How to implement Caffe LSTM layer by MKL-DNN

Caffe LSTM implement is different from other framwork.
I want use MKL-DNN implement a Caffe LSTM.
Could give me some suggestions? Thanks!!

Hi @xiaoweiChen,
It looks like IntelCaffe LSTM implementation rewrites LSTM cell in terms of inner product and elementwise operations, which should call corresponding DNNL layers. If you want to make more efficient implementation you will need to implement a new DNNL-based version of the layer using DNNL's RNN primitive. Do you see any differences in the cell definition or parameters?
An important thing to keep in mind is that IntelCaffe relies on Intel MKL-DNN v0.21, which is not supported anymore. Transition to v1.2 will require significant changes to the codebase.
If you have a Caffe model and looking for efficient inference deployment tool you may want to try OpenVINO, an Intel supported engine for deep learning model deployment. OpenVINO can run Caffe models with similar or better performance than IntelCaffe and thread safe.
OpenVINO langing page: https://github.com/opencv/dldt
Github: https://github.com/opencv/dldt

Thanks for your reply @vpirogov
In fact, I am a OpenVINO player. I have few Caffe models which have LSTM layer(s). While, OpenVINO is not support Caffe LSTM layer. So, I add the Caffe LSTM layer as the custom into OpenVINO(Base OpenVINO 2019 R3.1), also some modify for model-optimizer and inference-engine and make this custom layer work well.
I implement the Scale, Eltwise(SUM), FullyConnected, Slice by DNNL 1.2 in OpenVINO Source code. The performace is good for my team.
But, I also want to implement the LSTM cell by DNNL 1.2. May this can improve the Character Recognition.
And I have read RNN primitive, while I can not understand how to give the param to lstm_forward::desc(may caffe implement is different from other framework).
So, I open this issue, and want to get a lstm sample for caffe, or let me know how to give the param to lstm_forward::desc in Caffe LSTM layer.
Thanks again!

At first glance LSTM in Caffe matches the definition we have. @xiaoweiChen, could you please clarify what exactly you mean by saying "let me know how to give the param to lstm_forward::desc in Caffe LSTM layer"?
We recently added simple primitive examples for each primitive, and LSTM is one of them. Did you have a change to look at it?

Thanks for your work @emfomenk . I look your example just now. But, I also confusion for something.
Preparatory working....

The classical LSTM data flow is like https://arxiv.org/pdf/1906.06440.pdf (Figure 3). (Sorry, I not upload the image from local at present.)

The Caffe implement like below(without static input):

caffe LSTM unit implement:

const int num = bottom[0]->shape(1);
  const int x_dim = hidden_dim_ * 4;
  const Dtype* C_prev = bottom[0]->cpu_data();
  const Dtype* X = bottom[1]->cpu_data();
  const Dtype* cont = bottom[2]->cpu_data();
  Dtype* C = top[0]->mutable_cpu_data();
  Dtype* H = top[1]->mutable_cpu_data();
  for (int n = 0; n < num; ++n) {
    for (int d = 0; d < hidden_dim_; ++d) {
      const Dtype i = sigmoid(X[d]);
      const Dtype f = (*cont == 0) ? 0 :
          (*cont * sigmoid(X[1 * hidden_dim_ + d]));
      const Dtype o = sigmoid(X[2 * hidden_dim_ + d]);
      const Dtype g = tanh(X[3 * hidden_dim_ + d]);
      const Dtype c_prev = C_prev[d];
      const Dtype c = f * c_prev + i * g;
      C[d] = c;
      const Dtype tanh_c = tanh(c);
      H[d] = o * tanh_c;
    }
    C_prev += hidden_dim_;
    X += x_dim;
    C += hidden_dim_;
    H += hidden_dim_;
    ++cont;
  }

lstm_forward::desc constructor like below:

lstm_forward::desc(
    aprop,
    direction,
    src_layer_desc,
    src_iter_h_desc,
    src_iter_c_desc,
    weights_layer_desc,
    weights_iter_desc,
    bias_desc,
    dst_layer_desc,
    dst_iter_h_desc,
    dst_iter_c_desc);

OK, preparatory work completed.
I will give my opinion, may my opinion is not right.
Confusion 1:
lstm_forward::desc can implement Caffe LSTMUnit?
My opinion:
I think, I can not implement. Because Caffe LSTMUnit without weight, the weight process in InnerProduct Layer. It differenet from the classical LSTM implement. So, I can't replace LSTMUnit by this lstm_forward.
Confusion 2:
Base on Confusion 1, implement LSTM need re-implement, without Scale, InnerProduct, Eltwise... layers, just use many lstm_forward::desc constitute the LSTM layer.
Then, I meet the confusion...
How to process cont input(which you can see the caffe implement image, it shape is (16,3))? cont is match the src_iter_c_desc in lstm_forward::desc?  cont  is used in caffe implement, I can not ignore it.
In primitive example for LSTM and example rnn-inference-fp32 src_iter_c_desc is memory::desc().
So I don't know how to process cont for lstm_forward::desc.
My opinion:
This is my main question for this issue.
Hopeful, I explain my confusions clearly.
Looking for your reply! Thanks.

Hi @xiaoweiChen,
Sorry for the delay with respond and thanks for the nice explanation and pictures!
To be aligned on terms, let me try to map Caffe variables on DNNL names.

Caffe
DNNL
Comment

x [T, N, SLC]
src_layer [T, N, SLC]

c_0 [1, N, DIC]
src_iter_c [L, D, N, DIC]

c_$i, i = 1 .. 15
invisible intermediate cell states

c_16 [1, N, DIC]
dst_iter_c [L, D, N, DIC]

h_0 [1, N, SIC]
src_iter [L, D, N, SIC]

h_$i, i = 1 .. 15
invisible intermediate hidden states

h_16 [1, N, DIC]
dst_iter [L, D, N, DIC]

h [T, N, DIC]
dst_layer [ T, N, DIC ]
concat of h_1 ... h_16

Parameters table:
L = 1 # number of LSTM layers
D = 1 # number of directions
T = 16 # time stamps
N = 3 # batch
SLC = 4096
DIC = 256

Now, coming back to your questions:

I think, I can not implement. Because Caffe LSTMUnit without weight, the weight process in InnerProduct Layer. It differenet from the classical LSTM implement. So, I can't replace LSTMUnit by DNNL lstm_forward.

The weights that are processed in InnerProduct here, is what you showed on the first picture x_t * {W_o, W_f, W_c, W_i}, and will be processed by LSTM primitive.
The InnerProduct here, corresponds to h_t-1 * {R_o, R_f, R_c, R_i}, and will be handled by LSTM primitive as well.

Base on Confusion 1, the LSTM need re-implement by DNNL lstm_forward, and without Scale, InnerProduct, Eltwise... balabala layers, just use many lstm_forward::desc constitute the whole LSTM layer.
Then, I meet the confusion...
How to process cont input(which you can see the caffe implement image, it shape is (16,3))?
cont is match the src_iter_c_desc in lstm_forward::desc?
cont is used in caffe implement, I can not ignore it.

cont is an implementation detail of Caffe. It equals 0 for the first time stamp and 1 for all the other time stamps (see here). LSTM primitive will take care of it on its own. You should not worry about cont tensor from Caffe.

In primitive example for LSTM and example rnn-inference-fp32 src_iter_c_desc is memory::desc().
So I don't know how to process cont for lstm_forward::desc.

As I mentioned above, cont should not appear when you use DNNL LSTM primitive.  The tensors you need to use are the following:
// x
memory_desc src_layer_md({T, N, SLC}, dt::f32, format_tag::tnc);
memory src_layer(src_layer_md, engine, x);

// c_0
memory_desc src_iter_c_md({L, D, N, DIC}, dt::f32, format_tag::ldnc);
memory src_iter_c(src_iter_c_md, engine, c_0);

// c_T
memory_desc dst_iter_c_md({L, D, N, DIC}, dt::f32, format_tag::ldnc);
memory dst_iter_c(dst_iter_c_md, engine, c_T);

// h_0
memory_desc src_iter_md({L, D, N, SIC}, dt::f32, format_tag::ldnc);
memory src_iter(src_iter_md, engine, h_0);

// h_16
memory_desc dst_iter_md({L, D, N, DIC}, dt::f32, format_tag::ldnc);
memory dst_iter(dst_iter_md, engine, h_16);

// h
memory_desc dst_layer_md({T, N, DIC}, dt::f32, format_tag::tnc);
memory dst_layer(dst_layer_md, engine, h);

// W_xc
memory_desc weights_layer_md({L, D, SLC, 4, DIC}, dt::f32, format_tag::ldigo);
memory weights_layer(weights_layer_md, engine, W_xc);

// b_c
memory_desc bias_md({L, D, 4, DIC}, dt::f32, format_tag::ldgo);
memory bias(bias_md, engine, b_c);

// W_hc
memory_desc weights_iter_md({L, D, SIC, 4, DIC}, dt::f32, format_tag::ldigo);
memory weights_iter(weights_iter_md, engine, W_hc);

lstm_forward::desc(forward_inference, unidirectional_left2right,
    src_layer_md, src_iter_md, src_iter_c_md,
    weights_layer_md, weights_iter_md, bias_md,
    dst_layer_md, dst_iter_md, dst_iter_c_md);

Thanks for your word @emfomenk
I known how to map the caffe lstm to dnnl lstm interface.
While, base on your words, I have a question:
as I know:

L = 1 # number of LSTM layers
D = 1 # number of directions
T = 16 # time stamps
N = 3 # batch
SLC = 4096
DIC = 256

but...

// h_0
memory_desc src_iter_md({L, D, N, SIC}, dt::f32, format_tag::ldnc);
and
// W_hc
memory_desc weights_iter_md({L, D, SIC, 4, DIC}, dt::f32, format_tag::ldigo);

I don't know the SIC value.
This is jut a mistake?
SIC->DIC or SIC->SLC?
And I try to resolve this by myself.
h_0: SIC->DIC
W_hc: SIC->SLC
and use c++ try to implement this lstm layer, and make code compile-well.
En... Unfortunately, I meet runtime error

terminate called after throwing an instance of 'dnnl::error'
what():  could not create a descriptor for an LSTM forward propagation primitive

My code like below, may there have some argument(s) for lstm_forward::primitive_desc is wrong...
 // DNNL implement
  using tag = dnnl::memory::format_tag;
  using dt = dnnl::memory::data_type;

  dnnl::engine eng(dnnl::engine::kind::cpu, 0);
  dnnl::stream s(eng);

  std::size_t L = 1; // number of LSTM layers
  std::size_t D = 1; // number of directions
  std::size_t T = x_dims.at(0); // time stamps
  std::size_t N = x_dims.at(1); // batch

  std::size_t SLC = x_dims.at(2);
  std::size_t DIC = h_dims.at(2);

   // void compile-time warning as error
  long int L_i = L;
  long int D_i = D;
  long int T_i = T;
  long int N_i = N;
  long int SLC_i = SLC;
  long int DIC_i = DIC;

  // x
  dnnl::memory::desc src_layer_md(
  { T_i, N_i, SLC_i }, dt::f32, tag::tnc);
  dnnl::memory src_layer(src_layer_md, eng, x->buffer());

  SizeVector inner_c_h_shape{ L, D, N, DIC };
  auto inner_c_h_desc = TensorDesc(Precision::FP32, inner_c_h_shape, ANY);

  // c_0
  auto c_0 = std::make_shared<TBlob<float>>(inner_c_h_desc);
  c_0->allocate();
  dnnl::memory::desc src_iter_c_md(
  { L_i, D_i, N_i, DIC_i }, dt::f32, tag::ldnc);
  dnnl::memory src_iter_c(src_iter_c_md, eng, c_0->buffer());
  memset(
      static_cast<void*>(c_0->buffer()),
      0,
      L*D*N* DIC * sizeof(float));

  // c_T
  auto c_T = std::make_shared<TBlob<float>>(inner_c_h_desc);
  c_T->allocate();
  dnnl::memory::desc dst_iter_c_md(
  { L_i, D_i, N_i, DIC_i }, dt::f32, tag::ldnc);
  dnnl::memory dst_iter_c(dst_iter_c_md, eng, c_T->buffer());
  memset(
      static_cast<void*>(c_T->buffer()),
      0,
      L*D*N* DIC * sizeof(float));

  // h_0
  auto h_0 = std::make_shared<TBlob<float>>(inner_c_h_desc);
  h_0->allocate();
  dnnl::memory::desc  src_iter_md(
  { L_i, D_i, N_i, DIC_i }, dt::f32, tag::ldnc);
  dnnl::memory src_iter(src_iter_md, eng, h_0->buffer());
  memset(
      static_cast<void*>(h_0->buffer()),
      0,
      L*D*N* DIC * sizeof(float));

  // h_16
  auto h_16 = std::make_shared<TBlob<float>>(inner_c_h_desc);
  h_16->allocate();
  dnnl::memory::desc  dst_iter_md(
  { L_i, D_i, N_i, DIC_i }, dt::f32, tag::ldnc);
  dnnl::memory dst_iter(dst_iter_md, eng, h_16->buffer());
  memset(
      static_cast<void*>(h_16->buffer()),
      0,
      L*D*N* DIC * sizeof(float));

  // h
  dnnl::memory::desc dst_layer_md(
  { T_i, N_i, DIC_i }, dt::f32, tag::tnc);
  dnnl::memory dst_layer(dst_layer_md, eng, h->buffer());

  // W_xc
  dnnl::memory::desc weights_layer_md(
  { L_i, D_i, SLC_i, 4, DIC_i }, dt::f32, tag::ldigo);
  dnnl::memory weights_layer(weights_layer_md, eng, this->W_xc->buffer());

  // b_c
  dnnl::memory::desc bias_md({ L_i, D_i, 4, DIC_i }, dt::f32, tag::ldgo);
  dnnl::memory bias(bias_md, eng, this->b_c->buffer());

  // W_hc
  dnnl::memory::desc weights_iter_md({ L_i, D_i, SLC_i, 4, DIC_i }, dt::f32, tag::ldigo);
  dnnl::memory weights_iter(weights_iter_md, eng, this->W_hc->buffer());

  auto lstm_desc =
      dnnl::lstm_forward::desc(
          dnnl::prop_kind::forward_inference,
          dnnl::rnn_direction::unidirectional_left2right,
          src_layer_md,
          src_iter_md,
          src_iter_c_md,
          weights_layer_md,
          weights_iter_md,
          bias_md,
          dst_layer_md,
          dst_iter_md,
          dst_iter_c_md);

  auto lstm_prim_desc =
      dnnl::lstm_forward::primitive_desc(
          lstm_desc,
          eng);

  dnnl::lstm_forward(lstm_prim_desc).execute(
      s,
      {
          { DNNL_ARG_SRC_LAYER, src_layer },
          { DNNL_ARG_WEIGHTS_LAYER, weights_layer },
          { DNNL_ARG_WEIGHTS_ITER, weights_iter },
          { DNNL_ARG_BIAS, bias },
          { DNNL_ARG_DST_LAYER, dst_layer },
          { DNNL_ARG_SRC_ITER_C, src_iter_c },
          { DNNL_ARG_DST_ITER_C, dst_iter_c },
          { DNNL_ARG_SRC_ITER, src_iter },
          { DNNL_ARG_DST_ITER, dst_iter },
      });

  s.wait();

In your case SIC == DIC and should be used for both h_0 and W_hc.

Thanks @emfomenk , SIC == DIC can resolve the above exception.
Now, I just make the layer as a program without OpenVINO IE framework. The program would crash at LSTM execute.
I try to debug this base on DNNLRelease source coda 1.2.1 version(on windows):
\src\cpu\gemm\f32\jit_avx_gemm_f32.cpp:2586
sgemm_nocopy_driver(transa, transb, myM, myN, myK, p_alpha, myA,
                        lda, myB, ldb, &myBeta, myC, ld, myBias, ws);

The second hit would lead program crash
The crash location is
\src\cpu\gemm\f32\jit_avx_gemm_f32.cpp:2313
ker_(m, n, k, alpha, a, lda, b, ldb, beta, c, ldc, bias, ws);

The local values is

I found the bias is an empty pointer and I can not debug into this line any more...
Well, show my program code here:

#include <vector>

#include "dnnl.hpp"

using SizeVector = std::vector<std::size_t>;

int main() {

    std::vector<std::size_t> x_dims {16, 3, 4096};
    auto x_size = 1;
    for (auto num : x_dims) {
        x_size *= num;
    }
    auto x = new float[x_size];

    std::vector<std::size_t> h_dims {16, 3, 256};
    auto h_size = 1;
    for (auto num : h_dims) {
        h_size *= num;
    }
    auto h = new float[h_size];

    // DNNL implement
    using tag = dnnl::memory::format_tag;
    using dt = dnnl::memory::data_type;

    dnnl::engine eng(dnnl::engine::kind::cpu, 0);
    dnnl::stream s(eng);

    std::size_t L = 1; // number of LSTM layers
    std::size_t D = 1; // number of directions
    std::size_t T = x_dims.at(0); // time stamps
    std::size_t N = x_dims.at(1); // batch

    std::size_t SLC = x_dims.at(2);
    std::size_t DIC = h_dims.at(2);
    auto SIC = DIC;

    // void compile-time warning as error
    long int L_i = L;
    long int D_i = D;
    long int T_i = T;
    long int N_i = N;
    long int SLC_i = SLC;
    long int DIC_i = DIC;
    long int SIC_i = SIC;

    // x
    dnnl::memory::desc src_layer_md({T_i, N_i, SLC_i}, dt::f32, tag::tnc);
    //dnnl::memory src_layer(src_layer_md, eng, x->buffer());
    dnnl::memory src_layer(src_layer_md, eng, x);

    SizeVector inner_c_h_shape {L, D, N, DIC};
    //auto inner_c_h_desc = TensorDesc(Precision::FP32, inner_c_h_shape, ANY);
    auto inner_c_h_size = 1;
    for (auto num : inner_c_h_shape) {
        inner_c_h_size *= num;
    }

    // c_0
    //auto c_0 = std::make_shared<TBlob<float>>(inner_c_h_desc);
    //c_0->allocate();
    auto c_0 = new float[inner_c_h_size];
    dnnl::memory::desc src_iter_c_md(
            {L_i, D_i, N_i, DIC_i}, dt::f32, tag::ldnc);
    dnnl::memory src_iter_c(src_iter_c_md, eng, c_0);
    memset(c_0, 0, inner_c_h_size * sizeof(float));

    // c_T
    //auto c_T = std::make_shared<TBlob<float>>(inner_c_h_desc);
    //c_T->allocate();
    auto c_T = new float[inner_c_h_size];
    dnnl::memory::desc dst_iter_c_md(
            {L_i, D_i, N_i, DIC_i}, dt::f32, tag::ldnc);
    dnnl::memory dst_iter_c(dst_iter_c_md, eng, c_T);
    memset(c_T, 0, inner_c_h_size * sizeof(float));

    // h_0
    //auto h_0 = std::make_shared<TBlob<float>>(inner_c_h_desc);
    //h_0->allocate();
    auto h_0 = new float[inner_c_h_size];
    dnnl::memory::desc src_iter_md({L_i, D_i, N_i, DIC_i}, dt::f32, tag::ldnc);
    dnnl::memory src_iter(src_iter_md, eng, h_0);
    memset(h_0, 0, inner_c_h_size * sizeof(float));

    // h_16
    //auto h_16 = std::make_shared<TBlob<float>>(inner_c_h_desc);
    //h_16->allocate();
    auto h_16 = new float[inner_c_h_size];
    dnnl::memory::desc dst_iter_md({L_i, D_i, N_i, DIC_i}, dt::f32, tag::ldnc);
    dnnl::memory dst_iter(dst_iter_md, eng, h_16);
    memset(h_16, 0, inner_c_h_size * sizeof(float));

    // h
    dnnl::memory::desc dst_layer_md({T_i, N_i, DIC_i}, dt::f32, tag::tnc);
    dnnl::memory dst_layer(dst_layer_md, eng, h);

    // W_xc
    SizeVector W_xc_shape {L, D, SLC, 4, DIC};
    auto W_xc_size = 1;
    for (auto num : W_xc_shape) {
        W_xc_size *= num;
    }
    auto W_xc = new float[W_xc_size];
    dnnl::memory::desc weights_layer_md(
            {L_i, D_i, SLC_i, 4, DIC_i}, dt::f32, tag::ldigo);
    dnnl::memory weights_layer(weights_layer_md, eng, W_xc);

    // b_c
    SizeVector bias_shape {L, D, 4, DIC};
    auto bias_size = 1;
    for (auto num : bias_shape) {
        bias_size *= num;
    }
    auto b_c = new float[bias_size];
    dnnl::memory::desc bias_md({L_i, D_i, 4, DIC_i}, dt::f32, tag::ldgo);
    dnnl::memory bias(bias_md, eng, b_c);

    // W_hc
    SizeVector W_hc_shape {L, D, SIC, 4, DIC};
    auto W_hc_size = 1;
    for (auto num : W_hc_shape) {
        W_hc_size *= num;
    }
    auto W_hc = new float[bias_size];
    dnnl::memory::desc weights_iter_md(
            {L_i, D_i, SIC_i, 4, DIC_i}, dt::f32, tag::ldigo);
    dnnl::memory weights_iter(weights_iter_md, eng, W_hc);

    auto lstm_desc = dnnl::lstm_forward::desc(
            dnnl::prop_kind::forward_inference,
            dnnl::rnn_direction::unidirectional_left2right, src_layer_md,
            src_iter_md, src_iter_c_md, weights_layer_md, weights_iter_md,
            bias_md, dst_layer_md, dst_iter_md, dst_iter_c_md);

    auto lstm_prim_desc = dnnl::lstm_forward::primitive_desc(lstm_desc, eng);

    dnnl::lstm_forward(lstm_prim_desc)
            .execute(s,
                    {
                            {DNNL_ARG_SRC_LAYER, src_layer},
                            {DNNL_ARG_WEIGHTS_LAYER, weights_layer},
                            {DNNL_ARG_WEIGHTS_ITER, weights_iter},
                            {DNNL_ARG_BIAS, bias},
                            {DNNL_ARG_DST_LAYER, dst_layer},
                            {DNNL_ARG_SRC_ITER_C, src_iter_c},
                            {DNNL_ARG_DST_ITER_C, dst_iter_c},
                            {DNNL_ARG_SRC_ITER, src_iter},
                            {DNNL_ARG_DST_ITER, dst_iter},
                    });

    s.wait();

    return 0;
}

Because this is a small program for test, so I not use delete to release the new operator allocated memory.

En... auto W_hc = new float[bias_size]; to auto W_hc = new float[W_hc_size]; would resolve the crash.
Now, the test program running well. And Tomorrow I will test this under OpenVINO IE farmework.
If everything is OK, I will show the layer benchmark data here ;)

Glad to head you resolved the issues :)

hi @emfomenk ,
I use the test program code into framework, but get the wrong result.
I move my caffe LSTM implement from framework(also use DNNL to implement).
I shared my test code and implment code here:
xiaowei.tar.gz
Two questions here:

When use the std::fill to fill the input data and weight/bais data(you can see them in test code file), the test is passed. But, I don't think the result is right.
My output like below:

[PASSED] caffe result is little difference as native result!
caffe_h[0] : 0.761594 vs. native_h[0] : 0.761594
caffe_h[1] : 0.761594 vs. native_h[1] : 0.761594
caffe_h[2] : 0.761594 vs. native_h[2] : 0.761594
caffe_h[3] : 0.761594 vs. native_h[3] : 0.761594
caffe_h[4] : 0.761594 vs. native_h[4] : 0.761594
caffe_h[5] : 0.761594 vs. native_h[5] : 0.761594
caffe_h[6] : 0.761594 vs. native_h[6] : 0.761594
caffe_h[7] : 0.761594 vs. native_h[7] : 0.761594
caffe_h[8] : 0.761594 vs. native_h[8] : 0.761594
caffe_h[9] : 0.761594 vs. native_h[9] : 0.761594

caffe cost : 317.369 ms
native cost : 20.844 ms

E:\openSourceProjecrts\mirrors-MKL-DNN-v1.3\MKL-DNN\build-windows\xiaowei\Debug\LSTM-test.exe (进程 20092)已退出，代码为 0。
按任意键关闭此窗口. . .

When use the random number to fill the input data and weight/bais data, the test is failed.
My output like below:

caffe_h[0] : -0.761594 vs. native_h[0] : 0

E:\openSourceProjecrts\mirrors-MKL-DNN-v1.3\MKL-DNN\build-windows\xiaowei\Debug\LSTM-test.exe (进程 2036)已退出，代码为 -1。
按任意键关闭此窗口. . .

You can see the LSTM-test.cpp:58, there is a #if 1, make this to #if 0 can switch to random number mode.
If you have time, please help me. I don't know where is wrong.
Thanks!!!

I will close this issue. The last question is a precision question, may.
I find the way to resolve this question. and make the result same with my caffe lstm implementation.
I do some modify in jit_uni_rnn_common_postgemm.hpp, this part dnnl use 'kernel_' to compute the lstm-cell, and this use assembly with xbyak. I am not familar with xbyak, I can not dump the optimization code. So I have no idea for which part is wrong in assembly.
Well, I paste the modify code here.
jit_uni_rnn_common_postgemm.hpp
...
template <typename src_data_t, typename acc_data_t, typename scratch_data_t>
    rnn_postgemm_sig(execute_fwd) {
        using namespace rnn_utils;
        rnn_utils::ws_gates_aoc<src_data_t> ws_gates(rnn, ws_gates_);
        rnn_utils::ws_gates_aoc<scratch_data_t> scratch_gates(
                rnn, scratch_gates_);
        rnn_utils::weights_peephole_aoc_t<const float> weights_peephole(
                rnn, weights_peephole_);
        rnn_utils::bias_aoc_t bias(rnn, bias_);
        auto src_iter_ld = rnn.src_iter_ld(cell_position);
        auto src_iter_c_ld = rnn.src_iter_c_ld(cell_position);
        auto dst_iter_c_ld = rnn.dst_iter_c_ld(cell_position);
        auto dst_ld = rnn.dst_ld(cell_position);
        auto dst_copy_ld = rnn.dst_copy_ld(cell_position);
        rnn_utils::ws_states_aoc<src_data_t> states_t_l(
                rnn, states_t_l_, dst_ld);
        rnn_utils::ws_states_aoc<src_data_t> states_t_l_copy(
                rnn, states_t_l_copy_, dst_copy_ld);
        rnn_utils::ws_states_aoc<const src_data_t> states_tm1_l(
                rnn, states_tm1_l_, src_iter_ld);
        rnn_utils::ws_states_aoc<float> c_states_t_l(
                rnn, c_states_t_l_, dst_iter_c_ld);
        rnn_utils::ws_states_aoc<const float> c_states_tm1_l(
                rnn, c_states_tm1_l_, src_iter_c_ld);
        rnn_utils::ws_gates_aoc<scratch_data_t> scratch_cell(
                rnn, scratch_cell_);
        utils::array_offset_calculator<src_data_t, 2> ws_Wh_b(
                ws_grid_, rnn.mb, rnn.dic);

        static std::atomic_int cont = {0};

        auto caffe_lstm_cell = [](
         int dic,
         int cont,
         void *param2_,
         const void *param3_,
         void *param4_,
         const void *param6_,
         void *param7_) {

         float *X = static_cast<float *>(param2_);
         const float *biases = static_cast<const float *>(param3_);
         float *h = static_cast<float *>(param4_);
         const float *pre_c = static_cast<const float *>(param6_);
         float *c = static_cast<float *>(param7_);

         auto sigmoid = [](float x) {
           return 1.f / (1.f + expf(-x));
         };

         auto i_index_offset = 0 * dic;
         auto f_index_offset = 1 * dic;
         auto o_index_offset = 2 * dic;
         auto g_index_offset = 3 * dic;

         for (auto index = 0; index < dic; ++index) {
             auto i_index = i_index_offset + index;
             auto f_index = f_index_offset + index;
             auto o_index = o_index_offset + index;
             auto g_index = g_index_offset + index;

             auto i_gate_num = X[i_index] + biases[i_index];
             auto f_gate_num = X[f_index] + biases[f_index];
             auto o_gate_num = X[o_index] + biases[o_index];
             auto g_gate_num = X[g_index] + biases[g_index];

             auto i_gate = sigmoid(i_gate_num);
             auto f_gate = cont == 0 ? 0.f : sigmoid(f_gate_num);
             auto o_gate = sigmoid(o_gate_num);
             auto g = tanhf(g_gate_num);

             auto preC = pre_c[index];
             auto c_curr = f_gate * preC + i_gate * g;
             auto h_curr = o_gate * tanhf(c_curr);

             c[index] = c_curr;
             h[index] = h_curr;
         }
        };

        // Todo: add parallelization on dic for the batch 1 case
        // Assumption: the kernel runs a loop on dic elements
        parallel_nd(rnn.mb, [&](int i) {
            void *param1_ = &ws_gates(i, 0, 0); // RNN, LSTM, GRU
            void *param2_ = &scratch_gates(i, 0, 0); // RNN, LSTM, GRU
            const void *param3_ = &bias(0, 0); // RNN, LSTM, GRU
            void *param4_ = &states_t_l(i, 0); // RNN, LSTM, GRU
            void *param5_ = states_t_l_copy_
                    ? &states_t_l_copy(i, 0)
                    : states_t_l_copy_; // RNN, LSTM, GRU
            const void *param6_;
            void *param7_, *param8_;
            void *param9_ = nullptr;
            switch (pd_->cell_kind()) {
                case alg_kind::vanilla_lstm:
                    param6_ = &c_states_tm1_l(i, 0);
                    param7_ = &c_states_t_l(i, 0);
                    param8_ = (void *)&weights_peephole(0, 0);
                    break;
                case alg_kind::lbr_gru:
                    param6_ = &states_tm1_l(i, 0);
                    param7_ = &scratch_cell(i, 0, 0);
                    param8_ = &ws_Wh_b(i, 0);
                    break;
                case alg_kind::vanilla_gru:
                    param6_ = &states_tm1_l(i, 0);
                    param7_ = nullptr;
                    param8_ = nullptr;
                    break;
                default:
                    param6_ = nullptr;
                    param7_ = nullptr;
                    param8_ = nullptr;
                    break;
            }
            //kernel_(param1_, param2_, param3_, param4_, param5_, param6_,
            //        param7_, param8_, param9_);

            caffe_lstm_cell(
              rnn.dic,
              cont.load(),
              param2_,
              param3_,
              param4_,
              param6_,
              param7_);
        });

        cont.fetch_add(1, std::memory_order_relaxed);
    }
...

Thanks for the analysis! The only difference I see is:
auto f_gate = cont == 0 ? 0.f : sigmoid(f_gate_num);

while in oneDNN we have:
auto f_gate = sigmoid(f_gate_num);

According to oneDNN's RNN definition (link) the f_gate is always computed as sigmoid:

Maybe you could initialize the f-gate part of the first iteration of the input_layer data with -infinity, to make the corresponding sigmoid return 0.

Happy for seeing your reply and Thanks point the difference . @emfomenk
In fact, I don't want to modify the code in DNNL.
According your words "nitialize the f-gate part of the first iteration of the input_layer data with -infinity". And f_gate value equationauto f_gate_num = X[f_index] + biases[f_index];, I think I can modify the the biases to complete this. (I am not do it now.)
This would not modify anything for DNNL.
Emmm.... May, this is best way for me now.
In your opinion, this is a good idea? or, you can give some suggestions for doing this work?

You cannot modify bias, because it is the same for all timestamps: if you do so, you will get incorrect results for t=1 .. T.
I suggested to initialize h0, but now, looking at the formula I realized that it will be multiplied by U-matrix, which could lead to nans (as weights could be both positive and negative).
So, it seems this is the incompatibility that cannot be easily fixed. The only way to work-around this issue is to split the LSTM in to two steps: t = 0 and t = 1 ... T. For the first LSTM (with t=0) you can do the trick with the bias as you mentioned. For t = 1 .. T you just use the parameters from the model.


point websocketpp link to the actual fork you need

you'll need the unofficial fork because the official version is not supported

For what it's worth, I link that to the official one because it originally used it, and I do intend to eventually PR my fixes back to it. I do have the git submodule pointing to the correct one, but never really thought about the readme 😄 I think my fork is a bit outdated though. I should probably spend this weekend making a PR back upstream.


Add more HTTP/1.1 header parsing tests

Based on this article: https://portswigger.net/research/http-desync-attacks-request-smuggling-reborn
Just want to make sure we don't accept any of these malformed transfer-encoding headers, or that the well-formed headers don't hide that we are using chunked. Also worth following the link below that. I don't think there's any vulnerability here because Cowboy is super strict in its parsing.

Done. All good!


New Release

Could you make a new release, please? The changes are not yet available through the github actions market place. Thx :)

Done! I used to use @master so... enjoy v1.2.


Infinite Bits loop on error ?

Hello,
When issue arise, it tends to stick in infinite BITS loop:
Version 7.1.9

The way I found to break this loop without loosing current work was to restart BITS service 30mn later

Same problem there when trying to download drivers for Lenovo ThinkPad T14s Gen 2.

Unable to reproduce issue using details provided.


Require the file signer to specify the certificate chain

This is necessary because otherwise, a new cert chain would get generated in-memory on every startup.
Signed-off-by: Hayden Blauzvern<EMAIL_ADDRESS>
Summary

Release Note

Documentation

Codecov Report

Merging #137 (84a4b28) into main (e5cc4f2) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #137   +/-   ##
=======================================
  Coverage   53.24%   53.24%
=======================================
  Files          15       15
  Lines         678      678
=======================================
  Hits          361      361
  Misses        277      277
  Partials       40       40

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


Peer Review

Abstract and Introduction
Great topic! I think you can include further details of the study to make the title more descriptive, perhaps the model you utilized, the forecasting outcomes if you would like to in the title or the subtitle.
Your estimand is clearly stated in the beginning of the introduction. I think you can support some of your claims with literature, for example "..The 2018 election was chosen based on the high voter participation rate and clear partisan divisions.."
Data
I like your figure as it shows how the variables interact. You also have some insightful observations for the data distribution.
I think you can improve the section by including comments on data measurements. Since gender is a key variable, you may also comment on how non-binary gender data were considered.
Model and Results
Your model set up is detailed and you have explained the result thoroughly. If possible you might explore adjustments to the model to improve the R square value. Nonetheless I do think you have done well in this section.
Discussion
You have structured this section very well to discuss the outcomes and the limitations of the model. I think 5.2 can be substantiated with literature references, such as for "..the finding challenges some traditional assumptions about male voting patterns and suggests that gender may have a more nuanced impact on political preferences than previously assumed." A reader not very informed on the subject might want to know what the traditional assumptions are.
Other in the Repo
Your testing is thorough and your LLM is well documented. You also included a data sheet. It looks well composed!
Thank you and I hope you do very well in this paper!

added support for my claims at introduction
added subtitle
added data measurements about how non-binary been considered, n/a been processed
result been changed
added support for my claims at discussion


fix(curriculum): remove the period at the end of sentence

…the end of the sentence #52839
Checklist:

[x] I have read and followed the contribution guidelines.
[x] I have read and followed the how to open a pull request guide.
[x] My pull request targets the main branch of freeCodeCamp.
[x] I have tested these changes either locally on my machine, or GitPod.

Closes #52839

Hi @Kashinathpat. Congrats on your first pull request (PR)! 🎉
Thank you for your contribution to the page! 👍
We are happy to accept these changes and look forward to future contributions. 📝

Hi @gikf ! 🌟
Thank you so much for the warm welcome! I'm thrilled to have made my first contribution, and I'm eager to keep contributing to this amazing project. Your encouragement means a lot, and I'm excited to be part of the community. Looking forward to more coding adventures together! 🚀😊


Separate backwards compatibility testing in CI workflow

In the ./gradlew check process the build tasks for backwards compatibility of the previous version of elasticsearch are run.  During CI runs it would be better to block this part of the workflow until after the more basic tests are run.

@adnapibar this is saying something else than https://github.com/opensearch-project/OpenSearch/issues/799

@adnapibar this is saying something else than #799, which one's right?

As the description of #799 says -  ./gradlew check only runs a small number of bwc checks (only against compatible versions that are not yet released).
Our main goal of bwc check is to run it against compatible released versions e.g. against Elasticsearch 7.10.2. This is only run as part of ./gradlew bwcTest . See https://github.com/opensearch-project/OpenSearch/blob/main/TESTING.md#testing-backwards-compatibility for more details.

So this issue as stated is incorrect and should be closed?

So this issue as stated is incorrect and should be closed?

Yes

Closing, thanks! @peternied lmk if I shouldn't have ;)

Thanks for cleaning this up @dblock  - deferring to the newer issue!


Unicode identifiers not supported

It seems to me the flow parser does not support unicode identifiers correctly.
I get Unexpected token ILLEGAL.
Here are some examples:
let ℮;
var ℘\u2118
var ℘;
var 𞸆_$
var \u{1EE06}_$
var 𞸊𞸋
var \u{1EE0A}\u{1EE0B}
var _𞸃
var _\u{1EE03}
var 𞸀
var \u{1EE00}
var _፩፪፫፬፭፮፯፰፱
var \u{41}BC;
var A\u{42}C;
var \u{41}\u{42}\u{43};
゛+ ゜

All these examples work in espree, esprima, babylon, acorn

Oh I see this is a dup of #521.
Sorry for the noise.


flutter run --watch option

Is there an existing issue for this?

[X] I have searched the existing issues
[X] I have read the guide to filing a bug

Use case
Not everyone uses official editors to develop flutter apps, or may use scripts or other configurations that can't access the watch functionality of the IDE.
build_runner, which is maintained by the dart/flutter team offers --watch as an option too, but for machine-generated code.
Proposal
implement a  flutter run --watch option as mentioned in #6199.

My thoughts on this feature are captured in https://github.com/flutter/flutter/issues/62679, however I will leave this open as I wouldn't say we definitely won't implement it.

I'd like to see it if improvements come through in the future! I'll keep open for now to avoid the extra triage work since this seems to be requested every now and then.

Hey awesome people. Is this still a going concern?

I was just getting started setting up my flutter dev env and was looking for existance of such a command

Looks like as if the community came up with a solution https://github.com/erickzanardo/dashmon

Looks like as if the community came up with a solution https://github.com/erickzanardo/dashmon

Just tried but it didn't work =/
Apparently it's a know issue https://github.com/erickzanardo/dashmon/issues/3

This is how I'm doing it for now
run_app() {
  cd ./apps/pdv || exit
  tmux new-session \;\
  send-keys 'flutter run --debug -d linux --pid-file=/tmp/appdev.pid --dart-define-from-file=env-debug.json' Enter \;\
  split-window -v \;\
  send-keys 'cd ../ && npx -y nodemon -e dart -x "cat /tmp/appdev.pid | xargs -r kill -USR1"' Enter \;\
  select-pane -t 0 \;
}


brainiak run in miniconda on Windows?

Will brainiak run on Windows within miniconda? The "Win" link in Method 1 Step 1 on http://brainiak.org/#get-started makes me think it does, but elsewhere the documentation states that windows is not supported. (I tried the installation and got a PackagesNotFoundError.)
Thanks!

@mihaic Indeed I got the same error when trying on Windows. Shall we consider changing the website? Is the channel specific to certain OS?

@jaetzel, this is an error in our documentation. Thanks for bringing it up. We do not provide Conda packages for Windows.


please activate travis

https://travis-ci.org/jtimberman/smartmontools-cookbook
thanks!

well, https://travis-ci.org/sous-chefs/smartmontools

done, as demonstrated in #10


seq-collections-refactor branch

Contains WIP SeqData and AlignedData  module and classes.
#1860

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Details

178 of 183 (97.27%)  changed or added relevant lines in 3 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.04%) to 91.966%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

src/cogent3/core/new_alignment.py
171
176
97.16%

Totals

Change from base Build<PHONE_NUMBER>:
0.04%

Covered Lines:
30313

Relevant Lines:
32961

💛  - Coveralls


create SingleBin and BinRangeSelect tools

This PR starts to address #262
It modifies the existing hubble:binselect and hubble:singlebinselect tools to optionally create highlighted ROIs (only one can be shown at a time), and exposes the user's selected values. The selections can be accesses at tool.selection which returns a dictionary.
A subset can also be returned.
A possible mode of operation would be to turn ROI's off and then work with the subset created by the subset_state property.

This should be generalized and moved to cosmicds. We might not necessarily use it immediately in hubbleds given the new sequencing, but it will be good functionality to have handy for the future.

I made of this in #270 and am closing this for now.


build(devtools-view): Node16 resolution with ESM support

AB#7223
Add ESM build and always build with module Node16.

.js extensions from ts2esm
Change no-internal-modules to use standard lint config, then disable specific other places as needed.
Regorganize test fully under src/test and extract mock to src/test/utils
Setup jest to run against pre-built files to workaround ts-jest limitation with .cts files
Use Node16 resolution and build jest tests for CJS
ts-jest settings are commented out as a precaution.
Remove dead malformed interfaces (CommonInterfaces.ts)
Use a less standard build pattern with broad fluid-tsc to enable full module<->cjs switch support
Enable ability to change root package.json "type" with low effort as future investigations may be performed.
Besides changing that type it is recommended to update projects listed in lint config to reflect what is linted. test/screenshot will be inaccurately linted with "type": "common" at root but not a terrible hit.

Great changes. This was a gnarly one!


TG-1770: 1770 remove the fixed version packages when retrieving the icons

TG-1770

Also, can you sqash your commit into a single 🐛 fix app icons URL w/ version


Getting Started

Create Tutorial - Getting Started
Hello, I believe in the potential of the tool, I would like to help with the project in the documentation and create a brief use tutorial :smile:

We're working on creating a video tutorial actually. If you have any ideas we would be happy to see them:) Until then, we'll try to improve the gifs in the readme.

Ok! thank you! I'll try to help :)


Swift 5 support?

Is there any plan for an update to remove the public modifier warnings shown in Xcode 10.2.1?

@neilt I can try to do it quickly during the week

No problem, it is not a show stopper, but I had not seen any update in a while was just wondering.

@neilt Hello, sorry for a delay, I've been working on removing openssl and substituting it with own parser. I will push it today as a new release. Migration on Swift 5 will be included.

New version with swift 5+ support has been released

Thanks.


Addedd updateMult method

Allow update multiple elements using an array of commands.
Not sure if the name fits.
I thought about making update test if the first argument is an array, but IMO it's not a good idea.

Included in the 3.0.1 release. Just a little different then yours.

Thanks !
Just FYI, this breaks API compatibility

Oops. I have a patch for this, will upload today.

A patch has been released so it will support both single call update methods and the new multi call update methods

A patch has been released so it will support both single call update methods and the new multi call update methods

LGTM


Not compatible with Pencil version 3.0

This stencils package is amazing, but it has some compatible errors. When I am using this package with Pencil version 3.0, all UI Icons are gone. There must be some errors after several times adding components. All kinds of the elements can not add again. Other additional packages run correctly.
Version: 3.0.0-rc.1

https://github.com/evolus/pencil/issues/172
https://github.com/evolus/pencil/issues/178

Sorry for the dumb question and bringing it up here, but how do you install a stencil in the first place? I've been searching for a way and for some reason, I can't find any.

@sunxfancy Copy the content from the sub-folder found in the main install folder of the stencil up one level to the main install folder (C:\Users{username}.pencil\stencils\v1.1.0_648_1490656627447) and it will work. @toyongbanatero To install stencils click the manage collectiona icon right under the blue stencils bar and then click on the install from repository icno.

Closing as this is a duplicate (of #10 and #13) and is a bug in Pencil itself. From a quick test, it looks like the Pencil issue may have been fixed.


Fix Nested Form Cell Error Display

This PR fixes a small error where the nested form cell error is not actually shown.

@garrettmflynn Any way to write a test for this?


Feature Request: built-in ResizeBilinear Layer

I'm requesting a built-in Keras layer for resizing tensors.
Today:
I have to implement this layer either as a Lambda or a custom layer, both of which simply wrap tensorflow.compat.v1.image.resize_bilinear. This causes huge headaches for exporting to CoreML and TF-Lite. Existing layers like Upsampling2d and Reshape do not meet my needs, since I am downsampling to a fixed output size, so scaling by some factor does not work.
CoreML has a built-in resizeBilinear layer, but it is difficult to connect my custom Keras layer with that CoreML built-in.
Ideal solution:
There is a built-in Keras layer, I ask the CoreML team to use it, they implement a new layer conversion mapping, and I convert my model without digging into the guts of both Keras and CoreML.

tf.keras.layers.experimental.preprocessing.Resizing sounds like what you need. Looks like it was added in late 2019.


[question] what is the behavior  and return of function(NULL)

I think we can explain:

NOTE: If the parameter is NULL, the return is undefined.

Try to be consistent with openCypher.


Wrap RtcpReader in fresh objects per call to bind_rtcp_reader in default interceptors

As promised yesterday, a fix for #2

Well this seems to not break anything, but my read_rtcp call still falls on the floor so I'll open an issue for that.

https://github.com/webrtc-rs/webrtc/discussions/150

what about other interceptors? like NACK, TWCC?
I think they have the same issue, right?

what about other interceptors? like NACK, TWCC?
I think they have the same issue, right?
I will merge this PR first, you can continue to submit other PRs to fix others.

I think they're all in this PR, it's only in places where there was a parent_rtcp_reader - I thought there were more too, but grep disagreed!

yeah, you are right. :)
Thanks for the fix.


CSharp - Optimize Dictionaries

Do like Go, assume less than 100 elements.

Each of these two small fixes shaved about 5-8ms off each on my PC (more when run on the Github Runner).


Model Files only available on Baidu make them available in Releases.

I'm trying to get all the model files (.onnx) but I am unable to download them from the Baidu link, as I can not register an account on Baidu without a Chinese phone number. You can add them to the release you have made by editing the release and adding the onnx files. There is no limit to the size of the files attached to releases as this is how I release my software which is over 400mb.
Here is a example image from how I share large files on GitHub in Releases:

I'm trying to get all the model files (.onnx) but I am unable to download them from the Baidu link, as I can not register an account on Baidu without a Chinese phone number. You can add them to the release you have made by editing the release and adding the onnx files. There is no limit to the size of the files attached to releases as this is how I release my software which is over 400mb.
Here is a example image from how I share large files on GitHub in Releases:

Sorry for the late reply. I have followed your method. I am very happy to have learned a new skill. Thank you.

Thank you for all the model files.


The PMH Book and User Manual

Pictures of the UI portion of the application (not including the canvas) should not be included in the manual until after the UI Overhaul outlined in issue #318. Until then it is sufficient to describe what the application does, what AEGs are, and what all the different modes do via pictures of causes and effects from the canvas.
The book should be done in markdown. It will be built into a book using mdBook which is what the official rust and lean eBooks use.
Some important sections:

[ ] Chapter 1 Introduction to logic via AEGs
Contrary to the approach i take in my blog post, do not assume the user knows PC, introduce logic from first principals using AEGs.
[ ] Chapter 2 Propositional Calculus and its relation to AEGs
Talk about PC and how we may convert between it and AEGs for those who know propositional calculus.
[ ] Chapter 3 Introduce PMH and our features.

[ ] AEG construction in draw mode.
[ ] AEG theorem proving in proof mode.

We have a different repository for the book up and running, since mdbook involves Cargo, a different package manager. Check it out!
https://github.com/RAIRLab/EG-Theorem-Proving-in-PMH


vscode-extensions.ms-python.python: Fix packaging.

Motivation for this change
Previously, this extension used the JEDI backend for understanding
Python. Recently, Microsoft switched to using their Python Language
Server (https://github.com/Microsoft/python-language-server). The
extension attempts to download the latest version into the extension
folder, which failed for obvious reasons.
This change packages the language server into the extension, and
performs the necessary fixups and wrapping to make the dotnet binary
run.
Tested on Linux x86_64, but it should work on Darwin too.
Things done

[X] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[X] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[X] Tested execution of all binary files (usually in ./result/bin/)
[X] Determined the impact on package closure size (by running nix path-info -S before and after) Before: 2794344 After: 2869216
[X] Assured whether relevant documentation is up to date
[X] Fits CONTRIBUTING.md.

Haven't had any response in while, so I'm afraid it would customary for me to close this.
I you're still interested in contributing this change please open another pr with the requested changes.
Thanks :star2:


No license

The code doesn't specify any license which makes it hard to determine to what extent one is allowed to use the code.

License added.

Thanks!


Platform: update google project - to simplify infrastructure

Currently, the platform runs over 4 projects with a number of extra components. Based on what has been learnt from the genetics work recently, we want to migrate this to a single new combined project with a simplified infrastructure:
This means we need to:

[ ] elasticsearch image
[ ] elasticsearch instance groups (one per region)
[ ] elasticsearch internal load balance (one per region)
[ ] api instance templates
[ ] api instance groups (one per region)
[ ] nginx proxy docker image
[ ] nginx instance groups (one per region)
[ ] global load balancer
[ ] update external url api.opentargets.io

This has been done, new infrastructure deployed for 19.02 and old infrastructure cleanup is on going.


DAO Core events

Moving @EvgenRad81 proposal from learning center issues.

I want to add counter fo Core modules and events for any modification of module structure(adding, removing).

Added in 6024902c6863ec1cea6bc99b18a80cf9b1b786e9


"etree" is unknown import symbol (lxml)

Environment data

Language Server version: 2020.9.5
OS and version: macOS 10.15.6 (19G2021)
Python version (& distribution if applicable, e.g. Anaconda): python3.8 (venv)

Actual behaviour
I use the library lxml:4.5.2 and getting following error while importing, also autocomplete is not working. (https://lxml.de/)
from lxml import etree

"etree" is unknown import symbol Pylance (reportGeneralTypeIssues)

With language server set to jedi all is working fine.

lxml's etree module appears to be compiled, so we can't get any information out of it as there's no source to read (and we currently don't have a scraper to attempt to load the module and inspect it at runtime). At the moment, the best solution would be to have a set of stubs for lxml (as it appears to be almost entirely compiled code).
Note that reportGeneralTypeIssues only appears if your type checking mode is set to something other than "off", and off is the default for Pylance.

Installing the correct stubs fixed it for me, thanks for the hint.
https://github.com/lxml/lxml-stubs
pip install lxml-stubs

Pylance is able to make use of stubs packages, and there is one available for lxml: pip install lxml-stubs
It might work okay. I don't get any error on this simple code (in basic type checking mode):

Thanks for the tip; I didn't realize those had existed in typeshed but had been removed. We might be able to bundle them as we do with a few other commonly used libraries if lxml is common enough.

I fact I suspect pylance ability to import submodules is partially broken because I see similar errors when importing submodules from my own package. Neither mypy, pylint or flake8 complain about them but pylance wrongly produces them.

This issue is about lxml.etree not having any members; we have resolved this in the last release via special compiled coded stubs. @ssbarnea If you're having issues resolving imports, then please file a new issue. I'd be interested to see if downgrading Pylance fixes your issue.

This issue has been fixed in version 2021.4.0, which we've just released. You can find the changelog here: https://github.com/microsoft/pylance-release/blob/main/CHANGELOG.md#202140-7-april-2021


Cannot rescue the trolls and orcs on Frostwolf Starting island

@SavanticIO I'll pick this one up as well as some of the other Frostwolf stuff. I'll tag you in the others

@SavanticIO tomorrow I will get you properly assigned to all of these and add more details to the tickets. But consider them reserved for you. Sorry, hectic night

Updated description.


filter by gender

DOO-api gives now the gender of student. save that to db and enable the gender filter we once had

same as #321


Order imports update

Purpose
Based on lively conversation in Slack I've updated TSLint and all src files to accommodate our import preferences.
Enjoy!

Coverage remained the same at 27.008% when pulling cde9f8d0c82ca1ae6cdb03a643a41a734e929365 on rjstires:order-imports-update into 25ea708e5d0cfa205079a7862002a8536f05e0b7 on linode:develop.


fix(task): prevent spawn ENAMETOOLONG on Windows

As grunt.util.spawn is deprecated (see https://github.com/gruntjs/grunt-legacy-util/blob/621563dd8f6966af39546991bb8d710146430b79/README.md), we might as well switch to native node api. I've chosen child_process.fork() because it avoids looking for the correct node executable on windows (compare https://github.com/gruntjs/grunt-legacy-util/blob/621563dd8f6966af39546991bb8d710146430b79/index.js#L168-L173)
#Closes 161

I signed it!

Thanks a lot :octocat: I will test this out later today and then hopefully merge :)

@Dignifiedquire thanks for having a look at it. Do you work on a Linux machine? I'm currently working on a Windows machine. So maybe it makes sense to try it with Linux, too.
Regards,
Nicolas

@NicBright my main machine is OS X, but I have vms for windows and linux

Looks all good :)

Released in 0.12.1

Thanks for the quick review and integration. More than 20 developers in my client's organisation benefit from it :-)


Can't compile on React components on production with Vite but works on development

Command
build
Is this a regression?

[x] Yes, this behavior used to work in the previous version

The previous version in which this bug was not present was
webpack builder
Description
I can use react components okay with vite with ng serve, but it won't build
My org is a multi front-end team, and some teams work with react, others with angular, and is a big system, we can't migrate all react to angular.
During serve everything works flawlessly but during build it fails on compile time
Minimal Reproduction
[Create a new angular project with ssr
Add react, react dom ](https://github.com/alexandresebrao/ssr-19-angular-react)
This is a minimal repro.

I think if it doesn't have a problem during serve, it should work okay with build.

Just want to say thank you @clydin for the quick action!


Update auxlib, javalib, nativelib, nscplugin, ... to 0.4.3

Updates

org.scala-native:auxlib
org.scala-native:javalib
org.scala-native:nativelib
org.scala-native:nscplugin
org.scala-native:sbt-scala-native
org.scala-native:scalalib
org.scala-native:test-interface

from 0.4.0 to 0.4.3.
GitHub Release Notes - Version Diff
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Ignore future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "org.scala-native" } ]

labels: library-update, early-semver-minor, semver-spec-patch, commit-count:1

Hi.
It would be nice if this could be merged and then pushed to Maven since this allows building scala-native libraries with Scala 3. Thanks!

Yep, closing in favor of #477

Superseded by #498.


partials.py seems out of sync with main AMY

From discord user mathias:

Guess I found an error in partials.py:

import partials
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/mb/py_test/Amy/partials.py", line 40, in <module>
    def sequence(filename, max_len_s = 10, amp_floor=-30, hop_time=0.04, max_oscs=amy.OSCS, freq_res = 10, freq_drift=20, analysis_window = 100):
                                                                                ^^^^^^^^
AttributeError: module 'amy' has no attribute 'OSCS'

In amy.py the max_osc-Identifyer (now?) is called MAX_OSCS  After I changed the name of the referenced constant importing partials to use Loris as described in the docs works. Of course I am not entirely sure what's the intended solution - the change could be done to OSCS instead of MAX_OSCS in amy.py instead, of course. But to adjust partials.py accordingly appeared to be more logical to me...
There is another error in partials.py:

"bp0_target":amy.TARGET_AMP+amy.TARGET_LINEAR,
            "bp1_target":amy.TARGET_FREQ+amy.TARGET_LINEAR,

I deleted those lines because TARGET_AMP and TARGET_FREQ are not defined in amy.py
Plus during playback there is an error from amy.py, that bp0_target and bp1_target are no valid messages
(had to close the shell and only can report from the top of my head - sorry!)
Now the partials can be played back, but I did not hear anything and the next time there are errors:

partials.play(s, amp_ratio=2)
386.0
>>> partials.play(s, amp_ratio=2)
AMY queue is full
AMY queue is full

Working on this now.
I'm fixing #9 by forcing --prefix to be a local directory, which helps with an install (especially using virtualenv or conda or whatever)
Fixed up AMY_OSCS
The bp0 and bp1 have to be rewritten and using ControlCoefs, so we remove TARGET_AMP and TARGET_FREQ


fix(pagination): quick jumper not highlighted

fixed #37

Coverage remained the same at 52.163% when pulling a122238770743952a0f13f9bc8d9c17a9076c6be on HsuanXyz:fix-bug into 32234e45c4ab72768e474bc61ae0161a78e9bc26 on NG-ZORRO:master.


PSR errors and false Node status exception

Software
Version

PHP
8.2.10

Elasticsearch
8.10.2

Laravel
10

Laravel Scout
10.2

Elastic Scout Driver
3.1

Describe the bug
There are multiple incompatibilities and bugs on the library with latest version laravel.
I received these PSR Compatibility errors.
[2023-11-21 19:23:33] production.ERROR: Declaration of Elastic\Elasticsearch\Response\Elasticsearch::getHeader(string $name): array must be compatible with Psr\Http\Message\MessageInterface::getHeader($name) {"userId":1,"exception":"[object] (Symfony\\Component\\ErrorHandler\\Error\\FatalError(code: 0): Declaration of Elastic\\Elasticsearch\\Response\\Elasticsearch::getHeader(string $name): array must be compatible with Psr\\Http\\Message\\MessageInterface::getHeader($name) at /var/www/html/vendor/elasticsearch/elasticsearch/src/Traits/MessageResponseTrait.php:47)
Also
[2023-11-21 19:23:46] production.ERROR: No alive nodes. All the 1 nodes seem to be down. {"exception":"[object] (Elastic\Transport\Exception\NoNodeAvailableException(code: 0): No alive nodes. All the 1 nodes seem to be down. at /var/www/html/vendor/elastic/transport/src/NodePool/SimpleNodePool.php:77)
errors even though the node worked fine.
As a solution I replaced variable types from functions to resolve the first problem, and it worked. For example on getHeader(string $name) I replaced it with getHeader($name). It gave the same error on other functions and I replaced them as well.
Also for the second error, I have updated this section;
public function nextNode(): Node
    {
        $totNodes = count($this->nodes);
        $dead = 0;

        $next = $this->selector->nextNode();
        return $next;
        while ($dead < $totNodes) {
            $next = $this->selector->nextNode();
            if ($next->isAlive()) {
                return $next;
            }
            if ($this->resurrect->ping($next)) {
                $next->markAlive(true);
                return $next;
            }
            $dead++;
        }

        throw new NoNodeAvailableException(sprintf(
            'No alive nodes. All the %d nodes seem to be down.',
            $totNodes
        ));
    }

As you can see it will return the node without checking if it is up or down.
Doing so will allow it to access it. So there is a false exception going here.
Right now it is working fine.
As a summary there are two problems;

PSR Interface compatibility issues.
It gives false errors while checking if the node is alive or not.

Hey @ahmeteminkocal, I tried to reproduce the issue, but it seems to be working just fine with the latest versions of the packages. This is what I use on my local setup:
babenkoivan/elastic-adapter      v3.5.0  Adapter for official PHP Elasticsearch client
babenkoivan/elastic-client       v2.1.0  The official PHP Elasticsearch client integrated with Laravel
babenkoivan/elastic-scout-driver v3.1.0  Elasticsearch driver for Laravel Scout
elastic/transport                v8.8.0  HTTP transport PHP library for Elastic products
elasticsearch/elasticsearch      v8.11.0 PHP Client for Elasticsearch
guzzlehttp/guzzle                7.8.1   Guzzle is a PHP HTTP client library
guzzlehttp/promises              2.0.2   Guzzle promises library
guzzlehttp/psr7                  2.6.2   PSR-7 message implementation that also provides common utility methods
php-http/discovery               1.19.2  Finds and installs PSR-7, PSR-17, PSR-18 and HTTPlug implementations
php-http/httplug                 2.4.0   HTTPlug, the HTTP client abstraction for PHP
php-http/promise                 1.2.1   Promise used for asynchronous HTTP requests
psr/http-client                  1.0.3   Common interface for HTTP clients
psr/http-factory                 1.0.2   Common interfaces for PSR-7 HTTP message factories
psr/http-message                 2.0     Common interface for HTTP messages
psr/log                          3.0.0   Common interface for logging libraries
ralouphie/getallheaders          3.0.3   A polyfill for getallheaders.
symfony/deprecation-contracts    v3.4.0  A generic function and convention to trigger deprecation notices

Can you please ensure you use the latest versions of all dependencies (especially psr/http-message, which should be at least 2.0)?


Prevent tabs from automatically loaded on startup

Description
When you launch Ungoogled Chromium, a bunch of tabs are loaded automatically in the background, which causes performance issues. Only the currently active/selected tab should be loaded and the rest should be hibernated.
Who's implementing?

[ ] I'm willing to implement this feature myself

The problem
When you launch Ungoogled Chromium, a bunch of tabs are loaded automatically in the background, which causes performance issues. Only the currently active/selected tab should be loaded and the rest should be hibernated.
I have preloading disabled. But it still loads the tabs automatically in background. Vivaldi for example does this quite well, where it only loads one tab and that's it. Same thing in Firefox
Possible solutions
Copy the behaviour of Vivaldi or Firefox
Alternatives
No response
Additional context
No response

Same issue for me! It used to work fine until about 6-9 months ago I think. I've been using Slimjet mostly (chromium-based browser) and this is ruining it for me.


DO NOT MERGE: Redesigning settings surface

Current coverage is 61.04% (diff: 49.62%)

Merging #132 into master will decrease coverage by 0.87%

@@             master       #132   diff @@
==========================================
  Files            48         53     +5
  Lines          2227       2264    +37
  Methods           0          0
  Messages          0          0
  Branches        329        325     -4
==========================================
+ Hits           1379       1382     +3
- Misses          771        805    +34
  Partials         77         77

Powered by Codecov. Last update ca69095...5f519fb

cc @mziccard

What's the use of shouldAutoClose? Is it so that the close() method on the client close the channel/executor?

Yes.

A part of me believe that it will be easier for us and the user if close() unconditionally closes both the channel and the executor.

In the simplest case, the user just wants to write a CLI application. Whether the resources get closed doesn't greatly matter.
In the common case, the user wants to make a web server. The connection should stay around for the entire lifetime of the server anyway.
May the server wants to create fresh executor/channel/client for each request. But then you'd need to close out everything deterministically when you finish serving that request.

The last one here is very wasteful and slow. The whole point of channels in gRPC is to establish the connection and keep it open so that following calls are cheap.

If the executor and channel gets shared between multiple clients but the clients are used in the listed scenarios, the story of closing the resources stays the same.

Each channel has a cost. Each channel takes a non-trival amount of time to establish, and there are limits to sockets on the system. The optimal resource usage is a single channel per endpoint. If someone wants 1,000 client objects to a single service, they would still want only 1 channel to that service.

Are you thinking of any use case I'm missing?

We have two things we are trying to achieve simultaneously:

Don't leak channels when they're not being used any more
Try to keep the number of channels per service as close to 1 as possible

It's hard to actually achieve both of these, but we want to get as close as we can for now. I think (1) is the highest priority, and (2) is next. In the super simple case, we'll have to live with only achieving (1). In non-simple cases, we want to make it as easy as possible to achieve both (1) and (2).

@mziccard , I have settled on a new surface now - take a look and let me know what you think. I am going to start the process of cleaning it up for a merge - e.g. adding thread safety in certain places, adding/updating docs everywhere, filling out the unit tests - so don't pay attention to those gaps yet. Some key things to point out:

ConnectionSettings is now folded into InstantiatingChannelProvider. It was an extra level in the data model that didn't provide much help being independent.
The inner classes in ServiceApiSettings are moved to top-level public classes that can be used by users of GAX and generated GAPIC surfaces.
All of the provideChannelWithX and provideCredentialsWithX methods are gone; instead, users should use the defaultXBuilder methods that will be generated in GAPIC surfaces to get defaults, then set new values if necessary, then pass those to the appropriate component in the settings builder. I believe this makes everything more modular and clearer.
There is a new ProviderManager class to give first-class ability to lazily instantiate an executor and channel and then cache them for any additional gets.

I think this is ready for final review, PTAL

Maybe the server wants to create fresh executor/channel/client for each request. But then you'd need to close out everything deterministically when you finish serving that request.

The last one here is very wasteful and slow. The whole point of channels in gRPC is to establish the connection and keep it open so that following calls are cheap.

I completely agree. I guess my point is that even if someone wants to do this (highly not recommended), the cleanup still works if close unconditionally closes both the executor and the channel.

If the executor and channel gets shared between multiple clients but the clients are used in the listed scenarios, the story of closing the resources stays the same.

Each channel has a cost. Each channel takes a non-trival amount of time to establish, and there are limits to sockets on the system. The optimal resource usage is a single channel per endpoint. If someone wants 1,000 client objects to a single service, they would still want only 1 channel to that service.

Again, I completely agree that reusing channels is a good thing.
In my proposal, if someone wants 1,000 client objects to a single service, dealing with the resources is just as easy: Don't call client.close().
I think both use cases are easy:

Don't leak channels when they're not being used any more

 try (LibraryServiceApi libraryServiceApi = LibraryServiceApi.create()) {
   // ...
 }

Try to keep the number of channels per service as close to 1 as possible

ManagedChannel channel = getChannel();
Executor executor = getExecutor();
LibraryServiceSettings settings =
  LibraryServiceSettings.newBuilder().provide(channel, executor).build();
try {
  LibraryServiceApi api1 = LibraryServiceApi.create(settings);
  LibraryServiceApi api2 = LibraryServiceApi.create(settings);
} finally {
  channel.shutdown();
  executor.shutdown();
}

Compared to what it was before, we add 3 new lines to the program, but it makes close() a lot easier to reason about. Additionally, in the case there the channel and executor can live throughout the lifetime of the program (we think this is common), the shutdown code isn't necessary.
If you think this puts undue burden on the user though, I'm fine with this.

Again, I completely agree that reusing channels is a good thing.
In my proposal, if someone wants 1,000 client objects to a single service, dealing with the resources is just as easy: Don't call client.close().

I feel that this makes the client object surprising - it doesn't seem like if you close one client object, the other ones should be affected, but they all would be.

I think both use cases are easy:

Don't leak channels when they're not being used any more

 try (LibraryServiceApi libraryServiceApi = LibraryServiceApi.create()) {
   // ...
 }

Try to keep the number of channels per service as close to 1 as possible

ManagedChannel channel = getChannel();
Executor executor = getExecutor();
LibraryServiceSettings settings =
  LibraryServiceSettings.newBuilder().provide(channel, executor).build();
try {
  LibraryServiceApi api1 = LibraryServiceApi.create(settings);
  LibraryServiceApi api2 = LibraryServiceApi.create(settings);
} finally {
  channel.shutdown();
  executor.shutdown();
}

Compared to what it was before, we add 3 new lines to the program, but it makes close() a lot easier to reason about. Additionally, in the case there the channel and executor can live throughout the lifetime of the program (we think this is common), the shutdown code isn't necessary.

If you think this puts undue burden on the user though, I'm fine with the surface as it is.

The code would need to look more like the following - there are no global getChannel() / getExecutor() calls:
ScheduledExecutorService executor =
    LibraryServiceSettings.getDefaultExecutorProviderBuilder().build().getExecutor();
ManagedChannel channel =
    LibraryServiceSettings.getDefaultChannelProviderBuilder().build().getChannel(executor);
LibraryServiceSettings settings =
    LibraryServiceSettings.newBuilder()
        .setExecutorProvider(FixedExecutorProvider.create(executor))
        .setChannelProvider(FixedChannelProvider.create(channel))
        .build();
try {
  LibraryServiceApi api1 = LibraryServiceApi.create(settings);
  LibraryServiceApi api2 = LibraryServiceApi.create(settings);
} finally {
  channel.shutdown();
  executor.shutdown();
}

Here is some interesting prior art for our situation:
https://commons.apache.org/proper/commons-io/javadocs/api-1.4/org/apache/commons/io/input/TeeInputStream.html
When you construct a TeeInputStream, you can configure whether the "branched" OutputStream is closed or not when the TeeInputStream is closed. The key idea here is that streams that are passed to other streams are not always automatically closed when the containing stream is closed.
Another thing to consider is the executor. This is somewhat likely to live longer than the client objects, because many apps probably just make an executor that they pass around to a bunch of components of their app. Maybe they need to make a single simple API call in the course of a long-running backend process. This might be what they want:
 LibraryServiceSettings settings =
    LibraryServiceSettings.newBuilder()
        .setExecutorProvider(FixedExecutorProvider.create(APP_EXECUTOR))
        .build();
 try (LibraryServiceApi libraryServiceApi = LibraryServiceApi.create(settings)) {
   // ...
 }

The key idea with this example is that the lifetimes of executors and channels won't necessarily be anywhere near the same.

LGTM

@garrettjonesgoogle I think I understand the motivation now. Thank you for clarifying.


Reduce redundant warning messages in azure-cli credential provider

Changes
This warning is a bit noisy: in the happy path, we print it multiple times, and even if we don't use Azure CLI auth, we may print it anyways. Also, it will be printed always when using account-level authentication.
This PR reduces the noise for this warning by only printing it when we are certainly going to use Azure CLI auth, authenticating to a workspace, and can't get the subscription ID (e.g. because it is not configured).
Tests

[x] Ran locally: with accounts client, no warning is printed, and with workspace client, only one warning is printed.

Codecov Report
Attention: 1 lines in your changes are missing coverage. Please review.

Files
Coverage Δ

databricks/sdk/core.py
81.32% <83.33%> (+0.83%)
:arrow_up:

... and 1 file with indirect coverage changes
:loudspeaker: Thoughts on this report? Let us know!.

Related PR (which will have a conflict with this)
https://github.com/databricks/databricks-sdk-py/pull/409/files
I was trying to remember which stage and credentials I used in my testing. Since you just tested it, can you confirm that this doesn't happen in production?

@hectorcast-db I saw that PR which inspired me to do this one instead. After this change, this warning message will still be printed in production for users who don't specify the workspace resource ID. It can work if the workspace is in the same tenant as their current subscription, but for users who use multiple tenants, it can still be an issue, so I think the warning is well-placed to provide guidance for these otherwise inscrutable errors.


DIM2DataCite.xsl: update XML namespace to DataCite kernel version 4

Description
DIM2DataCite.xsl uses schema elements, e.g. givenName and familyName, that were added in DataCite kernel version 4 (http://schema.datacite.org/meta/kernel-4/metadata.xsd).
Currently, the XSL stylesheet DIM2DataCite.xsl refers to kernel version 3 (http://schema.datacite.org/meta/kernel-3/metadata.xsd) that does not contain givenName and familyName.

@abollini this PR should be back-ported to DSC7 (branch https://github.com/4Science/DSpace/tree/dspace-cris-2023_02_x) as well.

hi @saschaszott
thanks for proposing that. DSpace-CRIS is now using its internal export framework to generate datacite metadata both for DOI minting that just export of an item in the datacite metadata format. This change is not longer needed. We should remove the xsl completely but we are just leaving it unmaintained for who are still using it


HTTP 403 Forbidden when trying to save a topology

When I try to save a view I get error 403.

Solved by reinstalling the plugin.


Document list of standards used.

During our presentation for NERSC, someone asked for more detail about our claim to "leverage standards." We gave a decent summary off the cuff, but it would be good to have a list to point to.

Good question. I think that belongs in "contributing" / dev docs. The audience I have in mind for this page is someone who wants to know how "standard" the tiled service is, as opposed to the implementation/codebase.


Renaming default custom database and settings

What do you think about inserting the version number in the default name of the settings and the database file. Esp. the database has to be updated when OpenSlides is upgraded to a new version but users might not find the reason for database errors for they might not find the database file hidden in the deep of their home directory.

We decided to use migrations.
I let this pull request open as a reminder to add the migrations file before the release of the 2.0

We now have migrations, so this can be closed.


TMX format support

Is this parser on par with the latest version of Tiled?

It ought to work for Tiled v0.15 output, but I could have missed some things.  Are you seeing any issues?

I was just wondering... I was selecting a library from this page and i  gotten a little confused by the version compatibility thing i was unsure if it was format version or program version. Now I figured it, thanks. I am happily using TiledSharp right now: no problems so far. :+1:


macwalk and arpwalk not working in netdisco-docker

In the docker-compose variant macwalk and arpwalk fail with e.g.
 $ docker-compose run  netdisco-do macwalk
macwalk skipped: have not yet primed skiplist

forever, since there seems to be a disagreement on what workers->BACKEND should be called.
The default hostname creates entries like
netdisco=> select * from device_skip where device = '<IP_ADDRESS>' order by last_defer;
          backend          |     device      | actionset | deferrals |         last_defer
---------------------------+-----------------+-----------+-----------+----------------------------
 139f5ab43b0f.xyz.local | <IP_ADDRESS> | {}        |        16 | 2024-03-01 13:02:57.926219
 1ada5baef406.xyz.local | <IP_ADDRESS> | {}        |        16 | 2024-03-01 13:05:58.435278
 b40850a7940a.xyz.local | <IP_ADDRESS> | {}        |        16 | 2024-03-01 13:07:45.545747
 019e0ca649cf.xyz.local | <IP_ADDRESS> | {}        |        16 | 2024-03-01 13:09:06.01825

Setting hostname in docker-compose at least gives a static entry there:
$ grep -C 3 hostname docker-compose.yml
  netdisco-backend:
    image: netdisco/netdisco:latest-backend
    hostname: netdisco-backend

But still somehow netdisco-do thinks it has to use the local hostname that just exists during the run of the command:
 $ docker-compose run netdisco-do arpwalk -D
Starting netdisco_bp_netdisco-postgresql_1_773c1ba70409 ... done
Attempting to create directory /home/netdisco/perl5
...
[1] 2024-04-05 14:05:12  info resolving backend hostname...
[1] 2024-04-05 14:05:12 debug Backend identity set to ff19293cfd83.itoper.local
...
[1] 2024-04-05 14:05:13 debug arpwalk skipped: have not yet primed skiplist
[1] 2024-04-05 14:05:13  info arpwalk: finished at Fri Apr  5 14:05:13 2024
[1] 2024-04-05 14:05:13  info arpwalk: status defer: arpwalk skipped: have not yet primed skiplist

Somehow this backend identity stuff seems not to resolve on the backend side via the job queue?
setting deployment.yml:workers->{BACKEND} = the value from docker-compose.yml:netdisco-backend->{hostname} fixes it.
Should the default docker release somehow specify this, or is there a better way to find the backend name at runtime when netdisco-do is running on another host(name)?

Addressed device_skip bloat with static hostname: in docker-compose. The netdisco-do image now also sets netdisco-backend as hostname so that the skiplist check and thus ...walk commands work.


Add support for site scope to templates and content types

Add to the templates a scope attribute to track if the template comes from a RootSite of a Site Collection, or from a SubWeb.
Provide the same capability for the content types.

Done!


added remaining list count number at show more #6731

Coverage remained the same at 73.58% when pulling 5388cefb1174ad31f9d2114bf849fb12836d718b on gautampanday:show-more into 12590a72c9e645f250729d46799787af590d9a0d on openlabs:develop.

Coverage remained the same at 73.58% when pulling 5388cefb1174ad31f9d2114bf849fb12836d718b on gautampanday:show-more into 12590a72c9e645f250729d46799787af590d9a0d on openlabs:develop.

Coverage remained the same at 73.58% when pulling 702ff873bddd6b1b037fdec1f436f43ea2fed289 on gautampanday:show-more into 12590a72c9e645f250729d46799787af590d9a0d on openlabs:develop.

Coverage remained the same at 73.58% when pulling 702ff873bddd6b1b037fdec1f436f43ea2fed289 on gautampanday:show-more into 12590a72c9e645f250729d46799787af590d9a0d on openlabs:develop.

Coverage remained the same at 73.58% when pulling 6546de15bfd82a86be421a93dc23cd99020cf2bc on gautampanday:show-more into 12590a72c9e645f250729d46799787af590d9a0d on openlabs:develop.

Coverage remained the same at 73.58% when pulling 6546de15bfd82a86be421a93dc23cd99020cf2bc on gautampanday:show-more into 12590a72c9e645f250729d46799787af590d9a0d on openlabs:develop.

Coverage remained the same at 73.58% when pulling 1b9a1ab45b5048f174c8442c61395022f582667d on gautampanday:show-more into 12590a72c9e645f250729d46799787af590d9a0d on openlabs:develop.


Blacklist dependencies/versions

Hi! I would like to know if you are thinking on adding blacklist for certain dependencies?
Ie I may want to update manually compose and kotlin as I know they have a certain relationship, but update the rest automatically.
I would be happy to work on this if you think it would be a nice addition to the plugin

That would come down to configuring the version updates plugin correctly IMHO, one way to approach that is by taking the version in the version catalog and require that version for dependency updates.
An example of that is in this PR
Would that be sufficient for what you are looking for? The plugin could maybe do something to configure the version updates plugin to "pin" versions like that, but I'm not sure if that would improve things.

Thanks for the quick answer. What you mentioned is exactly the reason why I did not jump straight into the code, I did not know how to notify back about the "ignored" version.
The PR you linked is what we discussed on the team as an alternative, however, we also found nice the concept of "blacklist" that can be found in this other similar plugin: https://github.com/patrikerdes/gradle-use-latest-versions-plugin.
Anyway, to answer your question. Yes, that solution works for us, so, feel free to close this issue, and thanks a lot for taking the time to find an example about how to achieve this without forking your plugin.
BTW nice work, we discovered this plugin a few weeks ago and we are loving it ❤

Thanks ❤ I'll leave this open, I think some mechanism of saying "hey, I'll upgrade this myself" would be useful, if the plugin also tells you that an update is available. That could be a simple list of libraries and plugins you want to pin to the version specified in the catalog for example.

Just in case someone gets here trying to solve this same problem. This is the code I ended up writing to ignore the dependencies defined on the ignoredGroups list.
def ignoredGroups= ["org.jetbrains.kotlin", "androidx.compose.ui", "com.squareup.moshi"]

dependencyUpdates {
    resolutionStrategy {
        componentSelection {
            all {
                if (isNonStable(it.candidate.version) && !isNonStable(it.currentVersion)) {
                    reject('Release candidate')
                }
                if (ignoredGroups.contains(it.candidate.group)) {
                    reject("Pinned dependency")
                }
            }
        }
    }
}

I just stumbled upon a similar problem where i would like to specify a version that should not be updated automatically. I would prefer denoting this in the .toml file directly using the strictly keyword: https://docs.gradle.org/current/userguide/rich_versions.html
It seems like this is not supported by this plugin. Example:
nebula-release = { strictly = "14.0.3" }
[plugins]
nebula-release = { id = "nebula.release", version.ref = "nebula-release" }

results in the following error:
Expected class java.lang.String for key nebula-release, but was class java.util.LinkedHashMap
The gradle build task itself works just fine.

For anyone who wants to try, the 0.3.0-SNAPSHOT has the new pinning behaviour as documented in the README too now. If you have any feedback or see any issues, let me know, otherwise I'll release these changes somewhere this week.

Oh, I missed that! Thanks for the quick response!

Released in 0.3.0


Run sumstats sync (from submission app) from codon crontab

Currently the sync pipeline runs from gitlab: https://gitlab.ebi.ac.uk/gwas/sumstats-depo-to-harmonisation-sync/-/pipelines
but it often fails due to issues with the gitlab runner. It would possible (and probably better) to have this run as a bash script from the codon crontab.

Rewritten as a bash script to be run nightly from spotbot crontab on codon

modified crontab to enable bsub to be callable. Confirmed working.


Flaky test: GetMemoryTryGetArrayOfDisposedPoolThrows

This test fails occasionally with the following error:
Assert.Throws() Failure
Expected: typeof(System.InvalidOperationException)
Actual:   typeof(System.IO.FileLoadException): Could not load file or assembly 'System.Runtime.CompilerServices.Unsafe, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)
   at System.Memory`1.op_Implicit(Memory`1 memory)
   at Microsoft.Extensions.Internal.Test.DiagnosticMemoryPoolTests.<>c__DisplayClass7_0.<GetMemoryTryGetArrayOfDisposedPoolThrows>b__0()

CanWriteWithOffsetAndLenght failed again.

ReturnsCorrectCursor failed again.

CtorFindsFirstNonEmptySegment failed again.

AdvanceToEndThenPeekReturnsMinusOne failed again.

PeekWorkesWithEmptySegments failed again.

ReturnsCorrectCursor failed again.

ReturnsCorrectCursor failed again.

CanWriteNothingToBuffer failed again.
This comment was made automatically. If there is a problem contact @ryanbrandenburg.

CtorFindsFirstNonEmptySegment failed again.
This comment was made automatically. If there is a problem contact @ryanbrandenburg.

ReturnsCorrectCursor failed again.
This comment was made automatically. If there is a problem contact @ryanbrandenburg.

EmptyWriteDoesNotThrow failed again.
This comment was made automatically. If there is a problem contact @ryanbrandenburg.

PeekReturnsByteWithoutMoving failed again.
This comment was made automatically. If there is a problem contact @ryanbrandenburg.

AdvancingPastLengthThrows failed again.
This comment was made automatically. If there is a problem contact @ryanbrandenburg.

CtorFindsFirstNonEmptySegment failed again.
This comment was made automatically. If there is a problem contact @ryanbrandenburg.

ThrowsForInvalidParameters failed again.
This comment was made automatically. If there is a problem contact @ryanbrandenburg.

ReturnsCorrectCursor failed again.
This comment was made automatically. If there is a problem contact @ryanbrandenburg.

ReturnsCorrectCursor failed again.
This comment was made automatically. If there is a problem contact @ryanbrandenburg.

PeekReturnsByteWithoutMoving failed again.
This comment was made automatically. If there is a problem contact @ryanbrandenburg.

PeekGoesToEndIfAllEmptySegments failed again.
This comment was made automatically. If there is a problem contact @ryanbrandenburg.

AdvanceSingleBufferSkipsBytes failed again.
This comment was made automatically. If there is a problem contact @ryanbrandenburg.

ReturnsCorrectCursor failed again.
This comment was made automatically. If there is a problem contact @ryanbrandenburg.

AdvanceSingleBufferSkipsBytes failed again.
This comment was made automatically. If there is a problem contact @ryanbrandenburg.

ReaderIndexIsCorrect failed again.
This comment was made automatically. If there is a problem contact @ryanbrandenburg.

WorkesWithEmptyBuffer failed again.
This comment was made automatically. If there is a problem contact @ryanbrandenburg.

@pakrym maybe the same as https://github.com/aspnet/Home/issues/3355? This seems to be exploding all over the place.

Closing this seems like an infrastructure issue that went away.


tests in tests/common/util.rs fails on OpenBSD 7.5

The cargo test fails on OpenBSD due to not supporting xattrs.
failures:

---- common::util::tests::test_compare_xattrs stdout ----
thread 'common::util::tests::test_compare_xattrs' panicked at tests/common/util.rs:3628:56:
called `Result::unwrap()` on an `Err` value: Custom { kind: Unsupported, error: UnsupportedPlatformError }

failures:
    common::util::tests::test_compare_xattrs

There are several other failing tests on utils section on OpenBSD but this one is kind of more important because its in the tests/common/util.rs and when it breaks tests do not continue to by-util section.

Opening this to prevent creating a closely related case.
After remove the actual compare_xattrs based on PRs discussions (which looks like a good decision), OpenBSD cargo test for mv and cp also breaks because they do have tests based on that function. So I will send another PR to add the target_os is not openbsd condition to these tests too.


textStyleDisabled property

How do I set a different text style for disabled buttons?
Thank you for help :)

With the disabledStyle prop https://github.com/APSL/react-native-button#api


Updated sys-dm-os-wait-stats-transact-sql.md

Updated the above mentioned file to fix the Markdown display to use html tags instead. as it knocked everything else out of line.

@dacpac : Thanks for your contribution! The author, @MashaMSFT, has been notified to review your proposed change.

Hi @dacpac, thank you for your contribution! This PR is superseded by 7631. Once the change goes live, I will notify you and close out this PR. Thanks again!

Hi @dacpac, thanks again for help with this! It's now live: https://docs.microsoft.com/en-us/sql/relational-databases/system-dynamic-management-views/sys-dm-os-wait-stats-transact-sql?view=sql-server-2017. I'll close out this PR,  but please let me know if I can be of further assistance :) Thanks!


Define table data structure

Define the table schema required to easily load all types of randomizers into a SQL database

Table schema was created and tables have been created on the PostgreSQL server. Some specialty tables will still need to be created ad hoc. Most table types from the reference material will fit into the schema, but there are a few cases where it will be more beneficial to spawn a unique table. One such case was keywords which -- while simplistic enough in design to fit with the core schema -- would have flooded the range table with 1000+ records.


Let the Chef dep in omnibus float to the latest

Omnibus should be able to run the very latest Chef since we're just using Chef here for the actual build of the application. None of these deps impact the artifact we build directly. In general we should pin omnibus deps to hold back changes to omnibus/omnibus-software, but everything should be able to build at the latest.

Remove the chef pin
Remove chef-cli dep entirely since we really just had this in here for chefspec
Add Test Kitchen dep since there's a kitchen config here, but not Test Kitchen
Cleanup the Test Kitchen config a little bit. Remove the chef version pin, the unused yum-epel dep, and old platform versions in the kitchen.yml
Move all the test aka non-build deps into the test group. There's rspec for the cookbooks here, which we probably want to get running in CI

Signed-off-by: Tim Smith<EMAIL_ADDRESS>

Adhoc: https://buildkite.com/chef/chef-chef-server-master-omnibus-adhoc/builds/2170


Updates to support Wagtail 5

This PR includes: https://github.com/torchbox/wagtail-footnotes/pull/48
Makes changes to:

Drop support for Django v4.0 and python 3.7

Adds testing to the test matrix for Wagtail 5.0 and Django 4.2

@zerolab This should be good to go now, what do you think please?


add keycloak - initial

This PR will add keycloak to RC4Community.

This is awesome.    @debdutdeb can you please update our keycloak server and let's get community.rocket.chat working with REAL AUTH.       (just can't wait to see the linkage with superprofile  - hint Ronlek ;-)  )

@abhinavkrin   please advise if this is good enough to merge for our upcoming Alumni Summit purpose

Please also make keycloak the default IDM provider (if possible) for RC4Community instead of Firebase.
I've checked out the ISC license.   Totally OK to include.   Thanks.

Please also make keycloak the default IDM provider (if possible) for RC4Community instead of Firebase.
I've checked out the ISC license. Totally OK to include. Thanks.

It's fine to merge this PR. The default IDM would be keycloak.
Currently.
The user will have the following attribute:

id
our custom access token
image
name
email

Please remove package-lock.json from the PR  (and perhaps add it to your .gitignore)  - thanks.

Please remove package-lock.json from the PR (and perhaps add it to your .gitignore) - thanks. @abhinavkrin

Removed package-lock.json @Sing-Li


Document why certain hardware metadata configuration is required

Followup to https://github.com/tinkerbell/cluster-api-provider-tink/pull/51#discussion_r612724952
Clarify to users why certain hardware metadata is required

part is to satisfy cloud-init requirements for an ec2 style metadata source
part is to allow CAPI to know which device to target when creating the template/workflow

Hey @detiber, just wanted to check and see if this is still an outstanding issue, yes?


⚠ ECDC Shop has degraded performance

In 2882253, ECDC Shop (https://shop.memmingen-indians.de) experienced degraded performance:

HTTP code: 200
Response time: 6514 ms

Resolved: ECDC Shop performance has improved in 0070dd8 after 47 minutes.