Create small decoding graph, and other training script changes.

We are going to want a smaller-than-normal decoding graph for training purposes.
The creation process will be similar to the regular one except:

use a unigram language model (later we can use a pruned bigram)...
we'll need this to be loaded in the training program.  Let's copy-and-modify the mmi_bigram_train.py so it also loads this graph, we cann call it mmi_mbr_train.py, since we'll be using MMI + MBR.
I would like to use inner_labels='phones' when calling the composition operation that composes the ctc_topo with the (LG, or L*transcript), so we have access to the phone labels without repeats.  We don't need this on the bigram lf_mmi den graph.  We might need to modify the graph-creation code to use compose instead of intersect.
We will need to do a decoding, to get lattices, on each minibatch.  Later we'll create phone-level posteriors from those and subtract them from the phone-level posteriors from the numerator, and take the sum-of-absolute-values of the difference of sparse matrix to form the MBR part of the objective function.

Fine to split these up.

Fine to split these up.

I have been trying to understand the above description and the comment in https://github.com/k2-fsa/k2/issues/579
The following are some figures demonstrating my understanding. @danpovey Please correct me if they are wrong.
If they are correct, I will continue to write the training scripts.

Assume we have only two phones: a and b.
ctc_topo

unigram LM
0.1 and 0.2 are just some random numbers.

ctc_topo_P
ctc_topo_P = k2.compose(ctc_topo, P)

linear_fsa
Assume the transcript is "ba"

ctc_topo_P_linear_fsa
ctc_topo_P_linear_fsa = k2.compose(ctc_topo_P, linear_fsa, inner_labels='phones')

dense_fsa

num (the numerator)
num = k2.intersect_dense(ctc_topo_P_linear_fsa, dense_fsa, 10.0, seqframe_idx_name='seqframe_idx')

num_sparse
num_sparse = k2.create_sparse(rows=num.seqframe_idx,
                              cols=num.phones,
                              values=num.get_arc_post(True, True).exp(),
                              size=(4, 3),
                              min_col_index=0)
print(num_sparse)

tensor(indices=tensor([[0, 0, 1, 1, 1, 1, 2, 2, 2, 2],
                       [0, 2, 2, 0, 2, 1, 1, 1, 0, 1]]),
       values=tensor([0.0763, 0.9237, 0.0763, 0.0763, 0.5637, 0.2837, 0.6400,
                      0.0763, 0.0763, 0.2074]),
       size=(4, 3), nnz=10, dtype=torch.float64, layout=torch.sparse_coo)

den (the denominator)
den = k2.intersect_dense(ctc_topo_P, dense_fsa, 10, seqframe_idx_name='seqframe_idx')

I am stuck on the following description:

We will need to do a decoding, to get lattices, on each minibatch. Later we'll create phone-level posteriors from those and subtract them from the phone-level posteriors from the numerator

Currently, I only have phone-level posteriors from the numerator. How can I get the other part to make
the subtraction?

I have been trying to understand the above description and the comment in https://github.com/k2-fsa/k2/issues/579
The following are some figures demonstrating my understanding. @danpovey Please correct me if they are wrong.
If they are correct, I will continue to write the training scripts.

Assume we have only two phones: a and b.
ctc_topo

unigram LM
0.1 and 0.2 are just some random numbers.

ctc_topo_P
ctc_topo_P = k2.compose(ctc_topo, P)

linear_fsa
Assume the transcript is "ba"

ctc_topo_P_linear_fsa
ctc_topo_P_linear_fsa = k2.compose(ctc_topo_P, linear_fsa, inner_labels='phones')

dense_fsa

num (the numerator)
num = k2.intersect_dense(ctc_topo_P_linear_fsa, dense_fsa, 10.0, seqframe_idx_name='seqframe_idx')

num_sparse
num_sparse = k2.create_sparse(rows=num.seqframe_idx,
                              cols=num.phones,
                              values=num.get_arc_post(True, True).exp(),
                              size=(4, 3),
                              min_col_index=0)
print(num_sparse)

tensor(indices=tensor([[0, 0, 1, 1, 1, 1, 2, 2, 2, 2],
                       [0, 2, 2, 0, 2, 1, 1, 1, 0, 1]]),
       values=tensor([0.0763, 0.9237, 0.0763, 0.0763, 0.5637, 0.2837, 0.6400,
                      0.0763, 0.0763, 0.2074]),
       size=(4, 3), nnz=10, dtype=torch.float64, layout=torch.sparse_coo)

den (the denominator)
den = k2.intersect_dense(ctc_topo_P, dense_fsa, 10, seqframe_idx_name='seqframe_idx')

I am stuck on the following description:

We will need to do a decoding, to get lattices, on each minibatch. Later we'll create phone-level posteriors from those and subtract them from the phone-level posteriors from the numerator

Currently, I only have phone-level posteriors from the numerator. How can I get the other part to make
the subtraction?

I think the denominator graph should be the composition of ctc_topo_P  and unigram LM,  instead of dense_fsa. dense_fsa would be used to generate lattices of numberator and denominator.

I think the denominator graph should be the composition of ctc_topo_P  and unigram LM,  instead of dense_fsa. dense_fsa would be used to generate lattices of numberator and denominator.

By unigram LM I mean word-level.  It's the same process as creating the
decoding graph as in decode.py, except with a smaller LM.
The current LM would work too though.
On Sun, Jan 24, 2021 at 7:01 PM yaguang<EMAIL_ADDRESS>wrote:

I think the denominator graph should be the composition of ctc_topo_P and
unigram LM, instead of dense_fsa. dense_fsa would be used to generate
lattices of numberator and denominator.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/k2-fsa/snowfall/issues/80#issuecomment-766328391, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAZFLO63NYZVPLHTISYLVR3S3P4XLANCNFSM4WOAKNWQ
.

By unigram LM I mean word-level.  It's the same process as creating the
decoding graph as in decode.py, except with a smaller LM.
The current LM would work too though.
On Sun, Jan 24, 2021 at 7:01 PM yaguang<EMAIL_ADDRESS>wrote:

I think the denominator graph should be the composition of ctc_topo_P and
unigram LM, instead of dense_fsa. dense_fsa would be used to generate
lattices of numberator and denominator.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/k2-fsa/snowfall/issues/80#issuecomment-766328391, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAZFLO63NYZVPLHTISYLVR3S3P4XLANCNFSM4WOAKNWQ
.

... also, the process of getting lattices and turning them into sparse
phone-level posteriors is very similar for den and num, except for den, we
use
a graph constructed in a different way, and we need to use the pruned
intersection code as in decode.py.
On Sun, Jan 24, 2021 at 7:19 PM Daniel Povey<EMAIL_ADDRESS>wrote:

By unigram LM I mean word-level.  It's the same process as creating the
decoding graph as in decode.py, except with a smaller LM.
The current LM would work too though.
On Sun, Jan 24, 2021 at 7:01 PM yaguang<EMAIL_ADDRESS>wrote:

I think the denominator graph should be the composition of ctc_topo_P and
unigram LM, instead of dense_fsa. dense_fsa would be used to generate
lattices of numberator and denominator.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/k2-fsa/snowfall/issues/80#issuecomment-766328391,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAZFLO63NYZVPLHTISYLVR3S3P4XLANCNFSM4WOAKNWQ
.

... also, the process of getting lattices and turning them into sparse
phone-level posteriors is very similar for den and num, except for den, we
use
a graph constructed in a different way, and we need to use the pruned
intersection code as in decode.py.
On Sun, Jan 24, 2021 at 7:19 PM Daniel Povey<EMAIL_ADDRESS>wrote:

By unigram LM I mean word-level.  It's the same process as creating the
decoding graph as in decode.py, except with a smaller LM.
The current LM would work too though.
On Sun, Jan 24, 2021 at 7:01 PM yaguang<EMAIL_ADDRESS>wrote:

I think the denominator graph should be the composition of ctc_topo_P and
unigram LM, instead of dense_fsa. dense_fsa would be used to generate
lattices of numberator and denominator.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/k2-fsa/snowfall/issues/80#issuecomment-766328391,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAZFLO63NYZVPLHTISYLVR3S3P4XLANCNFSM4WOAKNWQ
.

Is the function of IntersectDensePruned  function similar like beam search decoding? If so, the lattice of denominator would be easy to generate.

Is the function of IntersectDensePruned  function similar like beam search decoding? If so, the lattice of denominator would be easy to generate.

Yes.
On Sun, Jan 24, 2021 at 7:27 PM yaguang<EMAIL_ADDRESS>wrote:

Is the function of IntersectDensePruned function similar like beam search
decoding? If so, the lattice of denominator would be easy to generate.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/k2-fsa/snowfall/issues/80#issuecomment-766332035, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAZFLO3S7XHWXKRKD4B6VKTS3P72RANCNFSM4WOAKNWQ
.

Yes.
On Sun, Jan 24, 2021 at 7:27 PM yaguang<EMAIL_ADDRESS>wrote:

Is the function of IntersectDensePruned function similar like beam search
decoding? If so, the lattice of denominator would be easy to generate.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/k2-fsa/snowfall/issues/80#issuecomment-766332035, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAZFLO3S7XHWXKRKD4B6VKTS3P72RANCNFSM4WOAKNWQ
.

Then if the modeling unit is phone, I thinks the pruned intersect of dense_fsa and  the denominator graph is the lattices of denominator. And the denominator graph is composed with CTC topo, lexicon graph and word level ungram lm.
And which is the difference of lattice and sparse phone-level posteriors? In my opinion, they are the same thing.

Then if the modeling unit is phone, I thinks the pruned intersect of dense_fsa and  the denominator graph is the lattices of denominator. And the denominator graph is composed with CTC topo, lexicon graph and word level ungram lm.
And which is the difference of lattice and sparse phone-level posteriors? In my opinion, they are the same thing.

the process of getting lattices and turning them into sparse
phone-level posteriors is very similar for den and num

We have num.phones while calling k2.create_sparse. But den  does not have phones attribute.
What should we pass to k2.create_sparse for den?

the process of getting lattices and turning them into sparse
phone-level posteriors is very similar for den and num

We have num.phones while calling k2.create_sparse. But den  does not have phones attribute.
What should we pass to k2.create_sparse for den?

By unigram LM I mean word-level.

Do we still need the bigram phone LM if a word level LM is present?

By unigram LM I mean word-level.

Do we still need the bigram phone LM if a word level LM is present?

the process of getting lattices and turning them into sparse
phone-level posteriors is very similar for den and num

We have num.phones while calling k2.create_sparse. But den does not have phones attribute.
What should we pass to k2.create_sparse for den?

Why den does not have phones attribute? The input of den and num should be the same.

By unigram LM I mean word-level.

Do we still need the bigram phone LM if a word level LM is present?

I think word level LM is better because it's more "end-to-end". But it depends on the size of lattices, which may exceed the memory of GPU.

the process of getting lattices and turning them into sparse
phone-level posteriors is very similar for den and num

We have num.phones while calling k2.create_sparse. But den does not have phones attribute.
What should we pass to k2.create_sparse for den?

Why den does not have phones attribute? The input of den and num should be the same.

By unigram LM I mean word-level.

Do we still need the bigram phone LM if a word level LM is present?

I think word level LM is better because it's more "end-to-end". But it depends on the size of lattices, which may exceed the memory of GPU.

the process of getting lattices and turning them into sparse
phone-level posteriors is very similar for den and num

We have num.phones while calling k2.create_sparse. But den does not have phones attribute.
What should we pass to k2.create_sparse for den?

Make sure the phones attribute is there, by using inner_label='phones' to the appropriate composition operation when creating the den graph.

the process of getting lattices and turning them into sparse
phone-level posteriors is very similar for den and num

We have num.phones while calling k2.create_sparse. But den does not have phones attribute.
What should we pass to k2.create_sparse for den?

Make sure the phones attribute is there, by using inner_label='phones' to the appropriate composition operation when creating the den graph.

.. and no don't use the bigram phone LM.  The aim is to reproduce what happens in real decoding, and then we don't use the bigram phone LM.

.. and no don't use the bigram phone LM.  The aim is to reproduce what happens in real decoding, and then we don't use the bigram phone LM.

We don't need this on the bigram lf_mmi den graph.

Can you elaborate the above comment? What is bigram lf_mmi den graph in snowfall?

We don't need this on the bigram lf_mmi den graph.

Can you elaborate the above comment? What is bigram lf_mmi den graph in snowfall?

By "bigram lf_mmi den graph" I mean the decoding graph that we use for the denominator of LF-MMI, which is based on a phone bigram (P).

By "bigram lf_mmi den graph" I mean the decoding graph that we use for the denominator of LF-MMI, which is based on a phone bigram (P).

What is the difference between lattice and phone-level posteriors? Posteriors should be contained in lattices.

What is the difference between lattice and phone-level posteriors? Posteriors should be contained in lattices.

What is the difference between lattice and phone-level posteriors? Posteriors should be contained in lattices.

Do lattices contain likelihoods by default, not posteriors?

What is the difference between lattice and phone-level posteriors? Posteriors should be contained in lattices.

Do lattices contain likelihoods by default, not posteriors?

I'm not quite sure what we should call the output of CTC model. In the current decoding pipeline, we use it as likelihoods, then posteriors becames useless.

I'm not quite sure what we should call the output of CTC model. In the current decoding pipeline, we use it as likelihoods, then posteriors becames useless.

I would call the floats in a lattice scores, which clarifies that they are in log-space without being super-specific about what they represent.
Posteriors are not the same as scores, they are the result of doing forward backward on scores.

I would call the floats in a lattice scores, which clarifies that they are in log-space without being super-specific about what they represent.
Posteriors are not the same as scores, they are the result of doing forward backward on scores.

Are arc posteriors also known as arc occupation probabilities?

Are arc posteriors also known as arc occupation probabilities?

Yes.  Although be careful, most of the time we store them in log space.

Yes.  Although be careful, most of the time we store them in log space.

take the sum-of-absolute-values of the difference of sparse matrix to form the MBR part of the objective function.

Does the loss consist of two-part:

MMI loss
MBR loss

If yes, do the two types of loss contribute equally, i.e., final_loss = mmi_loss + mbr_loss?

take the sum-of-absolute-values of the difference of sparse matrix to form the MBR part of the objective function.

Does the loss consist of two-part:

MMI loss
MBR loss

If yes, do the two types of loss contribute equally, i.e., final_loss = mmi_loss + mbr_loss?

We'll weight them, since they'll likely have different dynamic ranges.  You
can let the weight on MBR be 1.0 initially though.
On Wed, Jan 27, 2021 at 12:15 PM Fangjun Kuang<EMAIL_ADDRESS>wrote:

take the sum-of-absolute-values of the difference of sparse matrix to form
the MBR part of the objective function.
Does the loss consist of two-part:

MMI loss
MBR loss

If yes, do the two types of loss contribute equally, i.e., final_loss =
mmi_loss + mbr_loss?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/k2-fsa/snowfall/issues/80#issuecomment-768015025, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAZFLO53EQMR7KGTWEIEEFTS36HNNANCNFSM4WOAKNWQ
.

We'll weight them, since they'll likely have different dynamic ranges.  You
can let the weight on MBR be 1.0 initially though.
On Wed, Jan 27, 2021 at 12:15 PM Fangjun Kuang<EMAIL_ADDRESS>wrote:

take the sum-of-absolute-values of the difference of sparse matrix to form
the MBR part of the objective function.
Does the loss consist of two-part:

MMI loss
MBR loss

If yes, do the two types of loss contribute equally, i.e., final_loss =
mmi_loss + mbr_loss?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/k2-fsa/snowfall/issues/80#issuecomment-768015025, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAZFLO53EQMR7KGTWEIEEFTS36HNNANCNFSM4WOAKNWQ
.

For the MMI part, is it unchanged, i.e., with bigram phone LM and without G?
Or MMI and MBR share the same process for graph construction?

For the MMI part, is it unchanged, i.e., with bigram phone LM and without G?
Or MMI and MBR share the same process for graph construction?

MMI part is unchanged for now, but later we can try replacing it with the
larger graph.
On Wed, Jan 27, 2021 at 1:00 PM Fangjun Kuang<EMAIL_ADDRESS>wrote:

For the MMI part, is it unchanged, i.e., with bigram phone LM and without
G?
Or MMI and MBR share the same process for graph construction?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/k2-fsa/snowfall/issues/80#issuecomment-768031581, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAZFLO3GUMCWV5YHYBLNQTTS36MXNANCNFSM4WOAKNWQ
.

MMI part is unchanged for now, but later we can try replacing it with the
larger graph.
On Wed, Jan 27, 2021 at 1:00 PM Fangjun Kuang<EMAIL_ADDRESS>wrote:

For the MMI part, is it unchanged, i.e., with bigram phone LM and without
G?
Or MMI and MBR share the same process for graph construction?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/k2-fsa/snowfall/issues/80#issuecomment-768031581, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAZFLO3GUMCWV5YHYBLNQTTS36MXNANCNFSM4WOAKNWQ
.

take the sum-of-absolute-values of the difference of sparse matrix to form the MBR part of the objective function.

It turns out PyTorch does not support
a.abs()
# or
torch.abs(a)

if a is a sparse tensor.
RuntimeError: Could not run 'aten::abs.out' with arguments from the 'SparseCPU' backend.
'aten::abs.out' is only available for these backends: [CPU, CUDA, BackendSelect,
Named, AutogradOther, AutogradCPU,
AutogradCUDA, AutogradXLA, AutogradPrivateUse1, AutogradPrivate
Use2, AutogradPrivateUse3, Tracer, Autocast, Batched, VmapMode].

I am going to use a.to_dense().abs(). @danpovey  What do you think?

take the sum-of-absolute-values of the difference of sparse matrix to form the MBR part of the objective function.

It turns out PyTorch does not support
a.abs()
# or
torch.abs(a)

if a is a sparse tensor.
RuntimeError: Could not run 'aten::abs.out' with arguments from the 'SparseCPU' backend.
'aten::abs.out' is only available for these backends: [CPU, CUDA, BackendSelect,
Named, AutogradOther, AutogradCPU,
AutogradCUDA, AutogradXLA, AutogradPrivateUse1, AutogradPrivate
Use2, AutogradPrivateUse3, Tracer, Autocast, Batched, VmapMode].

I am going to use a.to_dense().abs(). @danpovey  What do you think?

Seems to_dense will generate a copy? Not sure if it's applicable ro share memory between sparse tensor and ragged tensor and do abs on the ragged tensor.

Seems to_dense will generate a copy? Not sure if it's applicable ro share memory between sparse tensor and ragged tensor and do abs on the ragged tensor.

I believe the plan is to implement our own sparse tensors in the end.
But the current priority is to get it done first.

I am leaning towards relying mostly on Torch's, at least at first, because otherwise we'll have an ever-increasing number of things we need to implement, such as sparse by dense matrix multiplication; but I want to understand it first.

I believe the plan is to implement our own sparse tensors in the end.
But the current priority is to get it done first.

I am leaning towards relying mostly on Torch's, at least at first, because otherwise we'll have an ever-increasing number of things we need to implement, such as sparse by dense matrix multiplication; but I want to understand it first.

Abs is a very easy operation to implement for sparse tensors, since it just
affects the individual values.
It might be possible to just construct another sparse tensor from the
meta-info and the abs of the values.  (Hopefully autograd will work).
Definitely don't make it dense.
I'd rather prototype stuff without having to implement sparse tensors-- see
if what I said works (and if the backprop for that works).
On Wed, Jan 27, 2021 at 3:45 PM Fangjun Kuang<EMAIL_ADDRESS>wrote:

I believe the plan is to implement our own sparse tensors in the end.
But the current priority is to get it done first.
I am leaning towards relying mostly on Torch's, at least at first, because
otherwise we'll have an ever-increasing number of things we need to
implement, such as sparse by dense matrix multiplication; but I want to
understand it first.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/k2-fsa/snowfall/issues/80#issuecomment-768101020, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAZFLO42WSHI4QWI5XEUBR3S37AB3ANCNFSM4WOAKNWQ
.

Abs is a very easy operation to implement for sparse tensors, since it just
affects the individual values.
It might be possible to just construct another sparse tensor from the
meta-info and the abs of the values.  (Hopefully autograd will work).
Definitely don't make it dense.
I'd rather prototype stuff without having to implement sparse tensors-- see
if what I said works (and if the backprop for that works).
On Wed, Jan 27, 2021 at 3:45 PM Fangjun Kuang<EMAIL_ADDRESS>wrote:

I believe the plan is to implement our own sparse tensors in the end.
But the current priority is to get it done first.
I am leaning towards relying mostly on Torch's, at least at first, because
otherwise we'll have an ever-increasing number of things we need to
implement, such as sparse by dense matrix multiplication; but I want to
understand it first.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/k2-fsa/snowfall/issues/80#issuecomment-768101020, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAZFLO42WSHI4QWI5XEUBR3S37AB3ANCNFSM4WOAKNWQ
.

IDK how hard it would be to test more recent PyTorch versions...

IDK how hard it would be to test more recent PyTorch versions...

I am using torch 1.7.1, which is already the latest stable version.

I am using torch 1.7.1, which is already the latest stable version.

Abs is a very easy operation to implement for sparse tensors, since it just
affects the individual values.
It might be possible to just construct another sparse tensor from the
meta-info and the abs of the values.  (Hopefully autograd will work).

Looking into it.

Abs is a very easy operation to implement for sparse tensors, since it just
affects the individual values.
It might be possible to just construct another sparse tensor from the
meta-info and the abs of the values.  (Hopefully autograd will work).

Looking into it.

.. of course this assumes the sparse tensor is coalesced (no repeated elements), but I think it normally is, anyway there might be a property available is_coalesced or something which will confirm.

.. of course this assumes the sparse tensor is coalesced (no repeated elements), but I think it normally is, anyway there might be a property available is_coalesced or something which will confirm.

Related functions are:

https://pytorch.org/docs/stable/sparse.html#torch.sparse.FloatTensor.coalesce
https://pytorch.org/docs/stable/sparse.html#torch.sparse.FloatTensor.is_coalesced

I am trying to implement k2.abs for sparse tensors with autograd support. Not sure how difficult it is.

Related functions are:

https://pytorch.org/docs/stable/sparse.html#torch.sparse.FloatTensor.coalesce
https://pytorch.org/docs/stable/sparse.html#torch.sparse.FloatTensor.is_coalesced

I am trying to implement k2.abs for sparse tensors with autograd support. Not sure how difficult it is.

I just implemented k2.sparse.sum and k2.sparse.abs for sparse tensors. It works perfectly for (aussming a is a sparse tensor):
k2.sparse.sum(k2.sparse.abs(a)).backward()

Unfortunately, it does not work for the following case
k2.sparse.sum(k2.sparse.abs(a - a)).backward()

  File "/xxx/py38/lib/python3.8/site-packages/torch/tensor.py", line 221, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/xxx/py38/lib/python3.8/site-packages/torch/autograd/__init__.py", line 130, in backward
    Variable._execution_engine.run_backward(
RuntimeError: unsupported tensor layout: Sparse

Seems like it is a limitation of PyTorch's autograd:
(a - a).to_dense().sum().backward()

throws the same exception.

I just implemented k2.sparse.sum and k2.sparse.abs for sparse tensors. It works perfectly for (aussming a is a sparse tensor):
k2.sparse.sum(k2.sparse.abs(a)).backward()

Unfortunately, it does not work for the following case
k2.sparse.sum(k2.sparse.abs(a - a)).backward()

  File "/xxx/py38/lib/python3.8/site-packages/torch/tensor.py", line 221, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/xxx/py38/lib/python3.8/site-packages/torch/autograd/__init__.py", line 130, in backward
    Variable._execution_engine.run_backward(
RuntimeError: unsupported tensor layout: Sparse

Seems like it is a limitation of PyTorch's autograd:
(a - a).to_dense().sum().backward()

throws the same exception.

Mm.   We may have to implement our own sparse tensors, then.. Disappointing.
On Wed, Jan 27, 2021 at 5:56 PM Fangjun Kuang<EMAIL_ADDRESS>wrote:

I just implemented k2.sparse.sum and k2.sparse.abs for sparse tensors. It
works perfectly for (aussming a is a sparse tensor):
k2.sparse.sum(k2.sparse.abs(a)).backward()
Unfortunately, it does not work for the following case
k2.sparse.sum(k2.sparse.abs(a - a)).backward()
File "/xxx/py38/lib/python3.8/site-packages/torch/tensor.py", line 221, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File "/xxx/py38/lib/python3.8/site-packages/torch/autograd/init.py", line 130, in backward
Variable._execution_engine.run_backward(
RuntimeError: unsupported tensor layout: Sparse
Seems like it is a limitation of PyTorch's autograd:
(a - a).to_dense().sum().backward()
throws the same exception.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/k2-fsa/snowfall/issues/80#issuecomment-768170942, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAZFLO7WK5Q62UUBK4XH3BDS37PMNANCNFSM4WOAKNWQ
.

Mm.   We may have to implement our own sparse tensors, then.. Disappointing.
On Wed, Jan 27, 2021 at 5:56 PM Fangjun Kuang<EMAIL_ADDRESS>wrote:

I just implemented k2.sparse.sum and k2.sparse.abs for sparse tensors. It
works perfectly for (aussming a is a sparse tensor):
k2.sparse.sum(k2.sparse.abs(a)).backward()
Unfortunately, it does not work for the following case
k2.sparse.sum(k2.sparse.abs(a - a)).backward()
File "/xxx/py38/lib/python3.8/site-packages/torch/tensor.py", line 221, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File "/xxx/py38/lib/python3.8/site-packages/torch/autograd/init.py", line 130, in backward
Variable._execution_engine.run_backward(
RuntimeError: unsupported tensor layout: Sparse
Seems like it is a limitation of PyTorch's autograd:
(a - a).to_dense().sum().backward()
throws the same exception.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/k2-fsa/snowfall/issues/80#issuecomment-768170942, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAZFLO7WK5Q62UUBK4XH3BDS37PMNANCNFSM4WOAKNWQ
.

The code of abs and sum with autograd support for sparse tensors is available at https://github.com/k2-fsa/k2/pull/626

The code of abs and sum with autograd support for sparse tensors is available at https://github.com/k2-fsa/k2/pull/626

Oh-- great!!
On Wed, Jan 27, 2021 at 6:03 PM Fangjun Kuang<EMAIL_ADDRESS>wrote:

The code of abs and sum with autograd support for sparse tensors is
available at k2-fsa/k2#626 https://github.com/k2-fsa/k2/pull/626
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/k2-fsa/snowfall/issues/80#issuecomment-768175032, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAZFLO6DEIAAL6PAIM4TRYDS37QHTANCNFSM4WOAKNWQ
.

Oh-- great!!
On Wed, Jan 27, 2021 at 6:03 PM Fangjun Kuang<EMAIL_ADDRESS>wrote:

The code of abs and sum with autograd support for sparse tensors is
available at k2-fsa/k2#626 https://github.com/k2-fsa/k2/pull/626
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/k2-fsa/snowfall/issues/80#issuecomment-768175032, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAZFLO6DEIAAL6PAIM4TRYDS37QHTANCNFSM4WOAKNWQ
.

The code that will throw exceptions can be found at
https://github.com/csukuangfj/k2/blob/58cd3aecb1978d15412f09ddcc0c4bb526ab952c/k2/python/tests/sparse_abs_test.py#L112
and
https://github.com/csukuangfj/k2/blob/58cd3aecb1978d15412f09ddcc0c4bb526ab952c/k2/python/tests/sparse_abs_test.py#L122

The code that will throw exceptions can be found at
https://github.com/csukuangfj/k2/blob/58cd3aecb1978d15412f09ddcc0c4bb526ab952c/k2/python/tests/sparse_abs_test.py#L112
and
https://github.com/csukuangfj/k2/blob/58cd3aecb1978d15412f09ddcc0c4bb526ab952c/k2/python/tests/sparse_abs_test.py#L122

After some attempts, I find that
(a - a).to_dense().sum().backward()

raises an exception.
However,
(a + (-a)).to_dense().sum().backward()

works perfectly!
https://github.com/k2-fsa/k2/pull/626 is reopened now.

After some attempts, I find that
(a - a).to_dense().sum().backward()

raises an exception.
However,
(a + (-a)).to_dense().sum().backward()

works perfectly!
https://github.com/k2-fsa/k2/pull/626 is reopened now.


PBS auction endpoint: ext.prebid.targeting

Documentation for https://github.com/prebid/prebid-server/issues/2223 . Implemented in https://github.com/prebid/prebid-server-java/pull/1730

@muuki88 Can we merge this? It's been LGTM for 19 days. But it doesn't seem to be waiting on a code PR.

@MartianTribe there is an open PR for this, which is still not merged. And currently this PR has also merge conflicts.


Global log enable/disable (IDFGH-9712)

Any thoughts on adding a global variable that can be used to enable/disable all logs at runtime?
Use case:
In some applications, it may be desirable to have many logs compiled (e.g. set a high CONFIG_LOG_MAXIMUM_LEVEL), but not impose a performance hit during normal operation, and only do so when instructed. E.g. for remote debugging, or perhaps having conditions in code where you want to start seeing logs.
The log cache seems to be fairly performant, but as an example, in my project if I compile in logs at LOG_LEVEL_DEBUG, but disable them at runtime with esp_log_level_set("*", ESP_LOG_NONE);, there is enough of a performance hit that it warrants looking into.
const int64_t MEASUREMENTS = 1000;

int64_t start = esp_timer_get_time();

for (int retries = 0; retries < MEASUREMENTS; retries++) {
    ESP_LOGI(TAG, "some test data, %d, %d, %d", retries, MEASUREMENTS-retries, 12);
}

int64_t end = esp_timer_get_time();

printf("%lld iterations took %lld microseconds (%lld microseconds per invocation)\n",
   MEASUREMENTS, (end - start), (end - start)/MEASUREMENTS);

For me, this results in
1000 iterations took 9069 microseconds (9 microseconds per invocation)

Not too bad but if you have a lot of logs, it can start to add up.
If instead we add a global log variable and do something like this:
// log.c
bool g_logs_enabled = true;

// esp_log.h
extern bool g_logs_enabled;

#define ESP_LOG_LEVEL_LOCAL(level, tag, format, ...) do {               \
        if ( (g_logs_enabled) && (LOG_LOCAL_LEVEL >= level) ) ESP_LOG_LEVEL(level, tag, format, ##__VA_ARGS__); \
    } while(0)

And switch that on/off each time with the test code:
g_logs_enabled = !g_logs_enabled;

You get:
# g_logs_enabled = false
1000 iterations took 90 microseconds (0 microseconds per invocation)
# g_logs_enabled = true (but log_level enabled)
1000 iterations took 9069 microseconds (9 microseconds per invocation)

Which is much more acceptable, but adds about 15KB to program size in my case. Any pitfalls with this / is this a bad idea? Should I be measuring differently?

Have you tried setting the maximum log level via menuconfig? Similarly have you tried calling something like esp_log_level_set("*", ESP_LOG_ERROR); (perhaps with ESP_LOG_NONE)

Yes that's what I described. With lots of logs compiled in but few enabled, you incur a performance hit as each log is checked to see if it's enabled.

There is also the KConfig setting for logs to disable all logs that are below a certain level, they simply will not be available in the compiled binary.
But having a bit more control to disable logs (such as a NDEBUG build) would speed it up considerably.

The stated use case is to make all logs available and run-time select-able, but without the performance hit. I understand you can simply not compile them in but  that's not what I want.

I don't think it is feasible to have everything compiled in without having a performance hit, at least for lower level components built into IDF itself.

I benchmarked it, rudimentary, but clearly very feasible. I'm suggesting this as an optional feature for certain use cases and asking for feedback about pitfalls of doing things this way.

Try taking a look at the disassembled code.

You're right. There is going to be a big perf difference.
ESP_LOG_X supports a different log level per tag, so it does not do the runtime check later until this code:
void esp_log_writev(esp_log_level_t level, const char *tag, const char *format, va_list args)
{
    if (!esp_log_impl_lock_timeout()) {
        return;
    }
    esp_log_level_t level_for_tag = s_log_level_get_and_unlock(tag);
    if (!should_output(level, level_for_tag)) {
        return;
    }

    (*s_log_print_func)(format, args);
}

Which is already after we've done all this code too:
#define ESP_LOG_LEVEL(level, tag, format, ...) do {                     \
        if (level==ESP_LOG_ERROR )          { esp_log_write(ESP_LOG_ERROR,      tag, LOG_FORMAT(E, format), esp_log_timestamp(), tag, ##__VA_ARGS__); } \
        else if (level==ESP_LOG_WARN )      { esp_log_write(ESP_LOG_WARN,       tag, LOG_FORMAT(W, format), esp_log_timestamp(), tag, ##__VA_ARGS__); } \
        else if (level==ESP_LOG_DEBUG )     { esp_log_write(ESP_LOG_DEBUG,      tag, LOG_FORMAT(D, format), esp_log_timestamp(), tag, ##__VA_ARGS__); } \
        else if (level==ESP_LOG_VERBOSE )   { esp_log_write(ESP_LOG_VERBOSE,    tag, LOG_FORMAT(V, format), esp_log_timestamp(), tag, ##__VA_ARGS__); } \
        else                                { esp_log_write(ESP_LOG_INFO,       tag, LOG_FORMAT(I, format), esp_log_timestamp(), tag, ##__VA_ARGS__); } \
    } while(0)

I think going that route it would have to be
#define ESP_LOG_LEVEL_LOCAL(level, tag, format, ...) do {               \
        if ( (g_master_log_level >= level) && (LOG_LOCAL_LEVEL >= level) ) ESP_LOG_LEVEL(level, tag, format, ##__VA_ARGS__); \
    } while(0)

And a slight performance hit with the compare.
1000 iterations took 8674 microseconds (8 microseconds per invocation)
1000 iterations took 94 microseconds (0 microseconds per invocation)

Not sure why compiled-in but set to none perf is better, you'd think it'd be worse.

+1
Compiling with log level "debug" and despite disabling log output at run-time makes the SPI LCD routines slow. I noticed that, too, a few days ago. So there is a lot of optimization potential for speed and saving energy.

@someburner your comments are hard to follow. What are perf numbers are comparing here?

And a slight performance hit with the compare.
1000 iterations took 8674 microseconds (8 microseconds per invocation)
1000 iterations took 94 microseconds (0 microseconds per invocation)

@chipweinberger
In both cases I'm benching with max+default log level set to debug. The slow-running (8us / invocation) is when g_master_log_level = ESP_LOG_DEBUG;. And the faster running is when g_master_log_level = ESP_LOG_NONE;.
And the comment is saying it's strange that the "slow" case is oddly a little faster than when using the boolean check that I suggested in OP vs your recommendation.

Got it. So the esp_log_level_master_set(esp_log_level_t level) approach solves your perf issue. I think it's worth opening a PR with that function. I would like to have this perf win too.

Yeah I will open a PR soon. We'll see how it's received though. The API can be a little strange if you make it optional, and making it required adds a decent chunk to code size.

adds a decent chunk to code size.

Perhaps put the check in esp_log_writev? As opposed to putting it in the macro.

Here is my current 5.1 edits against 5.0
esp_log.patch.txt

Running a few more checks here:

confirmed placing the check in esp_log_write is slower, only 10x perf vs 100x

1000 iterations took 828 microseconds (0 microseconds per invocation)
1000 iterations took 7252 microseconds (7 microseconds per invocation)

Not surprising at all since this still doesn't skip timestamps if enabled and has to call a function. Placing in IRAM doesn't help much. I suppose there could be an argument for a 10x perf bonus with a bit smaller code size but IMO 5-20K ish is not a big deal if you really want perf.

but IMO 5-20K ish is not a big deal if you really want perf.

My concern is IRAM functions that call ESP_LOGX will also be larger. Many apps have little IRAM to spare. Of course, we could put this all behind KConfigs. Maybe the best approach is both approaches!

More discussion in the PR:
https://github.com/espressif/esp-idf/pull/11057
I am pretty sure most components dont put debug macros into IRAM methods. Although I should probably check my IRAM usage with/without this to see.
Essentially maintainer is asking to make this an optional feature but transparent to the API. From my perspective that adds a lot of "what ifs" to how I would like this feature to be used. Unless I can be sure that IDF components and closed source libraries are only using ESP_LOGx macros guarded by KConfig macros that always default to being off, then making it transparent to API is problematic for me.

Hi! Since we have a PR https://github.com/espressif/esp-idf/pull/11057 for this issue I close this issue.
Further discussion can be done there.


HTMLVideoElement.requestVideoFrameCallback()

It's requestAnimationFrame but for <video>.

Specification: WICG/video-rvfc
Chrome: Flagged since 84
Firefox: https://github.com/mozilla/standards-positions/issues/250

Available via MDN now: https://caniuse.com/?search=requestVideoFrameCallback
→ https://caniuse.com/mdn-api_htmlvideoelement_requestvideoframecallback

Closing as per above


angularjs: Fix the finally method type signature

The finally method does not change the type of the returned promise - instead the original promise value is passed through unchanged.
Here is a CodePen which demonstrates the expected behaviour.
Note: Finally finally handler will modify the value if they a) throw or b) return a promise which is rejected.

@deadalusai thanks mate!


Image not switching?

Hey Tzyganu, first of all - thank you for the free extension!
I understand this is unlikely to be a bug with the extension itself but I'm really struggling!
We use Porto (the most popular magento theme) which I believe uses Fancy Images for media, but no matter what solution I find I cannot get images to switch when an option is selected
Each simple product has an image defined and I've looked into the source of our product view but cant find the main images "IMG ID" .
Any ideas?
My source looks like this -

@rmvdev I'm not familiar with the Porto theme. Most likely the image and media block does not use the same markup as the default theme.
Read the configuration manual to see what each field does and see how you can configure the extension.


Ctrl +Tab shortcut doesn't switch between editor

Azure Data Studio Version: 1.15.0

Ctrl +Tab is the default short key to navigate in editor but it doesn't seem to work any more

Ctrl + Tab is working for me.

This is fixed in Feb release.


Tingyu's final project - Black-Scholes Equation

In your introductory paragraph, you mention the article "17 equations that changed the world," yet you did not include this in your list of references. You could have at least linked the text of the title to the location of the article on the web:
http://www.businessinsider.com/17-equations-that-changed-the-world-2014-3
The text of the first two sub-sections has full sentences copied verbatim from Wikipedia!! This is very bad form and indeed you commit academic misconduct when you use this text without attribution (the Wikipedia articles are not among your references).
Your presentation is still rather scant and insufficient for readers unfamiliar with the topic. For example, the terms strike price and exercise price are explained in passing, and it's easy to miss what they mean. It's also easy to be confused by the fact that your equation is written first with V as the dependent variable, then with C. You should have a helpful text there about the convention of using C and P for call and put options.
Even though you have a paragraph about call and put options, and a link to an explanatory video, it would be useful to explain better what happens. For example, the fact that if S>E at the final time, the buyer will want to exercise the option to make a profit of S–E (this is important also to understand the boundary conditions).
In the demo, you introduce "an artificial limit S_max," without explanation. What's the reason to introduce it? Is it just numerical convenience?
In the derivation of the discretized equation, I would have liked to see each term of (2) written with its appropriate finite-difference discretization, before jumping to (3). It's hard to see at first glance where the i^2 comes from, unless you write it all out.
There is also no explanation of the initial and boundary conditions. The initial condition C(S,0)=max(S–E,0) requires some commentary. Did you notice that it's not smooth? (its first derivative is a step function). Could that have any effects on numerical solutions?
It's strange that you decided to only demonstrate the explicit method in your notebook. The Black-Scholes equation is a convection-diffusion equation for which the most popular scheme is Crank-Nicolson. And since you used an explicit method, you should have discussed the question of stability.
"... since there is i in the equation, we can no longer use vectorized stencil operation instead of double loops" —> Surely, there must be a way to get rid of the double loops. This is Python!
You compared the numerical result with that obtained with a statistical method, but without explanation of the latter. This is rather unsatisfying ... what is the probabilistic interpretation of the model?
Finally, your numerical example is taken from the paper "Numerical approximation of Black-Scholes equation," by G. Dura and Ana-Maria Mosneagu ... yet this paper is not in your reference list. (A few sentences from the text also paraphrase that paper rather closely ...)
This is quite disappointing and a rather sloppy job.


make GlobalContext.GetUnforwardedLogger work with *logger.TestLogger

This hopefully fixes nil logger issues. The problem was that, when KEYBASE_DEBUG was not present, libkb.SetupTest returned a *logger.TestLogger, which broke GlobalContext.GetUnforwardedLogger() since it did a type assertion to *logger.Standard which *logger.TestLogger is not.
Eventually we should probably move to a much cleaner and simpler logger interface in a big refactoring or something. But for now, let GetUnforwardedLogger() panic if it fails to return a proper logger?

this is great work, thanks for ferreting this out!
👍

Thanks Max! Merging :D


Order schedules in select all

The schedule promises coroutine queries all promises where next_run_time is less than or equal to tick time, we need to ensure that these schedules come back in a deterministic order.

Closes #192
Closes #194
Closes #195
Closes #196
Closes #197
Closes #198
Closes #199


Swift 1.2

Please update the code to be compatible with Swift 1.2!
Thank you! :)

Yep, the example project on swift 1.2 on xcode 6.3.1 gives 4 errors and won't build.

Thanks for the heads-up!

Hi I really like the UIAlertView you design, could you please fix the 4 bugs in Swift 1.2? Thank you so much~~~

I just commented  //    required override init() {
//        super.init()
//    }
and at Line 387 removed the ?.
It seems working (I mean, at least it compiles!)

@QUB3X  yes if you delete it. It can compiles, but it seems that the alertview dialog will be cover by the current viewcontroller's view instead of display at the front.

This PR should solve it and is merged into master:
https://github.com/stakes/JSSAlertView/pull/9


Translated geom_series.py to Julia 1.2

From @yolsever's #727.

@yolsever I think this is almost complete. Comments:

Let's cut down on the broadcasting (i.e., use of .+, ./, .-, .*) to cases where it's necessary. Otherwise it makes the code hard to follow (what is a vector, what is a scalar, etc.)

There's a line with a comment marked ### not sure about this line. The line looks good and clear to me. Check and remove the comment if appropriate.

Do a final style guide pass. Nothing struck me as wrong when I skimmed, but good to check. I think the main points here would be:

Maintain this correspondence between math and code even if the code is less efficient. Only optimize if it is really necessary.

Ensure that all code can be copied and pasted without modification into functions for performance and modification without changes to scoping (e.g. no local or global ever required)

Once this is done, we can do a formal review on this.

Note: If the equations are written in scalar form, but they vectorized the python side, it may be clearer for us to use list comprehensions or something. e.g.:
discounted_values = [b^t * p(t) for t in 1:T]
present_value = sum(discounted_values)

or something like that.
But if you do go this route, make sure you follow the style guide points about comprehensions/iterations.

I believe, if you are only doing sums, that you can even try things like
present_value = sum(b^t * p(t) for t in 1:T)

If I recall... Comprehensions can be very clear.
But we definetely need to remove every single broadcast that isn't necessary


weechat: 4.2.2 -> 4.3.0

Changes: https://weechat.org/files/doc/weechat/ChangeLog-4.3.0.html
Description of changes

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Result of nixpkgs-review pr 315441 run on x86_64-linux 1

10 packages built:

weechat
weechat-unwrapped
weechat-unwrapped.guile
weechat-unwrapped.lua
weechat-unwrapped.man
weechat-unwrapped.perl
weechat-unwrapped.php
weechat-unwrapped.python
weechat-unwrapped.ruby
weechat-unwrapped.tcl


TypeError on apidoc:generate

I just install by composer require the package, but when i try to create the doc using artisan apidoc:generate, this error appears :
` TypeError
Argument 1 passed to Mpociot\ApiDoc\Tools\DocumentationConfig::__construct() must be of the type array, null given, called in /var/www/vhosts/xxxxxxxxxxxxxx/vendor/mpociot/laravel-apidoc-generator/src/Commands/GenerateDocumentation.php on line 61
at vendor/mpociot/laravel-apidoc-generator/src/Tools/DocumentationConfig.php:9
5| class DocumentationConfig
6| {
7|     private $data;
8|

9|     public function __construct(array $config = [])
10|     {
11|         $this->data = $config;
12|     }
13|

  +15 vendor frames

16  artisan:37
Illuminate\Foundation\Console\Kernel::handle(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))`
Note: apidoc.php was created in config directory, but nothing was changed in this file.

Seems like the same problem here #685
It worked after running php artisan config:clear

Seems like the same problem here #685
It worked after running php artisan config:clear

it worked


Fix lib-cpp build issue

Signed-off-by: Blake Devcich<EMAIL_ADDRESS>

Add an equality check CHECK_ENUM_EQUALITY to Cancelling state for verification and code coverage


embedded power bi results in microsoft login box

there is an excellent example for how to embed powerbi App Owns Data here:
https://powerbi.microsoft.com/en-us/documentation/powerbi-developer-embed-sample-app-owns-data/
however this example is for running on the .net platform
and some of the routines used in the example dont exist for aspnetcore
there is another excellent post about someone attempting to do the same thing:
https://community.powerbi.com/t5/Developer/Embed-Power-BI-dashboard-in-ASP-Net-core/td-p/273279
so first i built the standard .net version according to the first example
and then i tried to migrate to aspnetcore incorporating fixes from the second link
here are the urls im using:
private static readonly string AuthorityUrl = "https://login.windows.net/common/oauth2/authorize/";
private static readonly string ResourceUrl = "https://analysis.windows.net/powerbi/api";
private static readonly string ApiUrl = "https://api.powerbi.com/";
private static readonly string EmbedUrl = "https://app.powerbi.com/";
and the result is -
the microsoft login box - not cool
can anyone tell me if they have actually embedded powerbi into aspnetcore?
if you have - can you give me a coupla hints on how to get real data back?

This issue was moved to aspnet/Home#3004


Remove gin specific probes

Since we switched to putting the server HTTP probes on net/http.serverHandler.ServeHTTP the Gin specific probe is not needed. The handler for Gin is called from the default HTTP server handler and effectively we get duplicate probes. The e2e tests are purposefully left in place so that we can still test that Gin instrumentation is working. I modified the gin tests to do the same level of validation as the net/http tests and I removed the double server span.
This is the backtrace from Delve attached to our Gin example:
   569:	// ServeHTTP conforms to the http.Handler interface.
=> 570:	func (engine *Engine) ServeHTTP(w http.ResponseWriter, req *http.Request) {
   571:		c := engine.pool.Get().(*Context)
   572:		c.writermem.reset(w)
   573:		c.Request = req
   574:		c.reset()
   575:
(dlv) bt
0  0x0000000000770a4e in github.com/gin-gonic/gin.(*Engine).ServeHTTP
   at /home/nino/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/gin.go:570
1  0x000000000065242e in net/http.serverHandler.ServeHTTP <--- /** the default HTTP server handler **/
   at /usr/local/go/src/net/http/server.go:3137
2  0x000000000064ea08 in net/http.(*conn).serve
   at /usr/local/go/src/net/http/server.go:2039
3  0x0000000000652c48 in net/http.(*Server).Serve.gowrap3
   at /usr/local/go/src/net/http/server.go:3285
4  0x000000000046ece1 in runtime.goexit
   at /usr/local/go/src/runtime/asm_amd64.s:1695

Closes #171 as not needed

Hm, the test failure shouldn't have happened, when I test manually it works as expected with the same main.go test file in the e2e test:
{"level":"info","ts":1712962564.0520866,"logger":"Instrumentation.Controller","caller":"opentelemetry/controller.go:54","msg":"got event","attrs":[{"Key":"http.request.method","Value":{"Type":"STRING","Value":"GET"}},{"Key":"url.path","Value":{"Type":"STRING","Value":"/hello-gin"}},{"Key":"http.response.status_code","Value":{"Type":"INT64","Value":200}},{"Key":"network.peer.address","Value":{"Type":"STRING","Value":"::1"}},{"Key":"network.peer.port","Value":{"Type":"INT64","Value":48980}},{"Key":"server.address","Value":{"Type":"STRING","Value":"localhost"}},{"Key":"server.port","Value":{"Type":"INT64","Value":8080}},{"Key":"network.protocol.version","Value":{"Type":"STRING","Value":"1.1"}}],"status":{"Code":0,"Description":""},"context":{"TraceID":"1a9bb4d3e636d5657bb47b167431aa24","SpanID":"7f0bddaa6495f765","TraceFlags":"01","TraceState":"","Remote":false},"parent":{"TraceID":"1a9bb4d3e636d5657bb47b167431aa24","SpanID":"8853ec505dd0e9f8","TraceFlags":"01","TraceState":"","Remote":true}}
{"level":"info","ts":1712962564.0524924,"logger":"Instrumentation.Controller","caller":"opentelemetry/controller.go:54","msg":"got event","attrs":[{"Key":"http.request.method","Value":{"Type":"STRING","Value":"GET"}},{"Key":"url.path","Value":{"Type":"STRING","Value":"/hello-gin"}},{"Key":"http.response.status_code","Value":{"Type":"INT64","Value":200}},{"Key":"server.address","Value":{"Type":"STRING","Value":"localhost"}},{"Key":"server.port","Value":{"Type":"INT64","Value":8080}},{"Key":"network.protocol.version","Value":{"Type":"STRING","Value":"1.1"}}],"status":{"Code":0,"Description":""},"context":{"TraceID":"1a9bb4d3e636d5657bb47b167431aa24","SpanID":"8853ec505dd0e9f8","TraceFlags":"01","TraceState":"","Remote":false},"parent":null}
2024/04/12 18:56:07 traces export: Post

I had to update the gin test at the end, we were checking for the scope to match the gin instrumentation and it was looking for a double server span.

LGTM, not sure we should keep the e2e test since we have both http and http_custom e2e tests, so I don't see the added value of keeping this test.

LGTM, not sure we should keep the e2e test since we have both http and http_custom e2e tests, so I don't see the added value of keeping this test.

I think keeping them as a forward compatibility test with Gin. We can always remove them in the future if they become a burden.

LGTM, not sure we should keep the e2e test since we have both http and http_custom e2e tests, so I don't see the added value of keeping this test.

I thought it would be useful to keep them around to prove we can still correctly instrument Gin, we are not dropping support for it, we are just handling it correctly with the server http instrumentation.


No definition found when ts code is in a subfolder

Hi there!
So I tried to installing atom and atom-typescript to see how it compares with Visual Studio / WebStorm for nodejs web projects. What I immediately noticed is that I'm getting Atom Ts: No definition found.
"What the heck does this mean?" though I. Well after searching the issues here, it became apparent that this is because of missing tsconfig.json. (Could the error message be improved? A user has no fighting chance dealing with this cryptic error message).
But the thing is, my tsconfig.json was not missing. It was there all right. It was working perfectly in WebStorm. What I quickly realized is that atom-typescript expects tsconfig.json at the project root.
I prefer to have all typescript files in a subfolder (as opposed to js, etc). Firstly I feel it's cleaner. Secondly, from time to time ts files appear inside node_modules, and ts compiler (at least in webstorm) dutifully walks the whole tree and compile them. And then complains. I did not want that to happen so I moved all my TS in a subfolder, put there tsconfig.json and it worked perfectly, but not in atom.
How do I configure atom-typescript so that it looks for tsconfig.json in a different place and starts transpiling from that folder? Is it possible to improve the error message?

Atom Ts: No definition found

That is for if you pres f12 aka goto definition. The message seems right for that.

What I quickly realized is that atom-typescript expects tsconfig.json at the project root

Not true.

How do I configure atom-typescript so that it looks for tsconfig.json in a different place and starts transpiling from that folder?

It looks at all folder above the current open .ts file. This is the way tsc is designed to work as well :rose:
Feel free to discuss any other issues here, we'd be hapy to help :rose:

That is for if you pres f12 aka goto definition. The message seems right for that.

That's weird. In the bottom pane there is following text:
No Build. Press ( F12 ) to start a build for an active TypeScript file's project.

This message suggests that F12 is for starting a build not for going to definition. Note that if I press F6 nothing happens at all. F12 builds the typescript. What am I missing?

It looks at all folder above the current open .ts file. This is the way tsc is designed to work as well

So it looks like it's broken. It does not build with tsconfig.json in the same folder as .ts. Anything I can do to help diagnose?

Thank you for this. It must have been artifact from very old Atom installation. I removed it completely and reinstalled from scratch, after that all started working as expected.

I'm also getting this error 'No Definition Found'.  It seems to compile the *.ts files fine until I add the 'out' options to the tsconfig.json then it stops compiling the ts files on save.  If I press F12 from with a ts file I then get the 'No Definition Found' error.
Here is a simplified version of the project:
root
models
modelOne.ts
modelTwo.ts
module.name.ts
tsconfig.json

Sorry for wasting your time it is working now adding the 'buildOnSave' options then uninstalling and reinstalling atom it is working as expected.  Seems odd I had to uninstall though?

Seems odd I had to uninstall though

You are not on windows are you? https://github.com/TypeStrong/atom-typescript/issues/195

Yeah, that looks like the issue

I was getting the same problem, but all I did was set "buildOnSave" from false to true, and restart the atom IDE. It was working fine with me without reinstalling.


Controller input not picking up

Not  sure what it takes for steam to transmit controller input to remote play app. doesn't come through with this.
maybe just run the xbox controller module on the app and it will pick up

i dont think controller works for this app
have you tried using the RemotePlayWhatever app?


fix doc generation setup

motivation: fix doc generation - sourcekitten requires Swift 5.2
changes: expose the doc generation task on the 5.3 docker-compose file so we can trigger it there

alternatively we can change the script to pin to the older version
@Lukasa @weissi probably need to check nio as well

alternatively we can change the script to pin to the older version
@Lukasa @weissi probably need to check nio as well

Uff yeah, I may have hit this before 🤔
Will check in other repos

Uff yeah, I may have hit this before 🤔
Will check in other repos

alternatively we can change the script to pin to the older version
@Lukasa @weissi probably need to check nio as well

@tomerd NIO seems okay, last release is 2.25.1 and the docs are for that very release.

alternatively we can change the script to pin to the older version
@Lukasa @weissi probably need to check nio as well

@tomerd NIO seems okay, last release is 2.25.1 and the docs are for that very release.

alternatively we can change the script to pin to the older version

@Lukasa @weissi probably need to check nio as well

@tomerd NIO seems okay, last release is 2.25.1 and the docs are for that very release.

looking at the code i think its susceptible to the same issue. it likely using a cached version which is why have not surfaced yet

alternatively we can change the script to pin to the older version

@Lukasa @weissi probably need to check nio as well

@tomerd NIO seems okay, last release is 2.25.1 and the docs are for that very release.

looking at the code i think its susceptible to the same issue. it likely using a cached version which is why have not surfaced yet


keystore failing on transactions Testflight 3.06 (351)

Was because the team change meant that the keychain is inaccessible. Solved by reverting back to the original app owner


does 'react-native-svg' library support d3-selection npm module ?

Thanks for the update. Is there any way to plot bar graph with having both X and Y axis on screen?

Take a look at: https://github.com/JesperLekland/react-native-svg-charts/


Add missing keywords to autocomplete for Snowflake

The following keywords for snowflake are missing (autocompletion does not work)

ilike
qualify
organization
gscluster

I never used organization or gscluster, but the first two are used very often.
Link to complete snowflake keyword list:
https://docs.snowflake.com/en/sql-reference/reserved-keywords.html

@hubertwimmer hi, do you have more information about using keywords "gscluster" ore "organization"? Are these functions or keywords? I found disappointingly few information about them

"qualify" added, "ilike" we have already

Can you please double check "ilike" because the syntax highlighting for it is not working in dBeaver 7.1.0 (and as far i can remember also not in all versions before).
And i also couldn't find out more about "gscluster" and "organization". Maybe you skip them for this issue. If i can find out more i can create another request. Ok?

@hubertwimmer ok, "ILIKE" added.

If i can find out more i can create another request. Ok?

sounds like a great plan

Tested 'qualify' and 'ILIKE'. Please create new feature requests for 2 others.

Fixed in version 7.1.3


docs(*) add request validator icon, banner, and nav link

Adds a beautiful icon to the Request Validator page, as well as an Enterprise banner and a side nav link in Enterprise 0.35+ navigation.

@chuca may have a different icon, putting this on hold for now...


Fix Final Death Not Showing in Live Fire/Fastball/other round based modes

The bug was that the final kill of each round would not show on the scoreboard as a death for the victim.
The regression was introduced in #72 and applied to all round-based gamemodes
Fixes the "Whatever fixed end of round survivors still receiving a death on the scoreboard when round ends, also introduced a bug where the last player killed doesn't receive a death either now." issue made by @GeckoEidechse in #132
Tested in Live Fire and Fastball, also tested in Attrition to make sure it didn't break anything else.
Clearing entities on round end or game end still does not add deaths to scoreboard and everything functions as intended.

I think this needs some more testing perhaps.
Looking at the SetWinner code in _gamestate_mp.nut both round based and normal modes enter the WinnerDetermined state, so I'm not sure if this change doesn't have any unintended side-effects

I played an Attrition round with this change and it seemed unaffected.
In theory it wouldn't count any deaths after the round ends and the screen starts fading to black, however I think (I might be mistaken here) that vanilla already does that, and I know that even so, the game either goes straight to epilogue or fades to black and post match after 1-2 seconds.


Shorten boilerplate

?setcolor=ffffff?replace;00000000=ffffff;ffffff00=ffffff?setcolor=ffffff = ?setcolor=fff?replace;fff0=fff
As suggested by TheFurryDevil on Discord.

Added in https://github.com/Silverfeelin/Starbound-DrawablesGenerator/commit/fdf46f276cce1ad4ff03e6fd630181646abe3886


remove fetch polyfill

remove fetch polyfill after 30/04/2024

@H01001000 If u get a chance please remove this as node-fetch is now causing issues and no issues are arising when using undici.


Improve theme style

make it responsive (support mobile devices)
increase font size
use svg logotype (it scales now)

Fixed #111

Coverage remained the same at 97.53% when pulling 5b8ef2e5aeeedfe6b411c763659cdd810d6de64d on theme-improvements into 2c137a5271cb0ed8d680c5a8230f37d6e215596c on master.

Coverage remained the same at 97.53% when pulling ae00e0ce9b23652623fbd17916da4cb853a8552f on theme-improvements into ea0a7bc0368576046f4d87382711b2fe390587ae on master.


Ordering issues with associative array

Fix ordering issues by using regular arrays instead of associative arrays. Existing code should work as is, thanks to PHP handling both kinds of arrays the same way.
This is a proposition of fix that highlights a problem: There's an issue with sorting.
Sorting is always correct when done in the create/edit view. But once saved and reloaded, the items were sorted by UUID (i.e. by array keys) instead of keeping their manually attributed order.

My guess is that since JSON objects are unordered sets of name/value pairs, there's no guarantee it'll keep ordering across systems (e.g. PHP-level serde, DB-level serde)


fix: update regex for browser name to repace backslash '/' with '_'

In my setup with SlimerJS, karma-junit-fails to create the xml files because the browser.name contains back-slashes '/' with 'Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 SlimerJS/0.9.6' . This results in an error similar to the following:
17 09 2015 23:55:39.016:WARN [reporter.junit]: Cannot write JUnit xml
	ENOENT: no such file or directory, open '/target/xunit/unit-tests/Mozilla/5.0_(X11__Linux_x86_64__rv_38.0)_Gecko/20100101_SlimerJS_0.9.6/TESTS-SlimerJS.xml'

Replacing the backslash with '_' fixes this issue.

Thanks, could you rebase on to the latest master please?

Looks like the changes here were already adopted upstream at some point, as seen in https://github.com/karma-runner/karma-junit-reporter/blob/eacf8bbbde0edd0d837d21e8671fc484eacac8cb/index.js#L67, so closing as a result.

Sorry, I made a mistake here - it is not upstream atm.


cube: Add support for VK_EXT_headless_surface

Adds HEADLESS as an option for CUBE_WSI_SELECTION
This builds cube using the VK_EXT_headless_surface
extension.
Change-Id: I3ec0d323fce996f15f3f85682a68758b1a213408

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

The spec says with regards to VK_EXT_headless_surface:

The VK_EXT_headless_surface extension is an instance extension. It provides a mechanism to create VkSurfaceKHR objects independently of any window system or display device. The presentation operation for a swapchain created from a headless surface is by default a no-op, resulting in no externally-visible result.

Source
Keeping in mind this quote and the fact that cube's entire point is to display an animated cube to the user, what is the reasoning for adding an option to cube that explicitly prevents it from displaying its images?

The main reason for this request is to have a sample application that can be used to verify support for the headless surface extension. We would like to use a sample application in testing a Vulkan layer which can implement WSI support: https://gitlab.freedesktop.org/mesa/vulkan-wsi-layer
I do appreciate that most of cube's usefulness is lost when using this WSI extension, but it seemed to be the simplest sample app for which to add headless WSI support. In addition a frame capture layer can be used to obtain the output of any application that is using the headless extension. Please correct me if there is a more appropriate Vulkan sample app for this.

The Vulkan Info application in this same repo should be able to verify whether an extension is supported or not.
This should include VK_EXT_headless_surface.

Vulkan Info can check if the headless surface extension is reported as supported, but would not use it to present any Vulkan rendered images. The advantage of having a sample application that uses headless surface to present its rendering is to demonstrate how Vulkan applications can work on a headless system. It can serve as an example of working with Vulkan on headless platforms.
If you think there is no value of using cube for this purpose then feel free to reject this request.

Yes, it would probably be better to demonstrate the headless surface extension with its own dedicated sample.  A great deal of the code in cube needs to deal with the various windowing systems and all the details of updating the window.  I think that the headless additions would easily get lost among all of this.
Perhaps a more valuable approach would be to make a simple demo that draws a simple scene like a color interpolated triangle and then dump the contents of the target into a PPM file or something. That way, the user could actually "see" the results of the headless rendering.  The headless aspects would also be more "front and center".
As far as distribution goes, you could try a PR to the LunarG Samples repo.  There's also a discussion going on within Khronos that suggests they might revive a samples repo.


uWSGI stats server collector

@avadhutp @dichiarafrancesco Please, review.
Btw. I'm not sure whether those long real JSON outputs in test fixtures are acceptable. If they are too big, I can grep just lines which contain "state":".

@drolando

@bill-hewitt

lg2m. ship it!


Library().delete_library does not delete the library

The following code outputs "1" and "Library exists" whereas the expected output would be "1" and "Library does not exist".
LMWareConfig().set_active_db("sqlite")
LLMWareConfig().set_vector_db("chromadb")
library_name = "lib1_library"
print(Library().delete_library(library_name=library_name)) # prints 1
if Library().check_if_library_exists(library_name):
    print("Library exists")
else:
    print("Library does not exist")

Hi @geverl!
Thank you for reporting that issue!
May I ask which llmware version you used?

@geverl - thanks for raising this - and appreciate a nice clear simple example (sometimes the most simple example produces the bug that you missed!) ....   I am running the same script (copy and paste) - and getting the correct answer "Library does not exist" ... there is a typo in your script LMWareConfig().set_active_db("sqlite") which I did correct to LLMWareConfig().set_active_db("sqlite") ... could you please try to run this again?    Will keep the issue open.

I think we should add a test case for this first.

@geverl - thanks for highlighting this - it is a good catch.   It has been fixed in the main branch if you pull the latest code, and will be in the next pip install release (in the next couple of days).   Two issues:
(1) to delete the library, please pass "confirm_delete=True", e.g.,      Library().delete_library(library_name=library_name,confirm_delete=True)
(2) the bug in LLMWare was brought out by creating a fresh Library() object (which has been fixed now), rather than using the 'library state' built when creating the library.   Instead, if you modify your current code to:
library = Library().create_new_library(library_name)
print(library.delete_library(library_name=library_name, confirm_delete=True))

then, I believe it should work even with llmware==0.2.15.
Could you please check and confirm back and then we will close out the issue.


Error in React project when formatOnSaveMode not set to "file"

Describe the bug
When using the extension in a React project, the extension will error when formatOnSaveMode is set to modifications or modificationsIfAvailable.
To Reproduce
Steps to reproduce outlined in example repo
Expected behavior
The extension should not error
Screenshots
If applicable, add screenshots to help explain your problem.
Versions (please complete the following information):

VS Code Prettier ESLint extension 5.0.3
Visual Studio Code: 1.68.1
Node: 16.15.1
Package Manager yarn 1.22.19
prettier: 2.7.1
eslint: 8.19.0

System Specifications (please complete the following information):

OS: Mac OS Mojave 10.14.6
Processor: 2.2 GHz Intel Core i7
RAM Size: 8 GB

The problem seems to be specific to React as I was unable to reproduce in a vanilla js project. A potential solution (at least temporarily) would be to make sure formatOnSaveMode is always set to file.
I have my workspace .vscode/settings.json set like this and it works great.
{
  "editor.defaultFormatter": "rvest.vs-code-prettier-eslint",
  "editor.formatOnPaste": false,
  "editor.formatOnType": false,
  "editor.formatOnSave": true,
  "editor.formatOnSaveMode": "file",
  "files.autoSave": "onFocusChange"
}

@mattbrannon  I was able to reproduce the behavior by wrapping the

 in App.js with an empty fragment and then saving.
Before Modification

After Modification
<>

</>
VS Code will send <> and then </> to my format function. Prettier is giving an error when trying to format this since it uses Babel. Babel won't successfully parse <> alone. I can update the documentation but since "file" is the default for "formatOnSaveMode", I'm going to close this issue.

I think updating the documentation is enough. I was banging my head against the wall for a while and then once I changed the setting back to "file" it worked like a charm. I think having some explicit instructions letting people know that it will break if formatOnSaveMode is set to anything but "file" would be helpful to others who may find themselves in a similar situation.
In general, the docs have all the information someone needs to get the extension working. But there's a few places that I think cause a little friction for users like myself when first attempting to install and use the it. If you're open to it I could submit a pull request with some ideas.

@mattbrannon I'd love a PR. I'll reopen this issue so you can close it with your PR.

Closed by #49


Rewrite data transformations to use the AFTER_DATALOADER event

I am very skeptical that the algorithms that use add_dataset_transform in composer.utils.data work as intended, as the dataloader workers are already created before the INIT event runs (and the algorithms attempt to monkeypatch the dataset). Instead, add_dataset_transform should be replaced by modifying the data on the AFTER_DATALOADER event. This will require that composer.utils.augmentation_primiatves are reimplemented to operate on batches of images (rather than on individual PIL images), and likely switch to using torchvision instead of PIL
It would be good to run a test to confirm that randaugment, augmix, and colout do not work as intentend.

🤔 cc: @coryMosaicML @growlix
I dont think the proposed solution would work, as the AFTER_DATALOADER event now runs on GPU. We've been using add_dataset_transform since launch, so unless something broke, it was working previously.

I think things are working fine on main, but going to look into this on dev now. @hanlint is correct that this would require doing augmentations on GPU.

Testing this made me aware that randaugment and augmix don't work with our CIFAR10 dataset, but not for the reason that @ravi-mosaicml mentioned. It's because the the cifar10 dataset already has to_tensor() in the list of transformations, and add_dataset_transform() just appends randaugment, so it ends up acting on tensor. About to test w/ ImageNet.

Seems to work on ImageNet for main and dev. Going to update add_dataset_transform() to be cifar10-friendly.

Should we support a
add_dataset_transform(.., before_to_tensor=True)

option?

@ravi-mosaicml OK with closing this issue then? the CIFAR-10 issue is a different issue.

Good to close, cifar10 compatibility PR incoming!

@growlix can you elaborate how you confirmed this?

Ah, since pytorch dataloader lazily forks on the first iteration, this works. However, I would not like to assume this will always be the case, as there is no guarantee that a user didn't already spin their dataloader before passing it into the trainer.
If we would like to do CPU data augmentations as a first-class API, then I think our trainer init should take the (dataset, batch size, sampler, drop_last) rather than a dataloader. We can then construct the dataloader at the appropriate time.

I confirmed by running run_mosaic_trainer.py -f {resnet50, resnet56} --algorithms randaugment --dataloader.num_workers 8 on a cota-g1-3080 interactive instance for dev and main and set a breakpoint in algorithms/randaugment/randaugment.py and checking that randaugment() was executing and the tensor was being modified.

Is there a fix that would accommodate the scenario you're describing? (pre-spun, persistent workers, n workers > 0)

The easiest 'fix' would be to recreate the dataloader e.g. something like dataloader = Dataloader(**vars(dataloader)), but I am not a fan of this approach since this wouldn't support a user's custom dataloader.

Yeah, agree i think the edge case can be not supported.


Issues related to disabling ABDesigner with CocoaPod

It's related to #221 and #220, sort of. Now that MPLog is replaced by MixpanelError, which is great, we no longer see logs from Mixpanel. However, it's still running in the background, just not ouputing it. I would love an easier way to disable ABDesigner. It's a feature that I will probably never use. I don't want it running in the background when I am debugging and optimizing. It's fine if Mixpanel decides it should be on by default, but at least provide us a way to opt out without having to resort to forking the project.
Yes, I am aware of the DISABLE_MIXPANEL_AB_DESIGNER flag we can add to our preprocessor macros in the project settings. However, I am using CocoaPod to manage all the dependencies in my project. Adding the flag to my project does nothing, it has to be added to the Pods-Mixpanel target, which is created by CocoaPod automatically. It's overwritten every time we run "pod update". It means it has to be a manual process every time just to turn it off.
Can we add a boolean value on Mixpanel object, which can default to YES, but expose it for developers to set it to NO if choose to? There must be an easier way. Thanks a lot.

Yes I don't really understand the idea behind using Preprocessor Macros in a library. Most of the time you're either using a static lib or building a CocoaPod.
While there is a way to set build flags with Cocoapods in the post_install method, it'd be way easier to be able to toggle it in the Mixpanel + (instancetype) sharedInstanceWithLaunchOptions:(NSDictionary *)launchoptions message.

hey guys, using the macro allows some customers to compile versions of the lib that do not include the ab testing functionality (vs just shutting it off) so it was a must. it also makes it easy for them to use the macro differently with different build targets.
that said, we're in a small debate now about whether the addition of a flag would be good to add as well, but it looks likely we'll add it if a better solution doesn't present itself. sorry again about all the console spam.

@LanceFu You can add the macro to the Podfile and it will prevent it from being overwritten when you perform a pod update.
http://guides.cocoapods.org/syntax/podfile.html#post_install
Closing this issue, please re-open if this doesn't solve your problem.


[Docs] Miniflare v2 docs removed?

Given that Miniflare 3 currently does not integrate well with Jest and vitest, a lot of us are still using Miniflare v2 for unit test simulation of the cloudflare tools.
However, it looks like those old docs have all been removed? Previous links like https://miniflare.dev/testing/jest no longer work.
The official docs link also suggest using miniflare v2, but the link on the testing page (https://miniflare.dev/testing) redirects straight back to the miniflare v3 docs (the link in question is https://v2.miniflare.dev/testing)
Not sure if this is an oversight or not, but it would be really great if we could still access the old miniflare v2 docs.

Hey! 👋 You should be able to access the v2 docs here: https://legacy.miniflare.dev/ Will need to get that link fixed. 👍

Ahh gotcha, thank you 😄

Actually, this seems like a simple enough change, I'll make a PR 👍


test(import) Test that all WAsm types can be used in imported functions

Related to #18.

bors r+


packet order incorrect

While connecting to the same server referenced for this mysql simple issue https://github.com/blackbeam/rust-mysql-simple/issues/160 I get an error about unexpected packet order using mysql_async. My first thought is perhaps the feature that was added in mysql simple just hasn't been implement in mysql_async. I can get any extra details you need. This is using SSL without a server public certificate and with domain verification off.

Hi!
Yeah. I believe it's blackbeam/rust-mysql-simple#160.
I have plans to fix this until Jan 8.

Great I'm glad a fix is in the roadmap.

Thanks for the quick reply as well.

Support of AuthSwitchRequest is implemented in v0.17.0.
Could you please verify that this issue is now fixed?

I will get verification done on Monday unfortunately I don't have access to the system in order to test over the weekend.


C++ desktop api calls is broken by 1.0 preview2 single-project MSIX packaging

Describe the bug
I created an initial Desktop C++/WinRT WinUI project using the newest template "Blank App, Packaged (WinUI3 in Desktop)", which should be using the preview MSIX self-packaging tooling.
But I noticed that the project name in solution explorer is annotated with suffix (Universal Windows), which is of course wrong. I checked the .vcxproj file and found this:
<AppContainerApplication>true</AppContainerApplication>
<AppxPackage>true</AppxPackage>

But no matter the two switches is on or off, calling functions like GetWindowsDirectory, RegOpenKey will get a compiler error saying that identifier not found.
The function declarations in WindowsSDK include files was grayed by intellisense, because #if WINAPI_FAMILY_PARTITION(WINAPI_PARTITION_DESKTOP | WINAPI_PARTITION_SYSTEM) is false.
As you can guess, the reason was possibly the desktop partition of API family was disabled.
I don't know where to change this settings, if there won't be any hot fix, could you tell me the workaround?
Steps to reproduce the bug

Steps to reproduce the behavior:
Create a C++ project with the template "Blank App, Packaged (WinUI3 in Desktop)", and call desktop-specific APIs.
Expected behavior
Desktop projects should have capabilities to call all native APIs.
Version Info
Windows 11 Build 22000.194
Visual Studio 2022 Preview 4.1
WindowsAppSDK 1.0 preview2

Windows version
Saw the problem?

Insider Build (xxxxx)
Yes

May 2020 Update (19041)

November 2019 Update (18363)

May 2019 Update (18362)

October 2018 Update (17763)

Additional context

Hi @seven-mile, this is a known issue in Preview 2 that we're working on fixing: https://docs.microsoft.com/en-us/windows/apps/windows-app-sdk/preview-channel#other-limitations-and-known-issues

C++ template for Visual Studio 2019 and Visual Studio 2022. In this release, these projects are restricted to calling the subset of Win32 APIs that can be called by UWP apps. The Blank App, Packaged with WAP (WinUI 3 in Desktop) template is not affected by this issue.

This only affects the single-project templates that are new in Preview 2. You can instead use the Blank App, Packaged with WAP (WinUI 3 in Desktop) template to create a project the old way that includes two projects and it'll work correctly.
I'll leave this bug open to track the status of this issue since we didn't have a public bug tracking it yet (we had internal bugs tracking it).

Hi @seven-mile, I believe this was fixed in 1.0 Preview 3. It has definitely been fixed in the upcoming 1.0 GA, marking as closed. Thanks!


🔄 synced file(s) with seedcase-project/seedcase-theme

synced local file(s) with seedcase-project/seedcase-theme.

Changed files

synced local directory _extensions/seedcase-project/seedcase-theme/ with remote directory _extensions/seedcase-theme/

This PR was created automatically by the repo-file-sync-action workflow run #9645236496

We have agreed that the puml theme shouldn't be synced, so I have moved it in seedcase-project/seedcase-theme#40 and will close this PR.


Fix/peek/file size

Summary of the Pull Request
Peek used to display 1 "bytes", which is grammatically incorrect. I have changed it to display "1 byte".
PR Checklist

[ ] Closes: #xxx
[x] Communication: I've discussed this with core contributors already (CC @Jay-o-Way).
[x] Tests: No need to add test cases for such a fix
[x] Localization: Localization is not affected
[x] Dev docs: N/A
[x] New binaries: No need to update following binaries

[x] JSON for signing for new binaries
[x] WXS for installer for new binaries and localization folder
[x] YML for CI pipeline for new test projects
[x] YML for signed pipeline

[x] Documentation updated: No need

Detailed Description of the Pull Request / Additional comments
I have added a new symbol, byte, in the resources.resw file and used this in the required function using a simple if type condition.
Validation Steps Performed
I built the peek project and verified it by opening files of various sizes. It correctly displays the file size.

@microsoft-github-policy-service agree

@Deepak-Sangle please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.
@microsoft-github-policy-service agree [company="{your company}"]

Options:

(default - no company specified) I have sole ownership of intellectual property rights to my Submissions and I am not making Submissions in the course of work for my employer.

@microsoft-github-policy-service agree

(when company given) I am making Submissions in the course of work for my employer (or my employer has intellectual property rights in my Submissions by contract or applicable law). I have permission from my employer to make Submissions and enter into this Agreement on behalf of my employer. By signing below, the defined term “You” includes me and my employer.

@microsoft-github-policy-service agree company="Microsoft"

Contributor License Agreement


UseOrEmpty: fix false positive for indexing operator calls with type parameters

Fix for https://github.com/detekt/detekt/issues/4791

Thanks! We had multiples cases here :)


Update MD file on new branch 2017-25-25_05-11-30

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

split/updateTestCon.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Storage account sas used with URI that includes a container name

Package Name: @azure/storage-blob
Package Version: ^12.9.0
Operating system: windows 11 internal
[X ] nodejs v16.13.1

version:

[ ] browser

name/version:

[ ] typescript

version:

Is the bug related to documentation in

[ ] README.md
[ ] source code documentation
[X] SDK API docs on https://docs.microsoft.com

Describe the bug
A clear and concise description of what the bug is.
If I create an Account sas token, then use the sas token in a URI which includes the blob storage container name, I get a meaningless error of 'The resource doesn't support specified Http Verb'.
FYI - RequestId:32987b5f-101e-0072-3ca1-90ab72000000\nTime:2022-07-05T18:58:51.4305537Z'
To Reproduce
Steps to reproduce the behavior:

Create Account SAS token

async function createAccountSas() {

    // where is signed encryption scope?
    // all resources, all services except files, all permissions
    const sasOptions = {

        // ref doc issue or sdk issue because I thought I would need
        // `AccountSASPermissions` to create property values

        services: AccountSASServices.parse("btqf").toString(), //"bqt", // not files
        resourceTypes: AccountSASResourceTypes.parse("sco").toString(), // service, container, object
        permissions: AccountSASPermissions.parse("rwdlacupi"), // FULL permissions
        protocol: SASProtocol.Https,
        startsOn: new Date(),
        expiresOn: new Date(new Date().valueOf() + (60 * 60 * 1000)), // 1 hour

    };

    const sasToken = generateAccountSASQueryParameters(
        sasOptions,
        sharedKeyCredential // is this optional, when should I use it?
    ).toString();

    constants.sasToken = sasToken;
}

Use sas token with container name in URI

async function getBlobServiceClientProperties() {

    try {
        // ? if I have a container name on this - I get an error of
        // 'UnsupportedHttpVerb'
        // 'The resource doesn't support specified Http Verb.\nRequestId:32987b5f-101e-0072-3ca1-90ab72000000\nTime:2022-07-05T18:58:51.4305537Z'
        // why not just ignore the container name?

        // log issue -
        const blobServiceUriWithSasToken = `https://${constants.accountName}.blob.core.windows.net/${constants.containerName}${(constants.sasToken[0] === '?' ? constants.sasToken : `?${constants.sasToken}`)}`;
        console.log(blobServiceUriWithSasToken);

        const blobServiceClient = new BlobServiceClient(
            blobServiceUriWithSasToken
        );

        const blobServicePropertiesResponse = await blobServiceClient.getProperties();

        console.log(
            `Sas Permission: ${JSON.stringify(blobServicePropertiesResponse)}`
        );
    } catch (err) {
        console.log(err);
    }

}

Expected behavior
An account sas token should be able to do everything in the account without an error. Please ignore the container name if included in the URI.
This also applies to file, queues, and tables - if a sub-object is in the URI - just ignore it.
Screenshots
If applicable, add screenshots to help explain your problem.
Additional context
Add any other context about the problem here.

@jeremymeng @EmmaZhu

Hi @diberry ,
The issue is not because of SAS token, it's because you tried to create a service client with a container URL. The operation to fetch Blob Service properties against a container is not valid.
Thanks
Emma


[QUERY]  azure-keyvault 1.x vs azure-security-keyvault-* 4.x

Query/Question
There seems to be multiple Java  client libraries for accessing keys within an Azure Key Vault, whilst I have previously been using azure-keyvault the new examples seem to direct users to the azure-security-keyvault-* libraries. Is azure-keyvault going to be deprecated in favour of the newer libraries?
Should projects being using the latter going forwards? IMHO: They're not equivalent at this point, which might be by design.
Why is this not a Bug or a feature Request?
General question on usage of the libraries.
Setup (please complete the following information if applicable):
N/A
Information Checklist
Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

[ x] Query Added
[ x] Setup information Added

Thanks for posting the question @magooster.  Yes, the v4.x azure-security-keyvault-* libraries are the latest supported SDK libraries for KeyVault.  We do recommend upgrading to this latest stable, supported release.  More information can be found  here https://techcommunity.microsoft.com/t5/azure-sdk/february-2020-unified-sdk-release/ba-p/1174661 and also on the AzureSDK twitter feed here https://twitter.com/AzureSdk/status/1229856571020083200.
We have a work item to publish relocation POMs to Maven to make the upgrade path more clear.  https://github.com/Azure/azure-sdk-for-java/issues/7968
@g2vinay can you please follow up with more details on track2?

@magooster
Thank you for your query and actively trying out our 4.x KV libraries, really appreciate the effort for identifying potential issues in our Track 2 SDKs.
Yes, 4.x azure-security-keyvault-libraries are the ones which we recommend going forward.
4.x azure-security-keyvault-libraries are part of our Track 2 SDKs, which are completely redesigned to offer better and friendly user experience to our developers.
It supports pluggable HTTP clients , runs on a re-designed java runtime and offers better performance compared to older Track 1 SDKs.
Let us know, if you've any more questions.


stack support

Hello, it would be nice to have ekg and therefore ekg-json in Haskell LTS to be fully supported by stack.

Stackage shouldn't require any upstream changes for it to work. It should just work with a .cabal file. Check with the Stackage maintainers.


Support for generating JSON formatted substrait plan

A few other substrait producers are able to generate a JSON formatted substrait plan (Isthmus, DuckDB).  It would be nice if datafusion could do this as well.  It would make it much easier to compare the generated substrait plans.

One alternative would to be to use the protobuf json_format package to export/import the JSON.

The substrait crate supports the Protobuf JSON Mapping via the serde feature: https://docs.rs/substrait/latest/substrait/#serde-support.


bad import "unsafe"

I tried to run luar on an Go - Appengine project. However I encountered
"o-app-builder: Failed parsing input: parser: bad import "unsafe" in github.com/stevedonovan/luar/luar.go"
I searched and found that the package is trimmed on GAE. Does this mean I cannot use luar with golua on GAE?
Thank you.

Also see https://github.com/layeh/gopher-luar.
I've removed the 'unsafe' import in https://github.com/stevedonovan/luar/commit/63246cf9b270904e56720123a986065e63141ae6.


fixed minor typos

resolves #6529
No issue created (updating comments and docs)
Description
Corrects some minor typos that have been bugging me for a while.
Checklist

[ ✓] I have read the contributing guide and understand what's expected of me
[ ✓] I have signed the CLA
[ ✓] I have run this code in development and it appears to resolve the stated issue
[ ✓] This PR includes tests, or tests are not required/relevant for this PR
[ ✓] I have opened an issue to add/update docs, or docs changes are not required/relevant for this PR
[ ✓] I have run changie new to create a changelog entry

@nshuman1 Thanks for the PR! I know this is just fixing some typos. Could I still ask you to open a quick issue for it, and add a changelog entry? We want to be able to give you credit in the release notes after merging!

@nshuman1 Looks like there's just a last little bit of reformatting:
If you are seeing this message in CI, reproduce locally with: `pre-commit run --all-files`.
To run `pre-commit` as part of git workflow, use `pre-commit install`.
All changes made by hooks:
diff --git a/core/dbt/cli/params.py b/core/dbt/cli/params.py
index 77cbf0e..b739e88 1006[44](https://github.com/dbt-labs/dbt-core/actions/runs/3847710753/jobs/6555228716#step:5:45)
--- a/core/dbt/cli/params.py
+++ b/core/dbt/cli/params.py
@@ -270,7 +270,7 @@ show = click.option(
 )

 skip_profile_setup = click.option(
-        "--skip-profile-setup", "-s", envvar=None, help="Skip interactive profile setup.", is_flag=True
+    "--skip-profile-setup", "-s", envvar=None, help="Skip interactive profile setup.", is_flag=True
 )

You can make that change manually, or (better) install pre-commit and run it to make the change for you: https://github.com/dbt-labs/dbt-core/blob/main/CONTRIBUTING.md#pre-commit

@jtcohen6 ran pre-commit, fixed linter error.


feat: update server to use express instead of restify

Also consolidate server creation to same file.
DOL still starts server internally so that's not easy to fix unless we move dependency out to sample repo.

:tada: This PR is included in version 0.273.0 :tada:
The release is available on:

npm package (@next dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Fix host header in req

What does this PR do?

This PR allows to set Host header to http providers.
providers:
  http:
    endpoint: "http://localhost:4000/config"
    headers:
      Host: "some.custom.domain"

Motivation

Resolves https://github.com/traefik/traefik/issues/11210
More

[ ] Added/updated tests
[ ] Added/updated documentation

Additional Notes

Hello @nikonhub and thanks for your contribution,
As this pull request is an enhancement, could you rebase it on the master branch?


Update deliver

looks good!

Maybe add a debug log here: https://github.com/bitrise-io/steps-deploy-to-itunesconnect-deliver/pull/6/files#diff-954e6a6d1c6501f0929fb225c7cf9f00R14 or just remove the "already installed" line if "update=true" -> right now it's a bit misleading that although it prints "deliver is already installed" it'll still go ahead and update (if "update=true"), without any log / note

👍


Automated testing of hello world

There are a lot of related bits that are not finished. A one-second mandatory max runtime won't work forever. But I think this works for now.

Oh, this was against the wrong branch. Argh! Sorry.


Add Command Name to FLS

Critical Changes
Add Command Name to Admin and OCC Standard profiles for Dev Org and Customer Orgs so that field appears on install.
Add OCC Fields to OCC Standard Profile so that all packaged fields will appear in OCC Standard Profile.
Changes
Issues Closed
#85

@mkolodner got the Sailor FLS issue figured out too.  For the Admin profile, I kept it to only that field to avoid being invasive to the app.  For OCC Standard Profile, I did an expanded FLS add by adding Case, Contact, Event, Task, Opportunity and Sailor fields to the profile.