Feature: Threading support

Summary
Operations that take a long time to run currently lock up the entire app. It should be possible to run all the operation logic within a separate thread, allowing the user to continue interacting with the app, even if the recipe is still executing. The user should also be able to cancel a bake that is taking too long or has crashed.
Possible solutions
Web Workers provide rudimentary JavaScript threading within browsers. The entire CyberChef core could sit in a Web Worker and communicate with the web app via message passing. As the core has been designed to operate distinctly from the front-end app, this shouldn't be too complex to implement.

I've drawn out design diagrams for the current and proposed systems. They show how each of the components gets instantiated in each model, which it is important to understand in order to ensure that there is minimal duplication between the core and app. Note that this does not include Utils.js which is used almost everywhere and will have to be compiled into both the worker and the front-end (in future it may make sense to split Utils.js into two if there isn't much crossover).
Current design diagram

Proposed design diagram

We've discussed this before: I think it is a good thing to do in general (UI problems,etc), and also very good for the long-term client/server vision (where operations can run in node or another language).
IIRC browser compatibility is pretty good for Web Workers. Do you plan on including a fallback at all?
Looking forward to this!

Browser support is fairly good and covers all the versions that CyberChef officially supports, so there won't be any fallbacks. If the user is running a browser that doesn't support web workers, the chances are that other parts of the app will also fail anyway.
The roadmap looks roughly like this (no timescales yet):

Threading - add Web Workers to support threading of the entire baking process.
Modules - only download the code for a core subset of common operations initially, then import the code to run other types of operation as requested. importScripts() should make this possible.
Abstract the core - give the user the ability to specify where operations are run, i.e. locally, on a specified server, or maybe even in the "cloud".

I had never seen importScripts before but now I can imagine how it would make modules much easier to implement. Good find.


Wrap entry optimization

Alternative implementation of wrap entry optimization proposed by @wendigo in #5
Instead of using global sync.Pool with locks, entry byte buffer is allocated per shard.
Bench test results before and after optimization:
go test -bench=WriteToCacheWith -benchtime=5s

BenchmarkWriteToCacheWith1Shard-4                       5000000	     1520 ns/op
BenchmarkWriteToCacheWith500Shards-4                    	10000000	      647 ns/op
BenchmarkWriteToCacheWith1kShards-4                     10000000	      627 ns/op
BenchmarkWriteToCacheWith10kShards-4                    	10000000	      646 ns/op
BenchmarkWriteToCacheWith1kShardsAndSmallShardInitSize-4	10000000	      789 ns/op

BenchmarkWriteToCacheWith1Shard-4                       5000000	     1320 ns/op
BenchmarkWriteToCacheWith500Shards-4                    	20000000	      594 ns/op
BenchmarkWriteToCacheWith1kShards-4                     20000000	      616 ns/op
BenchmarkWriteToCacheWith10kShards-4                    	10000000	      629 ns/op
BenchmarkWriteToCacheWith1kShardsAndSmallShardInitSize-4	10000000	      717 ns/op

Total time spent on entries wrapping during bench tests:
//before opt
1.13s     15.34s    104:	w := wrapEntry(currentTimestamp, hashedKey, key, entry)

//after opt
2.50s      3.84s    106:	w := wrapEntry(currentTimestamp, hashedKey, key, entry, &shard.entryBuffer)

Wow, nice one :) It seems that using sync.Pool has no sense here and this kind of buffer is fine in wrapEntry

I am glad you like it, although credits goes to you with this pr as you discovered the optimization possibility and proposed a solution with comparable results, thanks :)

:+1:

I leave one comment but it looks good. :+1:


chore(release): Automated agent release notes (v1.0.53)

Automated PR

testing automated release notes


Support autodiff of Eigendecomposition with repeated eigenvalues

On v0.1.25 on OSX, I get the following error when computing gradients from the following jit-compiled function.
import numpy as onp
import jax.numpy as np
from jax import grad, jit

def test(x):
    val, vec = np.linalg.eigh(x)
    return np.real(np.sum(val))

grad_test = jit(grad(test))
grad_test_jc = jit(grad(jit(test)))

x = onp.eye(3, dtype=onp.double)
xc = onp.eye(3, dtype=onp.complex)

print(test(x))
print(grad_test(x))
print(grad_test_jc(x))
print(grad_test(xc))

3.0
[[1. 0. 0.]
 [0. 1. 0.]
 [0. 0. 1.]]
[[1. 0. 0.]
 [0. 1. 0.]
 [0. 0. 1.]]
[[1.+0.j 0.+0.j 0.+0.j]
 [0.+0.j 1.+0.j 0.+0.j]
 [0.+0.j 0.+0.j 1.+0.j]]

So far so good. But computing the gradient of the jit-compiled function with complex inputs errors
print(grad_test_jc(xc))

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-1-10b24cdf8a93> in <module>
     19
     20
---> 21 print(grad_test_jc(xc))

/usr/local/lib/python3.7/site-packages/jax/api.py in f_jitted(*args, **kwargs)
    105     _check_args(jaxtupletree_args)
    106     jaxtree_fun, out_tree = pytree_fun_to_jaxtupletree_fun(f, in_trees)
--> 107     jaxtupletree_out = xla.xla_call(jaxtree_fun, *jaxtupletree_args)
    108     return build_tree(out_tree(), jaxtupletree_out)
    109

/usr/local/lib/python3.7/site-packages/jax/core.py in call_bind(primitive, f, *args, **kwargs)
    543   if top_trace is None:
    544     with new_sublevel():
--> 545       ans = primitive.impl(f, *args, **kwargs)
    546   else:
    547     tracers = map(top_trace.full_raise, args)

/usr/local/lib/python3.7/site-packages/jax/interpreters/xla.py in xla_call_impl(fun, *args)
    452   fun, out_tree = flatten_fun(fun, in_trees)
    453
--> 454   compiled_fun = xla_callable(fun, *map(abstractify, flat_args))
    455   try:
    456     flat_ans = compiled_fun(*flat_args)

/usr/local/lib/python3.7/site-packages/jax/linear_util.py in memoized_fun(f, *args)
    206       if len(cache) > max_size:
    207         cache.popitem(last=False)
--> 208       ans = call(f, *args)
    209       cache[key] = (ans, f)
    210     return ans

/usr/local/lib/python3.7/site-packages/jax/interpreters/xla.py in xla_callable(fun, *abstract_args)
    473     jaxpr, (pval, consts, env) = pe.trace_to_subjaxpr(fun, master).call_wrapped(pvals)
    474     assert not env  # no subtraces here (though cond might eventually need them)
--> 475     compiled, result_shape = compile_jaxpr(jaxpr, consts, *abstract_args)
    476     del master, consts, jaxpr, env
    477   handle_result = result_handler(result_shape)

/usr/local/lib/python3.7/site-packages/jax/interpreters/xla.py in compile_jaxpr(jaxpr, const_vals, *abstract_args)
    135 def compile_jaxpr(jaxpr, const_vals, *abstract_args):
    136   arg_shapes = list(map(xla_shape, abstract_args))
--> 137   built_c = jaxpr_computation(jaxpr, const_vals, (), *arg_shapes)
    138   result_shape = xla_shape_to_result_shape(built_c.GetReturnValueShape())
    139   return built_c.Compile(arg_shapes, xb.get_compile_options(),

/usr/local/lib/python3.7/site-packages/jax/interpreters/xla.py in jaxpr_computation(jaxpr, const_vals, freevar_shapes, *arg_shapes)
    173             map(c.GetShape, map(read, const_bindings + freevar_bindings)),
    174             *in_shapes)
--> 175         for subjaxpr, const_bindings, freevar_bindings in eqn.bound_subjaxprs]
    176     subfuns = [(subc, tuple(map(read, const_bindings + freevar_bindings)))
    177                for subc, (_, const_bindings, freevar_bindings)

/usr/local/lib/python3.7/site-packages/jax/interpreters/xla.py in <listcomp>(.0)
    173             map(c.GetShape, map(read, const_bindings + freevar_bindings)),
    174             *in_shapes)
--> 175         for subjaxpr, const_bindings, freevar_bindings in eqn.bound_subjaxprs]
    176     subfuns = [(subc, tuple(map(read, const_bindings + freevar_bindings)))
    177                for subc, (_, const_bindings, freevar_bindings)

/usr/local/lib/python3.7/site-packages/jax/interpreters/xla.py in jaxpr_computation(jaxpr, const_vals, freevar_shapes, *arg_shapes)
    167   for eqn in jaxpr.eqns:
    168     in_nodes = map(read, eqn.invars)
--> 169     in_shapes = map(c.GetShape, in_nodes)
    170     subcs = [
    171         jaxpr_computation(

/usr/local/lib/python3.7/site-packages/jax/util.py in safe_map(f, *args)
     41   for arg in args[1:]:
     42     assert len(arg) == n, 'length mismatch: {}'.format(list(map(len, args)))
---> 43   return list(map(f, *args))
     44
     45

/usr/local/lib/python3.7/site-packages/jaxlib/xla_client.py in GetShape(self, operand)
    876
    877   def GetShape(self, operand):
--> 878     return _wrap_shape(self._builder.GetShape(operand))
    879
    880   def SetOpMetadata(self, op_metadata):

RuntimeError: Invalid argument: Binary op add with different element types: c64[3,3] and f32[1,3].

Jax built from source produced the same error.

Might not be related, but even without jit compilation and complex inputs, gradient computation of a function of eigenvectors fails.
import numpy as onp
import jax.numpy as np
from jax import grad

def test(x):
    val, vec = np.linalg.eigh(x)
    return np.real(np.sum(vec))

grad_test = grad(test)

x = onp.eye(3, dtype=onp.double)

print(test(x))
print(grad_test(x))

3.0
[[nan nan nan]
 [nan nan nan]
 [nan nan nan]]

PR #670 fixes the first bug; we were incorrectly declaring that the eigenvalues of a complex matrix were complex, leading to a type error when under a jit scope.
Not sure about the second bug yet.

I'm wondering if the second case is happening because one of the assumptions of the JVP rule for eigh is that the eigenvalues are distinct. In this case, all the eigenvalues are 1.
This is at the limits of my linear algebra knowledge, @mattjj do you have insights here?

See the discussion on the implementation here: https://github.com/google/jax/blob/master/jax/lax_linalg.py#L155

Wow, that's a nice comment! Thanks @levskaya.
I don't have any insights.  I think the case of repeated eigenvalues might come down to "contributions welcome".

Thanks @hawkinsp for the quick response and fix! Looks like repeated eigenvalues is probably the cause of the second bug
import numpy as onp
import jax.numpy as np
from jax import grad

def test(x):
    val, vec = np.linalg.eigh(x)
    return np.real(np.sum(vec))

grad_test = grad(test)

onp.random.seed(42)
x = onp.diag(
    onp.ones(3, dtype=onp.double) +
    onp.random.normal(0, 1e-6, size=3)
)

x2 = onp.diag(
    onp.ones(3, dtype=onp.double) +
    onp.random.normal(0, 1e-8, size=3)
)

print(grad_test(x))
print(grad_test(x2))

[[-5.9604641e-08  0.0000000e+00  0.0000000e+00]
 [ 0.0000000e+00 -5.9604641e-08  0.0000000e+00]
 [ 0.0000000e+00  0.0000000e+00  1.1920929e-07]]
[[nan nan nan]
 [nan nan nan]
 [nan nan nan]]

@sdaxen do you need autodiff of eigendecomposition with repeated eigenvalues? If not, we should probably close this issue until someone actually asks for it. (That way we can keep all the "enhancement" issues tracking things that users have specifically asked for.)

@mattjj no, it's not a priority. I do need repeated eigenvalues, but I'm only test driving jax for the moment while doing my main work with a different system. Feel free to close.

Thanks for the info!
We're very interested to hear about JAX's shortcomings so that we can work to fix them, so if there's something about JAX that makes it unsuitable for your work, please let us know!
@hawkinsp is it a better policy to close this issue, or leave it open and just keep in mind that we don't have any users specifically asking for it yet?

I think we should leave these kinds of issues open. It makes them more easily searchable should someone else have the same problem; I'd rather have one issue than two.

Cool, makes sense to me! At least we clarified how to prioritize it, then.

I also observe that TF has the same limitation: https://github.com/tensorflow/tensorflow/blob/f33aa592f92e233aeb00198d0caab80eaa89afe9/tensorflow/python/ops/linalg_grad.py#L314

Forgive my general lack of knowledge, I'm just beginning to look into jax/remind myself of some linear algebra. But on this issue:
import numpy as onp
import jax.numpy as np
from jax import grad

def test(x):
    val, vec = np.linalg.eigh(x)
    return np.real(np.sum(vec))

grad_test = grad(test)

x = onp.eye(3, dtype=onp.double)

print(test(x))
print(grad_test(x))

Is the derivative here even well defined? It seems to me like in the degenerate case, there is no unique eigenvector so the np.real(np.sum(vec)) could be a whole range of possible values depending on your choice of basis, no? This is interesting stuff, I'd be curious how you go about learning more.

As you note the general problem is quite tricky.  To use physics parlance, there are two cases: you can have a matrix w. degenerate eigenvalues where the perturbation (gradient direction) "breaks the symmetry" and causes the degenerate eigenvalues to split, and then you have the case where the perturbation preserves the degeneracy... which generally makes talking about eigenvector derivatives very tricky / ill-defined with simple approaches.  Especially if you're dealing with the general complex case where the eigenvector phase has additional freedom.  There are a few papers that seem to offer general algorithmic approaches, but they're complicated enough that no one has sat down to try to implement them to see how they'd work...

The hard case is differentiating eigenvectors in the presence of degeneracies. Eigenvalue derivatives are still fine, either way.
I believe my pull request #1665 actually has a working JVP (forward mode) gradient implementation for eigh with degeneracies, but it can't be transposed, which means it doesn't work for backward mode differentiation.
In general, I don't think it's possible to define backwards mode gradients of eigenvectors for arbitrary functions of degenerate eigenvectors -- the gradients simply don't always exist. I'll see if I can work-up a good counter-example.
From a practical perspective, it seems like the better idea is to differentiate a higher level function like a power-series that does always have a well defined derivatives. Typically your program that uses eigenvectors corresponding to degenerate eigenvalues is ultimately using eigenvectors to calculate something like this anyways, because otherwise it's output would depend on arbitrary choices from the linear algebra library.

@shoyer Just so I'm clear, you're saying that if we have a matrix A = P D P^-1, typically the reason that you would want to do the eigendecomposition is so that you can evaluation a function f(A) by doing f(A) = P f(D) P^-1 which is independent of the arbitrary choice of eigenvector?

@shoyer Just so I'm clear, you're saying that if we have a matrix A = P D P^-1, typically the reason that you would want to do the eigendecomposition is so that you can evaluation a function f(A) by doing f(A) = P f(D) P^-1 which is independent of the arbitrary choice of eigenvector (and other uses would be out of scope)?

That's exactly right. I hypothesize that you're calculating eigenvectors in order to evaluate a matrix-valued function of some form.

If so, I'm still confused as to how the np.sum operation that is at issue here would be defined?

The example in the first post was with eigenvalue directives. As noted in https://github.com/google/jax/issues/669#issuecomment-489328464, it's been fixed.

It's probably worth noting that the example failure case for eigenvector derivatives from https://github.com/google/jax/issues/669#issuecomment-489303348 is not well-defined matrix-valued function:
def test(x):
    val, vec = np.linalg.eigh(x)
    return np.real(np.sum(val))

E.g., suppose x = np.eye(2). Then normalized eigenvectors vec could be either [[1, 0], [0, 1]] or [[1/sqrt(2), 1/sqrt(2)], [1/sqrt(2), -1/sqrt(2)]], so test(x) could be either 2 or 2/sqrt(2).

FWIW, I originally encountered this while playing around with the matrix exponential of hermitian matrices, which is a power series function. For such functions that internally use the eigendecomposition, we can nevertheless write forward- and reverse-mode rules that almost completely account for the degeneracy of the eigendecomposition.

I recently stumbled across this paper, which seems to provide exactly the algorithm we need here:

Derivatives of partial eigendecomposition of a real symmetric matrix for degenerate cases

It looks like we could use the forward derivative for a JVP rule in JAX, which would suffice for auto-diff as long as we know how to implement and transpose a Sylvester solve (i.e., scipy.linalg.solve_sylvester).

It looks like we could use the forward derivative for a JVP rule in JAX, which would suffice for auto-diff as long as we know how to implement and transpose a Sylvester solve (i.e., scipy.linalg.solve_sylvester)

Shouldn't be too hard. The JVP for a Sylvester solve is just another Sylvester solve. If
X = sylvester_solve(A, B, Q), then Ẋ = sylvester_solve(A, B, Q̇ - Ȧ@X - X@Ḃ) (the solve is linear wrt its 3rd argument, so this is fine).
But Sylvester solves usually compute a Schur decomposition of A and B and then use a triangular Sylvester solve like LAPACK's trsyl, so this JVP would compute the Schur decompositions twice. It's more efficient to reuse the Schur decomposition, but I imagine in JAX that could pose problems with higher order AD? Or does JAX have a traceable Schur decomposition?

I spent some time going through the paper today, and as far as I can tell, this approach only handles exactly degenerate matrices, not almost-degenerate matrices, and in the case of exactly degenerate matrices, a Sylvester solver is not actually necessary and does not help things. Using their notation, for a standard eigenvalue problem M=I and M'=0.
In the paper's notation, the usual JVP in Giles' paper and elsewhere is
F_ij = {
    0                if i = j
    inv(λ_j - λ_i)   otherwise
}
K' = X^T A' X
Λ' = I ∘ K'
X' = X (F ∘ K')

After simplification, their contribution wrt degeneracy amounts to a simple modification to the matrix F:
F_ij = {
    0                if i = j
    0                if λ_i = λ_j and K'_ij = 0
    inv(λ_j - λ_i)   otherwise
}

When M=I and M'=0, their JVP simplifies to
K' = X^T A' X. # intermediate
Λ' = I ∘ K'
A Y' - Y'Λ = X (K' - (I ∘ K'))  # solve for Y' using sylvester_solve
X' = X (D ∘ (X^T Y') - X^T Y')

But we can take this further by multiplying both sides of Sylvester's equation by X^T. Then we have
let W' = X^T Y' and Z' = K' - (I ∘ K')
Λ W' - W' Λ = Z'  # sylvester's equation, but now the two matrices on left-hand side are diagonal
λ_i W'_ij - W'_ij λ_j = Z'_ij  #  the same equation in scalar form
W'_ij (λ_i - λ_j) = Z'_ij  # rearranged to a scalar sylvester's equation

This Sylvester appears in the standard derivation, the same as in Giles and elsewhere. We can solve for W'_ij = Z'_ij / (λ_i - λ_j), but this is what causes instability. Their approach does not handle this. It only handles the case where λ_i - λ_j = 0. In that case, for Z'_ij≠0, there are no solutions, and overflow is unavoidable. If Z'_ij=0, then W'_ij can be any finite number, and their approach would choose W'_ij=0 (this is what the elementwise product with D does).

So as far as I can tell, this does not resolve the issue of degeneracy, especially since even if a matrix is constructed to have exactly equal eigenvalues, due to floating point error, the eigenvalues will usually be nonequal. All it does is ensure that for certain programs, when a matrix with exact eigenvalues is factorized to produce exact eigenvalues, and when the tangent has a certain structure, that 0/0 does not happen. Which is useful but I think will only happen in very extreme and rare cases.
One last point is that this modification applies to the standard eigendecomposition as well, not just for symmetric or real matrices.

I recently stumbled across this paper, which seems to provide exactly the algorithm we need here:
* [Derivatives of partial eigendecomposition of a real symmetric matrix for degenerate cases](https://arxiv.org/abs/2011.04366)

I noticed the author of the paper, @mfkasim1 is on GitHub, and might be interested in this discussion.

Thanks for the tag. The same issue is also discussed in pytorch: https://github.com/pytorch/pytorch/issues/47599 with PR https://github.com/pytorch/pytorch/pull/50942.
I also implemented the algorithm in my library, xitorch, starting from this line: https://github.com/xitorch/xitorch/blob/a8ce4bf234fe839682586b7dc56f869ee1dc51d3/xitorch/linalg/symeig.py#L324 or this line for a simple eigendecomposition: https://github.com/xitorch/xitorch/blob/a8ce4bf234fe839682586b7dc56f869ee1dc51d3/xitorch/_impls/linalg/symeig.py#L47
The algorithm in the paper basically only works if the loss function does not depend directly on the degenerate eigenvectors, but it can depends on the space spanned by the degenerate eigenvectors.
Therefore, additional checks are necessary (eq. 2.8 for forward diff, 2.13-2.15 for backward diff).
If the additional checks are passed, then it also works for near-degenerate case.
However, I haven't thought about complex and non-symmetric matrix, so it might not work.
I have tried the algorithm in my differentiable density functional theory (DFT) simulation (there are a lot of degenerate eigenvalues) and it works nicely (i.e. it passes pytorch's gradcheck and gradgradcheck).

@mfkasim1 thanks for joining us!
I share @sethaxen's concern about almost degenerate eigenvalues. In such cases, the standard auto-diff rule for eigh can be numerically unstable, because it includes terms like 1/(λ_j - λ_i). Have you thought about this case, or perhaps it does not arise for your applications in density functional theory?

@shoyer The case in my DFT application is that it is supposed to have exactly the same eigenvalues theoretically, but numerically, the retrieved eigenvalues are only close to each other (near-degenerate).
In the near-degenerate case, the denominator (λ_j - λ_i) is close to 0, so unless you have near 0 nominator (i.e. eqs 2.8 and 2.13-2.15), then the numerical instability is unavoidable (just like tan(x)).
In the paper, I just consider the case where the nominator is supposed to be 0.
If it's not 0 (due to numerical error), it is assumed to be 0.
If you want to consider a case where the nominator is supposed to be a very small value but not 0, then this is not covered in the paper.
In my DFT case, this is sufficient, because the loss function does not depend directly on the degenerate eigenvectors (it depends on the space spanned by the eigenvectors), so the nominator (eq 2.13) is supposed to be 0.

I think it is fine to restrict the autodiff rules for eigh to the case where the output is invariant to the choice for basis within the degenerate subspace. If this isn't true, then the calculation isn't a well defined function (and would differ for different Lapack implementations).

I came across this conversation and wanted to leave this note as a reference for near-degenerate eigenvectors, specifically the transformation of Eq. 10 into Eq. 11 to account for near-degeneracy: https://github.com/mitmath/18335/blob/master/notes/adjoint/eigenvalue-adjoint.pdf . Hopefully you find it useful, though some translation may be in order.

(posting a comment from last week, that I thought I had already already submitted!)

Shouldn't be too hard. The JVP for a Sylvester solve is just another Sylvester solve. If
X = sylvester_solve(A, B, Q), then Ẋ = sylvester_solve(A, B, Q̇ - Ȧ@X - X@Ḃ) (the solve is linear wrt its 3rd argument, so this is fine).

Yes, indeed, this does look pretty straightforward!
(Note for Googlers: here's a chat thread that contains links to a prototype for a differentiable sylvester solve in JAX, written in terms of SciPy operations)

But Sylvester solves usually compute a Schur decomposition of A and B and then use a triangular Sylvester solve like LAPACK's trsyl, so this JVP would compute the Schur decompositions twice. It's more efficient to reuse the Schur decomposition, but I imagine in JAX that could pose problems with higher order AD? Or does JAX have a traceable Schur decomposition?

I think we can use the same trick we use for linear solves: compute the matrix factorization/decompositions first (without gradients), and then pass it into the auto-diff primitive:
https://github.com/google/jax/blob/6e1cd395e890b122008af2384b33c6aaa5374c28/jax/_src/lax/linalg.py#L262-L269
Technically, the auto-diff primitive becomes "Sylvester solve from a Schur decomposition" rather than "Sylvester solve from scratch".

(posting a comment from last week, that I thought I had already already submitted!)

Shouldn't be too hard. The JVP for a Sylvester solve is just another Sylvester solve. If

X = sylvester_solve(A, B, Q), then Ẋ = sylvester_solve(A, B, Q̇ - Ȧ@X - X@Ḃ) (the solve is linear wrt its 3rd argument, so this is fine).

Yes, indeed, this does look pretty straightforward!
(Note for Googlers: here's a chat thread that contains links to a prototype for a differentiable sylvester solve in JAX, written in terms of SciPy operations)

But Sylvester solves usually compute a Schur decomposition of A and B and then use a triangular Sylvester solve like LAPACK's trsyl, so this JVP would compute the Schur decompositions twice. It's more efficient to reuse the Schur decomposition, but I imagine in JAX that could pose problems with higher order AD? Or does JAX have a traceable Schur decomposition?

I think we can use the same trick we use for linear solves: compute the matrix factorization/decompositions first (without gradients), and then pass it into the auto-diff primitive:
https://github.com/google/jax/blob/6e1cd395e890b122008af2384b33c6aaa5374c28/jax/_src/lax/linalg.py#L262-L269
Technically, the auto-diff primitive becomes "Sylvester solve from a Schur decomposition" rather than "Sylvester solve from scratch".

Hi! I was wondering if there are any plans to implement the Sylvester solver in Jax, perhaps based on the draft implementation you mentioned? I would be interested in an implementation. Any tips or status update would be appreciated!

Any progress on this? Encountering problems with almost degenerate eigenvalues...

Hi, there is regularization technique to circumvent this degeneracy. You can refer those in docstring.
Am not an expert at math, but this seems to work so far.
code from my repo (https://github.com/kc-ml2/meent/blob/main/meent/on_jax/emsolver/primitives.py)
import jax
import jax.numpy as jnp

from functools import partial

def conj(arr):
    return arr.real + arr.imag * -1j
    # return arr.conj()

@partial(jax.custom_vjp, nondiff_argnums=(1, 2, 3))
def eig(x, type_complex=jnp.complex128, perturbation=1E-10, device='cpu'):

    _eig = jax.jit(jnp.linalg.eig, device=jax.devices('cpu')[0])

    eigenvalues_shape = jax.ShapeDtypeStruct(x.shape[:-1], type_complex)
    eigenvectors_shape = jax.ShapeDtypeStruct(x.shape, type_complex)

    result_shape_dtype = (eigenvalues_shape, eigenvectors_shape)
    if device == 'cpu':
        res = _eig(x)
    else:
        res = jax.pure_callback(_eig, result_shape_dtype, x)

    return res

def eig_fwd(x, type_complex, perturbation, device):
    return eig(x, type_complex, perturbation), (eig(x, type_complex, perturbation), x)

def eig_bwd(type_complex, perturbation, device, res, g):
    """
    Gradient of a general square (complex valued) matrix
    Eq 2~5 in https://www.nature.com/articles/s42005-021-00568-6
    Eq 4.77 in https://arxiv.org/pdf/1701.00392.pdf
    Eq. 30~32 in https://www.sciencedirect.com/science/article/abs/pii/S0010465522002715
    https://github.com/kch3782/torcwa
    https://github.com/weiliangjinca/grcwa
    https://github.com/pytorch/pytorch/issues/41857
    https://jax.readthedocs.io/en/latest/notebooks/autodiff_cookbook.html#Complex-numbers-and-differentiation
    https://discuss.pytorch.org/t/autograd-on-complex-numbers/144687/3
    """

    (eig_val, eig_vector), x = res
    grad_eigval, grad_eigvec = g

    grad_eigval = jnp.diag(grad_eigval)
    W_H = eig_vector.T.conj()

    Fij = eig_val.reshape((1, -1)) - eig_val.reshape((-1, 1))
    Fij = Fij / (jnp.abs(Fij) ** 2 + perturbation)
    Fij = Fij.at[jnp.diag_indices_from(Fij)].set(0)

    # diag_indices = jnp.arange(len(eig_val))
    # Eij = eig_val.reshape((1, -1)) - eig_val.reshape((-1, 1))
    # Eij = Eij.at[diag_indices, diag_indices].set(1)
    # Fij = 1 / Eij
    # Fij = Fij.at[diag_indices, diag_indices].set(0)

    grad = jnp.linalg.inv(W_H) @ (grad_eigval.conj() + Fij * (W_H @ grad_eigvec.conj())) @ W_H
    grad = grad.conj()
    if not jnp.iscomplexobj(x):
        grad = grad.real

    return grad,

eig.defvjp(eig_fwd, eig_bwd)

Hi, there is regularization technique to circumvent this degeneracy. You can refer those in docstring. Am not an expert at math, but this seems to work so far.
code from my repo (https://github.com/kc-ml2/meent/blob/main/meent/on_jax/emsolver/primitives.py)
import jax
import jax.numpy as jnp

from functools import partial

def conj(arr):
    return arr.real + arr.imag * -1j
    # return arr.conj()

@partial(jax.custom_vjp, nondiff_argnums=(1, 2, 3))
def eig(x, type_complex=jnp.complex128, perturbation=1E-10, device='cpu'):

    _eig = jax.jit(jnp.linalg.eig, device=jax.devices('cpu')[0])

    eigenvalues_shape = jax.ShapeDtypeStruct(x.shape[:-1], type_complex)
    eigenvectors_shape = jax.ShapeDtypeStruct(x.shape, type_complex)

    result_shape_dtype = (eigenvalues_shape, eigenvectors_shape)
    if device == 'cpu':
        res = _eig(x)
    else:
        res = jax.pure_callback(_eig, result_shape_dtype, x)

    return res

def eig_fwd(x, type_complex, perturbation, device):
    return eig(x, type_complex, perturbation), (eig(x, type_complex, perturbation), x)

def eig_bwd(type_complex, perturbation, device, res, g):
    """
    Gradient of a general square (complex valued) matrix
    Eq 2~5 in https://www.nature.com/articles/s42005-021-00568-6
    Eq 4.77 in https://arxiv.org/pdf/1701.00392.pdf
    Eq. 30~32 in https://www.sciencedirect.com/science/article/abs/pii/S0010465522002715
    https://github.com/kch3782/torcwa
    https://github.com/weiliangjinca/grcwa
    https://github.com/pytorch/pytorch/issues/41857
    https://jax.readthedocs.io/en/latest/notebooks/autodiff_cookbook.html#Complex-numbers-and-differentiation
    https://discuss.pytorch.org/t/autograd-on-complex-numbers/144687/3
    """

    (eig_val, eig_vector), x = res
    grad_eigval, grad_eigvec = g

    grad_eigval = jnp.diag(grad_eigval)
    W_H = eig_vector.T.conj()

    Fij = eig_val.reshape((1, -1)) - eig_val.reshape((-1, 1))
    Fij = Fij / (jnp.abs(Fij) ** 2 + perturbation)
    Fij = Fij.at[jnp.diag_indices_from(Fij)].set(0)

    # diag_indices = jnp.arange(len(eig_val))
    # Eij = eig_val.reshape((1, -1)) - eig_val.reshape((-1, 1))
    # Eij = Eij.at[diag_indices, diag_indices].set(1)
    # Fij = 1 / Eij
    # Fij = Fij.at[diag_indices, diag_indices].set(0)

    grad = jnp.linalg.inv(W_H) @ (grad_eigval.conj() + Fij * (W_H @ grad_eigvec.conj())) @ W_H
    grad = grad.conj()
    if not jnp.iscomplexobj(x):
        grad = grad.real

    return grad,

eig.defvjp(eig_fwd, eig_bwd)

This code works fine! BTW, I'm pondering is there any "jvp"-version. (Cause I'm working on a frame only supporting forward-auto-diff)

I am also running into issues with needing degenerate eigenvalues for the computation of molecular energies using wavefunction methods. Degenerate eigenvalues are quite common in quantum chemistry as evidenced by Kasim's work on differentiable density functional theory and if you just think of the symmetry present in many molecules.
Is the fact that backward-mode AD can't handle degenerate eigenvalues a hindrance to implementing the forward-mode JVP?


[SAMZA-2786] remove the docs gradle project

As explained in SAMZA-2786 a missing dependency used by the docs gradle project is blocking builds.  This PR will remove references to the docs gradle project until we figure out how to resolve the dependency problem.

It appears there was an issue with artifact downloads from Jcenter.
https://github.com/jruby-gradle/jruby-gradle-plugin/issues/445
I just ran ./gradlew docs:jekyllBuild and ./gradlew docs:jekyllServeLocal and was able to load Samza documentation at localhost.  Closing this as we don't need to remove the docs project just yet.
Should it be an issue in the future, we probably want to write a task that bypasses the jruby-gradle plugin by creating a task to download jruby & run locally with a Gem file for dependency management.

Small comment to folks who, like me, are first time importers of the Samza project.
If you clone the directory and attempt to load it as a Gradle project into IntelliJ, it seems that this jruby dependency still blocks the entire codebase from being correctly loaded by the IDE.
This PR comes up as the top result on googling the error.
On attempting to apply the fixes in the PR, the Gradle project import completed successfully.
Hope this helps anyone else who faces this issue.


Project Configuration UI

Resolves #57. Resolves #58.
This PR adds a Project Configuration UI and improves the sidebar UI according to these Figma Mockups by @faiza-a.
Demo video on Drive:
https://drive.google.com/file/d/1M0vY9fUM-gY7AXQXNeyzN7kHj6TwVvRF/view?usp=sharing

The lint failures above are due to an unrelated problem; I have created #82 to resolve it.


QUESTION: Top Nav Menu

Can I change the layout options, to only one page of my website?

Probably yes, since you can change configuration at runtime on Laravel. You can change the configuration on the controller, before loading (rendering) your view, for example:
config([
    'adminlte.layout_topnav' => true,
    'adminlte.layout_fixed_navbar' => true,
]);

I'm not sure if the changes will persists for every request, in that case you will need to restore the configuration value for all the other views.

Oh that's pretty cool, but how can I change this at my controller? only add this code? or I'll need change more things?

Oh! that's look good! Thanks! :)


Minor improvement to duration checks

Put duration checks into its own OpenFermion-Cirq function.

I'm not sure about this change ... it seems to be implying that there is a unit called "duration" which is equal to half of a rad. That sounds like nonsense honestly. In reality we are defining "duration" on an ad hoc basis, which just happens to have the same formula in the cases implemented so far.

I'd say degrees and rads make less sense when it comes to gates defined by evolution!
I'm trying to better understand your concern though - is it that we have to (artificially?) align or rescale the generator of evolution to make this work? I think that might actually be net helpful, and we already do it for e.g. XXYYGate. Having both exp(-i rads (X⊗X + Y⊗Y) / 4) and exp(-i t (X⊗X + Y⊗Y) / 2) gives different consistent definitions that are more approachable to different people.

Maybe call it half_rads instead of duration? Then in the documentation, we can make clear that "half_rads" is what most people what think of as theta.

@kevinsung its been a month: what's going on with this PR? Should we merge it or close it or are we waiting for something to happen?

Closing for now, since this will be obsoleted when we get rid of half_turns (see https://github.com/quantumlib/Cirq/issues/865)


Notify of referral programme proposal in  toast

Story
As a console user
I want to know when there is a proposed update to the referral programme or volume discount programme
So that I go find out more about it and vote either for or against it
Acceptance Criteria

[ ] I can see a toast when a proposal for an update to the referral programme has been made
[ ] There is a way to go to see the proposal to vote i.e. a link to the proposal on the governance dapp

Tasks

[ ] Specs reviewed
[ ] Implementation
[ ] Testing (unit and/or e2e)
[ ] Code review
[ ] QA review

Sketch

Additional details / background info

Spec: https://github.com/vegaprotocol/specs/pull/1829
There is ONE new type of governance proposal for these referral programs (covering both create / update)
This task simply needs to ensure that we have toasts that can handle this type of governance proposal

I am not sure this is a ticket....
Referral programs are governance proposals so existing toasts will cover them.  Not sure we need anything else.

Covered by another ticket now


Update coverage threshold script

What
What does this PR add or remove?
Why
Why are these changes needed?
How
How do these changes achieve the goal?
Test
How has this been tested? How can a reviewer test it?
Checklist

[ ] I have self-reviewed this PR
[ ] I have added tests that prove the feature works or the fix is effective

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.1 out of 2 committers have signed the CLA.:white_check_mark: maggie98choy:x: github-actions[bot]You have signed the CLA already but the status is still pending? Let us recheck it.


Unable to store signature.

Unable to Store Signature and Unable to store svg signature these two messages are populating when I am trying to save the image.

Hi,
could you be more specific, maybe pasting a stack trace and Android version?

Unable to store signature message is populating when i'am trying hit the save button. How to resolve this issue?


ci: Stop rabbitmq when making CI images.

The production CI image starts rabbitmq-server but does not stop it,
which leaves a stale /var/run/rabbitmq/pid file in the image.
rabbitmqctl wait --timeout 600 /var/run/rabbitmq/pid, which is run
after starting the rabbitmq node, reads the PID file and waits for the
PID to be running, and for rabbitmq's port to be responding to pings.
If it reads an old PID file before the new PID is written, it
aborts (all but the first and last lines are output from rabbitmqctl wait that is hidden by /etc/init.d/rabbitmq-server):
 * Starting RabbitMQ Messaging Server rabbitmq-server
Waiting for pid file '/var/run/rabbitmq/pid' to appear
pid is 341
Waiting for erlang distribution on node 'rabbit@fc8f64d6acdb' while OS process '341' is running
Error:
process_not_running
 * FAILED - check /var/log/rabbitmq/startup_\{log, _err\}

If it failed, the production-upgrade script tried to start
rabbitmq again -- despite it already still starting in the
background.  These two attempts conflicted, and often one or both
failed.
Stop rabbitmq-server when building the image, which removes the
stale PID file.

Self-review checklist

[x] Self-reviewed the changes for clarity and maintainability
(variable names, code reuse, readability, etc.).

Communicate decisions, questions, and potential concerns.

[ ] Explains differences from previous plans (e.g., issue description).
[ ] Highlights technical choices and bugs encountered.
[ ] Calls out remaining decisions and concerns.
[ ] Automated tests verify logic where appropriate.

Individual commits are ready for review (see commit discipline).

[x] Each commit is a coherent idea.
[x] Commit message(s) explain reasoning and motivation for changes.

Completed manual review and testing of the following:

[ ] Visual appearance of the changes.
[ ] Responsiveness and internationalization.
[ ] Strings and tooltips.
[ ] End-to-end functionality of buttons, interactions and flows.
[ ] Corner cases, error conditions, and easily imagined bugs.

Merged, thanks for debugging this!


docs: Add cmake and ninja-build to Linux packages

These packages are required to run the build instructions in the doc.

Thanks! merging once the presubmit are finished.


Data types not validated

Hello,
Here's some code:
var orm = require('orm');

orm.connect('mongodb://localhost/dummy', function (err, db) {
  if (err) throw err;

    var Person = db.define('person', {
        name: String
    });

    var person = new Person({
    	name: 1234
    });

    person.validate(function (err, errors) {
    	if (err || errors) {
    		throw new Error('validation failed');
    	}

    	person.save();
    });
});

I expected to see validate and save both fail, but that's not the case. After running this code, my MongoDB database has a new object with an Integer field for name with the value 1234.
Is this a bug? If not, is there a built-in way for enforcing types?
Thanks!

You need to manually add validators:
var Person = db.define('person', {
  name: String
}, {
  validations: {
    name: function (data, next, thisArg, contexts) {
      if (typeof data != 'string') {
        return next(new Error("%s is not a valid type for the field 'name'!", typeof data));
      }
      next();
    }
  }
});


bump dependencies

Applied mkpr on package.json
/tmp/mkpr.sh

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Details

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 90.137%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
578

Relevant Lines:
633

💛  - Coveralls


Multiple 'rest of world' shipping options not working

The user will only be presented with one of the 'rest of world' options on the frontend checkout

Hi @nick,
What I've learned so far is that when a shop admin saves a form under Admin > Settings > Shipping with the 'Rest of the world' option set, a shipping.json file is generated/updated on the backend. For example:
[
  {
    "countries": [
      "US"
    ],
    "label": "",
    "processingTime": "",
    "detail": "",
    "amount": 0,
    "rates": [
      {
        "type": "free",
        "amount": 0
      },
      {
        "type": "fast",
        "amount": 1000
      }
    ]
  },
  {
    "label": "Free Shipping",
    "detail": "Arrives in 10 to 14 days",
    "amount": 0,
    "rates": [
      {
        "type": "slow",
        "amount": 2000
      },
      {
        "type": "fast",
        "amount": 4000
      },
      {
        "type": "expedited",
        "amount": 8000
      }
    ]
  },
  {
    "countries": [
      "AF"
    ],
    "label": "",
    "processingTime": "",
    "detail": "",
    "amount": 0,
    "rates": [
      {
        "type": "free",
        "amount": 0
      },
      {
        "type": "slow",
        "amount": 2000
      },
      {
        "type": "fast",
        "amount": 4000
      }
    ]
  }
]

In this example, the middle element refers to 'Rest of the world'. Given that the key countries is not populated here, I'm considering updating a file under https://github.com/OriginProtocol/dshop/tree/master/shop/src/pages/admin/settings/shipping to hopefully fix this issue. Would you please leave a comment if you believe the source of the issue lies elsewhere?

Hi again @nick,
I'm cc'ing @shahthepro because according to Github, they share authorship of the file https://github.com/OriginProtocol/dshop/blob/master/shop/src/pages/admin/settings/shipping/Edit.js with you.
In looking at this issue from the point of view described above, I suggest editing the onChange prop of the above file to make the fields in shipping.json consistent. Particularly, when a shop is newly built,
the contents of shipping.json is as follows:
[
  {
    "id": "international",
    "label": "Free Shipping",
    "detail": "Arrives in 10 to 14 days",
    "amount": 0
  }
]

Correct me if I'm wrong, but as I understand, when an Admin updates the Shipping settings for the first time, the "source country" object is moved to the top. If they choose to add a second destination, the details of that are combined into the object above.
The result here is a JSON file whose contents may look like:
[
  {
    "countries": [
      "US"
    ],
    "label": "",
    "processingTime": "",
    "detail": "",
    "amount": 0,
    "rates": [
      {
        "type": "free",
        "amount": 0
      },
      {
        "type": "expedited",
        "amount": 2000
      }
    ]
  },
  {
    "countries": [
      "AX"
    ],
    "label": "Free Shipping",
    "detail": "Arrives in 10 to 14 days",
    "amount": 0,
    "rates": [
      {
        "type": "fast",
        "amount": 4000
      },
      {
        "type": "slow",
        "amount": 2000
      }
    ]
  }
]

From my testing, I saw that the fields
    "label": "Free Shipping",
    "detail": "Arrives in 10 to 14 days",

are displayed during checkout only when the shop is new, and shipping settings are not yet configured by the Admin.


[HELP] how to add Anti-aim chams ?

the title said it.

Coded. 😃

Coding. smiley

"My Goodness Why Didn't I Think of That"

lul..........

Coding. smiley

"My Goodness Why Didn't I Think of That"

What do you think people will said?

i agree ^


[THREESCALE-10278] upgrade lua-resty-http to 0.17.1

What
Fix https://issues.redhat.com/browse/THREESCALE-10278
lua-resty-http 0.17.1 is also required for https://issues.redhat.com/browse/THREESCALE-5105
This PR is mainly a refactoring of existing code so no additional integration tests/unittests are added.
Verification Steps

Connect via proxy

Build docker image from this git branch

make runtime-image IMAGE_NAME=apicast-test

Run proxy dev environment

cd dev-environments/https-proxy-upstream-tlsv1.3
make certs
make gateway IMAGE_NAME=apicast-test

Send request to APIcast

curl --resolve get.example.com:8080:<IP_ADDRESS> -v "http://get.example.com:8080/?user_key=123"

Request should return 200

* Added get.example.com:8080:<IP_ADDRESS> to DNS cache
* Hostname get.example.com was found in DNS cache
*   Trying <IP_ADDRESS>...
* TCP_NODELAY set
* Connected to get.example.com (<IP_ADDRESS>) port 8080 (#0)
> GET /?user_key=123 HTTP/1.1
> Host: get.example.com:8080
> User-Agent: curl/7.61.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: application/json
< Transfer-Encoding: chunked
< Connection: keep-alive
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Credentials: true
< Date: Tue, 30 Jan 2024 04:53:30 GMT
< Server: gunicorn/19.9.0
<
{
  "args": {
    "user_key": "123"
  },
  "headers": {
    "Accept": "*/*",
    "Host": "example.com",
    "User-Agent": "curl/7.61.1"
  },
  "origin": "<IP_ADDRESS>",
  "url": "http://example.com/get?user_key=123"
}
* Connection #0 to host get.example.com left intact

Proxy should receive a CONNECT method with upstream hostname not IP:PORT

docker compose -p https-proxy-upstream-tlsv13 logs -f proxy

proxy-1  | CONNECT   Jan 30 04:55:57.831 [1]: Connect (file descriptor 4): <IP_ADDRESS>
proxy-1  | CONNECT   Jan 30 04:55:57.831 [1]: Request (file descriptor 4): CONNECT example.com:443 HTTP/1.1

Fetching configuration file from 3scale

Start dev environment

make development
make dependencies

Run APIcast locally

THREESCALE_DEPLOYMENT_ENV=staging APICAST_LOG_LEVEL=debug APICAST_WORKER=1 APICAST_CONFIGURATION_LOADER=lazy APICAST_CONFIGURATION_CACHE=0<EMAIL_ADDRESS>./bin/apicast

Run query

# capture apicast IP
APICAST_IP=$(docker inspect apicast_build_0-development-1 | yq e -P '.[0].NetworkSettings.Networks.apicast_build_0_default.IPAddress' -)

curl -i -k -H "Host: default-product.staging.example.com:443" "http://${APICAST_IP}:8080/?user_key=<user_key>"

Replace <user_key> with the actual user key.  The response should be HTTP/1.1 200 OK
HTTP/1.1 200 OK
Server: openresty
Date: Tue, 30 Jan 2024 05:08:26 GMT
Content-Type: application/json
Content-Length: 702
Connection: keep-alive
x-3scale-echo-api: echo-api/1.0.3
vary: Origin
x-content-type-options: nosniff
x-envoy-upstream-service-time: 0

From the APIcast log, the request to fetch configuration file return 200
2024/02/14 01:42:12 [debug] 763706#763706: *2 remote_v2.lua:268: proxy_configs_per_page(): proxy configs get status: 200 url: https://user_key@3scale-admin.example.con/admin/api/account/proxy_configs/production.json?host=default-product.staging.example.com&page=1&per_page=500&version=latest body: {"proxy_configs":  .... }

{
    "services": [
        {
            "proxy": {
                "error_headers_auth_failed": "text\/plain; charset=us-ascii",
                "error_headers_limits_exceeded": "text\/plain; charset=us-ascii",
                "error_headers_auth_missing": "text\/plain; charset=us-ascii",
                "error_headers_no_match": "text\/plain; charset=us-ascii",
                "error_status_no_match": 404,
                "error_status_auth_failed": 403,
                "error_status_limits_exceeded": 429,
                "error_status_auth_missing": 403,
                "secret_token": "Shared_secret_sent_from_proxy_to_API_backend_09df7e84d9ba36d8",
                "hostname_rewrite": null,
                "oidc_issuer_endpoint": null,
                "jwt_claim_with_client_id": null,
                "jwt_claim_with_client_id_type": null,
                "auth_user_key": "user_key",
                "auth_app_id": "app_id",
                "auth_app_key": "app_key",
                "oauth_login_url": null,
                "proxy_rules": [
                    {
                        "http_method": "GET",
                        "pattern": "\/",
                        "delta": 1,
                        "redirect_url": null,
                        "querystring_parameters": {},
                        "position": 1,
                        "parameters": {},
                        "metric_system_name": "hits",
                        "last": false,
                        "owner_type": "Proxy"
                        ...
                    }
                ],
                "error_auth_missing": "Authentication parameters missing",
                "api_test_path": "\/",
                "api_test_success": null,
                "apicast_configuration_driven": true,
                "oidc_issuer_type": "keycloak",
                "staging_domain": "default-product.staging.example.com",
                "production_domain": "default-product.production.example.com",
                "endpoint": "https:\/\/default-product.production.example.com: 443",
                "error_limits_exceeded": "Usage limit exceeded",
                "deployed_at": null,
                "backend": {
                    "endpoint": "https:\/\/su1.3scale.net",
                    "host": "su1.3scale.net"
                },
                "error_no_match": "No Mapping Rule matched",
                "valid?": true,
                "service_backend_version": "1",
                "hosts": [
                    "default-product.production.example.com",
                    "default-product.staging.example.com"
                ],
                "error_auth_failed": "Authentication failed",
                "lock_version": 1,
                "policy_chain": [
                    {
                        "name": "apicast",
                        "configuration": {},
                        "version": "builtin"
                    }
                ],
                "endpoint_port": 443,
                "sandbox_endpoint": "https:\/\/default-product.staging.example.com: 443",
                "authentication_method": "1",
                "api_backend": "https:\/\/echo-api.3scale.net: 443",
                "credentials_location": "query",
                "hostname_rewrite_for_sandbox": "echo-api.3scale.net"
            },
            "backend_authentication_type": "service_token",
            "description": "",
            "name": "Default Product",
            "backend_version": "1",
            "proxiable?": true,
            "system_name": "default_product",
            ...
        },
    ]
}

resty-http was downgraded to 0.15 for ARM support https://github.com/3scale/APIcast/commit/6a09893a7277225542a040afd62c35f5c55e40ee
Can we test that this image can be built out of this PR using ARM arch host? If you do not have available, I can ask some other member of the team to test it.

So I tried to build ARM image using docker from masterbranch
 ▲ APIcast make dev-build
/usr/bin/docker buildx build --platform linux/arm64 -t apicast-development:latest \
        --build-arg OPENRESTY_RPM_VERSION=1.19.3 \
        --build-arg LUAROCKS_VERSION=2.3.0 \
        /3scale/APIcast -f Dockerfile.devel

 => [ 8/16] RUN yum config-manager --add-repo http://packages.dev.3sca.net/dev_packages_3sca_net.repo                                                                                3.7s
 => ERROR [ 9/16] RUN yum install -y         openresty-1.19.3         openresty-resty-1.19.3         openresty-opentelemetry-1.19.3         openresty-opentracing-1.19.3         opentracing-cpp-devel-1.3.0      6.9s
------
 > [ 9/16] RUN yum install -y         openresty-1.19.3         openresty-resty-1.19.3         openresty-opentelemetry-1.19.3         openresty-opentracing-1.19.3         opentracing-cpp-devel-1.3.0         libopentraci
cpp1-1.3.0         jaegertracing-cpp-client-0.3.1-13.el8:
4.901 Devel packages from 3Scale                       81 kB/s | 220 kB     00:02
6.661 Error:
6.661  Problem 1: conflicting requests
6.661   - package jaegertracing-cpp-client-0.3.1-13.el8.x86_64 from packages.dev.3sca.net does not have a compatible architecture
6.661   - nothing provides libpthread.so.0(GLIBC_2.2.5)(64bit) needed by jaegertracing-cpp-client-0.3.1-13.el8.x86_64 from packages.dev.3sca.net
6.661   - nothing provides libpthread.so.0(GLIBC_2.3.2)(64bit) needed by jaegertracing-cpp-client-0.3.1-13.el8.x86_64 from packages.dev.3sca.net
6.661   - nothing provides libc.so.6(GLIBC_2.14)(64bit) needed by jaegertracing-cpp-client-0.3.1-13.el8.x86_64 from packages.dev.3sca.net
6.661  Problem 2: cannot install the best candidate for the job
6.661   - package openresty-1.19.3-23.el8.x86_64 from packages.dev.3sca.net does not have a compatible architecture
6.661   - nothing provides openresty-pcre >= 8.42-1 needed by openresty-1.19.3-23.el8.x86_64 from packages.dev.3sca.net
6.661   - nothing provides openresty-zlib >= 1.2.11-3 needed by openresty-1.19.3-23.el8.x86_64 from packages.dev.3sca.net
6.661  Problem 3: cannot install the best candidate for the job
6.661   - nothing provides openresty >= 1.19.3-23.el8 needed by openresty-resty-1.19.3-23.el8.noarch from packages.dev.3sca.net
6.661  Problem 4: cannot install the best candidate for the job
6.661   - package openresty-opentelemetry-1.19.3-23.el8.x86_64 from packages.dev.3sca.net does not have a compatible architecture
6.661   - nothing provides libpthread.so.0(GLIBC_2.2.5)(64bit) needed by openresty-opentelemetry-1.19.3-23.el8.x86_64 from packages.dev.3sca.net
6.661   - nothing provides libpthread.so.0(GLIBC_2.3.2)(64bit) needed by openresty-opentelemetry-1.19.3-23.el8.x86_64 from packages.dev.3sca.net
6.661   - nothing provides ld-linux-x86-64.so.2()(64bit) needed by openresty-opentelemetry-1.19.3-23.el8.x86_64 from packages.dev.3sca.net
6.661   - nothing provides ld-linux-x86-64.so.2(GLIBC_2.3)(64bit) needed by openresty-opentelemetry-1.19.3-23.el8.x86_64 from packages.dev.3sca.net
6.661   - nothing provides libm.so.6(GLIBC_2.2.5)(64bit) needed by openresty-opentelemetry-1.19.3-23.el8.x86_64 from packages.dev.3sca.net
6.661   - nothing provides libpthread.so.0(GLIBC_2.12)(64bit) needed by openresty-opentelemetry-1.19.3-23.el8.x86_64 from packages.dev.3sca.net
6.661  Problem 5: cannot install the best candidate for the job
6.661   - package openresty-opentracing-1.19.3-23.el8.x86_64 from packages.dev.3sca.net does not have a compatible architecture
6.661   - nothing provides libc.so.6(GLIBC_2.14)(64bit) needed by openresty-opentracing-1.19.3-23.el8.x86_64 from packages.dev.3sca.net
6.661   - nothing provides libopentracing.so.1()(64bit) needed by openresty-opentracing-1.19.3-23.el8.x86_64 from packages.dev.3sca.net
6.661  Problem 6: cannot install the best candidate for the job
6.661   - package opentracing-cpp-devel-1.3.0-26.el8arches.x86_64 from packages.dev.3sca.net does not have a compatible architecture
6.661   - nothing provides libopentracing.so.1()(64bit) needed by opentracing-cpp-devel-1.3.0-26.el8arches.x86_64 from packages.dev.3sca.net
6.661   - nothing provides libopentracing_mocktracer.so.1()(64bit) needed by opentracing-cpp-devel-1.3.0-26.el8arches.x86_64 from packages.dev.3sca.net
6.661   - nothing provides libopentracing-cpp1 = 1.3.0-26.el8arches needed by opentracing-cpp-devel-1.3.0-26.el8arches.x86_64 from packages.dev.3sca.net
6.661  Problem 7: cannot install the best candidate for the job
6.661   - package libopentracing-cpp1-1.3.0-26.el8arches.x86_64 from packages.dev.3sca.net does not have a compatible architecture
6.661   - nothing provides libc.so.6(GLIBC_2.14)(64bit) needed by libopentracing-cpp1-1.3.0-26.el8arches.x86_64 from packages.dev.3sca.net
6.661   - nothing provides ld-linux-x86-64.so.2()(64bit) needed by libopentracing-cpp1-1.3.0-26.el8arches.x86_64 from packages.dev.3sca.net
6.661   - nothing provides ld-linux-x86-64.so.2(GLIBC_2.3)(64bit) needed by libopentracing-cpp1-1.3.0-26.el8arches.x86_64 from packages.dev.3sca.net
6.661   - nothing provides libdl.so.2(GLIBC_2.2.5)(64bit) needed by libopentracing-cpp1-1.3.0-26.el8arches.x86_64 from packages.dev.3sca.net
6.662 (try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
------
Dockerfile.devel:30

Checking http://packages.dev.3sca.net/ I don't see any aarch64 packages. Perhaps you have a better way to build the image?

I’ve tried this but the verification steps won’t work OOTB on my system, due to missing RPMs for arm64 (aarch64). The packages we need (i.e. OpenResty 1.19.3-x and related ones within the same “family”, e.g. OpenTelemetry, OpenTracing, etc) are not available for this arch, neither in the default repos, nor in http://packages.dev.3sca.net/.
Searching my notes after #1381, I found out a few interesting things. Starting with the fact that, back then, I only touched Dockerfile.devel, but never Dockerfile. This means I was able to build the devel/ci container image on darwin/arm64, but never the runtime image, which is the one we use to build for prod IIRC.
Another important piece is that apparently I failed then to build the devel/ci image while targeting linux/arm64 platform. What I have succeeded doing was building on darwin/arm64 for linux/amd64.
To not completely diminish that as 100% useless, I recon it may have helped people to boot up the dev env container and run the test suite on a MacOS with Mx chip. But, in the end, we were just working around the limitations for running an arm64 image on an arm64 platform for dev purposes, but that’s all.

Back to this PR... While still unable to make runtime-image, there was only so much I could do. Here’s a summary:
1. Smoke tests (devel/ci image)
make dev-build IMAGE=quay.io/3scale/apicast-ci:openresty-1.19.3-pr1434-amd64
make development IMAGE=quay.io/3scale/apicast-ci:openresty-1.19.3-pr1434-amd64

Then, inside the development container:
make dependencies
make busted
make prove

Result: SUCCESS
2. “Connect via Proxy” (runtime image, Proxy w/ upstream using TLSv1.3)
make runtime-build # <========= FAILED
cd dev-environments/https-proxy-upstream-tlsv1.3
make certs
make gateway
curl --resolve get.example.com:8080:<IP_ADDRESS> -v "http://get.example.com:8080/?user_key=123"
docker compose -p https-proxy-upstream-tlsv13 logs -f proxy

Result: FAILED
3. “Fetch config from 3scale” (devel/ci image)
Inside the development container:
THREESCALE_DEPLOYMENT_ENV=staging APICAST_LOG_LEVEL=debug APICAST_WORKER=1 APICAST_CONFIGURATION_LOADER=lazy APICAST_CONFIGURATION_CACHE=0<EMAIL_ADDRESS>./bin/apicast

From the host:
APICAST_IP=$(docker inspect apicast_build_0-development-1 | yq e -P '.[0].NetworkSettings.Networks.apicast_build_0_default.IPAddress' -)
curl -i -k -H "Host: example.com:443" -H "Accept: application/json" -H "Authorization: Bearer ${ACCESS_TOKEN}" "http://${APICAST_IP}:8080/foo"

Output:
HTTP/1.1 404 Not Found
Server: openresty
Date: Thu, 01 Feb 2024 10:56:20 GMT
Content-Type: text/plain
Transfer-Encoding: chunked
Connection: keep-alive

Result: SUCCESS (?)

Thanks @guicassolato, for the last test, did you replace the placeholder values with actual 3scale values? It may also fail if you are using an older version of lua-resty-http (0.15), maybe delete the lua_modules folder and rerun make dependencies?
@eguzki so I guess we are good to merge this one?

building on darwin/arm64 for linux/amd64

Thanks @guicassolato
That was what I was missing. I was wondering how the hell you build dev image. building on darwin/arm64 for linux/amd64 answers that.
The aim is to allow arm64 based users to develop APIcast. If they cannot build  runtime image for arm64, that's unfortunate but not a blocking issue if they can still build and run amd64 images.

There is a failing check about codecov. It says that the tests only cover the 72% of the added code in this PR (not the overall code coverage). While we could be more restrictive about this, I tend to think that 70% of code coverage either for the patch or the project is good enough and should not be blocking the merge.

codecov is a strange one. It's fine now :sweat_smile:

Rebased!

@tkan145 the second part of the verification steps cannot be run. You did not specify the 3scale configuration fetched from 3scale API, hence I cannot reproduce.
Nevertheless, I consider the first part good enough. And together with the (old) e2e tests passing, I consider the PR tested.

Updated verification steps.
Because this has been approved. I will merge now


Update theano Op import.

theano.Op -> Op + from theano.graph.op import Op
This was needed to run the library on a fresh Anaconda install using a M1 (macOS).
Note to reviewer: please take this CL with a grain of salt. I am not at all familiar Theano and have not tested this carefully.

BUG=https://github.com/ColCarroll/imcmc/issues/7


Fix the EXPECT_EQ parameter's order

EXPECT_EQ is expecting "expected_value, actual_value". But code is having the parameters in reverse order. Fix is to swap the parameter's order.

#101 - PR reference


[Improvement][Hive]: When I set base.hive.auto-sync-data-write is true, optimize task executes slowly

Search before asking

[X] I have searched in the issues and found no similar issues.

What would you like to be improved?
I set base.hive.auto-sync-data-write is true.
The arctic table will refresh when optimize plan starts, there is hive data sync work needed to do, and this costs a lot of time.
Now, the optimize plan is a single thread, so costing too much time in the optimize plan will block the optimizer poll task.
How should we improve?
No response
Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

Subtasks
No response
Code of Conduct

[X] I agree to follow this project's Code of Conduct

@hzluting Can you provide the change in time taken by optimizing plan with this parameter on?
The check process will go through the properties of all the hive partitions, which will be somewhat expensive, but I'm not sure how much extra overhead it will incur in a real-world scenario.

@hzluting Can you provide the change in time taken by optimizing plan with this parameter on?
The check process will go through the properties of all the hive partitions, which will be somewhat expensive, but I'm not sure how much extra overhead it will incur in a real-world scenario.

OK, I have an optimizing group that contains 4 tables, and a table has more than 700 optimizing tasks. Once I set "base.hive.auto-sync-data-write" to be true for all 4 tables, this table can't do optimizing anymore. And I print the plan thread stack, it is running and always waiting to get files from HDFS.
I guess the optimizing plan refreshes tables too frequently, and it maybe causes great pressure on the HDFS.


Tag column showing 1 tag only even if there is enough space for more.

Description
As the title said.
Steps To Reproduce

Tag a file with 2 or more tags.
Make sure the tag column have enough space to show 2 or more tags.
Only the first tag is shown

Steps to fix
show as many tags as possible to fill the width of tag column
Files Version
Version: <IP_ADDRESS> OS Version: 10.0.22621.755
Windows Version
Windows 11 22H2 22621.755
Relevant Assets/Logs
not applicable

Merging with #9584


Introduce logging functionality

Animate is currently lacking logging functionality, which can be very useful for debugging. It would be a very useful addition.

Perhaps we could just port goalie.log into animate, and then use animate.log in goalie?

Perhaps we could just port goalie.log into animate, and then use animate.log in goalie?

I think it would be good to have separate logging systems, but for Goalie's to pick up Animate's. Similar to Firedrake and Thetis.


Make sure DB sessions are properly closed for metadata and agent storage

Partially fixed in  #869
We need to make sure we can have concurrent and/or multiple connections to the database.

#888


Declaration of XLSSC-57 is failing now that I have manually edited regions for the first mass paper

Specifically:
IndexError                                Traceback (most recent call last)
/tmp/ipykernel_33494/2047891876.py in
8 # Declaring the actual ClusterSample instance for the XXL sample
9 # This is the only sample whose original analysis used the WMAP9 cosmology
---> 10 xxl_srcs = ClusterSample(ra, dec, z, n, r500=r500, cosmology=xxl_cosmo, load_fits=False, use_peak=False,
11                          clean_obs=True, clean_obs_reg='r500', clean_obs_threshold=0.7, back_inn_rad_factor=3.33,
12                          back_out_rad_factor=5)
/mnt/pact/dt237/code/PycharmProjects/XGA/xga/samples/extended.py in init(self, ra, dec, redshift, name, r200, r500, r2500, richness, richness_err, wl_mass, wl_mass_err, custom_region_radius, use_peak, peak_lo_en, peak_hi_en, back_inn_rad_factor, back_out_rad_factor, cosmology, load_fits, clean_obs, clean_obs_reg, clean_obs_threshold, no_prog_bar, psf_corr, peak_find_method)
139                     #  bunch to make GalaxyCluster declaration quicker
140                     try:
--> 141                         self._sources[n] = GalaxyCluster(r, d, z, n, r2, r5, r25, lam, lam_err, wlm, wlm_err, cr,
142                                                          use_peak, peak_lo_en, peak_hi_en, back_inn_rad_factor,
143                                                          back_out_rad_factor, cosmology, True, load_fits, clean_obs,
/mnt/pact/dt237/code/PycharmProjects/XGA/xga/sources/extended.py in init(self, ra, dec, redshift, name, r200, r500, r2500, richness, richness_err, wl_mass, wl_mass_err, custom_region_radius, use_peak, peak_lo_en, peak_hi_en, back_inn_rad_factor, back_out_rad_factor, cosmology, load_products, load_fits, clean_obs, clean_obs_reg, clean_obs_threshold, regen_merged, peak_find_method)
111             raise UnitConversionError("R2500 radius must be in either angular or distance units.")
112
--> 113         super().init(ra, dec, redshift, name, custom_region_radius, use_peak, peak_lo_en, peak_hi_en,
114                          back_inn_rad_factor, back_out_rad_factor, cosmology, load_products, load_fits,
115                          peak_find_method)
/mnt/pact/dt237/code/PycharmProjects/XGA/xga/sources/general.py in init(self, ra, dec, redshift, name, custom_region_radius, use_peak, peak_lo_en, peak_hi_en, back_inn_rad_factor, back_out_rad_factor, cosmology, load_products, load_fits, peak_find_method)
97
98         # This uses the added context of the type of source to find (or not find) matches in region files
---> 99         self._regions, self._alt_match_regions, self._other_regions = self._source_type_match("ext")
100
101         # Making a combined list of interloper regions
/mnt/pact/dt237/code/PycharmProjects/XGA/xga/sources/extended.py in _source_type_match(self, source_type)
191             return Quantity(np.sqrt(abs(ra - self._ra_dec[0]) ** 2 + abs(dec - self._ra_dec[1]) ** 2), 'deg')
192
--> 193         results_dict, alt_match_dict, anti_results_dict = super()._source_type_match('ext')
194
195         # The 0.66 and 2.25 factors are intended to shift the r200 and r2500 values to approximately r500, and were
/mnt/pact/dt237/code/PycharmProjects/XGA/xga/sources/base.py in _source_type_match(self, source_type)
1327                 # print(self._initial_regions[obs][self._initial_region_matches[obs]])
1328                 print(obs)
-> 1329                 if len(self._initial_regions[obs][self._initial_region_matches[obs]]) == 0:
1330                     results_dict[obs] = None
1331                 else:
IndexError: boolean index did not match indexed array along dimension 0; dimension is 2 but corresponding boolean dimension is 1
This is specifically happening for ObsID<PHONE_NUMBER>, which as it turns out has no region file in XCS.
I also gave this source a new custom region, which I suspect must be what's causing the issue here. I'm going to temporarily remove the custom source region and see if that allows declaration.

I was correct, that is whats causing the problem. Now I just need to track down the actual source - presumably when the custom regions are added to the existing ObsID regions.

So I think it's because the custom region is being appended to an array which just contains None (that is what XGA does when no regions exist), which of course throws things off.

This may have highlighted another issue where the initial matching function hates region lists with only one entry:
TypeError                                 Traceback (most recent call last)
/tmp/ipykernel_40237/2047891876.py in
8 # Declaring the actual ClusterSample instance for the XXL sample
9 # This is the only sample whose original analysis used the WMAP9 cosmology
---> 10 xxl_srcs = ClusterSample(ra, dec, z, n, r500=r500, cosmology=xxl_cosmo, load_fits=False, use_peak=False,
11                          clean_obs=True, clean_obs_reg='r500', clean_obs_threshold=0.7, back_inn_rad_factor=3.33,
12                          back_out_rad_factor=5)
/mnt/pact/dt237/code/PycharmProjects/XGA/xga/samples/extended.py in init(self, ra, dec, redshift, name, r200, r500, r2500, richness, richness_err, wl_mass, wl_mass_err, custom_region_radius, use_peak, peak_lo_en, peak_hi_en, back_inn_rad_factor, back_out_rad_factor, cosmology, load_fits, clean_obs, clean_obs_reg, clean_obs_threshold, no_prog_bar, psf_corr, peak_find_method)
73
74         # Using the super defines BaseSources and stores them in the self._sources dictionary
---> 75         super().init(ra, dec, redshift, name, cosmology, load_products=True, load_fits=False,
76                          no_prog_bar=no_prog_bar)
77
/mnt/pact/dt237/code/PycharmProjects/XGA/xga/samples/base.py in init(self, ra, dec, redshift, name, cosmology, load_products, load_fits, no_prog_bar)
74
75                 try:
---> 76                     temp = BaseSource(r, d, z, n, cosmology, load_products, load_fits)
77                     n = temp.name
78                     self._sources[n] = temp
/mnt/pact/dt237/code/PycharmProjects/XGA/xga/sources/base.py in init(self, ra, dec, redshift, name, cosmology, load_products, load_fits)
143             self._lum_dist = None
144             self._ang_diam_dist = None
--> 145         self._initial_regions, self._initial_region_matches = self._load_regions(region_dict)
146
147         # This is a queue for products to be generated for this source, will be a numpy array in practise.
/mnt/pact/dt237/code/PycharmProjects/XGA/xga/sources/base.py in _load_regions(self, reg_paths)
1183                 #  to do this contains match...
1184                 within = np.array([reg.contains(SkyCoord(*self._ra_dec, unit='deg'), w)
-> 1185                                    for reg in reg_dict[obs_id][diff_sort[0:5]]])
1186
1187                 # Make sure to re-order the region list to match the sorted within array
TypeError: only integer scalar arrays can be converted to a scalar index

This appears to be resolved now


[AutoRelease] t2-compute-2023-03-31-26264(can only be merged by SDK owner)

https://github.com/Azure/sdk-release-request/issues/4005
Live test success
https://dev.azure.com/azure-sdk/internal/_build?definitionId=978
BuildTargetingString
azure-mgmt-compute
Skip.CreateApiReview
true

issue link:https://github.com/Azure/sdk-release-request/issues/4005


Collection and Work migration

I putting this out for comment early.  The work is not complete, but I want to be certain I have broken things up in a logical fashion.  (not just logical to me)
The CreatorMigrator is the main class that calls all the rest to migrate the creators for both Collections and Works.
The SolrListMigrator and the CreatorList are still only partially implemented since we do not have the Alias PR merged.  The CreatorList will convert all the creator Strings to Aliases using the AliasManagementService once that is included into develop.  The SolrListMigrator will use the new Aliases to convert the string creators currently in the system for each object to pointers to Aliases.
LocalUserLookup is a helper class to identify creators as local users to link or create them with the PSU login information.
SolrObjectlist, SolrWorkList and SolrCollectionList loads all the object ids from solr and then loads each item from fedora as we iterate through the ids.  The idea is to save on memory since Collection.all or GenericWork.all loads all objects into memory when only one is needed at a time.

Coverage decreased (-2.8%) to 71.445% when pulling 1a6b5283d54a1788a7a123b6ad60458a8c51c8d0 on i#989 into c6825c2ac37b0328800df83608a008e135d5d6e1 on develop.

@awead I was not really thinking about rerunning it, but that is a good thought.  I wonder if we could create the aliases once and then store and load them from file.  That way if the process fails we could restart.

Coverage decreased (-2.4%) to 71.841% when pulling 605523ebb881ee9d2541c84c1b601bcc912e077e on i#989 into 0cde2c6789a8edd0ebaeb471e898315cbc973091 on develop.

Coverage decreased (-3.5%) to 70.733% when pulling 1d1fa38e173953449ad1be20f05350998ad4897f on i#989 into 0cde2c6789a8edd0ebaeb471e898315cbc973091 on develop.

Coverage decreased (-1.8%) to 72.44% when pulling f0e05a5a4b22df2ca8327e157087f4af6c514919 on i#989 into 917d4bd1f0066459cd771e39e8812a5f756a0da2 on develop.

Coverage decreased (-1.9%) to 72.331% when pulling 5f3313e8b632711d61157b00e44992a498c78da9 on i#989 into 917d4bd1f0066459cd771e39e8812a5f756a0da2 on develop.

Coverage decreased (-1.8%) to 72.425% when pulling 0446ae226e6308ea1a9e6576a22a15e67ae3c777 on i#989 into 917d4bd1f0066459cd771e39e8812a5f756a0da2 on develop.

Coverage decreased (-1.8%) to 72.425% when pulling 0446ae226e6308ea1a9e6576a22a15e67ae3c777 on i#989 into 917d4bd1f0066459cd771e39e8812a5f756a0da2 on develop.

Coverage decreased (-1.6%) to 72.566% when pulling 431ceb18c93086afdaeffcbd8ab87b362e0176b5 on i#989 into 917d4bd1f0066459cd771e39e8812a5f756a0da2 on develop.

Coverage decreased (-1.6%) to 72.566% when pulling 431ceb18c93086afdaeffcbd8ab87b362e0176b5 on i#989 into 917d4bd1f0066459cd771e39e8812a5f756a0da2 on develop.

Coverage decreased (-1.5%) to 72.673% when pulling acc9f49d4106b4671bee4420f1279f534629cd9d on i#989 into 917d4bd1f0066459cd771e39e8812a5f756a0da2 on develop.

Coverage decreased (-1.5%) to 72.673% when pulling acc9f49d4106b4671bee4420f1279f534629cd9d on i#989 into 917d4bd1f0066459cd771e39e8812a5f756a0da2 on develop.

Coverage increased (+1.1%) to 75.312% when pulling acc9f49d4106b4671bee4420f1279f534629cd9d on i#989 into 917d4bd1f0066459cd771e39e8812a5f756a0da2 on develop.

Coverage increased (+1.1%) to 75.312% when pulling 1bba302da29d4c767e111ef0aa2e2dfb2fcb32fb on i#989 into 917d4bd1f0066459cd771e39e8812a5f756a0da2 on develop.

Coverage increased (+1.1%) to 75.312% when pulling 1bba302da29d4c767e111ef0aa2e2dfb2fcb32fb on i#989 into 917d4bd1f0066459cd771e39e8812a5f756a0da2 on develop.

Coverage increased (+1.1%) to 75.265% when pulling 735d792b08d3a122cb18149b58127f78cb63f055 on i#989 into 081b49e0d0c915a7c5f00a77026b856fbb136652 on develop.

Coverage increased (+1.1%) to 75.265% when pulling 735d792b08d3a122cb18149b58127f78cb63f055 on i#989 into 081b49e0d0c915a7c5f00a77026b856fbb136652 on develop.

Coverage increased (+1.08%) to 75.65% when pulling 0f5a118bba55710df812952b3574af598a98ac92 on i#989 into e6551dc6df714728490b1e2fe108a27ebc26e505 on develop.

Coverage increased (+1.1%) to 75.703% when pulling f62dda8fb5e21fd5b24855ba7b20de096f0c2ebd on i#989 into e6551dc6df714728490b1e2fe108a27ebc26e505 on develop.

Coverage increased (+1.1%) to 75.703% when pulling f62dda8fb5e21fd5b24855ba7b20de096f0c2ebd on i#989 into e6551dc6df714728490b1e2fe108a27ebc26e505 on develop.

Coverage increased (+1.2%) to 74.248% when pulling 5851325bf72fc6235573b91aa64e0c34a1e417b8 on i#989 into e8849c115cccb531cc93e5ce23f8dd5e46fc4d31 on develop.

Coverage increased (+1.3%) to 74.382% when pulling 6c18cb821e9a24b549911ff26b87dca5502c598a on i#989 into e8849c115cccb531cc93e5ce23f8dd5e46fc4d31 on develop.

Coverage increased (+1.3%) to 74.382% when pulling 299c655738b592c130093f8e52e1efe4a1303d1b on i#989 into e8849c115cccb531cc93e5ce23f8dd5e46fc4d31 on develop.

Coverage increased (+1.4%) to 74.453% when pulling c81a140e7617f2344c28aa94974b236cbf781908 on i#989 into e8849c115cccb531cc93e5ce23f8dd5e46fc4d31 on develop.

Coverage increased (+1.4%) to 74.453% when pulling d6ba670359bbfea2f5b41e239b2aba3a00ae1520 on i#989 into e8849c115cccb531cc93e5ce23f8dd5e46fc4d31 on develop.

Coverage increased (+1.4%) to 74.453% when pulling ebfeadbda140da1efc0976823cc4b0a0a06f7989 on i#989 into e8849c115cccb531cc93e5ce23f8dd5e46fc4d31 on develop.

Coverage decreased (-1.6%) to 71.5% when pulling 8881e20c02ae0252f5d2ba2bc91b67c0733d0c92 on i#989 into 89e6e64d2e158ad6542c81dd6f5a0a0364e58783 on develop.

Coverage decreased (-1.6%) to 71.443% when pulling 404c554318acb3d67732240fcec57721b37f1a83 on i#989 into 89e6e64d2e158ad6542c81dd6f5a0a0364e58783 on develop.

Coverage decreased (-1.6%) to 71.443% when pulling 404c554318acb3d67732240fcec57721b37f1a83 on i#989 into 89e6e64d2e158ad6542c81dd6f5a0a0364e58783 on develop.

Coverage decreased (-1.6%) to 71.443% when pulling 404c554318acb3d67732240fcec57721b37f1a83 on i#989 into 89e6e64d2e158ad6542c81dd6f5a0a0364e58783 on develop.

Coverage increased (+1.4%) to 74.489% when pulling 0b47066b5ad09669d19f941d3610ff5271564614 on i#989 into 89e6e64d2e158ad6542c81dd6f5a0a0364e58783 on develop.

Coverage increased (+1.4%) to 74.48% when pulling c20615e2dd700b22508f26711fca32c4bc8186e9 on i#989 into 89e6e64d2e158ad6542c81dd6f5a0a0364e58783 on develop.

Coverage increased (+1.4%) to 74.48% when pulling c20615e2dd700b22508f26711fca32c4bc8186e9 on i#989 into 89e6e64d2e158ad6542c81dd6f5a0a0364e58783 on develop.

Coverage increased (+1.4%) to 74.463% when pulling 79ecfd70d04ac62e30ba0e03c0bee2dde58c2bd0 on i#989 into 89e6e64d2e158ad6542c81dd6f5a0a0364e58783 on develop.

Coverage increased (+1.3%) to 74.346% when pulling 4e10c4ce4ece8a162c12087e921f6915f9f538f7 on i#989 into 89e6e64d2e158ad6542c81dd6f5a0a0364e58783 on develop.

Coverage increased (+1.2%) to 74.311% when pulling 4e10c4ce4ece8a162c12087e921f6915f9f538f7 on i#989 into 89e6e64d2e158ad6542c81dd6f5a0a0364e58783 on develop.

Coverage increased (+1.5%) to 74.534% when pulling b36ab302d837f199e53bbd721736b8c88befc463 on i#989 into 89e6e64d2e158ad6542c81dd6f5a0a0364e58783 on develop.

Coverage increased (+1.5%) to 74.534% when pulling b36ab302d837f199e53bbd721736b8c88befc463 on i#989 into 89e6e64d2e158ad6542c81dd6f5a0a0364e58783 on develop.


Automatically find IOKit

This will symlink IOKit (and libkern's OSTypes.h) headers from the Xcode installation used to open the project into yalu102/include, unless the links already exist. Nothing is checked into the git repository and the user is free to still use his own copy if he so pleases.

👍🏻

Close this.
IOKit includes the "headers" you're looking for.
It is like Microsoft Visual C++ or .NET 4.0: it's bundled with the macOS SDK.

@RypeDub420: From your comment, I read that you seem to misunderstand the purpose of this PR. This does not add any headers to the project itself.

@mologie works perfectly. Thank you

work fine, thx

qwertyoruiop was trolling us all along. IOKit is actually not used at all by yalu102. I feel shame now.


LocalEndpointImageBuildError: - Local deployment with custom Dockerfile due to wrong path

https://github.com/Azure/azure-sdk-for-python/blob/c1d1b69e05b70521ddec14adf31f88b10424fc3c/sdk/ml/azure-ai-ml/azure/ai/ml/_local_endpoints/validators/environment_validator.py#L131
On this line, absolute_build_directory should be referenced instead of absolute_dockerfile_path
Reproduce:
env = Environment(     build=BuildContext(path=".", dockerfile_path="Dockerfile") ) deployment_config=ManagedOnlineDeployment(     environment=env,     ... ) MLClient.online_deployments.begin_create_or_update(deployment_config, local=True)

Adding Service team to look into this.
@azureml-github, @azure-ml-sdk. Could you please look into this issue, once you get a chance ? Thanks in advance.

Thanks for the fix


Misalignment of 'Get Started' and 'Install' Boxes: Needs Padding and Margin Adjustment

Description
The two boxes, "Get Started" and "Install", are not properly aligned on the page. This misalignment makes the page look unbalanced and can affect the user experience. Both boxes need adjustments in padding and margin to be properly aligned.
Expected Behavior

The "Get Started" and "Install" boxes should be evenly aligned and spaced on the page.
Appropriate padding and margin should be applied to create consistent spacing between the boxes.

Actual Behavior

The two boxes appear misaligned, with uneven spacing.

Steps to Reproduce

Navigate to the section of the page where the "Get Started" and "Install" boxes are displayed.
Observe the misalignment and inconsistent spacing between the two boxes.

please assign this issue to me !


Microphone in 3.5.2

Since I updated to version 3.5.2 I am having issues with the microphone (with 3.5.1 it works fine).
I tried running the example project RecorderDemo and I got the following error:
cleanTempDirectory() 0 files deleted
setSession(category:with:) Error: Error Domain=NSOSStatusErrorDomain Code=-50 "(null)" Cannot set AVAudioSession Category to AVAudioSessionCategoryPlayback with options:AVAudioSessionCategoryOptions(rawValue: 8)
Errored setting category.
initialize() AKAudioPlayer Warning:  "5445C671-473E-4754-912A-3DCDC867364F.caf" is an empty file
2017-02-07 12:58:21.957767 RecorderDemo[436:40125] [mcmx] 337: input bus 0 sample rate is 0
2017-02-07 12:58:21.958961 RecorderDemo[436:40125] [central] 54:   ERROR:    [0x1aa8a0c40] >avae> AVAudioEngineGraph.mm:2515: PerformCommand: error -10875
2017-02-07 12:58:21.959609 RecorderDemo[436:40125] *** Terminating app due to uncaught exception 'com.apple.coreaudio.avfaudio', reason: 'error -10875'
*** First throw call stack:
(0x184fbd1b8 0x1839f455c 0x184fbd08c 0x19ead5300 0x19eae8b3c 0x19eaf2688 0x19eae57fc 0x19eae5684 0x19eae5774 0x19eae6868 0x19eae9a54 0x19eb5702c 0x19eb563d0 0x10027cf40 0x100096e48 0x1000971dc 0x18ae72924 0x18ae724ec 0x18ae78c98 0x18ae76138 0x18aee868c 0x18b0f4cb8 0x18b0fa808 0x18b10f104 0x18b0f77ec 0x186b9392c 0x186b93798 0x186b93b40 0x184f6ab5c 0x184f6a4a4 0x184f680a4 0x184e962b8 0x18aedd7b0 0x18aed8534 0x100094b88 0x183e795b8)
libc++abi.dylib: terminating with uncaught exception of type NSException
(lldb)

I tried commenting out the line with the call to setSession, and then the app runs but I get no input from the mic (recordings are empty).

I get exactly the same issue, but only on an actual device (my iPhone 6S). It works fine in the simulator.
Here's my log, if it helps (pretty much identical except for some identifiers/addresses):
cleanTempDirectory() 0 files deleted
setSession(category:with:) Error: Error Domain=NSOSStatusErrorDomain Code=-50 "(null)" Cannot set AVAudioSession Category to AVAudioSessionCategoryPlayback with options:AVAudioSessionCategoryOptions(rawValue: 8)
Errored setting category.
initialize() AKAudioPlayer Warning:  "9F351F97-525E-4CB9-9740-C30CB0692217.caf" is an empty file
2017-02-07 17:52:33.507757 RecorderDemo[524:100905] [mcmx] 337: input bus 0 sample rate is 0
2017-02-07 17:52:33.508450 RecorderDemo[524:100905] [central] 54:   ERROR:    [0x1b102dc40] >avae> AVAudioEngineGraph.mm:2515: PerformCommand: error -10875
2017-02-07 17:52:33.508811 RecorderDemo[524:100905] *** Terminating app due to uncaught exception 'com.apple.coreaudio.avfaudio', reason: 'error -10875'
*** First throw call stack:
(0x18b7611b8 0x18a19855c 0x18b76108c 0x1a526d300 0x1a5280b3c 0x1a528a688 0x1a527d7fc 0x1a527d684 0x1a527d774 0x1a527e868 0x1a5281a54 0x1a52ef02c 0x1a52ee3d0 0x1002c4f30 0x100076e48 0x1000771dc 0x191616924 0x1916164ec 0x19161cc98 0x19161a138 0x19168c68c 0x191898cb8 0x19189e808 0x1918b3104 0x19189b7ec 0x18d33792c 0x18d337798 0x18d337b40 0x18b70eb5c 0x18b70e4a4 0x18b70c0a4 0x18b63a2b8 0x1916817b0 0x19167c534 0x100074b88 0x18a61d5b8)
libc++abi.dylib: terminating with uncaught exception of type NSException

I've poked around with some stuff and there doesn't seem to be any workaround. Removing the setSession avoids the error, but even something as simple as this always produces completely silent output (but works perfectly in the simulator):
let mic = AKMicrophone()
AudioKit.output = mic
AudioKit.start()

Is there a good way to get a test app running against the source (not a compiled framework) so I could look into this? The examples/playgrounds all seem to depend on the compiled framework, which isn't helpful for debugging.

@jconst @isaacengel  stick with 3.5.1 or move to the develop branch 3.5.2 has a bug in the session managemetn that has been fixed.

@jconst @isaacengel 3.5.3 has been released.  Probably still has issues, but hopefully fewer.

Thanks! Upgrading to 3.5.3 solved the issue for me. (The signal is pretty quiet on the device though, which is totally fine, but anyone checking if the mic's working should just make sure to turn their volume up.)

3.5.3 is working for me too. Thank you!


Documentation or examples

Hello,
Can you provide more documentation about usage , also that link for Pycon 2016 is not working (asking permission), maybe you might at least fix that

Thanks for pointing this out, turns out I couldn't share it from where it was so I copied it over and now you should be able to view it. Let me know if you can't.
Documentation is a bit sparse, sorry. I'd be happy to help if you said what you needed to do?


The class_id in data processing 1_prepare_data/docker/code/utils/tf_record_util.py doesn't match the label_map.pbtxt

The class_id saved in list of classes in line 69 of file 1_prepare_data/docker/code/utils/tf_record_util.py is not matching the class_id of the label_map generation. Is this normal ?

Hi @t-T-s ,
Thanks for using the repository.
The line you're pointing to (code below), reads the class_id from the annotations file provided, which contains the bounding boxes and the class_id (which is 0 in our case)
        for a in annotations:
            ...
            class_id = a['class_id']
            ...
            classes.append(class_id)

In the data preparation notebook, we're defining the label_map as  label_map = '{"0": "bee"}. I just double checked and the annotation file and it only contains the label class "0", which is then mapped to "bee" for better interpretation during inference for example. Let me know if I answered your question or If there is something I missed.

Thank you for your reply @Othmane796
That part is clear for me. But the code below from 1_prepare_data/docker/code/utils/tf_record_util.py adds 1 to the class_id when generating pbtxt.
print('GENERATING LABEL MAP FILE')
    with open(f'{output_folder}/label_map.pbtxt', 'w') as label_map_file:
        for item in label_map:
            label_map_file.write('item {\n')
            label_map_file.write(' id: ' + str(**int(item) + 1**) + '\n')
            label_map_file.write(" name: '" + label_map[item] + "'\n")
            label_map_file.write('}\n\n')

Can this be problematic during training ?

Hi @t-T-s ,
Apologies for the late reply. You're right, this would typically be an issue, the reason why it was working is because during inference (3_predict/deploy_endpoint.ipynb) the label map that was used is the following:
category_index = {1:{'id': 1, 'name': 'bee'}}
Below if what would happen (right image) if we used a different category_index than the one used during training (example : category_index = {0:{'id': 0, 'name': 'bee'}})

I will update the code accordingly to standardise the label_map. Thanks for pointing this out !

Hi @t-T-s ,
Apologies for the late reply. You're right, this would typically be an issue, the reason why it was working is because during inference (3_predict/deploy_endpoint.ipynb) the label map that was used is the following:
category_index = {1:{'id': 1, 'name': 'bee'}}
Below if what would happen (right image) if we used a different category_index than the one used during training (example : category_index = {0:{'id': 0, 'name': 'bee'}})

I will update the code accordingly to standardise the label_map. Thanks for pointing this out !


Retouch Storage refdoc section

Mention dependency on Pivotal Milestones repo.

It doesn't exist... yet.

Bad title, fixed.


⚠ SceneTime has degraded performance

In 4f1a337, SceneTime ($ST) experienced degraded performance:

HTTP code: 200
Response time: 1105 ms

Resolved: SceneTime performance has improved in a954120 after 6 minutes.


Fix Series.groupby on the Series from different DataFrames.

Fixing Series.groupby on the Series from different DataFrames.
>>> import databricks.koalas as ks
>>> kdf1 = ks.DataFrame({"C": [0.362, 0.227, 1.267, -0.562], "B": [1, 2, 3, 4]})
>>> kdf2 = ks.DataFrame({"A": [1, 1, 2, 2]})
>>> ks.options.compute.ops_on_diff_frames = True
>>> kdf1.B.groupby(kdf2.A).sum()
Traceback (most recent call last):
...
pyspark.sql.utils.AnalysisException: 'Resolved attribute(s) __tmp_groupkey_0__#118L missing from ...\n'

This is based on #1459 and the first commit here is the same as it.

Codecov Report

Merging #1460 into master will decrease coverage by 0.05%.
The diff coverage is 82.71%.

@@            Coverage Diff             @@
##           master    #1460      +/-   ##
==========================================
- Coverage   93.88%   93.82%   -0.06%
==========================================
  Files          36       36
  Lines        8373     8392      +19
==========================================
+ Hits         7861     7874      +13
- Misses        512      518       +6

Impacted Files
Coverage Δ

databricks/koalas/groupby.py
90.45% <79.03%> (-1.46%)
:arrow_down:

databricks/koalas/generic.py
97.02% <87.50%> (+3.60%)
:arrow_up:

databricks/koalas/utils.py
96.58% <100.00%> (+0.13%)
:arrow_up:

databricks/koalas/frame.py
95.28% <0.00%> (-0.30%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 565d472...ade47a9. Read the comment docs.


update CSP setting

Refs:

https://github.com/stolostron/backlog/issues/23853
https://github.com/stolostron/backlog/issues/23854

Signed-off-by: Chunxi Luo<EMAIL_ADDRESS>

@ChunxiAlexLuo while the CSP is good, the X-XSS-Protection (even if largely deprecated) I expect will be a problem for the customer. The CSP may prevent XSS, but the Protection header certainly shouldn't be set to disabled.
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-XSS-Protection
I'm also not an owner so my approval will not trigger any merge.

Checking the triage doc it looks like X-XSS-Protection was not a header they were concerned with.
/lgtm

Determined in https://github.com/stolostron/backlog/issues/23853#issuecomment-1198611555 that this PR does not address issue #23853


Lack of support for multiplayer undo/redo (but Replicache is fundamentally multiplayer)

I read the code and noticed it is implemented as a single client undo/redo queue (it doesn't subscribe to the Replicache store) even though Replicache is a multiplayer system. This can lead to all sorts of unexpected behaviours:
Simple example (that I reproduced on TodoMVC):

Add two todo items from client A
Delete the second todo on client B
Click undo on client A, nothing happens (from the user perspective - bug, if they click again the first item is removed)

It can get much more complicated than that when a series of conflicting changes are done on two (or more) clients.
If the undoing client is offline things get even more gnarly.
I understand this is not an easy problem in multiplayer system but even if this lib doesn't solve it I think it needs to create an affordance that would allow the consuming app to solve it.
If the lib does attempt to solve this for everyone, one approach is to adopt the solution implemented by Figma (see here):

Undo history has a natural definition for single-player mode, but undo in a multiplayer environment is inherently confusing. If other people have edited the same objects that you edited and then undo, what should happen? Should your earlier edits be applied over their later edits? What about redo?
We had a lot of trouble until we settled on a principle to help guide us: if you undo a lot, copy something, and redo back to the present (a common operation), the document should not change. This may seem obvious but the single-player implementation of redo means “put back what I did” which may end up overwriting what other people did next if you’re not careful. This is why in Figma an undo operation modifies redo history at the time of the undo, and likewise a redo operation modifies undo history at the time of the redo.

Happy to brainstorm and/or contribute a PR if needed.
Thanks.

Other useful resources
https://liveblocks.io/blog/how-to-build-undo-redo-in-a-multiplayer-environment
https://www.researchgate.net/publication/2241844_Multi-User_UndoRedo

We have an existing bug (confusing in the Replicache repo) for this here:
https://github.com/rocicorp/replicache/issues/1008
I'm familiar with the articles you mention, but I don't believe they help with the case you describe:

Add two todo items from client A
Delete the second todo on client B
Click undo on client A, nothing happens (from the user perspective - bug, if they click again the first item is removed)

In the Figma implementation, this will also do nothing at step 3. Here's an example:
https://github.com/rocicorp/undo/assets/80388/c281d340-9bb7-43e8-b0c1-a43174d27329
The same thing happens in Notion. And I think it makes sense – if I press paste twice, then cmd+z twice, I expect to go back to the state before I copied anything. This unfortunately means the first cmd+z appears to do nothing, but that is less weird than the number of actions not matching up.
Our implementation does have some problems, you can see the Notion doc for details, and for the beginning of an idea how to fix. Would love any contributions if you want to work on that.