ENH: stats.gaussian_kde: support degenerate data; avoid inv

My issue is about the gaussian kde. I wanted to run an example of multivariate kernel density estimation I saw on StackOverflow, but I experience the error described below. This error seems to have a small random chance of not appearing, but appears most of the time.
Reproducing code example:

import numpy as np
from scipy import stats
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D

mu = np.array([1,10,20])
sigma = np.matrix([[4,10,0],[10,25,0],[0,0,100]])
data = np.random.multivariate_normal(mu,sigma,1000)
values = data.T

kde = stats.gaussian_kde(values)
density = kde(values)

fig, ax = plt.subplots(subplot_kw=dict(projection='3d'))
x, y, z = values
ax.scatter(x, y, z, c=density)
plt.show()

Error message:

---------------------------------------------------------------------------
LinAlgError                               Traceback (most recent call last)
<ipython-input-112-7ab47763eef5> in <module>
     10
     11 kde = stats.gaussian_kde(values)
---> 12 density = kde(values)
     13
     14 fig, ax = plt.subplots(subplot_kw=dict(projection='3d'))

~/.local/lib/python3.6/site-packages/scipy/stats/kde.py in evaluate(self, points)
    243         result = zeros((m,), dtype=float)
    244
--> 245         whitening = linalg.cholesky(self.inv_cov)
    246         scaled_dataset = dot(whitening, self.dataset)
    247         scaled_points = dot(whitening, points)

~/.local/lib/python3.6/site-packages/scipy/linalg/decomp_cholesky.py in cholesky(a, lower, overwrite_a, check_finite)
     89     """
     90     c, lower = _cholesky(a, lower=lower, overwrite_a=overwrite_a, clean=True,
---> 91                          check_finite=check_finite)
     92     return c
     93

~/.local/lib/python3.6/site-packages/scipy/linalg/decomp_cholesky.py in _cholesky(a, lower, overwrite_a, clean, check_finite)
     38     if info > 0:
     39         raise LinAlgError("%d-th leading minor of the array is not positive "
---> 40                           "definite" % info)
     41     if info < 0:
     42         raise ValueError('LAPACK reported an illegal value in {}-th argument'

LinAlgError: 1-th leading minor of the array is not positive definite
  ...

Scipy/Numpy/Python version information:

Scipy 1.3.0 and 1.2.1
Numpy 1.16.3
Python 3.6.7
I have ran this example on Google Colab, as well as on my Ubuntu-based OS on my laptop.

@rkern is gaussian_kde designed to work when the data lies in a lower dimensional subspace? If not, would you consider this issue to be an enhancement request rather than a defect?

Nominally, yes, it was designed to work on degenerate data, at least for a certain subset of methods and up to certain numerical limits. I.e. naturally, .evaluate() is going to be practically useless, but the various integrations and resampling were supposed to work, more or less. This was a requirement for the commercial software that this code originally contributed from (changing scipy won't affect that software, and scipy doesn't have to share that requirement, just a historical note). I didn't notice gh-12989 when it came along to introduce the Cholesky decomposition, which has more stringent (and earlier) positive-definiteness constraints. I don't care a huge amount either way, now.

Thanks @rkern. Unfortunately, gaussian_kde no longer supports degenerate data, and a nice fix (especially one incorporating the suggestion of gh-5087) will take some work. I'd suggest that gh-16631 would be helpful in the meantime and that we can change this issue to an enhancement request. Do you have a moment to see if gh-16631 looks good?

There are two issues here; I think they should be treated together.
gaussian_kde currently fails with degenerate data of the sort reported above: when the data lies in a subspace (hyperplane) of the space in which it is expressed. Apparently, it used to work on this sort of data, so we'd like to get that capability back.
I think this can be achieved by using scipy.linalg.ldl instead of scipy.linalg.cholesky. For instance, currently the diagonal of cholesky's L matrix is used to get the determinate of the matrix; instead we would use the D matrix. Some work is still needed to figure out how to get whitening from the LDL decomposition instead of Cholesky. Perhaps that is as simple as using:

to get the L expected by the current code.
The second enhancement idea is to eliminate the uses of inv and replace them with triangular solves on the LDL factors. I think https://github.com/scipy/scipy/pull/15493/files#r908725848 and https://github.com/scipy/scipy/pull/15493/files#r908818183 suggest that it is possible for Cholesky, and I bet it could be extended to work with LDL.
So to close this issue, I'd suggest:

Use the comments https://github.com/scipy/scipy/pull/15493/files#r908725848 and https://github.com/scipy/scipy/pull/15493/files#r908818183 to use a LDL decomposition instead of inv and cholesky throughout gaussian_kernel_estimate.
Make inv_cov a cached_property

If/when this is resolved, the condition detected in gh-17436 should result in a warning rather than an error.

@mdhaber I'm using gaussian_kde right now and the absence of support for singular matrices is a real pain. I'm willing to do a PR to fix this.
As you wrote above, just using LDL won't really work. The point is that if the matrix is numerically singular in general it is impossible to recover the eigenvalues below the noise level. So a choice has to be made between inflating the below-noise eigenvalues (regularization) and removing them (pseudoinversion).
Inflating means that eigenvalues are set to a minimum value. This can be done with Cholesky by adding a multiple of the identity to the matrix. This "nugget" variance can be chosen with Gershgorin:
def singular_chol(a):
    maxeigv = np.max(np.sum(np.abs(a),axis=1))) # upper bound on the maximum eigenvalue
    eps = maxeigv * len(a) * np.finfo(a.dtype).eps # estimated numerical error floor
    a = a + np.eye(len(a), dtype=a.dtype) * eps # make all eigenvalues at least eps
    return linalg.cholesky(a)

If the matrix has been inflated, vectors that have components along the small eigenspaces will produce very low probabilities: kde.logpdf(x) will decrease very quickly as x deviates from the subspace of the data. Also, the logdet will get a lot of log(eps) terms, so the value of the pdf is heavily influenced by the choice of regularization.
Instead, removing eigenspaces means computing a pseudoinverse. This can be done accurately with a diagonalization, although it is could be possible to compute an approximate version with a correction to the Cholesky factor (diagonalization costs 30 choleskies computationally.)
With a pseudoinverse, vectors that have components along the removed eigenspaces are projected orthogonally into the allowed subspace: kde.logpdf(x) will give a large value even if x is totally implausible. The logdet changes abruptly from one dataset to another as eigenvalues cross the threshold for inclusion.
So there is a substantial behavioral choice between regularization and pseudoinversion.
I'd suggest using regularization, for these reasons:

From the point of view of the user, it would be unexpected that implausible vectors get a large probability
Pseudoinversion requires a quite different linalg algorithm
Pseudoinversion is computationally more expensive

I would do the following:

remove matrix inversion and use only the Cholesky factor
add the option allow_singular: bool to gaussian_kde.__init__
if allow_singular=False (default), proceed as usual
if allow_singular=True regularize the matrix before Cholesky. Save the regularization to a public attribute eps (which is None otherwise)
document that for numerically singular matrices the probability is not well defined and comparisons between different datasets may be invalid

Decision: should I always regularize if allow_singular=True, to lead to predictable behavior, or try first a decomposition without regularization? I think the former is better.
Other decision: should I try again with a larger eps if the regularization is not sufficient to get a positive definite matrix? In my experience the regularization I wrote above has always been sufficient, so for starters I'd avoid retries to have a predictable behavior.
Other changes I would do:

support a np.random.SeedSequence as seed argument to gaussian_kde.resample
if there are no weights, assign _neff in gaussian_kde.__init__ to avoid having neff different from sample size due to roundoff in gaussian_kde.resample, which is unexpected

I think I understand the difference, but I didn't know the names. See gh-5286 (maybe starting here) and gh-5288 for some background with multivariate_normal.
If I understand correctly, multivariate_normal has always taken the "pseudoinverse" approach; the difference is that now there is a "threshold for inclusion" (in your words). Now, after gh-5288, if the argument vector lies outside the threshold to be considered part of the supported subspace, it is assigned a PDF of zero. The threshold is wide enough to accommodate the limitations of floating point arithmetic, but otherwise pretty tight, so it seems like a reasonable compromise for a bug fix (even if we might have made different decisions if we were starting from scratch).
Which methods are you most interested in? My intuition (which may be wrong) is that regularization will make the magnitude of the PDF meaningless, but the ratio between PDFs at different points will be correct. (In other words, reducing the magnitude used to inflate the eigenvalues will make all the nonzero lodpdf values slide upwards together.) Therefore, regularization seems like it could be useful for resample, but arguably incorrect (unless the user knows what they're getting into) for pdf. With that in mind, I don't think that SciPy's gaussian_kde should use regularization for pdf/logpdf/evaluate/__call__, but it could be useful to restore the behaviors that used to support singular matrices as mentioned above.

I'm interested in both resample and logpdf.
You are right that the value of the PDF is meaningless in an absolute sense because of the arbitrary regularization. But the same holds for pseudoinversion, since you are arbitrarily removing some directions. Unless the user made sure to always use the same directions, different pdfs would not be comparable as well. Actually, I think that working in a lower-dimensional subspace, and having errors for vectors estimated as out of space, is more surprising than having all very small eigenvalues inflated to a minimum threshold. Moreover, the calculations are not continuous w.r.t. the input matrix, while with regularization they are. Regularization won't break an optimization process, pseudoinversion will.
I think I already took into account all your concerns when I expressed a preference overall for regularization:

but arguably incorrect (unless the user knows what they're getting into) for pdf

This is why I propose as default allow_singular=False, to document the invalidity of absolute comparisons, and to add an attribute for the regularization used.

but it could be useful to restore the behaviors that used to support singular matrices as mentioned

I've looked at the 1.9 code and it does not really support well singular matrices, it doesn't have clear criteria for the inclusion/exclusion of directions, it is not numerically accurate, and is 5x slower than Cholesky.
I would not switch back to the previous behavior because it is inaccurate and I can't interpret what it is doing. I can't rely on it, I would always have to check manually the covariance matrix to make sure gaussian_kde won't mess up.
Doing a proper pseudoinversion would be slow: diagonalization is 30x slower than Cholesky. In general I, and I guess many users, care about speed; if gaussian_kde did that, I would write my own implementation with Cholesky.
Overall, this means that I prefer opt-in regularization as a small incremental modification to the current behavior.
If I had to redesign it from scratch, I would use scipy.stats.Covariance. But I would like to get a gaussian_kde that is able to deal with singular matrices in a well-specified way without slowing it down, and with a reasonable amount of work. I expect that integrating stats.Covariance would require some important design changes because its interface is not designed to deal with a function that expects samples as input, and I won't use gaussian_kde if it does pseudoinversion.

Note: my speed ratios are inflated because I'm counting only the matrix decomposition. Computing the covariance matrix takes (1-2 choleskies) x (number of samples / dimension).

speed of computing inverse covariance will be small relative to evaluating the kde on a sample, I guess.
In general I prefer pinv as default to Ridge because it leaves the results unchanged for the standard non-singular case, while a Ridge penalty always regularizes directions with larger eigenvalues.
I would make the option nonbinary allowing ridge as alternative, nondefault method.
(I never tried kde with singular matrices, so this is a generic comment.)

I've looked at the 1.9 code and it does not really support well singular matrices, it doesn't have clear criteria for the inclusion/exclusion of directions

Actually I mean a while back. It sounds like support for singular matrices was lost a bit further back with https://github.com/scipy/scipy/pull/12989.

I think I already took into account all your concerns when I expressed a preference overall for regularization

I agree that you did. It might take me some time to agree with it, but perhaps I will. It's not a decision for me alone anyway, though.
Let's see what some others think here, and ultimately this will need an email to the mailing list: @rkern @ilayn what do you think of adding support for singular covariance matrices back to gaussian_kde via regularization?

@josef-pkt by "ridge", do you mean the idea that the PDF drops to zero if the argument is beyond some numerical threshold outside the supported subspace?
And "pinv" means "pseudoinversion" as an idea, not necessarily the pinv function, right?

And "pinv" means "pseudoinversion" as an idea, not necessarily the pinv function, right?

Yes, svd based with cutoff on singular values or eigen values decomposition, i.e. only changing singular directions but leaving non-zero directions unchanged.
(for example statsmodels has helper functions for converting correlation matrices to be positive definite or positive semi-definite. This is useful when empirical correlation matrix is not definite because, for example, missing values.)

I can't follow the stats terminology very well but from pure linalg perspective the discussion is between add some multiple of identity until cholesky works OR do something else.
And if this is only for the logdet step then the simplest is to pick eigh (with evr, eigvals_only=True) and threshold the small eigenvalues with max(eigs)*max(m, n)*eps. Still not super nice but at least it moves the noise floor according to the condition number. Cholesky is (or should be) much faster than eigh but I don't know if you would care. Plus you won't be generating the factor itself so saves some memory overhead.

the discussion is between add some multiple of identity until cholesky works OR do something else.

Yes.

And if this is only for the logdet step

No, I don't think so. My understanding is that this Cholesky decomposition would be used throughout. (Currently, all the calculations use the Cholesky decomposition; the inverse is no longer used.)

IIUC, the suggestion is to add support for singular data simply by tweaking the covariance matrix until it is not numerically singular (so that Cholesky doesn't fail).

That's the "Ridge penalty" that @josef-pkt does not recommend. When he says he recommends a pseudoinverse method, he means not using Cholesky and instead using one of the other decompositions that doesn't require strict positive definiteness.

@rkern I'm asking for feedback on @Gattocrucco 's suggestion (https://github.com/scipy/scipy/issues/10205#issuecomment-1465238798). Josef's comment was his feedback on that topic; I'm also asking for yours.

The only thing I really care about is not applying a ridge penalty when the covariance matrix is already fine.

@Gattocrucco I think that answers your question:

Decision: should I always regularize if allow_singular=True, to lead to predictable behavior, or try first a decomposition without regularization? I think the former is better.

And Josef suggested:

I would make the option nonbinary allowing ridge as alternative, nondefault method.

So maybe instead of a boolean allow_singular, there should be an option for how to treat singular matrices. The default would be None, and the only alternative for now would be "regularization" (or "ridge", if that is the preferred term). I would be comfortable with that.

I would add the eigenvalue adjustment in the version of @ilayn as second option.
Aside: AFAIR, pinv, eigenvalue cutoff solution is the limit of Ridge regularization when the penalty weight goes to zero.
(theoretically, in practise we need cutoffs that account for for floating point precision).
If Ridge is only applied after checking for near-singularity, then it looses any or most of the computational advantage to just using always some form of pinv, I guess.

Ah I've read about it now I know ridge regression is Tikhonoff (sorry slow brain). I'm also surprised that stats folks are happy with such hacks and even gave it a name :smile
So if there is absolutely 100%  definitely a positive semidefinite array is sought after and the input cannot be negative definite, then scipy.linalg.lapack.dpstrf should be good. It is the complete pivoting version of Cholesky and probably fits the bill here. It factorizes as much as it can and then leaves the remaining problematic block intact and you can do whatever you want with it. But again, maybe I'm missing the point.
Look at what it is doing in http://www.netlib.org/lapack/lawnspdf/lawn161.pdf

Some toy example for your taste
# Rank deficient array then salted for ill-conditioning
zzz = np.random.rand(40,38) + 10*np.eye(40, 38)
Z = zzz @ zzz.T + 1e-12*np.eye(40)
X, piv, r, info = la.lapack.dpstrf(a=Z, tol=1e-6)

X is the permuted L factor, r is what algo is thinking about the rank, piv is the standard  1-indexed torture array from lapack folks.

Ah I've read about it now I know ridge regression is Tikhonoff (sorry slow brain). I'm also surprised that stats folks are happy with such hacks and even gave it a name

maybe researchers in the 1950s and 1960s didn't read Russian, or didn't need the heavy math.
(I only know my wife who uses Tikhonov and Landweber–Fridman regularizations in econometrics, but it's in Hilbert spaces, not some simple matrix algebra :)
https://www.ams.org/journals/mcom/1955-09-051/S0025-5718-1955-0074915-1/S0025-5718-1955-0074915-1.pdf

A thorough discussion. I'll try to reply to everyone.
@josef-pkt

speed of computing inverse covariance will be small relative to evaluating the kde on a sample, I guess.

Depends on the number of samples. I'm currently using high-dimension, few samples.

In general I prefer pinv as default to Ridge because it leaves the results unchanged for the standard non-singular case, while a Ridge penalty always regularizes directions with larger eigenvalues.

(I guess you mean smaller eigenvalues) I would like to leave the default behavior unchanged, and add ridge with an option.

I would make the option nonbinary allowing ridge as alternative, nondefault method.

The point is avoiding adding a lot of stuff to gaussian_kde because that should probably be handled together with stats.Covariance is some smarter way to be thought about, because there are so many possible options once one starts considering this problem (LOBPCG, Lanczos, LDL, partial Cholesky, corrected Cholesky, series pseudoinversion, who knows what else I'm not even an expert). opt-in ridge currently requires minimal changes.

If Ridge is only applied after checking for near-singularity, then it looses any or most of the computational advantage to just using always some form of pinv, I guess.

No, trying to do Cholesky and seeing it fail is much faster than doing diagonalization. If you really wanted to look directly at the largest and smallest eigenvalues, you could use LOBPCG, for somewhat large matrices it is faster even though the matrix is dense because it scales like $O(n^2)$.
@ilayn

And if this is only for the logdet step then the simplest is to pick eigh (with evr, eigvals_only=True) and threshold the small eigenvalues with max(eigs)*max(m, n)*eps. Still not super nice but at least it moves the noise floor according to the condition number. Cholesky is (or should be) much faster than eigh but I don't know if you would care. Plus you won't be generating the factor itself so saves some memory overhead.

If you want to threshold the small eigenvalues directly, then you are using the full decomposition, so you need to store the factor in memory. Instead, if you only use eigvalsh to estimate the higher eigenvalue to implement ridge, you are doing a lot of computations for something that can be estimated quite well with Gershgorin's theorem (max(sum(abs(x),axis=0)))). I care about speed in general, Cholesky is 30-40x faster than eigh. On my computer, on 10000x10000 matrices, eigh takes 1.5 minutes, cholesky 2 seconds. In any case, an nxn factor of some kind has to be stored, so actually I don't understand the comment about the factor, I may be confused.

So if there is absolutely 100% definitely a positive semidefinite array is sought after and the input cannot be negative definite, then scipy.linalg.lapack.dpstrf should be good. It is the complete pivoting version of Cholesky and probably fits the bill here. It factorizes as much as it can and then leaves the remaining problematic block intact and you can do whatever you want with it. You can still do two-passes since this is practically still Cholesky (one regular and one with this). But again, maybe I'm missing the point.

Very cool, I did not know about this function. But: I guess that, being pivoted, this is slower than normal Cholesky. On my computer, LDL is typically 5x slower than Cholesky (LDL is pivoted too). I need to check this function to be sure. But this algorithm would have an opaque an unpredictable behavior, and not be more justified than other choices, I address this kind of concern below in reply to @rkern.
@rkern

The only thing I really care about is not applying a ridge penalty when the covariance matrix is already fine.

If the user has not specified to allow singular matrices, yes. If the user has specified some option to deal with singular matrices, I'd actually prefer always applying the same calculation without branches.
Example: suppose that one sample was added, which moves a covariance matrix from "fine" to "bad". If the singular-enabled option was active, and it consisted in "try cholesky without ridge, try again with ridge if it fails," this could change some results abruptly.
More concrete example: consider an optimization that uses a simulation (something like simulated inference from the '90s, or a detector geometry Physics simulation I did once). So I have a deterministic function that maps fixed "source" random samples (drawn once and for all) and parameters to some simulated output. The goal is maximizing some function of this output samples w.r.t. the parameters. It is realistic to imagine some of these parameters would change the distribution of the output samples, making the covariance matrix more or less degenerate. If the target function involves a pdf estimation, done with gaussian_kde, it is much more useful to have comparable and continuously varying outputs as the parameters are varied, even if some eigenvalues go below the noise level, than satisfying some criterion for "most accurate possible given the circumstances." Ridge with a simple formula for the regularization has this property. Also consider that any method of dealing with below-noise eigenvalues must make some stuff up, there is no "truly correct" method.
@mdhaber

So maybe instead of a boolean allow_singular there should be an option for how to treat singular matrices. The default would be None (old behavior), and the only alternative for now would be "regularization" (or "ridge", if that is the preferred term). I would be comfortable with that.

Yup looks fine. I proposed allow_singular because I remember seeing it somewhere in numpy/scipy, but since the behaviors are actually different it makes sense to use a different keyword.
I make a ridge-only counter-proposal: regularization=None (default) does not use regularization, fails if Cholesky fails. regularization='auto' uses Gershgorin. regularization=<scalar> adds the given scalar to the diagonal of the matrix (useful for having pdfs comparable across datasets, if needed). The relevance of this choice would be documented under a Notes section.
Open-to-future-alternatives proposal: singular=None, singular="ridge", singular="ridge", eps=<scalar>, where eps could be used in the future as threshold/regularization for other algorithms too.

@ilayn I'm not familiar with LAPACK, could you tell me how to obtain the triangular factor of dpstrf out of X, piv, r?
Timing: it appears that worst case is about 2x cholesky, and can be faster if it stops early. Cool!

I proposed allow_singular because I remember seeing it somewhere in numpy/scipy, but since the behaviors are actually different it makes sense to use a different keyword.

Yup, allow_singular is used by the multivariate distributions like multivariate_normal. But there is no consistency between this function and multivariate_normal; e.g. one pdf method expects the transpose of the other.

@Gattocrucco You can't have both speed and accuracy for ill-conditioned problems. That's just not going to happen. Either you decompose with n**3 with a high scalar in front like 30 or 40 or you go with simpler things but axe your way out like cholesky. There is no in between. Your upper bound is also prone to wonky test data.
Anyways, if lower=0 then you get the upper trapezoid as the meaningful data. The bottom right (n-r) x (n-r) is identical to input data if n > r.
     [ __________ ]
     [  \        |]
 X = [    \______|]
     [            ]
     [            ]
     [            ]

Insane ASCII skills I know. So in the example above,
# Rank deficient array then salted for ill-conditioning
zzz = np.random.rand(40,38) + 10*np.eye(40, 38)
Z = zzz @ zzz.T + 1e-12*np.eye(40)
X, piv, r, info = la.lapack.dpstrf(a=Z, tol=1e-6)

# Here is the trapezoid
X[r:, :] = 0.
X[:r, :r] = np.triu(X[:r, :r])
P = np.zeros([40, 40])  # You don't need to create P, you can permute directly but better to show this way
P[piv-1, np.arange(40)] = 1.
Err = (P @ X.T) @ (X @ P.T) - Z  # Permuted and trapezoidal Cholesky factors
Err[np.abs(Err) < 1e-12] = 0  # Some cleanup
print(Err)

Look at the bottom right corner of Err. Funny numbers; almost as if giving the salt back 😉

@ilayn Thanks for the example with PSTRF, I could not guess that.

You can't have both speed and accuracy for ill-conditioned problems.

I'm explicitly giving up on accuracy to have a logpdf that works for real problems, since anyway I can't be "accurate" on that, it's not a linear system; I have to divide by a very small number, and that number just isn't computable accurately.
Now that I think about it, for resampling actually we could improve accuracy with PSTRF, since resampling does not involve an inversion, it has a notion of as accurate as possible. But I'd put that into "more complicated things to do in the future."

Changing the direction of the discussion:
Is there an option for user provided cov or cov_inv?
standard theory for kde is large nobs and small k.
For large dimensional problems, small nobs, large k, the covariance should use an improved estimator, e.g. shrinkage estimators like Ledoilt-Wolf or other ones available in sklearn.
I don't think gaussian_kde is supposed to handle that internally.
When I thought about singular, it's more the case of large nobs, smallish k but high collinearity.

aside: Tikhonoff versus Ridge
According to my wife, they are not the exactly the same outside least squares
system of equations: A x = y
what I called ridge: (A + k I) x = y
Tikhonoff: premultiplies by A.T: (A.T A + k I) x =A y
(the latter is the moment condition of a least squares problem where A is the data x-matrix, while in Ridge least squares A is the moment matrix)
:)

Take the fight to wiki, that's where I got the idea 😃 https://en.wikipedia.org/wiki/Ridge_regression
But in any case, it works with any positive definite perturbation matrix. So it doesn't have to be diagonally dominant or scalar times identity.

my previous point is that as in Ridge regression, the ridge penalization factor should not be just to barely make it positive definite in ill-posed problems.
shrinking towards identity, e.g. https://scikit-learn.org/stable/modules/generated/sklearn.covariance.LedoitWolf.html
So users might want to use a larger k, or proper estimator for large dimensional problems including options for sparse inverse covariance.
aside (continued)
I have Theil-Goldberger in statsmodels (sandbox) which is like Ridge but with a non-identity target matrix and predates "Ridge" regression by almost 10 years (1961)
https://github.com/statsmodels/statsmodels/blob/a35f7dc19617ef3c96e06fe54ae2dcb0c5eade93/statsmodels/sandbox/regression/penalized.py#L44
(But "Ridge" won the naming game)

@ilayn
I wanted to move the discussion from a numerical math view to "statistics".
It's not just system identification or numerically solving an ill-posed problem.
We want to get a "good" estimator that minimizes some evaluation criterion, eg. mean squared error relative to the "true" parameters, if we have repeatedly the same problem with different data.
Non-negligible (large) ridge factor, Bayesian priors, regularization, penalization provides a better estimator with smaller "loss" in accuracy or smaller variance in repeated sampling.
We want to have some kind of optimal regularization and not just minimally identified.
(
Most work in (frequentist) theoretical statistics and econometrics is in deriving the sampling properties of an estimator. Bayesians and machine learning guys can ignore it because they don't care. Bayes' law for the former and cross-validation for the latter solves all problems in the world. :)
In statsmodels it would sometimes be easy to compute a new estimator, but we don't have inference for it, standard errors and hypothesis test for estimated parameter or predictions. The estimation is relatively easy, the sampling properties are much too difficult. Easy to compute is very often because scipy has the needed tools for the numerical math.
)

The nice thing about the multiple of the identity is that it increases all eigenvalues by the same amount, even if you can't look at the individual eigenvalues. This makes it interpretable even though it is computationally simple. This allows you to understand cross-comparisons if you regularize different matrices with the same multiple of the identity.

but it increases the variances, which might already be bad estimators if there are outliers or very small samples.
I really think there should be an option for the user to provide a covariance or inverse covariance.
For example for univariate kde, a robust variance estimator like iqr is often used to reduce the influence of outliers.
For the multivariate case, the user could compute a outlier robust or large dimension appropriate cov or cov_inv.
This would avoid that scipy has to pick a solution to various problems with pearson cov.


Add Docker logo to sponsors

Blocked due to lack of logo in the ticket

Image has been made. Will finish adding in later tonight.


[Bug] Cannot convert whisper with pip package

System Info

Architecture: x86_64
CPU: AMD Ryzen Threadripper 1920X 12-Core Processor
NVIDIA RTX 3090
Driver Version: 545.29.06
Container Tag: cuda12.1-devel-ubuntu22.04

Who can help?
@byshiue
Information

[ ] The official example scripts
[ ] My own modified scripts

Tasks

[ ] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
[ ] My own task or dataset (give details below)

Reproduction
Hello, I tried building whisper after installing pip package in the cuda-12.1-devel-ubuntu22.04 container:
docker run --entrypoint /bin/bash -it nvidia/cuda:12.1.0-devel-ubuntu22.04

# Install dependencies, TensorRT-LLM requires Python 3.10
apt-get update && apt-get -y install python3.10 python3-pip openmpi-bin libopenmpi-dev

# Install the latest preview version (corresponding to the main branch) of TensorRT-LLM.
# If you want to install the stable version (corresponding to the release branch), please
# remove the `--pre` option.
pip3 install tensorrt_llm -U --pre --extra-index-url https://pypi.nvidia.com

# Check installation
python3 -c "import tensorrt_llm"

All of this works as expected. As soon as I run:
python3 build.py --output_dir whisper_large_v3 --use_gpt_attention_plugin --use_gemm_plugin --use_bert_attention_plugin

Expected behavior
To convert whisper to TensorRT Engine
actual behavior
It fails:
[02/01/2024-10:55:05] [TRT-LLM] [I] plugin_arg is None, setting it as float16 automatically.
[02/01/2024-10:55:05] [TRT-LLM] [I] plugin_arg is None, setting it as float16 automatically.
[02/01/2024-10:55:05] [TRT-LLM] [I] plugin_arg is None, setting it as float16 automatically.
[02/01/2024-10:55:07] [TRT] [I] [MemUsageChange] Init CUDA: CPU +13, GPU +0, now: CPU 3065, GPU 11135 (MiB)
[02/01/2024-10:55:11] [TRT] [I] [MemUsageChange] Init builder kernel library: CPU +1799, GPU +312, now: CPU 5000, GPU 11447 (MiB)
[02/01/2024-10:55:11] [TRT-LLM] [W] Parameter dtype is None, using default dtype: DataType.FLOAT, it is recommended to always specify dtype explicitly
[02/01/2024-10:55:11] [TRT-LLM] [W] Parameter dtype is None, using default dtype: DataType.FLOAT, it is recommended to always specify dtype explicitly
[02/01/2024-10:55:11] [TRT-LLM] [W] Parameter dtype is None, using default dtype: DataType.FLOAT, it is recommended to always specify dtype explicitly
[02/01/2024-10:55:11] [TRT-LLM] [W] Parameter dtype is None, using default dtype: DataType.FLOAT, it is recommended to always specify dtype explicitly
[02/01/2024-10:55:11] [TRT-LLM] [W] Parameter dtype is None, using default dtype: DataType.FLOAT, it is recommended to always specify dtype explicitly
[02/01/2024-10:55:11] [TRT-LLM] [W] Parameter dtype is None, using default dtype: DataType.FLOAT, it is recommended to always specify dtype explicitly
[02/01/2024-10:55:11] [TRT-LLM] [I] Loading encoder weights from PT...
[02/01/2024-10:55:11] [TRT-LLM] [W] Parameter was initialized as DataType.HALF but set to DataType.FLOAT
[02/01/2024-10:55:11] [TRT-LLM] [W] Parameter was initialized as DataType.FLOAT but set to DataType.HALF
[02/01/2024-10:55:11] [TRT-LLM] [W] Parameter was initialized as DataType.FLOAT but set to DataType.HALF
[02/01/2024-10:55:11] [TRT-LLM] [W] Parameter was initialized as DataType.FLOAT but set to DataType.HALF
[02/01/2024-10:55:11] [TRT-LLM] [W] Parameter was initialized as DataType.FLOAT but set to DataType.HALF
[02/01/2024-10:55:15] [TRT-LLM] [W] Parameter was initialized as DataType.FLOAT but set to DataType.HALF
[02/01/2024-10:55:15] [TRT-LLM] [W] Parameter was initialized as DataType.FLOAT but set to DataType.HALF
[TensorRT-LLM] TensorRT-LLM version: 0.8.0.dev2024012302Traceback (most recent call last):
  File "/TensorRT-LLM/examples/whisper/build.py", line 365, in <module>
    run_build(args)
  File "/TensorRT-LLM/examples/whisper/build.py", line 359, in run_build
    build_encoder(model, args)
  File "/TensorRT-LLM/examples/whisper/build.py", line 210, in build_encoder
    network.plugin_config.to_legacy_setting()
AttributeError: 'PluginConfig' object has no attribute 'to_legacy_setting'

TensorRT-LLM downloaded from pip has a version tag: TensorRT-LLM version: 0.8.0.dev2024012302
additional notes
NA

We have fixed this issue, please retry it. Thanks!


List failed and finished dataflows in dora list

List failed and finished dataflows on dora list, in addition to the active ones. This makes it easier to find out whether a started dataflow was successful or not.
Example Output:
❯ dora list
No dataflows are running
Failed dataflows:
- [grateful-song] 019010f5-9235-7d67-8fbc-1e82c16a8334
Finished dataflows:
- [nasty-statement] 019010f2-cb23-7274-bc2e-f383506770a0
- [pretty-carpenter] 019010f2-e297-74fc-a765-8b80e4e1ab97
- [encouraging-cough] 019010f3-a788-748c-9196-6104d58cd9d

Could be good to represent them within a table format instead of a list format in line with docker ps:
See: https://docs.docker.com/reference/cli/docker/container/ls/
We could list running dataflow first and finished dataflow at the end.

For the tab we can probably just use this: https://github.com/BurntSushi/tabwriter

Sure, I renamed the status fields in d2e2eef9. I also refactored the message a bit in 262f39fb to make the ordering more consistent (failed and succeeded dataflows are now listed chronologically instead of in separate groups).
Feel free to merge if this looks good to you now!


Create ".logs" directory for ibmi-bob

Is your feature request related to a problem? Please describe.
Slight convenience feature.
Describe the solution you'd like
When running "Launch Actions Setup" and checking "ibmi-bob", shouldn't it create a .logs directory if one doesn't exist?
Describe alternatives you've considered
I can easily create it manually

The .logs directory is created by the BOB tooling and vscode-ibmi download.  There is no need to manually create it.

@worksofliam
@edmundreinhardt
What am I doing wrong?  Here's the scenario:
Cloned the https://github.com/IBM/bob-recursive-example project from Github to a local directory on my PC.
Created Launch Actions Setup and selected ibmi-bob.
Did a build all, which deploys then fails because I forgot to change &lib1 to &LIB1 in iproj.json.
Fixed iproj.json.
Did build all again.
The build completes, but I get this failure:

The .logs directory does not exist on my PC.
The .logs directory is created on PUB400.

@SJLennon do you still get this error if you create the directory manually? Perhaps something is happening with the directory creation.

@worksofliam If I create it manually in File Explorer the download works.

What action are you running to run Bob? Please share the json, thanks!

[
  {
    "extensions": [
      "GLOBAL"
    ],
    "name": "Build all",
    "command": "OPT=*EVENTF BUILDLIB=&CURLIB /QOpenSys/pkgs/bin/makei build",
    "environment": "pase",
    "deployFirst": true,
    "postDownload": [
      ".logs/"
    ]
  },
  {
    "extensions": [
      "GLOBAL"
    ],
    "name": "Build current",
    "command": "OPT=*EVENTF BUILDLIB=&CURLIB /QOpenSys/pkgs/bin/makei compile -f &BASENAME",
    "environment": "pase",
    "deployFirst": true,
    "postDownload": [
      ".logs/"
    ]
  }
]

Built with:

@sebjulliand Edmund is out and I don't really use ibmi-bob. Is the .logs directory something you're familiar with? Are you interested in getting together Monday to debug together?

@worksofliam indeed I am; I do the same in Elias actually.
I tried @SJLennon use case and had the same issue. After a bit of debugging, it turns out this error shows when trying to create the folder locally:
Error (FileSystemError): Unable to resolve filesystem provider with relative file path 'c:\Temp\bob-recursive-example\.logs'

It seems that vscode.workspace.fs.createDirectory (at least on Windows) is not happy with what it's being given. I'll create a PR to fix that 😉


Contributed tasks don't run from Ctrl+P > task menu

Issue Type: Bug
When running a task contributed by an extension from Ctrl+P > "task", then expanding one of the contributed task groups and selecting a task, nothing happens.
To reproduce:

Clone https://github.com/joelspadin-garmin/vscode-private-extension-manager and open the code workspace.
Press Ctrl+P and type "task" followed by a space.
Under the "contributed" section, select "npm"
Select the "npm: install extension" task.

Expected behavior: a new terminal opens and runs "npm install" in the extension folder.
Actual behavior: nothing happens.
If I use Ctrl+Shift+P > Tasks: Run Task instead and then select the task the same way, the task runs.
This continues to happen even with all extensions disabled, just using the tasks provided by built-in extensions. It also happens for any contributed tasks, as I originally discovered this when trying to write a new extension that contributes taskDefinitions.
VS Code version: Code 1.45.1 (5763d909d5f12fe19f215cbfdd29a91c0fa9208a, 2020-05-14T08:27:35.169Z)
OS version: Windows_NT x64 10.0.17763

System Info

Item
Value

CPUs
Intel(R) Core(TM) i9-9980XE CPU @ 3.00GHz (36 x 3000)

GPU Status
2d_canvas: enabledflash_3d: enabledflash_stage3d: enabledflash_stage3d_baseline: enabledgpu_compositing: enabledmultiple_raster_threads: enabled_onoop_rasterization: disabled_offprotected_video_decode: unavailable_offrasterization: enabledskia_renderer: disabled_off_okvideo_decode: enabledviz_display_compositor: enabled_onviz_hit_test_surface_layer: disabled_off_okwebgl: enabledwebgl2: enabled

Load (avg)
undefined

Memory (System)
63.71GB (35.40GB free)

Process Argv

Screen Reader
no

VM
0%

@joelspadin-garmin thank you for filing this issue! You can get the fix ahead of our regular stable release by installing the insiders build. It installs side by side with stable.

Thanks!


solves issue  #24

Pull Request
This PR closes the issue #24
Description
I have linked the logo to the home page

@AMISH-KHAN
While running the checks, the lint and authorization checks failed. We only review when all checks are passed.

How can I solve this?

How can I solve this?

Take the latest pull from upstream/develop

git pull upstream develop

Install the dependencies

npm install

Run the lint check

npm run lint

Fix the errors
Stage your changes

git add .

 @AkashSharma001 please review it.

@AkashSharma001
Remove the contributor.md file from the pull request.

@AMISH-KHAN
Remove the contributor.md file from the pull request.

@AMISH-KHAN Remove the contributor.md file from the pull request.

are you suggesting to delete the file CONTRIBUTING.md or undo the changes I made accidentally?

@AMISH-KHAN Remove the contributor.md file from the pull request.

are you suggesting to delete the file CONTRIBUTING.md or undo the changes?

Removing the CONTRIBUTING.md file changes from your pull request is being suggested.
I think you have not taken the latest pull from upstream/develop.

hey @chandel-aman  please review it.
also when ever i commit I get this in my console which modifies the contributing.md file

@chandel-aman please add gssoc lable


carlito: Use stdenv.mkDerivation

As a stress test (of myself, my workstation, and of nixpkgs), I am trying to nixos-rebuild into nixos 22.11 without using any substituters, and fixing build failures along the way.
Description of changes
carlito used to call fetchzip directly, and the fact that it was a fixed-output derivation meant nobody noticed that:

the derivation no longer built from source (due to changes in fetchzip), and

the updated fontconfig xml never got used.

Things done

Built on platform(s)

[X] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
23.05 Release Notes (or backporting 22.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[ ] Fits CONTRIBUTING.md.

We should keep using fetchzip but fix the hash

Then this is a general issue for many fonts. The main idea behind this is to reduce the amount of derivations because they always add a small overlay which can be avoided when we are just copying files from $src to $out.

That's a clever trick, but optimisations must be correctness-preserving, and that fetchzip trick is not. We have massive silent breakage across the font packages as a result.

Disagree. fetchzip now splices the postFetch argument into its own postFetch and if it changes interface again, we won't know until someone tries to rebuild (because nobody would think to check the hash of its callers).
I could rewrite it to use runCommandNoCC or something instead?

Yeah, another behaviour change would break this again.


Fixes Fetch and Robotic environments initial state issues in 1.3.0

Adding back the lines described in #251 to fix the initial state changes introduced in gymnasium-robotics>=1.3.0, <=1.3.1.
Description
Adding back removed data setting lines removed in 1.3.0. Following Gymnasium's mujoco_env.py workflow, the procedure to reset an environment for Fetch and Robotic environments are:

mujoco._mj_resetData(self.model, self.data)
Update self.data to the saved initial data.
mujoco.mf_forward(self.model, self.data)

The second step changes for different environments. Changes in 1.3.0 removed the second step. The third step is called in set_state() in mujoco_env.py.
Update impacted Fetch environments to V4 and HandReach environment to V3. Other Shadow Dexterous Hand environments are not impacted since they inherit MujocoManipulateEnv which has a overridden _reset_sim().
Fixes #251
Type of change
Please delete options that are not relevant.

[ ] Documentation only change (no code changed)
[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

Screenshots
Please attach before and after screenshots of the change if applicable.

Checklist:

[x] I have run the pre-commit checks with pre-commit run --all-files (see CONTRIBUTING.md instructions to set it up)
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[x] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] New and existing unit tests pass locally with my changes

Should I update the test file as well? The one I'm using to test it is comparing the self.data.qpos and self.initial_qpos. Same goes for the qvel. This seems a little bit redundant.

In gymnasium we have this test, but I do not believe this applies to these environments because we do not have reset noise scale parameter.
https://github.com/Farama-Foundation/Gymnasium/blob/13230f4a120336b45c880b644585056d31f4c956/tests/envs/mujoco/test_mujoco_v5.py#L680-L712

In gymnasium we have this test, but I do not believe this applies to these environments because we do not have reset noise scale parameter. https://github.com/Farama-Foundation/Gymnasium/blob/13230f4a120336b45c880b644585056d31f4c956/tests/envs/mujoco/test_mujoco_v5.py#L680-L712

Specific to the impacted environments, the problem would be the randomized object location in Push, PickAndPLace and Slide. I think it's fair to ignore the object location since they are computed from the initial gripper position. If that's ok, I can modify the function and add a similar test function for Fetch environments.

Add the test

also tests/envs/hand/test_reach.py:11  should be updated to v3

I added the test in test_envs.py and update test_reach.py's test version.

I'm currently looking into how to fix starting position of the "Fetch" environment.
Is this PR valid? Could you let me know if this version is appropriate to apply in the latest source code, or should I stick with the current version? (if so, is this asymmetrical starting position is okay to achieve proper rewards?)

I'm currently looking into how to fix starting position of the "Fetch" environment. Is this PR valid? Could you let me know if this version is appropriate to apply in the latest source code, or should I stick with the current version? (if so, is this asymmetrical starting position is okay to achieve proper rewards?)

This PR is for fixing the Fetch-V3 environments' incorrect starting position introduced in the deterministic rollout fix in 1.3.0. I'm working with the maintainer to incorporate the fix to the main branch.
Are you facing the "asymmetrical starting position" in gymnasium-robotics<1.3.0? If not, then likely the issue is caused by the fix as well and you are welcome to try the code in this PR.
If you are still facing that issue in gymnasium-robotics<1.3.0, then it would be better for you to open a new bug report and provide more details so maintainers are aware of the issue.

Thank you for fixing this issue! I was able to train a PPO agent on Fetch Reach successfully by installing gymnasium robotics from source. Do you know when this might be available via pip?
FetchReachDense-v4

Thanks at @Kuds For the validation.
The next release should be around New Years.


Enhancement: Supports adding third-party dependencies from git repo with the version field

1. Does this PR affect any open issues?(Y/N) and add issue references (e.g. "fix #123", "re #123".):

[ ] N
[x] Y

fix : #266
2. What is the scope of this PR (e.g. component or file name):
pkg/client/client.go
3. Provide a description of the PR(e.g. more details, effects, motivations or doc link):

[x] Affects user behaviors
[x] Contains syntax changes
[x] Contains variable changes
[ ] Contains experimental features
[ ] Performance regression: Consumes more CPU
[ ] Performance regression: Consumes more Memory
[ ] Other

I simply changed the logic of AddDepToPkg function to handle version field at the time of adding the dependencies to the pkg.
4. Are there any breaking changes?(Y/N) and describe the breaking changes(e.g. more details, motivations or doc link):

[x] N
[ ] Y

5. Are there test cases for these changes?(Y/N) select and add more details, references or doc links:

[x] Unit test
[ ] Integration test
[ ] Benchmark (add benchmark stats below)
[ ] Manual test (add detailed scripts or steps below)
[ ] Other

I directly modified the test_data for testing the unit tests. No need to make any other changes in unit test.

Hi @Peefy I was getting conflicts in the PR so here I have opened another one. But I still see the unit tests failing which are passing in my local see :

cc: @zong-zhe

The CI failed. cc @Manoramsharma

The CI failed. cc @Manoramsharma

@zong-zhe told me to stop this PR until the #384 will be completed.


Array order check is too strict

Reporting a bug

[x] I am using the latest released version of Numba (most recent is visible in
the change log (https://github.com/numba/numba/blob/master/CHANGE_LOG).
[x] I have included below a minimal working reproducer (if you are unsure how
to write one see http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports).

Numpy identifies 1 x n or n x 1 matrices as both C- and F-contiguous. Such matrices fail to pass numba's contiguity check as F-contiguous.
Code example
>>> from numba import njit
>>> import numpy as np
>>>
>>> # This definition generates a warning
>>> @njit('f8[:, :], f8[:]')
... def prod(A, x):
...     return A.T @ x
...
<ipython-input-9-ed5020607679>:3: NumbaPerformanceWarning: '@' is faster on contiguous arrays, called on (array(float64, 2d, A), array(float64, 1d, A))
  return A.T @ x
/Users/lizli/anaconda3/lib/python3.7/site-packages/numba/typing/npydecl.py:967: NumbaPerformanceWarning: '@' is faster on contiguous arrays, called on (array(float64, 2d, A), array(float64, 1d, A))
  warnings.warn(NumbaPerformanceWarning(msg))
>>> # OK, I will be more specific
>>> @njit('f8[::1, :], f8[::1]')
... def prod(A, x):
...     return A.T @ x
...
>>> # All good
>>> prod(np.zeros((2, 2), order='F'), np.zeros(2))
array([0., 0.])
>>> # But there are cases where the matrix can be both C and F
>>> A = np.zeros((2, 1), order='F')
>>> print(A.flags)
  C_CONTIGUOUS : True
  F_CONTIGUOUS : True
  OWNDATA : True
  WRITEABLE : True
  ALIGNED : True
  WRITEBACKIFCOPY : False
  UPDATEIFCOPY : False
>>> prod(A, np.zeros(2))
Traceback (most recent call last):
  File "<ipython-input-18-e21ff4da4503>", line 1, in <module>
    prod(A, np.zeros(2))
  File "/Users/lizli/anaconda3/lib/python3.7/site-packages/numba/dispatcher.py", line 466, in _explain_matching_error
    raise TypeError(msg)
TypeError: No matching definition for argument type(s) array(float64, 2d, C), array(float64, 1d, C)

>>>

Further information:

Python 3.7.3
Numpy 1.16.4
Numba 0.44.1

@lzlarryli thanks for reporting this! I have just been able to reproduce this and can verify that is probably a bug in Numba.

Code to reproduce:
from numba import njit
import numpy as np

@njit('f8[:, :], f8[:]')
def prod(A, x):
    return A.T @ x

@njit('f8[::1, :], f8[::1]')
def prod(A, x):
    return A.T @ x

prod(np.zeros((2, 2), order='F'), np.zeros(2))

A = np.zeros((2, 1), order='F')
print(A.flags)
prod(A, np.zeros(2))

I've a feeling that this is a duplicate as I'm sure I've looked at this before. Fundamentally the issue is that the dispatcher checks the order of the array in the order of 'C', 'F' and if neither it assigns 'A'. I'm not sure how easy it would be to accommodate the above situation as I anticipate the above logic is implicitly assumed in a number of places in the code base.

Possibly related? #4585, #6998


Use HTTP instead of git otherwise it's doesn't work on dotcloud.

Actually I'm not sure if it even needs to depend on my fork of easyrss anymore... I recall I made some small changes that probably are not even needed anymore.


Development: Add helper script to convert Spring YAML configuration files to their equivalent ENV format

Checklist
General

[ ] I tested all changes and their related features with all corresponding user types on a test server.
[x] This is a small issue that I tested locally and was confirmed by another developer on a test server.
[x] Language: I followed the guidelines for inclusive, diversity-sensitive, and appreciative language.
[x] I chose a title conforming to the naming conventions for pull requests.

Motivation and Context
Spring configurations can be given either as YAML files, properties files, or as environment variables due to the configuration binding.
For Docker/Kubernetes setups it might be easier to use the ENV-variable option.
Description

To simplify the conversion from one setup type to the other, this script automates the conversion of the configuration from YAML to ENV.
The same script is also in https://github.com/b-fein/spring-yaml-to-env including some tests and linting set up. Therefore, instead of adding it here, I could also just host it in my own repo and just link it somewhere in the admin documentation here.
Steps for Testing

Try to convert your configuration files using the script.
The script should not print any warnings or errors to the console.
(Optional, might be trickier to test):

If you have got a Systemd-Service that starts Artemis, you can define an EnvironmentFile=… there that contains the content of the script output. Artemis should then still start when the yml configs are removed.
If you have got a Docker setup, you can use the environment variables there as well.
NOTE: This does not work currently if you have defined custom Docker images for the CI-system runners (options artemis.continuous-integration.build.images.…). This is not due to the script, but due to how Spring converts the ENV-variable names back into lowercase. The Artemis code that handles the loading in this case cannot handle the image names being in lowercase. ⇒ Fix in #5596

Review Progress

Code Review

[ ] Review 1
[ ] Review 2

Manual Tests

[x] Test 1
[x] Test 2

Test Coverage
unchanged

Thanks here already for this cool helper script! I already used it a lot for converting the yaml configs to env files. I still have to do stuff manually like re-adding comments and sorting keys out etc. But this is something you can't fully automate I guess so it is a nice helper! I would suggest putting the script somewhere external as it could also help other spring developers if the env file approach is working out?! Then I would just link it in the docs here when we merge our docker-compose setup?! @Mtze what do you think about this?
I tested this newer version of your script with Python3.8.10 again and didn't detect any issues. I also used the previous version already for starting up Docker-Compose and also didn't find any issues coming from the script yet.

I would suggest putting the script somewhere external as it could also help other spring developers if the env file approach is working out?!

In the meantime I already created a repo for that: https://github.com/b-fein/spring-yaml-to-env


Fix docs for Volcano integration

Docs had old chart flags for enabling batch scheduler/webhook, so it was not working.
@liyinan926 can you help merge this?

+1.
After some debugging, I finally found out that this helm install command was outdated :(
Hope this would get fixed soon.


Problem using oauth_code_login

I'm not sure if I am doing this right, but I wanted to capture the oauth code that I see when using oauth_desktop_login and pass it in a later session to oauth_code_login so that I don't have to keep logging in.
So what I do is, when I use oauth_desktop_login, a browser window pops up, I type in my sandbox username/password. The browser then goes to http://localhost:63342/fslogin?code=some_code_here, I copy the code (everything after the =), store it in a variable, then call oauth_code_login with that code as the parameter. Unfortunately, this results in a 400 Bad Request error.
Am I doing this right and there is some bug in the oauth_code_login? Is there a better way to not have to log in every time before getting the app approved (at which case I can just use login)?

Hmm. I'm a little busy with other things right now, but I'll poke around a bit.
The biggest issue I'm seeing is, as it currently stands, we're making a global variable of the code.


Added the icon to set dark and bright theme for the website (Issue #675)

Hello PA,
I have addedthe icon to set dark and bright theme for the website.Please review my code changes and approve it .
###screenshot-

Solve Conflicts


[Bug]: Tolerations have no validation

Is there an existing issue for this?

[X] I have searched the existing issues

Current Behavior
Tolerations field in the Cluster Settings page for an Admin do not protect against any misuse of the field.
There are restrictions to the field under the hood, like having a valid K8s name (https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names) that we can do to help the admin out.
Tracking the valid relationship of the toleration is out of scope.
Expected Behavior
Present the user with additional error help when they have an invalid name.

"name part must consist of alphanumeric characters, '-', '' or '.', and must start and end with an alphanumeric character (e.g. 'MyName', or 'my.name', or '123-abc', regex used for validation is '([A-Za-z0-9][-A-Za-z0-9.]*)?[A-Za-z0-9]')"

Steps To Reproduce

Be an admin
Go to Cluster Settings page
Type an invalid name with spaces or non-supported special characters

Workaround (if any)
No response
OpenShift Infrastructure Version
No response
Openshift Version
No response
What browsers are you seeing the problem on?
No response
Open Data Hub Version
2.1

Relevant log output
No response

If part of this work we can do a wider box to at least support the default value would be awesome.


[HW] Al-Amir Adegbuji-Onikoyi

Name: Al-Amir Adegbuji-Onikoyi
Email<EMAIL_ADDRESS>Linkedin Profile: https://www.linkedin.com/in/al-amir/
Attach the homework step I screenshots below:

Attach the homework step II screenshots below:

Thanks Al-Amir! Well done!

Hey Al, great job! Congrats! Here is your badge! https://api.badgr.io/public/assertions/mMNpRvBnQ5y9GY5kEaKhVA


fix(nuxt): exclude plugin declaration from non-existent files

🔗 Linked issue

❓ Type of change

[ ] 📖 Documentation (updates to the documentation, readme or JSdoc annotations)
[x] 🐞 Bug fix (a non-breaking change that fixes an issue)
[ ] 👌 Enhancement (improving an existing functionality like performance)
[ ] ✨ New feature (a non-breaking change that adds functionality)
[ ] 🧹 Chore (updates to the build process or auxiliary tools and libraries)
[ ] ⚠ Breaking change (fix or feature that would cause existing functionality to change)

📚 Description
This PR fixes an issue where non-existent (i.e. virtual) plugins would cause type inference for plugin injections to fail.
📝 Checklist

[ ] I have linked an issue or discussion.
[ ] I have updated the documentation accordingly.

I was wondering if this could be somehow backported to 3.4.x with some kind of user-land code? Type inference breaks in <3.5.1 due to InjectionType<typeof import("../components.plugin").default> which is fixed here.

OK, it was actually simple:
// Fix useNuxtApp() return type missing data injected by plugins.
//
// The problem was fixed in Nuxt 3.5.1, please remove after upgrading.
// See https://github.com/nuxt/nuxt/pull/20974#issuecomment-1592724945

import { addTemplate, defineNuxtModule } from '@nuxt/kit'

export default defineNuxtModule({
  setup(_option, nuxt) {
    addTemplate({
      getContents: () =>
        'declare const _default: any; export default _default;',
      filename: 'components.plugin.d.ts',
      dst: nuxt.options.buildDir + '/components.plugin.d.ts',
      write: true,
    })
  },
})


[Docker Hub] Consolidated signup and upgrade flow

Tell us about your request
Consolidate the Docker ID signup and upgrade flows into a single experience that enables Docker Hub users to quickly choose a plan that suits their needs.
Which service(s) is this request for?
Docker Hub
Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
We frequently hear Docker Hub users cite a lack of awareness around the advanced capabilities that Docker Hub offers for individual developers and development teams, including parallel builds and private repositories. This is due in part to the fact that the upgrade flow is located in the Billing section, which makes it difficult for Docker Hub users to learn about and sign up for advanced capabilities.
Are you currently working around the issue?
NA
Additional context
NA

Shipped!


⚠ Topfarmacia has degraded performance

In 8fd7e21, Topfarmacia (https://www.topfarmacia.it) experienced degraded performance:

HTTP code: 200
Response time: 7288 ms

Resolved: Topfarmacia performance has improved in 06dc295.


Add patch to fix the pagesize error in v8 on MIPS64EL

See https://github.com/electron/electron/pull/11208#issuecomment-346545546.

@zcbenz I have a simple modify, and only considered loongson platform.if you merge to libchromiumcontent, you must consider other platform(such as, x86 is 4K pagesize).
I think you set 16384 directly is not acceptable.

@ClarkWang-2013 This patch is only applied when building for mips64el, so it is fine.

I'm merging this since it solves the v8 crash and does not affect other builds.


chore: add prettier lint pre commit for markdown files

Summary
Add Markdown pre-commit linter
Changes
Added https://github.com/pre-commit/mirrors-prettier  as pre-commit linter for markdown files
User experience
End user experience does not change.
Changes made by developers are easier to align into single pattern.
Checklist
If your change doesn't seem to apply, please leave them unchecked.

[ ] I have performed a self-review of this change
[ ] Changes have been tested
[ ] Changes are documented
[ ] PR title follows conventional commit semantics

Unfortunately, this prettier mirror by pre-commit seems to be deprecated and archived.
Maybe something like markdown lint project?

Unfortunately, this prettier mirror by pre-commit seems to be deprecated and archived.
Maybe something like markdown lint project?

true i missed that one that it is deprecated, changed into that markdownlint , seems like it is one of most commonly used
https://github.com/igorshubovych/markdownlint-cli


Update reimplemented cw1-whitelist contract, so it uses the custom attribute

Part of #493, follow up of #495

This is part of Sylvia now. Closing!


Added followPlaylist and unfollowPlaylist

https://github.com/jwilsson/spotify-web-api-php/issues/21

I think we should pass public as an array instead and make it optional to keep it consistent with other methods (and just in case Spotify decides to add more options we can just extend the array), i.e.
$options = array(
    'public' => true
);

$api->followPlaylist('USER_ID', 'PLAYLIST_ID', $options);

updateUserPlaylist() is a great example of what I'm thinking of.

See latest change. Let me know if this is ok

Perhaps I was unclear, sorry about that.
I meant that the $options parameter should be optional as well.

I meant that the $options parameter should be optional as well.

I got confused because data is not optional in updateUserPlaylist($userId, $playlistId, $data)

Then it's definitely my fault, sorry mate!
I'll just run some tests tonight but everything looks good so far.


[v3]: Fixed scroll to top button bug

Fixed scroll-to-top button bug discussed here #2917

thanks for the attempt, but this solution is too overcomplicated, fixed by https://github.com/shuding/nextra/pull/2981


XDMF-Timeseries Example from pypi raises Attribute Error

Hi,
I cannot tell how much I appreciate the package. Thanks!
The following code from the exampel section of the pypi website raises an AttributeError:
import meshio
points = [
    [0.0, 0.0, 0.0],
    [0.0, 1.0, 0.0],
    [0.0, 0.0, 1.0],
]
cells = [("triangle", [[0, 1, 2]])]
filename = "./delete_me_xdmf_example.xdmf"
with meshio.xdmf.TimeSeriesWriter(filename) as writer:
    writer.write_points_cells(points, cells)

Meshio version: 4.3.8
From: conda-forge
Edits:

Add filename
Remove obviously not working solution suggestion

Indeed, whereas
meshio.Mesh(points, cells)

is fine.

    writer.write_points_cells(np.array(points), [(t, np.array(c)) for t, c in cells])

is a makeshift, but I reckon this should be fixed—should be easy enough, I'll take a look.

Super cool, thanks!


DS-702 - [B2B] When a user clicks on the multiple attributes formula option, the formula screen overlaps

Clicking on a new formula now closes the previous one, preventing overlap.

@jugnu-appveen


[BUG] settings.json is present, but settings.yaml is required?

Describe the bug
I've just cloned the project, then run docker-compose up from root. The following error appears:
nostream          | node:internal/fs/watchers:255
nostream          |     throw error;
nostream          |     ^
nostream          |
nostream          | Error: ENOENT: no such file or directory, watch '/home/node/.nostr/settings.yaml'
nostream          |     at FSWatcher.<computed> (node:internal/fs/watchers:247:19)
nostream          |     at Object.watch (node:fs:2343:34)
nostream          |     at SettingsStatic.watchSettings (/app/src/utils/settings.js:107:26)
nostream          |     at App.run (/app/src/app/app.js:35:51)
nostream          |     at Object.<anonymous> (/app/src/index.js:33:30)
nostream          |     at Module._compile (node:internal/modules/cjs/loader:1254:14)
nostream          |     at Module._extensions..js (node:internal/modules/cjs/loader:1308:10)
nostream          |     at Module.load (node:internal/modules/cjs/loader:1117:32)
nostream          |     at Module._load (node:internal/modules/cjs/loader:958:12)
nostream          |     at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:81:12) {
nostream          |   errno: -2,
nostream          |   syscall: 'watch',
nostream          |   code: 'ENOENT',
nostream          |   path: '/home/node/.nostr/settings.yaml',
nostream          |   filename: '/home/node/.nostr/settings.yaml'
nostream          | }
nostream          |
nostream          | Node.js v18.16.0
nostream exited with code 1

Digging a bit deeper, I see that .nostr.local/settings.json exists, thus is made available when volumes: - ${PWD}/.nostr:/home/node/.nostr is called in the the docker-compose.yml
However, there is no settings.yaml file included in that folder, which causes the error above.
A bit of inspection of the codebase shows that settings.spec.ts references settings.json a couple of times, and it's mentioned in the README, but nowhere else does it appear. The rest of the codebase expects settings.yaml.
Why is settings.json included, and not settings.yaml? The instructions state that settings.json is being auto-created, but then it is never consumed?
I'll copy resources/default-settings.yaml by hand as-per the instructions, but it seems it should be auto-generated instead of a never-used settings.json, eh?

README?

Got it to work with:
cp resources/default-settings.yaml ./.nostr/settings.yaml

@amunrarara I'll try and make this work automatically with docker compose up, but please use ./scripts/start to start Nostream. That script takes care of creating the settings.yaml you are missing.
The .nostr.local isn't used unless you are developing locally, I will get rid of that functionality as it has implemented but never used and causes confusion.
Closing this ticket for now.


findlayer err

on=@gethit
serv.log  <src.name> <SRC.FINDLAYER(layer_hand1).color> <SRC.FINDLAYER(layer_hand2).color>
SRC has no weapons

It is the right behavior, now Sphere is more strict with scripting and it's a new type of error.
You're trying to access the COLOR of a nonexistant item, because there is nothing in the layers 1 and 2. You have to check if there's an item, then you can check its properties

It is the right behavior, now Sphere is more strict with scripting and it's a new type of error.
You're trying to access the COLOR of a nonexistant item, because there is nothing in the layers 1 and 2. You have to check if there's an item, then you can check its properties

More than 10 years of writing habits have been overturned
A large number of error reporting scripts, and need a lot of modification.
Even the default script of spheresrv will report an error
Is this really necessary

More than 10 years of writing habits have been overturned
A large number of error reporting scripts, and need a lot of modification.
Even the default script of spheresrv will report an error
Is this really necessary

Is this really necessary

We had the same debate you have. Personnaly on my shard there TON of error too but. I fixed them and trust me my script are more optimal now. It permit me to find some bad reference on script that I was not aware (the major purpose of this modif).
I'm agree about Findid and findlayer there sometime some aberation when 1 line become 4 line. But all the refx, argo etc it's very usefull.
I think it's an adaptation for the best.  Few hour of fixing it and you'll have permanent more strong script.

Is this really necessary

We had the same debate you have. Personnaly on my shard there TON of error too but. I fixed them and trust me my script are more optimal now. It permit me to find some bad reference on script that I was not aware (the major purpose of this modif).
I'm agree about Findid and findlayer there sometime some aberation when 1 line become 4 line. But all the refx, argo etc it's very usefull.
I think it's an adaptation for the best.  Few hour of fixing it and you'll have permanent more strong script.

Won't more queries cause more processing and more overhead?

Won't more queries cause more processing and more overhead?

If you mean repeating more times stuff like FINDLAYER, one for checking the validity, and another to execute the operation then yes. In this case i suggest storing the result in REF1.
Also, yes, longer scripts require longer parsing, but it's a price to be paid to eliminate those obscure bugs with no console errors

If you mean repeating more times stuff like FINDLAYER, one for checking the validity, and another to execute the operation then yes. In this case i suggest storing the result in REF1.
Also, yes, longer scripts require longer parsing, but it's a price to be paid to eliminate those obscure bugs with no console errors


[Feature]:  tool scrip knowledge_init.py support multiple knowledge file path and skip some error in knowledge embedding

Is your feature request related to a problem? Please describe.
There are a lot of documents in my machine, and they are still being modified and changed. Currently knowledge_init.py only supports reading documents from the datasets directory.
Describe the solution you'd like

Add a file_path command line parameter, the default is DATASETS_DIR
Add an skip_doc_error command line parameter, the default is False,  skip some errors for some documents.

Additional context
I will submit a pr later.

Is your feature request related to a problem? Please describe.
There are a lot of documents in my machine, and they are still being modified and changed. Currently knowledge_init.py only supports reading documents from the datasets directory.
Describe the solution you'd like

Add a file_path command line parameter, the default is DATASETS_DIR
Add an skip_doc_error command line parameter, the default is False,  skip some errors for some documents.

Additional context I will submit a pr later.

thanks for your opinion, knowledge_init.py is knowledge tool before DBGPT0.3.0, now we use knowledge space  webui
to manage the knowledge.

Is your feature request related to a problem? Please describe.
There are a lot of documents in my machine, and they are still being modified and changed. Currently knowledge_init.py only supports reading documents from the datasets directory.
Describe the solution you'd like

Add a file_path command line parameter, the default is DATASETS_DIR
Add an skip_doc_error command line parameter, the default is False,  skip some errors for some documents.

Additional context I will submit a pr later.

thanks for your opinion, knowledge_init.py is knowledge tool before DBGPT0.3.0, now we use knowledge space webui to manage the knowledge.

Thanks for your reply, as far as I know, the current webui only supports selecting one file at a time, which is very inconvenient to use, and I expect to use dbgpt without webui, so I think we can keep the previous knowledge_init.py.

Is your feature request related to a problem? Please describe.
There are a lot of documents in my machine, and they are still being modified and changed. Currently knowledge_init.py only supports reading documents from the datasets directory.
Describe the solution you'd like

Add a file_path command line parameter, the default is DATASETS_DIR
Add an skip_doc_error command line parameter, the default is False,  skip some errors for some documents.

Additional context I will submit a pr later.

thanks for your opinion, knowledge_init.py is knowledge tool before DBGPT0.3.0, now we use knowledge space webui to manage the knowledge.

Thanks for your reply, as far as I know, the current webui only supports selecting one file at a time, which is very inconvenient to use, and I expect to use dbgpt without webui, so I think we can keep the previous knowledge_init.py.

According to the conclusion of my private communication with @Aries-ckt , I will modify knowledge_init.py to call knowledge rest  api to realize this feature.


Module update

Updates RNA modules; previous rna modules used unscaled combat results

We're seeing better performance if we don't scale rna prior to WGCNA, so we'll continue to use the old results.


Fixes and improvements to benchmark script

Motivation and Context
The benchmark.py script assumed the presence of a test.glb file and had unexpected behavior when the file was not present.  Moreover, multi-process benchmarking was not incorporated, making it hard to reproduce the benchmark table out-of-the-box.  @danielgordon10 thanks for bringing up this issue.
This PR removes defaulting of the --scene argument from both benchmark.py and example.py and requires the user to specify the intended scene, avoiding confusing behavior and unexpected errors.  It also incorporates multi-process benchmarking implemented by @erikwijmans .
How Has This Been Tested
Download the Matterport3D Habitat data and run examples/benchmark.py --scene /path/to/mp3d/17DRP5sb8fy/17DRP5sb8fy.glb. The benchmark script should output a table similar to the one below:
 ================ Performance (FPS) NPROC=1 ===================================
Resolution             rgb           rgbd     depth_only  semantic_only  rgbd_semantic
128 x 128         4288.3         2052.2         4266.9          540.3          429.1
256 x 256         3162.0         1704.4         3245.0          518.6          387.3
512 x 512         1611.6          803.4         1614.5          451.1          280.9
 ==============================================================================
 ================ Performance (FPS) NPROC=3 ===================================
Resolution             rgb           rgbd     depth_only  semantic_only  rgbd_semantic
128 x 128         8629.4         4258.8         8634.9          531.3          468.2
256 x 256         7483.4         3686.6         7462.1          522.5          456.1
512 x 512         4346.8         2228.6         4251.1          540.0          435.3
 ==============================================================================
 ================ Performance (FPS) NPROC=5 ===================================
Resolution             rgb           rgbd     depth_only  semantic_only  rgbd_semantic
128 x 128         8484.4         4178.5         8415.8          528.0          467.8
256 x 256         7444.6         3651.7         7423.0          522.4          455.7
512 x 512         7069.9         3503.1         7187.1          540.0          472.5
 ==============================================================================

Types of changes

[x] Docs change / refactoring / dependency upgrade
[x] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist

[x] My code follows the code style of this project.
[x] My change requires a change to the documentation.
[x] I have updated the documentation accordingly.
[x] I have read the CONTRIBUTING document.
[x] I have completed my CLA (see CONTRIBUTING)
[ ] I have added tests to cover my changes.
[ ] All new and existing tests passed.

Just follow up, what is our plan regarding this PR.

@mathfac thanks for the ping -- this PR is on my plate still. I will get it updated and ready for review soon.

Maybe we can use it in continues integration loop later.
On Wed, 15 May 2019 at 16:44, Manolis Savva<EMAIL_ADDRESS>wrote:

@mathfac https://github.com/mathfac thanks for the ping -- this PR is
on my plate still. I will get it updated and ready for review soon.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/facebookresearch/habitat-sim/pull/15?email_source=notifications&email_token=AAGWHAHRGSRQFKPY2HYMVF3PVSN63A5CNFSM4HFZT55KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVQICDQ#issuecomment-492863758,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAGWHAAYXVWKNALDUCH3LOTPVSN63ANCNFSM4HFZT55A
.

@mathfac @danielgordon10 @erikwijmans the PR is now updated with the following changes and ready for reviewing:

Fix output column header + fps alignment
Add parameter to enable semantic sensor (disabled by default)
Remove hardcoded resolution and nprocs and make into proper defaulted argparse arguments
Automatically download and extract test scene data if not present locally


Exception is wrapped by NoFallbackAvailableException even using Resilence4J @CircuitBreaker fallbackMethod

Describe the bug
I cannot find any information whether it is intended,
Exception is wrapped by NoFallbackAvailableException even using @CircuitBreaker fallbackMethod

Sample
This is my @FeignClient with @CircuitBreaker
@FeignClient(name = "ExchangeRateOpenFeign", url = "${exchange.currency.api.uri}")
public interface ExchangeRateOpenFeign {
     Logger log = LoggerFactory.getLogger(ExchangeRateOpenFeign.class);

    @CircuitBreaker(name = "currency", fallbackMethod = "fallback")
    @GetMapping
    ExchangeRateResponse call(
            @RequestHeader String apiKey,
            @RequestParam Currency source,
            @RequestParam Currency currencies);

    default ExchangeRateResponse fallback(
            @RequestHeader String apiKey,
            @RequestParam Currency source,
            @RequestParam Currency currencies,
            Throwable e) throws Throwable {

        log.error("Here it comes with error", e);
        throw e;
    }
}

Then test below is passed
@SpringBootTest
class ExchangeRateOpenFeignTest {

    @Autowired
    private ExchangeRateOpenFeign openFeign;

    @Test
    void errorWrappedWithNoFallbackAvailableException() {
        assertThatThrownBy(() -> openFeign.call("key", Currency.JPY, Currency.KRW))
                .isInstanceOf(NoFallbackAvailableException.class);
    }

}

Also I can get an error logs with NoFallbackAvailableException (Some logs have been omitted.)
2023-02-08 11:24:33.456 ERROR 65842 --- [pool-1-thread-1] c.m.o.a.openfeign.ExchangeRateOpenFeign  : Here it comes with error

org.springframework.cloud.client.circuitbreaker.NoFallbackAvailableException: No fallback available.
	at org.springframework.cloud.client.circuitbreaker.CircuitBreaker.lambda$run$0(CircuitBreaker.java:31) ~[spring-cloud-commons-3.1.5.jar:3.1.5]
	at io.vavr.control.Try.lambda$recover$6ea7267f$1(Try.java:949) ~[vavr-0.10.2.jar:na]
	at io.vavr.control.Try.of(Try.java:75) ~[vavr-0.10.2.jar:na]
	at io.vavr.control.Try.recover(Try.java:949) ~[vavr-0.10.2.jar:na]
	at org.springframework.cloud.circuitbreaker.resilience4j.Resilience4JCircuitBreaker.run(Resilience4JCircuitBreaker.java:133) ~[spring-cloud-circuitbreaker-resilience4j-2.1.5.jar:2.1.5]
	at org.springframework.cloud.client.circuitbreaker.CircuitBreaker.run(CircuitBreaker.java:30) ~[spring-cloud-commons-3.1.5.jar:3.1.5]
...

Caused by: feign.FeignException$BadRequest: [400] during [GET] to [http://localhost:8090/currency_data/live?source=JPY&currencies=KRW] [ExchangeRateOpenFeign#call(String,Currency,Currency)]: []
	at feign.FeignException.clientErrorStatus(FeignException.java:213) ~[feign-core-11.10.jar:na]
	at feign.FeignException.errorStatus(FeignException.java:194) ~[feign-core-11.10.jar:na]
	at feign.FeignException.errorStatus(FeignException.java:185) ~[feign-core-11.10.jar:na]
...

Since the fallbackmethod is specified, I don't think the error should be wrapped.
But it is wrapped where it is called from FeignCircuitBreakerInvocationHandler.java
What I found is that, it is happend by calling fallbackMethod from resilence4J FallbackDecorators.
As Object with @CircuitBreaker is proxy and when resilence4J calls fallbackMethod By DefaultFallbackDecorator,
FeignCircuitBreakerInvocationHandler.java is called and error is wrapped when fallback method is called
Thanks for reading!

Hey @MangKyu
Could you give more information about the spring version and cloud version you are using?
I created a sample repository to understand more about your problem. As far as I can see it doesn't throw NoFallbackAvailableException. One more question from my side why do you want to throw an exception on fallback?
Feel free to add a commit or create a branch so I can check on that.
https://github.com/cbezmen/can-feign/tree/b-MangKyu

@cbezmen hi can, Thanks for your kindness!
I uploaded my code on your project which is forked on my repository
Because 403 forbidden occurs
And i found that it occurs when spring.cloud.openfeign.circuitbreaker.enabled value is true.

Hey @MangKyu
I updated my branch. you can see diff via there or in here.
If you are using spring open feign, you shouldn't use @CircuitBreaker. There is already an implementation for fallbacks. There is documentation in here. Spring open-feign checks for fallback implementation in the @FeignClient annotation. If it is not implemented it will wrap it with NoFallbackAvailableException.

Thanks for reply!
I should avoid using @CircuitBreaker with Open Feign.
Thanks!


App Center Device Registration Service Certificate - Expired August 27, 2021

I noticed recently that using the 'Add device' from install.appcenter.ms downloads a App Center Device Registration Service Profile that lists the signing certificate for install.appcenter.ms as expired on August 27, 2021.

Hi @carmanj , the certificate is fixed now, could you please check it again?

Hi @carmanj , the certificate is fixed now, could you please check it again?

Hey Dmitriy - That took care of it! Thanks!


[BUG] - number of measurement points 0 returned by daq from device 'SHFQC/QA:DEV12188' does not match length of recipe measurement_map which is 524288

Describe the bug
Executing the following experiment results in the following error:
AssertionError: number of measurement points 0 returned by daq from device 'SHFQC/QA:DEV12XXX' does not match length of recipe measurement_map which is 524288

experiment.zip
"laboneq_compiler_settings": {
"HDAWG_MIN_PLAYWAVE_HINT": 400,
"HDAWG_FORCE_COMMAND_TABLE": false,
"SHFSG_FORCE_COMMAND_TABLE": true,
"SHFSG_MIN_PLAYWAVE_HINT": 128,
"SHFSG_MIN_PLAYZERO_HINT": 128
},
Versions used:

laboneq: 1.7

Hi @GlaserN, we are looking into this now.

@GlaserN - as discussed in our meeting, will close this ticket as implementing https://github.com/zhinst/laboneq/issues/26 should fix this issue.


Support caching streaming files that have known checksums

When working with remote files, we often have some metadata that we can use for caching. It would be nice to use gulp-cached to conditionally process streams based on that metadata, the same way it works for local files.
I propose setting a checksum property on the Vinyl file in a task that loads metadata, and checking that property in gulp-cached.
For example: https://github.com/uts-magic-lab/gulp-google-drive

Shouldn't this work for all files not just streaming ones?

Yes. If we can depend on file.checksum being set elsewhere, it makes sense to respect it in all circumstances.
I don't think it's possible to support automatically looking at common attributes such as file.stat.mtime or Etag without making the interface less clear.

Published as 1.1.0 - thanks!


wlfreerdp: mirror display resolution in fullscreen mode

This removes the need to specify both /w:1920 /h:1080 and /f. If obtaining the resolution fails the default will be used instead.

@freerdp-bot can you test that nice patch please ?

@hardening CI failed due to the TestConnect unit-test which I think is flaky. Can you kick off another build for me? 😬

@freerdp-bot test.

@hardening is this free to be merged?


Model-level validation corrupts field date serialization/validation for RESTful handler

We use CRUD API generated by the Zenstack with RESTful handler. We have never used the Model-level validation in our schema and decided to enable couple rules in one model. Apparently, there is some problem in Zod configuration or serialization layer in version 1.9.0. Without model-level validation rules we are able to add new entry via REST API but once we add "  @@validate" to the model it start throwing us error 403 with Zod validation message for fields with DateTime type.
schema.zmodel:
generator client {
  provider        = "prisma-client-js"
  previewFeatures = ["driverAdapters", "postgresqlExtensions", "views", "tracing"]
}

datasource db {
  provider    = "postgresql"
  url         = env("DATABASE_URL_WITH_SCHEMA")
  extensions  = [uuidOssp(map: "uuid-ossp")]
}

plugin openapi {
  provider = '@zenstackhq/openapi'
  flavor = 'rest'
  output = './dist/crud-api.json'
  title = 'CRUD REST API'
  version = '1.0.0'
  summary = 'CRUD API'
  description = 'This API can be used to make calls to the database using REST. Created with ZenStack.'
  prefix = '/api/crud'
  securitySchemes = {
        bearer: { type: 'http', scheme: 'bearer', bearerFormat: 'JWT' }
    }
}

model participant_interaction {
  participant_interaction_id String @id @default(dbgenerated("uuid_generate_v4()")) @db.Uuid

  summary String?

  interaction_start DateTime? @db.Timestamp(0)
  interaction_duration_minutes Int?

  coach_id String @db.Uuid
  participant_id String @db.Uuid

  created_by String?
  created_timestamp DateTime? @db.Timestamp(6) @default(now())
  last_modified_by String?
  updated_timestamp DateTime? @db.Timestamp(6) @updatedAt()
  mongo_db_id String? @db.VarChar(255) @allow("update", false)

  //Columns for Legacy interaction type
  legacy_interaction_task_type InteractionTaskType?

   Nulla non neque mollis, placerat risus sit amet, vehicula justo. Integer in ipsum id tortor porta malesuada. Maecenas ac neque ex. Morbi non est id risus posuere luctus. Vestibulum ac euismod felis. Sed eget imperdiet lorem. Pellentesque id accumsan enim.",
      "interaction_start": "2024-03-02T05:00:00.000Z",
      "interaction_duration_minutes":25,
    }
  }
}

Response:
{
    "errors": [
        {
            "status": 403,
            "code": "forbidden",
            "title": "Operation is forbidden",
            "reason": "DATA_VALIDATION_VIOLATION",
            "zodErrors": {
                "issues": [
                    {
                        "code": "invalid_type",
                        "expected": "date",
                        "received": "string",
                        "path": [
                            "interaction_start"
                        ],
                        "message": "Expected date, received string"
                    }
                ],
                "name": "ZodError"
            }
        }
    ]
}

As soon as I remove "@@validation" from our model, everything works properly.
Environment:
package.json
tsconfig.json

ZenStack version: 1.9.0
Prisma version: 5.10.2
Database type: Postgresql

Hi @vgarmash , thanks for reporting this. I'll look into it and update back here soon.

When @@validate is used, a zod schema is generated and used to validate the input data. Currently the schema is too strict for DateTime field and requires a Date object. I've made a fix to allow compatible strings (via z.coerce.date). It's included in the latest v1.10.0 release. Please help check if it's working for you. Thanks!


Creating SpacyTagger and SpacyChunker classes for using the transformer models in these two tasks

I developed SpacyPOSTagger class and SpacyChunker class that we can use it for training transformer models with spacy api.Please check if there is anything wrong or redundancy

Please check for new commits if they transmitted or not


Day-3 Progressive Deployment Lab with NGINX Plus Ingress Controller

demonstrate a canary deployment using NGINX Plus Ingress Controller.  Show how we can go farther than we did on Day 1 due to our limitations of NIC OSS.

@codygreen do you think we need to highlight some of these extra capabilities in the lab guide editorial, or is this something we should handle in the presentations? If the latter, I will close this as I think we accomplished this with the day 3 scenarios.

I think A/B and canary are good for the lab.  The others can be discussed in the presentation.


nixos-icons: 2017-03-16 -> 2021-02-24

Adds a white variant of the nixos logo

Motivation for this change
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 114276 at 8a89af81 run on x86_64-linux 1

1 package built:

nixos-icons

2 suggestions:

warning: license-missing
Package is missing a license
Near pkgs/data/misc/nixos-artwork/icons.nix:21:3:
   |
21 |   makeFlags = [
   |   ^

warning: maintainers-missing
Package does not have a maintainer. Consider adding yourself?
Near pkgs/data/misc/nixos-artwork/icons.nix:21:3:
   |
21 |   makeFlags = [
   |   ^


⚠ ClamAV has degraded performance

In a9371a2, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 101 ms

Resolved: ClamAV performance has improved in cd4d6ea.


pb when testing aws-sdk via npm run test

Hi Irvin,
I'm having troubles when testing via "npm run test" (docker-lambda)
I'm trying to return the aws ecs cluster list of my aws account/region, which contains 1 cluster.
but when displaying the cluster list returned by:
const ECS = new AWS.ECS();
var listClusters = await ECS.listClusters().promise();
console.log("PF cluster list: " + util.inspect(listClusters, false, null));

I get an empty list:
START RequestId: e2253bfa-dd82-13ca-7600-1290dc20ae81 Version: $LATEST
2018-04-02T11:06:19.472Z	e2253bfa-dd82-13ca-7600-1290dc20ae81	Event type: aws-ecs-example
2018-04-02T11:06:20.368Z	e2253bfa-dd82-13ca-7600-1290dc20ae81	PF cluster list: { clusterArns: [] }

When I check directely via aws cli, my cluster is returned correctly
11:52 $ aws ecs list-clusters --profile vetup-prod-user-terraform --region eu-west-1
{
    "clusterArns": [
        "arn:aws:ecs:eu-west-1:xxxxxxxxx:cluster/vma-cluster"
    ]
}

Note: My .env file contains the same credential refering by the above "vetup-prod-user-terraform" aws profile:
.env:
AWS_ACCESS_KEY_ID=xxxxxxxx
AWS_SECRET_ACCESS_KEY=xxxxxxxxx
AWS_REGION=eu-west-1

When I execute the lambda directly via aws lambda dashboard, the cluster is also correctly returned (specified the correct lambda "ecs:ListClusters" policy before):
START RequestId: caea0b50-3666-11e8-8584-d19946e08f2c Version: $LATEST
2018-04-02T11:12:55.583Z	caea0b50-3666-11e8-8584-d19946e08f2c	Event type: aws-ecs-example
2018-04-02T11:12:55.825Z	caea0b50-3666-11e8-8584-d19946e08f2c	PF cluster list: { clusterArns: [ 'arn:aws:ecs:eu-west-1:xxxxxxxxxxx:cluster/vma-cluster' ] }

Any idea what I'm missing ?
Thanks you

I believe you need to set the region in the AWS SDK using AWS.config.update({ region: process.env.AWS_REGION }). Could you try that and let me know if it works?
The reason why it's not automatically populated is because the environment variable is only used to specify the region that the Lambda function should reside in, rather than to assume the region with the use of the AWS SDK.
I should probably rename the environment variable, since it seems like it's causing confusion.

Thank you, it was exactly that, worked with this:
AWS.config.update({ region: 'eu-west-1' });

Quite confusing to me indeed since process.env.AWS_REGION gave me "us-east-1", where is it defined ? dunno where it came from.
Yes, maybe the AWS_REGION var in .env should be called AWS_LAMBDA_REGION.
I guess I have to set the right aws-sdk region in the test function since it's not needed when the lambda is executed in aws.

There are actually a few region settings that are possible:

Deployment target region

Uses AWS_LAMBDA environment variable when using npm run deploy

Local testing environment region
CI testing environment region
Production environment region

I'm thinking that for (1), since it's only for local deployment (rather than from CI) it should be called AWS_LAMBDA_DEPLOYMENT_REGION, along with a prompt when running the script.
For (2) and (3), the regions should be populated with an environment variable (preferably called AWS_SDK_REGION), while for (4) it automatically defaults to the region that the Lambda function resides in. It can also default to the environment variable if defined.
Not everyone is using this boilerplate with the AWS SDK, since it was intended as a general purpose Node.js boilerplate. I'll keep this in mind, though it feels a little out of scope at this point.

I'm also thinking of revising the scope of this project sometime soon, ever since learning about the Serverless framework, which essentially solves most of the problems that this project does. Hence I might just remove that deployment script in favour of Serverless' multi-platform deployment capabilities.

Thank you for your help Irvin,
Looking forward to trying your next work on the Serverless framework, sounds really cool  !

No problem, glad to be of help!
I'll leave this issue open for me to remind myself next time.


[wip] Use a single JSON per branch for jobs

This simplifies the HUD by shifting most of the data fetching logic to a single lambda that does the heavy lifting via the metrics.pytorch.org DB.

✔ Deploy Preview for pytorch-ci-hud ready!
🔨 Explore the source changes: 4b942e7
🔍 Inspect the deploy log: https://app.netlify.com/sites/pytorch-ci-hud/deploys/613ba970f5ba80000afdd8f3
😎 Browse the preview: https://deploy-preview-114--pytorch-ci-hud.netlify.app

This simplifies the HUD by shifting most of the data fetching logic to a single lambda that does the heavy lifting via the metrics.pytorch.org DB.

This simplifies the HUD by shifting most of the data fetching logic to a single lambda that does the heavy lifting via the metrics.pytorch.org DB.


Not working, asking for modules

I am trying to make it work. Could you give a combination of versions of python, oauth2client and gcalcli that make it work correctly? I think I tried several without luck. In my desktop it complains with
ERROR: Missing module - cannot import name run
thanks!

See #211 .  How did you setup gcalcli?

Thanks. I had seen it, and I thought I had tried it. Just to make sure, I tried again with oauth2client=1.4.12 and it worked!