ENH: linalg: array library interoperability

Is your feature request related to a problem? Please describe.
Plans for addressing https://github.com/scipy/scipy/issues/18286 in scipy.linalg.
Currently, scipy.linalg only supports the use of NumPy arrays. Work towards interoperability has the potential to bring large performance improvements now and, with adoption of the array API standard, will bring compatibility with any future library which complies with the standard.
linalg is an array API standard extension, which means that supporting it will bring automatic dispatching for all standard functions. array-api-compat has implemented linalg wrappers for numpy, cupy and torch, so that automatic dispatching is available today.
For those functions which use compiled code and are not in the standard, non-CPU device support will be limited to start off with. From the RFC:

When a non-NumPy array type sees compiled code in SciPy (which tends to use the NumPy C API), we have a couple of options:

dispatch back to the other library (PyTorch, CuPy, etc.).
convert to a NumPy array when possible (i.e., on CPU via the buffer protocol, DLPack, or array), use the compiled code in question, then convert back.
don't support this at all

I'll note that (1) is the long-term goal; how to implement this is out of scope for this proposal - for more on that, see the Appendix section. For now we choose to do (2) when possible, and (3) otherwise. Switching from that approach to (1) in the future will be backwards-compatible.

This does not mean that GPU support is off-limits for all of these functions right now. Firstly, if we can write non-standard functions in terms of standard functions, then they will become compatible with all compliant libraries. Secondly, we have the option to call to CuPy and PyTorch for some functions. While this is not the general, library-agnostic solution of (1) - the "long-term goal" - it may be worth doing this to unlock GPU support for some important functions.
Describe the solution you'd like.
A. Conform to the standard extension.
This should be (almost) possible in one PR, similar to https://github.com/scipy/scipy/pull/19005 in fft. This involves validating array inputs with array_namespace and using xp.linalg, if it is available, for all standard functions. If it is not available, a conversion to NumPy is attempted. For NumPy arrays, the existing SciPy implementation is used.
For completeness and ease of use, I propose to add the standard functions which are not currently in scipy. Therefore, if a user wants to write array-agnostic code using the standard extension, they can just use SciPy, without going back to the array library for the functions. This will not take many lines of code as it is just dispatching to xp.linalg, or attempting to use functions from np.
I say almost possible since, to fully conform, the standard requires that functions on matrices support “batching” (i.e., the ability to perform the operation over a “stack” of matrices). There is also a discrepancy between the default mode of qr (full/complete vs economical/reduced) in the APIs. This, and potentially other similar issues, may be unsuitable to address here. These issues are really details of the API though, and lots of interoperability will still be achieved, despite not completely conforming immediately.
B. Re-write some functions to be array-agnostic.
Any function which does not use compiled-code can be included here. Furthermore, any functions which can be rewritten in terms of standard functions will be included. There may be overlap here with the work to convert functions from Fortran to other languages (https://github.com/scipy/scipy/issues/18566), in the case that anything is converted to pure Python. As a result, this point may become multiple PRs, with future work coming as more compiled code is eliminated.
This (first PR) will look similar to the cluster bits of https://github.com/scipy/scipy/pull/18668, and a follow-up PR to come for fft.
C. Call to CuPy and PyTorch.
We could also add calls to CuPy and PyTorch for any functions which they share with SciPy, and to which points A and B do not apply. This would ultimately be superseded by a dispatching mechanism, but there are no active plans for that right now.
Which functions in each PR?
For A, the standard functions which are in SciPy are:
cholesky, det, eigh, eighvalsh, inv, pinv, qr, solve, svd and svdvals.
The standard functions which would be added are:
cross, diagonal, matmul, matrix_norm (can use SciPy's norm), matrix_power, matrix_rank, matrix_transpose, outer, slogdet, tensordot, trace, vecdot, vector_norm (can use SciPy's norm).
These functions to be added all either have implementations in np or np.linalg, or are aliases of other standard functions, so they will immediately work with NumPy arrays.
For B, the Special Matrices functions look doable. There may be others, but I haven't had a thorough look through.
For C, some functions which have been suggested are solve_triangular, lu, lu_solve and lstsq.
Describe alternatives you've considered.
The number of PRs to split this work into is undecided. If there are multiple PRs, the order of them is also undecided.
If this all looks reasonable, I plan work on a PR for A. The main question for this PR would be whether or not to touch non-standard functions. The advantage of this would be to have consistent validation with array_namespace and behaviour with arrays coercible by np.asarray, across the whole submodule. The disadvantage of this would be additional review overhead, where the same functions would be revisited in PRs for B and C. I have no preference, so will do whatever the maintainers prefer. Regardless of when the work occurs, I think that it makes sense to have this consistent validation and behaviour eventually.
Overall, I am not familiar with scipy.linalg yet, so I may be missing a lot! Please feel free to give any corrections or suggestions.
References

RFC for array library interoperability in SciPy: https://github.com/scipy/scipy/issues/18286
Tracker for array API support in SciPy: https://github.com/scipy/scipy/issues/18867
PR adding array API machinery and covering cluster: https://github.com/scipy/scipy/pull/18668
PR for conforming fft to the standard: https://github.com/scipy/scipy/pull/19005
Array API Standard: https://data-apis.org/array-api/latest/index.html
Array API linalg Extension: https://data-apis.org/array-api/latest/extensions/linear_algebra_functions.html
array-api-compat: https://github.com/data-apis/array-api-compat

I am a bit on the fringes for this but having some familiarity with linalg library, in my opinion array API is focusing on the wrong functionality to start with. The main argument is the following;
In this day and age, the algorithms are kinda sorta established. There are not many ways to do LU factorization or cholesky implementation. The compiled code speed varies however the algorithms are quite static and very rarely something else comes up. The performance since the last decade lies in the structure discovery and dispatching to the correct flavor of the algorithms.
In fact this is the key to many algorithms that you quickly run over the array to discover structure and then dispatch to triangular, hermitian, hessenberg, bidiagonal etc. This is why we introduced scipy.linalg.bandwidth, scipy.linalg.ishermitian, and so on to have relatively quicker way to discover. Regardless of the architecture these are the essential ones that almost all linalg should start from. In that regard, I don't find the Array API standard linalg extensions doing the right standardization. They are only standardizing the API we currently have but not what we are trying to get to.
However on the other hand, these have to be as fast as possible to get rid of the additional overhead. In other words, scipy.linalg.solve has to be faster than numpy.linalg.solve (which sends every array to the same algorithm) if there is triangular structure. The time saving for using a triangular structure should not be wasted on checking which array vendor is provided and all other related checks. Otherwise there is no point in checking the structure and we will be forever slow with no possibility of speeding up due to this overhead.
We already suffer from this with check_finite=True keyword in almost all linalg funcs. The slowdown is pretty much 2x due to NumPy elementwise checks with heavy NumPy overhead and we are hoping to get rid of it by quick exit functionality and shorter path to compiled code which has a rather more aggressive run-over-array behavior specifically written for SciPy.
Unfortunately we also send things to LAPACK internally and that also adds f2py overhead eating away all the advantage we gained from array structure to nanny the array to Fortran contiguous memory layout. Hardly ever, a user is creating a Fortran array hence this is pretty much a two array copies; one going down to LAPACK and one for receiving the result back from f2py.
Therefore the utmost important things we need to see is these helper function implementations to support array API. Otherwise this is just going to be a case-switch for if this library, use this function because these are going to be used extensively inside linalg.
Especially, check_finite and lower keywords should disappear at some point which are purely LAPACK artifacts and need to be gotten rid of. We are already considering switching to homemade getrf implementations just like Julia and Rust folks already did because of the reduced performance of OpenBLAS/MKL for approximately n<100. There is at least 4x performance we are leaving at the table by not attending to this but we are pretty much full maintainer capacity right now and things are going slow.
To that end we already started some limited effort to remove Fortran code as much as we can in SciPy in #18566, like you mentioned. And once we are close to having those removed we hope to switch to more compiled code without obfuscating the code too much.

Therefore the utmost important things we need to see is these helper function implementations to support array API.

I think that is a good point. I see you moved that conversation to https://github.com/data-apis/array-api/issues/675, so let's keep this part of the conversation there.

To that end we already started some limited effort to remove Fortran code as much as we can in SciPy in https://github.com/scipy/scipy/issues/18566, like you mentioned.

Yes, we should be careful here to not interfere with the efforts of converting Fortran code to Python/Cython/etc. I think it's a matter of @lucascolley avoiding to touch those functions. And in case there's overlap, let's help out and hopefully push the Fortran conversion forward faster.

I am a bit on the fringes for this but having some familiarity with linalg library, in my opinion array API is focusing on the wrong functionality to start with.

I think you bring up a number of good points around performance of functions in scipy.linalg and potential avenues to improve that. But it looks to me like that's an orthogonal topic. The goal here is to be able to write/run code with another library like CuPy or PyTorch. scipy.linalg functionality is used heavily, also in other SciPy submodules. Hence it having support for the array API, CuPy, PyTorch & co is very useful - without it any other code that uses linalg functionality is also blocked from using non-numpy arrays. As Tyler points out, there may be significant performance gains in some cases too, but even without that, it's very useful to do the work that @lucascolley proposes here.

Isn't the performance difference orders of magnitude vs. CPU there? I thought that was a big reason that GPU dispatch for matmul empowers sensible machine learning times on device vs. host.

There is indeed. Even on CPU, some of the merged scikit-learn work showed that there's significant performance gains with PyTorch (simply comparing in envs that install wheels from PyPI), because it parallellizes more and builds again MKL, while we bundle OpenBLAS which is significantly slower on average.

I think you bring up a number of good points around performance of functions in scipy.linalg and potential avenues to improve that. But it looks to me like that's an orthogonal topic. The goal here is to be able to write/run code with another library like CuPy or PyTorch. scipy.linalg functionality is used heavily, also in other SciPy submodules. Hence it having support for the array API, CuPy, PyTorch & co is very useful - without it any other code that uses linalg functionality is also blocked from using non-numpy arrays. As Tyler points out, there may be significant performance gains in some cases too, but even without that, it's very useful to do the work that @lucascolley proposes here.

Indeed, but the part that I don't understand is the Array API definitions probably. Because there is solve, 'LU' and other details. Does this mean that if an array comes in I basically use xp.solve(arr, b) or array.solve(b). I am not even really sure I understand correctly why Array API has a linalg section.

I suppose torch et al. probably do something powerful on GPU for matrix multiply with large arrays that we don't get from host OpenBLAS?

That is correct. But it gets slightly more complicated for matrix decompositions and so on. SciPy doesn't have much of array manipulations or ufuncs that operate over the array. Hence matmul's or fast transpose add's etc. are probably within the array vendor functionality catalogue (or should be).
We typically send it to somewhere (compiled code land) and get the result. But linalg section is confusing me in that vendors are also providing linalg libs or what SciPy's role would be.

Does this mean that if an array comes in I basically use compatible_vendor.namespace.solve(arr, b) ala numpy.linalg.solve or I use compatible_array.solve(b). ala ndarray.solve()

The former; they are functions of the namespace.
For a slightly more concise example than solve, here is what inv could look like, where _inv is the current SciPy implementation:
xp = array_namespace(a)
if is_numpy(xp):
    return _inv(a, overwrite_a=overwrite_a, check_finite=check_finite)
if overwrite_a:
    raise ValueError(arg_err_msg("overwrite_a"))
if not check_finite:
    raise ValueError(arg_err_msg("check_finite"))
if hasattr(xp, 'linalg'):
    return xp.linalg.inv(a)
a = np.asarray(a)
return xp.asarray(_inv(a))

(Support for parameters which are not in the standard could be looked into, but this would raise if a non-standard parameter is provided with a non-NumPy array.)

I am feeling a bit lost again. This is what we discussed extensively in the uarray proposal too. SciPy already has a linalg library. Why would I mediate another library array going into another library linalg function? Whoever is calling can also use xp.linalg.inv(a) directly.
Let's forget about this linalg part; Array API, as I understand it, has Array standardization right. All vendors agree that arrays should have x, y, z properties and A, B, C methods. That's all cool. Then if the consumers like SciPy when they are implementing stuff, they want to support these vendors through Array API. So I keep an eye on the standard, does it have Transpose cool. Does it have something like norm ah nice cool again. Does it have isfinite? No? OK then I need to find a way to do that.
Etc.
And independent from where it is coming from I can push it to scipy.linalg.inv if the implementation allows me to do so. This makes sense to me and indeed we should as much as we can.
But calling other linalg libraries makes no sense to me. This is exactly where we left off the previous discussion and seems like we are getting back to it again.

One reason to have scipy.linalg functions use their xp.linalg counterparts when given not-ndarrays is that it allows the rest of scipy to be upgraded to be Array API-compatible while still using scipy.linalg-accelerated implementations in the ndarray case. If they are rewritten to use the xp.linalg API, then they will get the np.linalg implementations instead of scipy.linalg implementations.
With our official numpy binaries, that might not be so bad since we put good effort into using accelerated linear algebra there, but I suspect there are still places where scipy.linalg shines.

I fully agree but why is this taking place under SciPy code? Let me put it another way, is every consumer is going to jump through the hoops for torch or jax etc. basically all edge cases just to implement array api compatible code?
I think this is something I am not able to convey the message clearly. In other words, the consumer is orchestrating the code path. This doesn't seem like an array API but a separate linalg API.

I agree that it is likely that only scipy (and specifically scipy.linalg and scipy.fft) will do this kind of dance. numpy occupies a unique space in the Array API ecosystem and the Python scientific computing ecosystem more generally, and scipy has always had a unique relationship to numpy. Specifically, scipy.linalg shadows numpy.linalg, and many functions inside of scipy itself use the scipy.linalg versions (and should continue to do so whenever given a true ndarray). But if we want as much of scipy as possible to be xp-agnostic (setting aside scipy.linalg considered by itself), then we either have to implement all of those functions using xp.linalg and give up on the scipy.linalg performance in the most common case (ndarray inputs) OR do a dance in each of those functions to pick between xp.linalg and scipy.linalg OR contain the xp.linalg-or-native dance inside of the scipy.linalg functions. The last choice is probably the best bang for buck.

So if we enumerate them

implement all of those functions using xp.linalg and give up on the scipy.linalg performance in the most common case
do a dance in each of those functions to pick between xp.linalg and  scipy.linalg
contain the xp.linalg-or-native dance inside of the scipy.linalg functions

The only remaining option is that

we don't support the linalg extension of Array API and only use the main body of it.

Because I really don't see the benefit of it for SciPy (yet!). And frankly, I don't understand why SciPy should be the arbiter of this to start with. I am full in for the Array API if I understand it correctly.
# xp = array_namespace(a)
arr = xp.array([1, 2, 3], [4, 5, 6])  # Whatever xp is

# following just works because the array has all the necessary
# methods used in the inv implementation
# And SciPy tries its best to use only those limited set of
# functionalities from Array API
arr_inv = scipy.linalg.inv(arr)

This is clearly a superior standardization that generalizes the ndarray of NumPy. But the following is working backwards pushing implementation details to consumers;
xp = array_namespace(a)
arr = xp.array([1, 2, 3], [4, 5, 6])  # Whatever xp is

# some code that checks if the kwargs are compatible
....

#then the dance we mentioned above
if numpy:
    arr_inv = scipy.linalg.inv(arr)
elif it is coming from some xp with linalg:
    call their linalg
elif gpu stuff:
    perform some host device dance
....
else:
    TypeError("don't know what to do with this input")

Why does SciPy has to host what other libraries do while other libraries don't need to care anything SciPy does? In other words, why should SciPy has to adopt this? It gains nothing other than a tiny bit of overhead and quite some if-else switches, because its most use case is NumPy (overall speaking). Moreover, SciPy binds itself to this switching code and its maintenance with 100% dependency to other libraries' implementation velocity. Have a look at Tyler's welch implementation in #19001. I consider it pretty convoluted as per my remarks. And also it seems like we didn't settle on the basic functionality either (as one piece of anectodal evidence; https://github.com/data-apis/array-api/issues/627)
Probably I am not able to peek from your perspective regarding the unique positions because this standard's goal is to remove that uniqueness (if I understand correctly) hence I would appreciate a bit more motivation to do this within SciPy. My current understanding is that linalg extension of this API is over-reaching to implementers and consumers which should not be the case. Gaining great performance just by shifting to GPU is great but if we are going to orchestrate the code such that it goes to another library then it is much easier to use that library already with its linalg extensions.
It is very much the same behavior if we don't change anything and let the xp.linalg namespace not to be used at all. In my first code example that would be wonderful to have; SciPy just consumes xp.array and its methods and tries really hard not to get out of those core functionalities and in return the array knows its own implementation properties.
Actually I am looking at ways to do array api conforming linalg already. That's maximal bang with the same buck such that we support all array inputs, and we don't bother if some library implements linalg or not.

My understanding is that the proposal being made in this issue (the one I labeled as the best bang for buck) is exactly the one that enables the first code example that you mark out as superior. The dance is hidden inside of scipy.linalg.inv() and not exposed to consumers. The proposal here is to avoid having consumers do that dance.

In case I didn't emphasize it, let me clarify; I consider SciPy to be a consumer and thus we, too, should not do this dance. I guess that's crucial point. In the terminology of other Array API issues, consumers are the array consumers not the end users. Or at least that's how I understood it. So there should be no dance in anywhere.
My question is why SciPy is the arbiter of this dance at all if it is a consumer.

Ask differently, who can implement linalg extension of Array API without this dance?

scipy.linalg is not an implementation of xp.linalg. Likely no implementation of xp.linalg will do The Dance. For other xp array objects, there is basically only one preferred linear algebra implementation, and that will be implemented as xp.linalg. xp.linalg.* functions won't be given non-xp array objects (because one gets the xp namespace object from the array object itself).
Because of numpy and scipy's long history together, we have 2 linalg libraries for ndarray objects, one shadowing the other. We (at least the rest of scipy) use the scipy.linalg version a lot because it's generally built with the faster versions of the LAPACK calls or a faster LAPACK in general. Our use of scipy.linalg functions is a blocker to making much of the rest of scipy xp-agnostic.
To highlight a point of agreement, at least between us two, I agree that we should not be doing backend-specific checks likes the ones you noted in #19001, so I'd strike the elif gpu stuff: clause in your example. For me, that's my line as to whether or not it's worth making one of our functions xp-agnostic. If it's not part of the standard or one implementation of the standard is broken, then we don't support it. That would indeed tightly couple us to arcane details of the implementation state of a whole host of other libraries, and that's too much of a burden, IMO. The Array API standard should be doing the cross-backend work.
The proposal does not involve backend-specific checks other than checking for true ndarrays. It constrains The Dance (sans-elif gpu stuff:) to just those functions in scipy.linalg that have standardized xp.linalg equivalents. No other part of scipy need do The Dance; they can just call scipy.linalg.inv() freely and rest assured that it will still be xp-agnostic AND get the best performance for ndarray inputs too. The Dance is not an all-pervasive pattern for proper use or implementation of xp.linalg. It's a response to a unique opportunity afforded by the existence of the faster scipy.linalg implementations for ndarray inputs.
Now, the ideal scenario would be that the np.linalg functions, which are the basis of the xp.linalg implementation for ndarray implementation of the Array API, would just be fast enough that xp-agnostic code could just use xp.linalg explicitly and unconditionally. Maybe they are good enough; we should check! Or maybe we should invest some effort into making them so. But for me, implementing The Dance inside of a few scipy.linalg functions is a cheap and effective way to unblock getting much of the rest of scipy xp-agnostic without sacrificing performance.

Much clearer thank you for the clarification. I think I completely agree though certain details, I just don't have enough information about. But they are not much relevant for this discussion.
So I am kinda sorta, thinking about the matfuncs when array API's are discussed. Here linalg uses itself quite a lot; say it goes roughly the following; I am just making things up so following is not necessarily the real function body, but many are like this
def sqrtm(A):

    # Using the xp.linalg and np.linalg by their namespace to cut it short
    if np:
        q, z = scipy.linalg.qz(A)
    elif xp:
        # possibly different API
        q, z, _ = xp.linalg.qz(A, some_kwarg=True)
    else:
         ValueError('Don't know how to handle A')

    # then do bunch of stuff with q and z
    # maybe call some cythonized code
    q_2 = some_cythonized_func(q)
    ....

    # Now say we need to call eigh
    if np:
        L, V = scipy.linalg.eigh(A, subset_by_index=[2, 5])
    elif xp:
        # possibly different API
        L, V, _ = xp.linalg.eigh(A, some_kwarg=True)
        # sorting is left open in the spec
        # https://data-apis.org/array-api/latest/extensions/generated/array_api.linalg.eigh.html
        L = xp.sort(L, descending=True)
    else:
         ValueError('Don't know how to handle A')

     # more code possibly with more scipy.linalg/xp.linalg cases
    ....
    return M

Alternatively, we can collect things under big sub functions
def sqrtm(A):

    if np:
        # scipy original implementation
        q, z = scipy.linalg._sqrtm(A)

    elif xp:

        q, z, _ = xp.linalg.qz(A, some_kwarg=True)

        # the array api compatible implementation
        ....
        L, V, _ = xp.linalg.eigh(A, some_kwarg=True)
        L = xp.sort(L, descending=True)
        ....

    else:
         ValueError('Don't know how to handle A')

    return M

This is what I can't wrap my head around yet. Seems to me that SciPy is selected as the common consumer to every vendor (well like you said, let's say we convinced folks about the GPU stuff but, Matt, at least, was impressed with the GPU results so that seems to be the killer feature). And we need to take care of array API quite comprehensively.
Maybe we can move array API namespace inside linalg with matching signatures? Say scipy.linalg.array_api.<func> where func signature is adjusted to array API linalg extension standard. But that seems to open other cans of worms. So I'm not sure how to proceed.

I do think that a separate namespace of xp.linalg-compliant shim functions that do The Dance might be a reasonable option. I was going to suggest it as well. In this scenario, we leave all of the scipy.linalg functions alone; they only do ndarray calculations on the CPU just as normal. But the shim functions will check for true ndarrays and call the scipy.linalg version in that case and otherwise defer to xp.linalg. That namespace might also be a place for xp-agnostic implementations of functions that are in scipy.linalg but not in the xp.linalg standard, much like the xp-agnostic implementation of cov() that's needed to unblock higher-level functions.
Among other reasons, keeping the xp.linalg-compliant shims elsewhere would avoid some of the confusion over the signature differences between the xp.linalg standard and the implementations in scipy.linalg. If we tried to make the scipy.linalg versions xp-agnostic, it would only be in the cases where the user didn't pass arguments that made use of scipy.linalg-specific features. That will be complicated to document and add a good amount of mental complexity to someone trying to write or maintain xp-agnostic code.
I think my guiding principle for thinking about scipy.linalg and scipy.fft with respect to the array standard is that we should not care very much about the core computational routines being xp-agnostic themselves (simple helper routines like fftshift() are another story). Their main purpose is to expose the performance and options of the compiled C/C++/Fortran code, and that can't be xp-agnostic. The main thing to think about is how to unblock the xp-agnosticism of other parts of scipy that happen to use scipy.linalg and scipy.fft. So I think both options are valid ways to do that, each with some drawbacks and benefits.
I will also note that my desire to remove things like if 'cupy' in xp.__name__ clauses is not a hard line if there's appetite for handling such cases by other devs (as #19001 proves). I acknowledge that for at least some amount of time, such switches may be necessary to actually achieve practical amounts of non-ndarray support in the near term. And if we just let their absence block us, it might be the case that we never surface the gaps in the Array API or individual implementations that need to be found for them to be fixed. But I definitely don't want them sprinkled all around scipy.signal and the rest. If they were constrained to shim/utility functions in scipy._lib.array_api or other such namespaces, I'd be more likely to tolerate them. I'd still probably try to minimize them and be more willing to accept that if the xp implementation doesn't support some things well enough, then they'll just hit a "convert-to-ndarray" codepath or fail (I'm thinking of the as_strided() fallback codepath in #19001, here).

I do think that a separate namespace of xp.linalg-compliant shim functions that do The Dance might be a reasonable option. I was going to suggest it as well. In this scenario, we leave all of the scipy.linalg functions alone; they only do ndarray calculations on the CPU just as normal.

The former may be worth considering I think, especially if it helps make a step in the direction of the long-standing issue of diverging behavior of scipy.linalg and numpy.linalg (e.g., for qr). However, I'm not sure about that part yet - it may just be extra API surface without fixing the existing divergence. Maybe it can be done but kept private, only for use by other SciPy submodules that may need it? I'd like to comment on the last sentence above though - also related to this comment:

I think my guiding principle for thinking about scipy.linalg and scipy.fft with respect to the array standard is that we should not care very much about the core computational routines being xp-agnostic themselves (simple helper routines like fftshift() are another story). Their main purpose is to expose the performance and options of the compiled C/C++/Fortran code, and that can't be xp-agnostic.

I don't really agree here. The key element of the proposed design (see gh-18286) is this sentence: The basic design principle I propose aiming for in all of SciPy for array libraries is: container type in == container type out. The reasoning here seems to not take that into account. linalg and fft are far from the only components in SciPy that contain a lot of compiled code, and I don't see why they should be the two special-case modules in the big picture (that there's array API extensions that overlap with scipy.fft and scipy.linalg functionality is a possible extra opportunity, that's all).
The design has several goals/consequences:

Correctness (e.g. no silently wrong answers for masked arrays, unit array or other such inputs, or pandas dataframe/series input that may behave unpredictably)
Composability (even without performance differences, it is a big gain to be able to use SciPy APIs in an array type-agnostic way from downstream libraries and end user code).
Performance (GPU execution, etc.)

The tentative argument here against changing scipy.linalg functions seems to only consider (3), Performance. I think some of this, in particular around (1), Correctness, is pretty well covered in  "Problems & Opportunities", so I'd much prefer to avoid discussing a deviation from that design for the linalg module only. Unless there's a really compelling reason of course - but I don't see it yet.
For (2), Composability, let me also point out a scikit-learn example: https://github.com/scikit-learn/scikit-learn/blob/80c39a4966ee88150d7e5849ab21a8e17bb1fbc8/sklearn/decomposition/_pca.py#L545-L555. They'd like scipy.linalg.svd for numpy arrays, and the xp version otherwise. That if-else dance works best if it lives inside scipy.linalg.svd rather than in scikit-learn and other users of scipy.linalg.svd.

I will also note that my desire to remove things like if 'cupy' in xp.__name__ clauses [...]

Agreed with the spirit of this part - we should minimize if-statements like this as much as we can. There'll be some cases where it's a tradeoff though, and a few lines more code for 10x or more performance difference (as in the as_strided case) should be okay. But we can't have that all over the place indeed.

Fair. I came into the discussion late, and the point of discussion at the time was whether or not the xp.linalg versions got used inside of the scipy.linalg equivalents, which was essentially a question about (3). On the assumption that we want "container type in == container type out" for all of our functions, then yeah, it would be a good idea for scipy.linalg.* to call the relevant xp.linalg.* when it's applicable and available.
That said, for the rest of scipy that uses scipy.linalg, it is going to be a little hard to know when calling scipy.linalg.* not only preserves the container type but also keeps the computation xp-native and avoids the conversion based on the options passed in. I think it might still behoove us to have xp.linalg API-compatible versions that use scipy.linalg implementations when given ndarrays and xp.linalg implementations otherwise. Then other parts of scipy that are really concerned about keeping everything xp-native can use that and be sure that we're not going through a "convert to ndarray" codepath. I'm happy with it being scipy-internal, though it might be useful for other projects that already have scipy as a dependency, so maybe data-apis/array-api-compat would be a good place for it, too.

Over the past few weeks a lot of improvements in test infra were implemented, and @lucascolley's PR (gh-19005) for support in fft was merged, as was @mdhaber's PR (gh-19023) which made a start on special with 14 of the most popular functions. So I think it's time to carefully move ahead on linalg.
linalg is a large module, so it'd be best to not have one giant PR for everything. Tackling first the functions that are also in the standard would be useful I think - @lucascolley gave a useful categorization (A-C) in the issue description. And the private "guaranteed to not convert to numpy.ndarray" as discussed in the comments above is a fourth category.
@lucascolley I know there's a few fft follow-ups to do still, but I think it'd be useful to see a PR with what you have for linalg so far, after updating it for any "best practices" from the fft/cluster/special work if needed.

I'm happy with it being scipy-internal, though it might be useful for other projects that already have scipy as a dependency, so maybe data-apis/array-api-compat would be a good place for it, too.

I do believe that there may be a case for a standalone library for this kind of thing. array-api-compat is scoped pretty tightly as a compat layer for APIs that are either in the latest standard but not yet implemented in libraries, or for things that have been discussed/accepted for inclusion in the standard and are pending inclusion. I think both SciPy and scikit-learn are slowly going to accumulate private versions of array-agnostic functions that are built on top of the standard rather than included in it (cov is a good example here, see gh-19051) and those could fairly easily be made reusable. I think it's a bit early for a separate package right now, but it may materialize over the next 6-12 months.

Thanks Robert, Ilhan, Ralf and Tyler for the discussion here! It looks like we've already answered some important questions, not least from my end.
In tandem with opening my WIP for linalg, I would like to briefly justify the scope of my PR here. Unfortunately, I will not have time to convert all of linalg this summer (my internship lasts until the end of September), so I would also like to set out a rough plan of how this PR will fit into the bigger picture of full module conversion.

my guess would have been that we'd try to improve the performance of our existing linalg surface area by supporting the array API under the hood, but expanding our API surface area to match the actual array backends is a bit less clear as the right direction to me

To clarify on this point, I am no longer considering any API additions.

I am currently writing a blog post which will describe the different changes required to convert a submodule. Testing aside, the changes are:

For pure Python + NumPy functions, rewrite everything to be array-agnostic. This was the case with the Fast Hankel Transforms in fft, and much of cluster.

When compiled code is hit, use np.asarray before it and xp.asarray after it. This will raise exceptions for GPU arrays to avoid silent device transfers. This was the case for the Discrete Sine and Cosine Transforms in fft.

For submodules which have a Standard Extension Module, check whether it exists in the namespace before falling back to (2). if hasattr(xp, 'linalg'):. This was the case for the basic functions in fft.

My PR will tackle the functions covered by (3). Future PRs will tackle (1) and (2).
After or alongside this work, separate PRs could implement calling out to specific libraries like CuPy and PyTorch where (2) would otherwise be hit. This is what Matt has done for some functions in special in gh-19023. This is a temporary solution in lack of a general dispatch structure, but arguably just as important as the array-agnostic work in the short term.

the private "guaranteed to not convert to numpy.ndarray" as discussed in the comments above

This is another type of work which can happen alongside what is above. Since this is just SciPy internal, I don't think that we need to figure this out fully before starting work on the other parts.

I hope that this helps to clarify things! I would recommend saving any specific implementation questions for my PR, but feel free to ask anything about the vision for conversion which I have laid out here.

I am currently writing a blog post which will describe the different changes required to convert a submodule.

This can be found at https://labs.quansight.org/blog/scipy-array-api.


Dear Sir / Madam shows up as Spanish with highest confidence.

The text "Dear Sir / Madam," is predicted to be in Spanish language, where it is clearly English content.

Yes, that can happen depending on

which languages you build the detector with
what the ngram probabilities of each language are for the respective input

So, this is not a bug. A statistical classifier can always fail for certain input.
Closed.


download

azure portal

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 9e204130-6582-4f07-5474-cbdce61ab694
Version Independent ID: b8819ed9-7c57-f85f-5f13-2c712bb9e755
Content: Azure portal documentation - Tutorials, API Reference
Content Source: articles/azure-portal/index.md
Service: azure-portal
GitHub Login: @czeumault
Microsoft Alias: carolz

@Susheel27 Thanks for reporting! I have assigned the issue to the author to check and add content to the document.

@Susheel27 What issue are you reporting?

#please-close

@Susheel27 We will now close this issue. If there are further questions regarding this matter, please reopen it and we will gladly continue the discussion.


Bug in the answer to exercise 14.27

Hello,
In the definition of StrBlobPtr& StrBlobPtr::operator--(), check is called with -1 as its first argument. Because the first parameter of check is a std::size_t and std::size_t is an unsigned type, so -1 would be treated as the biggest possible std::size_t, which will cause the check to always throw (because -1 >= ret->size() will always succeed). Indeed, the definition you provided for StrBlobPtr::operator--() is identical to the definition provided by the book and I believe it's a bug in the book. I think you should replace -1 with curr.

i think pass curr instead of -1 is right


Renew Issue

Im trying to renew a Let´s Encrypt Certificate on my Plesk Server
Im getting no real error message, only:
"Error: Let's Encrypt SSL certificate installation failed: Install certificate failure: Unable to set certificate name : "
After a few tries I am getting:
"Error: Let's Encrypt SSL certificate installation failed: Invalid response from https://acme-v01.api.letsencrypt.org/acme/new-cert1: Error creating new cert :: too many certificates already issued for exact set of domains: gitlab.micronited.de,www.gitlab.micronited.de. Type: urn:acme:error:rateLimited."
Because Let´s Encrypt blocks a domain after 10 certificates a week.
I also noticed that Plesk tries to renew the certificate of my other domain every day
https://crt.sh/?q=micronited.de

Hi Lucaber,
pls. see the provided suggestions from the WIKI - article:
https://github.com/plesk/letsencrypt-plesk/wiki/Error-%22Unable-to-set-certificate-name-:%22


set a maximum number of signs per line / collapse marks

Expected behavior
Let's say I have 3 warnings on the same line, neomake will add 3 marks. The behavior makes sense but with signcolumn=auto:3, neomake may end up using every column, leaving none for other plugins. I would prefer to have a maximum of one mark for neomake.
Consider this very low priority, just nice to have :)

Makes sense.
Also a good idea for performance in general, but should be coupled to a setting maybe.
I could also imagine to append a number of the total number of signs to the most dominating sign (only makes sense if it has a width of 1, but would be truncated otherwise I assume anyway).

See also https://github.com/neovim/neovim/pull/9788.

Started working on this.
But isn't it the case currently that there is only a single sign per line?!  (only looked at tests though)

Let's say I have 3 warnings on the same line, neomake will add 3 marks.

Really?
I've just checked this, and with e.g. "flake8" and "mypy", only one sign is displayed.
I'm experimenting with having a new setting for this, and then it would display more signs.
(I am using :set signcolumn=auto:3 here)
Are the warnings from the same maker?
Or from different modes (file / project)?

There was a repetition of the same sign symbol while I had tried to assign different symbols to each plugin. It is possible I made a mistake though so don't spend time on this xD
I've mostly switched to coc.nvim + lsp python client so it's not easy to reproduce but the pyls-mypy plugin crashes the server so I might use Neomake again for python as well.


Add E-mote

SteamDB app page links to a few games using this

NEKOPARA Vol. 0
NEKOPARA Vol. 1
NEKOPARA Vol. 2
NEKOPARA Vol. 3
NEKOPARA Vol. 4
NEKOPARA Extra
KukkoroDays
TroubleDays
IdolDays
NinNinDays
NinNinDays2
NekoMiko
Fantasy Tavern Sextet -Vol.1 New World Days-
Fantasy Tavern Sextet -Vol.2 Adventurer's Days-
Fantasy Tavern Sextet -Vol.3 Postlude Days-
Song of Memories
Corona Blossom Vol.1 Gift From the Galaxy
Maitetsu
Maitetsu:Pure Station
Maitetsu:Last Run!!
Witch's Garden

Brief explanation of the change
Added "E-mote", a Japanese software developed by M2 Co., Ltd, which is primarily used in visual novels to create highly animated and expressive character sprites. Notable visual novels like Nekopara have used it for their animations.

After running changed rules in this pull request (this bot only tests rules.ini changes):
Changed technologies: SDK.E-mote
These apps now match: (47)<PHONE_NUMBER>60<PHONE_NUMBER>10<PHONE_NUMBER>00<PHONE_NUMBER>40<PHONE_NUMBER>10<PHONE_NUMBER>20<PHONE_NUMBER>00<PHONE_NUMBER>60<PHONE_NUMBER>60<PHONE_NUMBER>50<PHONE_NUMBER>70<PHONE_NUMBER>70<PHONE_NUMBER>980 1171270 1258340 1372830 1406990 1434480 1461770 1523390 1576130 1580240 1642260 1698030 1770820 1770830 1770840 1934180 2060000 2162950 2243920 2347380 2467540 2825300
Please confirm that all these changed apps are correct.

After running changed rules in this pull request (this bot only tests rules.ini changes):
Changed technologies: SDK.E-mote
These apps now match: (47)<PHONE_NUMBER>60<PHONE_NUMBER>10<PHONE_NUMBER>00<PHONE_NUMBER>40<PHONE_NUMBER>10<PHONE_NUMBER>20<PHONE_NUMBER>00<PHONE_NUMBER>60<PHONE_NUMBER>60<PHONE_NUMBER>50<PHONE_NUMBER>70<PHONE_NUMBER>70<PHONE_NUMBER>980 1171270 1258340 1372830 1406990 1434480 1461770 1523390 1576130 1580240 1642260 1698030 1770820 1770830 1770840 1934180 2060000 2162950 2243920 2347380 2467540 2825300
Please confirm that all these changed apps are correct.

Correct. I can confirm that, excluding demos, all of the titles are tagged as 'Visual Novel' on Steam. The sole exception is Rune Factory 4 Special, which happens to be an RPG. "E-mote" can be seen at 1:13 in the second trailer (you may also refer to the .webm below):
e-mote_in_rune_factory_4.webm


Ansible Hook Unit Tests fail 100% of the time

The tests are failing randomly due to missing python packages - namely 'six' which is installed in the virtualenv.
The charm further compounds these failuires by having moved to a pip based charmhelper installation - this breaks the unit tests inherently, and requires refactoring of the unit_test packages. These tests seem more like the responsibility of the framework in charmhelpers vs embedded in the charm itself.
The unit test target should probably be refactored to run linting on the ansible routines, as well as any intermediate checks on the python <-> ansible call stack.

Closed by #15


drop machinery for eventing-rabbitmq

Part of https://github.com/knative/eventing/issues/7670

/assign @izabelacg

/lgtm


Not able to build with Vite

I am getting a heap out of memory error with Vite
I tried to increase heap size and also tried to use fewer workers to build.
Is there any solution?

@prasadkirpekar Hi, it only supports webpack yet. But I will try to update it to support vite.

Hello Do you have any plans for vite support?


runtimebp: Extend GOMAXPROCS logic to account for additional scenarios

On discussion with kevlar, we came to the conclusion that there should be
4 possible settings for GOMAXPROCS which should cover all needs and
ensure adequate automation.  These are:

GOMAXPROCS if provided,
limit if known (with configurable multiplier),
fallback to a known environment variable Infrared can provide,
exactly 2 (for minimum parallelism)

By providing this hierarchy, we give maximum flexibility for devs,
while also providing safe and sane guardrails to work with, and
never end up in a scenario where bp.go tries to use physical core count.

Going to be doing some additional tinkering here, but I wanted it in front of eyes.  I think I've got a sane way to do it off the request now that's going to require a bit more parsing.  Good feedback there, Yuxuan, I'll address.

We probably also want to hardcode the min arg passed in from InitFromConfig to be 2 to enforce the minimal of 2.

Did some significant changes here to make things more correct and follow the feedback.  Also added a bit of testing.

Sorry that things are a bit messy. I think we should take a step back to look at how things should behave from API perspective.
We have 4 APIs touched here, InitFromConfig, NumCPU, GOMAXPROCS, and GOMAXPROCSwithFormula:

InitFromConfig: this is the one used in baseplate.New, so we just want it to behave exactly hat we planned (first check GOMAXPROCS env var, then limit, then request*multiplier, then 2)
NumCPU: fallback to 2 certainly doesn't make sense here. fallback to request probably makes sense, but fallback to request*multiplier likely doesn't make sense.
GOMAXPROCS: this should be same as InitFromConfig except the GOMAXPROCS env var part, and the fallback to 2 part (that is handled by the min passed into GOMAXPROCS)
GOMAXPROCSwithFormula: this can get tricky, because we actually want different formulas in different cases in GOMAXPROCS now. we could just mark it as deprecated and make it behave exactly the same as GOMAXPROCS (ignore the formula passed in)

Superseded by #547.


I new code to sign to OneDrive

Provided code for expired. The expiration was announced at the last step and there is no error anywhere else.
Please provide a new code or what to do and me go update re-generated.
Thank you

+1
Same thing happens to me when I try to authenticate with the provided code.

Restarting the addon should generate a new code

That actually worked this time, but also by opening the Microsoft authentication link in a private Chrome window.

Cheers,
/Jonas


Translate doesn't work.

First of all, thank you for this plugin.
I used bower to download this plugin (https://libraries.io/bower/summernote-image-attributes) but translate doesn't work (I think bower doesn't download the last version of plugin)
Thanks for your help !

oh ok, I didn't add Bower abilitites to the plugin, so I'm not sure what's going on there.
I'll have a look into, and hopefully learn a little more about Bower.
Sorry, it didn't work for you. Have you tried downloading the plugin and using it from there, just to make sure it works for you with the translation you need?

Yes it works fine when I downloaded it from github. Thanks!

Cool.
I have no idea what's with Bower. I'm old skool, and prefer to download it and go from there. Must be getting old, maybe I need to look into it, and keep up with the cool kids, lol.


why software_provider "custom" is missing

as in title, "custom" provider exists in early version of puppet-logstash, but missing in current version. Can someone help to let me know why?

Hi. What did the custom provider do? Did it attempt to install from a .tar.gz file?

Yes. As in early version, I can specify to install logstash from a private url. https://github.com/elastic/puppet-logstash/blob/0.3.4/manifests/service.pp#L103

Ah OK. It's possible to install a packaged version from an explicit URL, but .tar and .jar files are not supported.
Is a packaged version OK? Logstash provides packages for most supported platforms.
https://www.elastic.co/support/matrix

Thanks a lot. But tar.gz is not supported? Could not retrieve catalog from remote server: Error 400 on SERVER: Unknown file extention "gz".

Very welcome!
Good luck, and Happy Logstashing.


TV giving No Response in HomeApp

When I add my Sony TV to HomeApp using you plugin for Homebridge I can only control it once. When it initial is added to HomeApp. After turning it off and on again the HomeApp is giving the TV status "No Response" any solution ideas for this?
BR, Hanno

BTW here you have the log that's given by Homebridge.
[3/28/2019, 10:17:08 AM] [BraviaOSPlatform] [INFO] Connecting to <IP_ADDRESS>
[3/28/2019, 10:17:08 AM] [BraviaOSPlatform] [INFO] Configuring accessory: Woonkamer Televisie
[3/28/2019, 10:17:08 AM] [BraviaOSPlatform] [INFO] Adding new Input Source: HDMI 1
[3/28/2019, 10:17:08 AM] [BraviaOSPlatform] [INFO] Adding new Input Source: HDMI 2
[3/28/2019, 10:17:08 AM] [BraviaOSPlatform] [INFO] Adding new Input Source: HDMI 3/ARC
[3/28/2019, 10:17:08 AM] [BraviaOSPlatform] [INFO] Adding new Input Source: HDMI 4
[3/28/2019, 10:17:08 AM] [BraviaOSPlatform] [INFO] Adding new Input Source: SmartTV
[3/28/2019, 10:17:08 AM] [BraviaOSPlatform] [INFO] Adding new Input Source: YouTube
[3/28/2019, 10:17:08 AM] [BraviaOSPlatform] [INFO] Woonkamer Televisie: Inputlist finished
[3/28/2019, 10:17:08 AM] [BraviaOSPlatform] [INFO] Woonkamer Televisie: Welcome to Sony Bravia TV OS 2.0.0 - Below your Identifier list to create rules in 3rd party apps
[3/28/2019, 10:17:08 AM] [BraviaOSPlatform] [INFO] Woonkamer Televisie: (0) HDMI 1
[3/28/2019, 10:17:08 AM] [BraviaOSPlatform] [INFO] Woonkamer Televisie: (1) HDMI 2
[3/28/2019, 10:17:08 AM] [BraviaOSPlatform] [INFO] Woonkamer Televisie: (2) HDMI 3/ARC
[3/28/2019, 10:17:08 AM] [BraviaOSPlatform] [INFO] Woonkamer Televisie: (3) HDMI 4
[3/28/2019, 10:17:08 AM] [BraviaOSPlatform] [INFO] Woonkamer Televisie: (4) SmartTV
[3/28/2019, 10:17:08 AM] [BraviaOSPlatform] [INFO] Woonkamer Televisie: (5) YouTube
HAP Warning: Characteristic 000000EA-0000-1000-8000-0026BB765291 not in required or optional characteristics for service 00000113-0000-1000-8000-0026BB765291. Adding anyway.
[3/28/2019, 10:17:08 AM] Homebridge is running on port 51826.

For me whole homebdrige is giving no response after installing the plugin. :/
Uninstall does not help.

Reset after uninstalling the plugin does work. Lots of work though

I did complete reinstall. 3h. 👎

Hello, im currently working on v3 with a rewritten code , i will check this in v3

Great looking forward to it!

Hello, im currently working on v3, the beta is now available!
How to download and configure:
https://github.com/SeydX/homebridge-bravia-tvos/issues/32

See https://github.com/SeydX/homebridge-bravia-tvos/issues/32


Content improvements and fixes

Why:
Some suggested improvements and fix for web app build job failing.
Closes #39
What's being changed:

Intro h3 heading to step 2 is confusing as no jobs have been triggered by label yet. Just remove heading.
Some of the commit instructions don't match with where the user is in the GitHub UI. Update instructions to match.
PATs can be either new fine-grained or classic. Point the learner towards using the classic token.
Workflow file content for deploy-staging.yml and spinup-destroy.yml is hidden in a <details> block, making it harder for the learner to find. The instructions for deploy-staging.yml in step 2 make it seem that copying the code is optional. (It could be added as an exercise for the learner to create the workflow from just a description of what the workflow should accomplish, but this would make the step significately more difficult and error prone.)
Added a breif description of the deploy-staging.yml workflow
Numbered first instruction for Step 5: Activity 1 and indented code block to align with page content.
Fix for web app not building and #39 . Actions runners are currently using Node v18.14.1 by default. Pinning the deploy workflows to using Node v16 seems to fix the problem. (npm audit fix --force also fixes the problem. Really, the node package versions need to be updated. I still think it's a good idea to pin the node version on the runner though).

Check off the following:

[ ] For workflow changes, I have verified the Actions workflows function as expected.
[ ] For content changes, I have reviewed the style guide.

Thank you for this pull request. Unfortunately we had to make a rather large structural change to the courses to meet updated accessibility requirements. We welcome you to create separate pull requests with these changes :)


NIFI-12772 Expose REMOTE_POLL_BATCH_SIZE property for ListSFTP

Summary
NIFI-12772
Allows use of the REMOTE_POLL_BATCH_SIZE property in the ListSFTP processor. Previously this was supported but not exposed for configuration.
Tracking
Please complete the following tracking steps prior to pull request creation.
Issue Tracking

[x] Apache NiFi Jira issue created

Pull Request Tracking

[x] Pull Request title starts with Apache NiFi Jira issue number, such as NIFI-00000
[x] Pull Request commit message starts with Apache NiFi Jira issue number, as such NIFI-00000

Pull Request Formatting

[x] Pull Request based on current revision of the main branch
[x] Pull Request refers to a feature branch with one commit containing changes

Verification
Please indicate the verification steps performed prior to pull request creation.
Build

[ ] Build completed using mvn clean install -P contrib-check

[x] JDK 21

Licensing

[ ] New dependencies are compatible with the Apache License 2.0 according to the License Policy
[ ] New dependencies are documented in applicable LICENSE and NOTICE files

Documentation

[ ] Documentation formatting appears as expected in rendered files

Need to do some manual testing

Thank you for working on this @tombrisland.
I'm really looking forward to this improvement and hope we can backport it to 1.x as well.
I haven't had time to do any testing, but just glanced over the code changes and they look straightforward.
It seems that even though the SFTPTransfer supports the property already, the underlying sshj does not support a limit to ls directly but instead we rather stop adding results to our list. The listing is continued nonetheless.
I think supporting the property is still an improvement, as we at least can limit the amount of FlowFiles / records created this way.
However, I wonder if it makes sense to move the existing max result check
if (listing.size() >= maxResults) {
    return false;
}

to the top of the RemoteResourceFilter? The limit only ever gets relevant, when there is a large amount of files at the remote source. Even though we cannot abort the listing prematurely, at least not without modifications to sshj, this way the checks needed for all files after the limit is reached is kept to a minimum.
What do you think about that?

Build above failed with a heap error.
Makes sense to me @EndzeitBegins, would also be great if we can get your early termination support in!
We may also need to have a think about which tracking strategies are supported when you select batch size. E.g. I don't think that time tracking will work with batch size since it relies on keeping track of the last timestamp processed?

Thank you for the adjustments.
While I hope we can get the changes to land in sshj I wouldn't wait for them. After introducing the general support for REMOTE_POLL_BATCH_SIZE in ListSFTP, we still can integrate a new version that includes the improvements in a following PR.
That's a very sensible remark. I think it makes sense to limit the support for REMOTE_POLL_BATCH_SIZE to the strategies BY_ENTITIES and NO_TRACKING only.  I'm not familiar with BY_TIME_WINDOW but I think combining the batch size with one of the other strategies might result in some files being skipped / never be picked up.

I don't have a lot to add to the comments I see already.  But the main gist is if the intent of the JIRA is now broader than the original title please change the JIRA.  If the intent has not changed then please narrow the PR.  So either this is about adding this option to ListSFTP or it is about others.  Would be important to validate each case which expands the scope of the review.
Given we don't actually get to, it appears, change the behavior of the SFTP commands executed on the remote end (ie: once you start a listing in a large directory...much of the damage is already done in terms of time/waiting) then the benefit of this property being available is mostly to protect the NiFi side.  Still useful just probably not the truly desired/most beneficial impact.
I'd avoid any changes for now that aren't central to the specific desired improvement.

Thanks for taking on the merge process @joewitt.
Could we backport these changes to the support/nifi-1.x branch as well?

@EndzeitBegins It would need a separate JIRA/PR would be needed and I dont think the 'removeIf' language mechanism would work as NiFi 1.x is Java 8 or higher.

Thank you for the fast response @joewitt and letting me know.
Hey @tombrisland, are you interested to work on a backport for 1.x or should I have a go at it?


Profile schema refactor

closes #394
This needs to be tested with this PR on the server:
https://github.com/OpenBazaar/openbazaar-go/pull/388
This also requires that you wipe your DB.

This looks good to me. @morebrownies and/or @hoffmabc can you test it as well?
Just as an FYI, I've added a task to the backlog to add the social media accounts interface to the settings profile tab. Currently there isn't a way in the UI to actually add any social media accounts.

Yep will test now

I blew away my DB and tried and I get this.

@hoffmabc you need to make sure you are running this server:
OpenBazaar/openbazaar-go#388

Thanks @rmisio.
What does this spinner indicate?

How come GUID is just floating here with no value? And didn't we agree to move to Peer ID?

Also how come Email address is not clickable and a mailto: link? Did we purposely decide not to do that?

That might be a question for @morebrownies

I'll review shortly. Making email clickable would be fine

@morebrownies @hoffmabc Could we please focus on stuff specifically related to the profile schema refactor and log bugs / JIRA stories for anything else?

I thought this was part of it (the email link) @rmisio sorry.

@hoffmabc re: chat spinner. That indicates that a previous page of chat messages are being fetched.

A JIRA link would have been nice as well since I didn't even know it existed until now.

How come GUID is just floating here with no value?

Good catch! That was a bug that I just fixed.

And didn't we agree to move to Peer ID?

I don't know anything about that, but if that's the case, I think we should get a bug / JIRA story and work on that separate from this. There's probably a lot of places in the UI, the term GUID is being used.

OK will file another issue on the GUID thing.

I tested adding a name, short desc, long desc, location, website, email, avatar, cover photo and it all seemed to work as expected.
My guid displays correctly too


Add version

I just updated my arcadia-pycolor version but besides checking my lock file I had no good way to confirm I was importing the correct version.
It is standard to put a package's version in the top-level __init__.py under the variable __version__:
>>> import importlib
>>> importlib.import_module("numpy").__version__
'1.26.4'
>>> importlib.import_module("pandas").__version__
'2.2.2'
>>> importlib.import_module("biotite").__version__
'0.40.0'
>>> importlib.import_module("dask").__version__
'2024.6.2'
>>> importlib.import_module("arcadia_pycolor").__version__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'arcadia_pycolor' has no attribute '__version__'

This PR adds __version__. Since we are now between 0.5.1 and 0.5.2, I've appended a pre-release identifier -dev. We could start doing -dev.X, -rc.X, -alpha.X, -beta.X, etc, but IMO this package doesn't require anything more complex than -dev.

Oops, yes, totally. This is my bad. I think the best way to do this is probably to use dynamic versioning, so that we can create new a release simply by creating a new tag on Github. This will avoids the need to hard-code the version anywhere.

Update: I've added dynamic versioning in #48.


docs(url-sync): adds missing parts in documentation fix #712

See my comment on the related issue, doc hotfixes

Will do (first time I have to close a PR because it targets develop instead of master ;P)

Yep I know, workflow gets a bit more complex I agree


add new primary color to palette

Closes #3877

What changed?

Add new color to the palette
Update disabled btn color and border color at Dark Mode

Adding Screen Shots:
-Image from Core

Image from EE


Renderer: Needs Code Review

Code review.
C++20.
Performance.
QuadTree spacial partitioning for rendering.

Done.

Done.


context.message will be modified after the Receive function returns, so the message in this closure may be invalid

https://github.com/AsynkronIT/protoactor-go/blob/dev/cluster/pid_cache.go#L65-L81

ctx.AwaitFuture captures the current message and runs the continuation with the message/context reset to the state it was in (it sends the continuation and current message as a message to itself)
Have you observed any case where this does not work as expected?

so sorry, my fork is not new. and i don't follow deeper. this bug is fixed at this line.
https://github.com/AsynkronIT/protoactor-go/blob/dev/actor/local_context.go#L447

this is a bug already fixed, so close this issue.


Directory of Files Importer

Create an importer to update existing works with files.

Remaining work:
-[ ] tests
-[ ] documentation


How to provision apiToken?

Hi!
I use provisioned DataSources via docker-compose and include file like this
  - name: HetznerCloud
    type: 'apricote-hcloud-datasource'
    access: proxy
    isDefault: false
    orgId: 1
    version: 1
    secureJsonData:
      apiToken: ${HETZNER_CLOUD_TOKEN}

But it's not working =(
How I should do this right way?
Moreover, if I paste working API Toket to field, I've got an error:
API Token was not configured or does not work, a valid API Token is required for the data source to access the Hetzner Cloud API
How to diagnose what's going wrong?
In Grafana Logs, after pressing Test button I see only this:
logger=context userId=1 orgId=1 uname=admin t=2024-08-07T13:30:12.529959137Z level=info msg="Request Completed" method=GET path=/api/datasources/uid/PC8C36DBD0101EDFD/health status=400 remote_addr=<IP_ADDRESS> time_ms=311 duration=311.669667ms size=159 referer=http://localhost:3000/connections/datasources/edit/PC8C36DBD0101EDFD handler=/api/datasources/uid/:uid/health status_source=server

Can you test the following?

Enable debug logging in the data source settings (Under "Development > Options")
Paste in the token
Test
Check the Grafana Logs again

You should see a request being made to the Hetzner Cloud API. Can you share that here?

I can't enable Debug, I assume this is because this datasource is provisioned.
Of course I've tried to enable Debug, but nothing happen.


boards/arm/nrf52_blenano2: add storage flash partition

This patch adds DT description of generic storage flash
partition. This fixes issue #7227.
Signed-off-by: Andrzej Puzdrowski<EMAIL_ADDRESS>

cc @leodesigner

Codecov Report

Merging #7345 into master will increase coverage by <.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #7345      +/-   ##
==========================================
+ Coverage   55.09%   55.09%   +<.01%
==========================================
  Files         474      474
  Lines       51786    51786
  Branches     9937     9937
==========================================
+ Hits        28530    28531       +1
+ Misses      19304    19303       -1
  Partials     3952     3952

Impacted Files
Coverage Δ

lib/posix/pthread.c
69.84% <0%> (+0.5%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update af8a0b1...59e13aa. Read the comment docs.

cc @carlescufi


Add a default shiftwidth for yaml files

Fixes #5697

Since 'expandtab' is set there is no point in setting 'tabstop' (which should hardly ever be change anyway).
I'll include it, Nikolai has not be actively maintaining files.

@brammool Isn't this included in the latest runtime files? (https://github.com/vim/vim/commit/eab6dff19f387469a200011bc6cf3508f5e43a4a)

It should have been, but somehow the file was accidentally deleted.  Fixed that now.


LGTM Alert cleanup

Have a few little burrs to sand off...
https://lgtm.com/projects/g/nccgroup/depthcharge

Changes for collection of 6 items staged innext @ 283f7dceb75d86204679597cfb8dad0d4142a9a3


Error on npm start

Output said to let you know, here is my debug log - any help would be appreciated! Thanks!!!
0 info it worked if it ends with ok
1 verbose cli [ '/usr/bin/node', '/usr/bin/npm', 'start' ]
2 info using<EMAIL_ADDRESS>3 info using<EMAIL_ADDRESS>4 verbose run-script [ 'prestart', 'start', 'poststart' ]
5 info lifecycle<EMAIL_ADDRESS><EMAIL_ADDRESS>6 silly lifecycle<EMAIL_ADDRESS>no script for prestart, continuing
7 info lifecycle<EMAIL_ADDRESS><EMAIL_ADDRESS>8 verbose lifecycle<EMAIL_ADDRESS>unsafe-perm in lifecycle true
9 verbose lifecycle<EMAIL_ADDRESS>PATH: /usr/lib/node_modules/npm/bin/node-gyp-bin:/home/ec2-user/workspace/angular-universal-serverless/node_modules/.bin:/sbin:/bin:/usr/sbin:/usr/bin
10 verbose lifecycle<EMAIL_ADDRESS>CWD: /home/ec2-user/workspace/angular-universal-serverless
11 silly lifecycle<EMAIL_ADDRESS>Args: [ '-c', 'npm run build && npm run server' ]
12 silly lifecycle<EMAIL_ADDRESS>Returned: code: 1  signal: null
13 info lifecycle<EMAIL_ADDRESS>Failed to exec start script
14 verbose stack Error<EMAIL_ADDRESS>start: `npm run build && npm run server`
14 verbose stack Exit status 1
14 verbose stack     at EventEmitter.<anonymous> (/usr/lib/node_modules/npm/lib/utils/lifecycle.js:255:16)
14 verbose stack     at emitTwo (events.js:106:13)
14 verbose stack     at EventEmitter.emit (events.js:191:7)
14 verbose stack     at ChildProcess.<anonymous> (/usr/lib/node_modules/npm/lib/utils/spawn.js:40:14)
14 verbose stack     at emitTwo (events.js:106:13)
14 verbose stack     at ChildProcess.emit (events.js:191:7)
14 verbose stack     at maybeClose (internal/child_process.js:891:16)
14 verbose stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:226:5)
15 verbose pkgid<EMAIL_ADDRESS>16 verbose cwd /home/ec2-user/workspace/angular-universal-serverless
17 error Linux 4.9.27-14.31.amzn1.x86_64
18 error argv "/usr/bin/node" "/usr/bin/npm" "start"
19 error node v6.11.0
20 error npm  v3.10.10
21 error code ELIFECYCLE
22 error<EMAIL_ADDRESS>start: `npm run build && npm run server`
22 error Exit status 1
23 error Failed at the<EMAIL_ADDRESS>start script 'npm run build && npm run server'.
23 error Make sure you have the latest version of node.js and npm installed.
23 error If you do, this is most likely a problem with the angular-universal-serverless package,
23 error not with npm itself.
23 error Tell the author that this fails on your system:
23 error     npm run build && npm run server
23 error You can get information on how to open an issue for this project with:
23 error     npm bugs angular-universal-serverless
23 error Or if that isn't available, you can get their info via:
23 error     npm owner ls angular-universal-serverless
23 error There is likely additional logging output above.
24 verbose exit [ 1, true ]

Thank you for your report. Could you confirm if now it is working?

Gets to the webpack section on npm start then I get errors again - I did the following just to see:

npm install -g @angular/cli
npm install -g typescript
npm install -g webpack

Here is my debug file this time:
0 info it worked if it ends with ok
1 verbose cli [ '/usr/bin/node', '/usr/bin/npm', 'start' ]
2 info using<EMAIL_ADDRESS>3 info using<EMAIL_ADDRESS>4 verbose run-script [ 'prestart', 'start', 'poststart' ]
5 info lifecycle<EMAIL_ADDRESS><EMAIL_ADDRESS>6 silly lifecycle<EMAIL_ADDRESS>no script for prestart, continuing
7 info lifecycle<EMAIL_ADDRESS><EMAIL_ADDRESS>8 verbose lifecycle<EMAIL_ADDRESS>unsafe-perm in lifecycle true
9 verbose lifecycle<EMAIL_ADDRESS>PATH: /usr/lib/node_modules/npm/bin/node-gyp-bin:/home/ec2-user/workspace/angular-universal-serverless/node_modules/.bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/aws/bin:/home/ec2-user/.local/bin:/home/ec2-user/bin:/opt/aws/bin:/home/ec2-user/.local/bin:/home/ec2-user/bin:/usr/local/bin:/home/ec2-user/.npm-global/bin
10 verbose lifecycle<EMAIL_ADDRESS>CWD: /home/ec2-user/workspace/angular-universal-serverless
11 silly lifecycle<EMAIL_ADDRESS>Args: [ '-c', 'npm run build && npm run server' ]
12 silly lifecycle<EMAIL_ADDRESS>Returned: code: 1  signal: null
13 info lifecycle<EMAIL_ADDRESS>Failed to exec start script
14 verbose stack Error<EMAIL_ADDRESS>start: `npm run build && npm run server`
14 verbose stack Exit status 1
14 verbose stack     at EventEmitter.<anonymous> (/usr/lib/node_modules/npm/lib/utils/lifecycle.js:255:16)
14 verbose stack     at emitTwo (events.js:106:13)
14 verbose stack     at EventEmitter.emit (events.js:191:7)
14 verbose stack     at ChildProcess.<anonymous> (/usr/lib/node_modules/npm/lib/utils/spawn.js:40:14)
14 verbose stack     at emitTwo (events.js:106:13)
14 verbose stack     at ChildProcess.emit (events.js:191:7)
14 verbose stack     at maybeClose (internal/child_process.js:891:16)
14 verbose stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:226:5)
15 verbose pkgid<EMAIL_ADDRESS>16 verbose cwd /home/ec2-user/workspace/angular-universal-serverless
17 error Linux 4.9.27-14.31.amzn1.x86_64
18 error argv "/usr/bin/node" "/usr/bin/npm" "start"
19 error node v6.11.0
20 error npm  v3.10.10
21 error code ELIFECYCLE
22 error<EMAIL_ADDRESS>start: `npm run build && npm run server`
22 error Exit status 1
23 error Failed at the<EMAIL_ADDRESS>start script 'npm run build && npm run server'.
23 error Make sure you have the latest version of node.js and npm installed.
23 error If you do, this is most likely a problem with the angular-universal-serverless package,
23 error not with npm itself.
23 error Tell the author that this fails on your system:
23 error     npm run build && npm run server
23 error You can get information on how to open an issue for this project with:
23 error     npm bugs angular-universal-serverless
23 error Or if that isn't available, you can get their info via:
23 error     npm owner ls angular-universal-serverless
23 error There is likely additional logging output above.
24 verbose exit [ 1, true ]

I'm going to go to stock angular universal and see if I can get that working and see if it sheds some light. Let me know if there is something else I can try for your repo. Thanks!

Does webpack section run correctly? Could you check 'npm run build'? If it will fail, please provide whole console output (staring from line with command).

Here is my entire CLI output, im very familiar with angular and aws, but this is my first try at angular universal. I'm actually having trouble with vanilla angular universal too so any help would be appreciated. I'm assuming my dev machine is missing something... Thanks!
[ec2-user@ip-10-0-0-41 workspace]$ git clone https://github.com/maciejtreder/angular-universal-serverless.git
Cloning into 'angular-universal-serverless'...
remote: Counting objects: 262, done.
remote: Compressing objects: 100% (122/122), done.
remote: Total 262 (delta 136), reused 255 (delta 129), pack-reused 0
Receiving objects: 100% (262/262), 675.22 KiB | 0 bytes/s, done.
Resolving deltas: 100% (136/136), done.
Checking connectivity... done.
[ec2-user@ip-10-0-0-41 workspace]$ cd angular-universal-serverless
[ec2-user@ip-10-0-0-41 angular-universal-serverless]$ PATH=$PATH:/usr/local/bin:~/.npm-global/bin
[ec2-user@ip-10-0-0-41 angular-universal-serverless]$ npm install
npm WARN prefer global<EMAIL_ADDRESS>should be installed with -g
npm WARN prefer global<EMAIL_ADDRESS>should be installed with -g

><EMAIL_ADDRESS>install /home/ec2-user/workspace/angular-universal-serverless/node_modules/node-sass
> node scripts/install.js

[ec2-user@ip-10-0-0-41 angular-universal-serverless]$ npm start

><EMAIL_ADDRESS>start /home/ec2-user/workspace/angular-universal-serverless
> npm run build && npm run server

><EMAIL_ADDRESS>prebuild /home/ec2-user/workspace/angular-universal-serverless
> npm run clean

><EMAIL_ADDRESS>clean /home/ec2-user/workspace/angular-universal-serverless
> rimraf dist

><EMAIL_ADDRESS>build /home/ec2-user/workspace/angular-universal-serverless
> webpack

sh: line 1:  2734 Killed                  npm run build

npm ERR! Linux 4.9.27-14.31.amzn1.x86_64
npm ERR! argv "/usr/bin/node" "/usr/bin/npm" "start"
npm ERR! node v6.11.0
npm ERR! npm  v3.10.10
npm ERR! code ELIFECYCLE
npm ERR<EMAIL_ADDRESS>start: `npm run build && npm run server`
npm ERR! Exit status 137
npm ERR!
npm ERR! Failed at the<EMAIL_ADDRESS>start script 'npm run build && npm run server'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the angular-universal-serverless package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     npm run build && npm run server
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs angular-universal-serverless
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!     npm owner ls angular-universal-serverless
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     /home/ec2-user/workspace/angular-universal-serverless/npm-debug.log
[ec2-user@ip-10-0-0-41 angular-universal-serverless]$ npm run build

><EMAIL_ADDRESS>prebuild /home/ec2-user/workspace/angular-universal-serverless
> npm run clean

><EMAIL_ADDRESS>clean /home/ec2-user/workspace/angular-universal-serverless
> rimraf dist

><EMAIL_ADDRESS>build /home/ec2-user/workspace/angular-universal-serverless
> webpack

Killed
[ec2-user@ip-10-0-0-41 angular-universal-serverless]$


Update Japanese localization on concepts/overview/working-with-objects/labels.md

What this PR does / why we need it:
content/ja/docs/concepts/overview/working-with-objects/labels.md is outdated.
File to update
https://github.com/kubernetes/website/tree/dev-1.19-ja.1/content/ja/docs/concepts/overview/working-with-objects/labels.md
Original
https://github.com/kubernetes/website/tree/release-1.19/content/en/docs/concepts/overview/working-with-objects/labels.md
diff between translated and v1.19
https://gist.github.com/da8d3b849aebd02969921ef3650ead01
Which issue(s) this PR fixes:
#26875

/assign @inductor

/lgtm

/lgtm

/approve


Missing commit function.

GitLens Version: v10.2.2

VSCode Version: all
OS Version: all

Steps to Reproduce:
**What is the current behavior?

Adding this feature is even more perfect!

Committing is already provided by the built-in Git extension in VS Code and I don't see a compelling reason to duplicate it.


Remove internal access level control (because default is internal)

👻  Remove internal access level control (because default is internal)

Codecov Report

Merging #376 into master will not change coverage.
The diff coverage is 0%.

@@           Coverage Diff           @@
##           master     #376   +/-   ##
=======================================
  Coverage   99.43%   99.43%
=======================================
  Files           5        5
  Lines         176      176
=======================================
  Hits          175      175
  Misses          1        1

Impacted Files
Coverage Δ

ReSwift/CoreTypes/Store.swift
100% <ø> (ø)
:arrow_up:

ReSwift/Utils/Assertions.swift
83.33% <0%> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b24ee18...6d6b187. Read the comment docs.


Displaying middleware messaging entities in the topology graph.

SSIA :]
todo:

[x] tests

cc @abonas
@miq-bot add_labels enhancement, providers/hawkular, ui, topology, wip, darga/no

great to see this PR!
the icon seems incorrect, it should be the same as defined here:
https://github.com/ManageIQ/manageiq/blob/master/app/decorators/middleware_messaging_decorator.rb#L6

Right, I wasn't sure whether to use the placeholder or something meaningful. Also the graph looks pretty crowded, what about disabling its the visibility by default?

up to uxd/PM if to hide or not. so far in all topologies afaik everything is shown by default.
@jdoyleoss @serenamarie125  wdyt?

IMO, I think it may be a good idea to hide most of the entities by default - unfortunately i don't understand the middleware environment, but if it's hierarchical maybe you could hide everything which is not the top 2 levels?  If you did that, I would suggest that you use an inline notification stating something to the effect that  "Currently only A and B are shown in the view.  Use the filter bar to view additional object types." @Jiri-Kremser @abonas what do you think?

My gut feel is that we should hide it by default. Message destinations are not likely a primary concern of a user. Some users have many datasources and applications deployed, which are primary concern, we should yield the space to them by default.
Question about the labels. Do we have to prefix everything with MIddleware? In this context it can be assumed.  I would prefer we just used Deployments, Datasources, Message Destinations, etc.

Question about the labels. Do we have to prefix everything with MIddleware? In this context it can be assumed. I would prefer we just used Deployments, Datasources, Message Destinations, etc.

I was just about to send exactly the same question regarding the prefix. I +1 to remove it (in a different PR)

@serenamarie125 @jdoyleoss thanks for your inputs.
hiding it can make sense. my only concern here is that this would be the only topology where something is hidden by default and it might confuse the users

@Jiri-Kremser other than the discussion above, LGTM + verified

I was just about to send exactly the same question regarding the prefix. I +1 to remove it (in a different PR)

@Jiri-Kremser could you please address it in a separate PR? should be super short n quick

@abonas I think that it's ok if it's different from the rest of the views.  I'm pretty sure that will be one of the things we suggest after the testing :)  As long as an inline notification or something to that effect is used, so the user understands what's going on, i think it is fine!

@abonas I think that it's ok if it's different from the rest of the views. I'm pretty sure that will be one of the things we suggest after the testing :) As long as an inline notification or something to that effect is used, so the user understands what's going on, i think it is fine!

another suggestion (not for this PR) I have is to have a dynamic legend ( in all topologies, not only middleware). For example - if there are no domains or server groups in the inventory - no need to display them in the legend

My suggestions for updated labels in another PR, from left to right in the screenshot:
EAP Instances
Deployments
Datasources
Message Destinations
VMs
EAP Domains
EAP Server Groups

My suggestions for updated labels in another PR, from left to right in the screenshot:
EAP Instances
Deployments
Datasources
Message Destinations
VMs
EAP Domains
EAP Server Groups

can you please open a github issue and describe it there? easier to handle and assign than taking it from here. thanks!

@miq-bot add_labels wip

EAP Instances, EAP Domains, EAP Server Groups

@jdoyleoss This is tricky, because it displays also entities comming from WildFly

@Jiri-Kremser what's the status of this PR? were the comments regarding hiding/notification addressed? (btw they can be done in a separate PR on top of this one as well if it's more convenient)

@abonas I'd rather do that in a separate PR

@miq-bot assign @chessbyte


Refresh switсhes multi-record view to single

System information:

Operating system (distribution) and version win10
DBeaver version 21.1.2 EA
related to #7014

Steps to reproduce, if exist:

Select some rows and switch to record view
Refresh table
Actual result: table switches from multi-view to single

Fixed

verified


Support for Custom build for CDH versions

I am using CDH 5.7.1 with the current version of Phoenix from CDH parcels. I am getting errors because of Jar mismatch. Can you please let us know of there is support for a custom build for the particular version of CDH used by the customer.

if you are using CDH5.7.1, then you ought to be using the Cloudera Labs packaging referenced in this announcement.