Auto-register fft backend to avoid context manager

Problem & Discussion
uarray makes multi backend support possible with modules like scipy.fft, but the current implementation still requires the user to set backends and use a context manager which is not exactly a very smooth user experience.
As @rgommers mentioned in (#10204 comment), it would be nice to auto register the backends on import.

Can we change uarray to let backends auto-register, so we don't have to make end-users use context managers? IIRC we only avoided auto-registration because NumPy devs didn't like it and there was an attempt to have NumPy adopt unumpy.

Quoting @hameerabbasi from (#10204 comment):

Anyone can Auto register today, it's only a question of changing it from "discouraged" to "allowed practice".

Solution

We need to update the vendored version of uarray used in SciPy. This is because the current uarray vendored version in SciPy doesn't support options like try_last=True which got added later in uarray #236.
Once we are done with the first step, even if the user has an older version of uarray installed, our backend will manage to utilize the newer vendored version. In any case we'll check and fall back to users uarray installation if the version is higher than the scipy vendored uarray. All this allows us to go ahead and update the scipy.fft._backend to always set a global backend for scipy with try_last=True.
For cupy, the final step would be to update the cupyx.scipy.fft module in the cupy library, making sure that on import of cupy, the scipy.fft module is tried and if it is found we can register the cupy backend there itself. All this will be wrapped in a try except block following EAFP.

Questions

I'm not really sure what happens when multiple libraries are used and imported. In that case if we go ahead to support auto-regostering for more than one library at the time of import, which backend to fall back to is not very clear and remains a question to think about.
Overhead

I was able to auto-register the backend (see Additional Context below) and already have the above solution working on my local build for scipy and cupy. I'll be happy to raise subsequent PRs to implement the solution mentioned.
Ps. I'm new to the SciPy community, apologies if I missed something. Would love to hear your feedback and thoughts on the solution and the questions. Thanks!
cc @rgommers @hameerabbasi @peterbell10 @grlee77
Additional context
Importing the following for demonstration.
import numpy as np
import cupy as cp
import scipy.fft
import cupyx.scipy.fft as cu_fft

# Basic
y_numpy = scipy.fft.fft(np.arange(10))

Current Correct Way:
# scipy with cupy array
x = cp.arange(10)
with scipy.fft.set_backend(cu_fft):
    y_cupy = scipy.fft.fft(x)  # Calls into cupyx and returns a cupy array

Proposed Way that currently fails:
# scipy with cupy array
x = cp.arange(10)
# without using `with set_backend(...)`
y_cupy = scipy.fft.fft(x)  # this will now raise a `TypeError`, but should be made to work (auto-dispatch)

TypeError: Implicit conversion to a NumPy array is not allowed. Please use .get() to construct a NumPy array explicitly.

Proposed Way that works after the solution:
# scipy with cupy array
x = cp.arange(10)
y_cupy = scipy.fft.fft(x)  #smooth experience for user due to auto-dispatching under the hood

3. For cupy, the final step would be to update the cupyx.scipy.fft module in the cupy library, making sure that on import of cupy, the scipy.fft module is tried and if it is found we can register the cupy backend there itself. All this will be wrapped in a try except block following EAFP.

This part is a bit concerning, as SciPy is an optional dependency of CuPy, and I feel too much magic has to happen behind the scene to make this proposal work. Can't we just ask users to explicitly call scipy.fft.set_global_backend() if this is desired?
One thing I don't like uarray (for now!) is that not much effort has been invested since it took the initial storm to SciPy/CuPy/etc. Only a few libraries support it, and I don't feel it's particularly welcomed among users and library maintainers compared to other dispatching mechanisms like __array_function__. Maybe @rgommers can correct me (I think we chatted about it recently?)
cc: @kmaehashi (not sure who else to ping for this subject...I've been maintaining CuPy's FFT support these days 😅)

Hi, @leofang thanks a tonne for your feedback :))
As you mention CuPy's optional dependence on SciPy, the only thing to make this work is to add a (similar to this) try-except block (still SciPy remains optional) here and register the backend with _scipy_fft.register_backend(_cupy_fft). I don't think this is too much magic.

One thing I don't like uarray (for now!) is that not much effort has been invested since it took the initial storm to SciPy/CuPy/etc. Only a few libraries support it, and I don't feel it's particularly welcomed among users and library maintainers compared to other dispatching mechanisms like array_function. Maybe @rgommers can correct me (I think we chatted about it recently?)

Yes, I completely agree with you that uarray has not been adopted widely, but since we already support it with CuPy new fft why not just make it a bit smooth for the user?

Yes, I completely agree with you that uarray has not been adopted widely, but since we already support it with CuPy new fft why not just make it a bit smooth for the user?

You still need to motivate why users can't just call set_global_backend() and register a backend themselves. Like you said it's just extra one or two lines of code, and I don't see you'd consider the status quo "not smooth".
From CuPy's perspective, if SciPy decides to do black magic behind the scene, CuPy will follow. The main question is if it's really necessary. Of course, the other two questions you raised (presence of multiple backends and performance overhead) are also important. I personally prefer transparency for this particular subject.

There's a couple of questions here:

Only a few libraries support it, and I don't feel it's particularly welcomed among users and library maintainers compared to other dispatching mechanisms like __array_function__. Maybe @rgommers can correct me (I think we chatted about it recently?)

Why isn't the uarray backend system used much at the moment?
I suspect the main reason is that there is no documentation. The tutorial (http://scipy.github.io/devdocs/tutorial/fft.html) doesn't mention it at all. The reference API docs (http://scipy.github.io/devdocs/reference/fft.html#backend-control) have very sparse examples, and those use SciPy itself only, so they don't really do anything useful. There is also no list of which libraries have a backend available. A few good examples would help a lot.
The other reasons I'd expect to be that:

having only fft available doesn't make for a very good pattern worth learning
if it takes more code than just importing cupy (or whatever other library), the benefits aren't obvious.

The one CuPy example I could find so quickly here is (from https://github.com/scipy/scipy/issues/12652#issuecomment-667348345):
>>> import cupyx.scipy.fft as cp_fft
>>> import scipy.fft
>>> import numpy as np
>>> a = cupy.arange(110).reshape((10, 11)).astype(float)
>>> with scipy.fft.set_backend(cp_fft):
...     b = scipy.fft.fft2(a)
...     c = scipy.fft.fft2(cupy.asnumpy(a))
>>> type(b)
cupy.core.core.ndarray
>>> type(c)
numpy.ndarray

Comparison to other dispatching mechanisms like __array_function__
There's a couple of different things here:

We made a large push with __array_function__ in 2018/2019 (blog posts, talks, release announcement). It got taken up by RAPIDS mainly, but showed shortcomings. As a result we got both the like= addition to creation functions (which is pretty ugly), and exploring better alternatives, like __array_module__ and unumpy (based on uarray).
Uarray is a generic backend mechanism, it's more appropriate to compare __array_function__ to unumpy. Part of what we needed is a backend mechanism for SciPy, because the limitation that you can have exactly one implementation per array type doesn't work. E.g. there would be no way to let mkl_fft or PyFFTW work with something like __array_function__. Also the __array_function__ machinery isn't even public for reuse by SciPy and other libraries; it's not really meant for that.

You still need to motivate why users can't just call set_global_backend() and register a backend themselves. Like you said it's just extra one or two lines of code, and I don't see why you'd consider the status quo "not smooth".

I think something like:
import cupyx.scipy.fft as cp_fft

with scipy.fft.set_backend(cp_fft):
    b = scipy.fft.fft2(a)

is already a decent amount of mental overhead for an average user. It gets worse though in real-world code. Say we want to use fft, linalg and ndimage in an application (not hypothetical, see the linked gh-10204). These have cupyx (and Dask, and JAX) counterparts. Now with the current way, we'd need three context managers. That's getting pretty painful.

One thing I don't like uarray (for now!) is that not much effort has been invested since it took the initial storm to SciPy/CuPy/etc.

This I agree with. Basically we all have suffered from a lack of resources for this complicated topic. Making other SciPy modules dispatchable we now finally have a bit of momentum for, because of the CZI grant for scikit-image (which heavily relies on ndimage) - but no one had serious bandwidth or funding to push this topic forward till now. This is also true for the exploratory efforts to use CuPy with scikit-learn. I'm hoping we can get some more momentum behind all of this soon - this auto-registration topic is kind of in preparation for that (or the first step towards it).

From CuPy's perspective, if SciPy decides to do black magic behind the scene, CuPy will follow. The main question is if it's really necessary. Of course, the other two questions you raised (presence of multiple backends and performance overhead) are also important. I personally prefer transparency for this particular subject.

There may be other options worth exploring, and we shouldn't be in a major hurry to push this decision through. What I would like is that it becomes possible to write "normal" code with a friendly UX (which is Uarray's main problem right now). In downstream library code:
from scipy import fft, linalg

def somefunc(x : Array, ...) -> Array:
    y = fft.fft2(x)
    s, v, d = linalg.svd(x)
    ....

and in end user code simple calls:
from scipy import fft, linalg

fft.fft2(x)
linalg.svd(x)

Having everything completely auto-register is not the only option, although it seems quite reasonable to me. But for example having one global opt-in could also be preferable to the current context manager pattern.
Finally, I don't think auto-registration is "black magic". Basically any other pattern (single/multiple dispatch, or registrering third-party functionality in PyTorch/TF) works like this. Having the context manager is just really poor UX, and it's wasn't the original design for Uarray. It was added specifically because Hameer's main focus was on Unumpy as a __array_function__ replacement, and NumPy devs insisted on it being opt-in in order to be considered for that purpose.

Also I think part of the complexity of the implementation is unnecessary. It came from (initially __array_ufunc__ and __array_function__) wanting to enable things like:
>>> import numpy as np
>>> import dask.array as da
>>> x = da.ones(4)
>>> y = np.full(4, 2)
>>> np.broadcast_arrays(x, y)   # NOTE: `x` and `y` are different types of arrays!
[dask.array<ones, shape=(4,), dtype=float64, chunksize=(4,), chunktype=numpy.ndarray>,
 dask.array<array, shape=(4,), dtype=int64, chunksize=(4,), chunktype=numpy.ndarray>]

I think that kind of mixed-kind array operation should just raise an exception like CuPy, PyTorch, TensorFlow, etc. all do - wanting to allow mixing everything together just ends up with lots of bugs and hard-to-predict behavior.

I think that simply importing a backend should not automatically enable it, at least not in all cases.
For example, if from pyfftw.interfaces import scipy_fft automatically enabled the corresponding backend, FFT performance would go down on average, since this backend implies FFTW_ESTIMATE (I think), which typically results in worse FFTs than with the builtin backend. This backend is great to have in specific circumstances (e.g. when using FFTW_MEASURE and working with plans), but not necessarily as a blanket replacement for the default backend.

Yes, I don't think mkl_fft or pyfftw should ever register on import, but for array libraries like cupy or dask there's no issue. Their backends should just return NotImplemented if they aren't called with their specific array type.
The only major down-side is increased dispatch overheads, since we'll need to call into every registered backend for it to check the array type.

Sorry I took a long long break and just begin to catch up with things slowly...
@rgommers Thanks for your nice and detailed reply on uarray and __array_function__. Just a few quick comments:

having only fft available doesn't make for a very good pattern worth learning

Indeed, FFT should be considered as an exception given that there are several implementations of FFT libraries out there, which is generally not the case for most modules in SciPy except for perhaps linalg and ndimage, so it makes little sense to think about backend mechanisms that back a common, well-defined set of APIs. Perhaps with the efforts in the array API and scikit-{learn, image} things will be different soon.

I suspect the main reason is that there is no documentation.
The one CuPy example I could find so quickly here is (from [#12652 (comment)]

Yeah it's not well documented. In CuPy it's mentioned here without a self-contained code snippet, which is kinda my overlook during my recent update to the FFT docs 😅 I'll work on that.

You still need to motivate why users can't just call set_global_backend() and register a backend themselves. Like you said it's just extra one or two lines of code, and I don't see why you'd consider the status quo "not smooth".

I think something like:
import cupyx.scipy.fft as cp_fft

with scipy.fft.set_backend(cp_fft):
    b = scipy.fft.fft2(a)

is already a decent amount of mental overhead for an average user.

I was referring to this usage:
import cupy as cp
import cupyx.scipy.fft as cp_fft
import scipy.fft

scipy.fft.set_global_backend(cp_fft)  # <--
a = cp.random.random(100)
b = scipy.fft.fft(a)

I think there's not much overhead for users to just add that extra line (set_global_backend) and it simply works without using any context manager as @AnirudhDagar originally hoped for, so I am still not convinced why we'd like to register on import solely by this argument...
By the way, I just saw #14359 got in. I feel it's a poor design mixing the roles of register_backend() with set_global_backend(). The distinction between them gets blurred and I am sure users would be confused...

There may be other options worth exploring, and we shouldn't be in a major hurry to push this decision through. What I would like is that it becomes possible to write "normal" code with a friendly UX (which is Uarray's main problem right now). In downstream library code:
from scipy import fft, linalg

def somefunc(x : Array, ...) -> Array:
    y = fft.fft2(x)
    s, v, d = linalg.svd(x)
    ....

and in end user code simple calls:
from scipy import fft, linalg

fft.fft2(x)
linalg.svd(x)

Is the Array object here complaint with the Array API? If so it's indeed the ultimate goal we are seeking.

Having everything completely auto-register is not the only option, although it seems quite reasonable to me. But for example having one global opt-in could also be preferable to the current context manager pattern.
Having the context manager is just really poor UX

This I don't disagree. Not only an overhead for users, sometimes context managers can also be hard to handle for library developers.

@rgommers this is definitely something we should look at irrespective of this issue. I can work on improving the docs here and maybe add a similar note for some supported libs to what's done in CuPy FFT's Code compatibility features point 4 and also show some real examples instead of setting the scipy backend (which doesn't really highlight the real use case).

I suspect the main reason is that there is no documentation. The tutorial (http://scipy.github.io/devdocs/tutorial/fft.html) doesn’t mention it at all. The reference API docs (http://scipy.github.io/devdocs/reference/fft.html#backend-control) have very sparse examples, and those use SciPy itself only, so they don’t really do anything useful. There is also no list of which libraries have a backend available. A few good examples would help a lot.

@leofang if you'd like I can also help with a follow up to cupy/cupy#5272 for adding documentation example code snippets.

Yeah it's not well documented. In CuPy it's mentioned here without a self-contained code snippet, which is kinda my overlook during my recent update to the FFT docs 😅 I'll work on that.

@leofang the idea behind adding try_last arg was to make it possible for the user to utilize both the backends (whichever is available, first checking for the user registered backend and in the end checking for the global backend registered inside scipy itself).

By the way, I just saw #14359 got in. I feel it's a poor design mixing the roles of register_backend() with set_global_backend(). The distinction between them gets blurred and I am sure users would be confused...

import numpy as np
import cupy as cp
import cupyx.scipy.fft as cp_fft
import scipy.fft

scipy.fft.register_backend(cp_fft)
a = cp.random.random(100)
b = scipy.fft.fft(a)
c = np.random.random(100)
d = scipy.fft.fft(c)  # <-- Raises `BackendNotImplementedError ` here since it can't find a backend.

Before the PR the above raised. IMO asking the user to repeatedly change or set backends to switch implementations is not UX friendly. But, if you still want to do it the old way, you can do that with set_global_backend.

I read the above thread and have some thoughts. Taking inspiration from this example, I think auto-registering can help the maintainers of packages which use SciPy as a dependency. Say, a package A performs linalg(or fft or ndimage, name doesn't matter here) operations. Now, SciPy has CPU implementation and several libraries have GPU implementations. The developer of A may not know which GPU library is present in their user's code so they cannot register everything beforehand. In auto-registering SciPy can check for the input type, find the associated library present in the user's system and if available register it. If it's already registered then no need to do it again. If the input is cupy arrays then cupy can be registered, similarly if pytorch tensors are input then pytorch can be registered.


Update goodness to include a link to JankyBorders.adoc

Thanks! I'm curious, would it be useful to mention that borders CLI can be called after-startup-command callback, like so?
after-startup-command = 'exec-and-forget borders active_color=0xffe1e3e4 inactive_color=0xff494d64 width=5.0'

JankyBorders readme mentions the same for yabairc, but I'm unfamiliar with yabai. Is yabairc run on every relaunch/config update? If so, don't users get two JankyBorders processes running?

It's not uncommon for a yabairc to include a killall borders before launching the process (same goes for SketchyBar).

There is no need to killall borders. It has a built-in detection of already running process

Yeah, good point! I must have been thinking of SketchyBar. There was a time when this approach (kill and restart) was commonly used, but that experience has also improved.


Autobuild on dockerhub?

Hi, I could not find the repo on dockerhub. You could add docker autobuild by linking the github repo here:
https://hub.docker.com/repository/create

Hey, just to make sure, I already have this repo linked with dockerhub:
https://hub.docker.com/repository/docker/spiros123/ukbiobank-utils
or is this not what you mean?

Yep, just did not find it. Modified the PR #3  with the correct path.


Release 1.6.0

There is a sizable chunk of CHANGES not tagged in a release. I think after the bug fix #57, which caused a performance degration, and the optimization #62 I think things are in a decent spot for a new tag to be created.
The tag will probably be made in the next couple days. After the freshest commits have cured for a short time.

Postponing this to include the optimization discussed in #63.
The release will not include #65 so there are no compatibility concerns.

Also added #67 because it's technically a defect, though it has been around for a while.


protect packetstream.read against falsy input

Context
In the stderr logs of one ssb-room server with muxrpc 6.5.0, I found these, quite frequently:
2|ssb-room | TypeError: Cannot read property 'name' of null
2|ssb-room |     at PacketStreamSubstream.stream.read (/root/ssb-room/node_modules/muxrpc/stream.js:57:25)
2|ssb-room |     at PacketStream._onstream (/root/ssb-room/node_modules/packet-stream/index.js:224:13)
2|ssb-room |     at PacketStream.write (/root/ssb-room/node_modules/packet-stream/index.js:135:41)
2|ssb-room |     at /root/ssb-room/node_modules/muxrpc/pull-weird.js:56:15
2|ssb-room |     at /root/ssb-room/node_modules/pull-stream/sinks/drain.js:24:37
2|ssb-room |     at /root/ssb-room/node_modules/pull-goodbye/node_modules/pull-stream/throughs/filter.js:17:11
2|ssb-room |     at Object.cb (/root/ssb-room/node_modules/packet-stream-codec/index.js:111:11)
2|ssb-room |     at drain (/root/ssb-room/node_modules/pull-reader/index.js:39:14)
2|ssb-room |     at more (/root/ssb-room/node_modules/pull-reader/index.js:55:13)
2|ssb-room |     at /root/ssb-room/node_modules/pull-reader/index.js:66:9

Problem
I did a search for .name in any of the modules, and found that one of the cases was in muxrpc. I put a console.log if data was falsy, and then re-ran the server and saw that console.log, which confirms that this code was not protected for cases where data is undefined and we try to access data.name.
Solution
If data is undefined, return an error instead of crashing Node.js.

PS: I still don't know in what circumstances do we get data undefined, but this PR helps us get closer to debugging, and at least protects us from a hard crash.

Oh, I figured it out. The first arg here is null, and that's the data arg: https://github.com/ssbc/packet-stream/blob/c3b4d75794dc6e89ea84d2c9fcb309259de723f6/index.js#L224
Why is it null?

Note, the ssb-room server was running muxrpc 6.5.0, I think if I would update it to the latest (containing PR #64), then this would be solved.
I think we can close this PR #67.

Thanks @christianbundy for taking a look at it, but I think this is outdated (see https://github.com/ssb-js/muxrpc/pull/67#issuecomment-822463043 )


Update Readme to discourage use of the baseminer

The baseminers are not competitive on the network, and it would be useful to new miners for our repo to reflect this.

Addressed in https://github.com/macrocosm-os/prompting/pull/244


Update of the DecayTree structure

I copied DecayTree structure from JPetGeant4 to the Framework, in order to be able to analyze simulations done by the new version of the software.

Found 2 violations:
Reporter: cppcheck-matrix-raporter
Rule: unmatchedSuppression
Severity: INFO
File: src/GeantParser/JPetGeantDecayTree/JPetGeantDecayTree.cpp L42
Unmatched suppression: unusedFunction

Reporter: cppcheck-matrix-raporter
Rule: unmatchedSuppression
Severity: INFO
File: src/GeantParser/JPetGeantDecayTree/JPetGeantDecayTree.cpp L65
Unmatched suppression: unusedFunction

Found 2 violations:
Reporter: cppcheck-matrix-raporter
Rule: unmatchedSuppression
Severity: INFO
File: src/GeantParser/JPetGeantDecayTree/JPetGeantDecayTree.cpp L42
Unmatched suppression: unusedFunction

Reporter: cppcheck-matrix-raporter
Rule: unmatchedSuppression
Severity: INFO
File: src/GeantParser/JPetGeantDecayTree/JPetGeantDecayTree.cpp L65
Unmatched suppression: unusedFunction

I had studied boost::graph and property_tree, but i do not think that these implementation would be useful,
as in simulations two hits could have the same multiplicity and the only way to distinguish them is by the trackID, so
creation new vertex/node would be complicated.
Tomorrow i would propose my implementation of the tree structure that would be better for our simulations.
If you do not merge these PR till tomorrow you could check new version of the decayTree.

Looks fine for me. Thanks!


Publish the licence(s) under which the application and services are provided

Should be places in the Licence section of the app, source files, API description maybe.
Verify that the licence conforms to the licences of the used components.

It is in the root folder, which would have to be enough for now I guess. Suggest to close


Unable to back out of the Settings screen

Describe the bug
When I go to the Settings screen I am unable to return
To Reproduce
Steps to reproduce the behavior:

Go to Settings screen
Click on the back button
Nothing happens

Expected behavior
It takes me back to the main view
Screenshots

This bug is a stupid copy/paste error, in your case the button is disabled because you don't have an image loaded.


Halt add-on with exit code 5

Hi all,
I'm trying to use the HAOS-SBFspot to read data from my Sunny Boy 3000TL-20 into HomeAssistant.
I'm trying to set it up to only integrate with HA, so no need for a Maria dB or PVoutput. I configured them with empty/fake values (I found in another ticket that this should do the trick).
Now, when starting, it did create the MQTT integration, but I do not get any devices or sensors.
Environment: HA on a RPi4.
HA Core version 2024.1.3
The startup logs of the add-on show the following:
s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service legacy-services: starting
services-up: info: copying legacy longrun sbfspot (no readiness notification)
s6-rc: info: service legacy-services successfully started
[00:00:08] INFO:
[00:00:08] INFO:  Linux 6.1.63-haos-raspi #1 SMP PREEMPT Tue Jan  9 10:43:14 UTC 2024 aarch64
[00:00:08] INFO:
|   INITIALISE MQTT SENSOR CREATION   |
sma
core-mosquitto
sma
homeassistant/sbfspot_{plantname}/sbfspot_{serial}
PrgVersion,Plantname,Timestamp,SunRise,SunSet,InvSerial,InvName,InvTime,InvStatus,InvSwVer,InvClass,InvType,InvTemperature,InvGridRelay,EToday,ETotal,GridFreq,PACTot,PAC1,UAC1,IAC1,OperTm,FeedTm,PDCTot,UDC1,UDC2,IDC1,IDC2,PDC1,PDC2,BTSignal,InvWakeupTm,InvSleepTm
[00:00:09] INFO: Using Existing Device info

PrgVersion Plantname Timestamp SunRise SunSet InvSerial InvName InvTime InvStatus InvSwVer InvClass InvType InvTemperature InvGridRelay EToday ETotal GridFreq PACTot PAC1 UAC1 IAC1 OperTm FeedTm PDCTot UDC1 UDC2 IDC1 IDC2 PDC1 PDC2 BTSignal InvWakeupTm InvSleepTm
[00:00:10] INFO: Setting Up PrgVersion
[23:00:11] WARNING: Halt add-on with exit code 5
s6-rc: info: service legacy-services: stopping
s6-rc: info: service legacy-services successfully stopped
s6-rc: info: service legacy-cont-init: stopping
s6-rc: info: service legacy-cont-init successfully stopped
s6-rc: info: service fix-attrs: stopping
s6-rc: info: service fix-attrs successfully stopped
s6-rc: info: service s6rc-oneshot-runner: stopping
s6-rc: info: service s6rc-oneshot-runner successfully stopped

The first startup showed something different, but I was to late to grab that.
In the home-assistant.log file, I found te following error:
2024-01-22 23:50:46.480 ERROR (MainThread) [homeassistant.components.hassio] Failed to to call /addons/a51a23d8_haos-sbfspot/stats - Container addon_a51a23d8_haos-sbfspot is not running
Thanks for your help

Hi @scauwe
I can't seem to find anything relating to error code 5 from s6 overlay or container errors. So I am not sure what that is pointing to.
It is stopping at the creation of the first MQTT sensor by the look of it.
A couple things to try.
A complete system reboot (not just restarting home assistant). That will clear out any ghost containers that might be related to the home assistant log error.
Then try starting the addon again and check the error. If it fails again, try starting the addon without the sensor creation option turned on.

Hi @habuild ,
Thanks for the answer. It guided me to what I needed.
The reboot did not help.
Disabling the MQTT topics showed that it was connecting to the inverter via bluetooth. When trying to publish the results, the following was logged:
Client null sending CONNECT
Connection error: Connection Refused: not authorised.
Error: The connection was refused.
Client null received CONNACK (5)
MQTT: Failed te execute '/usr/bin/mosquitto_pub' mosquitto client installed?
Error 1280 while publishing to MQTT Broker

In the mosquito help, I found that MQTT by default uses the HA users to authenticate. Since I was not using the default user, that was the problem! I created the user configured in HAOS-SBFspot in HA and restarted both MQTT and HAOS-SBFspot. SBFspot could now connect with te configured user and password without a problem. I reconfigured HAOS-SBFspot to create the topics and I found them as sensors in HA.
Thanks a lot!


High CPU Usage with NioEventLoopGroup

Expected behavior
The CPS usage should remain stable. The event loop groups should get closed on shutdown, and the number should not constantly increase.
Actual behavior
The number of eventloopgroups constantly increase inspite of closing them. Some of them endlessly wait before getting shut down.
"nioEventLoopGroup-8870-1" tid=13320
java.lang.Thread.State:  RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:281)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:105)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:98)
- locked <0x4f8900cd> (a io.netty.channel.nio.SelectedSelectionKeySet)
- locked <0x83732742> (a java.util.Collections$UnmodifiableSet)
- locked <0x288f21af> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:109)
at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)
at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:756)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:411)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:811)
Locked ownable synchronizers:
- None

Steps to reproduce
Keep creating eventloop groups and close them as well. Monitor for a long time.
Minimal yet complete reproducer code (or URL to code)

Netty version
4.1.25
JVM version (e.g. java -version)
1.8
OS version (e.g. uname -a)

thread_dump.zip
Have attached thread dump taken over 3 days. You can see the constant increase in the number of eventloopgroups.
For eventloop creation
eventLoopGroup = new NioEventLoopGroup();
For closing:
eventLoopGroup.shutdownGracefully().addListener(new GenericFutureListener() {
                        @Override
                        public void operationComplete(final Future future) throws Exception {
                           -------------
                        }
      });

I am pretty sure you not shutdown in all cases or continue creating new ones.

The first thing that i did was to check the number of shutdowns versus the number of creations of event loop groups. And they exactly match. Any workarounds you suggest?

@rohitsahay2000 the only thing I can tell you from the thread dumps is that there are many NioEventLoops that are not shutdown. Are you sure the shutdownGracefully() really completes ?
Also just generally I wonder why you create and shutdown so many groups. Normally you create one group at the start and re-use it during the life-time of the whole app.

Are you sure the shutdownGracefully() really completes ?
--> I can only call this method for shutting down the event loop group gracefully. Anything else you suggest to shutdown the event loop group and monitor the completes?
Also just generally I wonder why you create and shutdown so many groups. Normally you create one group at the start and re-use it during the life-time of the whole app.
--> One event loop group causes very high CPU usage because of the volume of data which needs to be processed.
I have seen similar issues like this:
https://github.com/netty/netty/issues/327
https://github.com/netty/netty/issues/5278
and there are many others which says the problem very similar to i have pointed out. I see there are lot of comments but the bug was generally closed because of lack of response or it was very old. Don't see any fixes which was done around this.
If anything you can suggest would be great.

Hi @normanmaurer , Did some more profiling. I have a event loop group of 100 threads. Seeing lot of threads going in awaiting state before CPU going more than 100% and app becomes unresponsive. Example
`Name: nioEventLoopGroup-2-11
State: RUNNABLE
Total blocked: 129  Total waited: 95
Stack trace:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:281)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:105)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:98)

locked io.netty.channel.nio.SelectedSelectionKeySet@8f0ca6c6
locked java.util.Collections$UnmodifiableSet@dba88c4
locked sun.nio.ch.EPollSelectorImpl@ed7a1c60
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:109)
io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)
io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:756)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:411)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:811)
`
This is one example, there are many threads like this and the count of such thread increases eventually. Can you please help?

@rohitsahay2000, I think you are discussing two different things: 1) high CPU usage and 2) old event loops not shutting down.
I don't think (2) is an "issue" or at least it is working as intended. Netty creates new threads pretty aggressively and doesn't really shut them down until the end. (And when shutting down an event loop, it even creates any threads that hadn't been created earlier!)
(1) is interesting though. Could you give the precise jvm version? Like at least including the patch version like "1.8.0_121", but the full output of java -version would be great.

@rohitsahay2000, could you also provide uname -a?

An easy way to see which threads are using CPU is to use top and type shift-H. That shows all threads. Thread ids in jstack are hex, so you will need to convert the thread id to hex to compare to jstack output.
http://www.boxjar.com/using-top-and-jstack-to-find-the-java-thread-that-is-hogging-the-cpu/ is an example walkthrough.

Attaching

Screenshot of the process with high CPU. (top -pXXXX)

Screenshot of all the threads for that process (shift -H result)

stack of one such thread is

nioEventLoopGroup-2-72 tid=183 	java.lang.Thread.State:  RUNNABLE 		at sun.nio.ch.DatagramChannelImpl.send0(Native Method) 		at sun.nio.ch.DatagramChannelImpl.sendFromNativeBuffer(DatagramChannelImpl.java:533) 		at sun.nio.ch.DatagramChannelImpl.send(DatagramChannelImpl.java:495) 		at sun.nio.ch.DatagramChannelImpl.send(DatagramChannelImpl.java:474) 		  locked <0x93a2fc8c> (a java.lang.Object) 		at io.netty.channel.socket.nio.NioDatagramChannel.doWriteMessage(NioDatagramChannel.java:293) 		at io.netty.channel.nio.AbstractNioMessageChannel.doWrite(AbstractNioMessageChannel.java:142) 		at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:934) 		at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.forceFlush(AbstractNioChannel.java:367) 		at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:640) 		at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:581) 		at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) 		at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460) 		at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884) 		at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) 		at java.lang.Thread.run(Thread.java:811) 	Locked ownable synchronizers: 	 None

These stacks look like you are just producing a lot of data that needs to be send. There is nothing we can do about it.

That said you may want to try to use EpollDatagramChannel as it supports using sendmmsg which will help to reduce the syscalls.

i find the same question. only 8 threads ,when cpu high , it won't fall.

@rohitsahay2000 I have some suggestions for you

check if the outbound bandwidth is enough, if it is not enough, CPU will be high sometimes.
check if your CPU cores are balanced well with your internet card, you may need this:
check the interrupts for each CPU core, if they are not balanced well, CPU will be high. You can use mpstat -I SUM -P ALL to check it out.

Strange ! You can put a disclaimer that high data usage is not advisable while using netty. I thought a issue is closed only when a resolution is provided not when "nothing can be done" and there is an existing bug.

Sure high data usage is supported but what you show is basically the overhead of sending data via UDP. At least this is what the stack trace is showing and as this is a syscall there is nothing we can do about this. The only thing you could do is to try to batch these. If you can share a reproducer that allows us to observe the same "problem" I am more then happy to have a look. But without seeing code etc there is really nothing we can do.
sun.nio.ch.DatagramChannelImpl.send0  is basically the native method of the JDK that will do the sys call.


feat(elements): split ui-patterns into lib and playground package

Closes #1100

Very cool 😎
Just one thing, the styles of tailwind seem to be missing in our playground app. You might need to import the stylesheet of the ui-patterns bundle in the root of the playground application.

ups, forgot the elements-react import. The Tailwind styles are applied correctly (see the change when removing @import<EMAIL_ADDRESS>in the global css file index.css
Its fixed now


add a Code of Conduct?

Perhaps we should have a code of conduct document for BBS, not just for contributing and participating in code contributions, but for the actual content of BBS itself? It's on my mind as I update the Nodeschool SF site, since the CoC extends to not only the website but the actual events themselves.

I'm generally for this, although I admit I don't know the best way to draft these things.

@selenamarie ?

@llkats django has a pretty reasonable one:
https://www.djangoproject.com/conduct/
https://www.djangoproject.com/conduct/enforcement-manual/.
https://www.djangoproject.com/conduct/reporting/
Its a lot but it helps cover all the dumb stuff that can happen


[WIP] Source Control Navigator

Description
Added support for viewing changed files.
Checklist

[ ] Add support for Repos
[ ] Add support for Remotes
[ ] Add support for Stashed Changes
[ ] Add support for Tags
[ ] Add support for Recent Locations
[ ] Add support for Branches
[ ] Add Git view in Inspector Navigator

[x] I read and understood the contributing guide as well as the code of conduct
[x] My changes generate no new warnings
[x] My code builds and runs on my machine
[ ] I documented my code
[ ] Review requested

Screenshots

Also add more info to the PR description.

And please add documentation to all the view models and views so we know what it does exactly without reverse engineering the code.

Like the title says WIP and the checkbox for documentation is unchecked... I'm aware I need to add documentation...

Also add more info to the PR description. I took a quick look at the coffee and it seems there are a lot more changes that are not mentioned in the description or shown on screenshots

Idk if you read the description and saw the unchecked checkboxes?! It's a whole navigator and editor being built in one PR there will be multiple changes

@nanashili This is a really big PR and it will be difficult to review once it's ready.


Update de.json

Thanks!!

I think that with this I have all I need for the next beta release (unless I've missed something :)). @vanja-san and @ahmed-0011 could you update your localization files? It'd would be great if you can, so the release has all the localization files up to date, but I understand if you can't anytime soon. Let me now in any case if you read this. I need to make more testing so, even if there are no complications, it will still probably take me a few days before I can build and upload the release.
Thanks all for your contributions!


Changed the default $group variable from '0' to ''

This fixes the bug outlined in #32

Thank you @charlesLF !


Update alt_serde_json as well🙏

@dtolnay
Would you be so kind to update serde_json as alt_serde_json crate as well? I need the latest feature of "alloc", but needing to include the serde package as alt_serde.
Thanks.

Hi @jimmychu0807, I am not an owner of that crate so I cannot publish updates to it.


Update CONTRIBUTING.md

Summary
update the contributing.md to add VSIX instructions.

Covered in #701

Closing this one as it was handled in another PR. Feel free to re-open as needed :)


minimum arguments?

Thanks for sharing.
I have a phase coherency matrix (2D) and I thought if I feed that into any of these functions, along with channel names, I may be able to get some plots out, but it seems that I was wrong.
So would any of the included plotting functions work with simply having this matrix and channel names as an input, or do I really need all other input arguments? If not, what would be the minimum number of inputs arguments one may need, to be able to plot any type of connectivity network, using this toolbox?

Hi @khashayar854,
Are you talking about the standard 2D heatmap? If yes, I don't think you need to provide all of the inputs. For example, here's a minimal working example :
from netchos import heatmap, circular
import numpy as np
import pandas as pd

# generate random data
n_nodes = 15
data = np.random.rand(n_nodes, n_nodes)
nodes_name = [f'roi_{k}' for k in range(n_nodes)]

# dataframe conversion
df = pd.DataFrame(data, index=nodes_name, columns=nodes_name)

# 2D heatmap
heatmap(df)

# circular plot
# circular(df)

Ah yes, you are right I need to include this file when installing it. I'll do it asap. If you want to use now without waiting, please try :
git clone https://github.com/brainets/netchos.git
cd netchos
pip install -e .

Thanks. Now I get no errors but see no plots! I use Spyder and re-running the last line, heatmap(df), does not produce any plots in the plot pane, and no error in the console.

Yes this is because the code I sent you produces automatically the figure inside notebooks (Jupyter). Can you try with the following lines :
# 2D heatmap
fig = heatmap(df)
fig.show()

Tried, still no plot.

Can you try :
import plotly.io as pio
pio.renderers.default = 'browser'

fig = heatmap(df)
fig.show()

and if still doesn't works, you can try :
pio.renderers.default = 'svg'

Thank you. Seem like 'browser' fixed it (sorry, python beginner here).
I can use both circular and heatmap with the example data you have above and both graphs show up perfectly.
However when I skip random data creation and create 'df' by replacing 'data' with my own adjacency matrix (2D), and 'node_name' with my list of channels (where the data type is also a list of string), and keep everything else the same, heatmap works, but circular seems to get stuck in a loop or something and never ends; the console has to be killed every time I run circle.
Any idea what might be happening?

Yep, you probably have a lot of links and that's why it keeps running. One workaround consist in removing some links that are not important. So for example in your case non-significant links are set to nan (not a number) or you can use a threshold. A quick example :
# random functional connectivity array
fc = np.random.rand(100, 100)

# use a 95th percentile threshold
th = np.percentile(fc, 95)

# set to nan every non-significant links
fc[fc < th] = np.nan

# dataframe conversion + circular plot
....

Yes, that was the problem.
Thanks so much for your help and clear examples. Once I fully compute my connectivity data and organise them, would love to come back to this package and try to visualise them, perhaps using more detailed arguments...
But for now this would do the job.
Thanks again.


FIX: Fix docstring tester

In my doc updates I broke our docstring checker, this should fix it.

I am happy if CIs are happy

[Current coverage][cc-pull] is 52.26%

Merging [#3352][cc-pull] into [master][cc-base-branch] will decrease coverage by 34.21%

@@             master      #3352   diff @@
==========================================
  Files           335        335
  Lines         57322      57317       -5
  Methods           0          0
  Messages          0          0
  Branches       8715       8655      -60
==========================================
- Hits          49569      29956   -19613
- Misses         5131      25557   +20426
+ Partials       2622       1804     -818

[][cc-pull]

Powered by Codecov. Last updated by [b1ec497...84bf1fe][cc-compare]
[cc-base-branch]: https://codecov.io/gh/mne-tools/mne-python/branch/master?src=pr
[cc-compare]: https://codecov.io/gh/mne-tools/mne-python/compare/b1ec497be3f265d58f8a25c983d7a509aa8a16d2...84bf1fe8be8ea977cfda85bcb3bb4df3e7dcdfd8
[cc-pull]: https://codecov.io/gh/mne-tools/mne-python/pull/3352?src=pr


32 Display scheduled date/time for event in pop-up dialog

In an event dialog box, the event's date and time are displayed above the description. The date is formatted as "Wed, Jun 14, 2023" and the time is formatted however it was in Hendrix Today's Excel sheet (it is stored as a string).
Closes 32

I noticed that the Event.time string is very inconsistent and often includes date and location info as well. Sometimes this is relevant, in cases such as theater performances which have different times on different days, but there are also some cases where the time is "5pm" rather than "5:00 PM" and cases with no time data at all.
There are also frequent conflicts between the event date as stored in the Event.date field and the date as listed in the event's title and description.


[WIP] No more pagination

no pagination might be a problem without lazy loading though....

what.. why no pagination?

'cause it's really bad UX; say I'm looking for a particular project; how am I supposed to find it? Browse through the pages? Would much rather use CMD+F as would every sane human being.

Or a search function. Removing pagination will cause insane load times

billables and reports both don't even have pagination!

Reports do, billables not i remember now.

'cause it's really bad UX; say I'm looking for a particular project; how am I supposed to find it? Browse through the pages? Would much rather use CMD+F as would every sane human being.

Just add search?

Seems to be easier than

can't we just do lazy_loading on the numbers/donuts etc of projects?

@tarzan could we close this PR?

yup


subquery doesn't support order by

InfluxDB version: 1.3.1
When I run the subquery with order by, it doesn't work:
> select * from cpu order by time desc
name: cpu
time                host    region  value
----                ----    ------  -----
1502764231978282504 serverA us_west 0.6
1502763228833832379 serverA us_west 0.61
1502763224144379689 serverA us_west 0.64
>
> select * from (select * from cpu) order by time desc
name: cpu
time                host    region  value
----                ----    ------  -----
1502763224144379689 serverA us_west 0.64
1502763228833832379 serverA us_west 0.61
1502764231978282504 serverA us_west 0.6

The order of the query is inherited by any subqueries. While there is no mechanic in place to catch changing the ordering between a subquery and the query itself, it's not tested and unlikely to do anything useful and should probably be an error to switch the order by within a subquery.
This could be clarified in the documentation. Do you have a specific use case in mind where you needed to specify the ordering in a separate direction?

You are right, I really don't have this use case. I just find this coincidentally. Maybe InfluxDB return some error or nothing, instead of the result that is not as user expected.


[AutoRelease] t2-eventgrid-2023-11-11-95255(can only be merged by SDK owner)

https://github.com/Azure/sdk-release-request/issues/4760
Live test success
https://dev.azure.com/azure-sdk/internal/_build?definitionId=993
BuildTargetingString
azure-mgmt-eventgrid
Skip.CreateApiReview
true

issue link:https://github.com/Azure/sdk-release-request/issues/4760


az vmss update fails JSON serialization on windows

Describe the bug
I am unable to run az vmss update with JSON parameters in windows; the equivalent works fine in bash. Here is a script I was working on...
param ($resource_group, $vmss_name, $vault_cert_name)

# Get the latest certificate from Azure Key Vault
$cert=(az keyvault secret list-versions --vault-name=vault_name --name=$vault_cert_name | ConvertFrom-Json) | Sort-Object -Property attributes.expires | Select-Object -First 1

# Format the cert ID into a "secrets" format usable by the azure CLI
$secret=$(az vm secret format --secrets=$($cert.id))

# Update the VMSS with the new secrets for the SSL cert
az vmss update --resource-group=$resource_group --name=$vmss_name --query virtualMachineProfile.osProfile.secrets --set="virtualMachineProfile.osProfile.secrets=$secret"

Command Name
az vmss update
Errors:
Unable to build a model: Cannot deserialize as [VaultSecretGroup] an object of type <class 'str'>, DeserializationError: Cannot deserialize as [VaultSecretGroup] an object of type <class 'str'>
Traceback (most recent call last):
pip-install-3pnsb50r\msrest\msrest\serialization.py, ln 577, in body
pip-install-3pnsb50r\msrest\msrest\serialization.py, ln 1298, in _deserialize
pip-install-3pnsb50r\msrest\msrest\serialization.py, ln 1298, in _deserialize
pip-install-3pnsb50r\msrest\msrest\serialization.py, ln 1298, in _deserialize
...
msrest.exceptions.SerializationError: Unable to build a model: Cannot deserialize as [VaultSecretGroup] an object of type <class 'str'>, DeserializationError: Cannot deserialize as [VaultSecretGroup] an object of type <class 'str'>

To Reproduce:
Steps to reproduce the behavior. Note that argument values have been redacted, as they may contain sensitive information.

Setup an azure vault
Store a secret in the vault
Run the powershell script above

Expected Behavior
Update the secrets in the VMSS image
Environment Summary
Windows-10-10.0.18362-SP0
Python 3.6.6

azure-cli 2.1.0 *

Additional Context

hi @qwordy could you pls help to have a look, thanks.

@AgSync-Aaron Thanks for the feedback. I hope this reply finds you well, although we know it's been a lit long since you opened the issue.
Short Answer
az vmss update --resource-group=$resource_group --name=$vmss_name --add virtualMachineProfile.osProfile.secrets @path_to_your_secrets.json
Long Answer
What you were encountering is an issue most possibly caused by quoting in powershell, see here.
When using the generic update methods to update a complex property, I recommend to use json file to avoid any possible string parsing problems that come from command line tools.
See here to learn the different ways to pass arguments to the generic update methods of AzureCLI.

Hi @AgSync-Aaron does above solutions work for you?

Closed due to no response for long time.


Add OptFrameCacheType option function

@yuce if you see the TODO in PR https://github.com/pilosa/pilosa/pull/1351 on this line I've noted that we should probably support all frame options in https://github.com/pilosa/pilosa/issues/1345:
type FrameOptions struct {
    Type        string      `json:"type,omitempty"`
    CacheType   string      `json:"cacheType,omitempty"`
    CacheSize   uint32      `json:"cacheSize,omitempty"`
    Min         int64       `json:"min,omitempty"`
    Max         int64       `json:"max,omitempty"`
    TimeQuantum TimeQuantum `json:"timeQuantum,omitempty"`
}

I'm working on that in several steps. Waiting for review each time so I don't push up one huge PR.
It would be good for you to review that PR and provide any feedback on the direction it's going.

@travisturner Thanks for letting me know about the upcoming changes in that PR.

It's already possible to pass CacheTypeRanked, CacheTypeLRU, or CacheTypeDefault as a frame option. Closing.


Shared throughput collection via Mongo Shell

Trying to create a shared throughput collection using mongo shell.
Database successfully created via
db = db.getSiblingDB('mydb');
db.runCommand({customAction: "CreateDatabase", offerThroughput: 400});

Then I am trying to create a collection with a shared throughput by running
db.runCommand({customAction: "CreateCollection", collection: "sharedtp" })
Althouth the docs are claiming offerThroughput and shardKey are optional, I still get this error
{
        "_t" : "OKMongoResponse",
        "ok" : 0,
        "code" : 2,
        "errmsg" : "Shared throughput collection should have a partition key\r\nActivityId: d8dd5725-0000-0000-0000-000000000000, Microsoft.Azure.Documents.Common/<IP_ADDRESS>",
        "$err" : "Shared throughput collection should have a partition key\r\nActivityId: d8dd5725-0000-0000-0000-000000000000, Microsoft.Azure.Documents.Common/<IP_ADDRESS>"
}

How can I create a shared throughput collection through mongo shell?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: e4845c0d-223c-7ad5-bca7-92d2012ba122
Version Independent ID: 0580c849-744f-6a37-6b2e-d7fb47ffe8c2
Content: MongoDB extension commands to manage data stored in Azure Cosmos DB’s API for MongoDB
Content Source: articles/cosmos-db/mongodb-custom-commands.md
Service: cosmos-db
GitHub Login: @SnehaGunda
Microsoft Alias: sngun

@rgherta Thanks for the feedback. We are actively investigating and will get back to you soon.

@rgherta Partition Key is required. I will have the documentation updated accordingly. There might be an effort already underway to have this document updated but please let me work with the content owner.

@SnehaGunda The Examples section (link) details the cmd to use to create an unsharded collection but it is my understanding and as the customer has pointed out, a partition key is now required for all collections.

@Mike-Ubezzi-MSFT  You are correct, this was my main confusion. I did not specify a shard key hence the error. The documentation should detail the fact that Shard Keys are optional UNLESS you are using an unlimited storage Database. In this case, due to auto-scaling, all collections must have a shard key. I will close this issue but many newcomers might have the same problem.


mismatch of the synced properties between documentation and user profile application

At the page https://docs.microsoft.com/en-us/sharepoint/user-profile-sync#properties-that-are-synced-into-sharepoint-user-profiles you find a table with properties which are synced into SharePoint user profiles. When I look in the SharePoint Admin Center at the User Profile Application in SharePoint Online I find more attributes that are marked by the icon which indicates that these are synced from AAD. So which of both sources is the truth? Example "Display Order" or "Web Site".

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 54ff426d-6bba-75ea-970b-03f16fe191b5
Version Independent ID: 51db9ad2-9e46-4391-601a-062b640b00d2
Content: About user profile synchronization - SharePoint in Microsoft 365
Content Source: SharePoint/SharePointOnline/user-profile-sync.md
Service: sharepoint-online
GitHub Login: @SerdarSoysal
Microsoft Alias: serdars

Hello @LennartaeHoefer how are you?
It's true the icon hints you to believe that the property is being synced from Azure AD, however, and I checked an On-Premises SharePoint to compare, those two that you mention have also that icon but they do not sync from local AD.
That leads me to believe the icon is not entirely correct and we should follow this article.
Thank you

Dear @get-itips, to get it right: the article we are talking about has the correct attributes named but the icon is not 100% correctly displayed in the SharePoint Admin Center for some of the attributes. Right?

Dear @get-itips, to get it right: the article we are talking about has the correct attributes named but the icon is not 100% correctly displayed in the SharePoint Admin Center for some of the attributes. Right?

That is my guess, but I'll try to confirm, Thank you

I managed to pass this information to the SPO team.
@scanum


-format=json example in README does not work

The example in the README for using -format=json does not work. https://github.com/tsenart/vegeta#json-format
jq -ncM '{method: "GET", url: "http://goku", body: "Punch!" | @base64, header: {"Content-Type": ["text/plain"]}}' |
  vegeta attack -format=json -rate=100 | vegeta encode

The body is not correctly sent to the server. Vegeta should be un-base64-encoding the string and passing to the server. It appears that it is sending nothing to the server in the body.
It's also weird that the body needs to be base64 encoded. It would be nice if that extra ceremony were not necessary.

Please open a Bug Report


Code release consulting

Hi, this work is awesome. I notice some part of the code contains unreleased part like sparse-view synthesis. I would like to consult for the release time of the rest part. Thank you!

+1

Thanks! We are still working hard on cleaning the code, please stay tuned.

Thanks! We are still working hard on cleaning the code, please stay tuned.

will the code of all training part be released ?

@Drexubery will the code of all training part be released ? I read many papers, I think your work is really the best baseline and codebase for future work.

Hi, thanks for your comments!
We are currently working on a new project to train a dynamic video novel view synthesis model. We will release the complete training code once the project is complete.


ofl_body_text : updated rationale with OFL from GF's Unified Font Repo

I updated the link for the example in the rationale. Is this the only thing to change or there is another place where it should be copy-pasted?

that's good. Thanks!


💡[Feature]:  Maternal Health risk prediction using Machine learning

Is there an existing issue for this?

[X] I have searched the existing issues

Feature Description
Objective:
To develop a predictive model and a web application that can classify the risk level (high, medium, low) for pregnant women based on various health parameters. The project will involve data analysis, preprocessing, feature engineering, model training, evaluation, and deployment using a Streamlit web application.
Dataset:
The dataset obtained from Kaggle includes the following columns:
https://www.kaggle.com/datasets/pyuxbhatt/maternal-health-risk?select=Maternal+Health+Risk+Data+Set.csv
Age: Age of the pregnant woman
SystolicBP: Systolic blood pressure
DiastolicBP: Diastolic blood pressure
BS: Blood sugar level
BodyTemp: Body temperature
HeartRate: Heart rate
RiskLevel: Risk level classification (high risk, medium risk, low risk)
Steps Involved:

Data Analysis:

Exploratory Data Analysis (EDA):
Understand the distribution of each feature.
Identify any patterns or trends in the data.
Detect outliers and anomalies.
Visualize relationships between features and the target variable (RiskLevel).

Data Preprocessing:

Handling Missing Values:
Identify and handle missing data through imputation or removal.
Data Transformation:
Normalize or standardize numerical features.
Encode categorical variables if any.
Feature Selection:
Select relevant features that contribute most to the prediction.

Feature Engineering:

Creating New Features:
Derive new features that might help in improving the model's performance.
Feature Scaling:
Apply scaling techniques like Min-Max scaling or Standard scaling to bring all features to a similar range.
4)Model Training

Model Deployment

Integrate the trained model with the Streamlit app.
Use Case
The Maternal Health Risk Prediction model can be used by healthcare providers in prenatal care facilities to assess the risk levels of pregnant women during routine check-ups. By inputting patient data such as age, blood pressure, blood sugar levels, body temperature, and heart rate into the Streamlit web application, healthcare professionals can quickly determine whether a pregnant woman is at high, medium, or low risk of complications. This real-time risk assessment tool can guide doctors in making timely and informed decisions about additional tests, treatments, or interventions to ensure the well-being of both the mother and the baby.
Benefits
Implementing this predictive model and web application can significantly enhance prenatal care by providing early warnings of potential health risks. It enables proactive management of maternal health, reduces the likelihood of severe complications, and improves overall pregnancy outcomes. Healthcare providers can allocate resources more effectively, prioritize high-risk cases, and offer personalized care plans. Additionally, the ease of use and accessibility of the Streamlit app ensure that even facilities with limited technical infrastructure can benefit from advanced predictive analytics, ultimately leading to better maternal and fetal health.
Add ScreenShots
none
Priority
High
Record

[X] I have read the Contributing Guidelines
[X] I'm a GSSOC'24 contributor
[X] I want to work on this issue

@TAHIR0110 ?

@RajKhanke go ahead.


Generator refactoring

With this PR I am trying to merge back some features I need for nuxt-generate-cluster so I wont need to maintain my own lib/builder/generator.js. One major feature is still missing (being able to pass custom variables to the options.generate.routes method) but I will create a separate PR for that.
The list of changes are:

Split generator logic in to separate methods
Separate the business logic from the presentation layer: make the Generator class a silent library class that doesn't communicate with the user itself (eg removed debug call's from generator.js)
Added/moved hooks

generate:distRemoved has been added to log the Destination folder cleaned statement
generate:distCopied has been added to log the Static & build files copied statement
generate:routeFailed has been added for when an unhandled exception occurred during renderRoute
generate:routeCreated has been moved to the generateRoute method so we can include the path

To be able to log errors during page rendering they are kept separately until all hooks have been called

--edit--
The build fails due to an error unrelated to this PR. The test might still fail for Node v6 as I copied the reporting method to nuxt-generate which still uses string literals and arrow functions.

Tell me when it's good to merge @pimlie
Great work on this refacto BTW

Codecov Report

Merging #2205 into dev will increase coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##              dev    #2205      +/-   ##
==========================================
+ Coverage   98.04%   98.04%   +<.01%
==========================================
  Files          15       15
  Lines         972      974       +2
==========================================
+ Hits          953      955       +2
  Misses         19       19

Impacted Files
Coverage Δ

lib/builder/generator.js
100% <100%> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update db5ecee...8081705. Read the comment docs.

If all checks pass then this PR is ready :)
As I have been adding some functionality/complexity to nuxt-generate, I though it would be wise to add an integration test for the cli commands to make sure they still output the things we want. E.g. in a later stage we could also now easily add tests on cli options etc.

Hmm, not sure why the Windows tests fail for v8? I dont see what any of the changes from this PR has to do with the children.patch » Search a country test :confused:
Could it maybe just be a (timing?) quirk in the test itself? Maybe the waitFor 100ms sometimes isn't long enough? Its a bit difficult for me to test on Windows as I dont have a Windows development pc.

Great, that fix seems to work. So ready when you are :grin:

Great work @pimlie


add friendly conversion + strip spring boot arguments + print stacktr…

…ace when verbose main parameter + spring boot update

Coverage increased (+0.5%) to 91.667% when pulling 8449619c6d24904240ed05c4eeeddf17fab6441e on friendlyConversion into c272070c5c66e45600594ce0104ff3e9199fd537 on master.

Coverage increased (+0.5%) to 91.667% when pulling 9f17b9848864946db53e7d9142d26e3d0190474b on friendlyConversion into c272070c5c66e45600594ce0104ff3e9199fd537 on master.

Closing #4


Validate data-type and aggregate attribute values

There must a validation when using data-type and aggregate attributes. Right now, I can use aggregate="dummy" without seeing any warning/error.

Is this done?

pos ya que


Add resource detection to checkout service

Changes
Similar to #648 & #651 this adds more resources being detected to the checkout service.

Merge Requirements

[x] CHANGELOG.md updated to document new feature additions
[x] Appropriate documentation updates in the docs folder

Thanks for addition, looks good to me.
BTW I wonder what about the metricsprovider, will you add in the future?

Thanks for addition, looks good to me. BTW I wonder what about the metricsprovider, will you add in the future?

Haven't thought about it to be honest, I fixed that now :)

@svrnm just out of curiosity, from where the container.id comes from?
I've tested the PR and only got host.name, which has the initial part of the container id, and not the whole ID as you have in the screenshot.
BTW: Tested it out just with docker.

Same for me.
container.id is not visible, host.name is visible
When looking at other services like paymentservice, I can see  container.id just fine, so shouldn't be a problem in the docker environment.

Same here, Limitation rg cgroup v2. PR is in the making as well:
https://github.com/open-telemetry/opentelemetry-go/pull/3508

I think the changelog has a conflict from the cart service pr but ready to merge when resolved

@cartersocha I fix the conflicts. I think the pr is ready to merged.

https://giphy.com/gifs/luffy-zKBcZUw49Sxtm
@fatsheep9146


[UI EHHANCEMENT]: Implement theming using shadcn ui

Is your feature request related to a problem? Please describe.
The overall theme of the app is too random. The best solution would be using a fixed theme for the entire site.
Describe the solution you'd like
I would like to fix the site's look by implementing shadcn-ui, one of the best UI theming libraries for this purpose.
Additional context
Please assign me this issue under GSSOC'24

Not looking for this right now. Thankyou for raising an issue.


pr/issue

1.My pid : 488767
2.Moxie protocol TVL : 372,595,013 MOXIE (My TVL : 104.664K MOXIE)
3.reason :
I have a Moxy Prison frame, so I didn't even make 30 interactions an hour while watching this. We need an explanation as to why we are going to jail.

account is not flagged


Add information about numpy<2 problem  on macOS arm

References and relevant issues
Initially reported here https://github.com/napari/napari/pull/7146#pullrequestreview-2231557186. But may happen for any plugin that uses threads.
requires https://github.com/napari/docs/pull/475 for proper link to documentation
Description
Add a warning about NumPy macOS problem to the welcome widget to help people who meet segfaults.

Oh I see, this is not about #7146, it's about other plugins etc. I might tweak the wording. This is an old problem though (?), so I'm going to leave to 0.5.3.

Closes because of napari/napari#7178 is merged with BLAS solution


Add support for RGBWW LEDs.

Cannot control just the white LED's.

Do you want to set it via the UI or via the incoming msg.payload?

Msg.payload would be fantastic.
Get Outlook for Androidhttps://aka.ms/ghei36

From: danecreekphotography<EMAIL_ADDRESS>Sent: Tuesday, August 18, 2020 7:07:02 AM
To: danecreekphotography/node-red-contrib-wled2<EMAIL_ADDRESS>Cc: Andrew<EMAIL_ADDRESS>Author<EMAIL_ADDRESS>Subject: Re: [danecreekphotography/node-red-contrib-wled2] Add support for RGBWW LEDs. (#66)
Do you want to set it via the UI or via the incoming msg.payload?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com/danecreekphotography/node-red-contrib-wled2/issues/66#issuecomment-675499540, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AJRRS45FMPL7ZKTMK672SRLSBKDINANCNFSM4QC3X2QA.

So I think it will via the payload already. Set payload.color1 to an array of four values with the 4th value being the W component:
"color1": [0, 0, 0, 128]

Let me know if that does/doesn't work.

That worked! Thank you very much.

From: danecreekphotography<EMAIL_ADDRESS>Sent: Tuesday, August 18, 2020 10:55 AM
To: danecreekphotography/node-red-contrib-wled2<EMAIL_ADDRESS>Cc: Andrew<EMAIL_ADDRESS>Author<EMAIL_ADDRESS>Subject: Re: [danecreekphotography/node-red-contrib-wled2] Add support for RGBWW LEDs. (#66)
So I think it will work via the payload already. Set payload.color1 to an array of four values with the 4th value being the W component:
"color1": [0, 0, 0, 128]
Let me know if that does/doesn't work.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com/danecreekphotography/node-red-contrib-wled2/issues/66#issuecomment-675625828, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AJRRS447UDFNHZP5W2RSH33SBK6B3ANCNFSM4QC3X2QA.

Nice! Now I just have to figure out how to support it via the UI.

Not gonna happen via the UI, but at least it's supported via the payload.

Hello, I understand that this issue has been closed, but is there any way that I could get a quick explanation on how to modify the msg.payload with the WLED node? I tried using the change node (before and after the WLED node) to set payload.color1 to "color1": [0, 0, 0, 128], but I didn't find any success in getting my WW LEDs to turn on. Unfortunately the online guides for changing the payload don't seem to be clicking with me today so I'm hoping that someone can point me in the correct direction on getting my WW leds to automatically turn on/off. Thanks for everything, I really appreciate the work you've done on this node making life easier for people like me.

Honestly I would ignore my node entirely and just directly send the JSON that WLED expects to receive using the http request node. My wled2 node adds no value once you are sending payloads with JSON.
Documentation for the JSON API is at https://github.com/Aircoookie/WLED/wiki/JSON-API. If you have trouble getting it working the NodeRed forum is super helpful: https://discourse.nodered.org/

Okay that make sense. I appreciate the assistance.


Suggestion:  add .LICENSE help topic for cmdlet licensing information

PowerShell's self-documentation capability is great, but there's no room to add license information at the top of the file.  If you try that, the entire help system breaks, and Get-Help ignores everything.
For example, I tried to include it before my regular help and my code:
<#
Test-FeatureRequest.ps1
by John Doe<EMAIL_ADDRESS>
This program is free software: you can redistribute it and/or modify it under the terms of
the GNU General Public License as published by the Free Software Foundation, either version
3 of the License.  This program is distributed in the hope that it will be useful, but
WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR
A PARTICULAR PURPOSE.  See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program.
If not, see <https://www.gnu.org/licenses/gpl-3.0.en.html>.

.SYNOPSIS
foo

.DESCRIPTION
foo

.PARAMETER result
foo
#>
code goes here

That caused the entire comment block to be ignored:
PS C:\Users\jdoe> Get-Help .\Test-FeatureRequest.ps1
Test-FeatureRequest.ps1 [-result] <Object> [<CommonParameters>]

I tried a few variants unsuccessfully, like splitting the comment block into two.  There was no difference between me putting the license information first or second.
<#
[license text goes here -- I won't retype it every time]
#>
<#
.SYNOPSIS
foo

.DESCRIPTION
foo

.PARAMETER result
foo
#>

I tried inventing a .LICENSE keyword and using that, in hopes PowerShell would think it's from a future version of the language and quietly ignore it.  That, too, made PowerShell ignore the entire block.
<#
.LICENSE
[license text goes here]

.SYNOPSIS
foo

.DESCRIPTION
foo

.PARAMETER result
foo
#>

In the end, I had to stick my license information into the .NOTES section.  This works, but it has the side effect of showing the information, which is meant to be a file header, every time someone uses the Get-Help command on my script -- not the end of the world.
I feel the next iteration of PowerShell Core should have a new documentation item, .LICENSE.  Its primary purpose would be to "decorate" the top of a file.  However, if implemented, it should also be shown when Get-Help -Full is called, as many licenses also specify how the program can be shared with others;  this is similar to finding most apps' license information under Help > About.
I tried this on PowerShell and Windows PowerShell, with identical results:
Name                           Value
----                           -----
PSVersion                      6.0.2
PSEdition                      Core
GitCommitId                    v6.0.2
OS                             Microsoft Windows 10.0.17134
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
PSRemotingProtocolVersion      2.3
SerializationVersion           <IP_ADDRESS>
WSManStackVersion              3.0

Name                           Value
----                           -----
PSVersion                      5.1.17134.112
PSEdition                      Desktop
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
BuildVersion                   10.0.17134.112
CLRVersion                     4.0.30319.42000
WSManStackVersion              3.0
PSRemotingProtocolVersion      2.3
SerializationVersion           <IP_ADDRESS>

Why not put that text in a regular comment block (as opposed to a doc comment block) at the top of the file e.g.:
<#
Test-FeatureRequest.ps1
by John Doe<EMAIL_ADDRESS>
This program is free software: you can redistribute it and/or modify it under the terms of
the GNU General Public License as published by the Free Software Foundation, either version
3 of the License.  This program is distributed in the hope that it will be useful, but
WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR
A PARTICULAR PURPOSE.  See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program.
If not, see <https://www.gnu.org/licenses/gpl-3.0.en.html>.
#>

<#
.SYNOPSIS
foo

.DESCRIPTION
foo

.PARAMETER result
foo
#>
function One {}

<#
.SYNOPSIS
foo

.DESCRIPTION
foo

.PARAMETER result
foo
#>
function Two {}

I already tried putting it in a separate block at the top of the file, before the self-help, but that didn't work.
However, I tried your example, with a blank line between the two blocks, and that worked successfully.


Huge memory consumption when setting history variable to a big number

This is on Linux (Ubuntu 17.10), X11.
While running alacritty, changing the history variable in the configuration to a big number (say 1,000,000, 10,000,000 and above) results in a huge memory consumption (allocation?) instantly that gets the computer stuck very quickly.

Currently all lines in the history are immediately allocated. So if you specify a million lines, this means that a million lines are actually already stored in the history, even if it's empty.
Alacritty also stores all lines in memory, so a million lines would take some space.
So the primary way to counteract this issue, would be to dynamically allocate scrollback lines. This is tracked in https://github.com/jwilm/alacritty/issues/1236, so subscribing to that is probably the best way to stay updated right now.
For now, reducing the history size to a smaller number is the only way to work around this.

How about putting some hard coded limit in the meantime or even crash with an error? It will be better than letting a user's computer get stuck.

That's a fair point, however it depends on the User's system.
Maybe putting in a limit of 100k will do the job. Have you tested at which point this causes freezing issues?

I'm not sure if it was 1,000,000 or 10,000,00 but I don't really want to try again. 100,000 was ok i think.

I've opened a PR to set an upper limit to the scrollback history size:
https://github.com/jwilm/alacritty/pull/1551
This should resolve the issue, appreciate the feedback!


[CORE][Relay] Swap and remove compile_engine with te_compiler followup of #8775

This is a follow-up work for #8775 to address the pending comments to remove the compile_engine.
As this is an intermediate step towards completely moving away from the engine, we exposed the following legacy functions from the compile engine to ensure backwards compatibility atm:

_TECompilerJIT
_TECompilerLower
_make_LoweredOutput
_make_CCacheKey

Now, compile_engine.py has been replaced with te_compiler.py and deprecated funcs, such as LowerShapeFuncs kept private within the te_compiler.cc.
@jroesch @mbs-octoml @electriclilies @comaniac @junrushao1994

PTAL @mbs-octoml @jroesch @electriclilies

Mostly this looks good, just found a few bits to tidy

Thanks, just pushed the fixes.


Add, update, and delete last watched items

Add ability to add, update, and delete last watched items through api.

No need for this. Integrated https://www.themoviedb.org/


Wasteland Biome disable?

More of a question before upgrading - can the biome be disabled? I ask because sometimes mod biomes do mess with the generator of minecraft, so that the generation of a seed does no longer give the output that was expected, maybe this is not the case with your biome, but if it is, i need to disable it, or the world will look pretty ugly.

Yes, the biome can be disabledin the configs, in the 'Other' section.


Add Post Widget

Description: Added Post widget for displaying and summarizing posts.
Tests Performed: N/A
Trello Card Link: https://trello.com/c/5JHdsNWZ/5-create-a-basic-post-widget

looks good to me! we will likely need to revisit this widget in a later task due to Figma changes, but this will work well in the meantime. great work!


[Bug] Deserialization/Parsing consumes more input than permitted.

🐛 Bug Report
A number of datatypes implement the FromBytes, ToBytes, and Parser traits.
Some implementations are initially more permissive, for example:

Function::read_le accepts more inputs than MAX_INPUTS before it is checked later.
Plaintext::read_le/parse accepts more than MAX_ENTRIES before it is checked later.

A potential attack vector is that a malicious actor can send a node large messages that may waste a node's compute. Of course, this can be mitigated by fees.
Ideally, the deserializer/parser should stop as soon as the input exceeds the bounds set by NETWORK.
Some further questions:

what is the maximum number of verifying keys in a deployment?

Should be resolved with https://github.com/AleoNet/snarkVM/pull/1988


Cannot display more than 7 items

I've noticed that I can only display 7 items (in some cases). If the array is longer, only the last 7 elements are displayed.
Here's my test data
val data = arrayOf("me","him","them","hate","love","mine","yours", "ours","tada")
val aa = ArrayAdapter(this@EngineerHome, android.R.layout.simple_spinner_item, ins)
aa.setDropDownViewResource(android.R.layout.simple_spinner_dropdown_item)
in this case, the first two elements are ignored
can you reproduce this on your end @tiper ?

It only affects the dropdown mode. I've switched to dialog. Thank you for the awesome library.

Hi @Marcosmaliki when you open the spinner with a selection, the popup list will try to focus on the chosen element. Please check if scrolling the list will show the missing elements.


Destruction of Algorithm that is presently running violates Vulkan spec - (Presently theoretical)

Algorithm's destroy function destroys:

A Pipeline
A DescriptorPool
A PipelineLayout

without waiting for completion.
There are other objects destroyed which may also cause errors in similar ways.
vkDestroyPipeline: VUID-vkDestroyPipeline-pipeline-00765: All submitted commands that refer to pipeline must have completed execution
vkDestroyDescriptorPool: VUID-vkDestroyDescriptorPool-descriptorPool-00303: All submitted commands that refer to descriptorPool (via any allocated descriptor sets) must have completed execution
There is also the theoretical potential for: vkDestroyPipelineLayout: VUID-vkDestroyPipelineLayout-pipelineLayout-02004: pipelineLayout must not have been passed to any vkCmd* command for any command buffers that are still in the recording state when vkDestroyPipelineLayout is called

Ok it seems that this test:
TEST(TestAsyncOperations, TestManagerAsyncExecutionDestroyDescriptors)
{
    {
        uint32_t size = 10;

        std::string shader(R"(
            #version 450

            layout (local_size_x = 1) in;

            layout(set = 0, binding = 0) buffer b { float pb[]; };

            shared uint sharedTotal[1];

            void main() {
                uint index = gl_GlobalInvocationID.x;

                sharedTotal[0] = 0;

                for (int i = 0; i < 100000000; i++)
                {
                    atomicAdd(sharedTotal[0], 1);
                }

                pb[index] = sharedTotal[0];
            }
        )");

        std::vector<uint32_t> spirv = kp::Shader::compileSource(shader);

        std::vector<float> data(size, 0.0);
        std::vector<float> resultAsync(size, 100000000);

        kp::Manager mgr;

        std::shared_ptr<kp::TensorT<float>> tensorA = mgr.tensor(data);
        std::shared_ptr<kp::TensorT<float>> tensorB = mgr.tensor(data);

        std::shared_ptr<kp::Sequence> sq1 = mgr.sequence();
        std::shared_ptr<kp::Sequence> sq2 = mgr.sequence();

        sq1->eval<kp::OpTensorSyncLocal>({ tensorA, tensorB });

        std::shared_ptr<kp::Algorithm> algo1 = mgr.algorithm({ tensorA }, spirv);
        std::shared_ptr<kp::Algorithm> algo2 = mgr.algorithm({ tensorB }, spirv);

        // AMD Drivers in Windows may see an error in this line due to timeout.
        // In order to fix this, it requires a change on Windows registries.
        // More details on this can be found here: https://docs.substance3d.com/spdoc/gpu-drivers-crash-with-long-computations-128745489.html
        // Context on solution discussed in github: https://github.com/EthicalML/vulkan-kompute/issues/196#issuecomment-808866505
        sq1->evalAsync<kp::OpAlgoDispatch>(algo1);
        sq2->evalAsync<kp::OpAlgoDispatch>(algo2);
    }
}

Can recreate the following validation violations:
[2021-05-03 17:14:22.963] [debug] [Sequence.cpp:28] Kompute Sequence Destructor started
[2021-05-03 17:14:22.963] [debug] [Sequence.cpp:208] Kompute Sequence destroy called
[2021-05-03 17:14:22.963] [info] [Sequence.cpp:217] Freeing CommandBuffer
[2021-05-03 17:14:22.964] [debug] [Manager.cpp:25] [VALIDATION]: Validation - Validation Error: [ VUID-vkFreeCommandBuffers-pCommandBuffers-00047 ] Object 0: handle = 0x55f527277820, type = VK_OBJECT_TYPE_COMMAND_BUFFER; | MessageID = 0x1ab902fc | Attempt to free VkCommandBuffer 0x55f527277820[] which is in use. The Vulkan spec states: All elements of pCommandBuffers must not be in the pending state (https://vulkan.lunarg.com/doc/view/<IP_ADDRESS>/linux/1.2-extensions/vkspec.html#VUID-vkFreeCommandBuffers-pCommandBuffers-00047)
[2021-05-03 17:14:22.964] [debug] [Sequence.cpp:229] Kompute Sequence Freed CommandBuffer
[2021-05-03 17:14:22.964] [info] [Sequence.cpp:233] Destroying CommandPool
[2021-05-03 17:14:22.964] [debug] [Sequence.cpp:246] Kompute Sequence Destroyed CommandPool
[2021-05-03 17:14:22.964] [info] [Sequence.cpp:250] Kompute Sequence clearing operations buffer
[2021-05-03 17:14:22.964] [debug] [OpAlgoDispatch.cpp:18] Kompute OpAlgoDispatch destructor started
[2021-05-03 17:14:22.964] [debug] [Algorithm.cpp:33] Kompute Algorithm Destructor started
[2021-05-03 17:14:22.964] [debug] [Algorithm.cpp:84] Kompute Algorithm Destroying pipeline
[2021-05-03 17:14:22.964] [debug] [Manager.cpp:25] [VALIDATION]: Validation - Validation Error: [ VUID-vkDestroyPipeline-pipeline-00765 ] Object 0: handle = 0x55f527353328, type = VK_OBJECT_TYPE_DEVICE; | MessageID = 0x6bdce5fd | Cannot call vkDestroyPipeline on VkPipeline 0x1a000000001a[] that is currently in use by a command buffer. The Vulkan spec states: All submitted commands that refer to pipeline must have completed execution (https://vulkan.lunarg.com/doc/view/<IP_ADDRESS>/linux/1.2-extensions/vkspec.html#VUID-vkDestroyPipeline-pipeline-00765)
[2021-05-03 17:14:22.964] [debug] [Algorithm.cpp:96] Kompute Algorithm Destroying pipeline cache
[2021-05-03 17:14:22.965] [debug] [Algorithm.cpp:108] Kompute Algorithm Destroying pipeline layout
[2021-05-03 17:14:22.965] [debug] [Algorithm.cpp:120] Kompute Algorithm Destroying shader module
[2021-05-03 17:14:22.965] [debug] [Algorithm.cpp:146] Kompute Algorithm Destroying Descriptor Set Layout
[2021-05-03 17:14:22.965] [debug] [Algorithm.cpp:158] Kompute Algorithm Destroying Descriptor Pool
[2021-05-03 17:14:22.965] [debug] [Manager.cpp:25] [VALIDATION]: Validation - Validation Error: [ VUID-vkDestroyDescriptorPool-descriptorPool-00303 ] Object 0: handle = 0x55f527353328, type = VK_OBJECT_TYPE_DEVICE; | MessageID = 0x4dad1ae8 | Cannot call vkDestroyDescriptorPool on VkDescriptorPool 0x140000000014[] that is currently in use by a command buffer. The Vulkan spec states: All submitted commands that refer to descriptorPool (via any allocated descriptor sets) must have completed execution (https://vulkan.lunarg.com/doc/view/<IP_ADDRESS>/linux/1.2-extensions/vkspec.html#VUID-vkDestroyDescriptorPool-descriptorPool-00303)
[2021-05-03 17:14:22.965] [debug] [OpBase.hpp:28] Kompute OpBase destructor started
[2021-05-03 17:14:22.965] [debug] [Sequence.cpp:28] Kompute Sequence Destructor started
[2021-05-03 17:14:22.965] [debug] [Sequence.cpp:208] Kompute Sequence destroy called
[2021-05-03 17:14:22.966] [info] [Sequence.cpp:217] Freeing CommandBuffer
[2021-05-03 17:14:22.966] [debug] [Manager.cpp:25] [VALIDATION]: Validation - Validation Error: [ VUID-vkFreeCommandBuffers-pCommandBuffers-00047 ] Object 0: handle = 0x55f527275a30, type = VK_OBJECT_TYPE_COMMAND_BUFFER; | MessageID = 0x1ab902fc | Attempt to free VkCommandBuffer 0x55f527275a30[] which is in use. The Vulkan spec states: All elements of pCommandBuffers must not be in the pending state (https://vulkan.lunarg.com/doc/view/<IP_ADDRESS>/linux/1.2-extensions/vkspec.html#VUID-vkFreeCommandBuffers-pCommandBuffers-00047)
[2021-05-03 17:14:22.966] [debug] [Sequence.cpp:229] Kompute Sequence Freed CommandBuffer
[2021-05-03 17:14:22.966] [info] [Sequence.cpp:233] Destroying CommandPool
[2021-05-03 17:14:22.966] [debug] [Sequence.cpp:246] Kompute Sequence Destroyed CommandPool
[2021-05-03 17:14:22.966] [info] [Sequence.cpp:250] Kompute Sequence clearing operations buffer
[2021-05-03 17:14:22.966] [debug] [OpTensorSyncLocal.cpp:23] Kompute OpTensorSyncLocal destructor started
[2021-05-03 17:14:22.966] [debug] [OpBase.hpp:28] Kompute OpBase destructor started
[2021-05-03 17:14:22.966] [debug] [OpAlgoDispatch.cpp:18] Kompute OpAlgoDispatch destructor started
[2021-05-03 17:14:22.966] [debug] [Algorithm.cpp:33] Kompute Algorithm Destructor started
[2021-05-03 17:14:22.966] [debug] [Algorithm.cpp:84] Kompute Algorithm Destroying pipeline
[2021-05-03 17:14:22.967] [debug] [Manager.cpp:25] [VALIDATION]: Validation - Validation Error: [ VUID-vkDestroyPipeline-pipeline-00765 ] Object 0: handle = 0x55f527353328, type = VK_OBJECT_TYPE_DEVICE; | MessageID = 0x6bdce5fd | Cannot call vkDestroyPipeline on VkPipeline 0x130000000013[] that is currently in use by a command buffer. The Vulkan spec states: All submitted commands that refer to pipeline must have completed execution (https://vulkan.lunarg.com/doc/view/<IP_ADDRESS>/linux/1.2-extensions/vkspec.html#VUID-vkDestroyPipeline-pipeline-00765)
[2021-05-03 17:14:22.968] [debug] [Algorithm.cpp:96] Kompute Algorithm Destroying pipeline cache
Makefile:90: recipe for target 'mk_run_tests' failed
make: *** [mk_run_tests] Segmentation fault

Yes, VUID-vkDestroyPipelineLayout-pipelineLayout-02004 is only if someone left a sequence still recording


Fix some errors not being reported on console

Fixes #934
It seems there were two issues:

Conversion from CargoMSRVError to InstanceError
InstanceError not being shown on console by default, but instead only being logged with tracing::error!
(and usage of tracing::error! might also have been problematic because the InstanceError might be about setup of tracing having failed)

It looks like this can be solved by removing Context::output_format (since this information is also available from opts), and then performing the Context::try_from call later and handling errors like normal execution errors.
For now I have marked this as draft and added some notes as comments below. Please let me know if I overlooked something, and what you think.

Hey, Thanks a lot for the PR!
I think I gave you the wrong pointer previously in #934, because once the context is built, it is assumed we have a RustVersion:
https://github.com/foresterre/cargo-msrv/blob/main/src/context/verify.rs#L16
Looking in more detail at this PR, I think you're on the right track. An error is produced after all (courtesy of Context::try_from).
The first step which I feel is a good step in the right direction, is to change the tracing::error! to a println, here:
https://github.com/foresterre/cargo-msrv/pull/963/files/2bee052a328af82a191d8712edaa63a0bd5cb06e#diff-72a8660bff7d91b9c1e7985d434a773cbe58c206ff5f337bcef8fb3ee8bace75L24-R28
There is a small problem though 🥲!
This works for human output methods, e.g. running cargo msrv verify on a project with no rust-version gives us:
     Running `cargo-msrv.exe msrv verify --path ..\codecrafters-redis-rust`
Unable to find key 'package.rust-version' (or 'package.metadata.msrv') in '..\codecrafters-redis-rust\Cargo.toml'

However, we don't want this kind of output if the output-format is, for example json (or: not human), while it currently does:
     Running `cargo-msrv.exe msrv verify --path ..\codecrafters-redis-rust\ --output-format json`
Unable to find key 'package.rust-version' (or 'package.metadata.msrv') in '..\codecrafters-redis-rust\Cargo.toml'

The problem is that I've initialized the https://github.com/foresterre/cargo-msrv/blob/main/src/bin/cargo-msrv.rs#L56 before the reporter.
It makes to do so because the context also tells us, which output format ought to be used (i.e. which reporter handler to use) so it's a bit of a chicken and egg problem.

One way to solve it would be to manually print json in the event that the output format is json. Using opts is perhaps not ideal over context, but we have to be pragmatic here.
The property is opts.shared_opts.user_opts. I believe you already accessed it too 😄.
I made a small proof of concept, similar to yours:
fn main() {
    let opts = parse_opts(std::env::args_os);
    let output_format = opts.shared_opts.user_output_opts.output_format;

    std::process::exit(
        match _main(opts) {
            Ok((_guard, exit_code)) => exit_code,
            Err(err) => {
                // Can't use tracing::error! here, because it may not have been initialized!
                report_init_err(err, output_format);

                ExitCode::Failure
            }
        }
        .into(),
    );
}

fn report_init_err(err: InstanceError, output_format: OutputFormat) {
    match output_format {
        OutputFormat::Human => eprintln!("{}", err),
        OutputFormat::Minimal => eprintln!("error\n{}", err),
        OutputFormat::Json => {
            eprintln!(
                "{}",
                serde_json::json!({
                    "type": "init",
                    "failure": format!("{}", err),
                })
            );
        }
        OutputFormat::None => {}
    }
}

fn parse_opts<I: IntoIterator<Item = OsString>, F: FnOnce() -> I + Clone>(
    args: F,
) -> CargoMsrvOpts {
    let matches = CargoCli::parse_args(args());
    let opts = matches.to_cargo_msrv_cli().to_opts();

    opts
}

And then fn _main can simply be:
fn _main(opts: CargoMsrvOpts) -> Result<(Option<TracingGuard>, ExitCode), InstanceError> {

Compared to yours I tried to keep as much intact (but as you can see it's very similar in concept), and did some manual effort to get the right output format.
P.S. If you want to rename the vague InstanceError to SetupError feel free to do so.

Are you sure that the output-format handling is really a problem with my proposed changes here? Note that the println! I added is only as fallback in case something goes really wrong (I assume), something which should never happen.
Because I am calling Context::try_from a bit later now, and expected program errors such as the "Unable to find key ..." error are properly handled and respect the --output-format; they don't reach the println!.
In my opinion calling Context::try_from later, as done in this pull request, is the most important fix to this, since it makes sure then that all expected program errors are handled at the same place. The switch from tracing::error! to println! is nice to have (also for correctness), but not strictly needed to fix the original issue here.

Thanks for the explanation! That will do it indeed.
Then I only have the request to rename the different _main_n functions to more descriptive names :)

Then I only have the request to rename the different _main_n functions to more descriptive names :)

Do you have any suggestions? _main_n might not be that descriptive but at least it makes the order clear.
I guess _main_3 can be named run_program or similar. But for _main_1 and _main_2 I am not sure if a more descriptive name is more helpful (also since they are doing multiple things), and just based on the names it won't be obvious in which order they run. I can't think of very good names for them at the moment.

Then I only have the request to rename the different _main_n functions to more descriptive names :)

Do you have any suggestions? _main_n might not be that descriptive but at least it makes the order clear. I guess _main_3 can be named run_program or similar. But for _main_1 and _main_2 I am not sure if a more descriptive name is more helpful (also since they are doing multiple things), and just based on the names it won't be obvious in which order they run. I can't think of very good names for them at the moment.

Just some ideas:

setup_opts_and_tracing
setup_reporter
setup_context

or

run_with_opts_and_tracing
run_with_reporter
run_with_context

An alternative could be to let setup_context (_main_3) return the context and call run_app from main.

Thanks for the feedback and the suggestions! I have adjusted the code now, and force-pushed the changes. Please let me know if I should change anything else.


Squash msal files

msal/ has too many files to look through and figure out what is what.  This reduces the files to just a handful grouped by function.

There is only one structural change, which is removing a mock that wasn't actually used.
There is one internal/ change that simple moved a type around.

This allows a better view to accomplish goals like:

Removing *Providers, which was an outstanding item
Looking at the structure to change our API surface

Fixed up 2 lints, but 1 was: Error: SA5011: possible nil pointer dereference (staticcheck).  It didn't give any information on where this was or what file.


perlPackages.ArchiveLibarchive: 0.08 -> 0.09

Description of changes
https://github.com/uperl/Archive-Libarchive/compare/v0.08...v0.09
I also did some cleanup in the postPatch

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Result of nixpkgs-review pr 343128 run on aarch64-darwin 1

13 packages built:

lanraragi
perl536Packages.ArchiveLibarchive
perl536Packages.ArchiveLibarchive.devdoc
perl536Packages.ArchiveLibarchiveExtract
perl536Packages.ArchiveLibarchiveExtract.devdoc
perl536Packages.ArchiveLibarchivePeek
perl536Packages.ArchiveLibarchivePeek.devdoc
perl538Packages.ArchiveLibarchive
perl538Packages.ArchiveLibarchive.devdoc
perl538Packages.ArchiveLibarchiveExtract
perl538Packages.ArchiveLibarchiveExtract.devdoc
perl538Packages.ArchiveLibarchivePeek
perl538Packages.ArchiveLibarchivePeek.devdoc

Thanks :)


Update build number

force rebuild on linux

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.