Merging LibTorchSharp in TorchSharp

Current situation:
@interesaaat has done quite a bit of work on his fork of TorchSharp. He has also created a separate repo with C/C++ code, LibTorchSharp on which his fork TorchSharp depends.
I have started integrating TorchSharp with ML.NET with a torch scoring transformer (which takes pretrained models) and a torch trainer which trains a user defined model. This work depends on @interesaaat's fork.
Plan:
Given the scattered work it seems like we should decide on a high level plan on how to streamline our development efforts. Here is what I think our goal should be:

Have a single repository TorchSharp which builds both the native components and the C# code this essentially means merging @interesaaat's TorchSharp and LibtorchSharp repositories
Have an official build that produces a signed nuget and publishes it to some nuget feed
Either as part of this nuget, or as part of a separate nuget we should redistribute the libtorch dlls that we use and publish them to a nuget feed

Expected outcome:
For the developers:

Access to the latest work by @interesaaat.
Simplification of future work involving both native and C# code since they would be both in the same repository (like adding new APIs or working on auto-generating the API).
On the ML.NET side we would be able to take a dependency on the produced nugets.

For the users:
Anyone who wants to use TorchSharp directly can use the nugets from the feed instead of having to

download libtorch and adding it to the environment variables
compile libtorchsharp and adding it to the environment variables
downloading the TorchSharp nuget

Execution (WIP):
I am working on these three points with @interesaaat  in my fork. It's still a work in progress but I am using ML.NET's build infrastructure to build both native and C# code, to have the official build that pushes to a feed, and as a model for the redistribution of libtorch (see Microsoft.ML.Tensoflow.Redist). I am currently able to build everything and run the tests. I have discussed with CELA about the redistribution of libtorch dlls. I am still figuring out the best way to do that, but it is possible.
cc/ @interesaaat @migueldeicaza

@artidoro we have build a full ML pipeline in C# based on CNTK (which is now done with version 2.7) and are very interested in alternatives that work on Windows too. From our experience with CNTK I would suggest splitting the nuget packages in a few underlying parts (naming and technicalities aside):
TorchSharp.Dlls.CPU - CPU only native dlls (if 32-bit x86 are also possible would be great for us)
TorchSharp.Dlls.GPU - GPU native dlls (might be split based on underlying API use e.g. cudnn, etc.)
TorchSharp.PInvoke - Managed only and with no dependencies to dll packages (can be referenced by class libraries that then do not require the dll packages and hence can reduce disk footprint etc)
TorchSharp - meta package that can combine all necessary packages and provide ease of use of most users.
Why split it like this, our experience from CNTK where the GPU package is quite heavy is that you get quite a big disk footprint and "drag" in class libraries where the dlls are not needed. This does have pros and cons.
If there are any up-for-grabs items for this I would be interested in helping out and in giving feedback. :)

@artidoro I'm also interested in helping with up-for-grabs items should you guys need it :-).

I like the plan.
On the build piece, I do not know what ML.NET build infrastructure is, is it Azure DevOps?   I would love to have this on Azure DevOps, so anyone forking the repo, could trivially reuse and rebuild things on their own.
Azure-DevOps wise, this can both do builds, and publish NuGet packages.

Yes absolutely will keep using Azure DevOps.
@nietras thanks for the suggestion, I think it's a good idea to keep the packages separate.
As soon as we have the repositories merged, I am sure there will be a lot of up-for-grabs items!

Thanks @artidoro for putting this together!
Regarding the native dlls, I believe we should redistribute all the (stable) versions of libtorch available on pytorch.org. My guess is that once we are done with one, for the others it should be straightforward.

Regarding the native dlls, I believe we should redistribute all the (stable) versions of libtorch available on pytorch.org. My guess is that once we are done with one, for the others it should be straightforward.

By "redistribute" you mean actually shipping the DLLs / .sos? Is there a NuGet or such which we can depend on instead?

@markusweimer the plan is to make that nuget!
The plan is to package the libtorch DLLs that we need in a NuGet (similar to the Microsoft.ML.Tensorflow.Redist NuGet which has some Tensorflow DLLs). This would enable a user to take a dependency on the Libtorch.Redist NuGet (name is still to be finalized).
Regarding the version of the DLLs I am starting with 1.0.1 since that's what LibtTorchSharp depends on, but we are planning to upgrade to 1.1 once we have the repos actually merged.

@ebarsoumMS might be able to help? He mentioned some time ago that he was looking into producing libtorch NuGets.

libtorch dll is in C++, so the ABI might be compiler depended. Should we use something like: https://github.com/microsoft/vcpkg instead?

VCPKG is a source format, right? That means to build TorchSharp, one would have to have a working C++ toolchain around. I am not sure whether that is a reasonable requirement to impose.
I believe the ABI issue is only a Linux issue. To overcome it, we can follow the model of .NET itself or (well behaved) Python packages: Compile on ancient Linux OSs, and trust that the ABI compatibility of new Linux is covering us. For .NET, they use CentOS 7 (RedHat 7) as the build environment. And Python packages using the manylinux1 moniker use CentOS 5, for even greater binary compatibility.
Also, PyTorch itself must face (and solve) the same issue, as it is available as a PIP package which doesn't impose a recompile upon install.
Lastly, it absolutely makes sense to have a vcpkg as well, to make it easy to consume this work in C++ projects.

We can do it this way, but there is an ABI break in GCC after 5.5, this is why we have _GLIBCXX_USE_CXX11_ABI flag.


Prompt pattern matches where it should NOT

Describe the bug
Prompt pattern brn\-boll111\-.*?(\(.*?\))?[\>|\#] matches in show interfaces description on an interface that has brn-boll111-ap-02 Gi0 <LS:LIR> in its description.
General Information
OS:
RHEL7, python3.6
Netdev version:
asyncssh==1.18.0
netdev==0.9.3
Device OS:
Cisco IOS XE Software, Version 16.12.03s
C9300
Debug information
[2020-12-10 10:30:09,152 DEBUG    base.py:388 - _read_until_prompt_or_pattern() pn:MainProcess pid:31799   t:MainThread  ] Host <SNIP>: Reading pattern 'brn\-boll111\-.*?(\(.*?\))?[\>|\#]' or 'brn\-boll111\-.*?(\(.*?\))?[\>|\#]' was found: 'show interfaces description\r\nInterface                      Status         Protocol Description\r\nVl1                            up             up       not used <LS:C>\r\n<SNIP> <LS:CV>\r\nTe3/0/20                       down           down     Access-Port with 802.1x - lan <LS:CV>\r\nTe3/0/21                       down           down     Access-Port with 802.1x - lan <LS:CV>\r\nTe3/0/22                       down           down     Access-Port with 802.1x - lan <LS:CV>\r\nTe3/0/23                       up             up       Fabric Physical Link - brn-boll111-ap-02 Gi0 <LS:LIR>\r\nTe3/0/24                       up             up       Fabric Physical Link - brn-boll111-ap-02 Gi0 <LS:LIR>\r\nGi3/1/1                        admin down     down     not used <LS:C>\r\nGi3/1/2                        admin down     down     not used <LS:C>\r\nGi3/1/3                        admin down     down     not used <LS:C>\r\nGi3/1/4                        admin down     down     not used <LS:C>\r\nTe3'
Additional context
Any ideas for a quick fix until the problem can be solved permanently?

I think I found the issue. In the code only the first 12 characters from the base prompt are being used for the pattern. Our hostname has 18 characters. I fixed the issue as follows.
netdev/vendors/base.py - line 252:
base_prompt = re.escape(self._base_prompt[:12])
should be
base_prompt = re.escape(self._base_prompt)
Now it works without problems.


Remove obsolete hook messages.

Reviewed in https://github.com/google/orbit/pull/725

This probably did not really reach you, because of the annoying timezone team meeting problems, but we agreed if its just the same PR on master instead of release, we don't require LGTM approvals. For the future: maybe you name the PR: [Rebase onto master] "PR message"; and then just merge it as soon as the CI checks are passed.
Anyways, thanks for fixing this bug

Ah great, thanks for the info @antonrohr !  Will do next time, cheers!


update react version

Need to update the react version, super simple

Thank you, @Wjpayne!

weird! @pavelkukov this is the error I get
npm ERR! While resolving<EMAIL_ADDRESS>                                     npm ERR! Found<EMAIL_ADDRESS>                                                  npm ERR! node_modules/react                                                     npm ERR<EMAIL_ADDRESS>from the root project                                npm ERR!                                                                        npm ERR! Could not resolve dependency:                                          npm ERR! peer<EMAIL_ADDRESS>from<EMAIL_ADDRESS>                      npm ERR! node_modules/react-dial-knob                                           npm ERR!   react-dial-knob@"*" from the root project

maybe update all npm packages. This is too perfect for my project right now ha

actually I see the error now @pavelkukov

build (12.x)
The ssh-private-key argument is empty. Maybe the secret has not been configured, or you are using a wrong secret name in your workflow file.

Yeah, the pipeline is old. The secret is there but for some reason, it is not working. I don't have time to fix this in the upcoming two weeks. You can fork and publish a new module to unblock your work.
Thank you for raising a PR and investigating the build issue! Please let me know if you publish it as a new module. I will put a link here to your alternative.


"Context deadline exceeded" for responsive target

What did you do?
Tried to run Prometheus server in a docker container and scrape metrics from a node_exporter instance running in a separate but linked container (see environment section.)
Used a basic config which scrapes the linked container, via its container name dns address, every 10 seconds.
Noticed that this very basic target was showing as DOWN in the Prometheus dashboard targets view, with the error: "context deadline exceeded"
I attached to the running Prometheus container and timed a manual scrape of the target using time and wget, this worked as expected:
$ docker exec prometheus time wget -qO- http://node_exporter:9100/metrics
# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0.000302033
go_gc_duration_seconds{quantile="0.25"} 0.0005395150000000001
... (other metrics) ...
real	0m 0.01s
user	0m 0.00s
sys	0m 0.00s

Created a separate target group with scrape_interval and scrape_timeout both increased to 30s but targeted still at the same node_exporter instance. Scrapes for this group are successful but appear to take a long time (Last Scrape reported as > 36s ago which I assume means the scrape took upwards of 6 seconds from Prometheus' perspective.)

What did you expect to see?
Both target groups reporting as UP and scrapes completing quickly
What did you see instead? Under which circumstances?
DOWN "context deadline exceeded" for the 10s scrape group
Environment
Recreate my exact environment:
# Run monitoring (node_exporter + prometheus server)
docker run --name "node_exporter" -d prom/node-exporter

# Write config into file for mounting into prometheus container
mkdir -p /tmp/prometheus/
cat << EOF > /tmp/prometheus/prometheus.yml
---
global:
  scrape_interval: 10s
  scrape_timeout: 10s

scrape_configs:
  - job_name: 'defaults'
    static_configs:
      - targets: ['node_exporter:9100']
        labels:
          group: 'defaults'

  - job_name: 'increased_timeout'
    scrape_interval: 30s
    scrape_timeout: 30s
    static_configs:
      - targets: ['node_exporter:9100']
        labels:
          group: 'increased_timeout'
EOF

docker run --name "prometheus" \
  -d \
  -p 9090:9090 \
  --link node_exporter:node_exporter \
  -v /tmp/prometheus/prometheus.yml:/prometheus.yml \
  -v /tmp/prometheus/data:/data \
  prom/prometheus:v1.5.2 \
  -storage.local.path=/data \
  -config.file=/prometheus.yml

# Open http://localhost:9090 in your browser and look at the Targets section.

# Stop and remove containers:
docker stop node_exporter prometheus
docker rm node_exporter prometheus

Prometheus version:

$ docker run -ti prom/prometheus:v1.5.2 -version
prometheus, version 1.5.2 (branch: master, revision: bd1182d29f462c39544f94cc822830e1c64cf55b)
  build user:       root@1a01c5f68840
  build date:       20170210-16:23:28
  go version:       go1.7.5

Prometheus configuration file:

---
global:
  scrape_interval: 10s
  scrape_timeout: 10s

scrape_configs:
  - job_name: 'defaults'
    static_configs:
      - targets: ['node_exporter:9100']
        labels:
          group: 'defaults'

  - job_name: 'increased_timeout'
    scrape_interval: 30s
    scrape_timeout: 30s
    static_configs:
      - targets: ['node_exporter:9100']
        labels:
          group: 'increased_timeout'

Logs:

time="2017-03-01T11:53:25Z" level=info msg="Starting prometheus (version=1.5.2, branch=master, revision=bd1182d29f462c39544f94cc822830e1c64cf55b)" source="main.go:75"
time="2017-03-01T11:53:25Z" level=info msg="Build context (go=go1.7.5, user=root@1a01c5f68840, date=20170210-16:23:28)" source="main.go:76"
time="2017-03-01T11:53:25Z" level=info msg="Loading configuration file /prometheus.yml" source="main.go:248"
time="2017-03-01T11:53:26Z" level=info msg="Loading series map and head chunks..." source="storage.go:373"
time="2017-03-01T11:53:26Z" level=info msg="1434 series loaded." source="storage.go:378"
time="2017-03-01T11:53:26Z" level=info msg="Listening on :9090" source="web.go:259"
time="2017-03-01T11:53:26Z" level=info msg="Starting target manager..." source="targetmanager.go:61"
time="2017-03-01T11:58:26Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:612"
time="2017-03-01T11:58:26Z" level=info msg="Done checkpointing in-memory metrics and chunks in 107.457463ms." source="persistence.go:639"
time="2017-03-01T12:03:26Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:612"
time="2017-03-01T12:03:26Z" level=info msg="Done checkpointing in-memory metrics and chunks in 97.668263ms." source="persistence.go:639"
...

Nothing which seems to be relevant in the logs.
Hopefully I'm not doing anything obviously wrong!
Thanks

Hm, @alexellis mentioned that he had similar problems with Prometheus on Docker - he couldn't even scrape Prometheus itself at localhost. Alex, did you find a solution to that?
@AlexRudd I used your exact steps to reproduce your scenario and it works for me. I wonder if Docker broke something networking-related in recent versions. I'm still on Docker 1.12.1. This might be one of these cases where you want to pull out Wireshark or tcpdump and look at what's really happening on the network.

I was seeing the same error with the latest docker images for Prometheus. I thought it was because I was running it as a docker stack i.e. docker swarm service.

It does work for me with exactly the same Prometheus version mentioned here btw., so I suspect it's Docker-related. But since I can't reproduce, it would be cool if someone who sees the problem can dig into actual packet traces to see what's going on.

Ah, I didn't think to check my docker version!
I just tried my reproduction steps at home and it worked fine:
$ docker version
Client:
 Version:      1.13.1
 API version:  1.26
 Go version:   go1.7.5
 Git commit:   092cba3
 Built:        Wed Feb  8 08:47:51 2017
 OS/Arch:      darwin/amd64

Server:
 Version:      1.13.1
 API version:  1.26 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   092cba3
 Built:        Wed Feb  8 08:47:51 2017
 OS/Arch:      linux/amd64
 Experimental: true

I'll check what version I'm on at work tomorrow and do some digging through Docker's issues.

So at work I'm running:
$ docker version
Client:
 Version:      1.12.6
 API version:  1.24
 Go version:   go1.6.4
 Git commit:   78d1802
 Built:        Tue Jan 10 20:32:39 2017
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.6
 API version:  1.24
 Go version:   go1.6.4
 Git commit:   78d1802
 Built:        Tue Jan 10 20:32:39 2017
 OS/Arch:      linux/amd64

I upgraded to the latest Docker (now Docker CE) and still get the same "context deadline exceeded" error in Prometheus:
$ docker version
Client:
 Version:      17.03.0-ce
 API version:  1.26
 Go version:   go1.7.5
 Git commit:   3a232c8
 Built:        Tue Feb 28 08:01:32 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.03.0-ce
 API version:  1.26 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   3a232c8
 Built:        Tue Feb 28 08:01:32 2017
 OS/Arch:      linux/amd64
 Experimental: false

Then I'll have to refer to "But since I can't reproduce, it would be cool if someone who sees the problem can dig into actual packet traces to see what's going on." :) It seems it's either a Docker problem, or some weird interaction between how Go resolves things and the way Docker does DNS.

Any chance you might be hitting https://github.com/docker/docker/issues/19866? Seems a lot has changed regarding default bridge network and --link.

I get the same error message (context deadline exceeded) on Travis-CI and Docker 1.12.3. I don't use Prometheus, but run integration tests from a Java client against the Docker daemon. The error appears when creating a service and directly afterwards requesting the current tasks. Locally on Docker for Mac 17.03.0-ce-mac2 I cannot reproduce the error.
In other words: I would confirm @juliusv's assumption that this isn't a Prometheus issue - but maybe the Prometheus setup helps to reproduce the issue?

Yeah - I think it's safe to close this issue for now, as it's likely a Docker issue. I'd rather ask the Docker folks about it.

Happening in Centos 7.3 in AWS without using docker. This is a very basic setup as the OP reported, with nothing fancy going on, except that i am running everything native. Would be happy to provide any details that might help resolution.

Ran into a similar problem on Docker 17.03-17.05 ce
These versions of docker have a bug such that connecting the service/container to the host bridge network is harder than it should be.
Work-around:
$ docker network connect bridge <cid_of_prometheus>

Ran into the same issue. Amazon Linux 2017.03 no docker. Basic setup.

@juliusv
Happening in Ubuntu 16.04  in  CDS cloud without using docker. Noticed that "node_exporter" target was showing as DOWN , with the error: "context deadline exceeded".
   19 minutes ago      Up 19 minutes       <IP_ADDRESS>:9100->9100/tcp   prometheus_node-exporter_1
a38bf51d25c6        prom/prometheus      "/bin/prometheus -..."   20 minutes ago      Up 20 minutes       <IP_ADDRESS>:9090->9090/tcp   vibrant_brown

prometheus
State | Labels                     | Last Scrape  | Error
DOWN |  instance="<IP_ADDRESS>:9100" | 56.407s ago | context deadline exceeded

Scrape interval and timeout set to 30s but no change

I am experiencing the same issue.
CONTAINER ID        NAME                CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS c4120abe7244        prometheus          0.03%               21.57MiB / 15.58GiB   0.14%               44.7kB / 128kB      393kB / 0B          21
My docker has 15GB of limit, adding more memory could solve the issue?

The fix for me was editing the NetworkPolicy associated with the Pod prometheus was trying to scrape.

@weskinner what exactly did you edit in the NetworkPolicy associated with the Pod?   I am seeing the same issue right now, so any advise would help.

@rcsavage sorry I'm not remember the specifics.  I just know the NetworkPolicy that applied to that pod was preventing prometheus from accessing it from the monitoring namespace.  Adjusting the labels on the monitoring namespace and/or the selectors in the NP should fix it if that is your problem as well.

Increasing value of scrape_timeout works for me in prometheus.yml file.

Increasing value of scrape_timeout works for me in prometheus.yml file global section.

I was having this issue due to the firewall settings in AWS. The port in Security Group was not open.

I'm trying to scrape kong metrics but the aws security group port is open and i used a scrape_interval: 30s. Any suggestion? Thanks

In my case, the issue was caused by pod which was in error state due to OOM. I don't know why but after removing the failing pod from the node the Prometheus started to be able to scrape metrics.

Hi, I've had the same problem

I had the same problem and i solved it opening tcp ports 9100 and 9101 on AWS security groups.
I hope it can be helpful for you also.

Thanks a lot, it worked for me.

I was facing this issue due to max connections reached. I increased the max_connections parameter in database and then released some connections.
It was able to scrape again.

For future users: I ran into this problem with Docker 20.10.7 on Ubuntu Server 20.04.3. The exporter instance/container was up and I could see the data by going to the /metrics URL. I tried every which way changing scrape times with no luck.
I finally opened up my local firewall ports with sudo ufw allow <port> for the exporter instances/containers, like @RdL87 did on AWS, and solved the issue.

For future users: I ran into this problem with Docker 20.10.7 on Ubuntu Server 20.04.3. The exporter instance/container was up and I could see the data by going to the /metrics URL. I tried every which way changing scrape times with no luck.
I finally opened up my local firewall ports with sudo ufw allow <port> for the exporter instances/containers, like @RdL87 did on AWS, and solved the issue.

Hi. I have the same problem .. I can follow the link, but it is not transmitted in the grafan. Can you tell us in more detail what you did?

Hi. I have the same problem .. I can follow the link, but it is not transmitted in the grafan. Can you tell us in more detail what you did?

My problem was that my grafana container wasn't able to reach the container with the exporter, despite bridging them. The exporter was doing its job, I was able to verify that by going to the metrics URL manually. So after reading @RdL87's post, I tried opening up the port on Ubuntu's firewall with that command it did the trick.

Is there is a way to monitor whenever the state goes to DOWN state.

mine is a network problem, I installed my node on VirtualBox.  I deactivated the NAT,  I used NAT to supply my internet inside my master, I turn it off and I change to a bridge connection. and everything works smoothly. I hope it will never happen again.


Add logging for pipeline

This PR adds some visual logging to watch frames go through pipelines.
Here's an example from sample 04. This sample uses two Pipelines with a SequentialMergePipeline, so the hardcoded one runs first. I think this makes it easy to see which frames are being passed along.
Note that I drop Audio frames. I might change this to show the first and then an ellipse or something until there's something that's not an audio frame.
10 2024-03-27 18:46:04,228   source -> TextFrame: "My friend the LLM is going to tell a joke about llamas" -> AzureTTSService
10 2024-03-27 18:46:04,229     AzureTTSService -> TextFrame: "My friend the LLM is going to tell a joke about llamas" -> AzureTTSService
10 2024-03-27 18:46:04,229   source -> EndPipeFrame -> AzureTTSService
10 2024-03-27 18:46:04,229     AzureTTSService -> EndPipeFrame -> AzureTTSService
10 2024-03-27 18:46:04,229       AzureTTSService -> TTSStartFrame -> sink
20 2024-03-27 18:46:04,229 🎬 Starting frame consumer thread
20 2024-03-27 18:46:04,229 Running azure tts
10 2024-03-27 18:46:04,230   source -> LLMMessagesQueueFrame -> ElevenLabsTTSService
10 2024-03-27 18:46:04,230     ElevenLabsTTSService -> LLMMessagesQueueFrame -> AzureLLMService
10 2024-03-27 18:46:04,230       AzureLLMService -> LLMResponseStartFrame -> ElevenLabsTTSService
10 2024-03-27 18:46:04,230         ElevenLabsTTSService -> LLMResponseStartFrame -> sink
10 2024-03-27 18:46:04,230 Generating chat via openai: [{"content": "tell the user a joke about llamas", "role": "system", "name": "system"}]
20 2024-03-27 18:46:04,863 === OpenAI LLM TTFB: 0.6326563358306885
20 2024-03-27 18:46:05,412 Got azure tts result
20 2024-03-27 18:46:05,412 Returning result
10 2024-03-27 18:46:05,412       AzureTTSService -> TTSEndFrame -> sink
10 2024-03-27 18:46:05,412       AzureTTSService -> TextFrame: "My friend the LLM is going to tell a joke about llamas" -> sink
10 2024-03-27 18:46:05,412       AzureTTSService -> EndPipeFrame -> sink
10 2024-03-27 18:46:05,827       AzureLLMService -> TextFrame: "Why" -> ElevenLabsTTSService
10 2024-03-27 18:46:05,828       AzureLLMService -> TextFrame: " don" -> ElevenLabsTTSService
10 2024-03-27 18:46:05,829       AzureLLMService -> TextFrame: "'t" -> ElevenLabsTTSService
10 2024-03-27 18:46:05,829       AzureLLMService -> TextFrame: " ll" -> ElevenLabsTTSService
10 2024-03-27 18:46:05,829       AzureLLMService -> TextFrame: "amas" -> ElevenLabsTTSService
10 2024-03-27 18:46:05,829       AzureLLMService -> TextFrame: " ever" -> ElevenLabsTTSService
10 2024-03-27 18:46:05,830       AzureLLMService -> TextFrame: " get" -> ElevenLabsTTSService
10 2024-03-27 18:46:05,830       AzureLLMService -> TextFrame: " lost" -> ElevenLabsTTSService
10 2024-03-27 18:46:05,830       AzureLLMService -> TextFrame: "?

" -> ElevenLabsTTSService
10 2024-03-27 18:46:05,830         ElevenLabsTTSService -> TTSStartFrame -> sink
10 2024-03-27 18:46:06,515         ElevenLabsTTSService -> TTSEndFrame -> sink
10 2024-03-27 18:46:06,515         ElevenLabsTTSService -> TextFrame: "Why don't llamas ever get lost?

" -> sink
10 2024-03-27 18:46:06,516       AzureLLMService -> TextFrame: "Because" -> ElevenLabsTTSService
10 2024-03-27 18:46:06,516       AzureLLMService -> TextFrame: " they" -> ElevenLabsTTSService
10 2024-03-27 18:46:06,516       AzureLLMService -> TextFrame: " always" -> ElevenLabsTTSService
10 2024-03-27 18:46:06,517       AzureLLMService -> TextFrame: " take" -> ElevenLabsTTSService
10 2024-03-27 18:46:06,517       AzureLLMService -> TextFrame: " the" -> ElevenLabsTTSService
10 2024-03-27 18:46:06,518       AzureLLMService -> TextFrame: " llama" -> ElevenLabsTTSService
10 2024-03-27 18:46:06,518       AzureLLMService -> TextFrame: "-n" -> ElevenLabsTTSService
10 2024-03-27 18:46:06,518       AzureLLMService -> TextFrame: "ated" -> ElevenLabsTTSService
10 2024-03-27 18:46:06,519       AzureLLMService -> TextFrame: " route" -> ElevenLabsTTSService
10 2024-03-27 18:46:06,519       AzureLLMService -> TextFrame: "!" -> ElevenLabsTTSService
10 2024-03-27 18:46:06,519         ElevenLabsTTSService -> TTSStartFrame -> sink
10 2024-03-27 18:46:07,194         ElevenLabsTTSService -> TTSEndFrame -> sink
10 2024-03-27 18:46:07,194         ElevenLabsTTSService -> TextFrame: "Because they always take the llama-nated route!" -> sink
10 2024-03-27 18:46:07,197       AzureLLMService -> LLMResponseEndFrame -> ElevenLabsTTSService
10 2024-03-27 18:46:07,198         ElevenLabsTTSService -> LLMResponseEndFrame -> sink
10 2024-03-27 18:46:07,198   source -> EndPipeFrame -> ElevenLabsTTSService
10 2024-03-27 18:46:07,198     ElevenLabsTTSService -> EndPipeFrame -> AzureLLMService
10 2024-03-27 18:46:07,198       AzureLLMService -> EndPipeFrame -> ElevenLabsTTSService
10 2024-03-27 18:46:07,198         ElevenLabsTTSService -> EndPipeFrame -> sink
20 2024-03-27 18:46:13,816 Stopping frame consumer thread

To be clear, is this logging frames as they are "between" services?
AzureLLMService -> TextFrame: "Why" -> ElevenLabsTTSService

This log line means "the AzureLLMService just emitted a TextFrame with the contents 'Why', and that frame is about to go into the ElevenLabsTTSService", right?

I really like the approach here, but one other thought for readability's sake: In addition to the "depth" approach, Could we assign a color to each service? Maybe we only emit the escape codes if we're connected to a tty or something. But I'm envisioning the frames themselves stay as white text, but we build an array of colors that correspond to the 'depth', which I think would make it even easier to scan the logs when you're debugging.

You're correct that each line is Frame Processor Yields -> Frame -> Frame Processor consumes.
The color thing is a cool idea -- I think I'll merge this and then maybe play with that in a separate PR. If we used a consistent color for each Frame Processor, maybe that could be more readable. I'll need to make sure that we can detect that the logger destination supports ANSI escape codes, too.


[pre-ll] Impl Display + Error for remaining error types

This addresses #506 for the pre-ll version, by filling the gaps on missing Error and Display implementations.
In the event that this procedure is also required in the latest versions, that can be arranged as well in a separate PR.

Oops, sorry 'bout that. It ought to be fixed now.

bors r+

On Dec 7, 2017, at 06:25, Eduardo Pinho<EMAIL_ADDRESS>wrote:
Oops, sorry 'bout that. It ought to be fixed now.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

Ok, I think I forgot to stage the fix. :disappointed_relieved:
Hopefully I got it right this time.

bors r+

On Dec 7, 2017, at 08:24, Eduardo Pinho<EMAIL_ADDRESS>wrote:
Ok, I think I forgot to stage the fix. 😥
Hopefully I got it right this time.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

bors r+

OSX failure from CI: https://travis-ci.org/gfx-rs/gfx/jobs/313020822#L2806

bors r+

bors r=kvark

Bors r+

On Dec 30, 2017, at 11:58, bors[bot]<EMAIL_ADDRESS>wrote:
Merge conflict
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

bors r+


Blazor Server formatters require 'unsafe eval'

Right now in Blazor SS with Blazor-ApexCharts, we need to enable unsafe eval in NGINX otherwise the pages with charts crash. Is it planned to also allow .NET Formatters for SS? Or is it possible to use CSP hash for the formatter?
We tried with a hash, but we are getting this error when opening a page with a chart: Refused to evaluate a string as JavaScript because 'unsafe-eval' is not an allowed source of script in the following Content Security Policy directive: "script-src 'self' 'sha256-[HASH]'".
Is it because in the published app the formatter script is saved differently from the source code and the hashes don't actually match?
Originally posted by @twojnarowski in https://github.com/apexcharts/Blazor-ApexCharts/issues/37#issuecomment-1932264816

As I have written, for now the only way we manage to run the chart with tooltips on data points is to have unsafe eval in CSP enabled. But there is no way that I now to check if actually the hash we generate from the tooltip formatter written in source is the same as the JS code that ends up in the running application. The formatter also gets only the code part, without <script> tags, so I don't see a way to use nonce in CSP. I think that keeping the unsafe eval for the whole Blazor application just because of only these formatters is not very safe idea...

In blazor server we cannot run the .net formatters because the format function in apexchart.js is synchronously, while we need async in order to call  blazor server.  In order to set a javascript formatter we need to run javascript eval function  to create the formatter function on the javascript side.
Currently we don't have any other solution to this problem.
I'm not familiar with CSP hash, could you explain how that could help.

Using CSP with hash means that the only JS code that can be evaluated at runtime is the code which hash matches the hash specified in the CSP.

Looking at the code and I believe we should be able to support custom tooltips with out any eval of javascript.
Would that help?

Using CSP with hash means that the only JS code that can be evaluated at runtime is the code which hash matches the hash specified in the CSP. That way if we hash this code:
function(value, opts) { if (value === undefined) {return '0';} return  value.toFixed(3)}

then only this piece of JS code can be run in the app. But it needs to be exactly the same when generating a hash and then in the deployed code - same whitespace, same capitalization, same everything for it to work. It protects from any kind of possible XSS attacks in the application.

I get that part, question is how would it work in the component?

We have upgraded the Blazor-ApexCharts nuget to 2.3.2 and currently we have this CSP:
add_header Content-Security-Policy "default-src 'self' http: https: data: blog: 'unsafe-inline'; script-src 'self'; always;";

And this is our custom formatter:
Tooltip = new Tooltip
            {
                Y = new TooltipY
                {
                    Formatter = @"function(value, opts) { if (value === undefined) {return '0';} return  value.toFixed(3)}"
                },
                HideEmptySeries = false,
            },

And still when the chart loads it producess this error:
EvalError: Refused to evaluate a string as JavaScript because 'unsafe-eval' is not an allowed source of script in the following Content Security Policy directive: "script-src 'self'".

    at eval (<anonymous>)
    at Object.<anonymous> (https://example.com/_content/Blazor-ApexCharts/js/blazor-apexcharts.js:413:34)
    at JSON.parse (<anonymous>)
    at Object.parseOptions (https://example.com/_content/Blazor-ApexCharts/js/blazor-apexcharts.js:408:21)
    at Object.renderChart (https://example.com/_content/Blazor-ApexCharts/js/blazor-apexcharts.js:202:28)
    at https://example.com/_framework/blazor.server.js:1:3244
    at new Promise (<anonymous>)
    at y.beginInvokeJSFromDotNet (https://example.com/_framework/blazor.server.js:1:3201)
    at Yt._invokeClientMethod (https://example.com/_framework/blazor.server.js:1:61041)
    at Yt._processIncomingData (https://example.com/_framework/blazor.server.js:1:58516)
   at Microsoft.JSInterop.JSRuntime.InvokeAsync[TValue](Int64 targetInstanceId, String identifier, Object[] args)
   at Microsoft.JSInterop.JSObjectReferenceExtensions.InvokeVoidAsync(IJSObjectReference jsObjectReference, String identifier, Object[] args)
   at ApexCharts.ApexChart`1.InvokeVoidJsAsync(String identifier, Object[] args)
   at ApexCharts.ApexChart`1.RenderChartAsync()
   at ApexCharts.ApexChart`1.OnAfterRenderAsync(Boolean firstRender)
   at Microsoft.AspNetCore.Components.RenderTree.Renderer.GetErrorHandledTask(Task taskToHandle, ComponentState owningComponentState)

Is there anything we are doing wrong right now?

Ok, thanks! That worked. We have added <ApexPointTooltip> inside <ApexChart> and that allowed us to create the custom tooltip that we need.
So to summarize, if someone stumbles upon this issue: There is no need to define Tooltip in ApexChartsOptions with a JS formatter, and keeping the CSP mentioned above works when just defining a custom tooltip with the ApexPointTooltip component.
P.S. the tooltip seems to be cached in Firefox, so we needed to clear all data to actually see the new one working.

Thanks


Options cannot be toggled

The radio buttons on the options page cannot be toggled or untoggled.

i have this problem as well


shell v <IP_ADDRESS> exits on every exception

After every mistake in expression the app exits and I have to re-run it, reopen db, do query again etc...
It's quite annoying.

This was addressed in #1013
There is a fix in master, however, it has not been released as a package yet.


Error on OpBitField{S,U}Extract to GLSL

Hi,
OpBitFieldSExtract and OpBitFieldUExtract are handled incorrectly in GLSL translation. They treated as a opcode with 4 parameters, wich is incorrect, both only take 3 parameters. This results in either a crash or invalid output.

Fixed on master.


Formatting is unusable

The formatting capability of this extension does not work well in a lot of situations. Is there any chance these issues are going to be addressed?

I don't think you'll have much luck here, this extension appears to be abandoned.
There is a fork here (GitHub) that could turn out to be promising.

I tried the above mentioned fork and although it works i'm notice serious slugishness in VSCode after install. Disabling the extension returns VS Code to smooth working order. There is no Issues tab for that repo so I don't know how reliable it will be for resolving issues like this.


Problem updating empty index with id in sort key [ Achieved by #150 ]

Hi,
I ran into a problem updating an empty index when a value in the PK of the index is not defined when creating the entity.
Here is my entity definition:
export const ProductPrefix = 'PRD'
@Entity({
  name: 'Product',
  primaryKey: {
    partitionKey: `ID#{{id}}`,
    sortKey: ProductPrefix,
  },
  indexes: {
    GSI1: {
      partitionKey: `${EventPrefix}#{{eventId}}`,
      sortKey: `${ProductPrefix}#{{id}}`,
      type: INDEX_TYPE.GSI,
    }
  }
})
export class ProductEntity extends BaseEntity {
  @Attribute({
    default: (entity: BaseEntity) => entity.id || ulid()
  })
  id: string

  @Attribute()
  eventId: string

  @Attribute()
  title: string
  ...
}

When I create an item without eventId which is listed in GSI1 PK, my record looks like this:
{
 "PK": "ID#01FZWVQ4JD5M2H3GSCP8JJZ48E",
 "SK": "PRD",
 "createdAt": "2022-04-05T12:38:14.128Z",
 "__en": "Product",
 "updatedAt": "2022-04-05T12:38:14.128Z",
 "id": "01FZWVQ4JD5M2H3GSCP8JJZ48E",
 "title": "Lorem ipsum dolor sit amet"
}

There is no GSI1 defined in the record, which is okay because I didn't provide eventId needed for GSI1PK. The problem is when I want to update the index and assign the Product entity to Event. This index needs two values to be updated properly. It's id and eventId. When I do update the record with these values I get InvalidPrimaryKeyAttributesUpdateError error:
Primary key attribute and non primary key attributes can not be
    updated together. When updating attributes that references a primary key, it can not have
    additional non key attributes.
Affected Primary key attributes: {"PK":"ID#01FZWQS0PHHSMHQYJ7M4WNTT3T"}
Tried to update non-key attributes: ["eventId"]

I understand that I can't update the primary key and index at the same time, but I need to use this id in the index without updating the primary key. If I do not specify the id in the update there is just GSI1PK added to the record "GSI1PK": "EVT#01FZWFCDFHEPH3CM87YV61SN4T" wihout sort key.
My current solution is to set eventId to NONE during creation and change this value on update. This works, but it's not ideal. Is there any other way to solve this?
Thanks 🙏

Adding NONE to the transformation temporarily solved my problem.
export class ProductEntity extends BaseEntity {
  @Attribute({
    default: (entity: BaseEntity) => entity.id || ulid()
  })
  id: string

  @Attribute({
    default: () => NoneValue,
  })
  @TransformFromDynamo(({value}: {value: string}) => value === NoneValue ? null : value)
  @TransformToDynamo(({value}: {value: string}) => !value ? NoneValue : value)
  eventId: string

  @Attribute()
  title: string
  ...
}

Hey @romcok
This is a known limitation of TypeDORM.
However, in your case, this should get resolved when https://github.com/typedorm/typedorm/issues/150 is implemented.
It will better analyze the update values and if the id by which we are trying to update is the same as the one being passed in as the new value, TypeDORM will make the improvised decision to allow updating attribute values.
Keep an eye out for the next release.
But the workaround you have mentioned above seems fine for now.

I am keeping this open to tracking the issue caused by not being able to identify updating attributes.

Oh, great. Just subscribed to #150. Thanks 👌


Song crashes with events

Describe your bug here.
If I remove the events, it doesn't crash. If I load the song without events, then add the events.json and make the chart editor load the events, that also doesn't crash but it crashes when I start the song. It didn't do this before so I have no idea why it does this now.

Command Prompt/Terminal logs (if existing)
No response
Are you modding a build from source or with Lua?
Lua
What is your build target?
Windows
Did you edit anything in this build? If so, mention or summarize your changes.
no

Nvm I just used onStep to trigger the events and it doesn't crash anymore. The events were change character and play animation


perlPackages.MHonArc: fix build

Motivation for this change
fix build problem
builder for '/nix/store/dkj8rc5zrh2x8scl1q9ra44x7b0k6z0v-perl-MHonArc-2.6.18.drv' failed to produce output path '/nix/store/kjyzhqxlnsv6vmfrygr7c472zhrlbdnr-perl-MHonArc-2.6.18-devdoc'

@GrahamcOfBorg build perlPackages.MHonArc

Success on x86_64-linux (full log)
Attempted: perlPackages.MHonArc
Partial log (click to expand)
post-installation fixup
moving /nix/store/5xvqhd5m36m4ichgk37dpk2a92knj3jb-perl-MHonArc-2.6.18/doc to /nix/store/5xvqhd5m36m4ichgk37dpk2a92knj3jb-perl-MHonArc-2.6.18/share/doc
shrinking RPATHs of ELF executables and libraries in /nix/store/5xvqhd5m36m4ichgk37dpk2a92knj3jb-perl-MHonArc-2.6.18
gzipping man pages under /nix/store/5xvqhd5m36m4ichgk37dpk2a92knj3jb-perl-MHonArc-2.6.18/share/man/
strip is /nix/store/4qvrxzxa535y8304mk195x50b6p9607d-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/5xvqhd5m36m4ichgk37dpk2a92knj3jb-perl-MHonArc-2.6.18/lib  /nix/store/5xvqhd5m36m4ichgk37dpk2a92knj3jb-perl-MHonArc-2.6.18/bin
patching script interpreter paths in /nix/store/5xvqhd5m36m4ichgk37dpk2a92knj3jb-perl-MHonArc-2.6.18
checking for references to /build in /nix/store/5xvqhd5m36m4ichgk37dpk2a92knj3jb-perl-MHonArc-2.6.18...
strip is /nix/store/4qvrxzxa535y8304mk195x50b6p9607d-binutils-2.30/bin/strip
/nix/store/5xvqhd5m36m4ichgk37dpk2a92knj3jb-perl-MHonArc-2.6.18

Success on aarch64-linux (full log)
Attempted: perlPackages.MHonArc
Partial log (click to expand)
post-installation fixup
moving /nix/store/8q8wspaxb9ik4fa9gdmh4drcd0g89k5m-perl-MHonArc-2.6.18/doc to /nix/store/8q8wspaxb9ik4fa9gdmh4drcd0g89k5m-perl-MHonArc-2.6.18/share/doc
shrinking RPATHs of ELF executables and libraries in /nix/store/8q8wspaxb9ik4fa9gdmh4drcd0g89k5m-perl-MHonArc-2.6.18
gzipping man pages under /nix/store/8q8wspaxb9ik4fa9gdmh4drcd0g89k5m-perl-MHonArc-2.6.18/share/man/
strip is /nix/store/0pjsgkxz0rp5baycq5sp2s72lrr5q9sg-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/8q8wspaxb9ik4fa9gdmh4drcd0g89k5m-perl-MHonArc-2.6.18/lib  /nix/store/8q8wspaxb9ik4fa9gdmh4drcd0g89k5m-perl-MHonArc-2.6.18/bin
patching script interpreter paths in /nix/store/8q8wspaxb9ik4fa9gdmh4drcd0g89k5m-perl-MHonArc-2.6.18
checking for references to /build in /nix/store/8q8wspaxb9ik4fa9gdmh4drcd0g89k5m-perl-MHonArc-2.6.18...
strip is /nix/store/0pjsgkxz0rp5baycq5sp2s72lrr5q9sg-binutils-2.30/bin/strip
/nix/store/8q8wspaxb9ik4fa9gdmh4drcd0g89k5m-perl-MHonArc-2.6.18


Upgrade to react-hot-loader 3 from react-transform

We should not be required react-hot-loader in the production, maybe we can separate to .prod.js and .dev.js, or just a NODE_ENV check?

I think that AppContainer and the entry patch amount to a noop when NODE_ENV is production by default.

@tyscorp oh, indeed. lol
But it will still saved some code for not require('react-hot-loader'), though not many.

We will revisit this PR after react-hot-loader v3 release!

@tyscorp I've implemented this functionality based on this PR in my project and it works in development env. But when running in production (npm run build && npm run start) I get the following:
bundle.js:1 Uncaught ReferenceError: AppContainer is not defined
Does this work for you?

What is the difference between react-hot-loader 3 and react-transform?

@tyscorp can you please rebase this PR

Closing, as react-transform has been deprecated


DISCARD_STREAM_TIMEOUT_MILLIS should be configurable

I'm using https://github.com/amitshekhariitbhu/Fast-Android-Networking(which uses okhttp) to prefetch some data.
Prefetch consumes the stream by calling close() but because of DISCARD_STREAM_TIMEOUT_MILLIS larger data is not downloaded(same issue with slower network and smaller data).

Consider reading the stream fully by calling body().source().skip(Long.MAX_VALUE).

I have this problem with a client using reftrofit2 which uses okhttp (version 3.8.0). When closing the ResponseBody (or actually an InputStream retrieved from it) an exception is raised server side. Setting HttpCodec.DISCARD_STREAM_TIMEOUT_MILLIS to 300 (instead of 100 from the distribution) solves the problem. This suggests that it takes more time than the allowed 100ms to clean up the socket and close it or return it to the pool, where 300ms suffices.
However, doing the skip as suggested (and handle the EOFException that gets raised), with the standard value of 100 for HttpCodec.DISCARD_STREAM_TIMEOUT_MILLIS also works.


Callback 'getattr' is called with non-existing path

I am using fuse-jna library to create a custom FS. When debugging my code, the callback readdir was called firstly when cd or ls a folder. It works fine. In my readdir implementation, some files and folder in  that dir path were found, for example,
/js
/css
/images

However the callback getattr were called many times for known(in above list) and unknown path(at least they were not returned by the callback readdir).
For example, I did see below path in getattr callback,
/.git  on both Linux and Mac OSX
/objects on Linux
/._. on MacOSX
/.DS_Store on Mac OSX

Do you have any idea why callback getattr were called with above non-existing path? They are annoying and might impact the performance.

I believe these are accesses done by the operating system which probes for "hidden" stuff that is used for some operating system specific functionality, e.g. caches, meta-data, trash-can, ...
I saw similar accesses in my project which uses fuse-jna and there I ignored those that I encountered, see IGNORED_DIRS

@centic9 Thanks for your explanation. Your project JGitFS really inspired me a lot when I am writing my project.

Yes, those are indeed just probes by your operating system, file manager, and whatever you have running on your machine. They do that to every filesystem, and they all have to handle that.


Allow autocompletion callback for whole command

First of, the existing API for auto completion is incredible, thank you so much for that :heart:. While implementing a command with a parameter that can be auto-filled either with values based on the current user or filled with values based on a member supplied in an extra parameter, I thought it would be great if I could access this member parameter in the auto complete function.
As far as I understand the discord documentation, in the auto complete event, everything is sent, not just the current/focused parameter. It would be great if the framework could take advantage of that maybe with an autocomplete function for the whole command instead of functions for each parameter.

How would an example command with an autocomplete function for the whole command look like? Having trouble imagining

I think the function would take three parameters: the Context, the partial input from all command parameters sent from discord (probably based on CommandData) and a selection for the currently requested/focused parameter that should be autocompleted (e.g. an enum).
The function would then process the data and return the data for the focused parameter. Since this function would need to return different completions based on the requested parameter, it would probably need to return an enum  with an inner value as well.
Example:
#[autocomplete = "complete_my_command"]
pub async fn my_command(ctx: Context<'_>, parameter1: i32, parameter2: String) -> Result<(), _> {
    // do something
}

enum CompleteSelection {
    Parameter1,
    Parameter2,
}

enum CompleteOutput {
     Parameter1(Vec<i32>), // basically either a Stream, Iterator. Just like the current output for a single parameter
     Parameter2(Vec<String>),
}

// the selection can be determined by the framework using the `focused` bool in the auto complete request from discord
async fn complete_my_command(ctx: Context<'_>, input: Vec<CommandData>, selection: CompleteSelection) -> CompleteOutput {
    match selection {
        // complete for the selected field optionally using the provided data.
    }
}

This seems very complex which I was afraid of. Maybe this use case is specialized enough that a manual autocomplete implementation is acceptable?
#[autocomplete = "complete_my_command"]
pub async fn my_command(ctx: Context<'_>, parameter1: i32, parameter2: String) -> Result<(), _> {
    // do something
}

async fn complete_my_command(ctx: Context<'_>, interaction: &serenity::AutocompleteInteraction) -> Result<(), Error> {
    // Find selected field in interaction.data.options

    // Send interaction.create_autocomplete_response()
}

Yes, I think that would work for me. I agree, that my proposal could get quite complex and hard to maintain.

I realized our two ideas are problematic since you still need to declare for each parameter if they are autocomplete (Discord needs to know this).
Directly afterwards I realized that poise's API already has all the capability you need since you can access the autocomplete interaction in the callback:
async fn foo_autocomplete(ctx: Context<'_>, partial: &str) -> Vec<String> {
    let interaction = match ctx {
        Context::Application(poise::ApplicationContext {
            interaction: poise::ApplicationCommandOrAutocompleteInteraction::Autocomplete(x),
            ..
        }) => x,
        _ => unreachable!("non-autocomplete interaction in autocomplete callback"),
    };

    // Find user param
    let user = interaction.data.options.iter().find(|o| o.name == "param_name").unwrap();
    // Extract user data
    let user = match user.resolved.unwrap() {
        serenity::CommandDataOptionValue::User(x, _) => x,
        _ => unreachable!("non-user value in user parameter"),
    }

    // Return autocomplete choices
}

Ah, didn't think of that, great!


test: initial tests for config loader

Codecov Report

Merging #45 into master will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master      #45   +/-   ##
=========================================
  Coverage     81.01%   81.01%
  Complexity      283      283
=========================================
  Files            57       57
  Lines          1085     1085
  Branches         72       72
=========================================
  Hits            879      879
  Misses          174      174
  Partials         32       32

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c7e9ac7...5cc4a18. Read the comment docs.


when conditional update is not met it's visible with REQUEST_ERROR metric

When conditional update is not met, a REQUEST_ERROR metric increases. For example:
feed_operations_rate{api="documentV1", deploymentType="stable",  operation="UPDATE", pod="vespa-stateless-container-0", service="vespa-stateless-container-monitoring", status="REQUEST_ERROR", tenant_id="default-tenant", vespaVersion="7.524.43", vespa_service="vespa_container"}
This is confusing, as it sounds like some error.
Should probably be changed to a different and more related label.
Also, not directly related to this ticket - it would be VERY useful if the feeding request would get back in the response what was the actual result of the update. For example if no update took place due to a not met condition (or if partial update had no impact due to a missing document).

I tested this myself using https://docs.vespa.ai/en/operations/monitoring.html and observed the same in the prometheus graphs. I think we should add the expected behaviour to https://docs.vespa.ai/en/reference/metrics.html
CC @gjoranv the metrics guru

We should not increase REQUEST_ERROR on condition-not-met.

This issue has been fixed in PR #23115, which will be included in Vespa version >= 8.2.17


Fix typo in comparing-direct2d-and-gdi.md

@lindexi : Thanks for your contribution! The author(s) have been notified to review your proposed change.


Add support for package.json5

Similar to https://github.com/mysticatea/npm-run-all/pull/266
Notes
Removed read-package-json-fast
read-package-json-fast doesn't support json5, and since it is an official NPM tool, it is probably not going to.
Thankfully, everything the package does that we care about is fairly simple.
Mostly, it uses the json-parse-even-better-errors and npm-normalize-package-bin packages.
I am now using these packages directly instead of read-package-json-fast.
None of the things read-package-json-fast does impact npm-run-all2.
Consistency
The return value of the readPackageJson function is still a promise even though the function is now entirely sync.
We could use fs/promise, but nothing else does yet.
This was all for consistency, but let me know if you would like to change it.
Tests
Tests are the same as npm-run-all so I will paste my note from there:
I have tried my best at writing tests for this change.
I don't think it is perfectly in line with the style of your other tests, but this was the simplest way I could see to test it.
If you have other ideas for tests, feel free to give guidance or change my tests when merging.

json5 isn't supported by the package.json ecosystem though?

You're right, NPM itself does not support JSON5. Other tools will also fail with package.json5.
However, we're not forcing anyone to use JSON5, just using it if it's there.
And since PNPM supports it, there are projects out there which do use it.
It's fairly likely at this point that it breaks something and someone has to use package.json, but why does that something have to be npm-run-all?
Is it just that it's not widely enough supported yet and you don't want the maintenance overhead?
Is there any path forward where you think npm-run-all should reconsider supporting package.json5?

I’m mostly worried that someone will then come and request that we support package.yml as well and then package.toml etc

Actually, PNPM supports package.yaml as well. Not sure which is more popular.
Bun supports comments in package.json last I checked as well, technically breaking the format.
All this just to say, I think at least some of the community is dissatisfied with this part of NPM right now, and certain projects will fail with npm-run-all.
In my opinion, the biggest issue is not being able to add comments to scripts (in case you need to document an obscure flag or command you're using to get around some issue), which is why I say it's a great fit.
I understand not wanting to support a ton of new formats, but maybe I can rewrite it in a way that would at least make new formats easier to support, and also make json5 an optional dependency?

First off thanks for the all the work on this. Going to mull over this if this is a good idea.
In general, secondary ecosystems like yarn and pnpm messing around with adding subtle breaking changes to established node/npm ecosystem design assumptions has been a gigantic time sink for maintainers and consumers with extremely dubious benefits.
Changing a load bearing assumption like package.json parsing as json seems like it is within this territory: jsonc/json5 have had years to establish themselves but have, in general, failed to still gain mass adoption, the argument going: by the time you add comments to json, you might as well use a more robust serialization language like yaml/toml. Also, Crockford says no. That said they have been picking up more pace lately (it seems).
Some questions I can answer for myself with research, but anyone feel free to chime in:

Comments in package.json: These can't published to npm, so what tool is accommodating them and stripping them?
For tools that do support jsonc/json5, are they using the correct file extension? e.g. package.jsonc/package.json5?
If we were to support package.jsonc/5, what is the full set of additional file extensions that people assume we support (yml/toml)?
Whats the status of run scripts in pnpm? Are they supporting a pnpm native approach to npm-run-all2 at this point? At which case, would that serve the needs of those interesting in package.json5 comments.


Run black linter on generated code

Either as an optional flag or just by default, run black on the generated client. It is the most popular auto-formatter and so can't hurt for us to use it by default.

Resolved by #18


Exit with usage message when nothing to do

The program would crash when run without a match argument.
Show the usage message instead and exit cleanly.

Thanks!


MINOR: Small javadoc/code cleanups in connect api and transforms

Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

Thanks for the reviews!


pkg/operators: Add combiner

Adding combiner operator
This PR introduces a new operator named "combiner". Only for TypeArray datasources, this operator will be in charge of merging outcomes from all targets into a single array on the client side.
Implementation details

The combiner operator will activate itself only if:

Gadget is run against more than one single target. For instance, it'll never get activated for ig used as standalone (without gadgetctl).
There is at least one TypeArray datasources. NOTE: From now on, this documentation will assume that the datasources are TypeArray.

The combiner operator requires the datasource to be annotated with FetchIntervalAnnotation, which is expected to be time.Duration type and is used as following:

If FetchIntervalAnnotation is 0, the combiner operator will emit a packet once it has received packets from all the targets (A maximum waiting time of 5 seconds has be hardcoded (Make it configurable?).
Instead, if FetchIntervalAnnotation is greater than 0, the combiner operator will emit a packet once the timer is triggered, no matter it has received data from all targets or not.
The combiner operator is not currently using the FetchCountAnnotation and it relies on the server to close the datasource once done (Still to be implemented cc @flyth).

The combiner operator is configured to have the highest priority on the client side. It is because, during the PreStart phase, the combiner operator will modify the datasources to force other operators on client side to subscribe to the datasource created by the combiner operator (From now on, called "combined-datasource") and not to the original ones coming from the server. The idea with this modification is that the combiner operator will subscribe to the original datasources and receive packets from each target separately, while it will emit one single packet on the combined-datasource only once it has received packets from all targets or timeout has been reached (This can be configured using FetchIntervalAnnotation as described above). Doing so, the rest of operators running on the server side will receive a packet where data from all nodes has already been combined, so they can sort, print and do any operation on the aggregated data arrays.

Looks great! I will need to play around with it a bit more.
The requested feature is being used slightly different to what I imagined initially (although it never got implemented, this was intended to be used as means to tell the server that the client is not interested in a particular datasource (or even fields) to save network bandwidth), maybe we should make it rather similar to the "unreferencing" we do for fields as well. But we can address that in a follow-up PR with more discussion.

Yeah. TBH, I used the requested feature because it was already there and wasn't used, but "requested" doesn't seem to be the proper name. Yeah, probably "unreferencing" or "enabling" would make clearer. Given that it's more about naming and not the feature itself, I also suggest managing in another PR.

I managed most comments. Ready for another round of reviews.


test-branch/execute-job/quoted-authorization/ee392292-4e9e-4069-804d-8d5764b7a130

PR for testing de109095e2146eb9438911fb6a0e8a0256f0bfb5

/canonical/self-hosted-runners/run-workflows de109095e2146eb9438911fb6a0e8a0256f0bfb5


Fix sysv files

Description
This PR fixes the sysv script under /etc/init.d/wazuh-indexer.
Issues Resolved
solves #340
Check List

[x] New functionality includes testing.

[x] All tests pass

[ ] New functionality has been documented.

[ ] New functionality has javadoc added

[ ] Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
[x] Commits are signed per the DCO using --signoff
[ ] Commit changes are listed out in CHANGELOG.md file (See: Changelog)
[ ] Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

After installing packages generated with the changes of this PR, the service commands operate as expected:
[root@alma9 ~]# curl -k -u admin:admin https://localhost:9200
{
  "name" : "node-1",
  "cluster_name" : "wazuh-cluster",
  "cluster_uuid" : "CQgc5AqSRzSg9HyWQd-u3g",
  "version" : {
    "number" : "7.10.2",
    "build_type" : "rpm",
    "build_hash" : "bf42358a96448dcf95cdde3cc82300267743a9e0",
    "build_date" : "2024-08-01T19:11:30.394888472Z",
    "build_snapshot" : false,
    "lucene_version" : "9.10.0",
    "minimum_wire_compatibility_version" : "7.10.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}
[root@alma9 ~]# service status wazuh-indexer
The service command supports only basic LSB actions (start, stop, restart, try-restart, reload, reload-or-restart, try-reload-or-restart, force-reload, status, condrestart). For other actions, please try to use systemctl.
[root@alma9 ~]# service wazuh-indexer status
● wazuh-indexer.service - wazuh-indexer
     Loaded: loaded (/usr/lib/systemd/system/wazuh-indexer.service; enabled; preset: disabled)
    Drop-In: /run/systemd/system/service.d
             └─zzz-lxc-service.conf
     Active: active (running) since Fri 2024-08-02 12:33:32 UTC; 34s ago
       Docs: https://documentation.wazuh.com
   Main PID: 6906 (java)
      Tasks: 123 (limit: 200975)
     Memory: 1.4G
        CPU: 33.254s
     CGroup: /system.slice/wazuh-indexer.service
             └─6906 /usr/share/wazuh-indexer/jdk/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 -Dopensearch.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -XX:+ShowCodeDetailsInExceptionMessages -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.security.manager=allow -Djava.locale.providers=SPI,COMPAT -Xms1g -Xmx1g -XX:+UseG1GC -XX:G1ReservePercent=25 -XX:InitiatingHeapOccupancyPercent=30 -Djava.io.tmpdir=/var/log/wazuh-indexer/tmp -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/lib/wazuh-indexer -XX:ErrorFile=/var/log/wazuh-indexer/hs_err_pid%p.log "-Xlog:gc*,gc+age=trace,safepoint:file=/var/log/wazuh-indexer/gc.log:utctime,pid,tags:filecount=32,filesize=64m" -Djava.security.manager=allow -Djava.util.concurrent.ForkJoinPool.common.threadFactory=org.opensearch.secure_sm.SecuredForkJoinWorkerThreadFactory -Dclk.tck=100 -Djdk.attach.allowAttachSelf=true -Djava.security.policy=file:///etc/wazuh-indexer/opensearch-performance-analyzer/opensearch_security.policy --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED -XX:MaxDirectMemorySize=536870912 -Dopensearch.path.home=/usr/share/wazuh-indexer -Dopensearch.path.conf=/etc/wazuh-indexer -Dopensearch.distribution.type=rpm -Dopensearch.bundled_jdk=true -cp "/usr/share/wazuh-indexer/lib/*" org.opensearch.bootstrap.OpenSearch -p /run/wazuh-indexer/wazuh-indexer.pid --quiet

Aug 02 12:33:25 alma9 systemd-entrypoint[6906]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/wazuh-indexer/lib/opensearch-2.13.0.jar)
Aug 02 12:33:25 alma9 systemd-entrypoint[6906]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
Aug 02 12:33:25 alma9 systemd-entrypoint[6906]: WARNING: System::setSecurityManager will be removed in a future release
Aug 02 12:33:26 alma9 systemd-entrypoint[6906]: Aug 02, 2024 12:33:26 PM sun.util.locale.provider.LocaleProviderAdapter <clinit>
Aug 02 12:33:26 alma9 systemd-entrypoint[6906]: WARNING: COMPAT locale provider will be removed in a future release
Aug 02 12:33:26 alma9 systemd-entrypoint[6906]: WARNING: A terminally deprecated method in java.lang.System has been called
Aug 02 12:33:26 alma9 systemd-entrypoint[6906]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/wazuh-indexer/lib/opensearch-2.13.0.jar)
Aug 02 12:33:26 alma9 systemd-entrypoint[6906]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
Aug 02 12:33:26 alma9 systemd-entrypoint[6906]: WARNING: System::setSecurityManager will be removed in a future release
Aug 02 12:33:32 alma9 systemd[1]: Started wazuh-indexer.
[root@alma9 ~]# service wazuh-indexer stop
Stopping wazuh-indexer (via systemctl):                    [  OK  ]
[root@alma9 ~]# service wazuh-indexer start
Starting wazuh-indexer (via systemctl):                    [  OK  ]
[root@alma9 ~]# ls -lsah /etc/init.d/wazuh-indexer
4.0K -rwxr-xr-x 1 root root 3.6K Aug  2 12:27 /etc/init.d/wazuh-indexer


How can i use this plug-in in commercial templates?

SlickNav uses the MIT license so you are free to use it for commercial use. If you are asking for implementation help, you will need to provide more information.

SlickNav uses the MIT license so you are free to use it for commercial use. If you are asking for implementation help, you will need to provide more information.


feature: add support for easy now deployments

Linked issue
Fixes #60

:tada: This PR is included in version 0.3.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


feat: (dnd5e 3.x) New Item Card format for the Container?

The container has a different kind of information within it.
Consider designing an item card the befits the container and provides the most relevant information in an easily digestible way.
Discuss with the community.

Unless a specific plan comes to mind later, we'll just let this go. Inline containers provide the most critical info, anyway.


build-osuosl-aix71-ppc64-1/2 missing libcurl.so ?

https://ci.adoptopenjdk.net/computer/build-osuosl-aix71-ppc64-1/
https://ci.adoptopenjdk.net/view/Failing Builds/job/build-scripts/job/jobs/job/jdk8u/job/jdk8u-aix-ppc64-hotspot/888/console
14:59:04  Caused by: hudson.plugins.git.GitException: Command "git fetch --tags --progress -- https://github.com/AdoptOpenJDK/openjdk-build.git +refs/heads/*:refs/remotes/origin/*" returned status code 141:
14:59:04  stdout:
14:59:04  stderr: exec(): 0509-036 Cannot load program git-remote-https because of the following errors:
14:59:04  	0509-150   Dependent module libcurl.a(libcurl.so.4) could not be loaded.
14:59:04  	0509-022 Cannot load module libcurl.a(libcurl.so.4).
14:59:04  	0509-026 System error: A file or directory in the path name does not exist.
14:59:04
14:59:04  	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2450)
14:59:04  	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2051)
14:59:04  	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$500(CliGitAPIImpl.java:84)
14:59:04  	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:573)

Using a passing test as a guidline, i ran
git init /home/jenkins/workspace/build-scripts/jobs/jdk8u/jdk8u-aix-ppc64-hotspot
cd /home/jenkins/workspace/build-scripts/jobs/jdk8u/jdk8u-aix-ppc64-hotspot
git fetch --tags --progress -- https://github.com/AdoptOpenJDK/openjdk-build.git +refs/heads/*:refs/remotes/origin/

remote: Enumerating objects: 10394, done.
Receiving objectsremote: Total 10394 (delta 0), reused 0 (delta 0), pack-reused 10394
Receiving objects: 100% (10394/10394), 4.12 MiB | 3.43 MiB/s, done.
Resolving deltas: 100% (6845/6845), done.
From https://github.com/AdoptOpenJDK/openjdk-build
 * [new branch]      andrew-m-leonard-patch-1 -> origin/andrew-m-leonard-patch-1
 * [new branch]      awx.sxa.org.uk -> origin/awx.sxa.org.uk
 * [new branch]      compileOnly -> origin/compileOnly
 * [new branch]      douph1-1834 -> origin/douph1-1834
....

And wasnt able to recreate the error. As mentioned in slack, I do suspect that ansible updating each package is related to this as the same error has appeared on build-osuosl-aix71-ppc64-1

As mentioned in slack, I do suspect that ansible updating each package is related to this

Perhaps not, because I ran the ansible playbook on build-osuosl-aix71-ppc64-2 aswell, which hit the same error, but
https://ci.adoptopenjdk.net/job/Test_openjdk17_j9_sanity.system_ppc64_aix/7/console has been running on it fine

In regards to libcurl.so, it doesnt seem to be on the machine. I just updated git on the machine manually, in an attempt to circumvent the ansible error that I posted above

@Haroon-Khel
build-osuosl-aix71-ppc64-2 is now failing the same:
https://ci.adoptopenjdk.net/view/Failing Builds/job/build-scripts/job/jobs/job/jdk16/job/jdk16-aix-ppc64-hotspot/55/console

Ive updated git on build-osuosl-aix71-ppc64-2 to 2.20.4

@Haroon-Khel What's the status of this one? I suspect the error referenced in the previous comment may be https://github.com/AdoptOpenJDK/openjdk-build/issues/2466 since it seems related to the jvmti area

This issue can be closed, since the original problem was fixed when I updated git on both machines


Issue from visual code

Does this occur consistently?
Repro steps:

I clicked over app of azure

Action: api.revealTreeItem
Error type: Error
Error Message: TreeError [azureAppService] Data tree node not found: [object Object]
Version: 0.19.0
OS: win32
OS Release: 10.0.19041
Product: Visual Studio Code
Product Version: 1.50.1
Language: en

Call Stack
U.getDataNode workbench.desktop.main.js:1159:607
U.expand workbench.desktop.main.js:1157:795
async Q.expand workbench.desktop.main.js:5739:726
async p.reveal workbench.desktop.main.js:4578:470

@enunez5 Could you provide more details to reproduce this?  It looks like you used the revealTreeItem command.  Is there anything of note about your app that could not be found? Did you create it through the portal or with any special conditions?


tox.ini: py.test instead of python setup.py test

and py.test is invoked with {posargs} so an advantage is that the user can pass arguments to py.test -- e.g.:
tox -e py27 -- --pdb tests/unit-tests/messages_test.py

It's possible to pass arguments to py.test inside setup.py test like this: https://pytest.org/latest/goodpractises.html#integration-with-setuptools-test-commands
It'll be only one way to run tests then

Ah okay, I didn't know you can do that. Thanks!
In that case, you might prefer https://github.com/JetBrains/teamcity-messages/pull/78 ...?


Basic autotests added, public API names made more consistent.

I added a sample test cases (using 'Catch' framework).
Also it seems the names of public API are now looking more consistent.
And the Test 3 now fails because the library doesn't like negative value in am_edit_suggest()
What do you think?

eh...
This is a Lua-style C library, so "heavily" naming is not very suit it's style :-(
and the test... I mean, I will add something like https://travis-ci.com/ to auto run current test when every commit, but not meaning use a test framework to run many tests.
for bugs, I will add https://coveralls.io/ support to this repo, to make 100% coverage test. this will eliminate  most of bugs.
so, sorry for we can not use your request... but still very thank you for help improve our code :-)

@starwing can you explain why this naming isn't better? It seems a lot better for autocompletion and IDE's.

@hfossli this naming indeed better, it's more clear. but for autocomplete for IDE's, naming without same prefix may better.
the original naming of amoeba has the style of [Lua language API][https://www.lua.org/manual/5.3/#index],  this is the original purpose for amoeba library: a extension module for Lua (and my GUI framework, [nui][https://github.com/starwing/nui]. amoeba has few APIs to complete it's work, so IMHO i think we needn't use a serious naming :-)

Ok, no problem.


fix: exit code management

Fix #52 and referring to this comment
Summary
With the redirection of stdout to the tee command, the code error return by lw-scanner isn't manage.
With this fix, we're able to fail and display the result of the scanner and fail with the waiting error code for Policy Violations.
How did you test this change?
There's already a test which verify that the fail is managed.
However, not very easy to test the behavior with a waiting error code for policy violations due to this the fast that's configure et trigger on the lacework instance.

@jycamier
we reverted the changes from the original pr to make sure ci / main branch is working again.
this resulted in some conflicts. can you have a look at the conflicts?

@timarenz alright, I'll do it.


Fix text following a link is still clickable

ensure the link can not be clicked after the link text itself ends due to a missed pop()

relates to: https://github.com/mikepenz/multiplatform-markdown-renderer/pull/129

You are right, that's the correct way! I noticed the annotation length was wrong (the end of the range was incorrect) but didn't realize the pop() was missing. Closing my PR and thanks for looking into it.


[Feat] Extend ChartLegendContent Api for interactivity

fixes #4188

@BalduinB I'll take a look at this. I think we should use the default onClick prop on <ChartLegend /> to do this...

This is also possible, but without creating a context it would look like that:
<ChartLegend
    onClick={selectBar}
    onMouseOver={handleLegendMouseEnter}
    onMouseLeave={handleLegendMouseLeave}
    content={({ onClick, onMouseOver, onMouseLeave }) => (
         <ChartLegendContent clicked={onClick} mouseOver={onMouseOver} mouseLeave={onMouseLeave} />
    )}
/>

not 100% shure, but after a quick search...


Update versions of io.jsonwebtoken

Update versions of io.jsonwebtoken dependencies. The current ones are 4 years old and cause a conflict with another frameworks, therefore requiring downgrade. The newer version can't be forced, because the function calls are incompatible.

@nosovm ^

@nosovm @NikolasMelui Is this library still maintained?

The job is done

That was fast, thanks! @NikolasMelui


Error on serializing response

I'm tryring your sample code and i got this error
Severity	Code	Description	Project	File	Line	Suppression State Error	CS0012	The type 'JsonElement' is defined in an assembly that is not referenced. You must add a reference to assembly 'System.Text.Json, Version=<IP_ADDRESS>
and
Severity	Code	Description	Project	File	Line	Suppression State Error	CS0411	The type arguments for method 'SocketIOResponse.GetValue<T>(int)' cannot be inferred from the usage. Try specifying the type arguments explicitly
from this line of code.
ReceivedText.text += "Received On " + name + " : " + response.GetValue().GetRawText() + "\n";

response.GetValue().GetRawText() become response.ToString(), i didn't update the example folder, sorry !

Bro what if the response is an array of int. How can i get their values?

response.GetValue().GetRawText() become response.ToString(), i didn't update the example folder, sorry !

var obj = response.GetValue<YOUR_TYPE>();

replace YOUR_TYPE with the class name or the type you want to serialize to
e.g YOUR_TYPE can be: string[], SomeClass, int and so on
already mentioned in the Readme:
https://github.com/itisnajim/SocketIOUnity#receiving

check if the server send the correct data, maybe its a string rather that list of objects,
this is not a list => "["item1", "item2"]"
and i recommend to use a list wrapper instead of a direct list
e.g a class named StringsList that has a property data of type string[]
without a debug log or a snippet of code i can't tell whats the cause of your problem.


[ECS] port more examples

add kill() and win_game() to client
add death example (not a port in the interest of making the example as minimal as possible)
port biomes example
add kick method to client
port parkour example

Yeah that might work a bit better.

woo parkour example ported wooo woow wo oo wooo


[BUG] 401 "claim is empty or token is invalid" when using Azure.Identity

Describe the bug
When I'm using Azure.Identity authentication, I can't get all topics & all queues from a service bus instance. The first call to get the queues will succeed but the second call to get the topics will always fail (or the other way around).
Expected behavior
Can get all topics & all queues from a service bus instance.
Actual behavior (include Exception or Stack Trace)
The second call to get all queues/topics will fail:
System.UnauthorizedAccessException
  HResult=0x80070005
  Message=claim is empty or token is invalid. TrackingId:7f293702-89fa-46ca-a04e-012f6104b863_G25, SystemTracker:xxx.servicebus.windows.net:$Resources/queues, Timestamp:2021-04-19T07:25:04
Status: 401 (Unauthorized)

Content:
<Error>401<Detail>claim is empty or token is invalid. TrackingId:7f293702-89fa-46ca-a04e-012f6104b863_G25, SystemTracker:xxx.servicebus.windows.net:$Resources/queues, Timestamp:2021-04-19T07:25:04</Detail></Error>

Headers:
Transfer-Encoding: chunked
Server: Microsoft-HTTPAPI/2.0
Strict-Transport-Security: REDACTED
Date: Mon, 19 Apr 2021 07:25:04 GMT
Content-Type: application/xml; charset=utf-8

  Source=Azure.Messaging.ServiceBus
  StackTrace:
   at Azure.Messaging.ServiceBus.Administration.HttpRequestAndResponse.<ThrowIfRequestFailedAsync>d__7.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable.ConfiguredTaskAwaiter.GetResult()
   at Azure.Messaging.ServiceBus.Administration.HttpRequestAndResponse.<SendHttpRequestAsync>d__14.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at Azure.Messaging.ServiceBus.Administration.HttpRequestAndResponse.<GetEntityAsync>d__11.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at Azure.Messaging.ServiceBus.Administration.HttpRequestAndResponse.<GetEntitiesPageAsync>d__10`1.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Azure.Core.PageResponseEnumerator.FuncAsyncPageable`1.<AsPages>d__2.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore`1.GetResult(Int16 token)
   at Azure.Core.PageResponseEnumerator.FuncAsyncPageable`1.<AsPages>d__2.System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult(Int16 token)
   at Azure.AsyncPageable`1.<GetAsyncEnumerator>d__6.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Azure.AsyncPageable`1.<GetAsyncEnumerator>d__6.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore`1.GetResult(Int16 token)
   at Azure.AsyncPageable`1.<GetAsyncEnumerator>d__6.System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult(Int16 token)
   at System.Runtime.CompilerServices.ValueTaskAwaiter`1.GetResult()
   at Xxx.TestController.<Test>d__0.MoveNext() in C:\Dev\repos\xxx\TestController.cs:line 51

Inner Exception 1:
RequestFailedException: claim is empty or token is invalid. TrackingId:7f293702-89fa-46ca-a04e-012f6104b863_G25, SystemTracker:xxx.servicebus.windows.net:$Resources/queues, Timestamp:2021-04-19T07:25:04
Status: 401 (Unauthorized)

Content:
<Error>401<Detail>claim is empty or token is invalid. TrackingId:7f293702-89fa-46ca-a04e-012f6104b863_G25, SystemTracker:xxx.servicebus.windows.net:$Resources/queues, Timestamp:2021-04-19T07:25:04</Detail></Error>

Headers:
Transfer-Encoding: chunked
Server: Microsoft-HTTPAPI/2.0
Strict-Transport-Security: REDACTED
Date: Mon, 19 Apr 2021 07:25:04 GMT
Content-Type: application/xml; charset=utf-8

To Reproduce
Steps to reproduce the behavior (include a code snippet, screenshot, or any additional information that might help us reproduce the issue)

Add your user to the "Azure Service Bus Data Owner" role

Try to get all topics & all queues of this service bus instance and the second call will fail.

For example, if I first get all topics than all queues, the call to get all queues will fail with a 401.
Here is a sample:
var administrationClient = new ServiceBusAdministrationClient("xxx.servicebus.windows.net", new VisualStudioCredential());
var topics = administrationClient.GetTopicsAsync();
await foreach (var topic1 in topics)
{
}

var queues = administrationClient.GetQueuesAsync(); // this line will fail with a 401
await foreach (var queue1 in queues)
{
}

If I try the other way around, the call to GetQueuesAsync will succeed but the call to GetTopicsAsync will fail with a 401.
I also tried to create a new instance of the ServiceBusAdministrationClient & of the TokenCredential (either DefaultCredential or VisualStudioCredential), but it still fails.
Please note that I tried with the old ManagementClient, and I have the same issue:
var managementClient = new ManagementClient("sb://xxx.servicebus.windows.net", new ManagedIdentityTokenProvider());
var topics = await managementClient.GetTopicsAsync();
var queues = await managementClient.GetQueuesAsync(); // 401 Unauthorized

Exception:
Microsoft.Azure.ServiceBus.UnauthorizedException
  HResult=0x80131500
  Message=claim is empty or token is invalid. TrackingId:7b487659-cd9d-4ad1-ae79-117899444f5b_G18, SystemTracker:xxx.servicebus.windows.net:$Resources/queues, Timestamp:2021-04-19T07:33:32
  Source=Microsoft.Azure.ServiceBus
  StackTrace:
   at Microsoft.Azure.ServiceBus.Management.ManagementClient.<SendHttpRequest>d__53.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at Microsoft.Azure.ServiceBus.Management.ManagementClient.<GetEntity>d__50.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at Microsoft.Azure.ServiceBus.Management.ManagementClient.<GetQueuesAsync>d__21.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Xxx.TestController.<Test>d__0.MoveNext() in C:\Dev\repos\Xxx\TestController.cs:line 37

  This exception was originally thrown at this call stack:
    [External Code]
    Xxx.TestController.Test() in TestController.cs

I also tried to create a new service bus instance with a few queues & topics, but it still fails.
Environment:

Name and version of the Library package used: Azure.Messaging.ServiceBus 7.1.2 (and Microsoft.Azure.ServiceBus 5.1.2 for the ManagementClient)
Hosting platform or OS and .NET runtime version (dotnet --info output for .NET Core projects): Windows 10 - .NET Core 3.1

.NET SDK (reflecting any global.json):
 Version:   5.0.200
 Commit:    70b3e65d53

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.19041
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\5.0.200\

Host (useful for support):
  Version: 5.0.3
  Commit:  c636bbdc8a

.NET SDKs installed:
  2.1.801 [C:\Program Files\dotnet\sdk]
  2.1.802 [C:\Program Files\dotnet\sdk]
  2.2.401 [C:\Program Files\dotnet\sdk]
  2.2.402 [C:\Program Files\dotnet\sdk]
  5.0.101 [C:\Program Files\dotnet\sdk]
  5.0.200 [C:\Program Files\dotnet\sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.All 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.13 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.25 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.2.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.2.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.2.8 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.13 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.25 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.2.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.2.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.2.8 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.3 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.13 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.25 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.2.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.2.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.2.8 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.12 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.1 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.3 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 3.1.12 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 5.0.1 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 5.0.3 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

IDE and version : Visual Studio 16.9.0

Thank you for your feedback.  Tagging and routing to the team member best able to assist.

Hello, any news about this issue? This is a blocker for us to use User Managed Identity instead of access keys.

@MarienMonnier Apologies for the delay in response here
@JoshLove-msft Can you please review this issue? Managed Identity should work as expected with the Service Bus library as far as I know. So, may be some set up problems?

Yes, I reproed the issue and I believe @yvgopal is going to look into it.

This is an issue for us too, seems like a fairly common use case

Hi Team,
Can we please have an update on this bug/issue.

We are actively investigating this issue. We will provide an update asap. cc @serkantkaraca

We ran into this using com.azure:azure-messaging-servicebus:7.6.0 library in Java on a new Service Bus namespace in West Europe.
In our case we tried to use a single ServiceBusAdministrationAsyncClient to create-if-missing several ASB Topics, and started getting HTTP 401 "claim is empty or token is invalid" errors somewhere in the middle.
Workaround for us was to create a new ServiceBusAdministrationAsyncClient for every ASB Topic we had to check and create if necessary.

@MarienMonnier Apologies for the late reply. Could you please confirm if you are still facing this issue ? Awaiting your reply.

Hi @navba-MSFT ,
I hadn't test since the last time. I can confirm that this is now working correctly like @MrKevHunter said.
I tested with the newest versions (Azure.Messaging.ServiceBus 7.7.0 & Microsoft.Azure.ServiceBus 5.2.0) and the versions I used a year ago when I faced the issue (Azure.Messaging.ServiceBus 7.1.2 & Microsoft.Azure.ServiceBus 5.1.2) and I confirm that this is working correctly, so the fix must have been made on Service Bus side and not inside the SDK.
Thanks,


[Server start] Improve message when elastic not launched/available

Currently we see this:
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/starlette/routing.py", line 526, in lifespan
    async for item in self.lifespan_context(app):
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/starlette/routing.py", line 467, in default_lifespan
    await self.startup()
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/starlette/routing.py", line 502, in startup
    await handler()
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/rubrix/server/server.py", line 101, in configure_elasticsearch
    raise ConfigError(
pydantic.errors.ConfigError: Your Elasticsearch endpoint at http://localhost:9200 is not available or not responding.
Please make sure your Elasticsearch instance is launched and correctly running and you have the necessary access permissions. Once you have verified this, restart the Rubrix server.
Error detail: [ConnectionError(<urllib3.connection.HTTPConnection object at 0x16fc94610>: Failed to establish a new connection: [Errno 61] Connection refused) caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x16fc94610>: Failed to establish a new connection: [Errno 61] Connection refused)]

For improving readability I'd recommend removing the Error detail as it does not add anything meaningful and makes the clear message more difficult to read.
Is this possible?

We can separate message in two logger calls with different logging level. From the point of view of a system manager, the error detail could be important to detect the problem. But it's true that we can enable by a --verbose logging param or something like that.

Not sure if this is possible, but I think for the "default user" it would already help if the "Error detail" came first, and then the "pydantic.errors.ConfigError".


Added the option to specify mouse position for the DND events

breaking change: defaults drags and drops to the center of the relevant elements.

@vzaidman Great to see that someone also needs this! Thanks for continuing on what I started! 👍
I was also thinking of creating a PR for this, though fixing integration tests is a bit of a hustle. And I'm not entirely sure whether my approach with createMouseEvent works nicely cross browser (https://developer.mozilla.org/en-US/docs/Web/API/MouseEvent/initMouseEvent).

this feature would be super useful. Hope this will be merge soon !

@vzaidman Any work still being done on this?

it works as far as i know.
I didn't used it in the end.
whoever wants, please fix the tests and merge it.

Still failing due to missing arguments. Will add those now.


Adds support for configuring error output. Closes #2325

Adds support for configuring error output. Closes #2325
Also, I centralized the list of available settings in the docs so that we don't need to maintain them in multiple locations.

Merged manually


patching / compiling kernel with Auvidea J20 CSI camera module support, missing files..

Hi,
I have patched the kernel source (jetpack 3.2) to include the Auvidea J20 module, selected and enabled it in xconfig.
But while compiling it looks for this files and cant find them:
+#include <media/v4l2-chip-ident.h>
+#include <media/camera_common.h>
+#include <media/soc_camera.h>
drivers/media/i2c/j20.c:32:35: fatal error: media/v4l2-chip-ident.h: No such file or directory
I can manually locate and copy the camera_common.h and soc_camera.h, but those again wants a lot of other header files that are suppose to be in media/
The v4l2-chip-ident.h is no where to be found, but i am sure i am doing something wrong, i should-en need to locate this files should i ?
I am compiling the kernel directly on the TX2 like this description:
https://github.com/jetsonhacks/buildJetsonTX2Kernel
I have also downloaded the kernel sources on my Ubuntu machine just in case there was some difference between building on the TX2 or a Ubuntu pc, but the v4l2-chip-ident.h is still no where to be found, going crazy here :)

The sources downloaded by these scripts are for the Jetson carrier boards. The kernel sources for the Auvidea boards are likely different. I do not know if Auvidea have updated their kernel sources for L4T 28.2. You will have to contact Auvidea for the source.

@Spawn32 @jetsonhacks There are any news about how to make work the Auvidea J20 on the Jetson TX2? Thanks


Remove pagination with cassandra

Pagination is not supported with JHipster + Cassandra. For the moment, if you generate an app with Cassandra and run npm test, it will fail

Please make sure the below checklist is followed for Pull Requests.

[ ] Travis tests are green

[ ] Tests are added where necessary

[ ] Documentation is added/updated where necessary

[ ] Coding Rules & Commit Guidelines as per our CONTRIBUTING.md document are followed

Please rebase to the latest version :)
Could be great to have an Azure configuration for this (Vue.js + Cassandra).
cc @pascalgrimaud @Hawkurane