"husarion_whoami" package not found (Demo README guide)

Describe the bug
When following the Demo README, running ros2 run rai_whoami rai_whoami_node --ros-args -p robot_description_package:="husarion_whoami" returns an error with missing husarion_whoami package.
ament_index_python.packages.PackageNotFoundError: "package 'husarion_whoami' not found, searching: [...]

To Reproduce
Steps to reproduce the behavior:

Follow rai Quick start
Follow the first point in Demo README: Running rai
See error

Expected behavior
The node starts correctly.
Platform

OS: Ubuntu 24.04
ROS 2 Jazzy

Version
ddb5c6eae3cb821fae2f708abd7c45427b24b034
Additional context
The first problem is the package name, the Demo README should mention the new name: rosbot_xl_whoami.
The second problem is that the package is not built and the ros2 run can't find it.
The fix is to rebuild this package only and re-source the workspace.
colcon build --symlink-install --packages-select rosbot_xl_whoami
. ./install/setup.bash

Thank you for PR. Merged!


Update depencencies

Do a go get -u and go mod tidy
This updates all deps to a newer version.

more generic update than #1048 (which may fix that issue - whatever it is/was)

Codecov Report

Merging #1049 into master will increase coverage by 0.03%.
The diff coverage is n/a.

@@            Coverage Diff            @@
##           master   #1049      +/-   ##
=========================================
+ Coverage   54.97%     55%   +0.03%
=========================================
  Files          41      41
  Lines        9872    9872
=========================================
+ Hits         5427    5430       +3
+ Misses       3423    3421       -2
+ Partials     1022    1021       -1

Impacted Files
Coverage Δ

server.go
67.4% <0%> (-0.28%)
:arrow_down:

msg.go
77.23% <0%> (+0.67%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update aae7df6...8dcea2d. Read the comment docs.

If my understanding is correct, I’m pretty certain this type of change is wrong when using modules. With MVS any importers requiring newer versions should update their own requirements. Updating here just forces consumers to pull newer dependencies unnecessarily. I’ll try and find a reference.

I think this is the comment I was thinking of https://github.com/golang/go/issues/35798#issuecomment-558358674.

Well, but, we should still regularly update ours anyway...
On Sat, 14 Dec 2019, 13:48 Tom Thorogood<EMAIL_ADDRESS>wrote:

I think this is the comment I was thinking of golang/go#35798 (comment)
https://github.com/golang/go/issues/35798#issuecomment-558358674.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/miekg/dns/pull/1049?email_source=notifications&email_token=AACWIW2FPUC6HVTIVMAGX6LQYTP2DA5CNFSM4J2A4NDKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEG4C6OQ#issuecomment-565718842,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AACWIW23BPHL3MOT2I5N5VDQYTP2DANCNFSM4J2A4NDA
.


runtime: excessive scavengeOne work slows mutator progress

Some background: I have an app that serves interactive HTTP traffic (typical response time is tens of milliseconds), and which also does periodic work to refresh the state necessary to serve that interactive traffic. Its live heap is usually around 8 GiB. The work to refresh the state spans over several GC cycles and can involve allocating hundreds of MiBs or even a couple GiBs over a few hundred milliseconds, which gives the pacer a hard time and can lead to the pacer choosing a high assist factor. To work around that, we're 1/ calling runtime.GC at a few key points in the state-refresh process, and 2/ running with GOMEMLIMIT=20GiB and GOGC=off. The app runs on a machine with 16 hardware threads (GOMAXPROCS=16).
The problem: There appears to be contention on the mheap lock in runtime.(*pageAlloc).scavengeOne. Execution traces show this can tie up all 16 Ps for hundreds of milliseconds.
CC @golang/runtime and @mknyszek
What version of Go are you using (go version)?
go1.19.3 linux/amd64
Does this issue reproduce with the latest release?
Yes: go1.19.3 is the latest stable release. I have not tried with tip.
What operating system and processor architecture are you using (go env)?
linux/amd64
What did you do?
GOMAXPROCS=16, GOMEMLIMIT=20GiB, GOGC=off
Manual call to runtime.GC, followed (after that call returns, indicating that the sweep work is done) by allocating memory especially quickly.
All while serving interactive (tens of milliseconds typical response time) HTTP requests.
What did you expect to see?
Consistent responsiveness for the interactive HTTP requests this process serves, regardless of which phase the GC is executing.
When the app has its own runnable goroutines, I expect the runtime to stay close to within its goal of 25% of P time spent on GC-related work.
What did you see instead?
The app's Ps are kept busy in a nearly-opaque form of work. Rather than running a G for around 100 µs and then moving on to the next G, each P has a single G that it claims to be running for hundreds of milliseconds without interruption. (Note, if those goroutines had "normal" work, I'd still expect them to be preempted by sysmon every 20 ms or so.) At times, every P in the app is kept busy this way.
When the Ps are busy this way, their CPUSample events in the execution trace (each of which represents 10 milliseconds of on-CPU time) are spread tens of milliseconds apart, and sometimes more than 100 ms. That suggests that these Ps are not spending their time executing any code, even within the kernel, and that instead they're asleep waiting for a lock that is not visible to the execution trace (such as a runtime-internal lock).
The CPUSample events during those periods show the Ps are trying to allocate new spans to use for new memory allocations. Most samples are within runtime.(*pageAlloc).scavengeOne and show a call to runtime.madvise. A few show time in that function's calls to runtime.lock and runtime.unlock.
Because those periods include so little on-CPU time outside of runtime.(*pageAlloc).scavengeOne, and also show so little on-CPU time in total, and show some on-CPU time interacting with the mheap lock, I take this to mean that the Ps are spending their with runtime.(*pageAlloc).scavengeOne on their stack, but sleeping, likely waiting for the mheap lock.
Here's one of the call stacks from a CPUSample event that arrived while a P was busy for 100+ milliseconds (showing from mallocgc):
runtime.madvise:546
runtime.sysUnusedOS:111
runtime.sysUnused:62
runtime.(*pageAlloc).scavengeOne:741
runtime.(*pageAlloc).scavenge.func1:647
runtime.(*pageAlloc).scavenge:646
runtime.(*mheap).allocSpan:1317
runtime.(*mheap).alloc.func1:910
runtime.systemstack:492
runtime.(*mheap).alloc:904
runtime.(*mcentral).grow:244
runtime.(*mcentral).cacheSpan:164
runtime.(*mcache).refill:181
runtime.(*mcache).nextFree:819
runtime.mallocgc:1018

Here's how the execution trace looks. The sweep work ends around 275 ms. That also means the app's manual runtime.GC call returns, which allows the allocation-intensive refresh procedure to continue. Procs 1, 12, 15 do some allocation-heavy work and then get stuck in scavengeOne from the 400 to 800 ms marks. (Some other Procs, like 3 and 11, do heavy allocations around the 400 ms mark but do not get stuck.) Around 800 ms, all 16 Procs get stuck and the queue of runnable goroutines grows. And around 1000 ms, a timer should have triggered to call runtime/trace.Stop, but the trace continues beyond the 1300 ms mark; the app is no longer responsive.

And here's the call_tree view from under mallocgc. This is the runtime/pprof output of the same CPU profile that is shown in the execution trace; these samples will cover the same time period as that entire execution trace, not filtered to any smaller time range.
$ ln -s /usr/local/go/bin/go /tmp/redacted
$ go tool pprof '-focus=^runtime\.mallocgc$' '-show_from=^runtime\.mallocgc$' -call_tree \
    -nodefraction=0 -edgefraction=0 -nodecount=1000 /tmp/redacted pprof/profile-during-trace

That's really interesting. It seems like just after a forced GC call, a lot of memory is suddenly available to scavenge, and in the rapid allocation the runtime thinks all that memory is going to cause it to exceed the memory limit. That's why all the scavengeOne calls are coming from allocSpan. I wonder how big the ranges are that it's trying to scavenge. It should be maximum 2 MiB, though in that case the runtime is only guessing that those memory regions are backed with huge pages (I'm hoping this will improve significantly with the solution to #55328, which I have in flight).
Back of envelope, I would expect the worst case for 2 MiB at once is a ~5 ms delay (which is kind of bad). However, the lock contention is a little weird though because the scavenger shouldn't ever be holding the lock across the madvise call, though it will grab it twice: once before madvise, and once after. Failing to make progress on this entirely is somewhat surprising; at some point the GC CPU limiter should kick in, though the window for that is large at 1 CPU-second per P. This makes me think that window should be smaller.
Does the application hang completely, or does it recover after some time? I'd love to get a GODEBUG=pagetrace=/path/to/file out of this (need to build with GOEXPERIMENT=pagetrace on tip), but unfortunately that currently requires starting the trace from the very start of execution, so it might require a lot of disk. I think that would essentially tell the full story.
On that topic fix to #55328 might help in other ways here. I think the fix there results in a much better scavenging heuristic too, which would cause less scavenged memory to be allocated (causing these calls) in the first place. However, it does not tackle the root of the problem (or problems).
One problem is definitely scalability of scavengeOne. Right now, it needs that very central lock. It doesn't hold it over the madvise, but it's still clearly having a large effect. I think this might be the easiest problem to tackle, but it's still not that easy as it problem requires some page allocator restructuring.
Another problem is getting into this state where the scavenger is being rapidly called into by every P because of the memory limit. This is harder to avoid because the memory limit still needs to be maintained. One idea I have here though is to have the sweeper actually help return some memory to the OS that it doesn't think will be used, to help prevent this situation. The trouble is identifying what to return to the OS.

cc @golang/runtime @mknyszek @aclements

It seems like just after a forced GC call, a lot of memory is suddenly available to scavenge

Can you say a bit more please? Does this mean that forcing a GC causes memory to become available to the scavenger, in a way that automatic GCs do not?

in the rapid allocation the runtime thinks all that memory is going to cause it to exceed the memory limit

I don't have data on the size of the allocated heap before this GC, or of how much the runtime has requested from the OS (like /memory/classes/total:bytes).

I wonder how big the ranges are that it's trying to scavenge.

Focusing on scavengeOne samples, the CPU profile associated with this execution trace shows time below runtime.(*mcache).nextFree, and it also shows time below runtime.(*mcache).allocLarge. The "large" allocations are in the neighborhood of 190 kiB, but looking at the heap profiles this process instance generated I see a lot of variation in that allocation's exact size. Because we're not rounding those up to, say, the nearest power of 2, I wonder if the runtime isn't often able to reuse those and if they're a big cause of the need for scavenging in the first place. I'm not very familiar with the scavenger or large allocations — does that sound right to you?
But I've also seen CPU profiles that show lots of time in runtime.(*pageAlloc).scavengeOne and  almost no samples in runtime.(*mcache).allocLarge. Though that could be a matter of timing, with the large allocs taking place either before or after that profile's time window.

Failing to make progress on this entirely is somewhat surprising; at some point the GC CPU limiter should kick in, though the window for that is large at 1 CPU-second per P. This makes me think that window should be smaller.

I'll see if the team is able to run with a modified toolchain that reduces that window.

Does the application hang completely, or does it recover after some time?

It recovers, yes. Certainly within two minutes (at its next scheduled telemetry dump time, which showed unremarkable behavior).

I'd love to get a GODEBUG=pagetrace=/path/to/file out of this (need to build with GOEXPERIMENT=pagetrace on tip), but unfortunately that currently requires starting the trace from the very start of execution, so it might require a lot of disk. I think that would essentially tell the full story.

I think this is going to be hard, especially before Go 1.20. I haven't seen this behavior outside of the team's production environment, and the team isn't set up well to run like this (tip, plus a GOEXPERIMENT, plus a large/growing file on disk) in production.

Another problem is getting into this state where the scavenger is being rapidly called into by every P because of the memory limit. This is harder to avoid because the memory limit still needs to be maintained.

This sounds like you're saying that our use of GOMEMLIMIT and GOGC=off is directly responsible for the large amount of scavengeOne work. Is that right?
Thank you!

It seems like just after a forced GC call, a lot of memory is suddenly available to scavenge

Can you say a bit more please? Does this mean that forcing a GC causes memory to become available to the scavenger, in a way that automatic GCs do not?
Kind of. Here's what I was thinking (but I'm less convinced this specifically is an issue now, see my other replies below): forced GCs ensure the sweep phase is complete in addition to the mark phase before continuing. Sweeping frees pages, which makes them available for scavenging. The runtime kicks the background scavenger awake at the end of each sweep phase, but there's potentially a multi-millisecond delay as sysmon is responsible for waking it.

If the background scavenger is asleep during a sweep phase, this can in theory also happen with automatic GCs. However, I think in your particular case where there's this sort of "calm before the storm" as a GC gets forced, I suspect the scavenger is more likely to be asleep, leaving all the maintenance work to allocating goroutines.
Writing this out makes me think that maybe if the memory allocator starts scavenging, it immediately kicks the scavenger awake. Though, we should confirm that the background scavenger is indeed asleep on the job in this case. It could be that even if it was constantly working, it wouldn't be doing enough to avoid this.

in the rapid allocation the runtime thinks all that memory is going to cause it to exceed the memory limit

I don't have data on the size of the allocated heap before this GC, or of how much the runtime has requested from the OS (like /memory/classes/total:bytes).

I wonder how big the ranges are that it's trying to scavenge.

Focusing on scavengeOne samples, the CPU profile associated with this execution trace shows time below runtime.(*mcache).nextFree, and it also shows time below runtime.(*mcache).allocLarge. The "large" allocations are in the neighborhood of 190 kiB, but looking at the heap profiles this process instance generated I see a lot of variation in that allocation's exact size. Because we're not rounding those up to, say, the nearest power of 2, I wonder if the runtime isn't often able to reuse those and if they're a big cause of the need for scavenging in the first place. I'm not very familiar with the scavenger or large allocations — does that sound right to you?
That's really interesting. Allocations that large definitely do have the potential to increase fragmentation a good bit, making it more likely that the allocator has to find new address space. That in turn will mean more frequent scavenging to stay below the memory limit.
But I've also seen CPU profiles that show lots of time in runtime.(*pageAlloc).scavengeOne and almost no samples in runtime.(*mcache).allocLarge. Though that could be a matter of timing, with the large allocs taking place either before or after that profile's time window.
Hm, yeah it could be timing. Fragmentation can still happen even if not in the allocLarge case, if a lot of your objects tend to be in the larger end of the size classes.

Failing to make progress on this entirely is somewhat surprising; at some point the GC CPU limiter should kick in, though the window for that is large at 1 CPU-second per P. This makes me think that window should be smaller.

I'll see if the team is able to run with a modified toolchain that reduces that window.
Thanks!

Does the application hang completely, or does it recover after some time?

It recovers, yes. Certainly within two minutes (at its next scheduled telemetry dump time, which showed unremarkable behavior).
Got it, just wanted to make sure this wasn't some kind of deadlock.

I'd love to get a GODEBUG=pagetrace=/path/to/file out of this (need to build with GOEXPERIMENT=pagetrace on tip), but unfortunately that currently requires starting the trace from the very start of execution, so it might require a lot of disk. I think that would essentially tell the full story.

I think this is going to be hard, especially before Go 1.20. I haven't seen this behavior outside of the team's production environment, and the team isn't set up well to run like this (tip, plus a GOEXPERIMENT, plus a large/growing file on disk) in production.
Yeah, I figured. A tool of last resort I suppose. I could also hack together a version that enables the page tracer with the execution trace. It should also be possible to make the page trace tooling a little more resilient to partial traces, so no crazy changes to the format are necessary. Let me know if this is of interest to you.

Another problem is getting into this state where the scavenger is being rapidly called into by every P because of the memory limit. This is harder to avoid because the memory limit still needs to be maintained.

This sounds like you're saying that our use of GOMEMLIMIT and GOGC=off is directly responsible for the large amount of scavengeOne work. Is that right?
Yeah. That call path (allocSpan -> scavenge -> scavengeOne) only kicks in when an allocation would cause the memory limit to be exceeded.

I think your description is painting a fairly clear picture: it sounds like fragmentation and the sudden need to allocate larger things is forcing the runtime to scramble and find memory to release to stay under the memory limit. The background scavenger in this scenario is instructed to scavenge until it's 5% under the memory limit which is supposed to help avoid these scenarios, but with GOMEMLIMIT and GOGC=off the heap limit can easily eat up that 5% each GC cycle.
I think the runtime might be doing something wrong here. I think that maybe it should be reserving a little bit more space off of the heap limit for hedging against future fragmentation. I suspect at your heap size this wouldn't really cost you very much in terms of GC frequency, but would make this transition to periodic work much smoother.
More specifically, the fixed headroom found at https://cs.opensource.google/go/go/+/master:src/runtime/mgcpacer.go;l=1036 should probably be proportional to the heap limit (something like 3-5%, so multiplied by 0.95-0.97). I was originally reluctant to do this, but I think your issue is solid evidence for doing this. I also think it might help with another memory limit issue, which is that it works less well for smaller heaps (around 64 MiB or less). It's only a 2-3 line change (well, probably a little bit more including a pacer test) and it might resolve this particular issue. Of course, a number here won't be perfect for every scenario, but it's a start. A more adaptive solution would be good to consider for the future.
Also, this follows a more general pattern of the pacer pacing for the "edge" (#56966) and not hedging enough for noise or changes in the steady-state in general.

Thank you!

Here's an attempt at a fix if you'd like to try it out: https://go.dev/cl/460375

@rhysh Do you think you'll have a chance to try this patch out for Go 1.21? I'm curious if it makes a meaningful difference in performance and/or if it helps make this situation less likely.

Here's the log output in an easier-to-read format:
$ cat after.txt | cut -d ' ' -f 5 | dist
N 604  sum 108840  mean 180.199  std dev 2209.87  variance 4.88355e+06

     min 0
   1%ile 0
   5%ile 0
  25%ile 0
  median 0
  75%ile 0
  95%ile 3.6
  99%ile 2968.08
     max 50136

⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡖ 0.000
⠐⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠂⡇
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠧ 0.000
⠈⠉⠉⠉⠉⠉⠋⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠙⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠋⠉⠁
      0                           5000                           10000
$ cat before.txt | cut -d ' ' -f 5 | dist
N 536  sum 5.79854e+06  mean 10818.2  std dev 28523.3  variance 8.13578e+08

     min 0
   1%ile 0
   5%ile 0
  25%ile 0
  median 56
  75%ile 5129.33
  95%ile 80088
  99%ile 114991
     max 242224

⠀⠀⠀⠀⠀⠀⠴⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡖ 0.000
⠀⠀⠀⠀⠀⠀⠤⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡇
⠠⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠄⠧ 0.000
⠈⠉⠉⠉⠉⠉⠙⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠋⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠁
      0                            100000

I believe my change mitigates the issue sufficiently. Please reopen if you see this again in production, @rhysh!


Hygieia publisher not working

Hi,
I have configure Hygieia Publiser in jenkins but nothing is displayed on hygieia dashboard.
After successful build i tried to check on Configure Deploy widget on Hygieia dashboard it shows No items were found.
Jenkins Global Configure:
Jenkins global configuration done got success message.
I have done all the Hygieia Publisher job level configuration also have some issue so not able upload the my configuration screen shots.
MY Console outoput:
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 4.894 s
[INFO] Finished at: 2016-12-21T05:57:38+00:00
[INFO] Final Memory: 23M/279M
[INFO] ------------------------------------------------------------------------
[JENKINS] Archiving /home/jenkins/workspace/...../pom.xml to com.comcast.rphy/devops/0.0.1-SNAPSHOT/devops-0.0.1-SNAPSHOT.pom
[JENKINS] Archiving /home/jenkins/workspace/...../target/devops.war to com......rphy/devops/0.0.1-SNAPSHOT/devops-0.0.1-SNAPSHOT.war
channel stopped
Hygieia: Published Build Complete Data. Response Code: 201. Response Value= 585a19c9f290810f1fb84199
Hygieia Build Artifact Publisher - Looking for file pattern 'devops.war' in directory /home/jenkins/workspace/.../.../ws/target
Hygieia: Published Sonar Result. Nothing to publish
Hygieia Deployment Publisher - Looking for file pattern 'devops.war' in directory /home/jenkins/workspace/...../..../ws/target
Finished: SUCCESS

@vetrivelan25
Hygieia Deployment Publisher - Looking for file pattern 'devops.war' in directory /home/jenkins/workspace/...../..../ws/target
Can you confirm that the file devops.war exists in the file system path as listed in the log ?

Hi,
Problem in deployment path.Now it's working.Thanks for your help.


would this work on a raspberry pi?

as in title: would this work on a raspberry pi?

I haven't test it on a raspberry pi.
You can try it.


Small changes

hey @xavijam quick CR?

Smells good.

Frontend tests were OK :+1: (details)

Frontend tests were OK :+1: (details)


Turbo daemon interrupts git rebase with git lock file

Verify canary release

[X] I verified that the issue exists in the latest Turborepo canary release.

Link to code that reproduces this issue
https://github.com/chris-olszewski/turbo-large-monorepo
What package manager are you using / does the bug impact?
pnpm / npm (haven't tested yarn)
What operating system are you using?
MacOS 14.4.1
Which canary version will you have in your reproduction?
2.0.0-canary.0
Describe the Bug
When the turbo daemon is running, and I perform a rebase of one of my git development branches, the rebase is interrupted repeatedly (every few rebased commits), with a git lockfile error:
error: Unable to create '/Users/dominic/workspace/my-project/.git/index.lock': File exists.

Another git process seems to be running in this repository, e.g.
an editor opened by 'git commit'. Please make sure all processes
are terminated then try again. If it still fails, a git process
may have crashed in this repository earlier:
remove the file manually to continue.

If I turbo daemon stop the problem no longer occurs.
Expected Behavior
It seems like it might be unsafe (in terms of breaking users' workflows) for a daemon to lock the .git/ directory after files have changed, because they'll be changing rapidly and repeatedly during a rebase. I'm not sure if that's something the turbo daemon does, but it seems like it might from having a quick browse of the code?
Perhaps related: I can't see any usage of --no-optional-locks nor GIT_OPTIONAL_LOCKS within turbo, so even things that look read-only, like git status, could cause a lock to be taken (git status updates the index as a side-effect).
To Reproduce
I have a large monorepo (although the branch I'm rebasing is relatively small - 5 commits, maybe 1000 lines, package and file changes mixed):
$ du -sh .git/
895M	.git
$ git rev-list --all --count
28050

I tried to reproduce the problem on a new repository made with npx create-turbo@latest -e with-shell-commands, however in such a repository, git operations complete much faster,  and the problem isn't reproducible. I also tried to add additional commits to slow the rebase down, but I couldn't add enough.
When testing, remember to add the --no-ff flag to git rebase so it actually rebases, if the branch is already up to date.
Here is an open source reproduction:
# obtain a large monorepo
git clone git@github.com:chris-olszewski/turbo-large-monorepo.git
cd turbo-large-monorepo/
npm install # this produces a modified package-lock.json

# commit that change to a new branch so we have a commit to rebase
git checkout -b new
git add package-lock.json
git commit -m 'ran npm install'

# start the daemon
turbo build
# can control-c this after a while, just so `turbo daemon status` shows the daemon is running

# cause the lock failure
git rebase --no-ff main
# repeat this last step (after git rebase --abort) until it fails, but it's a 100% failure rate for me
# turbo daemon stop allows the rebase to complete successfully

For canary:
$ npm add -DE turbo@canary
$ ./node_modules/.bin/turbo --version
2.0.0-canary.0
$ ./node_modules/.bin/turbo daemon stop
$ ./node_modules/.bin/turbo daemon start
$ git rebase --no-ff main # same failure

Additional context
turbo daemon logs contains many lines like:
2024-05-28T01:20:27.805686Z  WARN turborepo_lib::package_changes_watcher: file event lagged
2024-05-28T01:20:28.808170Z  WARN turborepo_lib::package_changes_watcher: file event lagged

however, it doesn't appear to log any of those at the time the lock occurs.
When I run the daemon with turbo daemon -vvv, there are a lot of logs, possibly too many to see what's going on. When I run with slightly less detail (-vv), I can see the watcher notice changed_packages and changed_files (this branch did change the pnpm lockfile). I can also see it receiving events on files from within .git/.
But I can't see is why it would be creating its own index.lock (or where it does any git operations, or interferes with the lockfile from some other process) - those don't seem to be logged in detail, or I'm not familiar enough with the daemon logging to spot them - I'm happy to email away my daemon logs if it helps.
I'm also unable to say what the git process (presumably) spawned by the daemon is doing in any detail, unfortunately: I'm running with SIP (so no good tooling for monitoring git process executions system-wide), and by the time lsof displays the process that has produced the lock, it is long gone (so an immediate ps doesn't show it, and MacOS lsof doesn't show process arguments!). Happy to be pointed to a method that might snarf the full process though.

We're seeing the same error regularly with our monorepo. I've seen it with rebasing, but also with lint-staged which stashes unstaged changes, and pops them afterwards. This can result in changes being lost since lint-staged runs as a pre-commit hook.

Yes, seeing the same issue when running turbo and trying to commit changes in tools like Sourcetree (Mac)

https://github.com/vercel/turbo/pull/8247

This should be fixed in 2.0 or 1.13.4. Just restart the daemon with turbo daemon restart


I am sharing pdf file but jpeg unsupported image is sent in ios flutter

I am using this lib for sharing a pdf file as an MMS. It works fine in Android but in iOS it shares a jpeg unsupported image. My code is:
 await SmsMms.send(
        recipients: phoneNo,
        message: 'some message',
        filePath: tempFile.path
      );

Hey @UmairSaqibBhutta Can you please share your device details. It'll be of great help

I have tested this on iPhones 14 and 11.


Setup arguments are ignored (for lua_ls and ltex)

Description
See #2797 (similar story except my issue is persistent with minimal config).
Any arguments passed to lua_ls on setup are ignored. In particular, if I inspect using :Mason, 'Lua.completion.callSnippet' is still the default "Disable", rather than "Replace" as specified.
See my minimal config here. This is a stripped back version of kickstart, where the issue is also present.
I have also tried to pass arguments to ltex, with the same issue. I am using lua_ls here as an example where I can be sure I haven't made some stupid mistake. I have not tested any other LSPs, the issue may or may not be present for them.
This has been tested on most recent nightly and stable nvim. I have tried a clean nvim install and clean install of all plugins with no change.
I would really appreciate some guidance on what might be going wrong, I am not experienced, but am fairly confident the people who wrote kickstart are! And that this ought to work.
Many thanks
Neovim verison:
NVIM v0.11.0-dev-411+geab535e10
Build type: RelWithDebInfo
LuaJIT 2.1.1720049189
OS:
Debian 12

works fine for me . there already have minimal for test https://github.com/neovim/nvim-lspconfig/blob/master/test/minimal_init.lua

works fine for me . there already have minimal for test https://github.com/neovim/nvim-lspconfig/blob/master/test/minimal_init.lua

Do you have any insight as to what might be going wrong?
I have tried the minimal config you linked, passing some random settings to pyright, without any change.

if something wrong you get some informations from log maybe. log in vim.fn.stdpath('log')


Submit Individual Overviews

For Assignment 2 fill out survey for Assignment 2 and post on Google Docs

Evaluations are completed. Adding a new ticket for reviewing before we submit.


Fix JWT request header name

JWT can be passed using the cf-access-token request header

thanks @cehrig for this. #693 applies your changes to the new Cloudflare for Teams documentation, they're now visible here. This is because in a couple of weeks we will deprecate Access, Gateway, Tunnel and WARP for Teams separate docs. Closing this

thanks @cehrig for this. #693 applies your changes to the new Cloudflare for Teams documentation, they're now visible here. This is because in a couple of weeks we will deprecate Access, Gateway, Tunnel and WARP for Teams separate docs. Closing this


Colour code makes differentiation difficult

While browsing through the explorer, I noticed the colour code of "short fibre bundles HCP" (see select layer -> parcellaction maps ->  fibre architecture). This appears in green, yellow and orange graduations which make it difficult to distinguish the individual fibre bundles from each other.
Before:

Now (v2.9):

It is not a must, but a multi-coloured colour code would improve the differentiation. Perhaps this could be adapted in the next release.

related to https://github.com/FZJ-INM1-BDA/siibra-explorer/issues/1013
we will need to revisit how region colours are generated


Fix apostrophes in zsh Tab completion

Fixes getnikola/nikola#1870.
Signed-off-by: Chris Warrick<EMAIL_ADDRESS>

thanks! I didnt actually test it... I am trusting you :)
I guess Nikola includes a generated zsh complention file that needs to be manually updated. That could be done now, no need to wait next doit release.

And of course it breaks the tests... :( I will fix that myself.

Nikola does not include a generated file at all, users just run nikola tabcompletion and take it from there. I’m not even sure if it’s documented anywhere.


add framework to raft store and transfer healthcheck to protobuf type

Current coverage is 37.44% (diff: 49.29%)

Merging #60 into master will increase coverage by 4.76%

@@             master        #60   diff @@
==========================================
  Files            29         30      +1
  Lines          3387       4775   +1388
  Methods           0          0
  Messages          0          0
  Branches          0          0
==========================================
+ Hits           1107       1788    +681
- Misses         2081       2625    +544
- Partials        199        362    +163

Powered by Codecov. Last update 12ca5cb...136ec33


DataFrame created has incorrect schema

I want to use a custom schema for creating the dataframe. On executing the below code
val customSchema = StructType(Array(StructField("firstName",StringType,true),StructField("lastName",StringType,true),StructField("age",IntegerType,true)))

val df = sqlContext.read.
        format("com.springml.spark.sftp").
        option("host", "localhost").
        option("username", "root").
        option("password", "****").
        option("fileType", "csv").
        option("inferSchema", "false").
        option("header","false").
        schema(customSchema).
        load("/home/files/data_people.csv")

df.printSchema()

df.show()

The output i get is as follows:
root
 |-- C0: string (nullable = true)
 |-- C1: string (nullable = true)
 |-- C2: string (nullable = true)

+-------+----------+---+
|     C0|        C1| C2|
+-------+----------+---+
|John   |     last | 24|
|Jack   |     last | 25|
+-------+----------+---+

Clearly the schema provided is not being considered. Any suggestions on how to get this working?

I am having same issue, is there a work around for this?

@mittalakhilesh
This is a bug and I am working on this. Current there is no workaround for this issue. I'll update this ticket once I push the fix.

@samuel-pt please take a look at the pull request submitted by me to fix this issue
Sent from my OnePlus ONEPLUS A3003 using FastHub

Currently I used workaround of converting DF to RDD and then again back to DF with schema.
val dfRdd = df.rdd
val newDf = sparkSession.createDataFrame(dfRdd, schema)
newDf

This issue is resolved by https://github.com/springml/spark-sftp/commit/55a6764e77b767d64835ed7c1ac32438d7023398


Destroy or reinitialize the Game

Hi,
I have a major performance issue with a game i'm creating. It's a game with multiple levels which are build in Tiled. It's a Vue JS website, each time I open a level I need to reinitialise Hexi with a new .json file and spritesheets. Is there in any way a possibility in Hexi to stop the processes in the background (destroy the current game) and start over?
Because after opening and restarting multiple levels my players starts walking very slow. I first thought it was because of the loop that's been called multiple times or maybe I had to work with requestAnimationFrame but these things had no difference whatsoever.
I'm out of options, my last option is to take away the VUE functionality and reload the page each time a player opens up a level, but this will have to reload a lot more assets and files. Which I obviously rather not do.
Kind regards,
Thomas

Thanks for posting this!
Unfortunately as I am not a Vue user I don't know how to help with this problem.
Can anyone out there please assist us??


refactor(perfmon): replace deprecated AsyncTask and Tasks.call()

This should replace 2 deprecations:

AsyncTask

In the Java quickstart, I replaced it with Executors.newSingleThreadExecutor().
In the Kotlin quickstart, I replaced it with a Kotlin Coroutine which I'm launching with the helper lifecycleScope KTX from the lifecycle-runtime-ktx library.

Tasks.call() - replaced with a TaskCompletionSource<T>.

@rosariopfernandes LGTM ... our first coroutines in the repo!


Nonscrollable LineChart

Is it possible to have a non-scrollable LineChart? I can't find a configuration to do so.

Did you find a solution?

Did you find a solution?

Nope. Switched to another library.

Wich do you use?

Did you find a solution?

Nope. Switched to another library.

which library are you using know?

Did you find a solution?

Nope. Switched to another library.

which library are you using know?

I've build my own library. Are you still intrested?

Did you find a solution?

Nope. Switched to another library.

which library are you using know?

I've build my own library. Are you still interested?

yes


Repeatedly running dialogs causes window rendering issues.

First noticed this in my app, Marmoset (https://github.com/sprang/marmoset), but it's reproducible in the "menu_bar" example application:

Launch "menu_bar" example
Choose File > About
Close the About dialog
Resize the main window
Choose File > About

Expected normal window behavior. Actual: Main window flashes back to previous size and is otherwise visually glitchy.
Reproducible with AboutDialog and MessageDialog in Marmoset (where the contents of the window also draw incorrectly).
Running on Arch Linux with latest Rust and Gtk-rs.

I don't know why this happens but this is going to be a bug in GTK. Please report it here: https://gitlab.gnome.org/GNOME/gtk/issues/new
On our side this can be worked around by not running a recursive main loop (i.e. not calling p.run() in this case). That's a better idea anyway because recursive main loops have interesting reentrancy problems.
I'll submit a PR for changing all the examples in a few minutes.

@GuillaumeGomez Can you close this one? The main bug here is somewhere in GTK

Sure!

Reported here: https://gitlab.gnome.org/GNOME/gtk/issues/2416


Rendering Issue with Lodestone/Malum/Core Shaders

Hi it seems when I use Ether and Core Shaders from the Malum mod that all that renders is a green box. I'm using the 1.20 dev branch but this also happens with v3.0.1
Any idea of what could be causing this? I know Core Shaders are a bit wonky sometimes.

resourcepack shaders have many incompatibilities with mods. you should use vanilla minecraft with this resourcepack

As godlander says. Use version < 3.0 like 2 or 2.5 if you want to better compatibility with mods that add additional shader effects.


Need a side navbar in the plugin

we need a side navbar to scroll between sections and will show the current or active section

Hi,
Scrollify is built in a away that makes it very easy for developers to implement things like navbars. There would really be no value for Scrollify to implement its own as it's such a simple thing to do.

Hi,
Yes, i need doted right side nav bar?
There is only having : $.scrollify.next(); & $.scrollify.previous();
Is there anyway to achieve that like i said?

Use the move method. $.scrollify.move();


Show branch name in Dashboard

Just a thought 💡
What if we showed the current branch name at the bottom of collections. Is there a reason why not to do this?

@erezrokah I would like to work on this feature. How shall I go about implementing this? Also, how can I get the branch name into react component? Thank you

The CMS has been very much about hiding git specifics from the UI (while keeping those in the config.yml under backend). Some backends are not even git based (test, fs proxy).
Not sure we would like to add this feature with the current CMS approach.

@erezrokah Understood. Thank you

Closing this issue. Don't see a good reason to break the current CMS approach of abstracting the backends


Polycom People+Content needed

Pull request has been opened to add this to the cask repo: https://github.com/caskroom/homebrew-cask/pull/11438
Once that has been accepted, it can be added.

Added in 38b9273


Is json_dto thread-safe?

What do you mean?

Are all classes in json_dto designed to be reentrant  ( like RapidJSON  )?
So I can call json_dto::from_json(), json_dto::to_json() from different threads ?

IIRC, json_dto-related stuff are stateless. It means that during the call of, for example, json_dto::from_json there is only local state, that doesn't go out of call scope. So it is safe to call json_dto::from_json or json_to::to_json from different threads.
But json_dto doesn't protect the object being serialized/deserialized. So if you call json_dto::to_json on one thread and modify the same object from another thread, then json_dto won't protect your object from data races.

Thx.


Synthetic param types

Support a reliable way to get a parameter type for #45
This throws if you try it on a local/anonymous type, but that's OK because you will never care about these constructors since you can't call them with anything useful.

Note to self: need to take into account that synthetic constructs come in 2 shapes, mandated (JLS says they must be present) and others.

Superseded by #192.


[1.15] Biofuel Generator makes free infinite power

Industrial Foregoing Version: 2.3.2
Titanium Version: 2.4.0
Biofuel Generator stops consuming biofuel and consequently generates infinite power for free.
(Note: I am using the Flux Networks mod (version 1.15.2-5.0.1-2) for FE transportation and storage).
-- Setup (4 x 3 x 3 area) --
Front View:
            [ Flux Plug ] [Flux Point]
[Flux Plug] [Biofuel Gen] [Bioreactor]
            [ Flux Plug ] [Water Condenser] [Water]

Top View:
            [ Flux Plug ] [   Water  ]
[Flux Plug] [ Flux Plug ] [Flux Point] [Water]
            [ Flux Plug ]

Reference Image:

(Note: in the image, there are two hoppers feeding items from chests into the Biofuel Reactor, but this does not affect the issue; the bug occurred before with this same setup without hoppers present).
The amount of FE being consumed is rather low (essentially only the Bioreactor and Water Condenser are drawing FE, maybe also a Latex Processor), while the output of the Biofuel Generator is an order of magnitude larger.
I am unsure at exactly what triggers the bug, but it has happened twice in two different worlds with this same setup. No more items in the Bioreactor are consumed after the bug triggers. My best guess is that it triggered when my Flux Storage and Biofuel Generator internal FE buffer became full.

Have you checked if its a Flux Plug issue?

I'm seeing the same issue.
I'm using Refined Pipes for energy transport. (https://www.curseforge.com/minecraft/mc-mods/refined-pipes)

I seem to be having a similar issue... it  seems to generate a lot more FE than it should.  Also using 1.15.2 and refined pipes.

Slightly unrelated: I added Mekanism to my pack and started using the generators it provides. Never looked back 🙂

I've recently got another person with the same issue so I replicated it using only 3 mods:
Industrial Foregoing 1.15.2-2.3.3
Titanium 1.15.2-2.4.2
For checking power generation
Mekanism 1.15.2-<IP_ADDRESS>2
Setup: https://imgur.com/a/9nZlou9
I'm pretty sure there's something wrong with the progress process that causes this to happen, it freezes at the value seen in the screenshot below.

Having the same issue with 1.16.4, using Cyclic energy pipes and Immersive Engineering wires for transport.
Forge 35.1.13
Industrial Foregoing 1.16.4-3.2.5-74675d6
Cyclic 1.16.4-1.0.0
Immersive Engineering 1.16.4-4.1.2-129

Should be fixed in the latest titanium

Should be fixed in the latest titanium


Go API?

Hi,
Nice work on Dolt and this Python API.
Is there a clean and simple Go API for performing Dolt operations and getting results back?
Obviously, the backend is Go, but I don't see an easily accessible API for a client program to utilise. Perhaps I overlooked something?
Apologies if this is not the best place to ask this question.
Thanks!

@reltuk @bheni?
The Go API is not polished but there's a way to get at it. We can polish it if you have a use case.

Yeah, we don't currently have a Go API that's expected to be stable across Dolt releases. Instead, the CLI and sql-server interfaces are (pretty) stable, although we're iterating towards better compliance with MySQL and Git and so sometimes we change things in backwards incompatible ways. The best way to interface with Dolt from Golang currently is to spawn a sql-server and connect to it using database/sql. Maybe not the most ergonomic approach for your use case...how painful does it sound?

@jimsmart let us know if this answers your question.

I'm conducting a feasibility study: client has a ETL-type workflow, including merging of datasets with a human-in-the-loop, but client is not themselves technically-minded enough to use command line tools nor write SQL, so we plan to build a UI of some kind, based around domain specific presentation regarding the dataset. Obviously, Dolt seems to offer pretty much the back-end functionality we need, so it might not be in our best interests to attempt to manually replicate this. And we are already using Go for other tools in the pipeline.
I don't mind too much about your proposal not being the most ergonomic, wrapping the CLI was already on my list as a possibility, and connecting via SQL is certainly do-able — and not something I had considered, thanks for the tip.


Start / End points - Inconsistencies from Android to iOS

Assuming the following:
<LinearGradient
	colors={
		['rgba(0,0,0,0.7)', 'rgba(136,136,136,0.5)', 'rgba(255,255,255,0)']
	}
	style={styles.linearGradient}
	start={{y: 1.0, x: 0.0}}
	end={{y: 0.5, x: 0.0}}
/>

On android, x is required, while on iOS you don't need to provide it.  Suggestion is to either make both throw or remove the requirement of both.

you bet. I'll go through and check to see if it's still the case then adjust accordingly.

Thanks @petrogad !


[Investigate] Investigate growth of paths in drives API when composable functions are supported

Related to PR: https://github.com/microsoft/OpenAPI.NET.OData/pull/431
Issue to investigate: https://github.com/microsoft/OpenAPI.NET.OData/pull/431#issuecomment-1777326987

Resolved


[Design]: User button in Users page

Is there an existing issue for this?

[X] I have searched the existing issues

SubTasks
Related to task : https://github.com/appsmithorg/appsmith/issues/19455
No response

Make the entire table cell clickable in the first column:


.executeWithSigner() not present for ContractExecuteTransaction()

Description
const contractExecTx = new ContractExecuteTransaction()
	.setContractId(contractId)
	.setGas(gasLim)
	.setPayableAmount(payableAmt)
	.setFunction(fcnName, paramsObj)
	.freezeWithSigner(wallet);
const contractExecSubmit = await contractExecTx.executeWithSigner(wallet);

Results in TypeError: contractExecTx.executeWithSigner is not a function
Note that this transaction does not need to be signed and is working as expected when using Client
Steps to reproduce
-"@hashgraph/sdk": "2.18",

See code in the next section. Generates the error below:

const contractExecSubmit = await contractExecTx.executeWithSigner(wallet);
TypeError: contractExecTx.executeWithSigner is not a function

Additional context
const myWallet = new Wallet(operatorId, operatorKey, new LocalProvider());
async function main() {

	//Execute a contract function
	const gasLimHi = 4000000;
	const payAmt = new Hbar(90);
	const fcnName = "createNonFungible";
	const fcnParams = new ContractFunctionParameters()
		.addString("myToken")
		.addString("MTK")
		.addString("No Memo")
		.addUint32(250)
		.addUint32(7700000);
	const rec = await contractExecuteFcn(contractId, gasLimHi, payAmt, fcnName, fcnParams, myWallet);
}

async function contractExecuteFcn(contractId, gasLim, payableAmt, fcnName, paramsObj, wallet) {
	//Execute a contract function (create NFT)
	const contractExecTx = new ContractExecuteTransaction()
		.setContractId(contractId)
		.setGas(gasLim)
		.setPayableAmount(payableAmt)
		.setFunction(fcnName, paramsObj)
		.freezeWithSigner(wallet);
	const contractExecSubmit = await contractExecTx.executeWithSigner(wallet);
	// const contractExecRec = await contractExecSubmit.getRecordWithSigner(wallet);

	// return contractExecRec;
	return 1;
}

Hedera network
testnet
Version
2.18
Operating system
Windows

freezeWithSigner() returns a promise, so you must await it before calling executeWithSigner which does exist: https://github.com/hashgraph/hedera-sdk-js/blob/cae04e57bdffc2dedff1f3b05aefe32d5c32e8a1/src/Executable.js#L482

Hey @danielakhterov - just for clarity, the line
const contractExecTx = new ContractExecuteTransaction()
does not require await and as you mention, it returns a promise.
The line
const contractExecSubmit = await contractExecTx.executeWithSigner(wallet);
does include await as suggested, but the error still occurs.
Does the Executable.js src file you linked apply .executeWithSigner even to the ContractExecuteTransaction class?
I know it applies for other classes I've tried before, but odd that the error occur when executing a contract function.

Executable applies to all transactions and queries, but does not apply to flows. new ContractExecuteTransaction does not require an await, but what you have above has a new ContractExecuteTransaction()...freezeWithSigner(), that last call freezeWithSigner() returns a promise and hence must be awaited before you can call executeWithSigner() on it. Hope that clears it up.


Use ranged changes trie configuration in CT functions

Extracted from #3201 . This PR introduces ChangesTrieConfigurationRange struct, which has 3 fields:

zero - it is the block where this configuration has been initially set. The first chanes trie that uses that configuration is built at block zero + One::one(). Like if configuration has been set at genesis, it will be Zero::zero() and first changes trie is built at One::one();
end - optional block, at which last changes trie that uses this configuration is built. It is also zero block of the next configuration;
config is the configuration itself.

The PR also introduces skewed digests - it is the digest that is built at the end block of configuration. It covers all blocks that have been emitted after last max-level digest has been built. Like if we have configuration { digest_interval: 3, digest_levels: 3 } (i.e. max level digest covers 27 blocks) and configuration ends at block 23, then configuration will point to: l2-digest at block#9 l2-digest at block#18, l1-digest at block#21 and blocks#{22, 23}.
This PR should have no effects until #3201 - that's why it has 4 TODOs where ChangesTrieConfigurationRange is built && it is currently built with zero = Zero::zero and end = None, meaning that there's only one configuration is allowed.

@cheme Thanks for review - I'll take a look at the rest of comments tomorrow - will mark as grumble for now.


Enable CI builds for mingw

I think this probably has no chance of building atm - shouldn't be too hard to fix. Needs to be enabled in CI so it stays in a working state.

Build script updated in "appv" branch - find SDL needs updating to work with mingw tar.gz file as it has some other structure.


Update direct-routing-plan-media-bypass.md

@NMuravlyannikov @CarolynRowe
Updating this document to reflect changes in service where we automatically change media bypass call to non media bypass call for all endpoints that don't support including 3PIP phones. I adjusted the 3PIP phone section to be about migrating to media bypass since this is still useful information. I modified the drawing and need to figure out how to upload it to replace the existing direct-routing-media-bypass-7.png.

Closed by mistake.  Reopening this pull request.


README.md - change deprecated '-h' to now favoured '-H ldap://'

According to new stable man:
http://www.openldap.org/software/man.cgi?query=ldapsearch&apropos=0&sektion=0&manpath=OpenLDAP+2.4-Release&format=html

   -H ldapuri
          Specify URI(s) referring to the ldap server(s); a list  of  URI,
          separated   by  whitespace  or  commas  is  expected;  only  the
          protocol/host/port fields are allowed.  As an exception,  if  no
          host/port  is  specified, but a DN is, the DN is used to look up
          the corresponding host(s) using the DNS SRV  records,  according
          to  RFC 2782.  The DN must be a non-empty sequence of AVAs whose
          attribute type is "dc" (domain component), and must  be  escaped
          according to RFC 2396.

   -h ldaphost
          Specify  an  alternate host on which the ldap server is running.
          Deprecated in favor of -H.

   -p ldapport
          Specify  an  alternate  TCP  port  where  the  ldap  server   is
          listening.  Deprecated in favor of -H.

Thanks !


[bug]: can't use !import_model

Getting event not found for import_model. I'm on m1 mac.
Was simply trying to follow the docs - https://invoke-ai.github.io/InvokeAI/installation/INSTALLING_MODELS/
robert@Roberts-MacBook-Pro InvokeAI % conda activate invokeai
(invokeai) robert@Roberts-MacBook-Pro InvokeAI % !import_model /Users/robert/Documents/InvokeAI/models/ldm/stable-diffusion-v1/model.ckpt
zsh: event not found: import_model
(invokeai) robert@Roberts-MacBook-Pro InvokeAI %

@Roblouisck Looks like you did not execute this in the invokeai command line client and just executed it in your normal terminal.
Looks like you may have missed step 2 of the Installation via the CLI
Launch the invoke.py CLI with python scripts/invoke.py.
You terminal should show something like  invoke>
Enter your command there.

Closing as the problem is clear and solution in comments - please let us know if there are other issues by creating a new GH issue.


Anyone knows the opcodes for chinese client 5.45?

chinese version is patch 5.45 now.Anyone knows the opcodes for it?

{
"C2S_ActionRequest": "0x034c",
"C2S_ActionRequestGroundTargeted": "0x00eb",
"S2C_ActionEffect01": "0x0240",
"S2C_ActionEffect08": "0x0376",
"S2C_ActionEffect16": "0x035b",
"S2C_ActionEffect24": "0x020b",
"S2C_ActionEffect32": "0x0353",
"S2C_ActorCast": "0x0392",
"S2C_ActorControl": "0x03c3",
"S2C_ActorControlSelf": "0x0077",
"S2C_AddStatusEffect": "0x0212"
}
I just got it

{
"C2S_ActionRequest": "0x034c",
"C2S_ActionRequestGroundTargeted": "0x00eb",
"S2C_ActionEffect01": "0x0240",
"S2C_ActionEffect08": "0x0376",
"S2C_ActionEffect16": "0x035b",
"S2C_ActionEffect24": "0x020b",
"S2C_ActionEffect32": "0x0353",
"S2C_ActorCast": "0x0392",
"S2C_ActorControl": "0x03c3",
"S2C_ActorControlSelf": "0x0077",
"S2C_AddStatusEffect": "0x0212"
}
I just got it

Hi, may I ask what's the difference of "C2S_ActionRequest" and "RequestUseAction"? The config file of patch 5.41 I downloaded before only contains the later one.

Upgrade to latest version you will know,leave old version

Hasn't this been resolved?

Hasn't this been resolved?

https://github.com/Soreepeong/XivAlexander/issues/91 Try this


Garbage collection bug in sg_shift

https://hackerone.com/hexodus reported the following bug.
Note that mruby must be built with the MRB_GC_STRESS for the crash to occur.

The following input demonstrates a crash:
a = {'a' => 'A'}
b = a.shift
printf b[1]

ASAN report:
mr@mruby:~/test/new/mruby/bin$ ./mruby poc.rb
AddressSanitizer:DEADLYSIGNAL
=================================================================
==27265==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x00000065b222 bp 0x7ffd2fa5a270 sp 0x7ffd2fa59ee0 T0)
==27265==The signal is caused by a READ memory access.
==27265==Hint: address points to the zero page.
    #0 0x65b221 in mrb_str_format /home/mr/test/new/mruby/mrbgems/mruby-sprintf/src/sprintf.c:569:28
    #1 0x65ae07 in mrb_f_sprintf /home/mr/test/new/mruby/mrbgems/mruby-sprintf/src/sprintf.c:517:12
    #2 0x55eaae in mrb_vm_exec /home/mr/test/new/mruby/src/vm.c:1463:18
    #3 0x55367d in mrb_vm_run /home/mr/test/new/mruby/src/vm.c:972:12
    #4 0x57545b in mrb_top_run /home/mr/test/new/mruby/src/vm.c
    #5 0x5a1ab3 in mrb_load_exec /home/mr/test/new/mruby/mrbgems/mruby-compiler/core/parse.y:6013:7
    #6 0x4e90bb in main /home/mr/test/new/mruby/mrbgems/mruby-bin-mruby/tools/mruby/mruby.c:280:11
    #7 0x7f7fc86f7f44 in __libc_start_main /build/eglibc-ripdx6/eglibc-2.19/csu/libc-start.c:287
    #8 0x41b05b in _start (/home/mr/test/new/mruby/bin/mruby+0x41b05b)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV /home/mr/test/new/mruby/mrbgems/mruby-sprintf/src/sprintf.c:569:28 in mrb_str_format
==27265==ABORTING

Root cause:
Since the key is set to mrb_undef_value() in sg_shift(), the key contained string memory would be freed by the GC.
After that it can access the freed null pointer.
Fix:
Add the following two lines between sg_shift() and mrb_assoc_new() in mrb_hash_shift().
mrb_gc_protect(mrb, del_key);
mrb_gc_protect(mrb, del_val);

Impact
Crash on mruby

A problem can also be observed even without MRB_GC_STRESS. When running the following code, it's apparent that the output is not always A:
loop do
  a = {'a' => 'A'}
  b = a.shift
  printf b[1]
end

AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAPAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA�AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA�AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA


Tracing with puffin!

Amazing framework. Thanks a lot to the the puffin developers. Thanks to @emilk for helping out
How to use it is described in the Readme:
cargo run --example create_test_file
cargo run --features trace -- binocle_test_file

In another terminal run:
cargo install puffin_viewer
puffin_viewer --url <IP_ADDRESS>:8585

FYI: @sharkdp

The build failure is persistent also on my forked repo: https://github.com/siedentop/binocle/runs/4194742703?check_suite_focus=true
Issue is on "windows-2019 (x86_64-pc-windows-gnu)", and zstd custom build command is failing.
I would like to just disable the tracing feature for this platform then.

Thank you for making me aware of this framework. This sounds really helpful. However, I'd rather not include tracing support by default here in this project - even if it's hidden behind a feature flag. I'd would try to avoid having to maintain this in the future. This PR/branch is still valuable for documentation purposes though!
Remark: if you want to make code like this optional, you can/should also make the necessary dependencies optional: https://doc.rust-lang.org/cargo/reference/features.html#optional-dependencies

Sounds good to me!


OSD-17611 : Adding BP-CLI release cycle info

What type of PR is this?
documentation
(bug/feature/cleanup/documentation)
What this PR does / Why we need it?
As a part of OSD-17611, adds info on BP-CLI release cycle
Which Jira/Github issue(s) does this PR fix?
OSD-17611
Resolves #
Special notes for your reviewer
Pre-checks (if applicable)

[ ] Ran unit tests locally
[ ] Validated the changes in a cluster
[ ] Included documentation changes with PR

Codecov Report

Merging #189 (628fa0b) into main (691eb6f) will not change coverage.
The diff coverage is n/a.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #189   +/-   ##
=======================================
  Coverage   51.11%   51.11%
=======================================
  Files          51       51
  Lines        3365     3365
=======================================
  Hits         1720     1720
  Misses       1364     1364
  Partials      281      281

/lgtm


Xarray open_mfdataset with engine Zarr

Work on enabling xr.open_dataset(..., engine="zarr"), to replace complement xr.open_zarr. This also allows `xr.open_mfdataset(..., engine="zarr") to be used.
Note: Credit should be given to @Mikejmnez, I'm just continuing this on from #4003.

[x] Closes #3668, closes #4003
[x] Tests added
[x] Passes isort -rc . && black . && mypy . && flake8
[x] User visible changes (including notable bug fixes) are documented in whats-new.rst

Hello @weiji14! Thanks for opening this PR.  There doesn't seem to be an obvious reason why this test is failing, since 1) this test isn't for Zarr and 2) this test shouldn't be affected by the new if blocks checking if engine=="zarr". Will need to double check the logic here.

Nevermind, I found it. There was an if that should have been an elif. Onward to the next error...

Is it ok to drop the deprecated auto_chunk tests here in this PR (or leave it to another PR)? The deprecation warning was first added in https://github.com/pydata/xarray/pull/2530/commits/ae4cf0ab19b3e563bde90a48b3e6ee615930d4a1, and I see that auto_chunk was used back in v0.12.1 at http://xarray.pydata.org/en/v0.12.1/generated/xarray.open_zarr.html.

Four more failures, something to do with dask?

TestZarrDictStore.test_vectorized_indexing - IndexError: only slices with step >= 1 are supported
TestZarrDictStore.test_manual_chunk - ZeroDivisionError: integer division or modulo by zero
TestZarrDirectoryStore.test_vectorized_indexing - IndexError: only slices with step >= 1 are supported
TestZarrDirectoryStore.test_manual_chunk - ZeroDivisionError: integer division or modulo by zero

=================================== FAILURES ===================================
__________________ TestZarrDictStore.test_vectorized_indexing __________________
self = <xarray.tests.test_backends.TestZarrDictStore object at 0x7f5832433940>
@pytest.mark.xfail(
    not has_dask,
    reason="the code for indexing without dask handles negative steps in slices incorrectly",
)
def test_vectorized_indexing(self):
    in_memory = create_test_data()
    with self.roundtrip(in_memory) as on_disk:
        indexers = {
            "dim1": DataArray([0, 2, 0], dims="a"),
            "dim2": DataArray([0, 2, 3], dims="a"),
        }
        expected = in_memory.isel(**indexers)
        actual = on_disk.isel(**indexers)
        # make sure the array is not yet loaded into memory
        assert not actual["var1"].variable._in_memory
        assert_identical(expected, actual.load())
        # do it twice, to make sure we're switched from
        # vectorized -> numpy when we cached the values
        actual = on_disk.isel(**indexers)
        assert_identical(expected, actual)

    def multiple_indexing(indexers):
        # make sure a sequence of lazy indexings certainly works.
        with self.roundtrip(in_memory) as on_disk:
            actual = on_disk["var3"]
            expected = in_memory["var3"]
            for ind in indexers:
                actual = actual.isel(**ind)
                expected = expected.isel(**ind)
                # make sure the array is not yet loaded into memory
                assert not actual.variable._in_memory
            assert_identical(expected, actual.load())

    # two-staged vectorized-indexing
    indexers = [
        {
            "dim1": DataArray([[0, 7], [2, 6], [3, 5]], dims=["a", "b"]),
            "dim3": DataArray([[0, 4], [1, 3], [2, 2]], dims=["a", "b"]),
        },
        {"a": DataArray([0, 1], dims=["c"]), "b": DataArray([0, 1], dims=["c"])},
    ]
    multiple_indexing(indexers)

    # vectorized-slice mixed
    indexers = [
        {
            "dim1": DataArray([[0, 7], [2, 6], [3, 5]], dims=["a", "b"]),
            "dim3": slice(None, 10),
        }
    ]
    multiple_indexing(indexers)

    # vectorized-integer mixed
    indexers = [
        {"dim3": 0},
        {"dim1": DataArray([[0, 7], [2, 6], [3, 5]], dims=["a", "b"])},
        {"a": slice(None, None, 2)},
    ]
    multiple_indexing(indexers)

    # vectorized-integer mixed
    indexers = [
        {"dim3": 0},
        {"dim1": DataArray([[0, 7], [2, 6], [3, 5]], dims=["a", "b"])},
        {"a": 1, "b": 0},
    ]
    multiple_indexing(indexers)

    # with negative step slice.
    indexers = [
        {
            "dim1": DataArray([[0, 7], [2, 6], [3, 5]], dims=["a", "b"]),
            "dim3": slice(-1, 1, -1),
        }
    ]

  multiple_indexing(indexers)

xarray/tests/test_backends.py:686:

xarray/tests/test_backends.py:642: in multiple_indexing
assert_identical(expected, actual.load())
xarray/core/dataarray.py:814: in load
ds = self._to_temp_dataset().load(**kwargs)
xarray/core/dataset.py:666: in load
v.load()
xarray/core/variable.py:381: in load
self._data = np.asarray(self._data)
/usr/share/miniconda/envs/xarray-tests/lib/python3.6/site-packages/numpy/core/numeric.py:501: in asarray
return array(a, dtype, copy=False, order=order)
xarray/core/indexing.py:677: in array
self._ensure_cached()
xarray/core/indexing.py:674: in _ensure_cached
self.array = NumpyIndexingAdapter(np.asarray(self.array))
/usr/share/miniconda/envs/xarray-tests/lib/python3.6/site-packages/numpy/core/numeric.py:501: in asarray
return array(a, dtype, copy=False, order=order)
xarray/core/indexing.py:653: in array
return np.asarray(self.array, dtype=dtype)
/usr/share/miniconda/envs/xarray-tests/lib/python3.6/site-packages/numpy/core/numeric.py:501: in asarray
return array(a, dtype, copy=False, order=order)
xarray/core/indexing.py:557: in array
return np.asarray(array[self.key], dtype=None)
xarray/backends/zarr.py:57: in getitem
return array[key.tuple]
/usr/share/miniconda/envs/xarray-tests/lib/python3.6/site-packages/zarr/core.py:572: in getitem
return self.get_basic_selection(selection, fields=fields)
/usr/share/miniconda/envs/xarray-tests/lib/python3.6/site-packages/zarr/core.py:698: in get_basic_selection
fields=fields)
/usr/share/miniconda/envs/xarray-tests/lib/python3.6/site-packages/zarr/core.py:738: in _get_basic_selection_nd
indexer = BasicIndexer(selection, self)
/usr/share/miniconda/envs/xarray-tests/lib/python3.6/site-packages/zarr/indexing.py:279: in init
dim_indexer = SliceDimIndexer(dim_sel, dim_len, dim_chunk_len)
/usr/share/miniconda/envs/xarray-tests/lib/python3.6/site-packages/zarr/indexing.py:107: in init
err_negative_step()

def err_negative_step():

  raise IndexError('only slices with step >= 1 are supported')

E       IndexError: only slices with step >= 1 are supported
/usr/share/miniconda/envs/xarray-tests/lib/python3.6/site-packages/zarr/errors.py:55: IndexError
_____________________ TestZarrDictStore.test_manual_chunk ______________________
self = <xarray.tests.test_backends.TestZarrDictStore object at 0x7f5832b80cf8>
@requires_dask
@pytest.mark.filterwarnings("ignore:Specified Dask chunks")
def test_manual_chunk(self):
    original = create_test_data().chunk({"dim1": 3, "dim2": 4, "dim3": 3})

    # All of these should return non-chunked arrays
    NO_CHUNKS = (None, 0, {})
    for no_chunk in NO_CHUNKS:
        open_kwargs = {"chunks": no_chunk}

      with self.roundtrip(original, open_kwargs=open_kwargs) as actual:

xarray/tests/test_backends.py:1594:

/usr/share/miniconda/envs/xarray-tests/lib/python3.6/contextlib.py:81: in enter
return next(self.gen)
xarray/tests/test_backends.py:1553: in roundtrip
with self.open(store_target, **open_kwargs) as ds:
/usr/share/miniconda/envs/xarray-tests/lib/python3.6/contextlib.py:81: in enter
return next(self.gen)
xarray/tests/test_backends.py:1540: in open
with xr.open_dataset(store_target, engine="zarr", **kwargs) as ds:
xarray/backends/api.py:587: in open_dataset
ds = maybe_decode_store(store, chunks)
xarray/backends/api.py:511: in maybe_decode_store
for k, v in ds.variables.items()
xarray/backends/api.py:511: in
for k, v in ds.variables.items()
xarray/backends/zarr.py:398: in maybe_chunk
var = var.chunk(chunk_spec, name=name2, lock=None)
xarray/core/variable.py:1007: in chunk
data = da.from_array(data, chunks, name=name, lock=lock, **kwargs)
/usr/share/miniconda/envs/xarray-tests/lib/python3.6/site-packages/dask/array/core.py:2712: in from_array
chunks, x.shape, dtype=x.dtype, previous_chunks=previous_chunks
/usr/share/miniconda/envs/xarray-tests/lib/python3.6/site-packages/dask/array/core.py:2447: in normalize_chunks
(),
/usr/share/miniconda/envs/xarray-tests/lib/python3.6/site-packages/dask/array/core.py:2445: in
for s, c in zip(shape, chunks)
/usr/share/miniconda/envs/xarray-tests/lib/python3.6/site-packages/dask/array/core.py:954: in blockdims_from_blockshape
for d, bd in zip(shape, chunks)

.0 = <zip object at 0x7f58332d9d48>
    ((bd,) * (d // bd) + ((d % bd,) if d % bd else ()) if d else (0,))

  for d, bd in zip(shape, chunks)

)

E   ZeroDivisionError: integer division or modulo by zero
/usr/share/miniconda/envs/xarray-tests/lib/python3.6/site-packages/dask/array/core.py:954: ZeroDivisionError
_______________ TestZarrDirectoryStore.test_vectorized_indexing ________________
self = <xarray.tests.test_backends.TestZarrDirectoryStore object at 0x7f5832a08a20>
@pytest.mark.xfail(
    not has_dask,
    reason="the code for indexing without dask handles negative steps in slices incorrectly",
)
def test_vectorized_indexing(self):
    in_memory = create_test_data()
    with self.roundtrip(in_memory) as on_disk:
        indexers = {
            "dim1": DataArray([0, 2, 0], dims="a"),
            "dim2": DataArray([0, 2, 3], dims="a"),
        }
        expected = in_memory.isel(**indexers)
        actual = on_disk.isel(**indexers)
        # make sure the array is not yet loaded into memory
        assert not actual["var1"].variable._in_memory
        assert_identical(expected, actual.load())
        # do it twice, to make sure we're switched from
        # vectorized -> numpy when we cached the values
        actual = on_disk.isel(**indexers)
        assert_identical(expected, actual)

    def multiple_indexing(indexers):
        # make sure a sequence of lazy indexings certainly works.
        with self.roundtrip(in_memory) as on_disk:
            actual = on_disk["var3"]
            expected = in_memory["var3"]
            for ind in indexers:
                actual = actual.isel(**ind)
                expected = expected.isel(**ind)
                # make sure the array is not yet loaded into memory
                assert not actual.variable._in_memory
            assert_identical(expected, actual.load())

    # two-staged vectorized-indexing
    indexers = [
        {
            "dim1": DataArray([[0, 7], [2, 6], [3, 5]], dims=["a", "b"]),
            "dim3": DataArray([[0, 4], [1, 3], [2, 2]], dims=["a", "b"]),
        },
        {"a": DataArray([0, 1], dims=["c"]), "b": DataArray([0, 1], dims=["c"])},
    ]
    multiple_indexing(indexers)

    # vectorized-slice mixed
    indexers = [
        {
            "dim1": DataArray([[0, 7], [2, 6], [3, 5]], dims=["a", "b"]),
            "dim3": slice(None, 10),
        }
    ]
    multiple_indexing(indexers)

    # vectorized-integer mixed
    indexers = [
        {"dim3": 0},
        {"dim1": DataArray([[0, 7], [2, 6], [3, 5]], dims=["a", "b"])},
        {"a": slice(None, None, 2)},
    ]
    multiple_indexing(indexers)

    # vectorized-integer mixed
    indexers = [
        {"dim3": 0},
        {"dim1": DataArray([[0, 7], [2, 6], [3, 5]], dims=["a", "b"])},
        {"a": 1, "b": 0},
    ]
    multiple_indexing(indexers)

    # with negative step slice.
    indexers = [
        {
            "dim1": DataArray([[0, 7], [2, 6], [3, 5]], dims=["a", "b"]),
            "dim3": slice(-1, 1, -1),
        }
    ]

  multiple_indexing(indexers)

xarray/tests/test_backends.py:686:

xarray/tests/test_backends.py:642: in multiple_indexing
assert_identical(expected, actual.load())
xarray/core/dataarray.py:814: in load
ds = self._to_temp_dataset().load(**kwargs)
xarray/core/dataset.py:666: in load
v.load()
xarray/core/variable.py:381: in load
self._data = np.asarray(self._data)
/usr/share/miniconda/envs/xarray-tests/lib/python3.6/site-packages/numpy/core/numeric.py:501: in asarray
return array(a, dtype, copy=False, order=order)
xarray/core/indexing.py:677: in array
self._ensure_cached()
xarray/core/indexing.py:674: in _ensure_cached
self.array = NumpyIndexingAdapter(np.asarray(self.array))
/usr/share/miniconda/envs/xarray-tests/lib/python3.6/site-packages/numpy/core/numeric.py:501: in asarray
return array(a, dtype, copy=False, order=order)
xarray/core/indexing.py:653: in array
return np.asarray(self.array, dtype=dtype)
/usr/share/miniconda/envs/xarray-tests/lib/python3.6/site-packages/numpy/core/numeric.py:501: in asarray
return array(a, dtype, copy=False, order=order)
xarray/core/indexing.py:557: in array
return np.asarray(array[self.key], dtype=None)
xarray/backends/zarr.py:57: in getitem
return array[key.tuple]
/usr/share/miniconda/envs/xarray-tests/lib/python3.6/site-packages/zarr/core.py:572: in getitem
return self.get_basic_selection(selection, fields=fields)
/usr/share/miniconda/envs/xarray-tests/lib/python3.6/site-packages/zarr/core.py:698: in get_basic_selection
fields=fields)
/usr/share/miniconda/envs/xarray-tests/lib/python3.6/site-packages/zarr/core.py:738: in _get_basic_selection_nd
indexer = BasicIndexer(selection, self)
/usr/share/miniconda/envs/xarray-tests/lib/python3.6/site-packages/zarr/indexing.py:279: in init
dim_indexer = SliceDimIndexer(dim_sel, dim_len, dim_chunk_len)
/usr/share/miniconda/envs/xarray-tests/lib/python3.6/site-packages/zarr/indexing.py:107: in init
err_negative_step()

def err_negative_step():

  raise IndexError('only slices with step >= 1 are supported')

E       IndexError: only slices with step >= 1 are supported
/usr/share/miniconda/envs/xarray-tests/lib/python3.6/site-packages/zarr/errors.py:55: IndexError
___________________ TestZarrDirectoryStore.test_manual_chunk ___________________
self = <xarray.tests.test_backends.TestZarrDirectoryStore object at 0x7f5831763ef0>
@requires_dask
@pytest.mark.filterwarnings("ignore:Specified Dask chunks")
def test_manual_chunk(self):
    original = create_test_data().chunk({"dim1": 3, "dim2": 4, "dim3": 3})

    # All of these should return non-chunked arrays
    NO_CHUNKS = (None, 0, {})
    for no_chunk in NO_CHUNKS:
        open_kwargs = {"chunks": no_chunk}

      with self.roundtrip(original, open_kwargs=open_kwargs) as actual:

xarray/tests/test_backends.py:1594:

/usr/share/miniconda/envs/xarray-tests/lib/python3.6/contextlib.py:81: in enter
return next(self.gen)
xarray/tests/test_backends.py:1553: in roundtrip
with self.open(store_target, **open_kwargs) as ds:
/usr/share/miniconda/envs/xarray-tests/lib/python3.6/contextlib.py:81: in enter
return next(self.gen)
xarray/tests/test_backends.py:1540: in open
with xr.open_dataset(store_target, engine="zarr", **kwargs) as ds:
xarray/backends/api.py:587: in open_dataset
ds = maybe_decode_store(store, chunks)
xarray/backends/api.py:511: in maybe_decode_store
for k, v in ds.variables.items()
xarray/backends/api.py:511: in
for k, v in ds.variables.items()
xarray/backends/zarr.py:398: in maybe_chunk
var = var.chunk(chunk_spec, name=name2, lock=None)
xarray/core/variable.py:1007: in chunk
data = da.from_array(data, chunks, name=name, lock=lock, **kwargs)
/usr/share/miniconda/envs/xarray-tests/lib/python3.6/site-packages/dask/array/core.py:2712: in from_array
chunks, x.shape, dtype=x.dtype, previous_chunks=previous_chunks
/usr/share/miniconda/envs/xarray-tests/lib/python3.6/site-packages/dask/array/core.py:2447: in normalize_chunks
(),
/usr/share/miniconda/envs/xarray-tests/lib/python3.6/site-packages/dask/array/core.py:2445: in
for s, c in zip(shape, chunks)
/usr/share/miniconda/envs/xarray-tests/lib/python3.6/site-packages/dask/array/core.py:954: in blockdims_from_blockshape
for d, bd in zip(shape, chunks)

.0 = <zip object at 0x7f58324b7f48>
    ((bd,) * (d // bd) + ((d % bd,) if d % bd else ()) if d else (0,))

  for d, bd in zip(shape, chunks)

)

E   ZeroDivisionError: integer division or modulo by zero
/usr/share/miniconda/envs/xarray-tests/lib/python3.6/site-packages/dask/array/core.py:954: ZeroDivisionError

I wonder if it's really worth deprecating open_zarr(). open_dataset(..., engine='zarr') is a bit more verbose, especially with backend_kwargs to pass optional arguments. It seems pretty harmless to keep open_zarr() around, especially if it's just an alias for open_datraset(engine='zarr').

We could also automatically detect zarr stores in open_dataset if:

the argument inherits from collections.abc.Mapping, and
it contains a key '.zgroup', corresponding to zarr metadata.

As for the annoyance of needing to write backend_kwargs={"consolidated": True}, I wonder if we could detect this automatically by checking for the existence of a .zmetadata key?

I wonder if it's really worth deprecating open_zarr(). open_dataset(..., engine='zarr') is a bit more verbose, especially with backend_kwargs to pass optional arguments. It seems pretty harmless to keep open_zarr() around, especially if it's just an alias for open_datraset(engine='zarr').

Depends on which line in the Zen of Python you want to follow - "Simple is better than complex", or "There should be one-- and preferably only one --obvious way to do it". From a maintenance perspective, it's balancing the cost of a deprecation cycle vs writing code that tests both instances I guess.

We could also automatically detect zarr stores in open_dataset without requiring engine='zarr' if:

the argument inherits from collections.abc.Mapping, and
it contains a key '.zgroup', corresponding to zarr metadata.

As for the annoyance of needing to write backend_kwargs={"consolidated": True}, I wonder if we could detect this automatically by checking for the existence of a .zmetadata key? This would add a small amount of overhead (one file access) but this probably would not be prohibitively expensive.

These are some pretty good ideas. I also wonder if there's a way to mimic the dataset identifiers like in rasterio, something like xr.open_dataset("zarr:some_zarrfile.zarr"). Feels a lot more like fsspec's url chaining too.
Counter-argument would be that the cyclomatic complexity of open_dataset is already too high, and it really should be refactored. Especially if new backend engines come online (e.g. #4142).

I wonder if it's really worth deprecating open_zarr(). open_dataset(..., engine='zarr') is a bit more verbose, especially with backend_kwargs to pass optional arguments. It seems pretty harmless to keep open_zarr() around, especially if it's just an alias for open_datraset(engine='zarr').

I agree. I think we should keep open_zarr around.
As reminder (because it took me a while to remember!), one goal with this refactor is to have open_mfdataset work with all backends (including zarr and rasterio) by specifying the engine kwarg.

As a note I'm working on implementing zarr spec v3 in zarr-python, still deciding how we want to handle the new spec/API.
If there are any changes that you would like or dislike in an API, feedback is welcome.

Just to note down a few things:

The deprecated "auto_chunk" kwarg was removed
open_zarr uses chunks="auto" by default, whereas open_dataset uses chunks=None (see my comment inline)

The different default chunk behaviour (point 2) is worth raising, and it might be best to postpone the deprecation of open_zarr until the next release, so that there's time to discuss what we want the default setting to be (None or auto).

These all sound good to me!
I agree that we shouldn't change the default behavior for open_dataset, and should keep open_zarr around for now -- there is no urgent need to deprecate it.

Thanks @weiji14 and @Mikejmnez for your contribution!

Note that zarr.open* now works with fsspec URLs (in master)

Thanks @weiji14  and @Mikejmnez . This is a great contribution.


How to remove credentials and authenticate again

https://docs.microsoft.com/en-us/azure/devops/organizations/accounts/disconnect-organization-from-azure-ad?view=azure-devops
What about telling how to remove old credentials from Windows and authenticate to Git repository again?
It must be specified on that page too.

This solution helped me
https://stackoverflow.com/posts/52749883/edit

Assigning to @chcomley.

Hi @xperiandri, I'm glad you found something that worked for you. Let me share another article that is a good starting point: https://docs.microsoft.com/en-us/azure/devops/repos/git/set-up-credential-managers?view=azure-devops. There are more specific manuals to the Git credential manager that are linked to within this doc, as well. I'll be closing this issue, however do let me know if you have any further questions or comments. Thanks so much for your feedback.

@chcomley I can't find how that article can help to solve my issue


chore: used named arguments instead of @DecidableRel

Enables https://github.com/leanprover/lean4/pull/6341, but seems independently neutral/positive.

That's one way to make it forward-compatible.
bors merge


path not found http://disy.github.com/perclipse/

While installing plugin in eclipse neon, getting error http://disy.github.com/perclipse/ path not found.

indeed, the path changed to http://sebastiangraf.github.io/perclipse/
Here did you find the old link? Eclipse marketplace?

deprecated links in eclipse marketplace: https://marketplace.eclipse.org/content/perclipse-perfidix-eclipse-plugin

links fixed.


[Question] What is the non-deprecated alternative for "cpu_offload"?

Environment details
python 3.8
deepspeed==0.6.1
pytorch-lightning==1.6.0
Problem description
I have been trying to fit a large model into memory in pytorch lightning using deepspeed. When I set my config with "cpu_offload": True, I get a substantial decrease in used memory (and increase to epoch time) as expected. However, as "cpu_offload" is deprecated, I tried to use "offload_optimizer", as recommended in https://www.deepspeed.ai/docs/config-json/, but it doesn't seem to do anything. Am I missing something?
Relevant part of my configs:
With cpu_offload
"zero_optimization": {
"stage": 2,
"allgather_partitions": True,
"allgather_bucket_size": 1e7,
"reduce_scatter": True,
"reduce_bucket_size": 1e7,
"overlap_comm": True,
"contiguous_gradients": True,
"cpu_offload": True
},
With offload_optimizer
"zero_optimization": {
"stage": 2,
"allgather_partitions": True,
"allgather_bucket_size": 1e7,
"reduce_scatter": True,
"reduce_bucket_size": 1e7,
"overlap_comm": True,
"contiguous_gradients": True,
},
"offload_optimizer": {
"device": "cpu",
"pin_memory": True,
"buffer_count": 4,
"fast_init": False
},

@Pringled, thanks for raising this issue. Can you try moving the "offload_optimizer" entry into the "zero_optimization" one?

@Pringled, more specifically should look like
 "zero_optimization" : {
  ...
      "offload_optimizer": {
          "device": "cpu",
          "pin_memory": True,
          "buffer_count": 4,
         "fast_init": False
    },
    ....
}

That was indeed the issue, thank you for the quick reply!

Thanks for quickly confirming resolution. This issue has it clear that the docs is confusing by not explicitly showing that "offload_optimizer" is a subfield of "zero_optimization". So that needs to be fixed.


Improve Python dependencies

Add Python libraries needed for FLARE-EDU
Ensure we install the python libraries in Python 9 with -py 39
Move stringsifter to its own package as it is not possible to install it in Python 3.9:
https://github.com/mandiant/stringsifter/issues/31
I have taken into account the following bugs to install stringsifter:

https://github.com/mandiant/stringsifter/issues/29
https://github.com/mandiant/stringsifter/issues/30

@mandiant/flare-vm does someone know how I can get rid of the following warning which is making the action fail? And why is the warning interpreted as an error?
WARNING: The script f2py.exe is installed in 'C:\hostedtoolcache\windows\Python\3.8.10\x64\Scripts' which is not on PATH.

I don't get this locally :confused:

f2py.exe appears to be part of numpy, not sure the install fails because of this error though

I have removed the update of stringsifter from this PR as the rest can be merged independently.


[Release] sdk/resourcemanager/hybridconnectivity/armhybridconnectivity/0.1.0

…y/0.1.0 generation from spec commit: 3c5135f8325eaf79be7b137cae363bdcba028c17

[ ] The purpose of this PR is explained in this or a referenced issue.
[ ] The PR does not update generated files.

These files are managed by the codegen framework at Azure/autorest.go.

[ ] Tests are included and/or updated for code changes.
[ ] Updates to CHANGELOG.md are included.
[ ] MIT license headers are included in each file.

/azp run prepare-pipelines

/azp run go - armhybridconnectivity - ci

/check-enforcer override


Fix for connection restart

This pull request fixes the following issues.

Allow re-registration (potentially with alternate credentials)
Recover from loss of connection to SIP server
Refactor Transaction classes for better maintainability
Move Transactions from a single source file to individual source files in a sub folder
Infer types where possible
Introduce Enum for Transaction State Machine's State

Just coming back from vacation, I will test this PR tomorrow.


Add proxy contract and upgrades lib

This PR supercedes #129 and fixes #124. This PR proposes to include upgradeable dummy contract implementations for testing (which is inspired by OpenZeppelin's Solidity implementation). Further, this PR includes an ERC20_Upgradeable preset.

I didn't mean we should not have the ERC20_Upgradeable tested! I just wanted to highlight the importance of having separate upgrades and proxy tests.

Ah thank you for clarifying :) on it!


Possible Memory Leak

-->
Description
Had previously noticed serious occasional heavy frame rate stuttering in game during Savage raids (from 141 FPS down to 60-70 fps then back to 141), decided to investigate.  Initially thought it was a DX11 issue tied to RTX GPU automatic overlocking logic.  Tried lowering fps cap via Nvidia GFCP from 141 to 131 to lower GPU load.  Played with that and no ACT, saw zero stuttering issues.
Loaded FFXIV with same 131 fps cap, ACT w/ Cactbot ENABLED.  Noticed as went around FFXIV world today that RAM usage rose from 6 GB to approximately 12.7 GB.  As soon as I closed ACT, RAM usage dropped back down to approx 6 GB RAM.  I suspect this is the issue as potentially maxed RAM over ~3 hours raiding.
Reloaded FFXIV with same 131 fps cap, ACT, and Cactbot DISABLED.  RAM has been steady around 7 GB for last couple hours.
Additional information
SYSTEM INFO:
WINDOWS 10 v.2004 build 19041.746
INTEL Core i5-9600K @ 5000 MHz
GIGABYTE Z390 PRO mobo
16 GB G.SKILL Ripjaws V RAM @ 3200 MHz
GIGABYTE RTX 2060 Super Gaming OC on Nvidia driver 461.40
CORSAIR TX650M PSU
If available, please include any logs or screenshots depicting the issue you're experiencing.

When you say "cactbot enabled" which cactbot overlays are you running?

Hi there,
The only overlay I was running was the raid overlay for the timer bars and
alerts w/ the TTS option.
Matt
On Tue, Feb 2, 2021 at 1:22 AM Adrienne Walker<EMAIL_ADDRESS>wrote:

When you say "cactbot enabled" which cactbot overlays are you running?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/quisquous/cactbot/issues/2424#issuecomment-771395987,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ASVPV5FW4AKTO4VEXJYME2DS46KY7ANCNFSM4W3Q5BMA
.

If this happens again, can you open up task manager and see which process is taking up the memory? Do you only have one overlay running? If so, if you reload that overlay via the OverlayPlugin reload button when memory is high, does that fix it?

I remembered when I afked overnight and left my ACT on, the memory usage was excessive when I got back. Here is the screenshot I saved (a few weeks ago). I might have the same problem as @Madscientist351.
However, I do not know whether this is cactbot's problem, because I had raid boss trigger+timeline, eureka, jobs overlay along with a few other plugins running. Also this only happens intermittently.

Yeah, this is not that useful to just see ACT memory on its own.  Can you look at the subprocesses next time? You might also be able to reload overlays one by one and see if that frees the memory and narrow down which one it is.

I had another excessive memory usage (~3GB) on CN server yesterday (raided a few hours, then afked at Limsa overnight). Reloading all overlays did not seem to free the excessive memory in my case. Disabling both OverlayPlugin.dll and CactbotOverlay.dll freed only ~300MB memory (reported by task manager), so I guess the leak is probably not from cactbot? Let me know if there is any update or suggestions.
ACT: <IP_ADDRESS>3 cactbot: <IP_ADDRESS> overlay: <IP_ADDRESS> ffxiv parsing plugin: <IP_ADDRESS> game: CN client patch 5.35

I looked into this a bit more today while waiting for raid, and I couldn't track down any memory leak that would cause this sort of jump in memory usage.
There does seem to be an extremely minor leak (potentially) upstream in OverlayPlugin which I'll detail below, but I'm not sure it's even something that can be fixed.
Whenever an overlay handler event is triggered, it looks like CEF is JIT compiling the JS for the event that gets passed from C#, and that JIT'd code isn't ever released. This will result in approximately 3kb of memory leaked per event called. I verified this by taking snapshots of memory over an approximately hour-long period, and tracking a specific JIT'd code by ID. It persisted in memory through multiple zone changes and even a swap to a different character.
The JIT'd code remained even after manually triggering GC in dev tools. It is specific to a given overlay, so reloading the overlay will clear that memory properly (tested and verified).
Here's an example screenshot so you know what to look for if you're interested in following up on it. I'm not including the full stack because it contains my public IPv4 address but it eventually tracks back to native_context.

This issue is pretty old and things have changed a lot in OverlayPlugin (new CEF version) and in cactbot (fixes for leaks for timers).  If this is still happening, please follow up here with more info.


cannot set MessageBoxInput default value

Please add property to set default value

check 1.5.3

Thanks again for your work. It works as expected.


Crash on My stats

If a user doesn't have any stats (probably the reason anyway), clicking on My stats will cause a crash.

User that breaks shit: https://hubbit.chalmers.it/stats/api

Doesn't fork for json though :-1:


fix: fixes incorrect handler signature; also checks for null since it…

… appears that sometimes the tween parameter is null when the handler is called

All good! Good fix, thanks!


docs: added disclaimer and table for Kubernetes support and cluster size

Closes #1612
This is the parent issue for this Pull request https://github.com/keptn/keptn/issues/9525 which is an official issue on the Keptn repo but the existing solution exists in this docs repo!!
cc: @thisthat @RealAnna

Changes done! @StackScribe @thisthat
Can you review this again?


update smdebug version for Jan DLC release to 0.5.0.post0

Description of changes:
Updated version of smdebug included into MXNet DLC training containers for Py3 to 0.5.0.post0.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Looks to be a botocore version conflict causing these failures

Can you try to update the awcli version to 1.16.314 and move the awscli pip install statement to the very end of the RUN pip commands? See below for an example
https://github.com/aws/sagemaker-tensorflow-container/pull/267/files


[SPARK-50423][PYTHON][DOCS] Refine the docstring for misc and hash functions

What changes were proposed in this pull request?
Refine the docstring for misc and hash functions
Why are the changes needed?
to improve docs and test coverage
Does this PR introduce any user-facing change?
doc-only changes
How was this patch tested?
updated doctests
Was this patch authored or co-authored using generative AI tooling?
no

Merged to master.


Busboy does not reliably parse file from FormData

I have encountered an issue when trying to parse larger FLAC files using Busboy. Sometimes, the file is parsed as one would expect, but other times, the stream remains stuck indefinitely. This is especially true when the file size increases.
Now, I must admit that I don't have an extensive JavaScript background, so it could very well be the case that I have overlooked something obvious. However, I did pose the question on Stack Overflow and a more seasoned developer could not figure it out either.
To help as much as I can, I have created a scenario that illustrates the issue first-hand. It consists of a minimal front-end that accepts a FLAC file, which is then sent to an endpoint as a FormData() object. The back-end receives the request and parses it using Busboy. The problem occurs when I try to pipe the file to createWriteStream(), as this can stall forever. Is the problem with my code, or is Busboy not streaming the file data? I ask because I cannot find any errors in my code.

I can't reproduce the issue. I generated a randomly filled 32MB file, uploaded it with cURL, and used the first example from the readme to parse the request. The amount of data received in the file stream matched exactly what was sent by cURL.
The only problem I see with your code is that you're assuming the file has been fully flushed to disk by the time 'close' is emitted on the busboy object. This may not the case. You should wait until the file's 'close' event before trying to upload it.

I encountered the same problem.

The problem occurs when I try to pipe the file to createWriteStream(), as this can stall forever.

My situation and code is almost exactly the same as @MihkelPajunen . I pipe the file stream to a writable stream(created by createWriteStream()).
I tested using different size of file, ranged from 24M to 250M. They all stuck, as shown in below debug screenshots(ordered by uploaded file size).

I'm able to reproduce it now. The issue manifests itself in a very specific situation. Fix inbound...


[Bug]: No Output / Output vanishes with Version 2.5.1

Short bug description
Since update to version 2.5.1 my console doesn’t shows any output
Extended bug description
After updating, my console shows output for a fraction of a second and after that the only output shown is:
// Laravel Tinker started at 2022-12-22 10:43:35
*** Execution finished ***

I use PhpStorm Version 2022.3 and have the new UI (beta) enabled.
For now i use Version 2.4.0 per manual install which works for me.
How to reproduce?

Open Tinker Console and dd something

Expected behavior
Output shown in result window
Screenshots

IDE & Version
PhpStorm 2022.3
PHP Version
7.4
OS and version
Windows 11
Laravel Tinker plugin version
2.5.1
Interpreter type
{"label"=>"Local"}
Laravel Version
v6.20.44
Tinker version
v1.0.10
Psysh version
v0.9.12
Additional Context
No response

Same issue here :(

Same as above. Not pin pointed which of the recent versions this started to occur in.
As a temp fix I've uninstalled the plugin and imported from disk using the most recent release from prior to the recent batches.
https://github.com/Roboroads/laravel-tinker/releases/tag/v2.4.0 release.

I feel like I know what this caused. Mind if I create a build for y'all to test?

No i don´t mind. I will try it if you like @Roboroads

Interestingly, I wasn't having any issues until maybe after a reboot.
I'm running
Windows 10
PHPStorm 2022.3.1
PHP 8.0.15
Laravel Tinker 2.5.1
Laravel Framework 9.42.2
I downgraded to Tinker 2.4.0 and it seems to have resolved the issue for the moment.

Alright. This might acutally be the same issue as #232 which is fixable.

Any update on this?

any update?

@algethamy I have been using the beta version he released here:
https://github.com/Roboroads/laravel-tinker/issues/232#issuecomment-1467106592
It works ok - I get double output sometimes, but at least it works.

I got a RC version, should work on all interpreter types and even dump/dd. Let me know if there's issues <3
Laravel Tinker-2.6.0-rc1.zip

@SaintPeter Thanks for checking this. However, i'm not able to reproduce this one.

Laravel Tinker-2.6.0-rc2.zip
@SaintPeter I added  to the HTML of the output.

@Roboroads Installed rc2, same output.
I am running PHP 8.0.15, don't know if there is a compatibility issue.
In truth, I don't know much about how unicode is handled or what might cause these characters to be mangled.  It does remind me of what happens to unicode in HTML - I think it get mangled in the same way.
I'll keep using this and let you know if I run into any other issues.

I'm testing with Laravel 10, with a min. req. for 8.1. In any case, it might be a your-project issue, but it's definately an encoding issue. I'm not sure how the plugin could be responsible since I'm not doing too weird of things.

I'll keep using this and let you know if I run into any other issues.

Thanks! When there's no feedback before friday I'll release it :)

Looks good!

Looks good!

Awesome! Thanks for testing as well 🙏

The Laravel Tinker-2.6.0-rc2.zip is still not working and the fallback to v2.4.0 is not valid anymore when using PHPStorm 2023.1.
How to reproduce?
When I try to run echo "abcd"; or  dd('somthing')
Expected behavior
Context:
OS: MacOS Ventura 13.2.1 (22D68)
Plattform: aarch64
IDE: PHPStorm 2023.1
Docker client: Docker version 23.0.1, build a5ee5b1dfc
Docker Runtime:  colima version 0.5.2
PHP Container: laravel/sail v1.21.2
PHP Version: PHP 8.1.17

@dennyweiss Unfortunately I do not have an M1/2 macbook. I'm also running the sail container but on linux - without issues.

Issue fixes in 2.6.0