java: Populate $out/lib/jvm

Description of changes
This change should make it easier to detect available JDKs by providing symlinks at the canonical location $out/lib/jvm/$name.
In the Java ecosystem, this is a common pattern. For example, the Gradle build tool supports "crawling" many of those locations:

/usr/lib{,64}/jvm, {/opt,/usr/{,local/}}java
$HOME/.jdks
$HOME/.sdkman/candidates

The Scala tool sbt follows does something very similar and an attempt to integrate with NixOS has stalled.
There's been some discussion about how to make integration of these tools with NixOS easier, see https://discourse.nixos.org/t/4677 and an earlier attempt at https://github.com/NixOS/nixpkgs/pull/76699
Since the attempts by @raboof, I have made some progress with Gradle in https://github.com/NixOS/nixpkgs/pull/119444. The approach gives more control but is not ergonomic since it requires overriding Gradle to pass in JDKs. Note that this proposal is not meant to replace passing toolchains to Gradle but proposed in addition.
Recently @kravemir has reignited the discussion and has suggested to implement detection based on environments/profiles in https://github.com/gradle/gradle/pull/29214#discussion_r1605954040. This PR is an attempt to standardize a detection location per-derivation, namely $out/lib/jvm/$name and to enable linking these paths whenever someone sets programs.java.enable.
A hook is provided so that all JDKs can use the same code for creating the symlink.
Once this has landed, it'd be possible to have tools like Gradle and sbt crawl /run/current-system/sw/lib/jvm, ~/.nix-profile/lib/jvm, or other locations via nix-shell.
For now I have only changed four JDKs, but that should be sufficient to see what's going on and test the approach. Of course I am happy to add the hook to other JDKs too.
Alternative Considered
I also considered generating a pkgs.linkFarm in the Java module, but this solution is not as portable. By creating the link in each derivation, downstream package consumers can also make use of the mechanism, e.g. home-manager could also link them somewhere without having to reimplement linking.
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/auto-detecting-java-installations/4677/11

I'm somewhat reluctant about this change:

for 'end-user' programs, the JDK should just be specified in the derivation, so for those you shouldn't need this module
for 'developer' environments, the 'appropriate' JDKs might differ per project, so installing them 'globally' as a module doesn't make too much sense.

Having let this simmer for a while, I think I would prefer the approach of having each JDK contribute itself to a JAVA_HOMES environment variable (similar to how it sets the JAVA_HOME, but allowing multiple values).
That said, as this topic has been stagnant for a while, I'd be supportive of this change as well, just to get things unstuck and gain some more experience on how this works out in practice ;)

for 'end-user' programs, the JDK should just be specified in the derivation, so for those you shouldn't need this module

I agree. This is not the intended use-case here. I mention how it is possible to pass JDKs to Gradle, for example, which fits this use-case.

for 'developer' environments, the 'appropriate' JDKs might differ per project, so installing them 'globally' as a module doesn't make too much sense.

This is why it is opt-in (currently tied to programs.java.enable). Note however, that the proposal also works with regular profiles, without NixOS. You don't have to use the "global" profile via /run/current-system/sw/share/jvm, you can point your detection mechanism at an arbitrary profile/environment. For example:
$ nix profile install .#adoptopenjdk-hotspot-bin-8 --priority 7
$ ls ~/.nix-profile/share/jvm
adoptopenjdk-hotspot-bin-8.0.372
$ nix profile install .#adoptopenjdk-hotspot-bin-11 --priority 7
$ ls ~/.nix-profile/share/jvm
adoptopenjdk-hotspot-bin-8.0.372  adoptopenjdk-hotspot-bin-11.0.19
$ nix profile list
Index:              0
Store paths:        [...]

Index:              1
Store paths:        /nix/store/...-home-manager-path

Index:              2
Flake attribute:    legacyPackages.x86_64-linux.adoptopenjdk-hotspot-bin-8
Original flake URL: git+file:///home/lorenz/src/github.com/NixOS/nixpkgs
Locked flake URL:   git+file:///home/lorenz/src/github.com/NixOS/nixpkgs?ref=refs/heads/jvm&rev=3a9db7691302d62a6d9020c927a791f0ddbfba86
Store paths:        /nix/store/4q4d79yp3sj9310kcxajh6y7gyjg5j4q-adoptopenjdk-hotspot-bin-8.0.372

Index:              3
Flake attribute:    legacyPackages.x86_64-linux.adoptopenjdk-hotspot-bin-11
Original flake URL: git+file:///home/lorenz/src/github.com/NixOS/nixpkgs
Locked flake URL:   git+file:///home/lorenz/src/github.com/NixOS/nixpkgs?ref=refs/heads/jvm&rev=3a9db7691302d62a6d9020c927a791f0ddbfba86
Store paths:        /nix/store/yd3x5x25k66byrwrl6qq7z3caqqqips0-adoptopenjdk-hotspot-bin-11.0.19
% nix profile remove /nix/store/4q4d79yp3sj9310kcxajh6y7gyjg5j4q-adoptopenjdk-hotspot-bin-8.0.372
removing 'git+file:///home/lorenz/src/github.com/NixOS/nixpkgs#legacyPackages.x86_64-linux.adoptopenjdk-hotspot-bin-8'
$ ls ~/.nix-profile/share/jvm
adoptopenjdk-hotspot-bin-11.0.19

I believe that this is also useful with nix-shell. Consider the following shell.nix placed at the root of nixpkgs:
let
  pkgs = import ./default.nix {};
  env = pkgs.buildEnv {
    name = "development";
    paths = with pkgs; [
      (adoptopenjdk-hotspot-bin-8  // { meta.priority = 6; })
      (adoptopenjdk-hotspot-bin-11 // { meta.priority = 5; })
    ];
  };
in
pkgs.mkShellNoCC {
  packages = [ env ];
  shellHook = ''
    echo "Your JVMs are in '${env}/share/jvm' check it out:"
    ls -o ${env}/share/jvm
  '';
}

This gives you:
$ nix-shell
Your JVMs are in '/nix/store/wn8c7mvnrlw83gsm6laq24haq94xp1b1-development/share/jvm' check it out:
total 8
lrwxrwxrwx 3 root 119 Jan  1  1970 adoptopenjdk-hotspot-bin-11.0.19 -> /nix/store/yd3x5x25k66byrwrl6qq7z3caqqqips0-adoptopenjdk-hotspot-bin-11.0.19/share/jvm/adoptopenjdk-hotspot-bin-11.0.19
lrwxrwxrwx 3 root 119 Jan  1  1970 adoptopenjdk-hotspot-bin-8.0.372 -> /nix/store/4q4d79yp3sj9310kcxajh6y7gyjg5j4q-adoptopenjdk-hotspot-bin-8.0.372/share/jvm/adoptopenjdk-hotspot-bin-8.0.372

Having let this simmer for a while, I think I would prefer the approach of having each JDK contribute itself to a JAVA_HOMES environment variable (similar to how it sets the JAVA_HOME, but allowing multiple values).

I know that this is your preferred approach, I have linked https://github.com/NixOS/nixpkgs/pull/76699 above, but I think that what I am proposing here both fits the "Nix way" (convention on derivation outputs instead of environment variables) and also the "Java ecosystem way" (crawling magic directories).

Forgive the bikeshedding, but why share? A JVM is mostly object files, so I'd have expected $out/lib/jvm (and this is a closer match to the /usr/lib/jvm path expected by Gradle and SBT).

Trying once more to get some attention. Pinging @NixOS/java after @tomodachi94's comment. Of course I do realize that there are merge conflicts currently, and I would of course resolve them, but I'd love to get some feedback before I invest more time.

That said, I'm generally supportive of the general direction this is going in.


MessageBuilder refactorings for tests

couple of refactorings to improve testability of MessageBuilder:

inject boundary generator
inject uuid generator (for message ids)
make send date and hideTimezone builder parameters

also adding a test with an attachment, and adding a check if writeTo output matches expected

I didn't like the passing around of dependencies and made some changes: https://github.com/k9mail/k-9/compare/message-builder-tests...cketti/message-builder-tests

Looks good to me~
(Although the whole reason I put that in the constructor was because you said to do that)

Yes, I didn't think this through. But putting the UuidGenerator reference in the constructor exposed the issue. So it was still a win ;)

Hooray


cmd/compile: add GOAMD64 architecture variants

This is a data-gathering bug to collect instances in which a GOAMD64 environment variable would be helpful. Please feel free to add to it. My hope is that over time, if/when these use cases accumulate, we will get a clearer picture of whether to add GOAMD64, and if so, what shape it should take.
Uses:

FMA: https://github.com/golang/go/issues/8037
Fast vs slow 3-part LEA instructions: https://github.com/golang/go/issues/21735#issuecomment-390805660
AES instructions: used on hot hashmap hashing calls (could be direct rather than indirect calls)
math/bits intrinsics support: see the discussion in the comments in https://lemire.me/blog/2017/09/05/go-does-not-inline-functions-when-it-should/
Various SSE levels: may speed up some hot runtime assembly, like memclr and memcpy

cc @randall77 @martisch @TocarIP @Quasilyte @dr2chase

I'd like to add:

non-destructive floating point math (X2 = X1 + X0, instead of X1 += X0)
faster duffcopy and OpMove via AVX
some scalar instructions like ANDN (and-not)
sane versions of Ctz, without undefined results on 0 argument
removing CPUID checks from intrinsified math functions (mostly rounding)
shifts/rotates that don't clobber flags and allow
AVX512-specific fp-to int64 conversions
extra AVX512 registers

There is also a problem with opting into 512-bit version of AVX512.
By default only 256-bit version should be used to avoid frequency drop, but there should be a mechanism to opt-in for 512-bit version for faster memmove/crypto/etc

It is preferable to use runtime detection when possible.  For any sizeable chunk of code, that's what we currently do.  But especially for individual instructions, the overhead for runtime checking can be large.  In that case, it can be preferable to "bake in" the decision during compilation.  That's what such an environment variable would do.

Apologues for asking more questions -  if we add an environment variable, wouldn't it be simpler and faster to simply use the optimal code for the hardware, getting rid of all run-time detection?
Unless the point is to do a best-effort combination of the both, so that very expensive code like crypto would still perform well on newer hardware even if the environment variable wasn't set up properly?

We want to use the environment variable as little as possible, as it bakes the optimization decision in at compile time.  As a consequence, the binary cannot then adapt at runtime; it would fail on an old machine, or not use optimal code on a new machine.
This will lead to people having to distribute multiple copies of their binary, possibly many depending on how many options are available.
It's also extra API surface to the compiler which we would have to spec & maintain.
So I think we'd need a really good reason to go down this road.

function multi-versioning was proposed some time ago as a compromise between less run-time checks and portability between older/newer CPUs.
I can't find discussion right now but the approach with checking CPUID features once and then using
specialized version looks good (at least on paper).
I have no specific proposals, more over, I'm not familiar with referenced technique well.
This is just a note that there is another approaches between "full compile-time" and "full run-time" selection. Thank you.

For some cases dynamic dispatch would be slower than branching though.
But there is also JIT-like approach where we re-write function code segment during startup if there are more optimized versions available. This is not possible for every imaginable platforms, but given this is mostly about AMD64, seems doable in that context.

Great ! I have been wanting this for a long time - https://groups.google.com/forum/#!topic/golang-nuts/b75xYuORaok :) My thinking was that using the non-destructive 3 operand form of VEX encoded instructions will alleviate register pressure and will make the binaries much smaller (Unfortunately I know nothing about compiler internals to do a PoC and confirm this).
@ianlancetaylor had mentioned -

I don't think it's worth adding a GOAMD64 environment variable for
this.  I don't think the gain is worth the complexity.  Happy to be
proven wrong.

I've implemented a prototype in https://go-review.googlesource.com/c/go/+/117295 to see how beneficial this can be.
@agnivade  can you check if it with your binaries?

With pleasure :)

I have done some testing.
On binary size, I am seeing a clear increase. I tested with 3 production applications. 2 network services and 1 webapp. This is without -ldflags="-s -w". Baseline was with current master.
Service1 - 12MB to 17MB
Service2 - 11MB to 16MB
Webapp - 11MB to 15MB
On the perf side, general application benchmarks do not show noticeable improvements. But if I do micro-benchmarks targeting very specific functions - for eg if I benchmark math.Atan2 which has lot of FMA statements, I can see improvement of 2ns.
I looked into the objdump of the binaries to investigate possible reasons for this (with whatever meagre assembly knowledge I have).
One thing that I noticed is that the V(ADD|SUB|MUL|DIV)SD operations always act on registers, and not on memory operands which they are especially useful for in reducing register pressure.
For eg. I see patterns like this throughout -
vmovsd 0x66323(%rip),%xmm3        # 477a38 <$f64.4059000000000000>
vdivsd %xmm3,%xmm2,%xmm2

vmovsd 0x662bf(%rip),%xmm3        # 4779e0 <$f64.3ff0000000000000>
vsubsd %xmm3,%xmm0,%xmm0

vmovsd 0xc6b83(%rip),%xmm4        # 4d82b0 <runtime.memstats+0x1730>
vsubsd %xmm4,%xmm2,%xmm5

which I believe can be optimized to -
vdivsd 0x66323(%rip),%xmm2,%xmm2

vsubsd 0x662bf(%rip),%xmm0,%xmm0

vsubsd 0xc6b83(%rip),%xmm2,%xmm5

That's all I could glean. Happy to provide more info if required.

Benchmarks that I ran Friday, got busy and forgot to post:
https://perf.golang.org/search?q=upload:20180611.1
Quiet machine, using Austin's perflock to make things even more sane.
Saw -0.5% geomean time change, one benchmark got worse for unclear reasons.
 I was comparing with 1.10.2. :disappointed: I have updated my comment. The results match with @dr2chase's observations.

No need for the s-word, benchmarking is picky work that's easy to get wrong, I ended up writing a tool to help me do it because I could not trust myself to do it all correctly and reproducibly, and I really ought to make that tool automate even more of the steps.

@dr2chase what machine are you using? I've tried to reproduce Growth_MultiSegment regression, but it showed no changes.

It is "Model name:          Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz"
I used Austin's perflock to reduce risk of heat-related performance changes, used a quiet machine.

It'd nice to be able to use BLSR from BMI1, e.g. to rapidly iterate over set bits, which is something that we do in a few hot functions in the runtime (adjustpointers, scanblock).

I haven't got data on the performance impact yet, because I have to hack the compiler to do it, but we have a tight loop that's oring two 8KB blocks together, and we also want to popcount them, and right now, on amd64, adding popcount more than doubles runtime for the loop according to naive benchmarks.
https://godbolt.org/z/1L6f_N
The difference from adding a popcount inside the loop is larger than I'd have expected:
        movq    (CX)(DX*8), BX
        movq    (AX)(DX*8), SI
        orq     SI, BX
        movq    BX, (CX)(DX*8)

becomes
        movq    8(CX)(DX*8), DI
        movq    8(AX)(DX*8), R8
        orq    R8, DI
        movq    DI, 8(CX)(DX*8)
        addq    SI, BX
        cmpb    runtime.x86HasPOPCNT(SB), $0
        jeq     unionBitmapBitmapInPlaceTemps_pc349
        xorq    SI, SI
        popcntq DI, SI

Note, the "addq" in the middle is actually adding a previous popcount, this loop's popcount is added to a counter about four instructions later. So this isn't the actual instructions for this step, but it's the same instructions as them.
I don't understand why there's an xorq SI, SI right before the popcount, because the popcount just overwrites that register anyway, I think? But my expectation would have been that popcount would have fairly low impact, because the entire operation is trivial enough that I wouldn't expect it to be running out of computation before it gets stuck waiting for cache lines. I did not expect it to double runtime cost. (I expected just the popcountq and addq instructions, basically.)
If time allows, I'm gonna hack my local compiler to ditch that and see what the impact is like.

For the XORQ SI, SI, if I recall correctly, this is because on some architectures popcnt has a false dependency on the output register.

So I made a funny mistake which actually produced informative results: I wrote a benchmark for this in which I omitted any actual use of the results of popcount operations.
As it happens, this has the curious effect of generating the conditional jumps to call the popcount function if necessary, but not generating the actual popcount instructions (as the compiler figured out that the value being set to their sum was unused).
So, for basically similar code, my reported times were:
no popcount: 450ns/op
conditionals but no popcount: 600ns/op
popcount included: 800ns/op
(somewhat approximate values)
I then went ahead and did the fairly obvious thing to just take away the runtime tests for OnesCount64, and with those, I get 444/444/520.
What's interesting about this is that the cost of adding the popcount instructions without the conditional tests is dramatically lower (~80ns) than the cost of adding the popcount instructions with the conditional tests (~200ns more than the conditionals without the popcounts).
So, for this specific function, it looks like the difference would be ~520ns vs. ~800ns. A 35% reduction in runtime for a hot loop is pretty tempting.
I don't really know enough about modern machines to know how common it is to find amd64 hardware without popcount. But I'd say that, yeah, I would be pretty interested in a target that allowed me to get that improvement on something that has been a pain point for us in the past.

As another data point, https://go-review.googlesource.com/c/go/+/299491 is a proof-of-concept CL that adds support for a handful of Haswell instructions to the compiler, which I (a not particularly savvy micro-optimizer) was able to use to improve encoding/binary.Uvarint's throughput by 166% (from 616ns to 231ns).
For comparison, the Go protobuf package includes a fully unrolled version of varint decoding (because varint decoding is a very hot code path within protobuf unmarshalling, which takes 347ns (+78% throughput). Applying the extra tricks from CL 299491 (using rotation instead of shifts, having a fast path for len(buf) >= 8 to eliminate bounds checks, early loading of the slice data pointer) bumps that a little further to about 315ns (+96%). Still measurably short of using Haswell extensions.
I think large-scale cloud deployments would benefit from being able to statically build Go binaries that take advantage of more recent x86 instruction set extensions. Haswell (2013) is older than POWER9 (2017), which is configurable via GOPPC64.

Varint decoding is also very hot code in the runtime, particularly during stack copying. See e.g. https://go-review.googlesource.com/43150.

Another data point: https://go-review.googlesource.com/c/go/+/299649 reworks scanobject to use some Haswell instructions, and gets a 4% geomean performance improvement in compilebench. (Not bad for another 1 day hack.)
I'm going to see if I can benchmark some longer-lived processes, and then also look into heapBitsSetType.

FWIW, heapBitsSetType would be much simpler with a 1-bit bitmap, something we've been thinking about. That may obviate the need for the fancy bit manipulation instructions.

I asked Google's C++ toolchain team for advice here, and they pointed out the x86-64 psABI now defines microarchitecture levels with specific feature sets: https://en.wikipedia.org/wiki/X86-64#Microarchitecture_Levels. Also, this blog post from Red Hat giving more context: https://developers.redhat.com/blog/2021/01/05/building-red-hat-enterprise-linux-9-for-the-x86-64-v2-microarchitecture-level/
It seems like the three immediately useful targets are "baseline", "v2", and "v3". Baseline is what we support today. v2 and v3 each offer new instructions that could be used for better optimizing current Go code (e.g., POPCNT in v2, BMI1/BMI2 in v3).
I don't see any immediate benefit from supporting v4 today. Go doesn't have native SIMD support, and AVX512 hardware support seems comparatively less common today still. (There are also lots of discussions about AVX512 having adverse system-level effects due to CPU down-throttling, etc.)
So my tentative proposal would be GOAMD64=baseline, GOAMD64=v2, and GOAMD64=v3. I'm open to alternatives to "baseline", but the psABI doesn't actually give this level a version name of its own. (But maybe we could suggest they retroactively name it "v1".)
There have also been suggestions that binaries should check at startup time whether the CPU supports the required feature, so we don't get lots of SIGILL issue reports from users. That seems reasonable to me.
Also, there's been some brainstorming about various ways we could automatically compile code multiple ways and efficiently dispatch to the right code paths at a higher-level to balance between binary size and runtime dispatch overhead. I think that's intriguing, but it's not obvious to me that we have a concrete solution here or anyone with time to investigate/implement that.

So my tentative proposal would be GOAMD64=baseline, GOAMD64=v2, and GOAMD64=v3

How about also GOAMD64=native, that would pick the highest one available on the machine that is compiling the code? Potentially that could be made the default for go run and go test.

Historically we've avoided things like GOAMD64=native.  Instead we let the bootstrap process set the default value of variables like GOAMD64 based on the machine on which the bootstrap is run.
Which isn't to say that we can't change that behavior, but I think that should be a separate decision, and one that should apply to all the GOARCH environment variables, not just GOAMD64.

I submitted #45453 as a proposal for this (GOAMD64 architecture variants), along with related proposals #45454 (build tags for distinguishing architecture variants) and #45455 (new intrinsics included in Haswell's BMI2 instruction set).


Could not find addon with name: ember-collection

Everytime I run ember install ember-collection I get the following error message. I find it hard to believe this bug is caused by this addon. However, I have not been able to reproduce this issue with any other ember addons.

As you can see, npm install --save-dev ember-collection correctly installs the addon.
Again, I do not know if there is anything that can be fixed here as it's very isolated and very odd. However, I figured I'd let you know all the same.
Thanks for the great addon!

Interesting I just ran into this same issue.
Tried to do
ember install ember-collection

and got:
Install failed. Could not find addon with name: ember-collection

Yep I'm getting this issue as well.

 I scanned the issues but didn't see one for this. I'll check in slack and see if anyone has an opinion.

Hey everyone. This should be fixed now. Turns out there was a 0.0.0 version published to npm which shouldn't of been there. Thanks @rwjblue for getting rid of that for us.

Awesome job tracking that down @raytiley!  I'm going to close this (happy to reopen if needed later)...


Made a few fixes to components under input_forms.

Minor TS screaming, but caused major headaches during deployment.
Speaking of which, there is something I was unable to resolve during deployment attempts:

Let's look into this ASAP.

Fixed. https://mm-demo.vercel.app/login :)


Allow setting name on accuracy op

Closes https://github.com/tensorflow/tensorflow/issues/7986

Can one of the admins verify this patch?

@tensorflow-jenkins test this please

PR merged. Thank you for the contribution, @cancan101 !


Factor out the implementations and re-add the existing constructor

This brings it closer to the shape described in https://github.com/sshnet/SSH.NET/pull/865#issuecomment-1793748151. I figured it was easier to show than tell.
There are no functional changes.
I have further feedback after this. Would you prefer I address them in PRs like this one or leave as a review for you to address?
Thanks
P.S for the avoidance of doubt: this PR targets your PR branch so if you merge this it will show up automatically in sshnet#865

Just add a code review on the original PR if needed.
The original constructor is not needed, why add it back?
Factoring out the CTR implementation, as I mentioned in another comment, is a bit weird unless the other modes also have their own class - but they would be empty classes.

Just add a code review on the original PR if needed.

I figured I should do the work rather than asking it of you.

The original constructor is not needed, why add it back?

I'm cautious about breaking changes.

Factoring out the CTR implementation, as I mentioned in another comment, is a bit weird unless the other modes also have their own class - but they would be empty classes.

It is a different implementation than the other modes. It doesn't seem weird to me.

Rob, I like this refactoring as it makes the code cleaner, and we did discuss refactoring CTR into its own class. Please make the small edits and I'll merge it in.

Oh dear, I shouldn't have merged develop into this branch. Will revert.


Fix bad reporting of excessive flagging

See #113

Note: added to 4.2.6 but will need a backport from master

@vietj Can I merged and backport?

Hum hold on, I need to investigate further


Active border

I would love to have an active border colour around the active window as a feature.
One you could maybe change the border-width and colour from.

Windows currently has borders and whether or not their width can be increased might be in the registry. However, the color can be chosen.
See under Settings -> Personalization -> Colors. Select the checkbox forTitle bars and window borders


activate_user doesn't run when no metadata is present

Using RC1/master, after a user has been registered, the activate_user call will never fire - because the join for metadata in $user->update() won't find any results.

This issue has been fixed.  Metadata is now always created when a user is created, so there will always be a metadata column.  If it doesn't exist for some reason (it always should) it will return a blank array still.


List of players in "Transfer queue" shows deleted players

What version of Music Assistant has the issue?
2.3.0b19
What version of the Home Assistant Integration have you got installed?
None
Have you tried everything in the Troubleshooting FAQ and reviewed the Open and Closed Issues and Discussions to resolve this yourself?

[X] Yes

The problem
When deleting a player provider, the associated players continue to show in the "Transfer queue to another player" screen
How to reproduce

Add the Airplay music provider
Go to the queue
Open the "Transfer queue to another player" screen
Notice that there is quite a long list of players, with a scroll bar at the bottom

In settings, delete the airplay provider
Go back to the queue and click on transfer queue once more
Observe that the player list is unchanged

Press F5 (if in a desktop browser)
Go back to the queue and click on transfer queue once more
Observe that the player list is now updated - the deleted players are gone

The issue is that step 8 (refresh page) should not be required.
Music Providers
Qobuz
Player Providers
Chromecast, Airplay
Full log output
ma_refresh.log
Additional information
No response
What version of Home Assistant Core are your running
2024.9.1
What type of installation are you running?
Home Assistant OS
On what type of hardware are you running?
Windows

should be fixed in 3.2.0b22

@madbrain76 please close this if it has been fixed.

Well, there was the issue with the one player that rose to the top. It required renaming to fix it. I don't think that should have been needed. The other players didn't require renaming to be sorted. I don't know why the one player was treated differently. I think that should be fixed before closing this issue.

This issue is about deleted players being shown. If that is fixed then this needs to be closed.
As for the other, unless you can recreate it, it is impossible to say why that happened. It doesn’t seem worth expending further effort in trying to work out why when it is a single report and was rectified by renaming.

Sorry, I had a brain fart last night, I confused the two issues. I just verified that unfortunately, this problem is not fixed . I tested in b23. I removed the DLNA and Airplay players. Then opened the queue, clicked "transfer queue" and they were still all there.
On the other report (unsorted entry), somebody new to MA running into this will not know that renaming is needed, without interacting with devs as I did. I can likely reproduce the issue if it will help fix it.

Ok deleted players still showing will continue to be looked at. As for the other issue if you can reliably recreate it then it will also be looked into further.

this should be fixed in 2.3.0b24

Unfortunately, it's still not fixed in b24 . The refresh is still required. I'm using Firefox 130 on Windows, if that matters.
There is a missing step 0 in the test case - start playing something first.
music-assistant_deleted_players.log

That might matter. Can you try Chrome?

I just did. It's broken in Chrome too.

This should b fixed in beta 33 or above. Please confirm.

This will be closed soon due to no followup


fallback_filename in api.add_static_route() always served as application/octet-stream

When fallback_filename is defined,
api.add_static_route('/', '/absolute/path', downloadable=False,
        fallback_filename='index.html')

suffix always comes up empty which results in the fallback file being sent over the wire as application/octet-stream (browser prompts for file download instead of rendering the HTML).
https://github.com/falconry/falcon/blob/master/falcon/routing/static.py#L109
The caveman-simple fix would be:

        # suffix = os.path.splitext(file_path)[1] # comes up empty
        suffix = os.path.splitext(self._fallback_filename)[1]
        resp.content_type = resp.options.static_media_types.get(
            suffix,
            'application/octet-stream'
        )

I guess this completely skips the normalization logic so a more elegant fix should be in place.

^ Scratch that, the solution needs a bit more research.
I could take a short at a PR once you validate these findings.
I'm on Python 3.5.2, Linux with the current master HEAD as my falcon module.

Fixed in https://github.com/falconry/falcon/pull/1348 (see also https://github.com/falconry/falcon/pull/1314, https://github.com/falconry/falcon/pull/1325 & https://github.com/falconry/falcon/pull/1341).


Does the plugin be suit for ios?

Hi, does the plugin be suit for ios?
It seems that the plugin only for android.

Hi, unfortunately iOS is not supported, because there is no public API for controlling WiFi. See duplicate here 32

iOS is supported in a limited manner. See the README for complete information.


Fix invokerId assignment path in zookeeper

Fixes #3856
Description
This PR fixes the id assignment path and also add some tests based on Curator Test Support. To enable test some refactoring was done to extract the existing id assignment logic to a new class InstanceIdAssigner.
Going forward this would also simplify work for #3858
Related issue and scope

[x] I opened an issue to propose and discuss this change (#3856)

My changes affect the following components

[ ] API
[ ] Controller
[ ] Message Bus (e.g., Kafka)
[ ] Loadbalancer
[x] Invoker
[ ] Intrinsic actions (e.g., sequences, conductors)
[ ] Data stores (e.g., CouchDB)
[ ] Tests
[ ] Deployment
[ ] CLI
[ ] General tooling
[ ] Documentation

Types of changes

[ ] Bug fix (generally a non-breaking change which closes an issue).
[ ] Enhancement or new feature (adds new functionality).
[x] Breaking change (a bug fix or enhancement which changes existing behavior).

Checklist:

[x] I signed an Apache CLA.
[x] I reviewed the style guides and followed the recommendations (Travis CI will check :).
[x] I added tests to cover my changes.
[ ] My changes require further changes to the documentation.
[ ] I updated the documentation where necessary.

Note that this would be a backward incompatible change as the path used in zookeeper would get modified. So for any existing setups the id calculation would result in a different result

Codecov Report

Merging #3866 into master will increase coverage by 0.32%.
The diff coverage is 91.66%.

@@            Coverage Diff             @@
##           master    #3866      +/-   ##
==========================================
+ Coverage   74.46%   74.79%   +0.32%
==========================================
  Files         138      139       +1
  Lines        6478     6479       +1
  Branches      409      407       -2
==========================================
+ Hits         4824     4846      +22
+ Misses       1654     1633      -21

Impacted Files
Coverage Δ

...er/src/main/scala/whisk/core/invoker/Invoker.scala
0% <0%> (ø)
:arrow_up:

.../scala/whisk/core/invoker/InstanceIdAssigner.scala
95.65% <95.65%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 238ad96...1d1965c. Read the comment docs.

manually confirmed it works as expected on kube (not checked by travis).  Will merge as soon as PG completes.

@chetanmeh -- would you mind rebasing this to latest master?  We need the reorg of the ansible db variables in #3867 to be in this PR for the PG testing to go through.  Thanks for your patience.

Rebased to lasted master

PG3 2525 🏃

Sorry this conflicted with @vvraskin's change, which I merged this morning 😢

Rebased to resolve conflicts

PG3 2526 🏃

PG3 2526 👍


Revisit RequestOptions.auto_parse_form_urlencoded

Create a media handler for application/x-www-form-urlencoded that can be used via req.media. As part of this work, deprecate the use of the auto_parse_form_urlencoded (which merges form params into query string params) by adding a note to the docs to that effect.
The implementation can be patterned after Request._parse_form_urlencoded()

Work remaining after https://github.com/falconry/falcon/pull/1537:

Decide whether we want to add this to default handlers (@nZac expressed support for this in another discussion IIRC, I agree)
Deprecate auto_parse_form_urlencoded in the docs


Adding sudo option for when forking a project

This adds the option to fork a project for another user. One use case for such feature is when you want to fork projects for new users as soon as their account is created.
Using Gitlab.sudo = 'someone_else' might not be suitable in this case, because concurrent requests with different values for sudo can have a negative impact on each other.
Note: g = Gitlab.client; g.sudo = 'someone_else' didn't have the expected effect. This PR is my attempt to overcome it.

Thanks, @asedge. How does it look like now?
Cheers!

@gregoriomelo Looks good, please squash the commits.

Cool, @asedge. Squashed.
Cheers!

Thanks!


[Bug] Duplicate property for map widget

Description
The map widget shows 2 properties mapCenter and center which both have the same values in the auto complete

Would you clarify this issue? I created the map, but I don't see the same value printed.

@Devedunkey when you try to bind the map to a text widget you will see this.
Drag a text widget, in the text field type {{Map1.
in the auto complete you will see 2 values center and map center

@Nikhil-Nandagopal
Thank you. I got it. Would you assign this to me?

@Devedunkey go for it!


The 'Deploy to Static Website' menu option is missing

I'm following this tutorial, but I've run into a problem. In my installation, I'm not getting the Deploy to Static Website menu item:

Is there a configuration item somewhere that I must set?

The tutorial is for Azure Storage, actually.  Please download that extension to follow the tutorial.

Aha!
Thank you for the link (and for your quick reply as well).


Handle switching in insert mode

Currently, one cannot navigate to another pane if you're in vim in insert mode. This change adds additional inoremap to handle this case.
Criticism welcome.

Hi @harterrt, while I appreciate the effort I don't think this is a direction we want to go. In general I favor the "do as little in insert mode as possible", and I also think plugins should claim as few mappings as possible. Thankfully, as your PR shows this is entirely available to the end user to map on their own.

Cool, thanks for the review and happy holidays. :slightly_smiling_face:


API key request using other devices

Is there a way to set the API key as fixed so that external users can interact with the chat?
I've installed it and it works perfectly in Chrome; however, when I open it in another browser or device, it asks me to enter the API KEY.?

It is designed by default to take the api key from the user and use it for generating the response

is there any way to keep my API key?

For me the only job I am looking for is the use of the chat by external people via my key.

Any news about this ? Would be nice to have the possibility to store an api key for all users. So when you share a custom gpt, the end user has not to put in the API key

Just thinking about the easiest way to do this, then I'll PR an "option" to make it public (without the key being visible to the client)

I got this working by changing all the client-side openai calls to server side API calls, so
This...
const messages = await openai.beta.threads.messages.list(threadId);

Becomes this...
const getMessagesResponses = await fetch(`/api/openai`, {
        method: "post",
        body: JSON.stringify({
          method: "messages",
          action: "list",
          threadId,
        }),
      });

      const messages = await getMessagesResponses.json();

And the API route runs this...
const listMessages = await openai.beta.threads[method].list(threadId);
      return NextResponse.json({
        ...listMessages,
      });

See this branch: https://github.com/mledwards/Open-Custom-GPT/tree/update/make-server-side

@hcdgsv @SamuelChabal @vonbarnekowa
Now added support for using api key of owner instead of requesting it everytime in the web version. Do try at customgpt.thesamur.ai


Propose Logo/Icon

Hi @akshay2211 . I'm a graphic designer and an open source enthusiast. I would like to contribute to your awesome project by proposing a logo/icon design for it if you allow me. I'll be doing it as a gift for free to your awesome project.
Best regard.
Mirza Zulfan.

@mirzazulfan  Thankyou for the proposal, I would be glad to accept any contribution in any of my opensource projects.

Thank you for your response sir. I'll start my design..

Hi @akshay2211 .. I've designed some logos,, which one do you prefer? is there anything you want to change please let me know. if you have an idea please share with me, it would be better for me. Thank you sir. Best Regard

Thank you @mirzazulfan for putting so much effort in MusicWave.
All the samples you have created all are good.
but can we keep the gradient visualization in logo, as it is the soul of library.
Some suggestions -

You can show the gradient visualization on a solid background with similar gradient name in it
or you can use a bright gradient background and provide a solid white colored small wave with or without its title.

Hi again. @akshay2211 . What do you think about this?

hi
I want to show the wave when recording voice but i cant
I'm going to help you
I'm sorry I do not know English well


[BUG]cronjob-controller  Cannot determine if job needs to be started

Describe the bug
Cronjob to release free ips are not getting executed on scheduled interval.
Expected behavior
Cronjob to clear free ips into the pool must have executed as per the schedule
To Reproduce
Steps to reproduce the behavior:

setup whereabouts v0.5.3 on k8s 1.20
create and delete multiple multus pods to see IPs are freed up or not.

Environment:

Whereabouts version : 0.5.3
Kubernetes version (use kubectl version): 1.20
Network-attachment-definition: N/A
Whereabouts configuration (on the host): N/A
OS (e.g. from /etc/os-release): Amazon Linux2
Kernel (e.g. uname -a):
Others:

Additional info / context
Log message
Type     Reason            Age                      From                Message                                                                                │
----     ------            ----                     ----                -------                                                                                │
Warning  FailedNeedsStart  2m49s (x14839 over 41h)  cronjob-controller  Cannot determine if job needs to be started: too many missed start time (> 100). Set or decrease .spec.startingDeadlineSeconds or check clock skew

Does this issue reproduce with a supported Kubernetes version (1.22 / 1.23 / 1.24) ?


Multi Combobox: Support persistent menu

We need the ability to allow users to maintain the combobox list of items open, so that users don't have to reopen the menu after each selection
How to test

Confirm that the hasPersistentMenu prop behaves as expected for Multi Comboboxes
Confirm that keyboard navigation behaves as expected (escape, tab, up/down, left/right arrow keys)
Confirm that the placeholder text disappears for Multi Comboboxes once at least one item has been selected

Verified the ticket  and found this to be working as desired.
See Screen-cast : http://somup.com/c3XnhTvBFm
Thanks!


fix: replace Unicode arrows with SVG

Description
The Unicode characters we were using for the Project Tree reveal-triangles don't exist on some systems. Given the very constrained nature of how CSS's list-item-style works, making them into images that could be read from inline data URIs was probably the best solution here; those long strings are a URL-safe encoding of the SVG for the triangles.
Task Item
#minor
Screenshots

Coverage remained the same at 52.165% when pulling 950e473152efd2442f03866f8bc33d9e484afdd7 on beyackle/fixListStyleTriangles into d0ce8fee175cce3edefca52102bcc74a2c9cae41 on main.


link build badge to master branch

Pull Request check-list
Please make sure to review and check all of these items:

[ ] Does npm test pass with this change (including linting)?
[ ] Is the new or changed code fully tested?
[x] Is a documentation update included (if this change modifies existing APIs, or introduces new ones)?

Description of change
Change the image to svg as it looks better on retina and link to master branch.

👍  thanks @a0viedo


add css file

I think you should remove the styles for elements like header, or h2 because it is going to conflict with your styles. The best is to add classes and then later replace those classes with the correct ones

HEY GUYS!!!
I just added styles for top-navigation. These styles are common to use for all the the 5 top navigation in all the teams.
I have put the css file inside css->navigation.css and an example html file inside teams->examples->top-navigation.html
Please review these files as well.
@iamcucusa @Zurina @juliusberthold


NestIO compatible with Nest 3.2

Hi all,
@rgutzen @jasperalbers and I were looking into the NestIO today and we came up with this edits that make the current NestIO compatible with both Nest 2.x and 3.x versions.
We incorporated all the changes suggested by @essink into this PR.
Essentially we force the IO to ignore all the headers (which raises a warning that it did so) as suggested by @rgutzen, which I also wrote into PR #739
Additionally, we found that the current NestIO decides whether to read neo.AnalogSignals or neo.SpikeTrain based on the file name alone. This is problematic since Nest 3.2 (maybe others too) saves spikes to the .dat format by default, which the IO expected to be analogsignals. I fixed this issue by adding a new keyword target_object which then determines what kind of object will be read. This does not allow to read analogsignals and spikes into the same segment (two different calls are needed), but this seems a rather good practice in my opinion, since spiketrains and analogsignales are rarely combined into the same segment.
I tested the code to load a .dat file generated with Nest 3.2 and it works, I can add a proper test somewhere if you want.
Best,
Aitor

Hi all,
thanks for taking care of this.
We created a NEST-elephant tutorial and we ran into this issue with the nestio and NEST 3.3.
Since this was tested with NEST 3.2, maybe add this info to the docstring(?):
https://github.com/NeuralEnsemble/python-neo/blob/50abfbce832edb1c0f8364ac34c8a6442e1a3edf/neo/io/nestio.py#L4

Just rebased the branch and some chaos has followed. My local history looks as one would expect, with this branch now splitting from the current master. I am not sure why Github interprets that all the previous commits are part of this PR :/

@JuliaSprenger I created a small NEST (3.3) simulation where I record both spikes and analogsignals. The results and simulation script are here: test_NEST_simulation_and_results.zip.
The current intended use is therefore something like this:
import neo
import quantities as pq
import matplotlib.pyplot as plt
# Load spikes
io = neo.NestIO(['sr1-142-0.dat', 'sr2-154-0.dat'], target_object='SpikeTrain')
bl = io.read_block([], t_start=0*pq.s, t_stop=20*pq.s)

# Plot spikes
plt.figure(figsize=(4, 3))
for i, st in enumerate(bl.segments[0].spiketrains):
    if st.annotations['source_file'] == 'sr1-142-0.dat':
        color = 'b'
    elif st.annotations['source_file'] == 'sr2-154-0.dat':
        color = 'r'
    plt.scatter(st.times, [i]*len(st.times), marker='|', c=color)
plt.xlabel('Time (ms)')
plt.ylabel('Neurons')
plt.savefig('nest_spikes.png', dpi=300)

# Load anasigs
io = neo.NestIO(['vm1-141-0.dat', 'vm2-153-0.dat'], target_object='AnalogSignal')
bl = io.read_block([], t_start=0*pq.s, t_stop=20*pq.s)

# Plot anasigs
plt.figure(figsize=(4, 3))
for i, ana in enumerate(bl.segments[0].analogsignals):
    if ana.annotations['source_file'] == 'vm1-141-0.dat':
        color = 'b'
    elif ana.annotations['source_file'] == 'vm2-153-0.dat':
        color = 'r'

    plt.plot(ana.times, 8*i + ana.magnitude, c=color, alpha=0.5)

plt.xlabel('Time (ms)')
plt.ylabel('Neurons')
plt.savefig('nest_anasigs.png', dpi=300)

Which creates these two plots in this case:

@JuliaSprenger I just fixed the history, are there any open issues with the current version of the NestIO?

@JuliaSprenger I just fixed the history, are there any open issues with the current version of the NestIO?

or @apdavison

Hi @heplesser , thanks for the comments and the detailed description of NEST outputs. I think this information is super helpful in driving this IO forwards. Also, thanks for your insights in problems that may arise at the "full scale" level.

Hello everyone!
After discussion with @mdenker and @apdavison yesterday, I provide here a little script that creates a network consisting of

aeif_cond_alpha neurons (to show multimeter recording)
iaf_psc_alpha_ps neurons (to show precise time recording)
weight_recorder (to show weight recording)

Running the script will generate a number of spike recorder, voltmeter, multimeter and weight recorder output files with different settings. Note in particular the effect of "time_in_steps": If false (default), time is recorded in ms under heading "time_ms", if true time will be recorded in columns "time_step" and "time_offset", where the actual spike time is given by time_step * resolution - time_offset. Unfortunately, the resolution is not stored in the file, so I suggest to not support these files for now.
Do not rely on ordering of columns or rows. In parallel simulations, there will be one file per virtual process, enumerated by the last part of the file name.
NEST 3 files can be safely identified by the
# NEST version: 3.7.0-rc1
# RecordingBackendASCII version: 2

header. NEST 2 files do not have these headers. I would suggest to use the header for identification, also to be prepared for future versions of the ASCII backend.
nest.ResetKernel()
nest.overwrite_files = True

g = nest.Create('poisson_generator', params={'rate': 100000})
n = nest.Create('aeif_cond_alpha', 10)
p = nest.Create('iaf_psc_alpha_ps', 5)

rnt = nest.Create('spike_recorder', params={'record_to': 'ascii',
                                            'label': 'aeif_spikes_times'})
rns = nest.Create('spike_recorder', params={'record_to': 'ascii',
                                           'time_in_steps': True,
                                            'label': 'aeif_spikes_steps'})

rpt = nest.Create('spike_recorder', params={'record_to': 'ascii',
                                            'label': 'precise_spikes_times'})
rps = nest.Create('spike_recorder', params={'record_to': 'ascii',
                                           'time_in_steps': True,
                                            'label': 'precise_spikes_steps'})

vt = nest.Create('voltmeter', params={'record_to': 'ascii',
                                      'label': 'voltmeter_times'})
vs = nest.Create('voltmeter', params={'time_in_steps': True,
                                      'label': 'voltmeter_steps'})

mm1t = nest.Create('multimeter', params={'record_from': ('g_ex', 'g_in', 'V_m', 'w'),
                                         'record_to': 'ascii',
                                            'label': 'multimeter_1ms'})
mmrt = nest.Create('multimeter', params={'record_from': ('g_ex', 'g_in', 'V_m', 'w'),
                                         'record_to': 'ascii',
                                         'interval': nest.resolution,
                                            'label': 'multimeter_res'})

w = nest.Create('weight_recorder', params={'record_to': 'ascii',
                                            'label': 'weight_rec'})

nest.SetDefaults('stdp_synapse', {'weight_recorder': w})

nest.Connect(g, n)
nest.Connect(n, n, syn_spec={'synapse_model': 'stdp_synapse', 'weight': 100.})
nest.Connect(n, p, syn_spec={'weight': 1000.})
nest.Connect(n, rnt + rns)
nest.Connect(p, rpt + rps)
nest.Connect(vt + vs + mm1t + mmrt, n)

nest.Simulate(100)

Hi @heplesser ! Thanks for pushing this forward and coming up with good test examples.
I think we should not introduce a NEST dependency to the neo tests, I would instead suggest to create several (small) files for multiple NEST versions. The files should have descriptive names like "nest3.7_voltmeter_ascii.dat". What do you think?

@morales-gregorio I agree, otherwise we could easily end up with circular dependencies and all-to-heavy test setups. Frozen example files for testing should work nicely. It should suffice to base the versioning on the ASCII Recording Backend version, which changes more slowly than the NEST version.

Hi @heplesser ! Thanks for pushing this forward and coming up with good test examples.
I think we should not introduce a NEST dependency to the neo tests, I would instead suggest to create several (small) files for multiple NEST versions. The files should have descriptive names like "nest3.7_voltmeter_ascii.dat". What do you think?

@morales-gregorio I agree, otherwise we could easily end up with circular dependencies and all-to-heavy test setups. Frozen example files for testing should work nicely. It should suffice to base the versioning on the ASCII Recording Backend version, which changes more slowly than the NEST version.

@heplesser could you create these files? 😇

Hey @morales-gregorio ,
the current test files for testing NEST are located on NeuralEnsemble /
ephy_testing_data in nest here: https://gin.g-node.org/NeuralEnsemble/ephy_testing_data/src/master/nest
The contents are:
0gid-1time-1256-0.gdf
0gid-1time-2gex-1262-0.dat
0gid-1time-2gex-3Vm-1261-0.dat
0gid-1time-2Vm-1259-0.dat
0gid-1time-2Vm-3gex-4gin-1260-0.dat
0gid-1time-2Vm-3Iex-4Iin-1264-0.dat
0gid-1time_in_steps-1258-0.gdf
0gid-1time_in_steps-2Vm-1263-0.dat
0time-1255-0.gdf
0time_in_steps-1257-0.gdf
brunel-delta-nest_mod.py  # <--- script that generated the files
N1-0gid-1time-2Vm-1265-0.dat
N1-0time-1Vm-1266-0.dat
N1-0Vm-1267-0.dat

I have copied and formatted the above script into generate_data.py:
"""
Script to generate test data for the nestIO.

This script creates a network simulation using the NEST simulator (https://www.nest-simulator.org/).
It generates output files for spike recorders, voltmeters, multimeters and weight recorders with
different settings. Note the effect of "time_in_steps": If false (default), time is recorded in ms
under heading "time_ms"; if true, time will be recorded in columns "time_step" and "time_offset",
where the actual spike time is given by time_step * resolution - time_offset. The resolution is not
stored in the file. In parallel simulations, there will be one file per virtual process, enumerated
by the last part of the file name.

Usage:
    python generate_data.py

"""

import nest

def main():
    """
    Executes the main logic of the script.

    This function sets up a simulation using the NEST simulator (https://www.nest-simulator.org/)
    for simulating neural networks. It creates a network consisting of aeif_cond_alpha neurons
    (to show multimeter recording), iaf_psc_alpha_ps neurons (to show precise time recording),
    and weight_recorder (to show weight recording). The simulation is run for a specified duration.

    Note: Do not rely on ordering of columns or rows. In parallel simulations, there will be
    one file per virtual process, enumerated by the last part of the file name.

    Parameters:
        None

    Returns:
        None

    """
    nest.ResetKernel()
    nest.overwrite_files = True

    g = nest.Create("poisson_generator", params={"rate": 100000})
    n = nest.Create("aeif_cond_alpha", 10)
    p = nest.Create("iaf_psc_alpha_ps", 5)

    rnt = nest.Create(
        "spike_recorder",
        params={"record_to": "ascii", "label": "aeif_spikes_times"},
    )
    rns = nest.Create(
        "spike_recorder",
        params={
            "record_to": "ascii",
            "time_in_steps": True,
            "label": "aeif_spikes_steps",
        },
    )

    rpt = nest.Create(
        "spike_recorder",
        params={"record_to": "ascii", "label": "precise_spikes_times"},
    )
    rps = nest.Create(
        "spike_recorder",
        params={
            "record_to": "ascii",
            "time_in_steps": True,
            "label": "precise_spikes_steps",
        },
    )

    vt = nest.Create("voltmeter", params={"record_to": "ascii", "label": "voltmeter_times"})
    vs = nest.Create("voltmeter", params={"time_in_steps": True, "label": "voltmeter_steps"})

    mm1t = nest.Create(
        "multimeter",
        params={
            "record_from": ("g_ex", "g_in", "V_m", "w"),
            "record_to": "ascii",
            "label": "multimeter_1ms",
        },
    )
    mmrt = nest.Create(
        "multimeter",
        params={
            "record_from": ("g_ex", "g_in", "V_m", "w"),
            "record_to": "ascii",
            "interval": nest.resolution,
            "label": "multimeter_res",
        },
    )

    w = nest.Create("weight_recorder", params={"record_to": "ascii", "label": "weight_rec"})

    nest.SetDefaults("stdp_synapse", {"weight_recorder": w})

    nest.Connect(g, n)
    nest.Connect(n, n, syn_spec={"synapse_model": "stdp_synapse", "weight": 100.0})
    nest.Connect(n, p, syn_spec={"weight": 1000.0})
    nest.Connect(n, rnt + rns)
    nest.Connect(p, rpt + rps)
    nest.Connect(vt + vs + mm1t + mmrt, n)

    nest.Simulate(100)

if __name__ == "__main__":
    main()

which generates: (.zip attached)
aeif_spikes_steps-18-0.dat
aeif_spikes_times-17-0.dat
generate_data.py
multimeter_1ms-23-0.dat
multimeter_res-24-0.dat
precise_spikes_steps-20-0.dat
precise_spikes_times-19-0.dat
voltmeter_times-21-0.dat
weight_rec-25-0.dat

nestIO_test_data.zip
Refer to this readme here: https://gin.g-node.org/NeuralEnsemble/ephy_testing_data to add the files.

Thanks @Moritz-Alexander-Kern ! Can you point to an example of how to set up the tests properly? I guess for CI one does not want to force the example data to be stored in the same repository, right?

There are 8 failing tests. I've looked into the first of these, and it seems to be that the case of id_column=None is not being correctly handled.


Dynamically adjust cache duration?

Is it possible to dynamically change the cache duration? I would like to have a configuration in app configuration to govern how frequently the cache should be renewed, and then I could adjust it without having to restart the app.
This is important because, for example, when we launch a new feature we decrease the cache to about 30s and as we get more confident about it, we up this to 1m, then 5m, then 10m... each one of these requires a whole new deploy

@Leonardo-Ferreira
Using AppConfiguration provider, the configuration is loaded at startup and revalidated via a sentinel key (aka ConfigureRefresh) on your desired probing interval. There isn't particular cache expiration of the already loaded configuration settings. If the provider detects a change (on the sentinel key), then refreshes the configuration. In any other case the configuration stays as is.
Do you prefer to dynamically adjust sentinel key probing interval?

@drago-draganov
When you plug your code to app configuration and passes on the connection string, you can set the local cache duration which defaults to 30 sec and that is the config I meant

This is indeed the probing interval for registered settings (sentinel key) for refresh. Basically, how long to wait before checking again if modified. It controls the frequence of checks for refresh. Your configuration will not be reloaded, unless the sentinel key is updated, regardless of that frequency.
I see how the naming may be confusing though.

thanks, but the question remains: can I adjust that dynamically?

There isn't anything built-in at the moment to allow that.
How would you prefer to adjust it dynamically? Ideal case.

I see two good options that somewhat overlap each other:

When configuring (at startup) you could provide the path to a configuration within that same app configuration indicating the cache duration, this way as the value changes on app configuration root, the duration of cache would reflect that change (TBD: should this value respect Sentinel rules?)
At portal level, you could define a global value for cache duration, which would work like suggestion 1

The order of overwriting would be:

No settings defaults to 30 sec (as is)
Global Value overwrites "no setting"
Code specific static value overwrites global value
Code specific dynamic value overwrites static value

it would also be nice if:

the SDK had an way to report when the next fetch will be (wether if its a check for sentinel change or a reload)
a event mechanism that the application could be notified that a refresh has occurred (bonus points if it provides the old/new values that were changed)
a "force reload" mechanism

Thanks for the suggestion, @Leonardo-Ferreira! We will look into it.
As far as diagnostics, please follow Logging and Monitoring
To detect reload of the configuration, we can use IConfiguration.GetReloadToken


multi: allow specifying min confs for on-chain transactions

As stated in my issue, would be nice to have this so channels can be CPFP to confirm quicker during a fee spike.

Still relevant?


Setup Netlify to preview rendered PRs

For every PR, Netlify will build an host the site at a custom domain just for that PR. Build failures and the links to the site preview are reported as status checks on the PR.
While Netlify is a commercial organization, the service is free for our use. We can continue to host the site on GitHub pages.
https://www.netlify.com/legal/open-source-policy

It works


Use pearl with custom pytorch model

Used to use pytorch + gym for RL.
Is there a way to make a custom neural net in pytorch and insert it into the pearl agent without using the pre-made settings for neural nets?

It is possible to directly provide a network instance to be used by policy learners implementing Temporal Difference (TD learning) methods:

all direct and indirect subclasses of DeepTDLearning (DeepQLearning, DeepSARSA, DoubleDQN, among others), since DeepTDLearning accepts an initialization parameter network_instance, which must be an instance of QValueNetwork.
While network_instance does not explicitly appear in the documentation of the __init__ methods of the subclasses, the parameter can be passed to them as a keyword argument (kwargs) and will be used by their super class.
Here's an example of network_instance being used for DeepQLearning.

All direct and indirect classes of QuantileRegressionDeepTDLearning (currently, this is only QuantileRegressionDeepQLearning).
This class also accepts a network_instance parameter, but this time it must be an instance of QuantileQValueNetwork.
There is currently no example of this in the code base because the test for QuantileRegressionDeepQLearning uses the default network type, but its use is analogous to the DeepQLearning example linked above.

Moreover, while it is not possible to directly provide network instances to the subclasses of ActorCriticBase, it is possible to request the use of custom user network classes. These policy learners accept actor_network_type and critic_network_type parameters which allows the user to specify the classes to be used for those networks. So the user can define their custom network classes and pass those to be used. They must be subclasses of ActorNetwork and QValueNetwork classes respectively.

@rodrigodesalvobraz
Thank you. it's clear.
Another small question: can i add custom properties to my agent? for example if i want my agent to have some properties that are properties of the agent, not of the environment, for example "money". Imagine i'm running a simulation of a market economy environment, and i want each agent to possess some float attributes like money, health, hunger etc.
Can i do this with pearl?
In gym i would simply insert these properties in my "agent" class

Adding a property to an object is always possible in Python in general, regardless of the libraries one might be using. If you have a object instance agent, writing agent.money = 10 will work and be accessible from then on.
However, I suspect this might not be what you would really want. If you want the RL algorithm to take these properties into account when making decisions for the agent (for example, the agent will be more likely to spend money if it has a lot of it, but will try to save when it does not), then what you really want is to include that information in the observations being received by the agent (if you happen to be using an environment, then the environment is a good candidate to keep this information and include it in the observations it provides). This is because RL algorithms will make decisions based on the information contained in observations (states).
It might be a bit odd to think that an agent's property is being included in the observations coming from the outside world since we tend to think of it as an internal property, but one way to think about it is that the agent is observing things, including how much money is in its pocket!
Because Pearl currently represents observations as tensors, you might want to use a representation such that the agent's properties of interest are concatenated with the rest of the observation into a single tensor.
To keep the property up-to-date (for example, to decrease the agent's money once a decision is made to spend some), the code receiving the action from the agent (in learning situations this will typically be the step method of the environment) must modify the property's value according to the action taken.
I hope this helps.

The solution you mention (keeping the properties inside the agent instance and concatenating them to the external observation) might actually work, too.
It seems to be a little less conventional. Usually in an RL problem one talks about a single type of observation, but here you have two types, the external one and the internal one obtained from concatenating agent's properties to the first one. Software libraries often makes assumptions based on conventions, so going a less conventional route may turn out tricky, but as far as I can tell right now your approach could work too.

thank you again


Support the new relics&charms/recipes/artifacts vault

Atlantis introduced a new shared vault. Support it in the UI

I'm working on it.

This feature is done, and ready for a merge on #116 .
Attached a sample UI:


"ponzu generate" with commented out interface implementations?

It would be a time saver to uncomment interface method implementations in generated code rather than have to find the code snippet and copy paste it? Though I'm sure some users can key it from memory, that's not me :)
Maybe ponzu generate could also add these methods but wrap them in comments? To use, just uncomment and extend a method if required.
A flag to populate or not on ponzu generate would also provide the user with a choice?

@olliephillips - interesting.. are you suggesting that we generate a commented out stub for interface APIs like Pushable on all generated types?
I had a thought a while back that we could decorate the ponzu gen c call with options like +push +hide etc to add this stub... but decided it was really not much work to just write that code. But I'll leave this open to get feedback from others..
cc/ @ponzu-cms/team - any thoughts?

@nilslice Yes, the function signatures are fairly simple but there are a quite a few interfaces that one might want to use. My thought was a single flag that would scaffold all of them. There's no bloat, since as commented, they're stripped at build. Just my thoughts. Hope others can jump in with their opinion

Adding a single flag to generate the commented out implementation stubs is something I feel much more comfortable with.. let's see what others think, but since it would be entirely opt-in, I don't have any problem with this at first thought. As for implementation, we'd just need to have some way to keep track of all the implementations, and/or use some tooling i.e https://github.com/josharian/impl

Closing this issue. No activity. Please reopen if need to.


Removed duplicate options from default config file (Fixes #63)

Thanks.
Just a minor nitpick. You should move the "Fixes #63" to the message body. E.g.:
Removed duplicate options from default config file

Fixes #63.


Revert "Update plugin io.sentry.jvm.gradle to v4.6.0"

Reverts ministryofjustice/hmpps-activities-management-api#881 because deploy is broken

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 87.27%. Comparing base (4967f2f) to head (165892f).

Additional details and impacted files
@@            Coverage Diff            @@
##               main     #882   +/-   ##
=========================================
  Coverage     87.27%   87.27%
  Complexity     3569     3569
=========================================
  Files           372      372
  Lines         12577    12577
  Branches       1024     1024
=========================================
  Hits          10977    10977
  Misses         1131     1131
  Partials        469      469

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


[BUG] Unistall can be broken in some scenarios

Describe the bug
Running the role with
- name: UnInstall tailscale
  ansible.builtin.include_role:
    name: ansible_role_tailscale
  vars:
    state: absent # noqa: var-naming[no-role-prefix]
    verbose: true # noqa: var-naming[no-role-prefix]
    tailscale_tags: ['sometag']

Might fail with the error:
TASK [ansible_role_tailscale : Uninstall | De-register Tailscale Node] *********
fatal: [host]: FAILED! => changed=false
  ansible_facts:
    discovered_interpreter_python: /usr/bin/python3
  cmd: tailscale status; tailscale logout
  delta: '0:00:00.104574'
  end: '2024-02-13 02:29:31.996263'
  msg: non-zero return code
  rc: 1
  start: '2024-02-13 02:29:31.891689'
  stderr: '500 Internal Server Error: invalid key: API key kzTEHB4CNTRL not valid'
  stderr_lines: <omitted>
  stdout: |-
    # Health check:
    #     - not logged in, last login error=invalid key: API key kzTEHB4CNTRL not valid

    unexpected state: NoState
  stdout_lines: <omitted>

That happens because during execution these tasks:
https://github.com/artis3n/ansible-role-tailscale/blob/dd09d2f1df9e09bcd1b5e51bbafe608082ff7e9e/tasks/uninstall.yml#L2-L18
The tailscale status exits with exit code 1 and then tailscale logout exit code 1:
root@host:~# tailscale status
# Health check:
#     - not logged in, last login error=invalid key: API key kX1yFH7CNTRL not valid

unexpected state: NoState
root@host:~# echo $?
1
root@host:~# tailscale logout
500 Internal Server Error: invalid key: API key kX1yFH7CNTRL not valid
root@host:~# echo $?
1

Expected behavior
Do not fail in such scenarios.
Additional context
PR is coming. :)

Actually, I am thinking about running tailscale logout only if tailscale_status.rc != 0

Why shouldn't these tasks fail, @McSim85 ? If we are unable to perform a tailscale logout (or tailscale status) then we cannot ensure that the infrastructure has been decommissioned from Tailscale


[Bug] author_signature not exists for non-anonymous admins

There is field which show author_signature for anonymous admins but this field isn't exists for non-anonymous and there is no other field to know their title in group

This is not a bug. Custom administrator title is not used as author signature for non-anonymous administrators. They have their first name and last name for that.
You can find administrator custom titles in ChatMember object, returned by various methods.


Making this project an actual template

Hi,
I'm trying template-haskell for my own project. I think it would be useful if template-haskell was an actual GitHub template instead of just a project. Here is an actual GitHub template: https://github.com/nix-dot-dev/getting-started-nix-template. As you can see, the main page of this project has a big green "Use this template" button.
Basically, you just have to check 1 checkbox in the GitHub settings page of your project: https://docs.github.com/en/github/creating-cloning-and-archiving-repositories/creating-a-template-repository.

Thanks for the tip, done!


no fast.ai?

I came across this wonderful repo. But I have a question:

why fastai isn't included in the list of resources?

I am thinking of spending time on their course as a newbies of ML, I do want to have quality feedbacks in advance. Thank you.

alright, it does have it.. my bad.


UUID 1-4 generator

Implement UUID 1-4 generator with the option to change the random seed by args

 UUID [1-4]


Code quality fix - Local Variables should not be declared and then immediately returned or thrown.

This pull request is focused on resolving occurrences of Sonar rule
squid:S1488 - Local Variables should not be declared and then immediately returned or thrown.
You can find more information about the issue here:
https://dev.eclipse.org/sonar/rules/show/squid:S1488
Please let me know if you have any questions.
Faisal Hameed

I liked all the other changes submitted and have merged them, but I'm hesitant on this one. I often like having a local variable before returning because it makes debug traces a little more straightforward to parse, because you can easily see the value about to be returned while still being in the stack context of the rest of the method to see how it got there.
The link you provided on it being bad practice doesn't seem to actually explain why it's bad practice (least not that I saw) and merely says that one reason developers do it isn't necessary, but that isn't the reason I like it anyway. Can you give an argument for why this should be changed?

Thanks @jmacglashan for merging my PRs. I agree with you that defining local variables makes debugging easy.
As it is mentioned in the link description, these local variables are internal representation and the method name should be sufficient for callers to know what is being returned.

Okay, since it otherwise doesn't harm anything, but has utility for debugging, I'll leave as is.


I met a question when running SOFA_si.sql

Sorry for bothering you.
When I ran the SOFA_si.sql, it informed me that:
ERROR:  column s.si_starttime does not exist
LINE 27:     between s.si_starttime
And I found the original code in SOFA_si.sql are as follows:
vaso_mv as
(
select ie.icustay_id
-- case statement determining whether the ITEMID is an instance of vasopressor usage
, max(case when itemid = 221906 then rate end) as rate_norepinephrine
, max(case when itemid = 221289 then rate end) as rate_epinephrine
, max(case when itemid = 221662 then rate end) as rate_dopamine
, max(case when itemid = 221653 then rate end) as rate_dobutamine
from suspinfect_poe s
inner join icustays ie
on s.icustay_id = ie.icustay_id
inner join inputevents_mv mv
on ie.icustay_id = mv.icustay_id
and mv.starttime
between s.si_starttime
and s.si_endtime
where itemid in (221906,221289,221662,221653)
But when I found and opened the table suspinfect_poe, I couldn't find the column _si_starttime_and si_endtime.
The sql code named as suspicion-of-infection.sql establishing the table suspinfect_poe is as follows:
DROP TABLE IF EXISTS suspinfect_poe CASCADE;
CREATE TABLE suspinfect_poe as
with abx as
(
select icustay_id
, suspected_infection_time
, specimen, positiveculture
, antibiotic_name
, antibiotic_time
, ROW_NUMBER() OVER
(
PARTITION BY icustay_id
ORDER BY suspected_infection_time
) as rn
from abx_micro_poe
)
select
ie.icustay_id
, antibiotic_name
, antibiotic_time
, suspected_infection_time
, specimen, positiveculture
from icustays ie
left join abx
on ie.icustay_id = abx.icustay_id
and abx.rn = 1
order by ie.icustay_id;
I am a new learner of sql and mimic, and the question bothers me a lot. Your help will be much appreciated. @alistairewj @tompollard

I have found the key to solve the problem.

hey how do you solve with the problem?


Remove definitions of old reporting round columns

https://dluhcdigital.atlassian.net/browse/FPASF-494
Change description
Removes the old reporting round columns from python model defintions. Once this is out, we know that no code is using those columns, and we can remove them safely in a future migration PR.
Note: I know that the DB migrations check will fail, because there are outstanding migrations. But this is intentional.

[ ] Unit tests and other appropriate tests added or updated
[ ] README and other documentation has been updated / added (if needed)
[ ] Commit messages are meaningful and follow good commit message guidelines (e.g. "FS-XXXX: Add margin to nav items preventing overlapping of logo")

How to test
If manual testing is needed, give suggested testing steps
Screenshots of UI changes (if applicable)

(requires https://github.com/communitiesuk/funding-service-design-post-award-data-store/pull/733 first)


HSL and HSLA are not supported yet?

color: green;
color: hsl(180, 50%, 30%);
background: hsla(120, 30%, 60%, 0.15);

The 1st color works well;
The 2nd and 3rd do not work.
HSL and HSLA are more friendly to adjust than RGB. Hope they be supported.
Thanks!

+1 for HSL and HSLA support. Otherwise - super tight package!

+1 !

+1


add field of affiliation for instructors

It's often useful to know what institute or company a person is at in a particular area. It sounds like we're collecting that data, but it would be good have that field available to view in Amy. It can be just a regular text field.
I know there will not be uniformity of how people enter their university - some might enter UCLA and some University of California, Los Angeles, but as administrators having that information available to view gives us helpful information for scheduling.

We've recently gathered affiliation via instructor profile survey.
What I'd like to know is if we need to track people who are affiliated with SCF Organizational Members or something more general (as in Tracy's example: UCLA, or AGH-UST as in my case).

Organizational membership can change over time, so let's record every
affiliation that people give us.

I agree we should just have a general organizational affiliation category.
Ultimately it would also be good to track which instructors are at Organizational Member universities. But, as Greg says, that will change, so I'm not quite sure how to track that.
I can see a use case where an Affiliate university wants to know who its current local instructors are when they're trying to schedule local workshops. Or we will want statistics on how many of our instructors are at member universities. Maybe we'll just have to go through periodically and categorize them ourselves.

Ref #375.

Closed via #433.


Get IP address of a website

How can we get the ip address of a website using httpie? I couldn't find any info on httpie documentation.

HTTPie currently doesn't show you the iP address, but there other tools you can use for that, for example:
ping example.org
resolveip example.org

can we expect this feature in new revisions?

Yes, it will probably become part of the --verbose, -v output.

Can we create addon/extensions to httpie? I see the following lines of code does the job
import socket

print socket.gethostbyname('google.com')
>> <IP_ADDRESS>