gradle: add setup hook

Description of changes

This adds previously missing Gradle lockfiles... although the official lockfile format is very bad - so instead we MITM Gradle with a caching proxy. Trust me, I've tried - MITMing Gradle is the only solution that consistently works, because Java devs have no idea what lockfiles are supposed to do (for starters, Gradle lockfiles don't always lock dependencies).
Said MITM caching proxy is added. Right now it's custom made for Gradle, but in theory it could be used for other software, to get rid of FODs. It offers basic configuration via regex.
A downside is the fact those "lockfiles" (meaning a JSON record of all files fetched by Gradle) now have to be stored in nixpkgs. I think this is much better than FODs anyway, and I've added an update script to generate them.
This PR also converts all packages that depend on Gradle to use the setup hook

In turn, this solves the following longstanding Gradle+Nix issues:

Package aliases like okio -> okio-jvm now work with no manual workarounds
SNAPSHOT dependencies don't have to be manually pinned - the "lockfile" takes care of it (because it includes maven repo metadata in text format, if Gradle accesses it at all)
FODs aren't used anymore, letting Nix fetch the files. This should increase the level of deduplication
Many Gradle scripts download stuff from random sources - that can be included in the lockfile. That said, there's no logic to exclude URLs from the cache currently, and no special logic for something like Github releases.
Using different dependencies (from different repos) with the same name and version number is now possible (yes, it is something Gradle actually does at times)
No code duplication is necessary anymore (currently there are long "deps" FODs for each Gradle app)

Open questions

Is the attr naming any good?
What is the best structure for the setup hooks added by this PR?

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

It would be great to get rid of # perl code mavenizes pathes indeed 😀

although the official lockfile format is very bad

Are there some improvements that could be suggested to the upstream, or is the format hopelessly bad?

As the lockfiles' size is quite significant, is there a way to make it noticeably smaller (i.e. 30%+)? Maybe a postprocessing script to put deps with a common prefix (e.g. from the same Gradle repo) to the same attrset?

Would this sort of change be simple enough to implement in the proxy directly? Change the type of Pages to
struct Pages(BTreeMap<String, BTreeMap<String, Page>>);

so that the first level is the URL scheme+authority:
{
 "https://jcenter.bintray.com": {
  "com/google/code/findbugs/jsr305/3.0.2/jsr305-3.0.2.jar": {
   "sha256": "dmrSoHg/JoeWLIrXTO7MOKKLn3Ki0IXuQ4t4E+ko0Mc="
  },
  "com/google/code/findbugs/jsr305/3.0.2/jsr305-3.0.2.pom": {
   "sha256": "GYidvfGyVLJgGl7mRbgUepdGRIgil2hMeYr+XWPXjf4="
  },
  "com/google/code/gson/gson-parent/2.9.1/gson-parent-2.9.1.pom": {
   "sha256": "fKCEXnNoVhjePka9NDTQOko3PVIPq5OmgDGK1sjLKnk="
  }
 }
}

In this tiny sample it saves about 8%, which is pretty far from the desired 30%+ 😦

Then maybe use the gradle coordination:
{
 "https://jcenter.bintray.com": {
  "com.google.code:findbugs.jsr305:3.0.2": {
   "jar": "dmrSoHg/JoeWLIrXTO7MOKKLn3Ki0IXuQ4t4E+ko0Mc=",
   "pom": "GYidvfGyVLJgGl7mRbgUepdGRIgil2hMeYr+XWPXjf4="
  }
 }
}

Related to #151524

Are there some improvements that could be suggested to the upstream, or is the format hopelessly bad?

Here's my list of issues:

it generates a lockfile tree, not a single lockfile. Each build.gradle produces 2 lockfiles (for the build script and for the project itself) and 2 "dependency verification" hash lists, and there may be multiple build.gradle files in a project. This means there can be lots of duplication
it doesn't store which repo dependencies come from, which actually makes it useless, not only for us to fetch deps from, but for Gradle itself, because different parts of a project may use the same dependency version from different repos, and it may have a different hash (that's not theoretical, it happens in practice)
It doesn't pin SNAPSHOT deps on purpose, pretty horrible huh?
There's no native way to bundle a project's dependencies. What we're doing right now is a massive hack - and it shows, because many manual workarounds are needed, like okio->okio-jvm or kotlin-stdlib-common->kotlin-stdlib redirections. "The Gradle Way" is copying the entire GRADLE_USER_HOME... which isn't reproducible at all
Maybe for the aforementioned reasons, maybe because it adds seemingly useless overhead, nobody actually uses lockfiles outside of locked down enterprise contexts
Many projects (like Ghidra or some Signal-related project I don't remember the name of) have additional parts of the build script fetch stuff from the internet.

At this point, isn't it normal to give up on Gradle lockfiles? It seems like the entire ecosystem is built in a way as to be as non-reproducible as possible. From what little interaction I had with upstream, it doesn't seem like they'd be very willing to improve it either, but you're welcome to try.
Good thing is the mitm cache, with some modifications (to add scripting), can actually be used for other build systems that currently use FODs, so I think this way of doing it is both pragmatic and useful in the long run. Even if there isn't a http proxy setting, we can hijack /etc/ssl (or whatever other paths) and use a proxifier, which should defeat even the most uncooperating build systems. As I already mentioned, this is far from ideal, but it's pragmatic.

Then maybe use the gradle coordination:

This gives an improvement of 45.835%! It would be possible to squeeze another 0.3% if snapshots didn't exist, but whatever, that doesn't matter much.

I think I got this in a shape suitable for most Gradle derivations currently in nixpkgs, should I try converting more derivations to the setup hook? I'm slightly familiar with them from my previous attempt at creating a builder for Gradle

@abbradar @averelld @charles-dyfis-net @chkno @delroth @emilytrau @expipiplus1 @fgaz @gavinrogers @gebner @jiegec @liff @linsui @lorenzleutgeb @Ma27 @mguentner @nagy @7c6f434c @rnhmjoj @roblabla @SharzyL @TheKostins @thoughtpolice @wentasah @wirew0rm
this PR is now ready for reviews (meaning it's complete from my point of view), and I'm pinging you as this affects the packages you're maintaining

Is maven-metadata really needed?

maven-metadata is required for pinning snapshot deps

Excellent work! Gradle has always been a thorn in the side of nixpkgs.
I have a security-related question: what happens if a malicious contributor edits the "lockfile" so that it pulls in a malicious dependency? Ideally it should make the build fail.

what happens if a malicious contributor edits the "lockfile" so that it pulls in a malicious dependency?

Good question. To inconspicuously add a malicious dependency you would either have to add it to the Maven repos the program uses (this is sadly possible), or create a new top-level key in deps.json (which is relatively easy to spot in review, but can just as easily go past the eyes of the maintainers). I guess it's better to make attempting to recreate this file part of review guidelines.

I guess it's better to make attempting to recreate this file part of review guidelines

It won't be enough, people aren't going to read/remember ecosystem-specific guidelines, especially for simple version bumps

maybe even add this as a comment to the start of each deps.json file.

This won't be enough either. Github hides long files in the review screen, and people aren't going to expand a lockfile. A comment right next to the version could work, but I can see how that could fail too.
To be sure, we'd have to find a way to check the lockfile (or notify reviewers) automatically.

maven-metadata is required for pinning snapshot deps. It isn't used anywhere else.

I thought you can generate the metadata file if it's really needed. Looks like the only info in it is the snapshot version.

To be sure, we'd have to find a way to check the lockfile (or notify reviewers) automatically.

@fgaz This would have to be part of GitHub CI - ofborg (for example) needs to run the updateDeps script and make sure the hash matches. I'll ask on Matrix.

I thought you can generate the metadata file if it's really needed. Looks like the only info in it is the snapshot version.

@linsui There are 9 snapshots in total in this PR. This adds a total of like 20kb. Compared to the total lockfile size, this is nothing. It's true that it can be generated, but I don't see much benefit in that, as it would have to be generated for every single dependency.

I'm sad that we're vendoring lockfiles and increasing the size of nixpkgs

@delroth this is the general direction of nixpkgs - when the original build tool provides hashes, use it, when it doesn't, lock the dependencies by ourselves. I do think it's unfortunate that Gradle has such poor support for dependency locking, but it's just a product of the wider Java ecosystem which, among other things, encourages vendoring and bundling everything in jars rather than building from source.

can we also have a default updateScript which works for most of them?

The update script has to essentially do the following - first, run nix-update (or any other updater), then if it changed anything, run updateDeps. It's better to allow using any updater in place of nix-update, what do you think would be a good interface for this? Off the top of my head, I can suggest gradle.updateScript { inherit updateDeps; } (with updateDeps falling back to a default value if it's not specified)

I understand JSON makes that annoying. I wonder how tied we are to JSON in this case

We aren't tied to JSON at all, it's just relatively easy to make JSON decently compact. I will add this comment when I collect more feedback (and the wording needs to be decided beforehand, as it's not very easy to update deps.json in all ~20-30 commits).

There are 9 snapshots in total in this PR. This adds a total of like 20kb. Compared to the total lockfile size, this is nothing. It's true that it can be generated, but I don't see much benefit in that, as it would have to be generated for every single dependency.

Yep, but I see no reason to vendor it.
I saw lots of waiting for lock message when downloading the deps. Can they be avoided?
Does the mitm-cache fetcher works with (mitm-)proxy? It seems that's not considered.

Yep, but I see no reason to vendor it.

@linsui The reason is that the file doesn't have to be generated. Generating it is much harder than just saving it. In this case, not saving it reduces a lot of convenience, as either maven-metadata would have to be generated in Nix for each dependency, or the snapshot/latest.integration pins would have to be brought back to derivations' code, which is bad.

I saw lots of waiting for lock message when downloading the deps. Can they be avoided?
Does the mitm-cache fetcher works with (mitm-)proxy? It seems that's not considered.

The deps fetcher simply uses fetchurl and merges it using runCommand. It doesn't use mitm-cache in any way. mitm-cache is only used for generating deps.json, and for feeding the fetched deps to Gradle.

The deps fetcher simply uses fetchurl and merges it using runCommand. It doesn't use mitm-cache in any way. mitm-cache is only used for generating deps.json, and for feeding the fetched deps to Gradle.

Oh, that's great! Then I thought those warnings are also from fetchurl.

JabRef builds and works for me. :+1:

jextract builds and works. Thanks for your work!

@fgaz I can alternatively forbid specifying redirects in the lockfile, simply always replacing them with the hash of the file to be redirected to. That should be enough to stop malicious edits. I guess maven-metadata.xml will stay susceptible to edits, and I'm not sure if it can be used for injecting code. If this is a big concern, I guess I could autogenerate maven-metadata.xml for all deps, though I'd really rather not.

fastddsgen fails to build. Am I doing something wrong?
$ nix build github:NixOS/nixpkgs/pull/272380/head#fastddsgen
...
Daemon will be stopped at the end of the build
> Task :buildIDLParser FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/build/source/thirdparty/idl-parser/build.gradle' line: 70

* What went wrong:
Could not determine the dependencies of task ':idl-parser:jar'.
> Could not resolve all files for configuration ':idl-parser:runtimeClasspath'.
   > Could not find org.antlr:antlr4:4.5.
     Searched in the following locations:
       - https://repo.maven.apache.org/maven2/org/antlr/antlr4/4.5/antlr4-4.5.pom
     If the artifact you are trying to retrieve can be found in the repository but without metadata in 'Maven POM' format, you need to adjust the 'metadataSources { ... }' of the repository declaration.
...

Trying to run fastddsgen.updateDeps. On a NixOS system, I get:
error: creating directory '/tmp/nix-shell-1-0': Read-only file system

Full log here.
When I do the same on non-NixOS distro, the error is different:
...
[I][2023-12-16T00:19:23+0100] Executing '/nix/store/x0xwspxbsysw7vkf2947yl4bzf7hlhr7-gradle-commands.sh' for '[STANDALONE MODE]'
/usr/bin/env: 'nix-shell': No such file or directory
[I][2023-12-16T00:19:23+0100] pid=2584153 ([STANDALONE MODE]) exited with status: 127, (PIDs left: 0)

Of course, nix-shell is in my $PATH:
$ which nix-shell
/nix/var/nix/profiles/default/bin/nix-shell

As far as I understand,  this nix-shell is called from nsjail, where the environment might be different.

I worked around error: creating directory '/tmp/nix-shell-1-0': Read-only file system by:
--- a/pkgs/build-support/gradle/update-deps.nix
+++ b/pkgs/build-support/gradle/update-deps.nix
@@ -135,7 +135,7 @@ writeShellScript "fetch-deps.sh" ''
   # nsjail isn't necessary, it's only used to prevent messy build scripts from touching ~
   ${nsjail}/bin/nsjail \
     -N --keep_caps --chroot / --disable_rlimits \
-    -T /home -T /root -R ${path} \
+    -T /home -T /root -T /tmp -R ${path} \
     ${lib.escapeShellArgs (map (x: "-E${x}") keep)} \
     -E NIX_PATH=nixpkgs=${path} \
     ${gradleScript} "$source"

Then updateDeps fails with:
FAILURE: Build failed with an exception.

* What went wrong:
java.lang.NullPointerException (no error message)

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

Full log with --stacktrace is here. The stack trace starts with:
* Exception is:
java.lang.NullPointerException
	at net.rubygrapefruit.platform.internal.FileSystemList.add(FileSystemList.java:31)
	at net.rubygrapefruit.platform.internal.jni.PosixFileSystemFunctions.listFileSystems(Native Method)
	at net.rubygrapefruit.platform.internal.PosixFileSystems.getFileSystems(PosixFileSystems.java:30)
	at org.gradle.internal.watch.vfs.impl.DefaultWatchableFileSystemDetector.detectUnsupportedFileSystems(DefaultWatchableFileSystemDetector.java:62)

Perhaps, this is also related to the sandbox.

People generally already complain about Cargo.locks and package-lock.json being to big. Do we really need to make that problem worse?

I'd like to remind of #254798 (libwebp vulnerability) where a major pain-point was the heavy usage of FOD for dependency resolution making it fairly tricky to find out which packages are actually affected and now this can be derived from the build closure. A follow-up would be to implement a facility that allows applying patches to these dependencies, but that's out of scope IMHO.
Then, we'd be able to patch out the next log4j on our own for each Java/Kotline package built from source with this facility.
Another note from me: I always hate packaging new versions of signald because it always escalates extraordinary (just take a look at what got removed from signald/default.nix and you'll know what I mean ;-) )
Please note that I don't have the time right now to review the actual fetcher, this is mostly about the approach and how it affects packages I'm maintaining, specifically signald.
This gets a :+1: from me under the premise that we find a reasonable answer to

I have a security-related question: what happens if a malicious contributor edits the "lockfile" so that it pulls in a malicious dependency?

that gets well-documented somewhere.

Can confirm that pdftk builds and some of the basic functionality was tested succesfully.

I understand that the fixed output derivation won't work if upstream didn't do its job properly, but I still don't like this trend of adding huge generated lock files to Nixpkgs.
Anyway cie-middleware-linux looks good and appears to be reproducible.

My 2 cents, having worked on and off on gradle2nix:

You can get dependency information (artifact IDs, filenames and hashes) from Gradle itself; I've done it in a couple of ways.

One is via dependency verification, and that's what I do in android-nixpkgs in an update-locks script.
Another way is to use Gradle's build listener facilities to detect configuration resolve events during the build. Gradle has a plugin meant for uploading software bill-of-materials info to GitHub called the GitHub Dependency Graph Plugin, and I'm currently working on repurposing this code to produce lockfiles that work for producing fixed-output derivations for each dependency. This has the most chance of success, IMO.

The way this MITM approach is designed probably works fine, but it ignores how Maven repositories work conceptually. The dependencies you specify in a Gradle build script are coordinates into a single Maven "namespace", i.e. the universe of available Maven dependencies, and repositories are kind of like views into this namespace. So if a buildscript specifies mavenCentral, then gradlePluginPortal in its repositories block, for example, Gradle will try each repository in turn for each dependency coordinate. Recording the first URL that returns a 200 will probably work in most cases, but as I have discovered, it does not if there are connectivity issues with the first repository and the second repository does not carry the same dependency versions; or worse, in the case of gradlePluginPortal, it mirrors the first repository but rewrites the POM/maven-metadata.xml so you don't have a reliable hash for the metadata. Basically I worry about "works on my machine" issues if you don't try to reproduce what Gradle does when resolving/fetching artifacts.
Gradle supports more than Maven and HTTP; for example, you can host repositories on S3, and you can have repositories that require authentication. You can pretty easily extract this info from Gradle itself, but I don't think a generic MITM proxy can.
By resolving every project configuration, this locks/fetches more dependencies than are strictly required for build/check tasks. Ideally you'd run the normal "build" task and only lock what Gradle resolves during that build, so you aren't locking all dependencies for all configurations of all subprojects.

All in all, I think this is a decent approach that probably works fine for nixpkgs, but there are likely going to be issues adapting this to complex projects (e.g. the Kotlin compiler) or out-of-tree projects.

I understand that the fixed output derivation won't work if upstream didn't do its job properly, but I still don't like this trend of adding huge generated lock files to Nixpkgs. Anyway cie-middleware-linux looks good and appears to be reproducible.

I think it makes sense to store them somewhere nix related, but I think it should also be possible without somehow. Still better than abomination of having a new derivation for each dependency.

Whats the general state of this? Should we continue to do cursed things like ```
$ echo "com.squareup.okio/okio/1.13.0/a9283170b7305c8d92d25aff02a6ab7e45d06cbe/okio-1.13.0.jar" |
awk -F/ '{gsub(/[0-9a-f]{30,40}/, "", $0); gsub(/./, "/", $1); printf "install -Dm444 %s $out/%s/%s/%s/%s\n", $0, $1, $2, $3, $NF}'
install -Dm444 com/squareup/okio okio 1.13.0  okio-1.13.0.jar $out/com/squareup/okio/okio/1.13.0/okio-1.13.0.jar
till this is merged, or just wait a month or so until this is in master?

@CutestNekoAqua I'm currently in the process of applying the feedback above, it will take some time. Please continue doing it the old way, I don't know how much time it will take.

Could you spin the package migrations into a separate PR to make this one easier to review?

I've applied a lot of the suggestions. Most importantly, redirects are now forbidden, and xml files are not included in the lockfiles anymore, they are instead generated based on the available jar/pom files. This means there should be no security issues with lockfiles (i.e. no way to embed malicious .jar/.pom files in them), which is important as lockfiles are much less likely to be thoroughly reviewed by humans than the derivation code.
There's definitely more stuff that could be done, most importantly in regards to the update scripts' compatibility with different system configurations. However, I'm not super knowledgeable on how nix-shell works, so I'd appreciate some help here.

The update script doesn't work for me if tmpfs isn't mounted to /tmp:
$ ./result
/tmp/tmp.EuM1YZ1Blz ~/nixpkgs
# Keys stuff ...
Mode: STANDALONE_ONCE
Jail parameters: hostname:'NSJAIL', chroot:'/', process:'/nix/store/vw3v4mjqfgda7sfkvmgapa6gyba71kvq-gradle-commands.sh', bind:[::]:0, max_conns:0, max_conns_per_ip:0, time_limit:0, personality:0, daemonize:false, clone_newnet:false, clone_newuser:true, clone_newns:true, clone_newpid:true, clone_newipc:true, clone_newuts:true, clone_newcgroup:true, clone_newtime:false, keep_caps:true, disable_no_new_privs:false, max_cpus:0
Mount: '/' -> '/' flags:MS_RDONLY|MS_BIND|MS_REC|MS_PRIVATE type:'' options:'' dir:true
Mount: '/home' flags: type:'tmpfs' options:'size=4194304' dir:true
Mount: '/root' flags: type:'tmpfs' options:'size=4194304' dir:true
Mount: '/nix/store/srkic5rxmd2n3l9wkdk36n99f6fr0ha2-nixpkgs' -> '/nix/store/srkic5rxmd2n3l9wkdk36n99f6fr0ha2-nixpkgs' flags:MS_RDONLY|MS_BIND|MS_REC|MS_PRIVATE type:'' options:'' dir:true
Mount: '/proc' flags:MS_RDONLY type:'proc' options:'' dir:true
Uid map: inside_uid:1000 outside_uid:1000 count:1 newuidmap:false
Gid map: inside_gid:100 outside_gid:100 count:1 newgidmap:false
Executing '/nix/store/vw3v4mjqfgda7sfkvmgapa6gyba71kvq-gradle-commands.sh' for '[STANDALONE MODE]'
error: creating directory '/tmp/nix-shell-1-1': Read-only file system
pid=257070 ([STANDALONE MODE]) exited with status: 1, (PIDs left: 0)

$ sudo mount -t tmpfs tmpfs /tmp
$ ./result
/tmp/tmp.EuM1YZ1Blz ~/nixpkgs
# Keys stuff ...
Mode: STANDALONE_ONCE
Jail parameters: hostname:'NSJAIL', chroot:'/', process:'/nix/store/vw3v4mjqfgda7sfkvmgapa6gyba71kvq-gradle-commands.sh', bind:[::]:0, max_conns:0, max_conns_per_ip:0, time_limit:0, personality:0, daemonize:false, clone_newnet:false, clone_newuser:true, clone_newns:true, clone_newpid:true, clone_newipc:true, clone_newuts:true, clone_newcgroup:true, clone_newtime:false, keep_caps:true, disable_no_new_privs:false, max_cpus:0
Mount: '/' -> '/' flags:MS_RDONLY|MS_BIND|MS_REC|MS_PRIVATE type:'' options:'' dir:true
Mount: '/home' flags: type:'tmpfs' options:'size=4194304' dir:true
Mount: '/root' flags: type:'tmpfs' options:'size=4194304' dir:true
Mount: '/nix/store/srkic5rxmd2n3l9wkdk36n99f6fr0ha2-nixpkgs' -> '/nix/store/srkic5rxmd2n3l9wkdk36n99f6fr0ha2-nixpkgs' flags:MS_RDONLY|MS_BIND|MS_REC|MS_PRIVATE type:'' options:'' dir:true
Mount: '/proc' flags:MS_RDONLY type:'proc' options:'' dir:true
Uid map: inside_uid:1000 outside_uid:1000 count:1 newuidmap:false
Gid map: inside_gid:100 outside_gid:100 count:1 newgidmap:false
Executing '/nix/store/vw3v4mjqfgda7sfkvmgapa6gyba71kvq-gradle-commands.sh' for '[STANDALONE MODE]'

Welcome to Gradle 8.8!
# ...
BUILD SUCCESSFUL in 4m 48s
5 actionable tasks: 5 executed
pid=260183 ([STANDALONE MODE]) exited with status: 0, (PIDs left: 0)

Does this have something to do with /home and /root being mounted with type:'tmpfs' in nsjail, perhaps?

I'm not sure. On my machine, root is mounted as tmpfs, but /tmp is a bind mount.

I've applied most of the remaining suggestions.
It can be hard to follow the previous reviews, so here are the links to the discussions that are still open:
we should check that everything builds on all platforms, especially signald on darwin, which was previously unsupported
is /opt or /share better?
(kinda bikesheddy) is attrOverrides confusing because of its similarity with overrideAttrs? maybe this should be removed completely as it's not currently used and can always be added back without rebuilds
should the docs be clearer on what "env vars" in the context of stdenv.mkDerivation are?
it's fine that these are sha256 rather than sri hashes, right?
do we remove code that handles the "no pname" case?
what to do with builtins.storePath?
potential race condition in the update script on super high load machines? I'd assume 0.5s is enough practically anywhere
should the lockfile name be forced to deps.json, so there can only be one gradle derivation per directory? 1 2
if no progress is done on any of these questions, I think it's better to merge this as-is and iterate on this solution later

I'm not too familiar with Nix on Darwin, am I correct in thinking that __darwinAllowLocalNetworking is always required for building Gradle projects with this method?

Another update:

I don't see any major issues anymore, I think I've resolved basically every open review there was.
I've switched from nsjail to bwrap, it should probably be a little tighter and I don't see any issues with the existing derivations
There was some confusion about why the update script uses nix-shell. This is because Gradle can call other commands, including wget, gcc or pkg-config. This requires bringing the entire stdenv and the package's dependencies and not just setting PATH. Instead of the old logic where you had to manually add packages for the update script to use, I just use srcOnly as the Nix shell.
SRI (rather than sha256) hashes are possible, but it would require a bit of effort to update all 30 packages' lockfiles again. If I get a second opinion about how they are better, I can do the switch.

SRI (rather than sha256) hashes are possible, but it would require a bit of effort to update all 30 packages' lockfiles again. If I get a second opinion about how a switch would be worth it, I can do it.

Maybe best of both worlds is to support both styles, but write SRI and let the lock files improve as they're updated later? You could regenerate one lock file to show that it all still works.

Speaking of different versions of lockfiles, it'd probably be a good idea to version them so that format updates in the future are possible.

I still have trouble building fastddsgen.
Running nix build .#fastddsgen fails due to missing org.antlr:antlr4:4.13.0 (full log). I guess that's because this is a dependency from idl-parser git submodule, which somehow doesn't propagate to deps.json. @chayleaf  Can you give me some hints how to fix that?

I've added lockfile versions and fixed the remaining build issues (unless I somehow introduced new issues while rebasing).
@wentasah in case of fastddsgen I've added the following to the gradle.updateDeps call:
postBuild = ''
  cd thirdparty/idl-parser
  # fix "Task 'submodulesUpdate' not found"
  gradleFlags=
  gradle nixDownloadDeps
'';

@wentasah in case of fastddsgen I've added the following to the gradle.updateDeps call:

Works well. Thanks!

Update scripts can now be created like this:
passthru.updateScript = _experimental-update-script-combinators.sequence [
  (gitUpdater { inherit pname; rev-prefix = "v"; })
  finalAttrs.passthru.updateDeps
];

I'll try to test this on Darwin later today to see what's wrong with the Ghidra build, it's best to fix these issues in this PR so we can add more info to the docs if necessary

The results of my check are as follows:

Basically all the packages build fine for Darwin with the sandbox off. With sandbox, YMMV - however I've now enabled local networking in all Darwin-supported packages, so it should be fine
Ghidra didn't build because its Gradle code for MacOS puts some files at the directory pointed at by the user.home Java property (this is different from the HOME env var), and it's set to /var/empty by default. Fixed with gradleFlagsArray+=("-Duser.home=$NIX_BUILD_TOP/home") in preBuild.
Corretto didn't build because it didn't build before, I can theoretically get it to work but it doesn't seem that easy as the code makes quite a few assumptions about where the SDK and frameworks are located, and has an entire autoconf infrastructure that's dormant on Linux but is necessary on MacOS. Anyway, darwin isn't even in its platforms, so I'm not sure why Nix doesn't complain about me trying to build it. I did uncover an issue there - the derivation didn't run the configure phase, so the setup hook couldn't set the Gradle options, that's fixed now by running the Gradle configure hook in preBuild.

I don't really see any remaining issues with this PR. Everything builds on Linux and Darwin, the update scripts work, the only thing I'm not sure about is the last commit in this PR (which can be reverted).

Freenet is mad about reproducible archives, disabling them fixes the build. Also, doesn't your log says openjfx17 has been built successfully?
I'll look into a good way of going about this.

Result of nixpkgs-review pr 272380 run on aarch64-darwin 1

1 package marked as broken and skipped:

scenic-view

8 packages failed to build:

armitage
corretto11
corretto17
corretto21
ghidra
ghidra-extensions.gnudisassembler
ghidra-extensions.machinelearning
ghidra-extensions.sleighdevtools

39 packages built:

apkleaks
apkleaks.dist
apksigcopier
apksigcopier.dist
apksigner
diffoscope
diffoscope.dist
diffoscope.man
experienced-pixel-dungeon
fastddsgen
fdroidserver
fdroidserver.dist
gradle (gradle_8)
gradle-unwrapped
gradle_6
gradle_6-unwrapped
gradle_7
gradle_7-unwrapped
jadx
jd-gui
jextract
jextract-21
key
kotlin-language-server
ma1sd
mindustry-server
mitm-cache
mucommander
pdf-sign
pdfchain
pdftk
rat-king-adventure
rkpd2
shattered-pixel-dungeon
shorter-pixel-dungeon
signald
signaturepdf
stirling-pdf
summoning-pixel-dungeon

How many of those are regressions?

Ghidra was the only regression, I just fixed it (it has to be JAVA_TOOL_OPTIONS rather than gradleFlagsArray), armitage build fails because of a dependency, corretto doesn't even have darwin in meta.platforms so I'm not sure why it's being built

a couple more openjfx fixes:

sed turns out to not be enough for openjfx fixup, changing back to perl
fixed openjfx22 not picking up the config file (the derivation structure changed a little from 21 to 22 so I missed ln -s $config gradle.properties which was in postPatch in 21 but in buildPhase in 22)
the ICU zips are now included in deps.json so their hashes don't have to be maintained separately

Updated the docs
diff
diff --git a/doc/languages-frameworks/gradle.section.md b/doc/languages-frameworks/gradle.section.md
index 7578d04122ad..95abf7be40e2 100644
--- a/doc/languages-frameworks/gradle.section.md
+++ b/doc/languages-frameworks/gradle.section.md
@@ -136,8 +136,9 @@ The update script flow:
 - Set the `IN_GRADLE_UPDATE_DEPS` environment variable to `1`
 - Run the derivation's `unpackPhase`, `patchPhase`, `configurePhase`
 - Run the derivation's `gradleUpdateScript` (the Gradle setup hook sets
-  it, by default it also runs `preBuild`, `preGradleUpdate` and
-  `postGradleUpdate` hooks)
+  a default value for it, which runs `preBuild`, `preGradleUpdate`
+  hooks, fetches the dependencies using `gradleUpdateTask`, and finally
+  runs the `postGradleUpdate` hook)
 - Finally, store all of the fetched files' hashes in the lockfile. They
   may be `.jar`/`.pom` files from Maven repositories, or they may be
   files otherwise used for building the package.
@@ -172,6 +173,13 @@ The Gradle setup hook accepts the following environment variables:
   package. Defaults to `assemble`.
 - `gradleCheckTask` - the Gradle task to be used for checking the
   package if `doCheck` is set to `true`. Defaults to `test`.
+- `gradleUpdateTask` - the Gradle task to be used for fetching all of
+  the package's dependencies in `gradle.updateDeps`. Defaults to
+  `nixDownloadDeps`.
+- `gradleUpdateScript` - the code to run for fetching all of the
+  package's dependencies in `gradle.updateDeps`. Defaults to running the
+  `preBuild` and `preGradleUpdate` hooks, running the
+  `gradleUpdateTask`, and finally running the `postGradleUpdate` hook.
 - `gradleInitScript` - path to the `--init-script` to pass to Gradle. By
   default, a simple init script that enables reproducible archive
   creation is used.

rebased
diff
diff --git a/pkgs/development/compilers/openjdk/openjfx/21/default.nix b/pkgs/development/compilers/openjdk/openjfx/21/default.nix
index f615bcef1ab1..706d62080ab9 100644
--- a/pkgs/development/compilers/openjdk/openjfx/21/default.nix
+++ b/pkgs/development/compilers/openjdk/openjfx/21/default.nix
@@ -8,7 +8,7 @@
 let
   pname = "openjfx-modular-sdk";
   major = "21";
-  update = "";
+  update = ".0.3";
   build = "-ga";
   repover = "${major}${update}${build}";
   jdk = openjdk21_headless;
@@ -20,9 +20,9 @@ in stdenv.mkDerivation {

   src = fetchFromGitHub {
     owner = "openjdk";
-    repo = "jfx";
+    repo = "jfx21u";
     rev = repover;
-    hash = "sha256-deNAGfnA6gwcAa64l0AWdkX+vJd3ZOfIgAifSl+/m+s=";
+    hash = "sha256-7z0GIbkQwG9mXY9dssaicqaKpMo3FkNEpyAvkswoQQ4=";
   };

   buildInputs = [ gtk2 gtk3 libXtst libXxf86vm glib alsa-lib ffmpeg_4 ];

I agree, some tests are necessary. I'll also write documentation regarding some design decisions in making the setup hook as well as Gradle's peculiarities.

Honestly, that sounds like goals for a v2. This should be good to go on its own; all the packages you converted are enough of a test for a v1.

I'm fine with getting this merged and submitting a followup PR with more docs and better compress-deps-json.py and fetch-deps.nix (it works, but now that I looked into Gradle docs I realized it doesn't yet handle all the cases)

Honestly, that sounds like goals for a v2.

I don't see how the version is relevant.
This adds a large chunk of functionality and it's unclear how other contributors can change it.
If the instructions are "We're sorry, but you'll have to find call sites yourself and build a few. We'd like to build a representative test suite; contributions welcome.", then that's better than nothing, because it acknowledges the problem and invites contributions.

all the packages you converted are enough of a test for a v1.

If you could gather a few builds in a tests.nix file or something, that already goes a long way.
I'll leave it up to @chayleaf to decide what the other contributors should do to make sure that they don't regress existing functionality.

I may have been unclear here. What I meant is that adding a test suite would be something for a second iteration; after this PR has landed.
While contributor's docs are absolutely welcome, I don't see them as strictly necessary. Anyone who would like to touch this will have to dig into the source in any case, docs would "merely" make that process easier.

contributor documentation is ready, right now I'm rechecking all the packages to check whether parallel building works for them (so far only openjfx failed to build), tests will come in a follow-up PR

I added contributor-oriented docs and slightly extended the lockfile format

ma1sd got removed from nixpkgs, dropped the commit

I just thought of an idea - there's currently lots of duplication, with mitmCache and passthru.updateDeps both being necessary. What if there was a mitmCache.update instead?

Not necessarily as part of this PR, but I wonder if there is a plan to have a buildGradlePackage wrapper at some point?

I don't see a point in that, look at e.g. the key package diff, it destroys all the annoying Gradle-specific overhead that was previously there

Good point; not adding a wrapper exposes the hacky behaviour. Nevermind.

Result of nixpkgs-review pr 272380 run on x86_64-linux 1

8 packages marked as broken and skipped:

corretto19
corretto19.debug
javaPackages.openjfx15
javaPackages.openjfx19
javaPackages.openjfx20
openjfx15
openjfx19
openjfx20

81 packages built:

apkleaks
apkleaks.dist
apksigcopier
apksigcopier.dist
apksigner
armitage
autopsy
bisq-desktop
bluej
cie-middleware-linux
corretto11
corretto11.debug
corretto17
corretto17.debug
corretto21
corretto21.debug
cryptomator
diffoscope
diffoscope.dist
diffoscope.man
experienced-pixel-dungeon
fastddsgen
fdroidserver
fdroidserver.dist
freenet
freeplane
ganttproject-bin
ghidra
ghidra-extensions.gnudisassembler
ghidra-extensions.machinelearning
ghidra-extensions.sleighdevtools
gradle (gradle_8)
gradle-unwrapped (gradle_8-unwrapped)
gradle_6
gradle_6-unwrapped
gradle_7
gradle_7-unwrapped
greenfoot
gscan2pdf
gscan2pdf.man
jabref
jadx
openjfx11 (javaPackages.openjfx11)
openjfx (javaPackages.openjfx17 ,openjfx17)
openjfx21 (javaPackages.openjfx21)
openjfx22 (javaPackages.openjfx22)
jd-gui
jextract
jextract-21
key
kotlin-language-server
maptool
mcaselector
microsoft-identity-broker
mindustry
mindustry-server
mindustry-wayland
mitm-cache
moneydance
mucommander
ns-usbloader
olvid
pattypan
pdf-sign
pdfchain
pdfsam-basic
pdftk
pidginPackages.purple-signald
quark-goldleaf
rat-king-adventure
rkpd2
scenebuilder
scenic-view
shattered-pixel-dungeon
shorter-pixel-dungeon
signald
signaturepdf
sparrow
sparrow-unwrapped
stirling-pdf
summoning-pixel-dungeon

Thanks a bunch for your work @chayleaf :)

This PR broke evaluation of nixpkgs for Nix 2.3.18.
syntax error, unexpected '.', at /nix/store/mbq17diwa38x96apmjp8maap2qf1yk01-source/pkgs/games/shattered-pixel-dungeon/generic.nix:56:54

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/nix-buildproxy-reproducible-http-https-responder-in-sandboxed-nix-builds/40081/12


fixes bug #17

What is the current behavior?
see bug description. It was not possible to disable the exporting button on ios given the new ios runtime
What is the new behavior?
exporting button is now correctly disabled. Required changing one line of code in plugin in order for boolean values to get correctly mapped to native NSNumber boolean valued objects.
Furthermore demo app now requires rc version of runtime which adds the required underlaying bugfix.
Fixes/Implements/Closes #[Issue Number].
fixes bug #17
BREAKING CHANGES:
no breaking changes
Migration steps:
no migration required

Thanks! @cjohn001 :)

@cjohn001
published as<EMAIL_ADDRESS>


Missing events even with flush

I am sending 13 events with logEvent at about the same time. I am not awaiting for the promise to resolve.
Later on, before I terminate, I am calling flush to make sure all events will be sent.
Flush responds with status: skipped
Expected Behavior

All events to be logged
Current Behavior

Some events are missing, sometimes 1, sometimes more
Possible Solution

I guess the request is still ongoing for some events?
But flush doesn't know about it?
Steps to Reproduce

Environment

SDK Version: 1.6.1
Node version: 12.x

@florian-milky the skipped response should only be returned from flush when either there are no events remaining in the buffer, or the events are incorrectly formatted. Where is it that you notice events are missing? Are they not appearing on the amplitude website?

@florian-milky are there any console warnings in your browser window when you try this?


Add options to minify document (query) when using documentMode String.

Is your feature request related to a problem? Please describe.
graphql has a utility function stripIgnoredCharacters that will effectively minify any string document. This could help with code size when strings representing queries are embedded in the source.
Describe the solution you'd like
I would like a config option that will run stripIgnoredCharacters right after print on documents.
Describe alternatives you've considered
None
Additional context
Thanks for this awesome suite of tools!

This would also be useful for the @graphql-codegen/schema-ast plugin. Some runtime environments such as Cloudflare Workers are limited to a code size of 1mb so every byte counts.

We are running into issues with this also. For now, since we are using Apollo client, I was able to use this method.
https://www.apollographql.com/docs/react/networking/advanced-http-networking/#overriding-the-default-print-function
It would be nicer/faster though if we could have the generated already have the stripped version.


Deprecate Differentiable.AllDifferentiableVariables.

Remove usages of AllDifferentiableVariables and
var allDifferentiableVariables.
Friend PR: https://github.com/apple/swift/pull/26527

All references should be removed.
$ grep -nr "AllDifferentiableVariables" Sources Tests
# No results.

@dan-zheng there's a mention of `AllDifferentiableVariables" in the readme.

@dan-zheng there's a couple of `AllDifferentiableVariables" in the readme.

I saw that, thanks! I'll fix that in a follow-up PR, since the https://github.com/apple/swift/pull/26527 is sensitive to the commit versions of tensorflow/swift-apis.

@dan-zheng is the loss function refactoring necessary in this PR?

@dan-zheng is the loss function refactoring necessary in this PR?

I'm not sure what exactly you're referring to, but it sounds orthogonal. This patch is meant to be minimal, removing references to AllDifferentiableVariables/allDifferentiableVariables.

The friend PR https://github.com/apple/swift/pull/26527 has been merged.
apple/swift now points to a commit of tensorflow/swift-apis from this PR.
Shall we hold off on merging this PR until the CI toolchain has https://github.com/apple/swift/pull/26527? cc @rxwei

The CI toolchain is already broken unfortunately due to build failures (marcrasi@ is looking into this AFAIK). I think we should merge this.


Pic #1 is tiny

Hi,
The first picture in this article is so tiny that it is next to impossible to see what it is trying to illustrate.
Please zoom in or make the picture clickable to get a larger version of the picture.
Thanks

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 1f09b043-ca4f-8d06-16b9-19f03e89b344
Version Independent ID: 8166553d-853a-204d-b0bf-e7380d4042d6
Content: Manage capacities within Power BI Premium and Power BI Embedded - Power BI
Content Source: powerbi-docs/service-admin-premium-manage.md
Service: powerbi
GitHub Login: @Minewiskan
Microsoft Alias: owend

Hi, @jonaselreweny -- thanks for asking! I've assigned your question to the author.

@jonaselreweny - Agreed. Guidelines require this first image include the whole browser, which make it tough to get a large enough screen grab in an article like this, but I'll see what I can do.

@jonaselreweny - Updated this image. This is about the best we can do here. Thanks for letting us know.
#please-close


Claims returned by jsonrpc responses should include the height of the original claim

Resolving a name returns the height of the latest update to the winning claim for that name, not the height of the original claim. But takeover calculations are based on the height of the original claim. That information needs to be made available by the daemon.

Alternatively, lbrynet should return takeover height information for claims.

For your 2nd comment, would https://github.com/lbryio/lbry/issues/1031 be enough?

@tzarebczan please confirm that claim take over use original claim height

@lyoshenka I did confirm that claim updates don't reset the takeover period.
However, I'm still skeptical why we need the root height. Isn't it invalid once the claim is taken over? Any takeovers at the vanity URL are then based on the new vanity takeover height. If that's abandoned and it goes back to the original claim, the root_height would no longer be used in the calculation, it would be the takeover_height (when it went back into effect).
We should be able to get all this info from the blockchain layer.

The goal was to be able to show (e.g. in the app) how long it will be before a claim you just made will become the winning claim. So if there's a longstanding claim with multiple updates to it, there's no way to tell how long the takeover process will take.

@lyoshenka Can this issue be closed or defined into an actionable issue?

The real issue is what I said above:

The goal was to be able to show (e.g. in the app) how long it will be before a claim you just made will become the winning claim. So if there's a longstanding claim with multiple updates to it, there's no way to tell how long the takeover process will take.

We can either turn this into that, or open it as a new issue. Up to you.

This is now shown in the meta section on resolve:
meta": {
        "activation_height": 307232,
        "claims_in_channel": 1109,
        "creation_height": 303200,


Polygon draw interaction buggy on mobile devices

There seems to be a problem with ol.interaction.Draw when using it on mobile devices (primary on iOS).
The problem occurs when drawing a polygon and (as far as we've noticed) slightly move your finger when touching a point - the point will be drawn but when you add another point the previous one simply disappears!
We use Openlayers with default interactions (dragging, zooming etc) and dynamically add the draw interaction. I thought the problem would occur only in combination of these but it was also reproducable after removing any other interactions. And the problem also occurs on the demo page: http://openlayers.org/en/v3.5.0/examples/draw-features.html
It seems like there is some problem in the "event-detection" - meaning whether it's a simple touch event or drag on the user side. The point gets drawn on the map, but is never added to the polygon coordinates(?), and therefore disappears on the next click.
I could reproduce this problem on Android as well, but it's "harder" to do and unlikely because I have to actively move my finger on touching down (I think this is because Android and iOS handle touch/drag detection differently). The problem never appeared on desktop devices.
I've also uploaded a video where you can see the problem in action:
https://dl.dropboxusercontent.com/u/23708747/ShareX/2015-07/VIDEO0122.mp4
I hope this is enough information. I couldn't find anything likely on the web. And I would dig through the code myself but I'm not exactly a Javascript pro :)

In the meantime I noticed that a new param "clickTolerance" was added in the last release (didn't find that a few hours ago..) - this might help us to mitigate the bug, but the main problem still persists: why do the points still remain if the touch has not been recognized as click? (as seen in the video)

Is anyone here who can reproduce or even track down the problem?

I've noticed a similar issue happening on desktop (I'm using firefox). With the example referenced, it is easily reproduce-able. http://openlayers.org/en/v3.10.1/examples/measure.html?q=measure

draw a large area rectangle about as long as the viewport, but don't double click or complete the rectangle
drag the viewport in the horizontal direction so that your start point for the rectangle is out of the viewport
the start point disappears and the rectangle turns into a triangle.

@scroach This is caused by the draw interaction trying to visualize a 'sketched' variant of the polygon being drawn. When for example the map is panned, the touch action triggering the move is visualized like the a mousemove action on the desktop.
I'll try to add a pull request for a possible fix, but I'm sure this is the correct way to go...

Closing due to lack of activity.


New Driver - Kogan Smart Heater

while this shows up in HA now, i have no control over its state
Device normal info

Kogan

Kogan Panel Heater - KAWFHTP15BA / KAHTP15WALA
{
"active_time":<PHONE_NUMBER>,
"biz_type": 0,
"category": "qn",
"create_time":<PHONE_NUMBER>,
"icon": "smart/icon/1470624702_0.jpeg",
"id": “xxxx ",
"ip": "",
"lat": "",
"local_key": "",
"lon": "",
"name": "Bedroom heater",
"online": true,
"owner_id": "23484236",
"product_id": "ynjanlglr4qa6dxf",
"product_name": "Panel heater",
"status": [
{
"code": "switch",
"value": false
},
{
"code": "temp_set",
"value": 25
},
{
"code": "temp_current",
"value": 19
}
],
"sub": false,
"time_zone": "+10:00",
"uid": "***",
"update_time":<PHONE_NUMBER>,
"uuid": "0014083010521c49f20a"
},
"sub": false,
"time_zone": "+10:00",
"uid": "***",
"update_time":<PHONE_NUMBER>,
"uuid": "0014083010521c49f20a"

},

Device specifications
2021-09-18 08:26:33 DEBUG (SyncWorker_6) [tuya iot] Request: method = GET, url = https://openapi.tuyaeu.com/v1.0/devices/xxxxx/specifications, params = None, body = None, t =<PHONE_NUMBER>164
2021-09-18 08:26:33 DEBUG (Thread-25) [tuya iot] _on_log: Received PUBLISH (d0, q0, r0, m0), 'cloud/token/in/ebc1d658bf210987387c9f4d12782ce2', ... (284 bytes)
2021-09-18 08:26:33 DEBUG (Thread-25) [tuya iot] payload-> b'{"data":"JFYot1AkACi8/Ur2AvTCi3qltBYwgPLSJE1YUp2CUYLVDfkFVEIu0HrN+DDEC/uA30I2NnsvKp0PqavUK5p6+2QiO6QlCEtkkzxqFzKYNWxMrTh7bOcdtlB60wzimmmtVhPwPLhtAwgQaCu5WeOT9m6lumAfCbOsThm8glBZvTgjlDYEccpKuYPzJEOBdMLP","protocol":4,"pv":"2.0","sign":"60e1f36e5c5c57e0246d96ace8e56e37","t":1631917592}'
2021-09-18 08:26:33 DEBUG (Thread-25) [tuya iot] on_message: {'data': {'dataId': 'c8013806-594e-4521-ba79-685fc9d3270a', 'devId': '7312002040f5201c02cc', 'productKey': 'GXoLAzy1gQUbckKe', 'status': [{'5': 53}]}, 'protocol': 4, 'pv': '2.0', 'sign': '60e1f36e5c5c57e0246d96ace8e56e37', 't':<PHONE_NUMBER>}
2021-09-18 08:26:33 DEBUG (Thread-25) [tuya iot] mq receive-> {'data': {'dataId': 'c8013806-594e-4521-ba79-685fc9d3270a', 'devId': '7312002040f5201c02cc', 'productKey': 'GXoLAzy1gQUbckKe', 'status': [{'5': 53}]}, 'protocol': 4, 'pv': '2.0', 'sign': '60e1f36e5c5c57e0246d96ace8e56e37', 't':<PHONE_NUMBER>}
2021-09-18 08:26:33 DEBUG (Thread-25) [tuya iot] mq _on_device_report-> [{'5': 53}]
2021-09-18 08:26:33 DEBUG (SyncWorker_6) [tuya iot] Response: {
"result": {
"category": "qn",
"functions": [
{
"code": "switch",
"type": "Boolean",
"values": "{}"
},
{
"code": "temp_set",
"type": "Integer",
"values": "{"unit":"℃","min":5,"max":40,"scale":0,"step":1}"
}
],
"status": [
{
"code": "switch",
"type": "Boolean",
"values": "{}"
},
{
"code": "temp_set",
"type": "Integer",
"values": "{"unit":"℃","min":5,"max":40,"scale":0,"step":1}"
},
{
"code": "temp_current",
"type": "Integer",
"values": "{"unit":"℃","min":0,"max":100,"scale":0,"step":1}"
}
]
},
"success": true,
"t":<PHONE_NUMBER>34

As per the other thread, remove the id from the samples you gave above.  This one looks like it should already be supported and allow you to turn it on/off and set the temperature, as well as report if its on/off, what the temperature is and what it is set to be.
According to this page
https://github.com/tuya/tuya-home-assistant/wiki/Supported-Device-Category?_source=f5f782752be3c4a9157ec47514d6091b support for that category of device was added in version 1.1 of the integration.

Thanks mate
As per the screenshot below. While it does pick up that it’s there, there is no options under operation to turn it on/off in Lovelace
[cid:C3ADA90B-81FC-4D44-AD42-EF50831A68EB-L0-001]
Sent from my iPhone
On 18 Sep 2021, at 1:02 pm, some-guy-in-oz @.***> wrote:

As per the other thread, remove the id from the samples you gave above. This one looks like it should already be supported and allow you to turn it on/off and set the temperature, as well as report if its on/off, what the temperature is and what it is set to be.
According to this page
https://github.com/tuya/tuya-home-assistant/wiki/Supported-Device-Category?_source=f5f782752be3c4a9157ec47514d6091b support for that category of device was added in version 1.1 of the integration.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com/tuya/tuya-home-assistant/issues/476#issuecomment-922168027, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ASPA4O3QJZO43UHH3PGMGETUCP6L5ANCNFSM5EIOLJFA.
Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

I dont see that screenshot... Have you customised your lovelace interface at all?  I usually have a "default" one that I let HA manager and it puts everything on it... The heater should be under a "Switch" section if you haven't assigned it to a room like this example that shows a holman WX8 irrigation system

Firstly thank you for your help with this
the image must of not come through via the email reply
no custom lovelace and just in my default test enviroment and nothing comes up under operation

What shows under configuration > entities?  There should be a switch for it. If you click the up/down arrows does the value next to the "Off" up the top change?

@some-guy-in-oz  the value changes no issue
under config > entitites all i get is a climate entity and not a switch entity

Does anything show when you try to pick a device via Developer Tools > Services for "Climate: Turn off"?

Yes but calling the service does nothing

Using the below I am able to turn it on
![AE5A01EC-1969-4789-BB43-E59C68350D86](https://user-images.githubusercontent.com/77467195/133892700-ae6ea992-a09b-42b7-a173-a3d7b0bbefb9.png)

So I have the following services

Calling these services does nothing
interestingly enough haven’t ng it set as below will turn it on

Hmm it should be the turn on/turn off commands - when you choose them are you able to select any devices or entities?
Is the heater already running in some sort of "auto" mode?  If so then just adjusting the temperature should turn it on/off.

So the on/off commands do nothing.
setting to auto turns it on but then none of the commands will turn it off (putting the temp down just does turns the temp down but not off)
from the looks of it several others are having the same bug as well with different branded panel heaters
From: some-guy-in-oz @.>
Sent: Sunday, 19 September 2021 9:11 AM
To: tuya/tuya-home-assistant @.>
Cc: rodgrech @.>; Author @.>
Subject: Re: [tuya/tuya-home-assistant] New Driver - Kogan Smart Heater (#476)
Hmm it should be the turn on/turn off commands - when you choose them are you able to select any devices or entities?
Is the heater already running in some sort of "auto" mode? If so then just adjusting the temperature should turn it on/off.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com/tuya/tuya-home-assistant/issues/476#issuecomment-922386753, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ASPA4OZW4OLTRXBBAVXZF3DUCUMBDANCNFSM5EIOLJFA.
Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

OK looking at the code I see it's expecting to find some "modes" that the heater supports and the API isn't sending any modes.  I imagine that if there are no modes supported but the "swsitch" is supported it should just offer an "off" and "auto" option.  I can patch the file for you test, but how are you running Home Assistant and are you comfortable replacing files to test it out?

Yeah mate, comfortable in testing this for you
im running home assistant supervised in a vm
From: some-guy-in-oz @.>
Sent: Sunday, 19 September 2021 4:29 PM
To: tuya/tuya-home-assistant @.>
Cc: rodgrech @.>; Author @.>
Subject: Re: [tuya/tuya-home-assistant] New Driver - Kogan Smart Heater (#476)
OK looking at the code I see it's expecting to find some "modes" that the heater supports and the API isn't sending any modes. I imagine that if there are no modes supported but the "swsitch" is supported it should just offer an "off" and "auto" option. I can patch the file for you test, but how are you running Home Assistant and are you comfortable replacing files to test it out?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com/tuya/tuya-home-assistant/issues/476#issuecomment-922422766, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ASPA4O3VOVI3G3IDGK5RTWTUCV7LFANCNFSM5EIOLJFA.
Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

OK.. unzip and replace this file in your install. If you installed tuya v2 via hacs it'll be in the hacs subfolder somewhere, otherwise it'll be in the home assistant components subfolder.
climate.zip
It'll just give you two options, "auto" and "off" with appropriate icons. In my test it looks like a "calendar" icon for the auto and a "power" icon for off.  It would be nice to just have a single button for on/off but it appears that's not how home assistants default climate widget works so if you just want a single button you'll probably need to code up the yaml yourself.
I used my watering irrigator to test it out, and this is what it looks like (obviously it doesn't provide a temperature)

You might also like to try using the "HEAT" icon instead... If the above works, edit lines 364 & 372 replacing HVAC_MODE_AUTO with HVAC_MODE_HEAT

Hey! That fixed the issue
changed the lines of code to HVAC_MODE_HEAT and that is much better :D
From: some-guy-in-oz @.>
Sent: Sunday, 19 September 2021 5:12 PM
To: tuya/tuya-home-assistant @.>
Cc: rodgrech @.>; Author @.>
Subject: Re: [tuya/tuya-home-assistant] New Driver - Kogan Smart Heater (#476)
You might also like to try using the "HEAT" icon instead... If the above works, edit lines 364 & 372 replacing HVAC_MODE_AUTO with HVAC_MODE_HEAT
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com/tuya/tuya-home-assistant/issues/476#issuecomment-922426986, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ASPA4O3YSSPUQWPYFBRE5PDUCWEMFANCNFSM5EIOLJFA.
Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

climate.zip
modfied file with heat button instead of auto.
ive rebooted HA a few times to make sure theres no oddities and it seems solid @some-guy-in-oz

OK.  The tuya devs won't accept it hard coded to heat as the climate entity also controls humidity and a/cs.  I'm just trying out some changes that sets it to heat only if its a heater.  Once I've finished testing I'll post here for you to try it out and if that works I'll submit the patch to the tuya devs so it can make its way to mainstream.

you're a legend mate.
shoot it through and ill test for ya

Can you try out the enclosed file, make sure it allows turning on and off and temperature adjustment.
climate.zip

This will probably fix #78, #149, #197, #208, #249 & #357

so close

so when you turn the heater on via HA, the device turns on, tuya app reports its on but home assistant state doesnt change and it still thinks its off

Did it show on using the first file I posted or has it never shown as on (the icon should change colour to show when it's on too).  What happens if you turn the device on and then restart home assistant?  Does it show on then?

Well not sure what is going on. Maybe an issue in tuya s end but I am able to use it correctly this evening using the 2nd lot of your files
Sent from my iPhone
On 20 Sep 2021, at 10:36 am, some-guy-in-oz @.***> wrote:

Did it show on using the first file I posted or has it never shown as on (the icon should change colour to show when it's on too). What happens if you turn the device on and then restart home assistant? Does it show on then?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com/tuya/tuya-home-assistant/issues/476#issuecomment-922567093, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ASPA4O6DZTYVOBZ44SCRYGDUCZ6YXANCNFSM5EIOLJFA.
Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

Do the "target temperature" reading change if you adjust them via the tuya app or on the device itself?

Yes it does
Sent from my iPhone
On 20 Sep 2021, at 7:50 pm, some-guy-in-oz @.***> wrote:

Do the "target temperature" reading change if you adjust them via the tuya app or on the device itself?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com/tuya/tuya-home-assistant/issues/476#issuecomment-922777329, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ASPA4O7TEOS7L6PMPPS4USLUC37VHANCNFSM5EIOLJFA.
Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

They change in home assistant too?  Does the first set of files I uploaded work with switching it on/off?

@some-guy-in-oz i have not had time to digest this thread fully but seems you may be able to assist us with a similar issue in #28

reopening issue as this nolonger works under the new version of Tuya V2


How to run the test suite?

Hi,
I load the solution in Visual Studio and, using Resharper, instruct it to run all tests.
The result is this:

No tests ever pass, I guess because they never connect to the local server? I'm pretty new to VS and C# though so I might be missing something obvious.. is there a guide to getting the test suite going?

You need to start the FluentAutomation.TestApplication site before you can run the tests.

Oh. Well that makes sense...
Sorry if this is a daft question, but how does one do that? :)


Horizontal Scroll Bar

Unnecessary horizontal scroll bar is present when viewing site.

Looks like multiple people with bootstrap has this problem.
Will attempt to fix. Will try this...
body{
overflow-x: hidden;
}
@alekskivuls @mbotsko7 @anthonyprograms

Horizontal bar issue fixed. Changes below.
f963d3fd97b8a87fc9b7269c8f01687e5675d6a6


Document required Python version

Ran into installation issues with Python 3.8, only to find out by looking at the pyproject.toml file that the project requires 3.9+, should help prevent some minor headaches :)

Hi @Paradoxis,
thank you for your suggestion on changing this particular part of the documentation. We are currently reworking some of the documentation and have chosen to consider the main documentation on https://docs.dissect.tools as the main source for the requirements as opposed to changing the requirements in the README.md in each of the 20+ repos.
As such, the following PR has been made to add a section on Python requirements: https://github.com/fox-it/dissect.esedb/pull/21
We will therefor close this PR and hope your issue with the documentation has been sufficiently addressed in the PR above.


Suggestion: Round "Duration" to nearest second

This would save a bit of vertical space, and the level of precision doesn't seem needed since it's not a realtime clock. (Also, maybe always format seconds as two digits?) Thank you!

thanks, it will be in the next release

done in v0.5


Replace modules_tmpdir fixture with pytest's built-in tmpdir fixture

I'm wondering if the modules_tmpdir fixture, could be removed/replaced in the tests with pytest's built-in tmpdir fixture?
Not 100% confident this is possible/the right move, but it's worth investigating.

its based uppon it and adds extra behaviour

https://github.com/pallets/flask/blob/master/tests/conftest.py#L68-L91 are pretty much layering behavior on top of it and its a good idea to give it a different name

Agreed, I just missed that it was already using it.


[bazel] Update the rules to build RAM artifacts

Update the rules to build RAM artifacts and dispatch JTAG tests.

#19719


naming of factory method

I'd suggest to not use the class naming convention in order to indicate the return value type of the function GetArchive. I think the usual convention would be something like get_archive or create_archive.

Yes, that was my mistake, please change it to get_archive.

done.


Replace git with https for install and setup scripts

openaps-install.sh and oref0-setup.sh installation bug due to git:// instead of https://
Line 74 of openaps-install.sh calls the git clone via git:// instead of https:// for some reason my PC doesn't like it an always fails.
Suggested change, replacing with git clone https://github.com/openaps/oref0.git
https://github.com/openaps/oref0/blob/ee154b044948c76f064d59db93cc3f7b7691a679/bin/openaps-install.sh#L74
After this line 74 call, i get the following:
Cloning into 'oref0'... fatal: unable to connect to github.com:
Work around is simply to use cd src && git clone git://github.com/openaps/oref0.git
Checking the code i found the following git call too, so requrest both changed at the same time.
https://github.com/openaps/oref0/blob/ee154b044948c76f064d59db93cc3f7b7691a679/bin/oref0-setup.sh#L736
Thanks.

Have you tested this change? What results did you get?

just found an issue with my install of Dev it's missing EdisonVoltage and i think related my git issue vs https, so have installed it manually running the git clone using https://
https://github.com/openaps/oref0/blob/acb0150021e7bf284b6741a9510076a13f690747/bin/oref0-setup.sh#L1017

@scottleibrand any issue for you if i make a PR to change all git://github.com to be https://github.com in the set up scripts?

Sure: I can’t think of any reason that would break anything.

thanks, will get on it.


add Korean translation for homepage

Thanks @mrchypark 😀

anytime @MadLittleMods 👍  Please let me know when you guys home page open. Some texts might be awkward.


Help

Hello Eng. Luca,
First of all, thank you for publishing such a valuable work. I would like to ask you some questions and I will appreciate your response

How long does this code needs to be trained?
What is the hardware that you used for training?
How can I use your code to train from scratch?
is your master's thesis available online?

Hi,

The full training tooks around ten days, and it also depends on your hardware.
It's described in the results section of the Readme, but basically was a: 18c CPU, 24GB GPU, and 128GB RAM.
Use main.py file, tune the batch size, number of epochs, timesteps, and network parameters as you wish.
I don't think so, but the work is described in the T-ITS paper.


Safari and input element for autocomplete

Bug, feature request, or proposal:
Bug
What is the expected behavior?
input should use background color
What is the current behavior?
default input color is shown
What are the steps to reproduce?
http://plnkr.co/edit/pjWowX60V0nRqoKlGoS9
Will produce this:

Here another: http://embed.plnkr.co/oQANuo/, same result
What is the use-case or motivation for changing an existing behavior?
Which versions of Angular, Material, OS, TypeScript, browsers are affected?
see #5004
Is there anything else we should know?

Specifically, it is the type="search" attribute on the input causing that. If you want a workaround, try this
input[type="search"] {
  -webkit-appearance: none;
}

@willshowell you're right. It is that. Is there a specific reason why type should be set?

seems to work without extra CSS now: http://plnkr.co/edit/96o6s1XuSVWwWPOk5U6t?p=info


Outdated sample code for Assistant

Operating System
Windows
Version Information
python version: 3.10
openai package version: 1.36.0
Steps to reproduce
I tried to import the requirement libraries
from openai import AzureOpenAI
from openai.types.beta.threads.message_content_image_file import MessageContentImageFile
from openai.types.beta.threads.message_content_text import MessageContentText
from openai.types.beta.threads.messages import MessageFile
from PIL import Image

Expected behavior
No error while excuting
Actual behavior
No module found
Addition information
I try to downgrade the openai library version to 1.11.0, it works, but plz fix it in requirement.txt

Relevant file location:
published documentation: https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/assistants#see-also
Sample .ipynb: https://github.com/hwasu/Assistants_Lab/tree/main/api-in-a-box/math_tutor
Requiement.txt: https://github.com/hwasu/Assistants_Lab/blob/main/api-in-a-box/requirements.txt

@RottenTangerine
With latest openai sdk, following fix can manually be applied while repo is updated by owners.
from openai.types.beta.threads.message import Message
from openai.types.beta.threads.text_content_block import TextContentBlock
from openai.types.beta.threads.image_file_content_block import ImageFileContentBlock

Then, update format_message() to refer the new classes.
def format_messages(messages: Iterable[Message]) -> None:
    message_list = []

    # Get all the messages till the last user message
    for message in messages:
        message_list.append(message)
        if message.role == "user":
            break

    # Reverse the messages to show the last user message first
    message_list.reverse()

    # Print the user or Assistant messages or images
    for message in message_list:
        for item in message.content:
            # Determine the content type
            if isinstance(item, TextContentBlock):
                print(f"{message.role}:\n{item.text.value}\n")
            elif isinstance(item, ImageFileContentBlock):
                # Retrieve image from file id
                response_content = client.files.content(item.image_file.file_id)
                data_in_bytes = response_content.read()
                # Convert bytes to image
                readable_buffer = io.BytesIO(data_in_bytes)
                image = Image.open(readable_buffer)
                # Resize image to fit in terminal
                width, height = image.size
                image = image.resize((width // 2, height // 2), Image.LANCZOS)
                # Display image
                image.show()

With this much change, it worked in my end.


[docs] Add additional context to Autocomplete asynchronous documentation

Summary
Closes #20068.
Basically implements the suggested documentation updates detailed in this comment: https://github.com/mui-org/material-ui/issues/20068#issuecomment-600915331

[x] I have followed (at least) the PR section of the contributing guide.

@jaebradley It's a great first pull request on Material-UI 👌🏻. Thank you for working on it!

@jaebradley thanks for your work this line saved my life.
filterOptions={(x) => x}
I don't know what exactly does, and I don't see that changes on official docs, but thanks a lot.

Weirdly enough there is still no mention of the filterOptions prop in the officisl docs. I knew that the filter problem arises from the internal working of Autocomplete but i had no idea i could bypass it with ths prop.

@sefrem It's mentioned in https://next.material-ui.com/api/autocomplete/#main-content


Actor generator that generates actors for models

Goal is to easily generate actor functionality for set of defined models. See specs.
Requirements:

[ ] Generate CRUD Actions support and listing root objects
[ ] Generate Global actors model with actor per circle
[ ] 3Script <-> Model symmetry and exporting
[ ] Generate actions parsing and handling code per actor

future:

stuff like parsing actions can occur in separate threads
sub object crud actions support

To be closed with working example demonstrating these requirements

in phase 2 we need openrpc compatibility over websockets/rest
also generation of openrpc spec

bcc025f creates the models as specified in the manual with the necessary attributes to generate actions executors for

lets first finish the openrpc generator https://github.com/freeflowuniverse/crystallib/issues/238
so we can potentially use this here as well

use the new baobab db implementation which is now being tested


Clarification on epoch based 'at least once' pipeline for distributed, workers > 1, batched scenario

I've been working with 0.2.x recently updating a project that was using 0.1 and adding support to another, I have a rough template for a custom pipeline but there are some details I am bit concerned about. I feel a pipeline similar to this would be valuable to add to documentation at some point (as an alternative to the resampling distributed example).
This is an epoch based pipeline, with the goal of 'each sample seen at least once per-epoch'. The eval path in this example is single-node where the train is multi. The setup of the pipeline is below, the big questions are regarding the use of the pipeline using with_epochs to acheive 'at least once'
    pipeline = [wds.SimpleShardList(input_shards)]
    # at this point we have an iterator over all the shards
    if is_train:
        pipeline.extend([
            wds.detshuffle(
                  bufsize=_SHARD_SHUFFLE_SIZE,
                  initial=_SHARD_SHUFFLE_INITIAL,
                  seed=seed),
            wds.split_by_node,
            wds.split_by_worker,
            # at this point, we have an iterator over the shards assigned to each worker at each node
            wds.tarfile_to_samples(handler=log_and_continue),
            wds.shuffle(
                bufsize=_SAMPLE_SHUFFLE_SIZE,
                initial=_SAMPLE_SHUFFLE_INITIAL,
                rng=random.Random(seed)),
            #wds.repeatedly,  # repeatedly here to avoid dropping samples at end of iteration for batching?
        ])
    else:
        pipeline.extend([
            wds.split_by_worker,
            # at this point, we have an iterator over the shards assigned to each worker
            wds.tarfile_to_samples(handler=log_and_continue),
        ])
    pipeline.extend([
        wds.select(filter_no_caption),
        wds.decode("pilrgb", handler=log_and_continue),
        wds.rename(image="jpg;png", text="txt"),
        wds.map_dict(image=preprocess_img, text=preprocess_txt),
        wds.to_tuple("image", "text"),
        wds.batched(batch_size, partial=not is_train),
    ])

For using with_epochs, I have two schemes that appear to work (but haven't confirmed by checking all samples yet)
In this first one, if we apply with_epochs on dataset (initial data pipeline) we need to calculate num batches per worker as each worker is iterating over it's copy of dataset correct?
    dataset = wds.DataPipeline(*pipeline)
    if is_train:
        global_batch_size = batch_size * world_size
        num_batches = math.ceil(num_samples / global_batch_size)
        num_workers = max(1, workers)
        num_worker_batches = math.ceil(num_batches / num_workers)  # per dataloader worker
        num_batches = num_worker_batches * num_workers
        num_samples = num_batches * global_batch_size
        dataset = dataset.with_epoch(num_worker_batches)  # each worker is iterating over this
    else:
        # last batches are partial, eval is done on single (master) node
        num_batches = math.ceil(num_samples / batch_size)

    dataloader = wds.WebLoader(dataset, batch_size=None, shuffle=False, num_workers=workers)

Alternatively it appears we could apply with_epochs on dataloader instead, is there an efficiency downside to this? I assume that I'd still have to round num_batches to take num_workers into consideration so that each worker will have a equivalent # of batches and we won't leave some batches on one of the workers?
    dataset = wds.DataPipeline(*pipeline)
    dataloader = wds.WebLoader(dataset, batch_size=None, shuffle=False, num_workers=workers)
    if is_train:
        global_batch_size = batch_size * world_size
        num_batches = math.ceil(num_samples / global_batch_size)
        num_workers = max(1, workers)
        num_batches = math.ceil(num_batches / num_workers) * num_workers
        num_samples = num_batches * global_batch_size
        dataloader = dataloader.with_epoch(num_batches)
    else:
        # last batches are partial, eval is done on single (master) node
        num_batches = math.ceil(num_samples / batch_size)

And then one last concern, when using batched() as I am here, it looks like as batched is iterating over source it will leave samples unconsumed at the end of iteration. In conjunction with with_epochs at the end of the pipeline, I gather this will leave those samples unseen when the wrap around is performed? Should I be inserting  repeatedly() somewhere before batched to prevent this? Either after or before the wds.shuffle in the is_train branch for example?

Hi, thanks for posting this question, @rwightman. Did you find a solution regarding the optimal "position" where to put with_epochs ?

So, in short, your analysis sounds right, including your use of "repeatedly" to try to fill up epochs. The recommended way of using WebDataset for large scale, distributed training is with shard resampling.
Since many people didn't want to have to convert their existing "epoch-based" and indexed Dataset code to resampled streaming code, there is now a separate API that provides an indexed Dataset interface to WebDataset. You get it by "import wids". You can see how it works in the ./examples subdirectory.


Update Orbit Components

Currently, we're using version 0.0.0-rc7 we need to update to last version 0.0.0-rc14

closing for duplicity and because https://github.com/kiwicom/smart-faq/issues/550 is a more up to date issue


GIX-1116: Check SNS Neuron balances

Motivation
Add methods to check and refresh neuron balances.
Changes

New methods "claimNeuron", "refreshNeuron" and "queryNeuron" in SNS Governance.
New methods "getNeuronBalance" in SNS wrapper.
"getNextNeuronAccount" uses "queryNeuron" instead of checking the balance. Neurons can have balance 0.

Tests

Tests for new methods.
Fix broken tests after refactor.

@peterpeterparker @mstrasinskis please review


feat: add Dockerfile; add docker-compose

This is a continuation of #269. Closes #202, closes #264. Depends on #367

[x] Add Dockerfile
[x] Add docker-compose
[x] Update readme
[ ] Remove rust-cargo as dependency (requires wheels for akatsuki-pp-py before making changes)
[ ] Add nginx maxminddb

@7mochi is this ready for review?

@7mochi is this ready for review?

yep

looks like nice work! sorry for delay as i'm a bit busy at the moment (akatsuki is rather popular at the moment & scaling it is taking most of my free time), so will need to wait a bit for a proper review

Hey there, #332 may be of interest to you guys

I'll wait until #332 and #333 are merged to make the changes

LGTM now add cloudflare flex SSL instructions upside_down_face

I just added the instructions, could you guys check it?

LGTM now add cloudflare flex SSL instructions upside_down_face

I just added the instructions, could you guys check it?

Uploading images on github would be nice instead imgur
get url from this, they wont expire i think

LGTM now add cloudflare flex SSL instructions upside_down_face

I just added the instructions, could you guys check it?

Uploading images on github would be nice instead imgur
get url from this, they wont expire i think

I thought there was no problem because the og README even has a discord image :tf:

#339 will be merged very shortly

it has now been merged

overall pr looks good! very close to mergeable. @7mochi would you like to update to account for #339 or should i?

overall pr looks good! very close to mergeable. @7mochi would you like to update to account for #339 or should i?

could you do it? I'm pepega solving conflicts with git

Will wait for wheels to be built for akatsuki-pp-py before making changes

[ ] Remove rust and cargo as dependencies
[ ] Switch to python:3.9-slim

Will wait for wheels to be built for akatsuki-pp-py before making changes
* [ ]  Remove rust and cargo as dependencies

* [ ]  Switch to `python:3.9-slim`

rust is needed to build pp system tho?

rust is needed to build pp system tho?

That's what the wheels fix - we build akatsuki-pp-rs on the side of the python bindings, so we don't need it here.

Because #367 the nginx image has to be changed to nginx-geoip2

as a small disclaimer, you shouldn't mark this PR as ready for review until wheels are available

use checklists from now on, no longer need to mark as draft

i barely remember stuff from this PR anymore, lmao. @tsunyoku any progress on bindings?

i barely remember stuff from this PR anymore, lmao. @tsunyoku any progress on bindings?

lmao, i'm still waiting 😭

i was meant to move over to poetry so i could have the wheels but i forgot.. i will get to it tmrw. sorry guys!

#411 ^ _ ^


Add additional convenience APIs to JSON model structure

Description
Adds additional convenience APIs to the JSON model structure to simplify usage.

APIs for adding properties and elements to JsonArray and JsonObject without needing the create the JSON primitive types.
APIs for converting between JsonElement subtypes without the need to typecast.

All SDK Contribution checklist:

[x] The pull request does not introduce [breaking changes]
[x] CHANGELOG is updated for new features, bug fixes or other significant changes.
[x] I have read the contribution guidelines.

General Guidelines and Best Practices

[x] Title of the pull request is clear and informative.
[x] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

[x] Pull request includes test coverage for the included changes.

API change check
APIView has identified API level changes in this PR and created following API reviews.
com.azure:azure-json


Milestone 3# Feedback

Specific expectations for this milestone: Mechanics
Comments
Commits on the Python repo are not balanced.

Hi Andy, @andytai7
Thanks for the feedback. Please let us explain. I believe we divided the work evenly, and we believe each of us has equal contributions to the project. Commit number is not an accurate measure of our efforts, and please accept this explanation.
Thanks!


Changing the text of an annotation via SDL returns InternalServerError: 'NoneType' object has no attribute 'get_name'

EdgeDB Version: Alpha 4
OS Version: Ubuntu 20

Create the following schema via SDL:
START MIGRATION TO {
	module default {
		abstract annotation description;

		abstract link Ordered {
			annotation description := "1";
		}
	}
};
populate migration;
commit migration;

then execute the same migration again, with a different value for the description (e.g. "2").
It fails with the following error:
ERROR: InternalServerError: 'NoneType' object has no attribute 'get_name'
  Hint: This is most likely a bug in EdgeDB. Please consider opening an issue ticket at https://github.com/edgedb/edgedb/issues/new?template=bug_report.md
  Server traceback:
      Traceback (most recent call last):
        File "/usr/lib/x86_64-linux-gnu/edgedb-server-1-alpha4/lib/python3.8/site-packages/edb/server/procpool/worker.py", line 75, in worker
          res = await meth(*args)
        File "/usr/lib/x86_64-linux-gnu/edgedb-server-1-alpha4/lib/python3.8/site-packages/edb/server/compiler/compiler.py", line 1704, in compile_eql_tokens_in_tx
          return self._compile(ctx=ctx, tokens=eql_tokens)
        File "/usr/lib/x86_64-linux-gnu/edgedb-server-1-alpha4/lib/python3.8/site-packages/edb/server/compiler/compiler.py", line 1319, in _compile
          comp: dbstate.BaseQuery = self._compile_dispatch_ql(ctx, stmt)
        File "/usr/lib/x86_64-linux-gnu/edgedb-server-1-alpha4/lib/python3.8/site-packages/edb/server/compiler/compiler.py", line 1242, in _compile_dispatch_ql
          return self._compile_ql_migration(ctx, ql)
        File "/usr/lib/x86_64-linux-gnu/edgedb-server-1-alpha4/lib/python3.8/site-packages/edb/server/compiler/compiler.py", line 937, in _compile_ql_migration
          ddl_query = self._compile_and_apply_ddl_stmt(
        File "/usr/lib/x86_64-linux-gnu/edgedb-server-1-alpha4/lib/python3.8/site-packages/edb/server/compiler/compiler.py", line 748, in _compile_and_apply_ddl_stmt
          block, new_types = self._process_delta(ctx, delta)
        File "/usr/lib/x86_64-linux-gnu/edgedb-server-1-alpha4/lib/python3.8/site-packages/edb/server/compiler/compiler.py", line 433, in _process_delta
          self._compile_schema_storage_in_delta(ctx, delta, subblock)
        File "/usr/lib/x86_64-linux-gnu/edgedb-server-1-alpha4/lib/python3.8/site-packages/edb/server/compiler/compiler.py", line 452, in _compile_schema_storage_in_delta
          s_refl.write_meta(
        File "/usr/lib/x86_64-linux-gnu/edgedb-server-1-alpha4/lib/python3.8/functools.py", line 875, in wrapper
          return dispatch(args[0].__class__)(*args, **kw)
        File "/usr/lib/x86_64-linux-gnu/edgedb-server-1-alpha4/lib/python3.8/site-packages/edb/schema/reflection/writer.py", line 147, in write_meta_delta_root
          _descend(
        File "/usr/lib/x86_64-linux-gnu/edgedb-server-1-alpha4/lib/python3.8/site-packages/edb/schema/reflection/writer.py", line 112, in _descend
          write_meta(
        File "/usr/lib/x86_64-linux-gnu/edgedb-server-1-alpha4/lib/python3.8/functools.py", line 875, in wrapper
          return dispatch(args[0].__class__)(*args, **kw)
        File "/usr/lib/x86_64-linux-gnu/edgedb-server-1-alpha4/lib/python3.8/site-packages/edb/schema/reflection/writer.py", line 649, in write_meta_create_object
          _descend(
        File "/usr/lib/x86_64-linux-gnu/edgedb-server-1-alpha4/lib/python3.8/site-packages/edb/schema/reflection/writer.py", line 112, in _descend
          write_meta(
        File "/usr/lib/x86_64-linux-gnu/edgedb-server-1-alpha4/lib/python3.8/functools.py", line 875, in wrapper
          return dispatch(args[0].__class__)(*args, **kw)
        File "/usr/lib/x86_64-linux-gnu/edgedb-server-1-alpha4/lib/python3.8/site-packages/edb/schema/reflection/writer.py", line 717, in write_meta_alter_object
          _descend(
        File "/usr/lib/x86_64-linux-gnu/edgedb-server-1-alpha4/lib/python3.8/site-packages/edb/schema/reflection/writer.py", line 112, in _descend
          write_meta(
        File "/usr/lib/x86_64-linux-gnu/edgedb-server-1-alpha4/lib/python3.8/functools.py", line 875, in wrapper
          return dispatch(args[0].__class__)(*args, **kw)
        File "/usr/lib/x86_64-linux-gnu/edgedb-server-1-alpha4/lib/python3.8/site-packages/edb/schema/reflection/writer.py", line 707, in write_meta_alter_object
          _update_lprops(
        File "/usr/lib/x86_64-linux-gnu/edgedb-server-1-alpha4/lib/python3.8/site-packages/edb/schema/reflection/writer.py", line 754, in _update_lprops
          target_ref = target_obj.get_name(schema)
      AttributeError: 'NoneType' object has no attribute 'get_name'

That was quick :+1:


Type conversion to Boolean

Minor issue with conversions on 0.6.3.  Passing True or False to interface taking and java.lang.Object is currently producing a java.lang.Long rather than an expected java.lang.Boolean.  Likely this is just an order of operations problem as I thought that we had logic to deal with this.

See #360 and #361 for tests showing the current behavior.

Fixed in master


Implement n sum

(Put an X inside the [ ] to denote check mark [X].)

[x] If creating a new file :

[x] added links to it in the README files ?
[x] included tests with it ?
[x] added description (overview of algorithm, time and space compleixty, and possible edge case) in docstrings ?

[ ] if done some changes :

[ ] wrote short description in the PR explaining what the changes do ?
[ ] Fixes #[issue number] if related to any issue

[ ] other

Pull Request Test Coverage Report for Build 656

67 of 67 (100.0%)  changed or added relevant lines in 2 files are covered.
1 unchanged line in 1 file lost coverage.
Overall coverage increased (+0.4%) to 71.499%

Files with Coverage Reduction
New Missed Lines
%

algorithms/linkedlist/is_palindrome.py
1
93.33%

Totals

Change from base Build 654:
0.4%

Covered Lines:
3783

Relevant Lines:
5291

💛  - Coveralls


/api/interpreter does not respond after initial clone, times out with 504

With the following docker-compose.yml file, after initially cloning and then restarting the service, the logs indicate that everything starts up correctly.
When I make an API call however, the service only responds with an HTTP 504 gateway timeout.

The logs after the second startup of the service (after clone)

Thank you for your timely reply. I used the <IP_ADDRESS> version of the image before, changed it to the latest image, and then accessed /api/interpreter, and then reported the error open64: 13 Permission denied /db/db//osm3s_osm_base Unix_Socket::7, Later, enter the container and run the chmod og+rx /db command. The problem is solved and you can access and use it normally. I hope OVERPASS_PLANET_URL can be modified to read local files, otherwise downloading data will be too slow. thank you

Now I am encountering new problems, visit http://<IP_ADDRESS>:12345/api/interpreter?data=[out:xml][bbox:30.52116,104.04774,30.54482,104.07829]; ( node; way; relation; ); out meta;, error runtime error: open64: 2 No such file or directory /db/db/user_data.bin File_Blocks::File_Blocks::1
, where is the user_data.bin file downloaded? Visit https://overpass-api.de/api/interpreter?data=[out:xml][bbox:30.52116,104.04774,30.54482,104.07829];
(
node;
way;
relation;
);
out meta; you can download the file normally

`package com.yifang.map.demo;
import java.io.BufferedInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.HttpURLConnection;
import java.net.URL;
import java.net.URLEncoder;
import java.nio.charset.StandardCharsets;
public class OverpassAPI {
public static void main(String[] args) {
    // 设置 Overpass API URL

//        String url = "http://overpass-api.de/api/interpreter";
//        String url = "http://<IP_ADDRESS>:12345/api/interpreter";
String url = "http://<IP_ADDRESS>:12345/api/interpreter";
    double latMin = 30.52116;
    double lonMin = 104.04774;
    double latMax = 30.54482;
    double lonMax = 104.07829;

    String query = String.format("""
            [out:xml][bbox:%f,%f,%f,%f];
            (
              node;
              way;
              relation;
            );
            out meta;
            """, latMin, lonMin, latMax, lonMax);

    try {
        // 对查询字符串进行 URL 编码
        String encodedQuery = URLEncoder.encode(query, StandardCharsets.UTF_8);

        // 设置连接
        HttpURLConnection connection = (HttpURLConnection) new URL(url + "?data=" + encodedQuery).openConnection();
        connection.setRequestMethod("GET");

        // 检查响应码
        int responseCode = connection.getResponseCode();
        if (responseCode == HttpURLConnection.HTTP_OK) {
            // 获取输入流并保存为 OSM 文件
            try (InputStream in = new BufferedInputStream(connection.getInputStream());
                 OutputStream out = new FileOutputStream("OverpassAPI.osm")) {
                byte[] buffer = new byte[1024];
                int bytesRead;
                while ((bytesRead = in.read(buffer)) != -1) {
                    out.write(buffer, 0, bytesRead);
                }
                System.out.println("OSM 数据已成功保存到 OverpassAPI.osm 文件中！");
            }
        } else {
            System.out.println("请求失败，响应码: " + responseCode);
        }
    } catch (IOException e) {
        System.err.println("发生错误: " + e.getMessage());
    }
}

}
`


⚠ Empornium has degraded performance

In 5ce63d1, Empornium ($EMP) experienced degraded performance:

HTTP code: 500
Response time: 1055 ms

Resolved: Empornium performance has improved in 9f3ac0c after 6 minutes.


List command

fixes #30
$  bundle exec mortar list
SHOT            RESOURCES
hello            10

RESOURCES refers to the count of resources fund from the given shot
$  bundle exec mortar list -q
hello

Also quiet option to allow easy scripting around.

@kke added completions and changed to use tty-table. also added some docs for this, plus docs for missing describe command. :) PTAL

@kke fixed all the comments, PTAL


Wrong address

While withdrawing my amount from alpha wallet for binance i pasted by mistakenly I put wrong address instead of binance wallet address kindly please help to revert back my 50 $

While withdrawing my amount from alpha wallet for binance i pasted by mistakenly I put wrong address instead of binance wallet address kindly please help to revert back my 50 $

Thanks for bringing this issue to our attention.The best way to resolve this is to contact our support team.
For the fastest resolution, please click on the link below to our dedicated support line:
Support: AlphaWallet Help Center
Note: Click on the live chat icon at the bottom corner of the page to start a conversation

While withdrawing my amount from alpha wallet for binance i pasted by mistakenly I put wrong address instead of binance wallet address kindly please help to revert back my 50 $

@Najamhaq15
This issue will be preview on pull requests.
While reviewing….. Visit the live support page to initiate a chat with an agent for further assistance via Alpha wallet live agent.
Regards

@Najamhaq15 Dear user
I understand that you have been experiencing some issues and I want to ensure you get the support you need as quickly as possible by using our live chat option.
Support : Support
1.	Access the Live Chat:
•	Navigate to our website’s home page.
•	Look for the chat icon located at the bottom right corner of the screen and start a 💬.

@Najamhaq15 Dear user
I understand that you have been experiencing some issues and I want to ensure you get the support you need as quickly as possible by using our live chat option.
Support : Support
1.	Access the Live Chat:
•	Navigate to our website’s home page.
•	Look for the chat icon located at the bottom right corner of the screen and start a chat.