python311Packages.{jax,jaxlib,jaxlib-bin}: 0.4.28 -> 0.4.30

Description of changes
Changelogs:

jaxlib: https://github.com/google/jax/releases/tag/jaxlib-v0.4.30
jax: https://github.com/google/jax/releases/tag/jax-v0.4.30

cc @samuela
Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

I cannot get jaxlib to build:
Analyzing: 2 targets (46 packages loaded, 225 targets configured)
INFO: Repository pypi_setuptools instantiated at:
  /build/source/WORKSPACE:27:13: in <toplevel>
  /build/output/external/pypi/requirements.bzl:49:20: in install_deps
Repository rule whl_library defined at:
  /build/output/external/rules_python/python/pip_install/pip_repository.bzl:697:30: in <toplevel>
ERROR: An error occurred during the fetch of repository 'pypi_setuptools':
   Traceback (most recent call last):
        File "/build/output/external/rules_python/python/pip_install/pip_repository.bzl", line 596, column 13, in _whl_library_impl
                fail("whl_library %s failed: %s (%s) error code: '%s'" % (rctx.attr.name, result.stdout, result.stderr, result.return_code))
Error in fail: whl_library pypi_setuptools failed:  (src/main/tools/process-wrapper-legacy.cc:80: "execvp(/build/output/external/python_x86_64-unknown-linux-gnu/bin/python3, ...)": >
) error code: '1'
ERROR: /build/source/WORKSPACE:27:13: fetching whl_library rule //external:pypi_setuptools: Traceback (most recent call last):
        File "/build/output/external/rules_python/python/pip_install/pip_repository.bzl", line 596, column 13, in _whl_library_impl
                fail("whl_library %s failed: %s (%s) error code: '%s'" % (rctx.attr.name, result.stdout, result.stderr, result.return_code))
Error in fail: whl_library pypi_setuptools failed:  (src/main/tools/process-wrapper-legacy.cc:80: "execvp(/build/output/external/python_x86_64-unknown-linux-gnu/bin/python3, ...)": >
) error code: '1'
ERROR: /build/source/jaxlib/tools/BUILD.bazel:25:10: //jaxlib/tools:build_wheel depends on @pypi_setuptools//:pkg in repository @pypi_setuptools which failed to fetch. no such packa>
) error code: '1'
ERROR: Analysis of target '//jaxlib/tools:build_wheel' failed; build aborted: Analysis failed
INFO: Elapsed time: 38.193s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (46 packages loaded, 225 targets configured)

I cannot get jaxlib to build:

Could be due to this change: https://github.com/google/jax/compare/jaxlib-v0.4.28...jaxlib-v0.4.29#diff-5493ff8e9397811510e780de47c57abb70137f1afe85d1519130dc3679d60ce5

I updated to v0.4.30 which was released today.
However, it crashes in a different way:

Running phase: unpackPhase
unpacking source archive /nix/store/sbky7hcqq756a9vyg7g76nq5l9z4wsz4-source
source root is source
Running phase: patchPhase
Running phase: updateAutotoolsGnuConfigScriptsPhase
Running phase: configurePhase
no configure script, doing nothing
Running phase: buildPhase
Extracting Bazel installation...
WARNING: The following configs were expanded more than once: [mkl_open_source_only]. For repeatable flags, repeats are counted twice and may lead to unexpected behavior.
INFO: Options provided by the client:
  Inherited 'common' options: --isatty=1 --terminal_columns=0
INFO: Reading rc options for 'build' from /build/source/.bazelrc:
  Inherited 'common' options: --experimental_repo_remote_exec
INFO: Reading rc options for 'build' from /nix/store/62azw4ysqscs49zybmbdl21yh1yra7s2-bazel-rc:
  'build' options: --distdir=/nix/store/c74pr6dwc94w8aynj631xkna5gvmg20c-bazel-deps --extra_toolchains=@bazel_tools//tools/jdk:nonprebuilt_toolchain_definition --tool_java_runtime_version=local_jdk_11 --java_runtime_version=local_jdk_11
INFO: Reading rc options for 'build' from /build/source/.bazelrc:
  'build' options: --nocheck_visibility --apple_platform_type=macos --macos_minimum_os=10.14 --announce_rc --define open_source_build=true --spawn_strategy=standalone --enable_platform_specific_config --experimental_cc_shared_library --define=no_aws_support=true --define=no_gcp_support=true --define=no_hdfs_support=true --define=no_kafka_support=true --define=no_ignite_support=true --define=grpc_no_ares=true --define=tsl_link_protobuf=true -c opt --config=short_logs --copt=-DMLIR_PYTHON_PACKAGE_PREFIX=jaxlib.mlir. --@xla//xla/python:enable_gpu=false
INFO: Reading rc options for 'build' from /build/source/.jax_configure.bazelrc:
  'build' options: --strategy=Genrule=standalone --repo_env PYTHON_BIN_PATH=/nix/store/6b1fqdwb3g56j5pazv8zkx9qd0mv3wiz-python3-3.11.9/bin/python --action_env=PYENV_ROOT --python_path=/nix/store/6b1fqdwb3g56j5pazv8zkx9qd0mv3wiz-python3-3.11.9/bin/python --distinct_host_configuration=false --define PROTOBUF_INCLUDE_PATH=/nix/store/bhr113ijfjij5z3l8h0dzxd0836647wc-protobuf-25.3/include --config=mkl_open_source_only
INFO: Found applicable config definition build:short_logs in file /build/source/.bazelrc: --output_filter=DONT_MATCH_ANYTHING
INFO: Found applicable config definition build:mkl_open_source_only in file /build/source/.bazelrc: --define=tensorflow_mkldnn_contraction_kernel=1
INFO: Found applicable config definition build:avx_posix in file /build/source/.bazelrc: --copt=-mavx --host_copt=-mavx
INFO: Found applicable config definition build:mkl_open_source_only in file /build/source/.bazelrc: --define=tensorflow_mkldnn_contraction_kernel=1
INFO: Found applicable config definition build:linux in file /build/source/.bazelrc: --config=posix --copt=-Wno-unknown-warning-option --copt=-Wno-stringop-truncation --copt=-Wno-array-parameter
INFO: Found applicable config definition build:posix in file /build/source/.bazelrc: --copt=-fvisibility=hidden --copt=-Wno-sign-compare --cxxopt=-std=c++17 --host_cxxopt=-std=c++17
Loading:
Loading:
DEBUG: /build/output/external/xla/third_party/py/python_repo.bzl:99:14:
HERMETIC_PYTHON_VERSION variable was not set correctly, using default version.
Python 3.11 will be used.
To select Python version, either set HERMETIC_PYTHON_VERSION env variable in
your shell:
  export HERMETIC_PYTHON_VERSION=3.12
OR pass it as an argument to bazel command directly or inside your .bazelrc
file:
  --repo_env=HERMETIC_PYTHON_VERSION=3.12
DEBUG: /build/output/external/xla/third_party/py/python_repo.bzl:110:10: Using hermetic Python 3.11
Loading:
INFO: Repository python_x86_64-unknown-linux-gnu instantiated at:
  /build/source/WORKSPACE:24:23: in <toplevel>
  /build/output/external/xla/third_party/py/python_init_toolchains.bzl:9:35: in python_init_toolchains
  /build/output/external/rules_python/python/repositories.bzl:580:26: in python_register_toolchains
Repository rule python_repository defined at:
  /build/output/external/rules_python/python/repositories.bzl:404:36: in <toplevel>
java.util.logging.ErrorManager: 1
java.io.IOException: No space left on device
        at java.base/java.io.FileOutputStream.writeBytes(Native Method)
        at java.base/java.io.FileOutputStream.write(FileOutputStream.java:354)
        at java.base/java.io.FileOutputStream.write(FileOutputStream.java:354)
        at java.base/java.io.FileOutputStream.write(FileOutputStream.java:354)
        at java.base/java.io.FileOutputStream.write(FileOutputStream.java:354)
        at java.base/java.io.FileOutputStream.write(FileOutputStream.java:354)
        at java.base/java.io.FileOutputStream.write(FileOutputStream.java:354)
        at java.base/java.io.FileOutputStream.write(FileOutputStream.java:354)
        at java.base/java.io.FileOutputStream.write(FileOutputStream.java:354)
        at java.base/java.io.FileOutputStream.write(FileOutputStream.java:354)
        at java.base/java.io.FileOutputStream.write(FileOutputStream.java:354)
        at java.base/java.io.FileOutputStream.write(FileOutputStream.java:354)
        at java.base/java.io.FileOutputStream.write(FileOutputStream.java:354)
        at java.base/java.io.FileOutputStream.write(FileOutputStream.java:354)
        at java.base/java.io.FileOutputStream.write(FileOutputStream.java:354)
        at java.base/java.io.FileOutputStream.write(FileOutputStream.java:354)
        at java.base/java.io.FileOutputStream.write(FileOutputStream.java:354)
        at java.base/java.io.FileOutputStream.write(FileOutputStream.java:354)
        at java.base/java.io.FileOutputStream.write(FileOutputStream.java:354)
        at java.base/java.io.FileOutputStream.write(FileOutputStream.java:354)
        at java.base/java.io.FileOutputStream.write(FileOutputStream.java:354)
        at java.base/java.io.FileOutputStream.write(FileOutputStream.java:354)
        at java.base/java.io.FileOutputStream.write(FileOutputStream.java:354)
        at java.base/java.io.FileOutputStream.write(FileOutputStream.java:354)
        at java.base/java.io.FileOutputStream.write(FileOutputStream.java:354)
        at java.base/java.io.FileOutputStream.write(FileOutputStream.java:354)
        at java.base/java.io.FileOutputStream.write(FileOutputStream.java:354)
        at java.base/java.io.FileOutputStream.write(FileOutputStream.java:354)
        at java.base/java.io.FileOutputStream.write(FileOutputStream.java:354)
        at java.base/java.io.FileOutputStream.write(FileOutputStream.java:354)
        at java.base/java.io.FileOutputStream.write(FileOutputStream.java:354)
        at java.base/java.io.FileOutputStream.write(FileOutputStream.java:354)
        at java.base/java.io.FileOutputStream.write(FileOutputStream.java:354)
        at java.base/java.io.FileOutputStream.write(FileOutputStream.java:354)
        at java.base/java.io.FileOutputStream.write(FileOutputStream.java:354)
        at java.base/java.io.FileOutputStream.write(FileOutputStream.java:354)
        at java.base/java.io.FileOutputStream.write(FileOutputStream.java:354)
        at java.base/java.io.FileOutputStream.write(FileOutputStream.java:354)
        at java.base/java.io.FileOutputStream.write(FileOutputStream.java:354)
        at java.base/java.io.FileOutputStream.write(FileOutputStream.java:354)
        at java.base/java.io.FileOutputStream.write(FileOutputStream.java:354)
        at java.base/java.io.FileOutputStream.write(FileOutputStream.java:354)
        at java.base/java.io.FileOutputStream.write(FileOutputStream.java:354)
        at java.base/java.io.FileOutputStream.write(FileOutputStream.java:354)
        at java.base/java.io.FileOutputStream.write(FileOutputStream.java:354)
        at java.base/java.io.FileOutputStream.write(FileOutputStream.java:354)
        at java.base/java.io.FileOutputStream.write(FileOutputStream.java:354)
        at java.base/java.io.FileOutputStream.write(FileOutputStream.java:354)
        at java.base/java.io.FileOutputStream.write(FileOutputStream.java:354)
        at java.base/java.io.FileOutputStream.write(FileOutputStream.java:354)
        at java.base/java.io.FileOutputStream.write(FileOutputStream.java:354)
        at com.google.common.io.CountingOutputStream.write(CountingOutputStream.java:52)        at java.base/sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:233)        at java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:303) at java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:281) at java.base/sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)     at java.base/sun.nio.cs.StreamEncoder.write(StreamEncoder.java:135)     at java.base/java.io.OutputStreamWriter.write(OutputStreamWriter.java:226)      at java.base/java.io.Writer.write(Writer.java:249)      at com.google.devtools.build.lib.util.SimpleLogHandler$Output.write(SimpleLogHandler.java:765)  at com.google.devtools.build.lib.util.SimpleLogHandler.publish(SimpleLogHandler.java:434)       at java.logging/java.util.logging.Logger.log(Logger.java:979>
        at com.google.common.io.CountingOutputStream.write(CountingOutputStream.java:52)        at java.base/sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:233)        at java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:303) at java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:281) at java.base/sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)     at java.base/sun.nio.cs.StreamEncoder.write(StreamEncoder.java:135)     at java.base/java.io.OutputStreamWriter.write(OutputStreamWriter.java:226)      at java.base/java.io.Writer.write(Writer.java:249)      at com.google.devtools.build.lib.util.SimpleLogHandler$Output.write(SimpleLogHandler.java:765)  at com.google.devtools.build.lib.util.SimpleLogHandler.publish(SimpleLogHandler.java:434)       at java.logging/java.util.logging.Logger.log(Logger.java:979>
   Traceback (most recent call last):
        File "/build/output/external/rules_python/python/repositories.bzl", line 175, column 34, in _python_repository_impl
                rctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error extracting /build/output/external/python_x86_64-unknown-linux-gnu/temp12096015298234464535/cpython-3.11.6+20231002-x86_64-unknown-linux-gnu-install_only.tar.gz to /build/output/external/python_x86_64-unknown-linux-gnu/temp12096015298234464535: write (No space left on device)
        at java.base/java.io.FileOutputStream.write(FileOutputStream.java:354)
        at com.google.common.io.CountingOutputStream.write(CountingOutputStream.java:52)        at java.base/sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:233)        at java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:303) at java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:281) at java.base/sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)     at java.base/sun.nio.cs.StreamEncoder.write(StreamEncoder.java:135)     at java.base/java.io.OutputStreamWriter.write(OutputStreamWriter.java:226)      at java.base/java.io.Writer.write(Writer.java:249)      at com.google.devtools.build.lib.util.SimpleLogHandler$Output.write(SimpleLogHandler.java:765)  at com.google.devtools.build.lib.util.SimpleLogHandler.publish(SimpleLogHandler.java:434)       at java.logging/java.util.logging.Logger.log(Logger.java:979>
   Traceback (most recent call last):
        File "/build/output/external/rules_python/python/repositories.bzl", line 175, column 34, in _python_repository_impl
                rctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error extracting /build/output/external/python_x86_64-unknown-linux-gnu/temp12096015298234464535/cpython-3.11.6+20231002-x86_64-unknown-linux-gnu-install_only.tar.gz to /build/output/external/python_x86_64-unknown-linux-gnu/temp12096015298234464535: write (No space left on device)
        at com.google.common.io.CountingOutputStream.write(CountingOutputStream.java:52)        at java.base/sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:233)        at java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:303) at java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:281) at java.base/sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)     at java.base/sun.nio.cs.StreamEncoder.write(StreamEncoder.java:135)     at java.base/java.io.OutputStreamWriter.write(OutputStreamWriter.java:226)      at java.base/java.io.Writer.write(Writer.java:249)      at com.google.devtools.build.lib.util.SimpleLogHandler$Output.write(SimpleLogHandler.java:765)  at com.google.devtools.build.lib.util.SimpleLogHandler.publish(SimpleLogHandler.java:434)       at java.logging/java.util.logging.Logger.log(Logger.java:979>
        at com.google.common.io.CountingOutputStream.write(CountingOutputStream.java:52)        at java.base/sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:233)        at java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:303) at java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:281) at java.base/sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)     at java.base/sun.nio.cs.StreamEncoder.write(StreamEncoder.java:135)     at java.base/java.io.OutputStreamWriter.write(OutputStreamWriter.java:226)      at java.base/java.io.Writer.write(Writer.java:249)      at com.google.devtools.build.lib.util.SimpleLogHandler$Output.write(SimpleLogHandler.java:765)  at com.google.devtools.build.lib.util.SimpleLogHandler.publish(SimpleLogHandler.java:434)       at java.logging/java.util.logging.Logger.log(Logger.java:979>
   Traceback (most recent call last):
        File "/build/output/external/rules_python/python/repositories.bzl", line 175, column 34, in _python_repository_impl
                rctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error extracting /build/output/external/python_x86_64-unknown-linux-gnu/temp12096015298234464535/cpython-3.11.6+20231002-x86_64-unknown-linux-gnu-install_only.tar.gz to /build/output/external/python_x86_64-unknown-linux-gnu/temp12096015298234464535: write (No space left on device)
        at com.google.common.io.CountingOutputStream.write(CountingOutputStream.java:52)        at java.base/sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:233)        at java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:303) at java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:281) at java.base/sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)     at java.base/sun.nio.cs.StreamEncoder.write(StreamEncoder.java:135)     at java.base/java.io.OutputStreamWriter.write(OutputStreamWriter.java:226)      at java.base/java.io.Writer.write(Writer.java:249)      at com.google.devtools.build.lib.util.SimpleLogHandler$Output.write(SimpleLogHandler.java:765)  at com.google.devtools.build.lib.util.SimpleLogHandler.publish(SimpleLogHandler.java:434)       at java.logging/java.util.logging.Logger.log(Logger.java:979>
        File "/build/output/external/rules_python/python/repositories.bzl", line 175, column 34, in _python_repository_impl
                rctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error extracting /build/output/external/python_x86_64-unknown-linux-gnu/temp12096015298234464535/cpython-3.11.6+20231002-x86_64-unknown-linux-gnu-install_only.tar.gz to /build/output/external/python_x86_64-unknown-linux-gnu/temp12096015298234464535: write (No space left on device)
        at java.base/java.io.FileOutputStream.write(FileOutputStream.java:354)
        at com.google.common.io.CountingOutputStream.write(CountingOutputStream.java:52)        at java.base/sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:233)        at java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:303) at java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:281) at java.base/sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)     at java.base/sun.nio.cs.StreamEncoder.write(StreamEncoder.java:135)     at java.base/java.io.OutputStreamWriter.write(OutputStreamWriter.java:226)      at java.base/java.io.Writer.write(Writer.java:249)      at com.google.devtools.build.lib.util.SimpleLogHandler$Output.write(SimpleLogHandler.java:765)  at com.google.devtools.build.lib.util.SimpleLogHandler.publish(SimpleLogHandler.java:434)       at java.logging/java.util.logging.Logger.log(Logger.java:979>
   Traceback (most recent call last):
        File "/build/output/external/rules_python/python/repositories.bzl", line 175, column 34, in _python_repository_impl
                rctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error extracting /build/output/external/python_x86_64-unknown-linux-gnu/temp12096015298234464535/cpython-3.11.6+20231002-x86_64-unknown-linux-gnu-install_only.tar.gz to /build/output/external/python_x86_64-unknown-linux-gnu/temp12096015298234464535: write (No space left on device)
        at com.google.common.io.CountingOutputStream.write(CountingOutputStream.java:52)        at java.base/sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:233)        at java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:303) at java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:281) at java.base/sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)     at java.base/sun.nio.cs.StreamEncoder.write(StreamEncoder.java:135)     at java.base/java.io.OutputStreamWriter.write(OutputStreamWriter.java:226)      at java.base/java.io.Writer.write(Writer.java:249)      at com.google.devtools.build.lib.util.SimpleLogHandler$Output.write(SimpleLogHandler.java:765)  at com.google.devtools.build.lib.util.SimpleLogHandler.publish(SimpleLogHandler.java:434)       at java.logging/java.util.logging.Logger.log(Logger.java:979>
        File "/build/output/external/rules_python/python/repositories.bzl", line 175, column 34, in _python_repository_impl
                rctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error extracting /build/output/external/python_x86_64-unknown-linux-gnu/temp12096015298234464535/cpython-3.11.6+20231002-x86_64-unknown-linux-gnu-install_only.tar.gz to /build/output/external/python_x86_64-unknown-linux-gnu/temp12096015298234464535: write (No space left on device)
Loading:
        at java.base/java.io.FileOutputStream.write(FileOutputStream.java:354)
        at com.google.common.io.CountingOutputStream.write(CountingOutputStream.java:52)        at java.base/sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:233)        at java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:303) at java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:281) at java.base/sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)     at java.base/sun.nio.cs.StreamEncoder.write(StreamEncoder.java:135)     at java.base/java.io.OutputStreamWriter.write(OutputStreamWriter.java:226)      at java.base/java.io.Writer.write(Writer.java:249)      at com.google.devtools.build.lib.util.SimpleLogHandler$Output.write(SimpleLogHandler.java:765)  at com.google.devtools.build.lib.util.SimpleLogHandler.publish(SimpleLogHandler.java:434)       at java.logging/java.util.logging.Logger.log(Logger.java:979>
   Traceback (most recent call last):
        File "/build/output/external/rules_python/python/repositories.bzl", line 175, column 34, in _python_repository_impl
                rctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error extracting /build/output/external/python_x86_64-unknown-linux-gnu/temp12096015298234464535/cpython-3.11.6+20231002-x86_64-unknown-linux-gnu-install_only.tar.gz to /build/output/external/python_x86_64-unknown-linux-gnu/temp12096015298234464535: write (No space left on device)
        at com.google.common.io.CountingOutputStream.write(CountingOutputStream.java:52)        at java.base/sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:233)        at java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:303) at java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:281) at java.base/sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)     at java.base/sun.nio.cs.StreamEncoder.write(StreamEncoder.java:135)     at java.base/java.io.OutputStreamWriter.write(OutputStreamWriter.java:226)      at java.base/java.io.Writer.write(Writer.java:249)      at com.google.devtools.build.lib.util.SimpleLogHandler$Output.write(SimpleLogHandler.java:765)  at com.google.devtools.build.lib.util.SimpleLogHandler.publish(SimpleLogHandler.java:434)       at java.logging/java.util.logging.Logger.log(Logger.java:979>
        File "/build/output/external/rules_python/python/repositories.bzl", line 175, column 34, in _python_repository_impl
                rctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error extracting /build/output/external/python_x86_64-unknown-linux-gnu/temp12096015298234464535/cpython-3.11.6+20231002-x86_64-unknown-linux-gnu-install_only.tar.gz to /build/output/external/python_x86_64-unknown-linux-gnu/temp12096015298234464535: write (No space left on device)
Loading:
        at com.google.common.io.CountingOutputStream.write(CountingOutputStream.java:52)        at java.base/sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:233)        at java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:303) at java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:281) at java.base/sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)     at java.base/sun.nio.cs.StreamEncoder.write(StreamEncoder.java:135)     at java.base/java.io.OutputStreamWriter.write(OutputStreamWriter.java:226)      at java.base/java.io.Writer.write(Writer.java:249)      at com.google.devtools.build.lib.util.SimpleLogHandler$Output.write(SimpleLogHandler.java:765)  at com.google.devtools.build.lib.util.SimpleLogHandler.publish(SimpleLogHandler.java:434)       at java.logging/java.util.logging.Logger.log(Logger.java:979>
        at com.google.common.io.CountingOutputStream.write(CountingOutputStream.java:52)        at java.base/sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:233)        at java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:303) at java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:281) at java.base/sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)     at java.base/sun.nio.cs.StreamEncoder.write(StreamEncoder.java:135)     at java.base/java.io.OutputStreamWriter.write(OutputStreamWriter.java:226)      at java.base/java.io.Writer.write(Writer.java:249)      at com.google.devtools.build.lib.util.SimpleLogHandler$Output.write(SimpleLogHandler.java:765)  at com.google.devtools.build.lib.util.SimpleLogHandler.publish(SimpleLogHandler.java:434)       at java.logging/java.util.logging.Logger.log(Logger.java:979>
   Traceback (most recent call last):
        File "/build/output/external/rules_python/python/repositories.bzl", line 175, column 34, in _python_repository_impl
                rctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error extracting /build/output/external/python_x86_64-unknown-linux-gnu/temp12096015298234464535/cpython-3.11.6+20231002-x86_64-unknown-linux-gnu-install_only.tar.gz to /build/output/external/python_x86_64-unknown-linux-gnu/temp12096015298234464535: write (No space left on device)
        at com.google.common.io.CountingOutputStream.write(CountingOutputStream.java:52)        at java.base/sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:233)        at java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:303) at java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:281) at java.base/sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)     at java.base/sun.nio.cs.StreamEncoder.write(StreamEncoder.java:135)     at java.base/java.io.OutputStreamWriter.write(OutputStreamWriter.java:226)      at java.base/java.io.Writer.write(Writer.java:249)      at com.google.devtools.build.lib.util.SimpleLogHandler$Output.write(SimpleLogHandler.java:765)  at com.google.devtools.build.lib.util.SimpleLogHandler.publish(SimpleLogHandler.java:434)       at java.logging/java.util.logging.Logger.log(Logger.java:979>
        File "/build/output/external/rules_python/python/repositories.bzl", line 175, column 34, in _python_repository_impl
                rctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error extracting /build/output/external/python_x86_64-unknown-linux-gnu/temp12096015298234464535/cpython-3.11.6+20231002-x86_64-unknown-linux-gnu-install_only.tar.gz to /build/output/external/python_x86_64-unknown-linux-gnu/temp12096015298234464535: write (No space left on device)
Loading:
        at com.google.common.io.CountingOutputStream.write(CountingOutputStream.java:52)        at java.base/sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:233)        at java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:303) at java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:281) at java.base/sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)     at java.base/sun.nio.cs.StreamEncoder.write(StreamEncoder.java:135)     at java.base/java.io.OutputStreamWriter.write(OutputStreamWriter.java:226)      at java.base/java.io.Writer.write(Writer.java:249)      at com.google.devtools.build.lib.util.SimpleLogHandler$Output.write(SimpleLogHandler.java:765)  at com.google.devtools.build.lib.util.SimpleLogHandler.publish(SimpleLogHandler.java:434)       at java.logging/java.util.logging.Logger.log(Logger.java:979>
        at java.base/java.io.FileOutputStream.write(FileOutputStream.java:354)
        at com.google.common.io.CountingOutputStream.write(CountingOutputStream.java:52)        at java.base/sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:233)        at java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:303) at java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:281) at java.base/sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)     at java.base/sun.nio.cs.StreamEncoder.write(StreamEncoder.java:135)     at java.base/java.io.OutputStreamWriter.write(OutputStreamWriter.java:226)      at java.base/java.io.Writer.write(Writer.java:249)      at com.google.devtools.build.lib.util.SimpleLogHandler$Output.write(SimpleLogHandler.java:765)  at com.google.devtools.build.lib.util.SimpleLogHandler.publish(SimpleLogHandler.java:434)       at java.logging/java.util.logging.Logger.log(Logger.java:979>
   Traceback (most recent call last):
        File "/build/output/external/rules_python/python/repositories.bzl", line 175, column 34, in _python_repository_impl
                rctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error extracting /build/output/external/python_x86_64-unknown-linux-gnu/temp12096015298234464535/cpython-3.11.6+20231002-x86_64-unknown-linux-gnu-install_only.tar.gz to /build/output/external/python_x86_64-unknown-linux-gnu/temp12096015298234464535: write (No space left on device)
        at com.google.common.io.CountingOutputStream.write(CountingOutputStream.java:52)        at java.base/sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:233)        at java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:303) at java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:281) at java.base/sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)     at java.base/sun.nio.cs.StreamEncoder.write(StreamEncoder.java:135)     at java.base/java.io.OutputStreamWriter.write(OutputStreamWriter.java:226)      at java.base/java.io.Writer.write(Writer.java:249)      at com.google.devtools.build.lib.util.SimpleLogHandler$Output.write(SimpleLogHandler.java:765)  at com.google.devtools.build.lib.util.SimpleLogHandler.publish(SimpleLogHandler.java:434)       at java.logging/java.util.logging.Logger.log(Logger.java:979>
        File "/build/output/external/rules_python/python/repositories.bzl", line 175, column 34, in _python_repository_impl
                rctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error extracting /build/output/external/python_x86_64-unknown-linux-gnu/temp12096015298234464535/cpython-3.11.6+20231002-x86_64-unknown-linux-gnu-install_only.tar.gz to /build/output/external/python_x86_64-unknown-linux-gnu/temp12096015298234464535: write (No space left on device)
Loading:
        at com.google.common.io.CountingOutputStream.write(CountingOutputStream.java:52)        at java.base/sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:233)        at java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:303) at java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:281) at java.base/sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)     at java.base/sun.nio.cs.StreamEncoder.write(StreamEncoder.java:135)     at java.base/java.io.OutputStreamWriter.write(OutputStreamWriter.java:226)      at java.base/java.io.Writer.write(Writer.java:249)      at com.google.devtools.build.lib.util.SimpleLogHandler$Output.write(SimpleLogHandler.java:765)  at com.google.devtools.build.lib.util.SimpleLogHandler.publish(SimpleLogHandler.java:434)       at java.logging/java.util.logging.Logger.log(Logger.java:979>
        at java.base/java.io.FileOutputStream.write(FileOutputStream.java:354)
        at com.google.common.io.CountingOutputStream.write(CountingOutputStream.java:52)        at java.base/sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:233)        at java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:303) at java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:281) at java.base/sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)     at java.base/sun.nio.cs.StreamEncoder.write(StreamEncoder.java:135)     at java.base/java.io.OutputStreamWriter.write(OutputStreamWriter.java:226)      at java.base/java.io.Writer.write(Writer.java:249)      at com.google.devtools.build.lib.util.SimpleLogHandler$Output.write(SimpleLogHandler.java:765)  at com.google.devtools.build.lib.util.SimpleLogHandler.publish(SimpleLogHandler.java:434)       at java.logging/java.util.logging.Logger.log(Logger.java:979>
   Traceback (most recent call last):
        File "/build/output/external/rules_python/python/repositories.bzl", line 175, column 34, in _python_repository_impl
                rctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error extracting /build/output/external/python_x86_64-unknown-linux-gnu/temp12096015298234464535/cpython-3.11.6+20231002-x86_64-unknown-linux-gnu-install_only.tar.gz to /build/output/external/python_x86_64-unknown-linux-gnu/temp12096015298234464535: write (No space left on device)
        at com.google.common.io.CountingOutputStream.write(CountingOutputStream.java:52)        at java.base/sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:233)        at java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:303) at java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:281) at java.base/sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)     at java.base/sun.nio.cs.StreamEncoder.write(StreamEncoder.java:135)     at java.base/java.io.OutputStreamWriter.write(OutputStreamWriter.java:226)      at java.base/java.io.Writer.write(Writer.java:249)      at com.google.devtools.build.lib.util.SimpleLogHandler$Output.write(SimpleLogHandler.java:765)  at com.google.devtools.build.lib.util.SimpleLogHandler.publish(SimpleLogHandler.java:434)       at java.logging/java.util.logging.Logger.log(Logger.java:979>
        File "/build/output/external/rules_python/python/repositories.bzl", line 175, column 34, in _python_repository_impl
                rctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error extracting /build/output/external/python_x86_64-unknown-linux-gnu/temp12096015298234464535/cpython-3.11.6+20231002-x86_64-unknown-linux-gnu-install_only.tar.gz to /build/output/external/python_x86_64-unknown-linux-gnu/temp12096015298234464535: write (No space left on device)
Loading:
        at com.google.common.io.CountingOutputStream.write(CountingOutputStream.java:52)        at java.base/sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:233)        at java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:303) at java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:281) at java.base/sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)     at java.base/sun.nio.cs.StreamEncoder.write(StreamEncoder.java:135)     at java.base/java.io.OutputStreamWriter.write(OutputStreamWriter.java:226)      at java.base/java.io.Writer.write(Writer.java:249)      at com.google.devtools.build.lib.util.SimpleLogHandler$Output.write(SimpleLogHandler.java:765)  at com.google.devtools.build.lib.util.SimpleLogHandler.publish(SimpleLogHandler.java:434)       at java.logging/java.util.logging.Logger.log(Logger.java:979>

The diff looks non-trivial: https://github.com/google/jax/compare/jaxlib-v0.4.29...jaxlib-v0.4.30
jax seems to now depend directly on jaxlib: https://jax.readthedocs.io/en/latest/changelog.html#jax-0-4-30-june-18-2024

However, it crashes in a different way:

It looks like it has just run out of disk space, eg write (No space left on device)

It looks like it has just run out of disk space, eg write (No space left on device)

I'm afraid that there is a real issue unfortunately:
Analyzing: 2 targets (45 packages loaded, 225 targets configured)
INFO: Repository pypi_build instantiated at:
  /build/source/WORKSPACE:30:13: in <toplevel>
  /build/output/external/pypi/requirements.bzl:49:20: in install_deps
Repository rule whl_library defined at:
  /build/output/external/rules_python/python/pip_install/pip_repository.bzl:697:30: in <toplevel>
ERROR: An error occurred during the fetch of repository 'pypi_build':
   Traceback (most recent call last):
        File "/build/output/external/rules_python/python/pip_install/pip_repository.bzl", line 596, column 13, in _whl_library_impl
                fail("whl_library %s failed: %s (%s) error code: '%s'" % (rctx.attr.name, result.stdout, result.stderr, result.return_code))
Error in fail: whl_library pypi_build failed:  (src/main/tools/process-wrapper-legacy.cc:80: "execvp(/build/output/external/python_x86_64-unknown-linux-gnu/bin/python3, ...)": No su>
) error code: '1'
ERROR: /build/source/WORKSPACE:30:13: fetching whl_library rule //external:pypi_build: Traceback (most recent call last):
        File "/build/output/external/rules_python/python/pip_install/pip_repository.bzl", line 596, column 13, in _whl_library_impl
                fail("whl_library %s failed: %s (%s) error code: '%s'" % (rctx.attr.name, result.stdout, result.stderr, result.return_code))
Error in fail: whl_library pypi_build failed:  (src/main/tools/process-wrapper-legacy.cc:80: "execvp(/build/output/external/python_x86_64-unknown-linux-gnu/bin/python3, ...)": No su>
) error code: '1'
ERROR: /build/source/jaxlib/tools/BUILD.bazel:25:10: //jaxlib/tools:build_wheel depends on @pypi_build//:pkg in repository @pypi_build which failed to fetch. no such package '@pypi_>
) error code: '1'
ERROR: Analysis of target '//jaxlib/tools:build_wheel' failed; build aborted: Analysis failed
INFO: Elapsed time: 39.451s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (45 packages loaded, 225 targets configured)

what's the status of this?

what's the status of this?

I'm stuck on this. We need to revisit the jaxlib derivation regarding the breaking changes introduced in this release.

From a quick glance, I think jaxlib and tensorflow broke with this change: https://github.com/google/jax/commit/517e299a9d41acc13b41252e52c5884ea3d1cfab. I think this forces bazel to download a "hermetic" python which is basically some prebuilt binary of questionable origin. Looking at the build logs, it also chooses the python release at random unless you export HERMETIC_PYTHON_VERSION. We want this feature off, random executables from the internet is not the kind of reproducibility Nixpkgs is after...
@GaetanLepage can you try fetchpatch { ...; revert = true; }?

Hello @SomeoneSerge,
We are sorry to hear that hermetic Python setup breaks your workflow . I would like to learn more about it, specifically if it is a technical breakage or more in the realm of depending on downloaded python being against your established policies.
Either way, your issues could be solved without non-hermetic behavior (please check my response in jax/issues/22216 for details).
Let me to address a few of the raised concerns directly in this thread

random executables from the internet

The downloaded Python interpreters are not of questionable origin and are not random at all. The actual packages to download, together with their SHA256 sums are specified directly in rules_python, which is an officially supported product. If you trust bazel to build your project (which must be the case here) you should trust rules_python as one of its core components as well. The actual python binaries used are based on https://github.com/indygreg/python-build-standalone, if that does not satisfy your standards, you may always provide your own python interpreter and exactly that one will be used in your build.

is not the kind of reproducibility Nixpkgs is after

I would like to know more about how reproducibility is defined in Nixpkgs, but either way one of hermetic Python's major goals is to actually increase reproducibility. Hermetic Python locks all of the key dependencies together with their SHA256 sums, so entire Python ecosystem in the build is fully locked and remains the same regardless of the surrounding environment and depends only on the corresponnding git commit  which updates/changes those dependencies directly in the source tree.
Downloading anything from the internet in general.
One way or another any bazel build always downloads lost and lots of dependencies form the internet every time. It is a cornerstone of bazel's design, and you should use bazel only if that is acceptable for your case. Python was just one of the few inconsistent with that paradigm parts of the dependency resolution mechanism which we are finally cleaning up here.

Hello @SomeoneSerge,
We are sorry to hear that hermetic Python setup breaks your workflow . I would like to learn more about it, specifically if it is a technical breakage or more in the realm of depending on downloaded python being against your established policies.
Either way, your issues could be solved without non-hermetic behavior (please check my response in jax/issues/22216 for details).
Let me to address a few of the raised concerns directly in this thread

random executables from the internet

The downloaded Python interpreters are not of questionable origin and are not random at all. The actual packages to download, together with their SHA256 sums are specified directly in rules_python, which is an officially supported product. If you trust bazel to build your project (which must be the case here) you should trust rules_python as one of its core components as well. The actual python binaries used are based on https://github.com/indygreg/python-build-standalone, if that does not satisfy your standards, you may always provide your own python interpreter and exactly that one will be used in your build.

is not the kind of reproducibility Nixpkgs is after

I would like to know more about how reproducibility is defined in Nixpkgs, but either way one of hermetic Python's major goals is to actually increase reproducibility. Hermetic Python locks all of the key dependencies together with their SHA256 sums, so entire Python ecosystem in the build is fully locked and remains the same regardless of the surrounding environment and depends only on the corresponnding git commit which updates/changes those dependencies directly in the source tree.
Downloading anything from the internet in general. One way or another any bazel build always downloads lost and lots of dependencies form the internet every time. Those dependencies are always specified with corresponding SHA256 sums, which should reduce the probability of any malicious attack. The fact of downloading the same thing every time regardless of the actual environment is explicitly designed to ensure reprehensibility of every build, not to make it worse. It is a cornerstone of bazel's design, and you should use bazel only if that is acceptable for your case. Python was just one of the few inconsistent with that paradigm parts of the dependency resolution mechanism which we are finally cleaning up here.

Thank you @vam-google for this very detailed answer.
Indeed, it is quite challenging (well impossible) for nix derivation to implicitly fetch some resources from the internet at build time. Nix is adopting a similar strategy as what you described regarding rules_python. The issue here is that we can't satisfy both build systems simultaneously.
I think that the most sensible option for us is to provide bazel with our own python interpreter and dependencies.
The goal for us is now to find a way around those restrictions in order to successfully build jaxlib from source.
The answer to @SomeoneSerge's issue looks like a good starting point !

Thanks @vam-google for providing this info!

I would like to know more about how reproducibility is defined in Nixpkgs, but either way one of hermetic Python's major goals is to actually increase reproducibility.

Nix has two types of build rules, called "derivations" in Nix parlance:

Normal derivations, the default: All inputs are pinned/fixed according to their hashes, and the build script itself is pinned according to its hash. The build environment restricts certain kinds of naughty behavior -- eg network access -- to promote stability in the final result. Example: most Nix package builds.
Fixed-output derivations (FODs): These derivations are based on a pinned build script, but are executed in an unrestricted environment, so eg network access is fair game. Buuuuut the hash of the output is specified at compile time and verified. Example: fetchFromGitHub.

Our Bazel rule, buildBazelPackage, works by splitting the build into two Nix derivations: a fixed-output derivation for bazel fetch, and a normal derivation for bazel build.
Btw I will shamelessly take this opportunity to draw your attention to https://github.com/NixOS/nixpkgs/issues/321920 in which we have found that the output of the bazel fetch step is actually not reproducible. Different users are seeing different hashes (https://github.com/NixOS/nixpkgs/issues/296737). I believe that we have fixed this with https://github.com/NixOS/nixpkgs/pull/323681 (s/o to @SomeoneSerge!), but I'd be curious to hear your thoughts on this since it seems to violate Bazel's claims to reproducibility. Perhaps we're missing something?

Thanks and welcome @vam-google! I'll try to be brief (EDIT: I failed) and not parrot what has already been said.
Regarding "our workflow", my response^1 in the jax repo touches on the subject a bit. What I didn't mention is parallel goals such as customization, inspectability, etc. There's some benefits we get from being able to make all of our software use exactly the same python interpreter arbitrarily customized at the user site. <HERE WAS A CHUNK OF TEXT REPEATING SAMUEL'S POST IN DIFFERENT WORDS>

The removed piece:

As Gaetan mentions, Bazel and Nix take on some of the same responsibilities. As Samuel says, Nix only really implements two concepts: an immutable "store", and the singular way to extend the store, using programs that convert a pre-defined set of inputs into a pre-defined set of outputs. These are run in isolation to ensure that no other inputs can be read and no other outputs can be written; this all of course must sound very familiar and resembling "Bazel ignoring <...> the actual environment". Maybe to emphasize the differences instead, neither Nix nor Nixpkgs have anything like cc_library or cc_executable: at this level of detail we just delegate to tools like meson and cmake, and rely on the sandbox. In this sense Nix is a subset of Bazel, but this is often a positive: e.g. a project that uses Nix as a development tool will likely use cmake/meson exactly the same way it would without Nix, and so is easier to consume in other environments.

(end of the removed piece)

to know ...if it is a technical breakage or more in the realm of ... policies

I'd say it's both. I highlighted some of the technical aspects in the other response^1, although I think that was just the first iteration. As for the policies, this has to do with your other question:

...more in the realm of ... policies? <...> ...how reproducibility is defined in Nixpkgs?

Some of our "derivations" are actually non-deterministic (parallelism, scheduling, etc) so it's more common that we'd talk about "repeatability"... but to really answer your question, I'd rather think in terms of "complexity" instead. In particular, the cost we must pay now to be able to bootstrap a piece of today's software from scratch, say, ten years later. In the first approximation (let's say, compatible hardware and such belong in the 0th), our main threat are these "FODs" that Samuel has mentioned, for two reasons: many of them are the roots of our dependency DAG (and all roots are FODs), and they also are impure (allowed network). Thus at the very least, one had better archived all of the FODs. Thus, we want fewer and smaller FODs. From this point of view, having a piece of source code as a root is preferable (less "complex") to having a collection of large non-inspectable platform-specific binaries. Our ideal would be a full-source bootstrap similar to what Guix does^2, but there are tradeoffs involved and we currently make different choices.
It was also in this limited sense that I called the downloaded binaries "questionable [in] origin" and "random"

Head's up: We're migrating to numpy 2.1 in https://github.com/NixOS/nixpkgs/pull/354627 and jax 0.4.28 is failing a number of tests on 3.12 and 3.13.
FAILED tests/lax_numpy_test.py::LaxBackedNumpyTests::testUnique1 - AssertionError: Tuples differ: (3,) != (3, 1)
FAILED tests/lax_numpy_test.py::LaxBackedNumpyTests::testUnique2 - AssertionError: Tuples differ: (2,) != (2, 1, 1)
FAILED tests/lax_numpy_test.py::LaxBackedNumpyTests::testUnique4 - AssertionError: Tuples differ: (3,) != (3, 1)
FAILED tests/lax_numpy_test.py::LaxBackedNumpyTests::testUniqueSize0 - AssertionError: Tuples differ: (4,) != (1, 4)
FAILED tests/lax_numpy_test.py::LaxBackedNumpyTests::testUniqueSize4 - AssertionError: Tuples differ: (4,) != (1, 4)
FAILED tests/lax_numpy_test.py::LaxBackedNumpyTests::testUniqueSize5 - AssertionError: Tuples differ: (1,) != (1, 1)
FAILED tests/lax_numpy_test.py::LaxBackedNumpyTests::testUniqueSize6 - AssertionError: Tuples differ: (3,) != (3, 1)
FAILED tests/lax_numpy_test.py::LaxBackedNumpyTests::testUniqueSize8 - AssertionError: Tuples differ: (4,) != (1, 1, 4)
FAILED tests/lax_numpy_test.py::LaxBackedNumpyTests::testUniqueSize9 - AssertionError: Tuples differ: (3,) != (3, 1)
FAILED tests/lax_numpy_test.py::NumpySignaturesTest::testWrappedSignaturesMatch - AssertionError: {'clip': {'np_params': ['a', 'a_min', 'a_max', 'min', 'max'...
FAILED tests/lax_numpy_reducers_test.py::JaxNumpyReducerTests::testCumulativeSum0 - AssertionError:


use Gradle for smoother contributor onboarding

We need a well known build process. Manual library hunting was fine for a lone developer but now people want to contribute and them being able to test changes is essential.

a basic dependency list works and example sketches from the repo can be run,
the next step is successfully building a jar from it that is the same as seen in the latest release

jar build works, initial setup tested on another pc and improved with adding java version=1.8 and utf-8 into build.gradle


k8s.openebscrd: Add CRD functionality for StoragePool

Signed-off-by: Sayan Chowdhury<EMAIL_ADDRESS>This commit adds List, Delete, and Read Functionality for the StoragePool.

@sayanchowdhury
Looks good to me in general.
Just a minor enhancement:
Add an opts argument in list get and delete functions. Following is a reference
https://github.com/openebs/CITF/pull/55/files

@sayanchowdhury the above reference provided by @sonasingh46 is good but has one mistake.
He is using something like opts *meta_v1.*, you should not do like that.
You should use it like opts meta_v1.*
For example:-
func (k8s K8S) GetStoragePool(storagePoolName string, opts meta_v1.GetOptions)

and if you want you can add the opts in all the 6 methods that are already present in this file.


alreadyExists returns the wrong code

As per @paulb777, here is the PR based on the issue I found: https://github.com/firebase/firebase-ios-sdk/issues/9942
Step 1: Describe your environment
Xcode version: 13.4.1
Firebase SDK version: 9.3.0
Installation method: CocoaPods
Firebase Component: Functions
Target platform(s): iOS
Step 2: Describe the problem
Functions that throw already-exists generate the right HTTP response (seen in Postman as HTTP 409 Conflict). However, in the iOS client bubbles the error to the client as notFound. The commit mentioned below shows the affected areas and includes a fix.
Steps to reproduce:
Simple. Just throw the following from a Cloud Function:
throw new functions.https.HttpsError('already-exists', 'Resource exists')
If you have a downloadable sample project that reproduces the bug you're reporting, you will
likely receive a faster response on your issue.
Relevant Code:
https://github.com/firebase/firebase-ios-sdk/commit/3e758246f6c25001cb9f36ac9b429101a59219c7
Regards,
-- Tito

Submitted. Thanks Paul!

Thanks Tito! Merged

Hey @tciuro I'm working on Google Open Source Peer Bonus nominations and would like to recognize your contribution to improving Firebase Functions' code quality. I couldn't find an email address for you, so if you're interested, please email it to chliang at google.com and I'll make the nomination.

Email sent. Thank you!


Setup CI

This PR closes #43 by making the following changes

move DSL tests from bin/main.ml to test/well_formed.ml
set up CI according to issue #43 and @anshumanmohan's comment below
fill in dune-project

I've set up a formatting check for the CI: now, commits for which dune fmt is not a no-op will not pass CI.
Before we hit the big green button, would you mind confirming if this is the desired behavior?

Yes looks good to me, go ahead and merge when ready :)


Fix: email link on sign in page

Background:
https://trello.com/c/QjcqfFL6/271-fix-email-link-on-sign-in-page
Solution:

The value for the support email is provided by the Rails configuration and exposed to the Application controller
Layout for the page is fixed, value is rendered on the server

 Fix email link on sign in page


[FEATURE]

It would be very useful / easier, if you would do the commands with prefixes.
Like that:
/g <command>

Hello,
from my point of view, it would be too much effort for a minimal difference. The space in between does not make a big difference in time, usage or efficiency.
Best regards


Allow automatic Minor updates

Reverse https://github.com/mapbox/mapbox-tile-copy/commit/bbfab1a43b62675e6614e81d871728dd795590b4#diff-b9cfc7f2cdf78a7f4b91a753d10865a2R32
Also solves https://github.com/mapbox/mapbox-tile-copy/pull/44/files
cc @yhahn @jakepruitt

:+1:

Deployed.


Add Dynamic Topic Discovery module [14000]

Merge after https://github.com/eProsima/DDS-Router/pull/130

#154

Add communication test for cases without builtin-topics


With my SDD cache hot (i.e. after a few runs) the time-to-photon for a 650MB video is ~350ms, which is acceptable.
Interestingly enough, "concatenating" a SINGLE arrow array (with nothing) can take 90ms.

Checklist

[x] I have read and agree to Contributor Guide and the Code of Conduct
[x] I've included a screenshot or gif (if applicable)

Interestingly enough, "concatenating" a SINGLE arrow array (with nothing) can take 90ms.

The 90ms part doesn't shock me since I assume it's just copying 650MB of data all over again -- but once again the root of the issue seems to bet yet another arrow kernel that is missing some obvious early outs, a concatenation of 1 should be a noop...
We should have our own kernel wrapper that check for these cases, like we do for filter() and `take().


Gh 1872 create GetVariables op

Codecov Report

:exclamation: No coverage uploaded for pull request base (gh-1860-Variables-Operations@5abd317). Click here to learn what that means.
The diff coverage is n/a.

@@                       Coverage Diff                       @@
##             gh-1860-Variables-Operations    #1875   +/-   ##
===============================================================
  Coverage                                ?   75.63%
  Complexity                              ?      342
===============================================================
  Files                                   ?       61
  Lines                                   ?      981
  Branches                                ?       58
===============================================================
  Hits                                    ?      742
  Misses                                  ?      192
  Partials                                ?       47

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5abd317...279d336. Read the comment docs.

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.3 out of 4 committers have signed the CLA.:white_check_mark: m55624:white_check_mark: d47853:white_check_mark: p013570:x: Travis CI UserTravis CI User seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.


Mention embedded-hal dependency in README

Hi, @barafael! First of all, I'd like to thank you for open-sourcing your work. It looks exactly like what I need for an upcoming project. In the next couple of weeks I'll test this crate with the E32-900T20D, and if all works fine you can add to the list of tested models :)
While reading the description, it wasn't clear to me how your lib interfaces with the MCU code, until I checked the source code/Cargo.toml and saw you use embedded-hal.
IMHO mentioning the usage of embedded-hal in the README (like you did in the ebyte-e32-ui repo) is important and would make it easier for potential users to evaluate how your library can be integrated with their Rust embedded projects, so this is my small two cents on it so far :)
Kind regards

Hi Gustavo, that's awesome! Let me know how it goes and if there's any way I can help with testing.
I'll be adding the docs like you said.

Just confirmed that this crate works fine with the E32-900T20D!
Here is a working example with an ESP32 MCU, in case you want to mention in the docs :)


intall external Unlocked : multiple package with same begining

Describe the bug
I have to external dependency:

"Nebula Logger Plugin -<EMAIL_ADDRESS>"04t5Y0000015nDJQAY"
"Nebula<EMAIL_ADDRESS>"04t5Y0000015nDxQAI"

only "Nebula<EMAIL_ADDRESS>"04t5Y0000015nDxQAI" is installed on org.
When I install a release with the following definition
...
packageDependencies:
    Nebula Logger Plugin -<EMAIL_ADDRESS>04t5Y0000015nDJQAY
    Nebula<EMAIL_ADDRESS>04t5Y0000015nDxQAI
...

package installation is skipped with message
  Checking whether package  Nebula Logger Plugin -<EMAIL_ADDRESS>with ID 04t5Y0000015nDJQAYis installed in<EMAIL_ADDRESS>  Package to be installed was found in the target org<EMAIL_ADDRESS>  Skipping Installing of package Nebula Logger Plugin -<EMAIL_ADDRESS>in<EMAIL_ADDRESS>
but the package is not present in the org. Ending with a non installed package.
To Reproduce
Steps to reproduce the behavior:

install 04t5Y0000015nDxQAI to an empty org
release the file

release: CRM_1.4.1
skipIfAlreadyInstalled: true
artifacts:
packageDependencies:
    Nebula Logger Plugin -<EMAIL_ADDRESS>04t5Y0000015nDJQAY
    Nebula<EMAIL_ADDRESS>04t5Y0000015nDxQAI

Expected behavior
Nebula<EMAIL_ADDRESS>04t5Y0000015nDxQAI should be skipped & Nebula Logger Plugin -<EMAIL_ADDRESS>04t5Y0000015nDJQAY installed.
Additional context
After looking at the code, it seems that #1245 added this behaviour
❯_ ❯  node
Welcome to Node.js v18.14.2.
Type ".help" for more information.
> a = '04t5Y0000015nDxQAI'
'04t5Y0000015nDxQAI'
> b = '04t5Y0000015nDJQAY'
'04t5Y0000015nDJQAY'
> a.substring(0,14) == b.substring(0,14)
true

I think we should take one more character to fix this. Thanks @alanjaouen

@azlam-abdulsalam I can raise PR if it can help you, just lmk


Document _DEP_DIR dependency overriding

[x] This contribution adheres to CONTRIBUTING.md.

What does this Pull Request accomplish?
Document using _DEP_DIR to override a dependency.
Why should this Pull Request be merged?
Using _DEP_DIR is needed for pull request builds when adding module support to https://github.com/ni/niveristand-scan-engine-ethercat-custom-device.
What testing has been done?
N/A

FYI @dbendele


Fix heatmap perf part 2

Description
This PR reduces the time to render the heatmap by about 40% on top of previous gains. In my test of a 49 sample project, time for the samples_taxons endpoint went down from 14s to 8s consistently.
The improvement was achieved mostly by ranking and filtering in SQL instead of ruby. See top_n_query.
Next, about 2 seconds was saved by converting two N+1 queries into a subquery to get the most recent pipeline run.
Thirdly, the bottleneck SQL query was filtered more with MINIMUM_ZSCORE_THRESHOLD and MINIMUM_READ_THRESHOLD.
In the process, I refactored HeatmapHelper and ReportHelper a bunch. I moved all methods used only by HeatmapHelper into that class. There is still a lot of duplication of code in the two classes. I will leave that for another day.
Other changes:

I added a suite of unit tests to cover the public methods of ReportHelper. The tests cover different permutations of inputs but not all edge cases.
I removed the aggregatescore from the heatmap UI and code on recommendation by @cdebourcy .

Notes

failed pipeline runs: The original logic which I reproduced in SQL is to use the most recent run regardless of status. This doesn't seem right to me.
missing pipeline runs: I literally spent 3 hours debugging an issue which turned out to be caused by taxon counts referencing pipeline runs that did not exist. There appears to be 87 of such counts in prod. See https://modeanalytics.com/editor/chanzuckerberg/reports/46d9ac9db658 . It would be nice to enforce data integrity with either database constraints or garbage collecting scripts to avoid such head-scratching situations.
I duplicated definitions of user facing metrics such as zscore for performance reasons. This could easily lead to inconsistencies if not now then later which may be spotted by users. That said, there were already several duplicate definitions in the reports codebase, which is more troubling.
 For example, phages are excluded by default without any indication in the UI.
It seems like the aggregate score is no longer loved. If so we should kill it in the report page as well.

Test
rails t to cover basic param functionality
To cover metric definitions:
print the values of zscore and rpm, computed in ruby and in SQL
see the values are the same
To cover correct top N behavior:
Load a large heatmap in dev branch
Load a large heatmap in master branch
See that the values on hover are the same
See that the grid of colors is very similar, differing only in equally ranked samples or taxons
Try all different metrics, different taxons per sample
To cover metric definitions:
Load a large heatmap in dev branch
print the values of zscore and rpm, computed in ruby and in SQL
see the values are the same

@cdebourcy @tfrcarvalho
I spent another 1.5 hours today trying to convert the custom threshold filters and all the remaining ruby filtering to SQL. However, I gave up because it was too hard to get exactly the same results in all cases. The key difference is that the ruby filters are applied at the level of taxons but the inner SQL query is applied on counts.
The good news is that custom filtering is a small percent of queries and we could easily route to the old code path, which I implemented in the lastest commit, so we get nearly all the performance benefit at the cost of code complexity.
Before working any further on heatmaps, I believe we need to refactor the code a lot more and write tests for each part, assuming we want to maintain the current behavior exactly.

I am ok with moving on with this change. I understand some refactors are too long for the current scope.


(local) bazel caching is using incorrect instance_name for release branches

Currently the release branches are resolving the instance_name as branch/release which is most likely causing cache poison issues, build fails, and slower ci times

seems only to be happening on 1.26+ and only in prs - 1.27 release pr has branch/main


Howto create minibatches with independent samples

TL;DR: I am looking for a way to mux streams and aggregate them in batches while making sure that each batch contains only 1 sample from each stream (if the batchsize is smaller than the number of streams).
For quite some time now I wonder about an aspect of data sampling in the context of training deep models for hierarchical-like data set. Lets return (#133) to the infamous scenario of music tracks that are trained on smaller excerpts instead of using the whole track so that non-dynamic models such as CNNs can be used.
So lets assume we have 10 tracks and each track yields 100 excerpts (with our without overlap), this would result in 1000 samples. Now we have a few ways to sample for one training epoch, and one way is to sample randomly and select batchsize samples for each batch and repeat the procedure (without replacement) for a fixed number of iterations in each epoch. While this is fast and easy to implement – even without pescador – it comes with two downsides:

it is likely that we are not going to see all data in one epoch.

This can be compensated by increasing the number of iterations or epochs. Also there is a proof that with-replacement sampling performs worse than without-replacement sampling, but this is only valid for non-convex problems. I forgot where I read it but for DNNs it could actually result in improved generalisation, which probably why it turned out to be successfully deployed in audio for such as singing voice detection paper from @f0k.

we end up having multiple excerpts from the same track in one batch

I currently don't know if this has an impact on performance, but there is some work that deals with these kind of problem for small-batch sizes and/or non-i.i.d samples. Also I know some methods in source separation use this kind of sampling (e.g. the ICASSP 2015 work from @TE-StefanUhlich). Therefore I want to run a small comparison and I hoped I could implement all  flavors using pescador.
So, with pescador we can easily make sure that all samples are seen once using StochasticMux. However if we use the buffer_stream we cannot easily make sure that all samples are from different tracks. A workaround would be to use RoundRobin if batchsize is smaller than the number of tracks and shuffle the streamers every epoch:
for epoch in range(k):
    shuffle(streams)
    mux = pescador.RoundRobinMux(track_streams, mode='exhaustive')
    batches = pescador.buffer_stream(mux, nb_batches)
    for batch in batches:
        ...

however than we would end up having repetition (e.g. for three tracks and batchsize 2: [[a, b], [c, a], [b, c], [a, b]]).
I would be glad if someone could show me how to implement this in pescador or can possibly point me to some theory papers were this is discussed. Also maybe this would something to consider for the pescador paper you are planning to write.

TL;DR: I am looking for a way to mux streams and aggregate them in batches while making sure that each batch contains only 1 sample from each stream (if the batchsize is smaller than the number of streams).

If batch size == k * # streams (for integer k), then RoundRobin will do the job.  Indeed, that's what we put it in for. :)
If batch size < # streams, then RoundRobin will guarantee at most 1 sample per stream per batch, but as you observe, you'll get interactions between batches that depends on the ratio between the two numbers, and the order of the streamers within the mux.
The epoch-shuffling idea you posted is interesting, and similar to something the RR already does.  A slightly more pescadorian approach to this would be to limit each streamer to produce a small number of samples (== the number before the order will change) and self-terminate, then use mode='permuted_cycle option of RoundRobinMux to re-order the streamers before reactivating.  This way you don't have to wait for a full epoch before the order changes.
If each streamer generates exactly 1 random example, this should give you exactly the desired behavior of fully randomized order with balanced presentation.  However, you'll have to pay in overhead of reactivating the streamers, so it might not be worth it.  My untested, gut feeling, is to set the per-streamer limit such that the permutation order changes several times per epoch.  That "several times" may depend on how many total streamers you have: if it's small, then epoch/n_streams! might be feasible; if it's large, then it probably doesn't matter too much because you'll never observe every order anyway.

Also there is a proof that with-replacement sampling performs worse than without-replacement sampling, but this is only valid for non-convex problems.

Side note: are you referring to section 4.2 of that paper?  Or a different part?  I'm not sure it applies in either case, but maybe I missed something in my cursory skim.

Side note: are you referring to section 4.2 of that paper? Or a different part? I'm not sure it applies in either case, but maybe I missed something in my cursory skim.

I got it from the conclusion 😆  I'll be honest here, I don't understand any of the proof and I will probably not have the time to look into it. But normally when this paper is cited, they will tell you that this only applies to convex problems.

The epoch-shuffling idea you posted is interesting, and similar to something the RR already does. A slightly more pescadorian approach to this would be to limit each streamer to produce a small number of samples (== the number before the order will change) and self-terminate, then use mode='permuted_cycle option of RoundRobinMux to re-order the streamers before reactivating. This way you don't have to wait for a full epoch before the order changes.

Excellent, thanks. This is the hack I was looking for. If this would be of greater interest, I would propose to add an option to RR to make this easy to use.

@bmcfee if you're interested how the RoundRobin flavors perform for source separation compared to StochasticMux, I can post the results here. Otherwise, feel free to close this out.

If this would be of greater interest, I would propose to add an option to RR to make this easy to use.

I think it is of interest, but I'm not sure adding it to RR is the best way to approach it.  I think we all want to keep the logic as simple as possible, and adding top-down limiting control to RR (like in poissonmux) sounds like a can of worms to me.
However, if you find that it does the job for you, it'd be great to add it as a use-case/example in the documentation.

if you're interested how the RoundRobin flavors perform for source separation compared to StochasticMux, I can post the results here

@faroit: I'd be interested how much of a difference the sampling strategy makes. I used to sample excerpts without replacement (https://github.com/f0k/ismir2015/blob/master/experiments/augment.py#L18-L51), but now often sample recordings without replacement (and then uniformly sample an excerpt within each chosen recording). I wasn't worried too much that a batch could contain multiple excerpts of the same recording, but a) that for datasets of wildly varying recording lengths, longer recordings would be favored over shorter ones, and b) that for large datasets, keeping track of all possible recording/position tuples adds a lot of overhead.

now often sample recordings without replacement (and then uniformly sample an excerpt within each chosen recording).

This is typically how I do it as well, using a pumpp sampler for each track, and then a StochasticMux over tracks.

a) that for datasets of wildly varying recording lengths, longer recordings would be favored over shorter ones,

This is true, but you can also either use rate-limiting (in the mux) or self-limiting (in the streamer's generator) to avoid this.  If you use single_active mode in the mux, then you still get without-replacement behavior in the short term, but the individual streamer can be reactivated at any time later.  This should prevent long tracks from being over-represented, but still allow you to cover the data well.

b) that for large datasets, keeping track of all possible recording/position tuples adds a lot of overhead.

This is true.  In general, we don't have a great mechanism for persisting state across activations of a streamer.  This is somewhat by design, as you mention, it could incur substantial memory overhead.  So far, it hasn't really been a problem, but I'm open to suggestions about how to generically support such a thing if it does become desirable.

@f0k @bmcfee just short status update: I didn't forget this issue. I still think this discussion could be really valuable. Also – specifically for source separation networks – we will present this as part of the ISMIR source tutorial. I will share the code and results later here when we fixed the remaining parameters.

Pinging back on this thread -- @faroit did you ever do the comparison of round-robin and shuffle mux?

did you ever do the comparison of round-robin and shuffle mux?

we did. In the plot above, the track sampling without replacement was done with round-robin.
For our music separation the difference (in validation loss) turned out to be negligible.
In the meantime we upgraded code to stick with the default pytorch dataloaders for performance issues. I will have look at pescador again as mentioned in #144.

Great, thanks!  I'm not 100% sure I understand the axes of this plot though, can you break it down a bit more?  Is the message that replacement for tracks doesn't matter, but it does for samples drawn per batch from the track set?
Side note: I'm currently doing a bit of issue gardening, and trying to clean up some rough edges before we start hammering on speed.

Great, thanks! I'm not 100% sure I understand the axes of this plot though, can you break it down a bit more? Is the message that replacement for tracks doesn't matter, but it does for samples drawn per batch from the track set?

sorry, of course: the message is that taking samples (here samples = chunks/excerpts within a track) without replacement improves performance as it guarantees that the model sees all parts of each tracks (when no overlap) exactly once, thus utilizing all the data. This is not concerning the batch but the epoch!
In contrast, tracks with/without replacement is concerning the batch as mentioned above. Does this make sense?

Did excerpts overlap in your experiment, or were they non-overlapping? (It doesn't really matter, but I'm curious.)

I think I always used non-overlapping segments. Here is the code I used back then

Great, thanks!  I suspect there's something really interesting going on here relating to the conditional independence assumptions of excerpts within a track (definitely dependent) vs across tracks (hopefully independent).


Adapt SDK for refactored schema

Changes in Schema refactor #9001 could affect on python sdk functionality
To take attention:

models contents also versions of models
models_version doesn't exist

This item

change list function queries from 'select * from ' to 'show '
should be done after:

Created tests for python sdk in mindsdb (which call not mocked mindsdb api)
removing Select * from <obj> will be planned in near future


AvaloniaProperty default value factory

Is your feature request related to a problem? Please describe.
It should be possible to set default value factory.
Describe the solution you'd like
public static StyledProperty<TValue> Register<TOwner, TValue>(string name, Func<TValue> defaultValueFactory, bool inherits = false, BindingMode defaultBindingMode = BindingMode.OneWay, Func<TValue, bool> validate = null, Func<IAvaloniaObject, TValue, TValue> coerce = null, Action<IAvaloniaObject, bool> notifying = null) where TOwner : IAvaloniaObject;
public static AttachedProperty<TValue> RegisterAttached<TOwner, THost, TValue>(string name, Func<TValue> defaultValueFactory, bool inherits = false, BindingMode defaultBindingMode = BindingMode.OneWay, Func<TValue, bool> validate = null, Func<IAvaloniaObject, TValue, TValue> coerce = null) where THost : IAvaloniaObject;
public static AttachedProperty<TValue> RegisterAttached<THost, TValue>(string name, Type ownerType, Func<TValue> defaultValueFactory, bool inherits = false, BindingMode defaultBindingMode = BindingMode.OneWay, Func<TValue, bool> validate = null, Func<IAvaloniaObject, TValue, TValue> coerce = null) where THost : IAvaloniaObject;

Additional context
WPF doesn't have this feature.
But UWP does:
CreateDefaultValueCallback Delegate
Property = DependencyProperty.Register("MyProperty",
    propertyType, ownerType,
    PropertyMetadata.Create(createDefaultValue));

Possible use cases

Storing mutable objects inside of property, and default value should be new for each avalonia object having it - https://devblogs.microsoft.com/oldnewthing/20191002-00/?p=102950 (actually it's a pretty BAD idea, and this topic describes why).
When default value is read from global settings class or any other static source that can be mutated, and default value should be recalculated.

BTW, right now developer can use DIrectProperty as a workaround, if it works for him (keeping in mind limitations of Direct Property).

BTW, right now developer can use DIrectProperty as a workaround, if it works for him (keeping in mind limitations of Direct Property).

The UWP implementation has a serious problem: https://devblogs.microsoft.com/oldnewthing/20191002-00/?p=102950
It may be possible for Avalonia to avoid that problem, but it's a bit tricky.

The UWP implementation has a serious problem: https://devblogs.microsoft.com/oldnewthing/20191002-00/?p=102950
It may be possible for Avalonia to avoid that problem, but it's a bit tricky.

When default value is read from global settings class or any other static source that can be mutated, and default value should be recalculated.

Could you explain this some more? I don't think I've ever encountered this use-case.

When default value is read from global settings class or any other static source that can be mutated, and default value should be recalculated.

Could you explain this some more? I don't think I've ever encountered this use-case.

Could you explain this some more? I don't think I've ever encountered this use-case.

It's pretty non-MVVM and non-Binding friendly design, but it's possible to have some static class\storage, that is intended by developer to provide default values for some View dependency properties. And moreover, that can be modified in runtime multiple times.
public static class Settings
{
    public static Color DefaultBackgroundColor { get; set; }
}

And you can't use any existing avalonia property overload, because it will capture only initial value at moment of property creation while one might expect it to has value of moment, when GetValue is called and there is no any other value (as it works in UWP).
But if one ports code like that from UWP, it is possible to use DirectProperty without huge changes. And it is possible to rewrite it  to use view model binding.

Could you explain this some more? I don't think I've ever encountered this use-case.

It's pretty non-MVVM and non-Binding friendly design, but it's possible to have some static class\storage, that is intended by developer to provide default values for some View dependency properties. And moreover, that can be modified in runtime multiple times.
public static class Settings
{
    public static Color DefaultBackgroundColor { get; set; }
}

And you can't use any existing avalonia property overload, because it will capture only initial value at moment of property creation while one might expect it to has value of moment, when GetValue is called and there is no any other value (as it works in UWP).
But if one ports code like that from UWP, it is possible to use DirectProperty without huge changes. And it is possible to rewrite it  to use view model binding.

And this feature actually makes more sense in UWP, where dependency objects are still depended on the UI thread, but in same time it's possible to have multiple UI threads. Following quote from the same topic:

The intended purpose of the callback is to let you have a default value that is itself a dependency object. Since dependency objects have thread affinity, you cannot use the same object for all callers, because the callers may come from different threads. The callback lets you look in some per-thread storage to obtain the correct object for that thread. In the worst case, you can have your own private cache of objects indexed by thread.

And this feature actually makes more sense in UWP, where dependency objects are still depended on the UI thread, but in same time it's possible to have multiple UI threads. Following quote from the same topic:

The intended purpose of the callback is to let you have a default value that is itself a dependency object. Since dependency objects have thread affinity, you cannot use the same object for all callers, because the callers may come from different threads. The callback lets you look in some per-thread storage to obtain the correct object for that thread. In the worst case, you can have your own private cache of objects indexed by thread.

The UWP functionality isn't exactly what you think it is. Although, your last quote from the docs is accurate so maybe I misread some things as I quickly skimmed through.
The create default value callback confused me for years myself. Basically, its only to create an initial value that is thread-dependent. The value will only be created once per thread. It's not a factory.
https://github.com/unoplatform/uno/issues/4347#issuecomment-715584107
Anyway, adding a real default value factory would be quite useful.

The UWP functionality isn't exactly what you think it is. Although, your last quote from the docs is accurate so maybe I misread some things as I quickly skimmed through.
The create default value callback confused me for years myself. Basically, its only to create an initial value that is thread-dependent. The value will only be created once per thread. It's not a factory.
https://github.com/unoplatform/uno/issues/4347#issuecomment-715584107
Anyway, adding a real default value factory would be quite useful.

The value will only be created once per thread. It's not a factory.

It might be only a part of real behavior. I quickly created simple application with two scenarios:

One dependency property, many dependency objects using it.
One dependency property, one dependency object using it, but multiple times.

In first case delegate will be called multiple times, so it is technically a factory.
In second case it will be reused, so it's more like "one value per one dependency object per thread" (what doesn't makes sense for me, because we can't share one dependency object between threads).
I haven't tried case with multiple UI threads though, but I believe it should produce different default values as expected.
App7.zip

The value will only be created once per thread. It's not a factory.

It might be only a part of real behavior. I quickly created simple application with two scenarios:

One dependency property, many dependency objects using it.
One dependency property, one dependency object using it, but multiple times.

In first case delegate will be called multiple times, so it is technically a factory.
In second case it will be reused, so it's more like "one value per one dependency object per thread" (what doesn't makes sense for me, because we can't share one dependency object between threads).
I haven't tried case with multiple UI threads though, but I believe it should produce different default values as expected.
App7.zip

I would be willing to bet Microsoft has changed the intended use/functionality of this. Originally, it might have been a full factory created to facilitate instantiating objects used on the UI thread from another thread. Then they did some backtracking and clarified in the documents it should only be used for the very narrow case. That could explain why everyone has different impressions and tests dont always match the docs. It could also be that it never worked correctly so they narrowed scope to the only tested and validated use case.
Reguardless, treating this as a factory in Avalonia would be great. It could be used to instantiate collections for each dependency object (equivalent in Avalonia) without doing so in the constructor which is cleaner IMO.

I would be willing to bet Microsoft has changed the intended use/functionality of this. Originally, it might have been a full factory created to facilitate instantiating objects used on the UI thread from another thread. Then they did some backtracking and clarified in the documents it should only be used for the very narrow case. That could explain why everyone has different impressions and tests dont always match the docs. It could also be that it never worked correctly so they narrowed scope to the only tested and validated use case.
Reguardless, treating this as a factory in Avalonia would be great. It could be used to instantiate collections for each dependency object (equivalent in Avalonia) without doing so in the constructor which is cleaner IMO.


[HUDI-7128] DeleteProcedures support batch mode

Change Logs
DeleteProcedures support batch mode
eg：
if user want to delete 100 or more markers，before the pr need execute sparksql job for 100 times，and just once after the pr，it would reduce much execute time in sparksql.
Impact
none
Risk level (write none, low medium or high below)
low
Documentation Update
Describe any necessary documentation update if there is any new feature, config, or user-facing change

The config description must be updated if new configs are added or the default value of the configs are changed
Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the instruction to make
changes to the website.

Contributor's checklist

[ ] Read through contributor's guide
[ ] Change Logs and Impact were stated clearly
[ ] Adequate tests were added if applicable
[ ] CI passed

@danny0405 @yihua PTAL  CI error seems not related to the changed code


Local variable descriptors are not currently populated in kernel IR

Work in progress: https://codereview.chromium.org/2521353004/

@kmillikin


fix: tuning fine-grained token permissions

No changes in the code, just testing new GitHub fine-grained token configuration

:tada: This PR is included in version 1.0.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Save Option not working after creating new files

Does this issue occur when all extensions are disabled?: Yes/No

VS Code Version:
OS Version:

Steps to Reproduce:

clone a project from git
try adding new files in it, they are always untracked,never changing into modified

/gifPlease

Thanks for reporting this issue! Unfortunately, it's hard for us to understand what issue you're seeing. Please help us out by providing a screen recording showing exactly what isn't working as expected. While we can work with most standard formats, .gif files are preferred as they are displayed inline on GitHub. You may find https://gifcap.dev helpful as a browser-based gif recording tool.
If the issue depends on keyboard input, you can help us by enabling screencast mode for the recording (Developer: Toggle Screencast Mode in the command palette). Lastly, please attach this file via the GitHub web interface as emailed responses will strip files out from the issue.
Happy coding!

This issue has been closed automatically because it needs more information and has not had recent activity. See also our issue reporting guidelines.
Happy Coding!


AST: Overrides for "extra params"

Overview
This PR lays the ground work for being able to replace at "run time" the "extra params" of AST Operation nodes that have any (like the breaks field in Classification).
JSON Samples
The newly extended form of EvalParams, which are embedded in ToolRun. Nothing about ToolRun itself had to change in this PR.
 {
   "sources" : {
     "ae3d5e1b-6333-4b2c-a365-f58b62ca1b16" : {
       "id" : "cd9840de-8806-4dc3-ac78-c9072edf5a02",
       "band" : 5,
       "type" : "scene"
     },
     "a4d64cc5-e3d0-48ac-91ea-5f545fa85ac7" : {
       "id" : "bce1b792-8bb0-4bf1-92a9-ffcf95506201",
       "band" : 5,
       "type" : "scene"
     },
     "e5a58572-4a50-440d-9084-a38e1d2914b1" : {
       "id" : "9e0ae12f-5ff0-46e4-b67c-c57b087ab9f4",
       "band" : 5,
       "type" : "scene"
     }
   },
   "metadata" : {

   },
   "overrides" : {
     "ae3d5e1b-6333-4b2c-a365-f58b62ca1b16" : {
       "constant" : 5.0
     },
     "e5a58572-4a50-440d-9084-a38e1d2914b1" : {
       "classMap" : {
         "classifications" : {
           "0.5" : 3,
           "0.0" : 1,
           "0.75" : 4,
           "1.0" : 5,
           "0.25" : 2
         },
         "options" : {
           "boundaryType" : "lessThanOrEqualTo",
           "ndValue" : -2147483648,
           "fallback" : -2147483648
         }
       }
     }
   }
 }

Closes #1991

+1


Enables changing 'UserAgent'

.Net does not allow modifying several known-headers using customHeaders. So, there's no way to change User-Agent with the current C# implementation unlike other languages can do.

Ma I ask you to also file a JIRA ticket for this?
http://thrift.apache.org/docs/HowToContribute


goroutine leak at cache.go:1079 - runJanitor

package main

import (
	"testing"
	"time"

	"github.com/patrickmn/go-cache"
	"go.uber.org/goleak"
)

func leaktest(t *testing.T) {
	defer goleak.VerifyNone(t)
	cache.New(5*time.Minute, 10*time.Minute)
}

func main() {
	testSuite := []testing.InternalTest{{Name: "leaktest", F: leaktest}}
	testing.Main(func(a, b string) (bool, error) { return a == b, nil }, testSuite, nil, nil)
}

Run above code (https://go.dev/play/p/PxN4Q4-F2rG)
Result:
        [Goroutine 6 in state select, with github.com/patrickmn/go-cache.(*janitor).Run on top of the stack:
        goroutine 6 [select]:
        github.com/patrickmn/go-cache.(*janitor).Run(0xc0002f16d0, 0x0?)
        	/home/wxh/go/pkg/mod/github.com/patrickmn/go-cache@v2.1.0+incompatible/cache.go:1079 +0x85
        created by github.com/patrickmn/go-cache.runJanitor
        	/home/wxh/go/pkg/mod/github.com/patrickmn/go-cache@v2.1.0+incompatible/cache.go:1099 +0xed
        ]

It looks like https://github.com/patrickmn/go-cache/blob/v2.1.0/cache.go#L1076-L1087 does not handle exit condition properly.

it's work
import (
	"runtime"
	"testing"
	"time"

	"github.com/patrickmn/go-cache"
	"go.uber.org/goleak"
)

func leaktest(t *testing.T) {
	defer goleak.VerifyNone(t)
	cache.New(5*time.Minute, 10*time.Minute)
	runtime.GC()
}

func main() {
	testSuite := []testing.InternalTest{{Name: "leaktest", F: leaktest}}
	testing.Main(func(a, b string) (bool, error) { return a == b, nil }, testSuite, nil, nil)
}

It is not memory leaks (or goroutine leak). "go.uber.org/goleak" test does not wait for "garbage collector".

Can you please explain why I am getting the error(cause of the error) and how will adding a routine.GC() after every cache creation helps to resolve the error.

The test counts the number of goroutines at the beginning and at the end. You need to clean up the memory (garbage collection) before terminating the test. Golang typically does this with a delay. Before releasing an object from memory, the Golang calls a finalizer that terminates the goroutine.
Finalizer registration in code: https://github.com/patrickmn/go-cache/blob/master/cache.go#L1123
About finalizer: https://medium.com/a-journey-with-go/go-finalizers-786df8e17687
About garbage collection: https://tip.golang.org/doc/gc-guide


How to Add arduino plotter sketch with another sketch

I want to monitor electrical parameters in real time. I have the following code with emon arduino sketch, I want to plot graph with the data from it usig arduino plotter, How to do that. The following code is below.
`#include <EmonLib.h>
// EmonLibrary examples openenergymonitor.org, Licence GNU GPL V3
#include "EmonLib.h"             // Include Emon Library
EnergyMonitor emon1;             // Create an instance
void setup()
{
Serial.begin(250000);
emon1.voltage(2, 10000, 1.7);  // Voltage: input pin, calibration, phase_shift
emon1.current(1, 500);       // Current: input pin, calibration.
}
void loop()
{
emon1.calcVI(20,2000);         // Calculate all. No.of half wavelengths (crossings), time-out
emon1.serialprint();           // Print out all variables (realpower, apparent power, Vrms, Irms, power factor)
float realPower       = emon1.realPower;        //extract Real Power into variable
float apparentPower   = emon1.apparentPower;    //extract Apparent Power into variable
float powerFActor     = emon1.powerFactor;      //extract Power Factor into Variable
float supplyVoltage   = emon1.Vrms;             //extract Vrms into Variable
float Irms            = emon1.Irms;             //extract Irms into Variable
}`

Check out some of the examples using multiple graphs and multiple variables.
You can't do any printing because the plotter relies on the serial port.

No, I want to use this code in your plotter sketch.Can you link both the sketches to get graph?

Yes, you just need to use the energy monitor values as your graphing variables.

Yes,but am not getting graph

Are you able to run the examples?

Yes, I am able to run your examples but how to integrate with my sketch

Please help me out how to integrate my sketch with your plotter sketch... I will add credits in my project

You can try something along these lines (Note this is untested):
// EmonLibrary examples openenergymonitor.org, Licence GNU GPL V3

#include "EmonLib.h" // Include Emon Library
EnergyMonitor emon1; // Create an instance

#include "Plotter.h"
Plotter p;
float realPower;

void setup()
{
  emon1.voltage(2, 10000, 1.7); // Voltage: input pin, calibration, phase_shift
  emon1.current(1, 500); // Current: input pin, calibration.

  p = Plotter(); // create plotter
  p.AddTimeGraph( "Real power vs time", 500, "real power", realPower ); // add any graphs you want
}

void loop()
{
  emon1.calcVI(20,2000); // Calculate all. No.of half wavelengths (crossings), time-out

  realPower = emon1.realPower; //extract Real Power into variable

  p.Plot();
}

graph is not visible i mean its not plotting

Can you post the output from the serial monitor? Baud rate needs to be 115200 and listener needs to be closed

Closing this. Feel free to reopen if you still need help.


[BUG] Cluster Management: Edit as YAML option not available

Rancher Server Setup

Rancher version: v2.7.9
Installation option (Docker install/Helm Chart): Helm Chart

If Helm Chart, Kubernetes Cluster and version (RKE1, RKE2, k3s, EKS, etc): RKE2

Proxy/Cert Details:

Information about the Cluster

Kubernetes version:  v1.26.10+rke2r2
Cluster Type (Local/Downstream): Local

User Information

What is the role of the user logged in? (Admin/Cluster Owner/Cluster Member/Project Owner/Project Member/Custom)

Admin

Describe the bug
"Edit as YAML" option is not available in Cluster Management.
To Reproduce
Open Cluster Management -> Clusters, click on context menu for cluster.
Result
"Edit Config" and "View YAML" are available, but "Edit as YAML" is not.
Expected Result
Edit as YAML option as described in the manual.
Screenshots

Additional context

#21866 describes a previous instance of the option going missing, but that was apparently fixed.

I have this same issue, just in 2.9.1


pynmrstar should use its own logger?!

It would be helpful if pynmrstar used its own logger so when silencing warnings such as
 WARNING:root:Loop with no data on line: xx
warnings could be silenced for pynmrstar and not the libraries

I'm pretty sure loop with no data is an invalid construct in STAR so if anything, that particular one should throw an exception -- or at least be logged as critical error

I'm pretty sure loop with no data is an invalid construct in STAR so if anything, that particular one should throw an exception -- or at least be logged as critical error

It logs it as a warning, which I think is more appropriate than a critical error or even an error. A critical error implies that the program may not be able to continue - but an empty log in no way impedes the operation of the program. Allowing empty loops when parsing seems to me to best conform to the https://en.wikipedia.org/wiki/Robustness_principle.
It used to be possible to specify a module-level raise_parse_warnings value to cause issues like this to cause the raising of an exception rather than just logging about them. In the major 3.0 refactor when these module-level variables went away it looks like that functionality became hidden to the end user - only available if using the Parser() class directly. I've pushed commits which in the next release will restore this functionality to the end user so that when calling Entry.from_file() (for example) the user can specify raise_parse_warnings=True to access this behavior.

t would be helpful if pynmrstar used its own logger

Work on that was in progress but never completed - thanks for the reminder. I've just taken a look again and will have an update rolled out shortly to resolve this issue.

Resolved in newly released version 3.3.1.
Release notes: https://pynmrstar.readthedocs.io/en/latest/release-notes.html#id1


Upgrade docker image of Document Reviewer web - ACS fix for Document Reviewer

Title of ticket:
ACS Workable Task - Document Reviewer web
Description
This ticket is to action on fixing the ACS Vulnerabilities in Document Reviewer web container.
Dependencies
Are there any dependencies?
DOD

[x] Identify all the docker Images to upgrade.
[x] Check and upgrade to the latest versions of node and nginx.
[x] Ensure the Alpine Linux Packages (e.g., bash, git, openssh) are up to date.
[x] Test the upgrade in local environment.

Closing this task as we upgraded the versions of node and ngnix to the latest versions and in the ACS reports all the fixable vulnerabilities related to web components are fixed.