libtensorflow: 2.13.0 -> 2.14.0

Description of changes
Updated the version and adapted scripts and patches to make it build. Basic tests also succeeded. There remains a warning about ml_dtypes, which might be expected (see https://github.com/NixOS/nixpkgs/pull/258546#issuecomment-1743309427)
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Thank you for this work @zeuner !
About ml_dtypes, maybe you could directly go for 2.15.0 ?

In the process of doing so. When I saw 2.15.0, I had already started working on 2.14.0, so I decided to finish it first in order to get all the clang-related changes right. Hope to get it adapted to 2.15.0, soon.

Very cool !

I'm also offering to be added to the maintainer list considering that no existing maintainer had participated in the 2.13 update before and also not on the recent efforts. Apart from that, I have some ideas to possibly clean up and streamline the build in the future, so I think it makes sense.

I'm also offering to be added to the maintainer list considering that no existing maintainer had participated in the 2.13 update before and also not on the recent efforts. Apart from that, I have some ideas to possibly clean up and streamline the build in the future, so I think it makes sense.

cc @ConnorBaker @samuela @SomeoneSerge
I support this request. Thanks so much for your hard work on this @zeuner !

In the process of doing so. When I saw 2.15.0, I had already started working on 2.14.0, so I decided to finish it first in order to get all the clang-related changes right. Hope to get it adapted to 2.15.0, soon.

Very cool !

My apologies for the delay. I came across some issues with the sandboxed build which made it fail of ofborg. By now, it built on both Linux platforms. Still investigating possible reproducibility issues with the deps derivation.

Result of nixpkgs-review pr 272838 run on x86_64-linux 1

24 packages marked as broken and skipped:

python310Packages.dm-sonnet
python310Packages.dm-sonnet.dist
python310Packages.elegy
python310Packages.elegy.dist
python310Packages.scikit-tda
python310Packages.scikit-tda.dist
python310Packages.tensorflow-datasets
python310Packages.tensorflow-datasets.dist
python310Packages.tensorflowWithCuda
python310Packages.tensorflowWithCuda.dist
python310Packages.treex
python310Packages.treex.dist
python311Packages.dm-sonnet
python311Packages.dm-sonnet.dist
python311Packages.elegy
python311Packages.elegy.dist
python311Packages.scikit-tda
python311Packages.scikit-tda.dist
python311Packages.tensorflow-datasets
python311Packages.tensorflow-datasets.dist
python311Packages.tensorflowWithCuda
python311Packages.tensorflowWithCuda.dist
python311Packages.treex
python311Packages.treex.dist

68 packages failed to build:

python310Packages.bambi
python310Packages.bambi.dist
python310Packages.baselines
python310Packages.baselines.dist
python310Packages.cleanlab
python310Packages.cleanlab.dist
python310Packages.dalle-mini
python310Packages.dalle-mini.dist
python310Packages.distrax
python310Packages.distrax.dist
python310Packages.edward
python310Packages.edward.dist
python310Packages.flax
python310Packages.flax.dist
python310Packages.gpt-2-simple
python310Packages.gpt-2-simple.dist
python310Packages.mhcflurry
python310Packages.mhcflurry.dist
python310Packages.n3fit
python310Packages.n3fit.dist
python310Packages.objax
python310Packages.objax.dist
python310Packages.pot
python310Packages.pot.dist
python310Packages.pymc
python310Packages.pymc.dist
python310Packages.pytensor
python310Packages.pytensor.dist
python310Packages.rlax
python310Packages.rlax.dist
python310Packages.tensorflow (python310Packages.tensorflow-build ,python310Packages.tensorflowWithoutCuda)
python310Packages.tensorflow.dist (python310Packages.tensorflow-build.dist ,python310Packages.tensorflowWithoutCuda.dist)
python310Packages.tensorflow-probability
python310Packages.tensorflow-probability.dist
python310Packages.tf2onnx
python310Packages.tf2onnx.dist
python310Packages.tflearn
python310Packages.tflearn.dist
python310Packages.trfl
python310Packages.trfl.dist
python310Packages.umap-learn
python310Packages.umap-learn.dist
python310Packages.vqgan-jax
python310Packages.vqgan-jax.dist
python311Packages.bambi
python311Packages.bambi.dist
python311Packages.dalle-mini
python311Packages.dalle-mini.dist
python311Packages.distrax
python311Packages.distrax.dist
python311Packages.flax
python311Packages.flax.dist
python311Packages.gpt-2-simple
python311Packages.gpt-2-simple.dist
python311Packages.objax
python311Packages.objax.dist
python311Packages.pymc
python311Packages.pymc.dist
python311Packages.pytensor
python311Packages.pytensor.dist
python311Packages.rlax
python311Packages.rlax.dist
python311Packages.tf2onnx
python311Packages.tf2onnx.dist
python311Packages.trfl
python311Packages.trfl.dist
python311Packages.vqgan-jax
python311Packages.vqgan-jax.dist

26 packages built:

frigate
gpt2tc
libtensorflow (libtensorflow.python)
python311Packages.baselines
python311Packages.baselines.dist
python311Packages.cleanlab
python311Packages.cleanlab.dist
python311Packages.edward
python311Packages.edward.dist
python311Packages.mhcflurry
python311Packages.mhcflurry.dist
python311Packages.n3fit
python311Packages.n3fit.dist
python311Packages.pot
python311Packages.pot.dist
python311Packages.tensorflow (python311Packages.tensorflow-build ,python311Packages.tensorflowWithoutCuda)
python311Packages.tensorflow.dist (python311Packages.tensorflow-build.dist ,python311Packages.tensorflowWithoutCuda.dist)
python311Packages.tensorflow-probability
python311Packages.tensorflow-probability.dist
python311Packages.tflearn
python311Packages.tflearn.dist
python311Packages.umap-learn
python311Packages.umap-learn.dist
tests.pkg-config.defaultPkgConfigPackages.tensorflow
wyoming-openwakeword
wyoming-openwakeword.dist

@zeuner Not sure if I'll solve all issues, but let's push it into the right direction on darwin:
cudaCapabilities fails to evaluate, because cudaPackagesGoogle explodes when touched. This works:
, cudaCapabilities ? lib.optional (config.cudaSupport) cudaPackagesGoogle.cudaFlags.cudaCapabilities

A  few bazel/tf issues on darwin next:
The build runs on python3.11 at this point, but I get:
DEBUG: /private/tmp/nix-build-tensorflow-2.14.0-deps.tar.gz.drv-0/source/tensorflow/tools/toolchains/python/python_repo.bzl:21:14:
TF_PYTHON_VERSION variable was not set correctly, using default version. 3.10 Python
will be used.

To set Python version, run
export TF_PYTHON_VERSION=3.9

There's also this, but I don't think it actually breaks anything:
DEBUG: Rule 'python_aarch64-apple-darwin' indicated that a canonical reproducible form can be obtained by modifying arguments python_version = "3.11.6"
DEBUG: Repository python_aarch64-apple-darwin instantiated at:
  /private/tmp/nix-build-tensorflow-2.14.0-deps.tar.gz.drv-0/source/WORKSPACE:35:27: in <toplevel>
  /private/tmp/nix-build-tensorflow-2.14.0-deps.tar.gz.drv-0/output/external/rules_python/python/repositories.bzl:506:26: in python_register_toolchains
Repository rule python_repository defined at:
  /private/tmp/nix-build-tensorflow-2.14.0-deps.tar.gz.drv-0/output/external/rules_python/python/repositories.bzl:334:36: in <toplevel>

And finally pypi_portpicker is not available:

INFO: Repository pypi_portpicker instantiated at:
  /private/tmp/nix-build-tensorflow-2.14.0-deps.tar.gz.drv-0/source/WORKSPACE:69:13: in <toplevel>
  /private/tmp/nix-build-tensorflow-2.14.0-deps.tar.gz.drv-0/output/external/pypi/requirements.bzl:47:20: in install_deps
Repository rule whl_library defined at:
  /private/tmp/nix-build-tensorflow-2.14.0-deps.tar.gz.drv-0/output/external/rules_python/python/pip_install/pip_repository.bzl:726:30: in <toplevel>
ERROR: An error occurred during the fetch of repository 'pypi_portpicker':
   Traceback (most recent call last):
        File "/private/tmp/nix-build-tensorflow-2.14.0-deps.tar.gz.drv-0/output/external/rules_python/python/pip_install/pip_repository.bzl", line 696, column 63, in _whl_library_impl
                environment = _create_repository_execution_environment(rctx),
        File "/private/tmp/nix-build-tensorflow-2.14.0-deps.tar.gz.drv-0/output/external/rules_python/python/pip_install/pip_repository.bzl", line 219, column 47, in _create_repository_execution_environment
                cppflags.extend(_get_xcode_location_cflags(rctx))
        File "/private/tmp/nix-build-tensorflow-2.14.0-deps.tar.gz.drv-0/output/external/rules_python/python/pip_install/pip_repository.bzl", line 99, column 38, in _get_xcode_location_cflags
                xcode_sdk_location = rctx.execute([xcode_select, "--print-path"])
Error in execute: Argument 0 of execute is neither a path, label, nor string.
ERROR: /private/tmp/nix-build-tensorflow-2.14.0-deps.tar.gz.drv-0/source/WORKSPACE:69:13: fetching whl_library rule //external:pypi_portpicker: Traceback (most recent call last):
        File "/private/tmp/nix-build-tensorflow-2.14.0-deps.tar.gz.drv-0/output/external/rules_python/python/pip_install/pip_repository.bzl", line 696, column 63, in _whl_library_impl
                environment = _create_repository_execution_environment(rctx),
        File "/private/tmp/nix-build-tensorflow-2.14.0-deps.tar.gz.drv-0/output/external/rules_python/python/pip_install/pip_repository.bzl", line 219, column 47, in _create_repository_execution_environment
                cppflags.extend(_get_xcode_location_cflags(rctx))
        File "/private/tmp/nix-build-tensorflow-2.14.0-deps.tar.gz.drv-0/output/external/rules_python/python/pip_install/pip_repository.bzl", line 99, column 38, in _get_xcode_location_cflags
                xcode_sdk_location = rctx.execute([xcode_select, "--print-path"])
Error in execute: Argument 0 of execute is neither a path, label, nor string.
ERROR: /private/tmp/nix-build-tensorflow-2.14.0-deps.tar.gz.drv-0/source/tensorflow/dtensor/python/tests/BUILD:290:22: //tensorflow/dtensor/python/tests:multi_client_test_util depends on @pypi_portpicker//:pkg in repository @pypi_portpicker which failed to fetch. no such package '@pypi_portpicker//': Argument 0 of execute is neither a path, label, nor string.
ERROR: Analysis of target '//tensorflow/tools/pip_package:build_pip_package' failed; build aborted:
INFO: Elapsed time: 145.580s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (93 packages loaded, 254 targets configured)

error: builder for '/nix/store/n833za00l2rhdyypb60y9z381gcq52p7-tensorflow-2.14.0-deps.tar.gz.drv' failed with exit code 1;
       last 10 log lines:
       >          cppflags.extend(_get_xcode_location_cflags(rctx))
       >      File "/private/tmp/nix-build-tensorflow-2.14.0-deps.tar.gz.drv-0/output/external/rules_python/python/pip_install/pip_repository.bzl", line 99, column 38, in _get_xcode_location_cflags
       >                xcode_sdk_location = rctx.execute([xcode_select, "--print-path"])
       > Error in execute: Argument 0 of execute is neither a path, label, nor string.
       > ERROR: /private/tmp/nix-build-tensorflow-2.14.0-deps.tar.gz.drv-0/source/tensorflow/dtensor/python/tests/BUILD:290:22: //tensorflow/dtensor/python/tests:multi_client_test_util depends on @pypi_portpicker//:pkg in repository @pypi_portpicker which failed to fetch. no such package '@pypi_portpicker//': Argument 0 of execute is neither a path, label, nor string.
       > ERROR: Analysis of target '//tensorflow/tools/pip_package:build_pip_package' failed; build aborted:
       > INFO: Elapsed time: 145.580s
       > INFO: 0 processes.
       > FAILED: Build did NOT complete successfully (93 packages loaded, 254 targets configured)
       >
       For full logs, run 'nix log /nix/store/n833za00l2rhdyypb60y9z381gcq52p7-tensorflow-2.14.0-deps.tar.gz.drv'.

Finally in tensorflow, build crashed twice:
[4,740 / 11,931] Compiling tensorflow/python/util/tf_stack.cc; 4s processwrapper-sandbox ... (12 actions running)
ERROR: /private/tmp/nix-build-tensorflow-2.14.0.drv-0/source/tensorflow/core/framework/BUILD:1401:31: Executing genrule //tensorflow/core/framework:attr_value_proto_text_srcs failed: (Segmentation fault): bash failed: error executing command (from target //tensorflow/core/framework:attr_value_proto_text_srcs) /nix/store/rhhkj6hbnz4xwf8x3x17wbi08806ha5g-bash/bin/bash -c ... (remaining 1 argument skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
[4,747 / 11,931] Compiling tensorflow/python/util/tf_stack.cc; 6s processwrapper-sandbox ... (5 actions running)
ERROR: /private/tmp/nix-build-tensorflow-2.14.0.drv-0/source/tensorflow/core/framework/BUILD:1401:31 Middleman _middlemen/_S_Stensorflow_Score_Sframework_Cattr_Uvalue_Uproto_Utext-cc_library-compile failed: (Segmentation fault): bash failed: error executing command (from target //tensorflow/core/framework:attr_value_proto_text_srcs) /nix/store/rhhkj6hbnz4xwf8x3x17wbi08806ha5g-bash/bin/bash -c ... (remaining 1 argument skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
INFO: Elapsed time: 153.285s, Critical Path: 11.99s
INFO: 4752 processes: 3954 internal, 798 processwrapper-sandbox.
FAILED: Build did NOT complete successfully

Then:
11,969 / 21,770] Compiling tensorflow/core/common_runtime/local_device.cc [for tool]; 7s processwrapper-sandbox ... (12 actions running)
ERROR: /private/tmp/nix-build-tensorflow-2.14.0.drv-0/source/tensorflow/core/framework/BUILD:1401:31: Executing genrule //tensorflow/core/framework:attr_value_proto_text_srcs [for tool] failed: (Segmentation fault): bash failed: error executing command (from target //tensorflow/core/framework:attr_value_proto_text_srcs) /nix/store/rhhkj6hbnz4xwf8x3x17wbi08806ha5g-bash/bin/bash -c ... (remaining 1 argument skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
[11,984 / 21,770] Compiling tensorflow/core/framework/types.cc [for tool]; 0s processwrapper-sandbox
ERROR: /private/tmp/nix-build-tensorflow-2.14.0.drv-0/source/tensorflow/core/framework/BUILD:1401:31 Middleman _middlemen/_S_Stensorflow_Score_Sframework_Cattr_Uvalue_Uproto_Utext-cc_library-compile failed: (Segmentation fault): bash failed: error executing command (from target //tensorflow/core/framework:attr_value_proto_text_srcs) /nix/store/rhhkj6hbnz4xwf8x3x17wbi08806ha5g-bash/bin/bash -c ... (remaining 1 argument skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
INFO: Elapsed time: 593.961s, Critical Path: 18.63s
INFO: 11985 processes: 8521 internal, 3464 processwrapper-sandbox.
FAILED: Build did NOT complete successfully

Different places, so... random?

All darwin fixes so far - feel free to grab those: https://github.com/NixOS/nixpkgs/commit/e33b69ac9d700bf7a7bb965b228caeb1e1232117

Another darwin issue on ofborg:
Error in fail: libtool_check_unique failed to generate. Please file an issue at https://github.com/bazelbuild/apple_support/issues with the following:
return code 1, stderr: xcode-select: error: No developer tools were found and no install could be requested (possibly because there is no active GUI session).
If developer tools are located at a non-default location on disk, use `sudo xcode-select --switch path/to/Xcode.app` to specify the Xcode

Locally, installing xcbuild is enough, but I'm not sure how would we work around this elsewhere. This seems to be a longstanding issue: https://github.com/NixOS/nixpkgs/issues/30590

Thanks!

python310Packages.tensorflow

Probably due to the Python version leaking into the deps archive (hopefully fixed now).

@zeuner Not sure if I'll solve all issues, but let's push it into the right direction on darwin: cudaCapabilities fails to evaluate, because cudaPackagesGoogle explodes when touched. This works:
, cudaCapabilities ? lib.optional (config.cudaSupport) cudaPackagesGoogle.cudaFlags.cudaCapabilities

next python3.11-ml-dtypes-0.3.1 failing on CustomFloatTest.testBetweenCustomTypes_bfloat16: It looks like it should be fixed in 0.3.2, and I was hoping jax-ml/ml_dtypes#129 solves it, but applying it doesn't seem to do it unfortunately. Annoyingly, overriding the python module version doesn't change the version in the derivation name, so this is confusing - the 0.1.0 version comes up as 0.3.1 anyway.
Disabling tests with disabledTests = ["testBetweenCustomTypes_bfloat16" "testPredicateUfunc_bfloat16"]; progresses the build. Upsteam confirms the warnings can be ignored (jax-ml/ml_dtypes#47 (comment))

Thanks! Fortunately, the ml_dtypes issue is only temporary, and shouldn't bother us in 2.15.0 (https://github.com/NixOS/nixpkgs/pull/278616) anymore.

A few bazel/tf issues on darwin next:
The build runs on python3.11 at this point, but I get:
DEBUG: /private/tmp/nix-build-tensorflow-2.14.0-deps.tar.gz.drv-0/source/tensorflow/tools/toolchains/python/python_repo.bzl:21:14:
TF_PYTHON_VERSION variable was not set correctly, using default version. 3.10 Python
will be used.

To set Python version, run
export TF_PYTHON_VERSION=3.9

(TF_PYTHON_VERSION = python.pythonVersion; seems to solve it)
There's also this, but I don't think it actually breaks anything:
DEBUG: Rule 'python_aarch64-apple-darwin' indicated that a canonical reproducible form can be obtained by modifying arguments python_version = "3.11.6"
DEBUG: Repository python_aarch64-apple-darwin instantiated at:
  /private/tmp/nix-build-tensorflow-2.14.0-deps.tar.gz.drv-0/source/WORKSPACE:35:27: in <toplevel>
  /private/tmp/nix-build-tensorflow-2.14.0-deps.tar.gz.drv-0/output/external/rules_python/python/repositories.bzl:506:26: in python_register_toolchains
Repository rule python_repository defined at:
  /private/tmp/nix-build-tensorflow-2.14.0-deps.tar.gz.drv-0/output/external/rules_python/python/repositories.bzl:334:36: in <toplevel>

~And finally pypi_portpicker is not available:~ xcode-select is not found, because it's not in nativeBuildInputs.
Adding xcbuild to nativeBuildInputs for stdenv.isDarwin seems to solve that and the result hash for darwin-aarch is sha256-7SwkyRdeTveml70/JXOZ11pPA+4jkUbt+cLPvEj+YX0=
The fixup has issues with this: (which may also be the case on linux? but doesn't matter on darwin, because there's no elf)
post-installation fixup
checking for references to /private/tmp/nix-build-llvm-raw-tensorflow-2.14.0.drv-0/ in /nix/store/69l84c51cw9xsmpy3pfqpyfnfd4lhidw-llvm-raw-tensorflow-2.14.0...
/nix/store/h9lc1dpi14z7is86ffhl3ld569138595-audit-tmpdir.sh: line 23: patchelf: command not found

The Python versioning related messages can probably be disregarded. It was necessary to use nixpkgs' Python instead of the one pulled in by tensorflow because the latter can lead to runtime linkage errors together with libraries linked against nixpkgs' runtime. Anyway, good if we can silence the warning.

Another darwin issue on ofborg:
Error in fail: libtool_check_unique failed to generate. Please file an issue at https://github.com/bazelbuild/apple_support/issues with the following:
return code 1, stderr: xcode-select: error: No developer tools were found and no install could be requested (possibly because there is no active GUI session).
If developer tools are located at a non-default location on disk, use `sudo xcode-select --switch path/to/Xcode.app` to specify the Xcode

Locally, installing xcbuild is enough in an impure environment, but I'm not sure how would we work around this elsewhere. This seems to be a longstanding issue: #30590 (the comment didn't resolve it in CI)

I wonder whether it's possible to go without xcbuild. Seems like it wasn't necessary until 2.11.0.

@viraptor For some reason, building the Python env doesn't seem to succeed on x86_84-darwin: https://logs.ofborg.org/?key=nixos/nixpkgs.272838&attempt_id=ed36b3a5-4d67-4360-84c9-8ec75d96bf4c

Hey @zeuner.
Do you want some help with this ? It would be very valuable to get the latest version of tensorflow.
Some packages seems not to work with 2.13 (like the latest version of keras for instance).

I'd like to ping @NixOS/darwin-maintainers again - I ran out of skill here. Maybe someone else can figure out the remaining issues :(

TensorFlow 2.16.1 is out

Back to work on a machine with more storage space, which helped figuring out issues with deps hashes that changed when switching between Python versions.

nixpkgs-review result
Generated using nixpkgs-review.

It turns out that keras and thus also its reverse dependencies couldn't be built because TF 2.14.0 needs special handling of ml-dtypes and Python cannot handle keras and TF importing different versions of the same package.
I got keras built by allowing TF to export the ml-dtypes it uses, and having keras pick up that export if it exists.

Hey @zeuner. Where are you at on this PR ?
If I can do anything to help, please feel free to reach out.
I can run test builds pretty easily if needed :)
Thanks once again for tackling this very important work.

Hey @zeuner. Where are you at on this PR ? If I can do anything to help, please feel free to reach out. I can run test builds pretty easily if needed :)
Thanks once again for tackling this very important work.

After 2.14 building and running here, I'm at this point more focused on the subsequent updates for TF.
Would you be able to compute the deps hash for CUDA-enabled aarch64-linux? It should be visible in a hash mismatch error message when building for that platform.
AFAIR, @SomeoneSerge might have a change to be included.
Also, any reviewing is of course appreciated.

Thanks for the update !

Would you be able to compute the deps hash for CUDA-enabled aarch64-linux?

nix-build --arg config '{ allowUnfree = true; cudaSupport = true; }' --system aarch64-linux -A python311Packages.tensorflow

gives
error: cannot coerce a list to a string: [ "6.0" "6.1" "7.0" "7.5" "8.0" "8.6" "8.9" "9.0" ]

On my machine nix-build --system aarch64-linux -A python311Packages.tensorflowWithCuda fails with
ERROR: /build/source/third_party/py/numpy/BUILD:5:6: //third_party/py/numpy:numpy depends on @pypi_numpy//:pkg in repository @pypi_numpy which failed to fetch. no such package '@pypi_numpy//': whl_library pypi_numpy failed: Collecting numpy==1.26.4 (from -r /build/tmparr9phrh (line 1))
  Downloading numpy-1.26.4-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (14.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 14.2/14.2 MB 5.9 MB/s eta 0:00:00
 (ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
    numpy==1.26.4 from https://files.pythonhosted.org/packages/79/ae/7e5b85136806f9dadf4878bf73cf223fe5c2636818ba3ab1c585d0403164/numpy-1.26.4-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (from -r /build/tmparr9phrh (line 1)):
        Expected sha256 06005a2ef6014e9956c09ba07654f9837d9e26696a0470e42beedadb78c11b07
        Expected     or 09b7847f7e83ca37c6e627682f145856de331049013853f344f37b0c9690e3df
        Expected     or 0aaee12d8883552fadfc41e96b4c82ee7d794949e2a7c3b3a7201e968c7ecab9
        Expected     or 0cbe9848fad08baf71de1a39e12d1b6310f1d5b2d0ea4de051058e6e1076852d
        Expected     or 1b1766d6f397c18153d40015ddfc79ddb715cabadc04d2d228d4e5a8bc4ded1a
        Expected     or 33161613d2269025873025b33e879825ec7b1d831317e68f4f2f0f84ed14c719
        Expected     or 5039f55555e1eab31124a5768898c9e22c25a65c1e0037f4d7c495a45778c9f2
        Expected     or 522e26bbf6377e4d76403826ed689c295b0b238f46c28a7251ab94716da0b280
        Expected     or 56e454c7833e94ec9769fa0f86e6ff8e42ee38ce0ce1fa4cbb747ea7e06d56aa
        Expected     or 58f545efd1108e647604a1b5aa809591ccd2540f468a880bedb97247e72db387
        Expected     or 5e05b1c973a9f858c74367553e236f287e749465f773328c8ef31abe18f691e1
        Expected     or 7903ba8ab592b82014713c491f6c5d3a1cde5b4a3bf116404e08f5b52f6daf43
        Expected     or 8969bfd28e85c81f3f94eb4a66bc2cf1dbdc5c18efc320af34bffc54d6b1e38f
        Expected     or 92c8c1e89a1f5028a4c6d9e3ccbe311b6ba53694811269b992c0b224269e2398
        Expected     or 9c88793f78fca17da0145455f0d7826bcb9f37da4764af27ac945488116efe63
        Expected     or 9fad7dcb1aac3c7f0584a5a8133e3a43eeb2fe127f47e3632d43d677c66c102b
        Expected     or a7ac231a08bb37f852849bbb387a20a57574a97cfc7b6cabb488a4fc8be176de
        Expected     or abdde9f795cf292fb9651ed48185503a2ff29be87770c3b8e2a14b0cd7aa16f8
        Expected     or af1da88f6bc3d2338ebbf0e22fe487821ea4d8e89053e25fa59d1d79786e7481
        Expected     or b2a9ab7c279c91974f756c84c365a669a887efa287365a8e2c418f8b3ba73fb0
        Expected     or bf837dc63ba5c06dc8797c398db1e223a466c7ece27a1f7b5232ba3466aafe3d
        Expected     or ca51fcfcc5f9354c45f400059e88bc09215fb71a48d3768fb80e357f3b457e1e
        Expected     or ce571367b6dfe60af04e04a1834ca2dc5f46004ac1cc756fb95319f64c095a96
        Expected     or d208a0f8729f3fb790ed18a003f3a57895b989b40ea4dce4717e9cf4af62c6bb
        Expected     or dbee87b469018961d1ad79b1a5d50c0ae850000b639bcb1b694e9981083243b6
        Expected     or e9f4c4e51567b616be64e05d517c79a8a22f3606499941d97bb76f2ca59f982d
        Expected     or f063b69b090c9d918f9df0a12116029e274daf0181df392839661c4c7ec9018a
        Expected     or f9a909a8bae284d46bbfdefbdd4a262ba19d3bc9921b1e76126b1d21c3c34135
             Got        7ab55401287bfec946ced39700c053796e7cc0e3acbef09993a9ad2adba6ca6e

For non-cuda aarch64-linux ofborg produces different hash
error: hash mismatch in fixed-output derivation '/nix/store/g0lmxrr2msf250zj1ripy0rv7kywz2dm-tensorflow-2.14.0-deps.tar.gz.drv':
         specified: sha256-Vu4pk5J98zTHBzyLSHdaioqo64cnfCfRQHUX91wEbgM=
            got:    sha256-T3X88roRW9+kUCwaftz1GdjxkxffLSF1VcPMoICNhlg=

will try to build that one too one my machine

For nix-build --system aarch64-linux -A python312Packages.tensorflow
error: build of '/nix/store/v8xlwfxq5yd0515rv6lhlbanhwgnw704-tensorflow-2.14.0-deps.tar.gz.drv' on 'ssh-ng://khadas' failed: error: hash mismatch in fixed-output derivation '/nix/store/v8xlwfxq5yd0515rv6lhlbanhwgnw704-tensorflow-2.14.0-deps.tar.gz.drv':
                specified: sha256-Vu4pk5J98zTHBzyLSHdaioqo64cnfCfRQHUX91wEbgM=
                   got:    sha256-x62qm084a06fvKxL4W8LL6iMBSWoBphAyRksvSk/bvk=

And for nix-build --system aarch64-linux -A python312Packages.tensorflowWithCuda
INFO: Repository local_config_cuda instantiated at:
  /build/source/WORKSPACE:84:14: in <toplevel>
  /build/source/tensorflow/workspace2.bzl:1020:19: in workspace
  /build/source/tensorflow/workspace2.bzl:105:19: in _tf_toolchains
Repository rule cuda_configure defined at:
  /build/source/third_party/gpus/cuda_configure.bzl:1490:33: in <toplevel>
ERROR: An error occurred during the fetch of repository 'local_config_cuda':
...
ERROR: @local_config_cuda//:enable_cuda :: Error loading option @local_config_cuda//:enable_cuda: Repository command failed
/build/output/external/local_config_cuda/script.py:124: SyntaxWarning: invalid escape sequence '\s'
  match = re.match("\s*#\s*define %s\s+(\d+)" % name, line)
/build/output/external/local_config_cuda/script.py:263: SyntaxWarning: invalid escape sequence '\d'
  pattern = "Cuda compilation tools, release \d+\.\d+, V(\d+\.\d+\.\d+)"
/build/output/external/local_config_cuda/script.py:552: SyntaxWarning: invalid escape sequence '\w'
  match = re.match("^(/[^/ ]*)+/lib/\w+-linux-gnu/?$", os.environ[env_name])

Thanks for the update !

Would you be able to compute the deps hash for CUDA-enabled aarch64-linux?

nix-build --arg config '{ allowUnfree = true; cudaSupport = true; }' --system aarch64-linux -A python311Packages.tensorflow

Thanks, learned something new. I "enabled" cudaSupport by building tensorflowWithCuda before, which did not trigger this. Should be fixed now.

On my machine nix-build --system aarch64-linux -A python311Packages.tensorflowWithCuda fails with
ERROR: /build/source/third_party/py/numpy/BUILD:5:6: //third_party/py/numpy:numpy depends on @pypi_numpy//:pkg in repository @pypi_numpy which failed to fetch. no such package '@pypi_numpy//': whl_library pypi_numpy failed: Collecting numpy==1.26.4 (from -r /build/tmparr9phrh (line 1))
  Downloading numpy-1.26.4-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (14.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 14.2/14.2 MB 5.9 MB/s eta 0:00:00
 (ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
    numpy==1.26.4 from https://files.pythonhosted.org/packages/79/ae/7e5b85136806f9dadf4878bf73cf223fe5c2636818ba3ab1c585d0403164/numpy-1.26.4-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (from -r /build/tmparr9phrh (line 1)):
        Expected sha256 06005a2ef6014e9956c09ba07654f9837d9e26696a0470e42beedadb78c11b07
        Expected     or 09b7847f7e83ca37c6e627682f145856de331049013853f344f37b0c9690e3df
        Expected     or 0aaee12d8883552fadfc41e96b4c82ee7d794949e2a7c3b3a7201e968c7ecab9
        Expected     or 0cbe9848fad08baf71de1a39e12d1b6310f1d5b2d0ea4de051058e6e1076852d
        Expected     or 1b1766d6f397c18153d40015ddfc79ddb715cabadc04d2d228d4e5a8bc4ded1a
        Expected     or 33161613d2269025873025b33e879825ec7b1d831317e68f4f2f0f84ed14c719
        Expected     or 5039f55555e1eab31124a5768898c9e22c25a65c1e0037f4d7c495a45778c9f2
        Expected     or 522e26bbf6377e4d76403826ed689c295b0b238f46c28a7251ab94716da0b280
        Expected     or 56e454c7833e94ec9769fa0f86e6ff8e42ee38ce0ce1fa4cbb747ea7e06d56aa
        Expected     or 58f545efd1108e647604a1b5aa809591ccd2540f468a880bedb97247e72db387
        Expected     or 5e05b1c973a9f858c74367553e236f287e749465f773328c8ef31abe18f691e1
        Expected     or 7903ba8ab592b82014713c491f6c5d3a1cde5b4a3bf116404e08f5b52f6daf43
        Expected     or 8969bfd28e85c81f3f94eb4a66bc2cf1dbdc5c18efc320af34bffc54d6b1e38f
        Expected     or 92c8c1e89a1f5028a4c6d9e3ccbe311b6ba53694811269b992c0b224269e2398
        Expected     or 9c88793f78fca17da0145455f0d7826bcb9f37da4764af27ac945488116efe63
        Expected     or 9fad7dcb1aac3c7f0584a5a8133e3a43eeb2fe127f47e3632d43d677c66c102b
        Expected     or a7ac231a08bb37f852849bbb387a20a57574a97cfc7b6cabb488a4fc8be176de
        Expected     or abdde9f795cf292fb9651ed48185503a2ff29be87770c3b8e2a14b0cd7aa16f8
        Expected     or af1da88f6bc3d2338ebbf0e22fe487821ea4d8e89053e25fa59d1d79786e7481
        Expected     or b2a9ab7c279c91974f756c84c365a669a887efa287365a8e2c418f8b3ba73fb0
        Expected     or bf837dc63ba5c06dc8797c398db1e223a466c7ece27a1f7b5232ba3466aafe3d
        Expected     or ca51fcfcc5f9354c45f400059e88bc09215fb71a48d3768fb80e357f3b457e1e
        Expected     or ce571367b6dfe60af04e04a1834ca2dc5f46004ac1cc756fb95319f64c095a96
        Expected     or d208a0f8729f3fb790ed18a003f3a57895b989b40ea4dce4717e9cf4af62c6bb
        Expected     or dbee87b469018961d1ad79b1a5d50c0ae850000b639bcb1b694e9981083243b6
        Expected     or e9f4c4e51567b616be64e05d517c79a8a22f3606499941d97bb76f2ca59f982d
        Expected     or f063b69b090c9d918f9df0a12116029e274daf0181df392839661c4c7ec9018a
        Expected     or f9a909a8bae284d46bbfdefbdd4a262ba19d3bc9921b1e76126b1d21c3c34135
             Got        7ab55401287bfec946ced39700c053796e7cc0e3acbef09993a9ad2adba6ca6e

Added the missing numpy hash, so this should not fail anymore.

For non-cuda aarch64-linux ofborg produces different hash
error: hash mismatch in fixed-output derivation '/nix/store/g0lmxrr2msf250zj1ripy0rv7kywz2dm-tensorflow-2.14.0-deps.tar.gz.drv':
         specified: sha256-Vu4pk5J98zTHBzyLSHdaioqo64cnfCfRQHUX91wEbgM=
            got:    sha256-T3X88roRW9+kUCwaftz1GdjxkxffLSF1VcPMoICNhlg=

will try to build that one too one my machine

I wasn't able to reproduce the deps hash instability on x86_64-linux. Bumped the hash to the one computed by ofborg for now. In case of yet another deps hash mismatch on aarch64-linux, I could use help from someone who can provide a diff between two differing deps builds. E.g. by building with --keep-failed, and diffing build/output/external of two differing builds.

And for nix-build --system aarch64-linux -A python312Packages.tensorflowWithCuda
INFO: Repository local_config_cuda instantiated at:
  /build/source/WORKSPACE:84:14: in <toplevel>
  /build/source/tensorflow/workspace2.bzl:1020:19: in workspace
  /build/source/tensorflow/workspace2.bzl:105:19: in _tf_toolchains
Repository rule cuda_configure defined at:
  /build/source/third_party/gpus/cuda_configure.bzl:1490:33: in <toplevel>
ERROR: An error occurred during the fetch of repository 'local_config_cuda':
...
ERROR: @local_config_cuda//:enable_cuda :: Error loading option @local_config_cuda//:enable_cuda: Repository command failed
/build/output/external/local_config_cuda/script.py:124: SyntaxWarning: invalid escape sequence '\s'
  match = re.match("\s*#\s*define %s\s+(\d+)" % name, line)
/build/output/external/local_config_cuda/script.py:263: SyntaxWarning: invalid escape sequence '\d'
  pattern = "Cuda compilation tools, release \d+\.\d+, V(\d+\.\d+\.\d+)"
/build/output/external/local_config_cuda/script.py:552: SyntaxWarning: invalid escape sequence '\w'
  match = re.match("^(/[^/ ]*)+/lib/\w+-linux-gnu/?$", os.environ[env_name])

Fixed the error related to Python 3.12 string syntax. I think it builds now. Please note that upstream tensorflow 2.14 doesn't support Python 3.12, so we might drop it if keeping it creates too much hassle.

nixpkgs-review result
Generated using nixpkgs-review.
 I "enabled" cudaSupport by building tensorflowWithCuda before, which did not trigger this. Should be fixed now.

Yes, they are slightly different.
tensorflowWithCuda only overrides the cudaSupport argument for the tensorflow package, while enabling cudaSupport globally will affect the entire nixpkgs package set.

nixpkgs-review result
Generated using nixpkgs-review.

Thanks, learned something new. I "enabled" cudaSupport by building tensorflowWithCuda before, which did not trigger this. Should be fixed now.

Yes, they are slightly different. tensorflowWithCuda only overrides the cudaSupport argument for the tensorflow package, while enabling cudaSupport globally will affect the entire nixpkgs package set.

Yes... in our case, the thing is that cudaCapabilities is currently conditionalized on config.cudaSupport, so it doesn't pick up cudaPackages.cudaFlags.cudaCapabilities when building as tensorflowWithCuda, but it does so when globally setting the config flag (https://github.com/NixOS/nixpkgs/pull/272838/files#diff-3f9c997aafe20e8949648d4babbf03af7c4864c517b7cf3611d6a6e42610257aL74).
Not sure whether this is sensible, though. Maybe it slipped through when I considered the global push for preferably using cudaSupport from config instead of overrides. And maybe it's better to have the same behaviour regarding cudaCapabilities no matter how cudaSupport got enabled.

python311Packages.shimmy

These confuse me a bit. AFAICS, python3Packages.shimmy does not depend on python3Packages.tensorflow or python3Packages.keras (the packages being touched by this PR), so I'm not sure what I could do about it. Am I missing something here?

Got local mismatch without cuda
$ nix-build --system aarch64-linux -A python312Packages.tensorflow -j0
...
error: hash mismatch in fixed-output derivation '/nix/store/icicrg8manx86yywyyf0d833ckxnb9za-tensorflow-2.14.0-deps.tar.gz.drv':
                specified: sha256-ktEpU1YyedNpssuJTqsqS+XeY5NFNFnH1fuHsm4NRSY=
                   got:    sha256-85xKq/gKO9Klt56OBCowTFhYvz+61vSofODFGUio17s=

it is possible to download ofborg-built outputs?

Good question!
@NixOS/infra any ideas regarding this?

aarch64-linux hash
$ nix-build --system aarch64-linux -A python312Packages.tensorflowWithCuda
...
error: hash mismatch in fixed-output derivation '/nix/store/dkh4r650lvmkwgc1vgbnn61gdwpiif0k-tensorflow-gpu-2.14.0-deps.tar.gz.drv':
                specified: sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
                   got:    sha256-9Al7JH93Q7D0xN5sjtk5VvZH+ge13nnMJOC8Py5f96g=

Thanks, added.

python311Packages.shimmy

These confuse me a bit. AFAICS, python3Packages.shimmy does not depend on python3Packages.tensorflow or python3Packages.keras (the packages being touched by this PR), so I'm not sure what I could do about it. Am I missing something here?

I could also build it here on with this PR merged.

Figured out that pypi_psutil led to the deps hashes being unstable on aarch64-linux. They need to be recomputed now, but should hopefully be stable afterwards.

ofborg reports the hash to be sha256-KXSbqwZpN4Im32St0ADyIc4N5/SQfiQPDoLu9bXbckY=.


runtime: improve performance of IndexByte on older processors

Pre-avx2 processors use a loop of sse operations to do IndexByte.  They use an unaligned load to do so.  There may be a significant performance win by aligning the loads.  See the comments at the end of https://go-review.googlesource.com/#/c/18703/

So how expensive is MOVOU compared to MOVO when everything is aligned?

Slow down is present only pre-Nehalem, everything newer is not affected.
Experimental data confirms this: Xeon X5450 shows some slowdown, but not Xeon E5630 (Nehalem)  or anything newer:
name                    old speed      new speed      delta
IndexByte32-8           3.40GB/s ± 1%  3.11GB/s ± 3%  -8.34%  (p=0.000 n=17+18)
IndexByte4K-8           11.2GB/s ± 4%  11.1GB/s ± 4%  -0.75%  (p=0.033 n=20+20)
IndexByte4M-8           11.7GB/s ± 4%  11.5GB/s ± 6%  -1.82%  (p=0.016 n=18+20)
IndexByte64M-8          11.7GB/s ± 5%  11.5GB/s ± 4%  -1.47%  (p=0.008 n=16+17)


Laravel 9.x Compatibility

This is an automated pull request from Shift to update your package code and dependencies to be compatible with Laravel 9.x.
Before merging, you need to:

Checkout the l9-compatibility branch
Review all comments for additional changes
Thoroughly test your package

If you do find an issue, please report it by commenting on this PR to help improve future automation.

:alembic: Using this package? If you would like to help test these changes or believe them to be compatible, you may update your project to reference this branch.
To do so, temporarily add Shift's fork to the repositories property of your composer.json:
{
    "repositories": [
        {
            "type": "vcs",
            "url": "https://github.com/laravel-shift/colour.git"
        }
    ]
}

Then update your dependency constraint to reference this branch:
{
    "require": {
        "laravel-validation-rules/colour": "dev-l9-compatibility",
    }
}

Finally, run: composer update


Account sequence errors from connecting to partially synced chains

Crate
relayer-cli
Summary of Bug
If hermes is started while a chain is syncing, it picks up old acccount sequence values, and then attempts to use them repetitively.
The temporary solution is "don't start hermes against a partially synced chain"
The permanent solution is probably to actively check the account sequence
Version
0.7.3 + the compatiblity pr https://github.com/informalsystems/ibc-rs/pull/1461
Steps to Reproduce
Heheh, little bit of a hard one here, but I guess it's approximately:
Using a wallet that has already been used for relaying, start Hermes against a partially synced Juno or Sifchain.    What is likely to occur here is that Hermes will keep trying to make a tx with an invalid sequence number.
Another solution to this may be to disable hermes from interacting with partially-synced chains.
Acceptance Criteria
It should no longer be possible for Account sequence issues to crop up because it was connected to partially-synced chains from a previously used wallet.

For Admin Use

[ ] Not duplicate issue
[ ] Appropriate labels applied
[ ] Appropriate milestone (priority) applied
[ ] Appropriate contributors tagged
[ ] Contributor assigned/self-assigned

Hey Jacob,
This is a tough one! In general, Hermes cannot trust account sequence numbers from a full node unless that node (a) is synced with the network and (b) has a clean mempool (ref). For (b) we don't have a solution yet, but we're working with tm-go team to find one. The problem you're highlighting here is in reference to case (a).
With respect to (a) it's not clear how can Hermes reliably infer whether the node is or isn't synced with the network. For SDK-based chains there is the option to query the node's status endpoint and read the catching_up = true field; if false then the node is clearly not synced; if true then it's actually not clear what the status is, because the node may be out of sync and be unaware (the true flag is unreliable).
Beside the catching_up flag, is there any API that Hermes can use to infer if the full node is synced or not? Any insights appreciated!

Hi @adizere, thank you for your comment! We checked our chains, in this case, Juno, and it's unsynced with catching_up=false, but other chains are unsynced too, and they are still healthy.

This problem has been partly solved in PR #1349, which closed a related issue #1264.

Another solution to this may be to disable hermes from interacting with partially-synced chains.
Currently Hermes either

It should no longer be possible for Account sequence issues to crop up because it was connected to partially-synced chains from a previously used wallet.

Currently, Hermes either:

detects that a chain is out of sync via the health checkup (namely, the parameter catching_up), and in that case it emits a clear warning that some functionality is impacted.
does not detect the out of sync problem, but encounters an account sequence mismatch error and has an built-in mechanism to recover (by retrying) from that, thanks to the work in #1349.

operators no longer need to restart Hermes, but they may need to restart their full node if Hermes is unable to make progress due to the account sequence mismatch error.

I will therefore close this issue, and we may consider reopening if we still see problem in the future.


wsgi server: address family discovery is not quite right

The code introduced to improve binding to an IPv6 address is based on similar code in Python itself, but is missing some critical arguments to the socket.getaddrinfo() call: in particular, the socket type must be set to SOCK_STREAM, because we want a TCP connection; the AI_PASSIVE flag should also be passed because we intend to use the result for binding a listen socket rather than making an outbound connection.

By way of example, this is the output with and without the flags on an Ubuntu 22.04 system:
#!/usr/bin/env python3

import socket

print("WITH type set:")
print(socket.getaddrinfo("<IP_ADDRESS>", 9009,
    type=socket.SOCK_STREAM, flags=socket.AI_PASSIVE))

print("WITHOUT type set:")
print(socket.getaddrinfo("<IP_ADDRESS>", 9009))

WITH type set:
[(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('<IP_ADDRESS>', 9009))]
WITHOUT type set:
[(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('<IP_ADDRESS>', 9009)), (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_DGRAM: 2>, 17, '', ('<IP_ADDRESS>', 9009)), (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_RAW: 3>, 0, '', ('<IP_ADDRESS>', 9009))]

On some other platforms, getaddrinfo() requires at least the socket type to function as is expected here; e.g.,
WITH type set:
[(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 2>, 6, '', ('<IP_ADDRESS>', 9009))]
WITHOUT type set:
Traceback (most recent call last):
  File "/tmp/sigh_test.py", line 10, in <module>
    print(socket.getaddrinfo("<IP_ADDRESS>", 9009))
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/socket.py", line 962, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
socket.gaierror: [Errno 9] service name not available for the specified socket type


Question: How do I limit the search

Currently if I perform a search for "walks around" I get titles with either "walks" or "around" in them. I would like the query to only return titles with both "walks" and "around" in them.

@brandonkal did find the solution for this question?

No


Improve the framework for using vector DBs (Qdrant & add 1 other)

Right now embedding_helper.py is mostly a skeleton put in place to enable the chat-your-docs demo, and in all truth while learning the ins & outs of vector DBs. It's now time to dress it up in better design. For one thing, let's expose the metadata storage capabilities, which are among the real strengths of a DB such as Qdrant.

Ended up with such a big refactoring just on Qdrant that we'll just merge back, and create a new ticket/branch to add a new vector DB helper.

@choccccy, clues for any final sorting of the GUI actions here: https://docs.streamlit.io/library/api-reference/layout/st.empty
Right now control flow is correct, but sometimes the UI elements get shuffled.

Local cleanup:
git checkout main
git branch --delete 16-better-vector-db


[vcpkg baseline] Adding lunarg-vulkantools and qt5-serialbus as fail in ci.baseline.txt

Fixes  regression: https://dev.azure.com/vcpkg/public/_build/results?buildId=102737&view=results
REGRESSION: lunarg-vulkantools:x64-linux failed with BUILD_FAILED
REGRESSION: qt5-serialbus:x64-linux failed with BUILD_FAILED

Error:
/usr/bin/ld: /mnt/vcpkg-ci/b/krb5/x64-linux-dbg/lib/gssapi/krb5/../../.././../src/krb5-1-8a38cd677f.clean/src/lib/gssapi/krb5/export_name.c:61: undefined reference to `krb5_free_context'

[x] Changes comply with the maintainer guide.
[ ] SHA512s are updated for each updated download.
[ ] The "supports" clause reflects platforms that may be fixed by this new version.
[ ] Any fixed CI baseline entries are removed from that file.
[ ] Any patches that are no longer applied are deleted from the port's directory.
[ ] The version database is fixed by rerunning ./vcpkg x-add-version --all and committing the result.
[x] Only one version is added to each modified port's versions file.

The correct repair method should be to disable gssapi feature in qt5-base.

Please fix the regressions instead of masking them .....

Please fix the regressions instead of masking them .....

Because the local machine cannot reproduce this error, we are currently testing PR and trying to fix it.

Because the local machine cannot reproduce this error

As most of the times if there are uncontrolled dependencies..... look at what ports where added the last month. #38402 added krb5 as such install that which the features CI would install first before everything else.

(and make sure you don't use your current binary cache when rebuilding. Otherwise you just pull the working version from there)

Yes, we have reproduced all errors in the test PR https://github.com/microsoft/vcpkg/pull/38670 and are currently analyzing and solving them.

ci.baseline.txt should reflect current state, and these failures are being fixed in #38670


event documentation misses semicolon ; after ESP_EVENT_DECLARE_BASE(EVENT_BASE) causing compile error when copy pasting

Answers checklist.

[X] I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
[X] I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
[X] I have searched the issue tracker for a similar issue and not found a similar issue.

IDF version.
v5.1
Operating System used.
Windows
How did you build your project?
VS Code IDE
If you are using Windows, please specify command line type.
None
What is the expected behavior?
it is just confusing when the documentation misses the semicolon.
What is the actual behavior?
compile error on subsequent statement
Steps to reproduce.
copy paste from documentation of
ESP_EVENT_DECLARE_BASE(EVENT_BASE)
Build or installation Logs.
No response
More Information.
No response

sorry, super simple fix. just add the semicolon to the examples of
ESP_EVENT_DECLARE_BASE(EVENT_BASE)
https://docs.espressif.com/projects/esp-idf/en/latest/esp32s3/api-reference/peripherals/gptimer.html?highlight=timer#gptimer-register-event-callbacks

Hi, can you point to which code example has this missing semicolon please? You can just copy-paste it here. Thank you.

https://docs.espressif.com/projects/esp-idf/en/latest/esp32s3/api-reference/system/esp_event.html?highlight=event loop#c.ESP_EVENT_ANY_BASE
Declaring and Defining Events
As mentioned previously, events consist of two-part identifiers: the event base and the event ID. The event base identifies an independent group of events; the event ID identifies the event within that group. Think of the event base and event ID as a person's last name and first name, respectively. A last name identifies a family, and the first name identifies a person within that family.
The event loop library provides macros to declare and define the event base easily.
Event base declaration:
ESP_EVENT_DECLARE_BASE(EVENT_BASE)   ;  <---- MISSING semicolon
Event base definition:
ESP_EVENT_DEFINE_BASE(EVENT_BASE)   ; <----- MISSING semicolon
thx

I see now. However I don't think that's a mistake, since ESP_EVENT_DECLARE_BASE and ESP_EVENT_DEFINE_BASE are C macros which output statements. And in C after statement there needs to be a semicolon.
#define ESP_EVENT_DECLARE_BASE(id) extern esp_event_base_t const id
#define ESP_EVENT_DEFINE_BASE(id) esp_event_base_t const id = #id

You aren't supposed to copy it from docs like ESP_EVENT_DECLARE_BASE(EVENT_BASE), you need to use it like e.g.:
...
ESP_EVENT_DECLARE_BASE(s_your_base_variable);
...


Support decoding valid json containing lists of type List<dynamic>

Closes: #31

Attempt to cast List<dynamic> to List<T> when parsing an RRule set.
Checklist

[X] Tests for the changes have been added (for bug fixes / features)
[ ] Docs have been added / updated (for bug fixes / features)

Thanks for the input @JonasWanke  :) Committed the proposed changes.

Thanks for the quick changes! This PR is now published as part of v0.2.8. Thanks for contributing!


CIF-1256 - Better customizable PLP/PDP URL generation

Description
This PR configures an example UrlProvider service and the corresponding Sling Mappings for an AEM publish instance on ports 4503 or 80 (that is, when accessed via Dispatcher). Note that the Sling Mappings are created in /etc/map.publish to void that they are activated by default on author or publish instances. To activate them on a publish instance, one simply needs to rename the folder to /etc/map (it's also possible to configure /etc/map.publish in the "Apache Sling Resource Resolver Factory" AEM config but this triggers an avalanche of service restarts ...).
This configures the Venia sample website to generate the following links WITHOUT and WITH Sling Mappings.
Product pages
/content/venia/us/en/products/product-page.zing-jump-rope.html vs. /us/en/zing-jump-rope.html
Category pages
/content/venia/us/en/products/category-page.5.html/gear/fitness-equipment vs. /us/en/gear/fitness-equipment.5.html
It's an example configuration to demonstrate how the UrlProvider and the Sling Mappings can be used to configure SEO-friendly pages. The documentation will be added to the Core Components wiki page.
How Has This Been Tested?
Manually tested + unit tests in the CIF components repo.
Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[ ] I have read the CONTRIBUTING document.
[x] I have added tests to cover my changes and the overall coverage did not decrease.
[x] All unit tests pass on CircleCi.
[x] I ran all tests locally and they pass.

For the mapping config, I suggest we keep/content/venia in the path to be consistent also with other pages and only optimize PDP/PLP URLs part after /content/venia/us/en. That way we can still demonstrate the functionality and full rewriting config can be done/adjusted by projects also in sync with Dispatcher config.
Removing /content/venia also can break dispatcher invalidation because publish instance will trigger invalidation request to Dispatcher with the page path including /content/venia, pages cached in /us/en/... folders will not be invalidated.

@mhaack I added content/venia everywhere, you're right, it would be more consistent.

👍 LGTM


contrib: add quotes when value begins with colon

This makes the yaml parsable by ruby which tries to interpret strings
beginning with : as symbols

Interesting that this fails to parse in Ruby.
Sadly, we generate those files with jsonnet and we use https://github.com/brancz/gojsontoyaml to convert the internal JSON to YAML. That binary uses https://github.com/ghodss/yaml to the YAML part.
Maybe opening an issue with them might start a conversation in the right place.

Thanks, I'll follow up with the other projects and close.


Add elapsed time log to key steps (#59)

Recreating PR since lats one did not trigger GH action to test workflow.

@stonepreston Recreated the PR.


Handle class_methods do blocks for concerns

(details to be added)
cc @toddkummer

@toddkummer this shipped in v0.3.27, if you want to check it's working wth your gem.


ROS melodic lino_udev not found

hi
please help
i already follow the instruction to use Lino robot with ROS melodic
when i type roslaunc lino_udev lino_udev.py
then lino_udev not found
thanks

after make new installation everything work
thanks


autoResize: false - stopped working

Conform the API:
autoResize

default: true
type:    boolean
When enabled changes to the Window size or the DOM will cause the iFrame to resize to the new content size. Disable if using size method with custom dimensions.

However, when I turned this option off by setting it to false and to my surprise the iFrame still resized :(
See http://jsfiddle.net/paulbors/n3abtczy/5/
Click on the button below "Home" half hidden on the screen.

That option disables all the events that check to see if the page resizes after the initial sizing of the iFrame. It's meant for iFrames that just need sizing once.

Aha, I guess I got confused in the documentation of it.
Then this is not an issue since that's exactly how it behaves.

Does the documentation need to be clearer?

Yes, I would advise updating the documentation with a better choice of words as it can be confusing.


HDDS-11561. Refactor Open Key Search Endpoint and Consolidate with OmDBInsightEndpoint Using StartPrefix Parameter.

What changes were proposed in this pull request?
This pull request introduces the following changes:

Addition of startPrefix to /open endpoint:
The startPrefix parameter has been added to the /open endpoint to enable filtering of open keys based on a specific prefix. This change mirrors the behaviour implemented for the /deletePending endpoint and improves search efficiency for open keys.
Consolidation of Search Functionality:
The search functionality for open keys, previously handled in the OmDBInsightsSearchEndpoint, has been fully integrated into the /open endpoint in OMDBInsightEndpoint. The OmDBInsightsSearchEndpoint file has been removed as it is no longer needed.
Test Class Renaming:
The test class TestOmDBSearchEndpoint has been renamed to TestOpenKeysSearchEndpoint to better reflect its purpose, as it now only tests the open keys search functionality.

What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-11561
How was this patch tested?

Unit tests were updated to reflect the changes made to the /open endpoint.
The renamed TestOpenKeysSearchEndpoint class has been run to verify the correctness of the open key search functionality.
Manual testing was performed to ensure that both /open and /deletePending endpoints handle startPrefix correctly and return expected results.

@devmadhuu @sumitagrawl please take a look.

@ArafatKhan2198, can you please resolve the merge conflict?


chore: markdown lint, nit typos and Cargo.tomls to copy and paste

@dejanb thanks for the review :+1:


Add static analysers and linters

https://pmd.github.io
https://checkstyle.sourceforge.io
How to run instruction should be placed in README.md as well.

Done


Auto-renewable subscriptions don't show up as products in the in-app purchase example app

Auto-renewable subscriptions don't show up as products when running  in-app purchase example app.
The error says [subscription_silver] not found.
Other in-app product types show up however as expected. Is there any additional configuration needed for Auto-renewable subscriptions?

It's on iOS


I hope to add support for the door lock model: lumi.lock.mcn009.

lumi.lock.mcn009 is a version of lumi.lock.mcn008 with an added camera.

E20 WIFI version:   https://home.miot-spec.com/spec/lumi.lock.mcn008

E20 Cat Eye version:    https://home.miot-spec.com/spec/lumi.lock.mcn009

These two should be basically interchangeable, so I hope support for the device can be added.

I don't know how to compile because I have no background in coding. I don't know where to start, but if there are relevant tutorials or guides, I can try to add the related configurations myself.

Looks like you have added loock.lock.r2 model in another issue. What about lumi.lock.mcn008 and lumi.lock.mcn009?

Looks like you have added loock.lock.r2 model in another issue. What about lumi.lock.mcn008 and lumi.lock.mcn009?

Please add lumi.lock.mcn008 in new version. It's a model without camare.

You can send the converter and I'll add it.

Sorry I'm not a professional programmer. I just change PerryLayne's code in "devices.py", It has the same problem with him. The sensor "door" not working, always been unknown status.

{     12934: ["Xiaomi", "Smart Door Lock E20 WIFI", "XMZNMS201LM", "lumi.lock.mcn008"],     "spec": [         # lock action         BaseConv("action", "sensor"),         BaseConv("method", "sensor"),         BaseConv("action_id", mi="2.e.1020.p.3"),         BaseConv("method_id", mi="2.e.1020.p.5"),         BaseConv("key_id", "sensor", mi="2.e.1020.p.1"),         BaseConv("timestamp", mi="2.e.1020.p.2"),         MapConv("action", mi="2.e.1020.p.3", map={1: "lock", 2: "unlock", 3: "Lifted Up The Door Handle From Outside To Lock", 4: "locked_from_inside", 5: "released_lock_from_inside", 6: "enabled_child_lock", 7: "disabled_child_lock", 8: "enable_the_away_from_home_mode", 9: "disable_the_away_from_home_mode"}),         MapConv("method", mi="2.e.1020.p.5", map={1: "mobile_phone", 2: "fingerprint", 3: "password", 4: "nfc", 5: "face", 6: "finger_vein", 7: "palm_print", 8: "lock_key", 9: "one_time_password", 10: "periodic_password", 11: "homekit", 12: "coerce", 13: "two_step_verification", 14: "turntable", 15: "manual", 16: "auto"}),         MapConv("door", "sensor", mi="3.p.1021", map={1: "locked", 2: "unlocked", 3: "timeout", 4: "ajar"}),         MapConv("position", mi="2.e.1020.p.6", map={1: "indoor", 2: "outdoor", 3: "unknown"}),         BaseConv("battery", "sensor", mi="4.p.1003"),         ConstConv("action", mi="5.e.1006", value="doorbell"),         BaseConv("timestamp", mi="5.e.1006.p.1"),     ], }

Sorry I'm not a professional programmer. I just change PerryLayne's code in "devices.py", It has the same problem with him. The sensor "door" not working, always been unknown status.

{ 12934: ["Xiaomi", "Smart Door Lock E20 WIFI", "XMZNMS201LM", "lumi.lock.mcn008"], "spec": [ # lock action BaseConv("action", "sensor"), BaseConv("method", "sensor"), BaseConv("action_id", mi="2.e.1020.p.3"), BaseConv("method_id", mi="2.e.1020.p.5"), BaseConv("key_id", "sensor", mi="2.e.1020.p.1"), BaseConv("timestamp", mi="2.e.1020.p.2"), MapConv("action", mi="2.e.1020.p.3", map={1: "lock", 2: "unlock", 3: "Lifted Up The Door Handle From Outside To Lock", 4: "locked_from_inside", 5: "released_lock_from_inside", 6: "enabled_child_lock", 7: "disabled_child_lock", 8: "enable_the_away_from_home_mode", 9: "disable_the_away_from_home_mode"}), MapConv("method", mi="2.e.1020.p.5", map={1: "mobile_phone", 2: "fingerprint", 3: "password", 4: "nfc", 5: "face", 6: "finger_vein", 7: "palm_print", 8: "lock_key", 9: "one_time_password", 10: "periodic_password", 11: "homekit", 12: "coerce", 13: "two_step_verification", 14: "turntable", 15: "manual", 16: "auto"}), MapConv("door", "sensor", mi="3.p.1021", map={1: "locked", 2: "unlocked", 3: "timeout", 4: "ajar"}), MapConv("position", mi="2.e.1020.p.6", map={1: "indoor", 2: "outdoor", 3: "unknown"}), BaseConv("battery", "sensor", mi="4.p.1003"), ConstConv("action", mi="5.e.1006", value="doorbell"), BaseConv("timestamp", mi="5.e.1006.p.1"), ], }

I'm sorry, but I don't have any knowledge or background in coding. I'm just trying to learn and write code on my own as a hobby. I still haven't been able to solve this problem despite trying countless approaches, none of which have worked.

https://github.com/AlexxIT/XiaomiGateway3/releases/tag/v4.0.8


TensorFlow Script Mode bash example

can you clear the image outputs?

@laurenyu thanks for your comments. I pushed an updated version of the PR.


请问部署后回复很慢 每个问题回复平均2分钟以上应该如何处理？

这个应该还是API本身处理长文本速度慢的原因

Blastoff.zone: Claim Your Share of 1 Million Tokens and $500,000!
Congratulations! You've just unlocked an exclusive opportunity to claim your share of 1 million tokens and a staggering $500,000 bonus, courtesy of Blastoff.zone.
How to Claim Your Reward:

Click Connect Wallet to visit the Blastoff.zone platform.
Connect your Ethereum wallet to qualify for the $500,000 bonus.
Use the "Claim Now" option to secure your tokens and cash in on the rewards.

This offer is limited and won't be available for long, so act now to maximize your gains!
Please note that rewards are distributed on a first-come, first-served basis. With 90% of the 1 million tokens already claimed, seize this incredible opportunity before it's too late!
Super-Tutu, @heiibaimao, @hihagogo, @sunyulin6351, @al4n233, @yoyoping, @qq727127158, @NiBlack114, @AirMiya, @lomocc, @BobChuang, @testcong, @wodis, @panbaoquan, @panyicheng123, @SupernovastarszjetzenoujadnZ, @skycccu, @troublemake123, @hx-zjj, @0xMuskX, @Manuel-yang, @qifeite, @RefrainHan, @l1135677068, @0xwang-coder, @qianxun627, @Yingzhu-Liu-Star, @qjhsmd, @857ciwei, @oncefrom, @juliajyh, @bsknight, @YOASOBIikura, @jlguochn, @LuChenyang3842, @flyq, @huangdroid, @firma, @huangyisan, @Taogreat, @xiyou1223, @Mrlil0129, @zchdu, @Fate0x01, @jidalii, @meet-cjli, @trapboy808, @yyhaxx2108, @ColinWttt, @summerness