cudaPackages: multiple outputs for redistributables

Description of changes

General idea:

Find out, ahead of time, what "features" each package provides.

"Feature" means library (static or dynamic), headers, binaries, samples, or documentation.
This is handled by the find_outputs Python script.

Have Nix create an output for each feature.

Use the complementary manifest output file generated by find_outputs to avoid requiring IFD.
Use the same logic behind symlinkJoin to ensure the default output, out, is the union of all other outputs.

This effectively makes the new outputs "opt-in"

Some numbers, all found by building against master and this branch, with --impure and a local Nixpkgs configuration of
{
  allowUnfree = true;
  cudaSupport = true;
  cudaCapabilities = [ "8.9" ];
  cudaForwardCompat = false;
}

cuDNN closure/NAR size went from 2.4G to 1.1G.

Comparing master's cudnn to this branch's cudnn.lib.

Magma closure size went from 2.9G to 1.6G.

Comparing master with this branch (same attribute name).
Build targeted capability 8.9 (Ada Lovelace) with CUDA 11.8.
Mostly as a result of dropping dependencies on static cuBLAS and cuSPARSE.

PyTorch closure size went from 9.7G to 8.4G.

Comparing master with this branch (same attribute name).
Build targeted capability 8.9 (Ada Lovelace) with CUDA 11.8.
Much greater gains to be had in migrating away from CUDA Toolkit.

Decrease comes entirely from switch to cudnn.lib instead of cudnn.
Magma does not contribute to the closure size, because we link against it statically.

Additional changes:

Overrides for CUDA 12.x packages to ensure patchelf succeeds.
Refactored overrides for CUDA 12.x packages to use only the outputs they require from other CUDA packages.
autoPatchelfIgnoreMissingDeps set to ignore libcuda.so.1 specifically instead of all libs, removing the possibility of breakages by accidentally ignoring libraries we should be linking.
Moved addBuildInputs to all site.

Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Wow, this is huge (no pun intended xD)!
I suppose this PR is to be reviewed later, after you mark it ready? Just a comment before then: it seems like this change deals with aarch64 support, cross-compilation support, and splayed outputs support at once. When we start merging this, it might be easier to merge in smaller chunks? I'd also be wary of exposing addBuildInputs&c as attributes: pipe looks like a great improvement compositionality-wise, but it's also important that our codebase doesn't end up looking alien to a randomly sampled nixpkgs maintainer. I think this usage of pipe is kind of similar to that of the module system's lib.mkMerge, which we might be able to move to in time.

Wow, this is huge (no pun intended xD)!

You're right, there was a fair amount of scope creep as I was working on this. It needs to be split up.

I suppose this PR is to be reviewed later, after you mark it ready?

Yep! I try not to mark PRs as ready for review until I'm near-done or want to make it a collaborative effort/trigger some CI checks.

Just a comment before then: it seems like this change deals with aarch64 support, cross-compilation support, and splayed outputs support at once. When we start merging this, it might be easier to merge in smaller chunks?

Definitely! I can see a few different components in this:

Multi-output CUDA redistributables

The actual goal of this PR

Multi-arch support

Support for just Jetson is a specific code-path in the more general multi-arch support feature

Cross-compilation support

Implemented through splicing

I'd also be wary of exposing addBuildInputs&c as attributes: pipe looks like a great improvement compositionality-wise, but it's also important that our codebase doesn't end up looking alien to a randomly sampled nixpkgs maintainer. I think this usage of pipe is kind of similar to that of the module system's lib.mkMerge, which we might be able to move to in time.

I haven't played much with the module system -- most of the inspiration I took came from utility functions the Haskell packaging has: https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/haskell-modules/lib/compose.nix.
There's a lot of boilerplate in what's being done in the overrides which is why I started introducing those abstractions.
As I was writing some of this code I caught myself trying to make everything point-free (a bad habit from my Haskell tendencies).
You're right of course that addBuildInputs and friends should be much more limited in scope.

Focused only on multiple outputs now and rebased on master.
Prior to stripping the multiple arch work, I made a copy of it here: https://github.com/ConnorBaker/nixpkgs/tree/feat/cuda-redist-multiple-arch.

Result of nixpkgs-review pr 240498 --extra-nixpkgs-config '{   allowUnfree = true;   cudaSupport = true;   cudaCapabilities = [ "8.9" ];   cudaForwardCompat = false; }' run on x86_64-linux 1

9 packages marked as broken and skipped:

cudaPackages.nvidia_driver
cudaPackages.nvidia_driver.bin
cudaPackages.nvidia_driver.lib
python310Packages.caffeWithCuda
python310Packages.caffeWithCuda.bin
python310Packages.theanoWithCuda
python310Packages.theanoWithCuda.dist
python311Packages.theanoWithCuda
python311Packages.theanoWithCuda.dist

21 packages failed to build:

caffeWithCuda
caffeWithCuda.bin
cudaPackages.cuda_demo_suite
cudaPackages.cuda_documentation
cudaPackages.nvidia_fs
cudaPackages.tensorrt (cudaPackages.tensorrt_8_5_3)
cudaPackages.tensorrt.dev (cudaPackages.tensorrt_8_5_3.dev)
cudaPackages.tensorrt_8_5_1
cudaPackages.tensorrt_8_5_1.dev
cudaPackages.tensorrt_8_5_2
cudaPackages.tensorrt_8_5_2.dev
cudaPackages.tensorrt_8_6_0
cudaPackages.tensorrt_8_6_0.dev
katagoTensorRT
mathematica-cuda
python310Packages.tensorrt
python310Packages.tensorrt.dist
python311Packages.caffeWithCuda
python311Packages.caffeWithCuda.bin
python311Packages.tensorrt
python311Packages.tensorrt.dist

89 packages built:

cudaPackages.cuda_cccl (cudaPackages.cuda_cccl.dev)
cudaPackages.cuda_cudart (cudaPackages.cuda_cudart.dev ,cudaPackages.cuda_cudart.lib ,cudaPackages.cuda_cudart.static)
cudaPackages.cuda_cuobjdump (cudaPackages.cuda_cuobjdump.bin)
cudaPackages.cuda_cupti (cudaPackages.cuda_cupti.dev ,cudaPackages.cuda_cupti.lib ,cudaPackages.cuda_cupti.sample ,cudaPackages.cuda_cupti.static)
cudaPackages.cuda_cuxxfilt (cudaPackages.cuda_cuxxfilt.bin ,cudaPackages.cuda_cuxxfilt.dev ,cudaPackages.cuda_cuxxfilt.static)
cudaPackages.cuda_gdb (cudaPackages.cuda_gdb.bin)
cudaPackages.cuda_memcheck (cudaPackages.cuda_memcheck.bin)
cudaPackages.cuda_nsight (cudaPackages.cuda_nsight.bin)
cudaPackages.cuda_nvcc (cudaPackages.cuda_nvcc.bin ,cudaPackages.cuda_nvcc.dev ,cudaPackages.cuda_nvcc.static)
cudaPackages.cuda_nvdisasm (cudaPackages.cuda_nvdisasm.bin)
cudaPackages.cuda_nvml_dev (cudaPackages.cuda_nvml_dev.dev ,cudaPackages.cuda_nvml_dev.lib)
cudaPackages.cuda_nvprof (cudaPackages.cuda_nvprof.bin ,cudaPackages.cuda_nvprof.lib)
cudaPackages.cuda_nvprune (cudaPackages.cuda_nvprune.bin)
cudaPackages.cuda_nvrtc (cudaPackages.cuda_nvrtc.dev ,cudaPackages.cuda_nvrtc.lib ,cudaPackages.cuda_nvrtc.static)
cudaPackages.cuda_nvtx (cudaPackages.cuda_nvtx.dev ,cudaPackages.cuda_nvtx.lib)
cudaPackages.cuda_nvvp (cudaPackages.cuda_nvvp.bin)
cudaPackages.cuda_profiler_api (cudaPackages.cuda_profiler_api.dev)
cudaPackages.cuda_sanitizer_api (cudaPackages.cuda_sanitizer_api.bin)
cudaPackages.cudnn (cudaPackages.cudnn.dev ,cudaPackages.cudnn.lib ,cudaPackages.cudnn.static ,cudaPackages.cudnn_8_9 ,cudaPackages.cudnn_8_9.dev ,cudaPackages.cudnn_8_9.lib ,cudaPackages.cudnn_8_9.static)
cudaPackages.cudnn_8_6 (cudaPackages.cudnn_8_6.dev ,cudaPackages.cudnn_8_6.lib ,cudaPackages.cudnn_8_6.static)
cudaPackages.cudnn_8_7 (cudaPackages.cudnn_8_7.dev ,cudaPackages.cudnn_8_7.lib ,cudaPackages.cudnn_8_7.static)
cudaPackages.cudnn_8_8 (cudaPackages.cudnn_8_8.dev ,cudaPackages.cudnn_8_8.lib ,cudaPackages.cudnn_8_8.static)
cudaPackages.fabricmanager (cudaPackages.fabricmanager.bin ,cudaPackages.fabricmanager.dev ,cudaPackages.fabricmanager.lib)
cudaPackages.libcublas (cudaPackages.libcublas.dev ,cudaPackages.libcublas.lib ,cudaPackages.libcublas.static)
cudaPackages.libcufft (cudaPackages.libcufft.dev ,cudaPackages.libcufft.lib ,cudaPackages.libcufft.static)
cudaPackages.libcufile (cudaPackages.libcufile.dev ,cudaPackages.libcufile.lib ,cudaPackages.libcufile.sample ,cudaPackages.libcufile.static)
cudaPackages.libcurand (cudaPackages.libcurand.dev ,cudaPackages.libcurand.lib ,cudaPackages.libcurand.static)
cudaPackages.libcusolver (cudaPackages.libcusolver.dev ,cudaPackages.libcusolver.lib ,cudaPackages.libcusolver.static)
cudaPackages.libcusparse (cudaPackages.libcusparse.dev ,cudaPackages.libcusparse.lib ,cudaPackages.libcusparse.static)
cudaPackages.libnpp (cudaPackages.libnpp.dev ,cudaPackages.libnpp.lib ,cudaPackages.libnpp.static)
cudaPackages.libnvidia_nscq (cudaPackages.libnvidia_nscq.lib)
cudaPackages.libnvjpeg (cudaPackages.libnvjpeg.dev ,cudaPackages.libnvjpeg.lib ,cudaPackages.libnvjpeg.static)
cudaPackages.nccl
cudaPackages.nccl.dev
cudaPackages.nsight_compute (cudaPackages.nsight_compute.bin)
cudaPackages.nsight_systems (cudaPackages.nsight_systems.bin)
faissWithCuda
faissWithCuda.demos
katagoWithCuda
magma (magma-cuda ,magma_2_7_1)
magma-cuda-static
magma_2_6_2
nvidia-thrust-cuda
python310Packages.bitsandbytes
python310Packages.bitsandbytes.dist
python310Packages.cupy
python310Packages.cupy.dist
python310Packages.encodec
python310Packages.encodec.dist
python310Packages.jaxlibWithCuda
python310Packages.jaxlibWithCuda.dist
python310Packages.openai-triton
python310Packages.openai-triton-bin
python310Packages.openai-triton-bin.dist
python310Packages.openai-triton.dist
python310Packages.tensorflowWithCuda
python310Packages.tensorflowWithCuda.dist
python310Packages.tiny-cuda-nn
python310Packages.torchWithCuda
python310Packages.torchWithCuda.dev
python310Packages.torchWithCuda.dist
python310Packages.torchWithCuda.lib
python310Packages.torchWithRocm
python310Packages.torchWithRocm.dev
python310Packages.torchWithRocm.dist
python310Packages.torchWithRocm.lib
python310Packages.torchaudio-bin
python310Packages.torchaudio-bin.dist
python310Packages.torchvision-bin
python310Packages.torchvision-bin.dist
python310Packages.trainer
python310Packages.trainer.dist
python311Packages.cupy
python311Packages.cupy.dist
python311Packages.encodec
python311Packages.encodec.dist
python311Packages.jaxlibWithCuda
python311Packages.jaxlibWithCuda.dist
python311Packages.openai-triton
python311Packages.openai-triton-bin
python311Packages.openai-triton-bin.dist
python311Packages.openai-triton.dist
python311Packages.torchaudio-bin
python311Packages.torchaudio-bin.dist
python311Packages.torchvision-bin
python311Packages.torchvision-bin.dist
tiny-cuda-nn
tts
tts.dist

Failed derivations

[ ] Mathematica_13.3.0_BNDL_LINUX.sh.drv (derivation hash: 7wzwnszjqipv89adrssk22ri38msfg4f)

https://gist.github.com/ConnorBaker/d6f89ef3100596d7ff5facffe7ac0427

[ ] TensorRT-<IP_ADDRESS>.Linux.x86_64-gnu.cuda-11.8.cudnn8.6.tar.gz.drv (derivation hash: s39qfr7dfccm8xssjl1120xjzix01m0p)

https://gist.github.com/ConnorBaker/95b15cf21f1ef63f02345ff848d09932

[ ] TensorRT-<IP_ADDRESS>.Linux.x86_64-gnu.cuda-11.8.cudnn8.6.tar.gz.drv (derivation hash: dlpp7vgdfk5qwp61wkyi7pmczgyml224)

https://gist.github.com/ConnorBaker/b4aff4fc3c943a068fcb06d5efc35a57

[ ] TensorRT-<IP_ADDRESS>.Linux.x86_64-gnu.cuda-11.8.cudnn8.6.tar.gz.drv (derivation hash: xycvfh5gj7rq51ij9dyk43wcffddcd2g)

https://gist.github.com/ConnorBaker/a085e37953408b60149be8e0dec1b288

[ ] TensorRT-<IP_ADDRESS>.Linux.x86_64-gnu.cuda-11.8.tar.gz.drv (derivation hash: rjr5gr05bm11sb00n4zamhy2c2l3h8z5)

https://gist.github.com/ConnorBaker/a82b369f62aee09ef2601d8f18f33fe5

[ ] cuda_documentation-11.8.86.drv (derivation hash: yyyj1j3xiy6zn7vlqc7byw161m1s7h50)

https://gist.github.com/ConnorBaker/97bbc9264b5f27b0ddf65ea02c209984

[ ] nvidia_fs-2.13.5.drv (derivation hash: 83w0w33iglsixz89jcdqyy6jddaim6z6)

https://gist.github.com/ConnorBaker/7387bf1dd71437b714df5f5c40dba7c7

[ ] cuda_demo_suite-11.8.86.drv (derivation hash: 5c3vzbnc85dzf6h1sfg3kb6hg142ahbw)

https://gist.github.com/ConnorBaker/48a55697d797d138e0a1aec664ad5684

[ ] cudatoolkit-10.1.243.drv (derivation hash: 478ghyg23khgp97nqwhnry0lpvidqy9g)

https://gist.github.com/ConnorBaker/03428498610ad3c87a0e8d4b18368a3e

PR is marked as draft, but I was requested to review, so leaving a few comments here.

Thank you for taking a look! You were incredibly thorough and I really appreciate that :)

Overall, I like the idea of reducing closure sizes but these changes do introduce a substantial amount of new code and additional complexity. I feel obligated to raise the question of whether this additional complexity is worth it. Reducing the pytorch closure size from 9.7G -> 8.4G is nice, but there's a lot of new code to maintain here.

The PyTorch closure size reduction is due solely to splitting cuDNN's static libraries out. Because it still relies on the CUDA Toolkit, we haven't seen any big reductions in closure size, but we will when we switch to the redistributables.

A few general thoughts, poorly organized:

find_outputs has far more code than I initially expected based on the functionality described in the README (Huge thanks for including a README btw!!). Could find_outputs be simplified -- possibly into a single script?
Is find_outputs necessary? Why not have each redist package offer all outputs, with logic to copy things into place. Some of these output directories would end up empty, but that doesn't seem problematic.

Perhaps I'm missing something?

You're right, find_outputs (now cuda-redist-find-features) is fairly large for what it does. Part of that is because I tend to split out Python projects into many smaller modules, and part of that is because of the organic growth of the script over time.
Although, going forward, more functionality will likely be added to it. @SomeoneSerge and I had discussed including information about the architectures each package supports (or if they support any at all; some have no device code!). The motivation for this particular addition is the ability to substitute linux-sbsa (ARM Linux server) packages for linux-aarch64 (NVIDIA Jetson devices) when it does not have its own package.
Personally, I'm strongly opposed to empty outputs. I feel that outputs signal the components a package provides, and that having an empty output is misleading. Aside from that, the multiple-outputs setup hook automatically deletes empty directories (including outputs): https://github.com/NixOS/nixpkgs/blob/7e67ece86476a82ea0e663c20c8190754c02d447/pkgs/build-support/setup-hooks/multiple-outputs.sh#L140 so I would need to patch that, or re-create the empty outputs in a very late stage.

Squashed, rebased, and force-pushed.

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/tweag-nix-dev-update-51/30870/1

Marked as ready for review. Still running nixpkgs-review on this, but it looks mostly clean. Aside from the usual TensorRT and Mathematica failures, I'm seeing a cudatoolkit-10.1 and cudatoolkit-10.2 failure, and failures related to xformers. I'll see if I can reproduce those on master.

Result of nixpkgs-review pr 240498 --extra-nixpkgs-config '{ allowUnfree = true; cudaSupport = true; cudaCapabilities = [ "8.9" ]; cudaForwardCompat = false; }' run on x86_64-linux 1

10 packages marked as broken and skipped:

cudaPackages.nvidia_driver
cudaPackages.nvidia_driver.bin
cudaPackages.nvidia_driver.lib
python310Packages.caffeWithCuda
python310Packages.caffeWithCuda.bin
python310Packages.theanoWithCuda
python310Packages.theanoWithCuda.dist
python311Packages.theanoWithCuda
python311Packages.theanoWithCuda.dist
truecrack-cuda

31 packages failed to build:

caffeWithCuda
caffeWithCuda.bin
cudaPackages.nsight_systems (cudaPackages.nsight_systems.bin)
cudaPackages.tensorrt (cudaPackages.tensorrt_8_5_3)
cudaPackages.tensorrt.dev (cudaPackages.tensorrt_8_5_3.dev)
cudaPackages.tensorrt_8_5_1
cudaPackages.tensorrt_8_5_1.dev
cudaPackages.tensorrt_8_5_2
cudaPackages.tensorrt_8_5_2.dev
cudaPackages.tensorrt_8_6_0
cudaPackages.tensorrt_8_6_0.dev
gpu-screen-recorder
gpu-screen-recorder-gtk
katagoTensorRT
mathematica-cuda
python310Packages.tensorrt
python310Packages.tensorrt.dist
python310Packages.xformers
python310Packages.xformers.dist
python311Packages.caffeWithCuda
python311Packages.caffeWithCuda.bin
python311Packages.tensorflowWithCuda
python311Packages.tensorflowWithCuda.dist
python311Packages.tensorrt
python311Packages.tensorrt.dist
python311Packages.torchWithRocm
python311Packages.torchWithRocm.dev
python311Packages.torchWithRocm.dist
python311Packages.torchWithRocm.lib
python311Packages.xformers
python311Packages.xformers.dist

136 packages built:

colmapWithCuda
cudaPackages.cuda-samples
cudaPackages.cuda_cccl (cudaPackages.cuda_cccl.dev)
cudaPackages.cuda_cudart (cudaPackages.cuda_cudart.dev ,cudaPackages.cuda_cudart.lib ,cudaPackages.cuda_cudart.static)
cudaPackages.cuda_cuobjdump (cudaPackages.cuda_cuobjdump.bin)
cudaPackages.cuda_cupti (cudaPackages.cuda_cupti.dev ,cudaPackages.cuda_cupti.lib ,cudaPackages.cuda_cupti.sample ,cudaPackages.cuda_cupti.static)
cudaPackages.cuda_cuxxfilt (cudaPackages.cuda_cuxxfilt.bin ,cudaPackages.cuda_cuxxfilt.dev ,cudaPackages.cuda_cuxxfilt.static)
cudaPackages.cuda_demo_suite
cudaPackages.cuda_documentation
cudaPackages.cuda_gdb (cudaPackages.cuda_gdb.bin)
cudaPackages.cuda_memcheck (cudaPackages.cuda_memcheck.bin)
cudaPackages.cuda_nsight (cudaPackages.cuda_nsight.bin)
cudaPackages.cuda_nvcc (cudaPackages.cuda_nvcc.bin ,cudaPackages.cuda_nvcc.dev ,cudaPackages.cuda_nvcc.static)
cudaPackages.cuda_nvdisasm (cudaPackages.cuda_nvdisasm.bin)
cudaPackages.cuda_nvml_dev (cudaPackages.cuda_nvml_dev.dev ,cudaPackages.cuda_nvml_dev.lib)
cudaPackages.cuda_nvprof (cudaPackages.cuda_nvprof.bin ,cudaPackages.cuda_nvprof.lib)
cudaPackages.cuda_nvprune (cudaPackages.cuda_nvprune.bin)
cudaPackages.cuda_nvrtc (cudaPackages.cuda_nvrtc.dev ,cudaPackages.cuda_nvrtc.lib ,cudaPackages.cuda_nvrtc.static)
cudaPackages.cuda_nvtx (cudaPackages.cuda_nvtx.dev ,cudaPackages.cuda_nvtx.lib)
cudaPackages.cuda_nvvp (cudaPackages.cuda_nvvp.bin)
cudaPackages.cuda_profiler_api (cudaPackages.cuda_profiler_api.dev)
cudaPackages.cuda_sanitizer_api (cudaPackages.cuda_sanitizer_api.bin)
cudatoolkit (cudaPackages.cudatoolkit ,cudatoolkit_11)
cudatoolkit.doc (cudaPackages.cudatoolkit.doc ,cudatoolkit_11.doc)
cudatoolkit.lib (cudaPackages.cudatoolkit.lib ,cudatoolkit_11.lib)
cudaPackages.cudnn (cudaPackages.cudnn.dev ,cudaPackages.cudnn.lib ,cudaPackages.cudnn.static ,cudaPackages.cudnn_8_9 ,cudaPackages.cudnn_8_9.dev ,cudaPackages.cudnn_8_9.lib ,cudaPackages.cudnn_8_9.static)
cudaPackages.cudnn_8_6 (cudaPackages.cudnn_8_6.dev ,cudaPackages.cudnn_8_6.lib ,cudaPackages.cudnn_8_6.static)
cudaPackages.cudnn_8_7 (cudaPackages.cudnn_8_7.dev ,cudaPackages.cudnn_8_7.lib ,cudaPackages.cudnn_8_7.static)
cudaPackages.cudnn_8_8 (cudaPackages.cudnn_8_8.dev ,cudaPackages.cudnn_8_8.lib ,cudaPackages.cudnn_8_8.static)
cudaPackages.cutensor
cudaPackages.cutensor.dev
cudaPackages.fabricmanager (cudaPackages.fabricmanager.bin ,cudaPackages.fabricmanager.dev ,cudaPackages.fabricmanager.lib)
cudaPackages.libcublas (cudaPackages.libcublas.dev ,cudaPackages.libcublas.lib ,cudaPackages.libcublas.static)
cudaPackages.libcufft (cudaPackages.libcufft.dev ,cudaPackages.libcufft.lib ,cudaPackages.libcufft.static)
cudaPackages.libcufile (cudaPackages.libcufile.dev ,cudaPackages.libcufile.lib ,cudaPackages.libcufile.sample ,cudaPackages.libcufile.static)
cudaPackages.libcurand (cudaPackages.libcurand.dev ,cudaPackages.libcurand.lib ,cudaPackages.libcurand.static)
cudaPackages.libcusolver (cudaPackages.libcusolver.dev ,cudaPackages.libcusolver.lib ,cudaPackages.libcusolver.static)
cudaPackages.libcusparse (cudaPackages.libcusparse.dev ,cudaPackages.libcusparse.lib ,cudaPackages.libcusparse.static)
cudaPackages.libnpp (cudaPackages.libnpp.dev ,cudaPackages.libnpp.lib ,cudaPackages.libnpp.static)
cudaPackages.libnvidia_nscq (cudaPackages.libnvidia_nscq.lib)
cudaPackages.libnvjpeg (cudaPackages.libnvjpeg.dev ,cudaPackages.libnvjpeg.lib ,cudaPackages.libnvjpeg.static)
cudaPackages.nccl
cudaPackages.nccl.dev
cudaPackages.nsight_compute (cudaPackages.nsight_compute.bin)
cudaPackages.nvidia_fs
cudaPackages.saxpy
cudaPackages.setupCudaHook
faissWithCuda
faissWithCuda.demos
forge
gpu-burn
gromacsCudaMpi
gwe
hip-nvidia
hip-nvidia.doc
katagoWithCuda
librealsenseWithCuda
librealsenseWithCuda.dev
magma (magma-cuda ,magma_2_7_1)
magma-cuda-static
magma_2_6_2
nvidia-thrust-cuda
nvtop
nvtop-nvidia
python310Packages.bitsandbytes
python310Packages.bitsandbytes.dist
python310Packages.cupy
python310Packages.cupy.dist
python310Packages.encodec
python310Packages.encodec.dist
python310Packages.jaxlibWithCuda
python310Packages.jaxlibWithCuda.dist
python310Packages.numbaWithCuda
python310Packages.numbaWithCuda.dist
python310Packages.openai-triton
python310Packages.openai-triton-bin
python310Packages.openai-triton-bin.dist
python310Packages.openai-triton.dist
python310Packages.pycuda
python310Packages.pycuda.dist
python310Packages.pynvml
python310Packages.pynvml.dist
python310Packages.pyrealsense2WithCuda
python310Packages.pyrealsense2WithCuda.dev
python310Packages.tensorflowWithCuda
python310Packages.tensorflowWithCuda.dist
python310Packages.tiny-cuda-nn
python310Packages.torchWithCuda
python310Packages.torchWithCuda.dev
python310Packages.torchWithCuda.dist
python310Packages.torchWithCuda.lib
python310Packages.torchWithRocm
python310Packages.torchWithRocm.dev
python310Packages.torchWithRocm.dist
python310Packages.torchWithRocm.lib
python310Packages.torchaudio-bin
python310Packages.torchaudio-bin.dist
python310Packages.torchvision-bin
python310Packages.torchvision-bin.dist
python310Packages.trainer
python310Packages.trainer.dist
python311Packages.bitsandbytes
python311Packages.bitsandbytes.dist
python311Packages.cupy
python311Packages.cupy.dist
python311Packages.encodec
python311Packages.encodec.dist
python311Packages.jaxlibWithCuda
python311Packages.jaxlibWithCuda.dist
python311Packages.openai-triton
python311Packages.openai-triton-bin
python311Packages.openai-triton-bin.dist
python311Packages.openai-triton.dist
python311Packages.pycuda
python311Packages.pycuda.dist
python311Packages.pynvml
python311Packages.pynvml.dist
python311Packages.pyrealsense2WithCuda
python311Packages.pyrealsense2WithCuda.dev
python311Packages.tiny-cuda-nn
python311Packages.torchWithCuda
python311Packages.torchWithCuda.dev
python311Packages.torchWithCuda.dist
python311Packages.torchWithCuda.lib
python311Packages.torchaudio-bin
python311Packages.torchaudio-bin.dist
python311Packages.torchvision-bin
python311Packages.torchvision-bin.dist
python311Packages.trainer
python311Packages.trainer.dist
tiny-cuda-nn
tts
tts.dist
xgboostWithCuda
xpraWithNvenc
xpraWithNvenc.dist

Failed derivations

[ ] nsight_systems-2<IP_ADDRESS>.drv (derivation hash: ij0s879hg2pkhnw6lnlkf9fji63akva6)

https://gist.github.com/ConnorBaker/262867b5f401e38b333d46e2d340a2f2

[ ] Mathematica_13.3.0_BNDL_LINUX.sh.drv (derivation hash: c1i2vddgkkajjwqc574gvcvhv2qi3bbd)

https://gist.github.com/ConnorBaker/ea29cec2e5bc34aafb6971062070fa35

[ ] TensorRT-<IP_ADDRESS>.Linux.x86_64-gnu.cuda-11.8.cudnn8.6.tar.gz.drv (derivation hash: z0p5rhrilvalqkz2n0qda5zgrzqq12va)

https://gist.github.com/ConnorBaker/221af16d6add17a5f599c0aa3be9eadf

[ ] TensorRT-<IP_ADDRESS>.Linux.x86_64-gnu.cuda-11.8.cudnn8.6.tar.gz.drv (derivation hash: kafa8qrrfp0fq4p93wpzb7fxn31l72ms)

https://gist.github.com/ConnorBaker/217a2465dab6cf9a574a509187343752

[ ] TensorRT-<IP_ADDRESS>.Linux.x86_64-gnu.cuda-11.8.cudnn8.6.tar.gz.drv (derivation hash: sdayyl861icp0wd777y5j7chnr1f2vhs)

https://gist.github.com/ConnorBaker/0223715559aad40835e0ba9df63613be

[ ] TensorRT-<IP_ADDRESS>.Linux.x86_64-gnu.cuda-11.8.tar.gz.drv (derivation hash: j5b3vqwfq2ydzbgllrkpqh5d9njrzhmk)

https://gist.github.com/ConnorBaker/24c225933d37b23073c5e10750f9e429

[ ] cudatoolkit-10.1.243.drv (derivation hash: qgp4j5jss45m6iyrq96530awgq7dgqpr)

https://gist.github.com/ConnorBaker/50d24683d536b6cde3f70f8d2513b9ba

[ ] cudatoolkit-10.2.89.drv (derivation hash: lyg5xh9j43jrjh023qyikqglq7vmqpwl)

https://gist.github.com/ConnorBaker/c6444e5ad67414f6aecd302ede265123

[ ] python3.10-xformers-0.0.20.drv (derivation hash: hkfisbjxanj6zjiw8a760lk5if4b1zq6)

https://gist.github.com/ConnorBaker/ca624c70f87a1dc8d2134383bf10915c

[ ] python3.11-torch-2.0.1.drv (derivation hash: 9mbvpsss4brhl88jr78j5hp1gif86lvm)

https://gist.github.com/ConnorBaker/b6642d2bb3f123a79375753dddb72f7f

[ ] python3.11-xformers-0.0.20.drv (derivation hash: zggwqbf2p6qzfpn7n606pxfzs6xl47gs)

https://gist.github.com/ConnorBaker/82c806738d0961a27c72b9245934a0e0

[ ] tensorflow-gpu-2.11.1.drv (derivation hash: jlzvq17q3wg8lmpbv93dy6wh0hnmmr6s)

https://gist.github.com/ConnorBaker/ef89c6c3fd1f0de1c8a6f2024775ab23

Had to change autoPatchelfIgnoreMissingDeps = ["libcuda.so.1"]; back to autoPatchelfIgnoreMissingDeps = true; for nsight_systems. This "fixes" the build in the sense that it just ignores all the missing dependencies instead of causing an error outright.

Awesome work, thanks again! I hate to start off with this, but I think we'd want to leave out the formatting changes, so as to keep the diffs reviewable and possibly preserve the git blame?
P.S. As mentioned before, I don't mind alejandra, and I'd easily vote for switching to it if I saw the rest of nixpkgs was already using it. Just that this PR is about split outputs rather than aesthetics?
P.P.S. I'm still away, so I'll be a slow and clumsy reviewer making a lot of typos because screen keyboard

Rebased and updated comment.
Next up, reducing diff size.

Minimized the diffset for edits as best as I was able to to make this easier to review.

That's right! Docs! https://github.com/ConnorBaker/nixpkgs/blob/feat/cuda-redist-multiple-outputs/doc/languages-frameworks/cuda.section.md

@samuela wanted to check if you had a chance to review this, want to review this, or know who else would be a good person to review :)

Rebased, squashed, and force-pushed.

if we go through with this, part of the critical path for CUDA packaging will live outside of nixpkgs... are we ok with that?

Good question!
To me, the answer is yes -- if doing so provides us with added value which isn't outweighed by the downside.
I think that split outputs are well worth this maintenance cost, and that having the code outside Nixpkgs provides us with an opportunity to iterate more quickly than we might otherwise.
@Samuela @SomeoneSerge what are your thoughts on the following? Personally, I'd like to see this merged sooner rather than later, as it's hung out in review for some time (understandable given the size and scope of the PR).
Followup Questions
How comfortable are we with:

having the source for cuda-redist-find-features out-of-tree?
having the source on my personal account (@ConnorBaker) instead of a company account (@Tweag) or the community account (@nix-community)?
introducing code to bring about more features?

My Thoughts

having the source for cuda-redist-find-features out-of-tree?

I'm vaguely concerned by the fact that, as it is out-of-tree, it is not subject to the same review process as the rest of nixpkgs
However, I think that the code is simple enough that this is not a major concern.
I am concerned that it is not subject to the same CI as the rest of nixpkgs. I think that a combination of having CI run on the cuda-redist-find-features repo and including a derivation for cuda-redist-find-features in nixpkgs would be a good solution to this.

having the source on my personal account (@ConnorBaker) instead of a company account (@Tweag) or the community account (@nix-community)?

I have no issue with transferring ownership elsewhere :)

introducing code to bring about more features?

Without use of IFD, I don't know how else to do some of the things I've done or planned.
As an example of future work, I think it'd be neat if we were able to scan the redistributable libraries to find out what overrides they'll need, ahead of time, avoiding the need for us to track the majority of them manually.
This will become more important as we move towards supporting more architectures (mid-September) and a wider version of CUDA releases, given that overrides differ between relases and architectures.

Personally, I'd like to see this merged sooner rather than later,

☝🏻 ☝🏻 ☝🏻

if we go through with this, part of the critical path for CUDA packaging will live outside of nixpkgs... are we ok with that?

First of all, this is more like an automation or code-generation tool to simplify the maintenance, I wouldn't call it "critical". Sort of like poetry2nix or nix-update (also live out-of-tree) only it's nvidia2nix. Bash, for that matter, is more critical to us and isn't maintained in-tree. Once we've generated these augmented manifests, the users don't even need to know the tool was ever involved, unless maybe they look to override something. I also like that the the tool being under Connor's account it's easier to treat it as a blackbox: I was about to complain about pydantic being to heavy, but now that it's isolated I literally don't care.

unless maybe they look to override something

Now that's an important use-case we may be damaging and which we maybe want to handle later. I suspect these extra jsons may make it harder to diverge (as in, using overlays and overrides) from the nixpkgs tree

having the source on my personal account (@ConnorBaker) instead of a company account (https://github.com/tweag) or the community account (https://github.com/nix-community)?

If the script sticks around and evolves further maybe we can look into moving it to some organization's namespace, for better consolidation and discoverability. Otherwise, I don't a even a bit mind you having it under yours, as long as it's permissively licensed and fork-able 🤷🏻
To conclude,
...from my point of view the only thing holding this PR back is that it touches the documentation and I'd like to see that documentation updated first, because I think it could confuse people in its current form

...from my point of view the only thing holding this PR back is that it touches the documentation and I'd like to see that documentation updated first, because I think it could confuse people in its current form

Any particular portion of the docs you'd want to see updated, @SomeoneSerge? I purposefully didn't want to change things outside of adding a section on update procedures, but I will if you feel it's necessary.

Updated with feedback on the docs section (CUDA Toolkit -> CUDA Toolkit runfile installer; explicitly mention cudaPackages.cudatoolkit), rebased, and squashed.

Rebased, updated, squashed, and force-pushed.

@SomeoneSerge I'm confident this is ready to be merged; have I addressed all your doc concerns?
@samuela any blockers you see which would prevent merging?

I intend to merge this tomorrow morning (August 31st) at 13:00 UTC barring any strong objects.

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/cuda-team-roadmap-update-2023-08-29/32379/1


How to handle macOS SDK requirement from Conda build?

When running our sanity check we realised that it didn't work on MacOS. This turns out to be because: "The macOS compilers require the macOS 10.9 SDK".
This is explained in: https://docs.conda.io/projects/conda-build/en/latest/resources/compiler-tools.html together with a solution.
The questions is: should we handle this, by making a guide to Mac users, or shouldn't we handle it, since it is something that is already documented in conda build?

It might be hard for the user to handle this themselves, since our setup method before the sanity check (bioconda_utils_build_setup), creates the folder inside Bioconda-recipes/recipes and adds the meta.yaml and build.sh that it have generated, before running the sanity check.
The user would have to manually add their conda_build_config.yaml into this folder after it is created but before we start running the test (which is right after, making it impossible for the user to do).

From this link: https://docs.conda.io/projects/conda-build/en/latest/resources/variants.html#conda-build-variant-config-files, it looks like it should be possible for the user to put their conda_build_config.yaml file inside their HOME directory as a permanent location where conda always looks first. This solves the problem introduced in the previous comment.
Furthermore, I just realised that the method I looked at bioconda_utils_build_setupisn't the one used for this purpose, but this is instead done in mini_sanity_check. (It is actually not used at all. We might want to remove all code that was used before we switched to our own conda-build-mini)

This should be solved in #50


Firefox capture screen is affected by pointer-events: none on body

Hello,
when the dialog is opened is not possibile capture the screen until the dialog is closed.
I've used the Firefox capture screen, the capture object is behind the dialog object.
Vincenzo

It turns out pointer-events: none also causes other undesirable side-effects.
For instance, the element can't be selected with Web Inspector: https://bugzilla.mozilla.org/show_bug.cgi?id=745489
So let me consider this a browser issue and close as won't fix.


Location Guard seems not working

Hi @frederikme!
First of all, thank you for your hard work and for your marvelous bot, everything works like a charm.
However, I noticed that the LocationGuard extension doesn't work with the search service and raise this error :
"The search service is not working right now. Please try again later."
Is there an alternative to this deficient extension or another better extension for changing the location?
Thanks in advance man!
EDIT :
It seems that I have found a workaround to LocationGuard extension when it doesn't work, simply add these lines after defined the browser
        self.browser.execute_cdp_cmd(
            "Emulation.setGeolocationOverride",
            {
                "latitude": 48.856613,  # latitude here
                "longitude": 2.352222, # longitude here
                "accuracy": 100, # accuracy here
            },
        )

Hi @Anopy,
I'm glad you like Tinderbotz so far!
Let me start by thanking you for opening a new issue.
Contributing and noticing flaws or bugs is what makes this repository grow.
Back to topic: There seems to be an issue with the API of the extension that converts a search query into latitude and longitude values. However, your suggestion opens up a lot of possibilities to fix this problem. I was thinking about using this website https://geocode.xyz/?locate={location_query}&geoit=JSON to get the longitude and latitude and then inserting it into the browser using your suggested code. This way, as I understand it, the extension wouldn't even be needed anymore.
So basically this is how I would implement it.
In the running script it would look like this:
session.setCustomLocation("New York, USA", accuracy="100%")

and that would result into a get-request like this:
url = "https://geocode.xyz/?locate=New%20York%20USA&geoit=JSON"
import requests
r = requests.get('url')
data = r.json()

That request returns this json data:
{   "standard" : {      "addresst" : {},      "city" : "New York",      "prov" : "US",      "countryname" : "United States of America",      "postal" : {},      "confidence" : "0.90"   },   "longt" : "-73.95861",   "alt" : {},   "elevation" : {},   "latt" : "40.68908"}

So placing this results into the browser would then be done like you suggested
self.browser.execute_cdp_cmd(
            "Emulation.setGeolocationOverride",
            {
                "latitude": data['latt'],  # latitude here
                "longitude": data['longt'], # longitude here
                "accuracy": 100, # accuracy here
            },
        )

This is just some guessing, I don't know if this will work out. I'll try to implement it and get back to you.
Thanks again for contributing!
Greetings,
Fred

Hi @Anopy,
I'm glad you like Tinderbotz so far!
Let me start by thanking you for opening a new issue.
Contributing and noticing flaws or bugs is what makes this repository grow.
Back to topic: There seems to be an issue with the API of the extension that converts a search query into latitude and longitude values. However, your suggestion opens up a lot of possibilities to fix this problem. I was thinking about using this website https://geocode.xyz/?locate={location_query}&geoit=JSON to get the longitude and latitude and then inserting it into the browser using your suggested code. This way, as I understand it, the extension wouldn't even be needed anymore.
So basically this is how I would implement it.
In the running script it would look like this:
session.setCustomLocation("New York, USA", accuracy="100%")

and that would result into a get-request like this:
url = "https://geocode.xyz/?locate=New%20York%20USA&geoit=JSON"
import requests
r = requests.get('url')
data = r.json()

That request returns this json data:
{   "standard" : {      "addresst" : {},      "city" : "New York",      "prov" : "US",      "countryname" : "United States of America",      "postal" : {},      "confidence" : "0.90"   },   "longt" : "-73.95861",   "alt" : {},   "elevation" : {},   "latt" : "40.68908"}

So placing this results into the browser would then be done like you suggested
self.browser.execute_cdp_cmd(
            "Emulation.setGeolocationOverride",
            {
                "latitude": data['latt'],  # latitude here
                "longitude": data['longt'], # longitude here
                "accuracy": 100, # accuracy here
            },
        )

This is just some guessing, I don't know if this will work out. I'll try to implement it and get back to you.
Thanks again for contributing!
Greetings,
Fred

UPDATE
You can now set a custom location by using the following code:
session.setCustomLocation("Barcelona, Spain", accuracy="100%")
Where accuracy is an optional parameter which is by default 100%.
NOTE: As you can see, the extension has become obsolete so it has been deleted from the project.
If any problem occurs, feel free to open a new issue!
For now, I'm closing this one.
Have a nice day!
Fred

UPDATE
You can now set a custom location by using the following code:
session.setCustomLocation("Barcelona, Spain", accuracy="100%")
Where accuracy is an optional parameter which is by default 100%.
NOTE: As you can see, the extension has become obsolete so it has been deleted from the project.
If any problem occurs, feel free to open a new issue!
For now, I'm closing this one.
Have a nice day!
Fred


Spotify client not found.

I followed the steps exactly in the README to in order to be able to pull play counts from Spotify (the publicly hosted API works, I'm simply trying to get it to work on my own machine). After completing every single step carefully, (including replacing .spa file, running zlink script ect.) and then running Spotify using the command given in the README (with the appropriate args e.g.  "--allow-running-insecure-content" ect.) I get a "Spotify client could not be found" error when trying to make a request to the endpoint which I have specified in the config.json (which looks like this btw):
{     "port": 8080,     "useSecure": false,     "secure": {         "cert": "",         "key": "",         "chain": "'"     },     "http": {         "endpoint": "/albums"     },     "websocket": {         "ipAddress": "<IP_ADDRESS>"     } }
I'm using index.js to start the simple node server so obviously https isn't necessary, so I left properties blank in "secure".
The URL I was trying to make a GET request to was: http://<IP_ADDRESS>:8080/albums?albumid=49K82qcZN4u109e6Zw8io1. I tried other forms of this url to no avail.
Also, when I use the command spotify --ignore-certificate-errors --allow-running-insecure-content it loads the Spotify app normally but I immediately get a Mac notification saying "Session: Spotify (sleep) just terminated" and my terminal shuts down.
Do you have any advice or tips in order to get this to work for myself on my own machine? Since your publicly hosted API works I imagine it may just be something I'm doing wrong. Let me know if more information is needed in order to get to a solution.
Thanks.

Hi, can you send me your zlink.spa file that was generated from spotify_zlink.js?

Thanks for the reply. I uploaded it to Google Drive with a shareable link here: https://drive.google.com/file/d/19KVyymbhB2H9eakQqW3hFrnSyGXz26mp/view
After running the spotify_zlink.js script it DID say it was successful also.

I tried your zlink file on Mac and Linux and it works fine for me. Can you try launching Spotify without any arguments? It should still work without the extra arguments using your config.
Also, if you can, install spicetify-cli and use it to enable the Chrome DevTools in Spotify (spicetify enable-devtool, it should then be available under the menu bar: Develop > Developer Tools). Then, screenshot the log under Console and post it here.

I followed your steps. Here is the error in the Spotify developer console that appears on starting up Spotify:
Uncaught (in promise) Error: Cosmos Request Error. Status: 502     at zlink.bundle.js:160     at zlink.bundle.js:57     at zlink.bundle.js:17     at zlink.bundle.js:17
When I attempt to make a request to the endpoint, I get the message
{ "success": false, "data": "An unknown error has occurred; logged to server console" }
And then on the console it prints an object like this:
{uuid: "fe4a89d3-a385-4727-9de9-9494a25e5d70", id: "1TKCBDHIWLiqjWTxomqniD"}
And at the same time my terminal window that's running the index.js server outputs the error
ReferenceError: print is not defined at WebSocket.listener (/Users/<username>/desktop/Spotify-PlayCount-master/app.js:29:13)
So it seems I have made progress...index.js shows that my client is now connected, and I'm obviously not having the Spotify client not found error anymore.
I used your command to open up Spotify with and without the arguments. Which Spotify CLI are you using to open up Spotify with the spotify command though? I'm using Shpotify I didn't see anything about a Spotify CLI in your README so maybe we're using different ones and this matters or something?
Also just a sidenote, I tried using a different port (8000 instead of 8080) and I noticed that when I try a different port, I get an additional error in the Spotify developer console window saying WebSocket connection to 'ws://<IP_ADDRESS>:8080/' failed: Error in connection establishment: net::ERR_CONNECTION_REFUSED. (Notice it still says 8080 even though I changed config.json to be port 8000. When I run index.js it says it's running on port 8000 as well). Not sure if this is relevant to the issue just thought that was strange.
Let me know if you have any other suggestions to try! Or if any other information is needed. Thanks for the help so far.

Hey I figured out the issue and was able to get it to work. On line 29 of app.js there is a print statement. I commented it out and everything works great thank you for developing this.

https://ibb.co/C7sJsXX


question: to root or not to root; security? best practice?

At least with the initial WSL + Ubuntu beta, we are given a root prompt. Many things that work with unprivileged users on a real Ubuntu box (e.g. SSH, etc) strictly only work under root in WSL + Ubuntu.
Does the usual "principle of least privilege" that I would normally follow in Linux apply here?
It seems like all the dangers of using root all the time still apply here: e.g. running npm as root could execute a malicious install script, capable of doing much more nasty stuff as root.
Should we be adding a new user immediately and switching to that lesser-privileged user?
Should WSL + Ubuntu offer to make such a non-root user for us on first run? Should a non-root user become the default?

Well, I created an user with adduser and put him in the sudo group using adduser <user> sudo.
Since the WSL is aimed to us, developers, and we all know that root isn't ideal to use in a daily basis, I think that this is fine, for now.
Once the WSL is well stablished, maybe this would present itself as a problem.

I also immediately made myself a new user out of habit with:
adduser <username>
adduser <username> sudo

And as opening bash results on login to root, I created this simple alias to jump into my created user: alias u='su - hfaran'
All that being said, I believe the semantics of what root means in WSL are just a little off because the install actually lives isolated in the app data of each specific user. Unless you're an admin on the machine, you can't look at other Windows users' Linux filesystems.

the install actually lives isolated in the app data of each specific Windows user

I thought as much. What does rm -rf / do from within WSL as root? If it isn't possible to completely hose (or install malicious software within) the host Windows environment, then maybe root-only within WSL isn't so bad. /shrug

You are root on a restricted environment: your user.
The only things you can arm on is WSL and is your own process.
For exploits or other stuff the environment run on another layer.
If you want to hack around, make a guess all typical calls are protected.

Interesting conversation.  We will be adding some documentation around how users work in the future, but I wanted to call out a few things here.

The idea of root is separate from the Windows user.  That is, running as root in a a normal cmd prompt cannot access anything restricted to Windows admin privileges.

The WSL will honor root & user access within Linux directory structure.

What does this mean for running as root.  Basically, our guidance will be to create a user and run under the context.  That said, we also believe in choice and know that there are times when you'll want to run as root so that should be possible as well.
So the community knows where we are going here: we will be releasing an update soon with better user support.  In short we'll ask people to create a Linux user on first run.  When launched, the WSL will start under that user's context.  If someone does not want to run as a user, they can make their default user 'root' and have the same experience we have today.
I'm going to close this one since it's a discussion and not really an issue.  We have people working on getting a good forum in place for these types of conversations.  Expect to see something from us soon on this specific topic and then we can have the discussion on what people like / don't like about the implementation.

Yep. Closing makes sense. I wasn't reporting a bug, and we're still free to converse of course (until the other forum exists).
Good to see some feedback. I think we all understand that this is very new, and it's super exciting to see the direction WSL is heading in. Cheers! :D

@jokeyrhyme after a fresh lxrun install I have 1000:1000 owner on /data and root:1000 on /mnt. When I create a user via adduser it assings uid 1000 to it, so that that folders are owned by it. Is it ok or should I assign another uid for this user manually?

Great question on the uid and gid values. We have some hardcoded uid / gid values that need to be fixed. I'll be going through these today and change the owners so they match real Ubuntu. I think that will mean these will be changed to root in most cases.

Oh, that's nice. Thanks for fast feedback.


General revamp of Dagger usage

Use of Builders+Dagger 2.7 way to define sub-components
Abstraction layer between injections and Dagger framework

Cool @mdelolmo!! I'm going to start a review of this PR giving you as much feedback as possible just to wonder myself if I'm understanding the way you implemented it. Feel free to correct my misunderstandings.


i guess we do this already using chunk/indent but nonetheless...

if we can get this in chunk (one node) / indent (whole file) it will be great:

so adding fold support will do good:
Inspiration: https://github.com/gh-liu/fold_line.nvim

Feels interesting, but will doing this cause serious performance issues? I'll try out the plugin you mentioned first.

Feels interesting, but will doing this cause serious performance issues? I'll try out the plugin you mentioned first.

check this: gh-liu/fold_line.nvim#1

can this display effect meet your need

can this display effect meet your need

wait, thats real cool :lmao: tho the start of line as in return got messed up but we can use the rounded_border glyph for that anyway
can we do this for the current context only tho? coz it may coz perf drops if we do this with the whole file so i guess this is going towards chunk :damn:
tho this almost looks like the plugin

this is impl by using ibl.hook and statucol.nvim, I have write a demo like below
the first one

the second one

which one you like better

the first one (what happened to ur symbols in the second one tho 💀?)
whats the difference?

what ~godly~ font are you using?

i have just edited the image but is this possible (more similarity to the plugin) tho only when the respective fold is closed

the first one (what happened to ur symbols in the second one tho 💀?)

the second one use different char and highlight

whats the difference?

the first render in the max level indent line, while the second only render in the first level

what ~godly~ font are you using?

[font]
size = 12

[font.normal]
family = "JetBrainsMono NFM"
style = "Regular"

[font.bold]
family = "JetBrainsMono NFM"
style = "Bold"

[font.italic]
family = "VictorMono NFM"
style = "Italic"

i have just edited the image but is this possible (more similarity to the plugin) tho only when the respective fold is closed

no, hlchunk only set extmark, not monitor fold info... maybe you can search for other plugin

no, hlchunk only set extmark, not monitor fold info... maybe you can search for other plugin

can we just add another spec for indent like chunk having chars, something like:
chars = {
        vertical_line = "│", -- or `vertical_line` to be passed as a table for multiple indent chars
        left_top = "╭",
        left_bottom = "╰",
    },

no, under the current conditions, it is impossible to achieve the style effect of fold-line. It is necessary to inherit the indent mod and rewrite the render function.

no, under the current conditions, it is impossible to achieve the style effect of fold-line. It is necessary to inherit the indent mod and rewrite the render function.

no im just asking if we can add indent start and indent end characters

in the dev brach, you can use filter_list to define your own hook to achieve this, but this field still unstable, maybe change in future...

After considering for a while, I feel that the above implementation has a significant impact on performance and is not conducive to code maintenance. temporarily close this issue, and if I have any ideas for refactoring in the future, I may reopen it.


SendAndWait works only the first time

Hi jchristin,
today I'm trying to upgrade my program to use the new 4.8.1 version of WatsonTcp from the far 4.1.5 one that always worked finely for me.
I've noticed that now it's very very quick to send messages, with my old 4.1.5 version i could send up to 65 messages per second (async messages), and now with the 4.8.1 i can reach the outstanding speed of 1500 msg/sec!! so awesome!
But sadly thre is also a problem... :-(
If I call on client a SendAndWait it works fine, but only the first time, if then I call another SendAndWait this hangs until the timeout exception...
this is my sample client code:
            cli = new WatsonTcpClient("localhost", 23456);
            cli.Events.ServerConnected += Cli_ServerConnected;
            cli.Events.MessageReceived += Cli_MessageReceived;
            cli.Settings.DebugMessages = false;
            cli.Connect();
            cli.Send("Single message from client-1");
            cli.Send("Single message from client-2");
            cli.Send("Single message from client-3");

            SyncResponse srsp = cli.SendAndWait(30000, "1");
            Console.WriteLine(Encoding.UTF8.GetString(srsp.Data));
            srsp = cli.SendAndWait(30000, "1");  <<----- this row hangs and after 30 secondt fires a TimeoutException
            Console.WriteLine(Encoding.UTF8.GetString(srsp.Data));

and this is the server side that works only for the first time, for the second SendAndWait this is never fired
        private static SyncResponse SyncRequestReceived(SyncRequest req)
        {
            Console.WriteLine($"Syncrequest received from {req.IpPort}: {Encoding.UTF8.GetString(req.Data)}");
            SyncResponse ret;

            int delay = 0;
            int.TryParse(Encoding.UTF8.GetString(req.Data), out delay);
            if (delay == 0)
                delay = 1;

            System.Threading.Thread.Sleep(1000*delay);
            ret = new SyncResponse(req, $"Long work done :-) {delay}");

            Console.WriteLine("Syncrequest completed");
            return ret;
        }

many thanks in advance for your great work!
Michele

Hi @ozrecsec good to hear from you.  What is your environment i.e. runtime (.NET Framework, .NET Core, .NET 5) and version?  Which operating system?
I just tried using the Test.Server and Test.Client projects and they work without an issue sending a series of sequential SendAndWait requests (the Test.Server project automatically responds with a SyncResponse).
Cheers, Joel

Here's the result on Windows 10 w/ .NET 5.0.  Right side is the client.

Windows 10 w/ .NET 4.6.1

Windows 10 w/ .NET Core 3.1

Are you running Windows 10 or a different version?  Linux/Mac?

I don't know if this is another issue, but on 4.8.1, when I do WatsonTcpServer.Send (or SendAsync) multiple messages from the server, the client MessageReceived is only fired once, I've rolled back to 4.8.0 and it fires for every message.

thank you @davidhays I've tried to reference 4.8.0 version in my test project and SendAndWait works fine.
My develop environment is on Windows server 2012R2 and Framework .NET 4.6.1.

Got it - was able to reproduce.  Seems client DataReceiver isn't having its SemaphoreSlim released.

Could you give v4.8.2 a try and let me know if this resolves the issue?
NuGet: https://www.nuget.org/packages/WatsonTcp/4.8.2
Commit: https://github.com/jchristn/WatsonTcp/commit/5ed315665ae71a7c9b0be7a47883aaf0542e4bbf

4.8.2 works for me.

Awesome. @ozrecsec please close if this fixes it!  Cheers Joel

I've tested 4.8.2 and works fine for me too.
thanks!


Header Component Integration leads to title being invisible

I was using the design system to build the new header and the header title was invisible. Tracked this down to the css example for the h1 visibility: hidden;
I corrected this by removing that property

i missed the media queries nvm !


Fixed DH _jcompat to jcompat module rename

Deephaven's python module has made _jcompat public as jcompat.  Appropriate changes were made.
Resolves #41

Required for DH v0.12


Essential updates for Onboarding through Serval

Long term onboarding through SILNLP is unsustainable.  The jobs required for onboarding new teams need to be run in Serval (long term).  The final solution is still undecided, but there are some known enhancements that Serval will need before this can be done.  These are the ones that would be required to move the current onboarding app from SILNLP to Serval, namely:
Known necessary:

Validation split - and Bleu/Chrf++ scores

Control over validation split books and chapters

Alignment scores - overall score # (already being worked on)

Strong Assumption:

Basic information on how the corpora are used to train the models

Paratext project name, language, number of verses, Keyterm count
Number of segments from each source corpora/file used for training

A porting of the SILNLP enhanced data on paratext projects

Tokenization stats
Verses per book

This really needs more definition.  I'm closing this as not really an end product.  It is likely still needed, but let's actually have a goal we are aiming for.


JSON considers that if it is not a single value it should be ignored

//json.c:621
    if (state.current_token.type == JSON_TOKEN_OBJECT_START) {
        root = parse_object(&state);

    }
    else if (state.current_token.type == JSON_TOKEN_ARRAY_START) {
        root = parse_array(&state);
    }
    else {
        fmt_fprintf(stderr, "Error: Root element must be an object or array at position %zu.\n", state.position);
        free(state.input);
        return NULL;
    }

But a json can be valid as a single and unique value
"value"

53

etc...

Dear Mattis,
Currently, I am engaged in the implementation of JSON and I am aware of the
issue you've mentioned. You are absolutely correct that a JSON can be valid
as a single, unique value, such as a simple string ("value"), a number
(53), etc.
I have some ideas for resolving this issue, but I need some time to
implement them. This matter requires a review and possibly a rewrite of
parts of the JSON parsing code to ensure that various types of JSON values
are properly supported.
As soon as I am able to make these changes, I will be sure to inform you.
Thank you for pointing out this important detail.
Best regards,
On Tue, Jan 30, 2024 at 2:41 PM Mattis DALLEAU @.***>
wrote:

//json.c:621
if (state.current_token.type == JSON_TOKEN_OBJECT_START) {
root = parse_object(&state);
}
else if (state.current_token.type == JSON_TOKEN_ARRAY_START) {
    root = parse_array(&state);
}
else {
    fmt_fprintf(stderr, "Error: Root element must be an object or array at position %zu.\n", state.position);
    free(state.input);
    return NULL;
}

But a json can be valid as a single and unique value
"value"
53
etc...
—
Reply to this email directly, view it on GitHub
https://github.com/KaisenAmin/c_std/issues/4, or unsubscribe
https://github.com/notifications/unsubscribe-auth/AI23CKJB2CAZ2OMJJABT4HTYRDIMRAVCNFSM6AAAAABCRBP3V2VHI2DSMVQWIX3LMV43ASLTON2WKOZSGEYDONJWGYZTAOI
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

now json_parser can handle single value as json.


Recorded file can't be saved(Experiments)

Describe the bug
Recorded file can't be saved(Experiments)
Steps to reproduce the behavior

Navigate to https://arcade.makecode.com/beta#editor
Click "More..." -> "About..." -> Experiments -> enable "Simulator Gifs"
Back to editor page and click "Share"
Click "Start recording" to record

Expect behavior
Recorded file should can be saved
Actual behavior(Firefox)
Recorded file can't be saved(Experiments)

Additional context

OS: Windows(rs6)
arcade version: 0.14.4
Microsoft MakeCode version: 5.21.6

I am not sure I understand the issue @Helen09. Recorded gif would be part of the published project. It will appears when you refer link in social media like twitter.

To accommodate @Helen09 's intention, could we add a Record button to the simulator? Would that be too confusing?

@abchatra When user refers link in other website or import url on home page, Recorded gif doesn't show.

By design.


Send choosen value to didSelect action

This is a good idea. However,

There needs to be a test
It should not only pass the (new) value, but also a reference to the component itself

I think I will fix this in the next days, but thanks for your contribution, still!


Add an option to expose devices on /off

Expose devices (all  / some ?) with their on / off function (devices, not activities)

Since tv support is on its way I will close that

Reopen still it still make sense to have them aside , will think of it

Thank you very much!

@sommo @carsten-h  @bverkron  : Done in 0.4.0, let me know if it suits your needs !
I exposed devices as push button switch since there is no way to get their state (especially if they were put on / off by any other action than the switch itself).
If your device is configured with powertoggle button (on/off, same command), it will be exposed as one push switch in homekit. However, if there is only power on or/and power off defined, it will expose a switch for power on / power off.

I am getting this message for my shutters:
[2/16/2019, 7:13:05 PM] [HarmonyHub] Discovered Device : Rollladen [2/16/2019, 7:13:05 PM] [HarmonyHub] Error - No Power function found for Rollladen
There is no switch for On/Off for them, only up/down is available. In my Harmony App these are visible around the OK button on the first page.

I just mapped on / off as it is what make sense for me. I think you can try to map on / off function in harmony with up / down and it might work.

There is only Powertoggle available and what I need is DirectionUp and DirectionDown.
Maybe it is possible to specify the "correct" On/Off commandname via config.json?

It is a pleasure :)
Did you try this : https://support.logitech.com/en_za/article/12732 ?

Will also try this new feature because I have 2 light strips which are not part of an activity.
What about the syntax for config.joson?
„devicesToPublishAsAccessoriesSwitch„ : [„Stripe 1“, „Stripe 2“]
Thanks

Yes, I did this. I used DirectonUp for PowerOff and DirectionDown for PowerOn. But the error message is still the same. I will look into it this evening when I have more time.

Will also try this new feature because I have 2 light strips which are not part of an activity.
What about the syntax for config.joson?
„devicesToPublishAsAccessoriesSwitch„ : [„Stripe 1“, „Stripe 2“]
Thanks

You have to put the name of the devices in harmony app, as an array indeed .

Yes, I did this. I used DirectonUp for PowerOff and DirectionDown for PowerOn. But the error message is still the same. I will look into it this evening when I have more time.

LEt me know, you will have to restart homebridge for this to take effect. If you could launch in debug mode and send me your config which will be displayed it might help.

Okay, I tried ... restarted homebridge.
Accessories will not apear...
Did I miss something?
        {
            "platform": "HarmonyHubWebSocket",
            "name": "HarmonyHub",
            "hubIP": "XXX",
            "showTurnOffActivity": true,
            "turnOffActivityName": "All off",
            "devicesToPublishAsAccessoriesSwitch": [
                "Beleuchtung TV",
                "Beleuchtung Theke"
            ]
        }

I used exactly the names out of the Harmony App.

After I did the changes in Harmony app I restarted homebridge. Can you tell me how to start in debug mode?

Add a -D option in the command line

Ok, I will do it this evening as I have to leave now.

Will Lso deliver data  with debug mode.
I hab a look to the app:
One device is powered on/off with the same button, the other with 2 buttons.
So I think whis will not be the problem as far as I understood, right?

Yes, right. I have a doubt if it works outside of tv mode (it should, but maybe Inmissed something) . Feel free to post debug log and I will check this evening

Hi Nicoduj, thanks for this nice feature! Would it be posible to expose a certain command on the device as toggle on/off button (it’s the same command for on and off). I have a light switch which is controlled by B&O sound system, and the switch works regardless the sound system is turned on/off.
Thank you in advance!

Hi @maisun, will think of this use case originally I just wanted to handle power feature originally. Being able to map other commands involves begin able to list them all in a certain way and map them to each « switch » .  Definitely more work and config to do but why not :)

Hi @nicoduj, maybe a generic function to add a toggle switch (switch on send command and then set off) in HomeKit for DeviceName.CommandName - not sure if Harmony client sends command name or translated IR code to the hub though.
Thank you in advance!

Hi! I managed to start homebridge with -D.
The part from the logfile, that is interesting for this, should be the following (I extracted only the part for the shutter, if you need more I can send it):
Rollladen.txt

Great. Will take a look at it from what I see there is no power control group and that is why the plugin can’t find the command . Will see if I can base the search on something else or maybe another way to do this (would like the conf to be not too « difficult »)

The only thing that I am see is:
"powerFeatures":{"PowerOffActions":[{"__type":"IRPressAction","IRCommandName":"DirectionUp","Order":0,"Duration":0,"ActionId":0}],"PowerOnActions":[{"__type":"IRPressAction","IRCommandName":"DirectionDown","Order":0,"Duration":0,"ActionId":0}]},
These are the two commands I set.
For my Panasonic TV there can be the following found:
"powerFeatures":{"PowerOffActions":[{"__type":"IRPressActi Feb 17 15:05:36 HomePi homebridge[23019]: on","IRCommandName":"PowerOff","Order":0,"Duration":1000,"ActionId":0}],"PowerOnActions":[{"__type":"IRPressAction","IRCommandName":"PowerOn","Order":0,"Duration":1000,"ActionId":0}]},"isManualPower":"false","controlGroup":[{"name":"Power","function":[{"action":"{\"command\":\"PowerOff\",\"type\":\"IRCommand\",\"deviceId\":\"60257134\"}","name":"PowerOff","label":"Power Off"},{"action":"{\"command\":\"PowerOn\",\"type\":\"IRCommand\",\"deviceId\":\"60257134\"}","name":"PowerOn","label":"Power On"},{"action":"{\"command\":\"PowerToggle\",\"type\":\"IRCommand\",\"deviceId\":\"60257134\"}","name":"PowerToggle","label":"Power Toggle"}]},

Yes, that is what I saw. For all my devices I have a power controlGroup and I based my search on this. Will think of using powerFeatures maybe instead .

Interestingly there is a powerFeatures group for my TV also, although I did not add anything here. Maybe it would be the best to search for powerFeatures only if there is no power controlGroup available.

I think powerFeatures is a link to the command itself since it doesn’t have the command . Don’t really know what the group stands for / if it si defined through the configuration. I might try to check the power features available and then parse the whole list of commands to find them .

OK guys I pushed a new version with a new optional behaviour where you can specify the command you want on the switch for the device. Please read README for syntax :)
Let me know if it suits your needs !

Will try tomorrow... thanks...

OK guys I pushed a new version with a new optional behaviour where you can specify the command you want on the switch for the device. Please read README for syntax :)
Let me know if it suits your needs !

Just tested the custom button with my setup and it works like charm! The only problem I noticed is when I press button too quickly it doesn't work sometimes, but I can live with that. Really good work, thanks for bringing my Harmony alive!

The only problem I noticed is when I press button too quickly it doesn't work sometimes, but I can live with that. Really good work, thanks for bringing my Harmony alive!

Could you tell me more ? There is currently a delay for switching the switch back to off position (since it is a push button). The delay is about 1s, so I expect that if you hit twice it won't catch the second action. Is it in this time interval when you say quickly ?

Thank you for this solution!
Now I have two switches in Homekit for my shutters. If they work I can tell this evening as they are too loud to test them now where all others here are sleeping.

Yes! Thanks for the great work.
I installed the latest version, but norhing changed until now.
THe devices are not visible. Before I start deeper cheecks I want to clarify if it should basicaly work with this settings:
This is my device, page 2, 2 buttons, one for power on one for power of, they are working inside harmony app

And these are the general power settings for the device:

Dies mean, power them off if not in use. But this device is not a part of any action ...
So in my opinion this value does not effect anything.
I did not change anything to this:
        {
            "platform": "HarmonyHubWebSocket",
            "name": "HarmonyHub",
            "hubIP": "XXX",
            "showTurnOffActivity": true,
            "turnOffActivityName": "All off",
            "devicesToPublishAsAccessoriesSwitch": [
                "Beleuchtung TV",
                "Beleuchtung Theke"
            ]
        }

The device is controlled via IR ... and as far as I saw I am not able to change the buttons that I for example use others for power off and on ...

The only problem I noticed is when I press button too quickly it doesn't work sometimes, but I can live with that. Really good work, thanks for bringing my Harmony alive!

Could you tell me more ? There is currently a delay for switching the switch back to off position (since it is a push button). The delay is about 1s, so I expect that if you hit twice it won't catch the second action. Is it in this time interval when you say quickly ?

That's most likely the problem - to be honest I pressed the button without looking at it.
Is it possible to configure the delay? 1s is ok for on/off but for volume it should be lower I think.

@maisun

That's most likely the problem - to be honest I pressed the button without looking at it.
Is it possible to configure the delay? 1s is ok for on/off but for volume it should be lower I think.

This mode was not made for commands like Volume. I definitely prefer to focus on TV Mode with IOS 12.2 for such use case, since for this kind of commands a "remote" is much more reliable I think.
@Phoenix-DH
I need your config (which is shown in debug mode) to investigate. From what I see at lest something like :

"devicesToPublishAsAccessoriesSwitch" : ["Beleuchtung TV;PowerOn","Beleuchtung TV;PowerOff"] should work.

@nicoduj:
Thank you, will try .. bad thing .. then I will have 2 switches, right?

Understood, thanks! Look forward to iOS 12.2

Hello!
@nicoduj, you are my hero!
The commands are recognized as I wrote this morning. And they are working without any problem, so now I can control my shutters depending on sunrise/sunset. That is really great!
Thank you!

then I will have 2 switches, right?

@nicoduj: ...

@Phoenix-DH yes you will have 2 switches. I can have a switch that knows the state of the device since it would not be able to react to external events. However if your device have a PowerToggle function then you can set only this one and you will have only one switch
Sent with GitHawk

Great, thanks!
I get the one with Off/On seperat working ... with 2 switches!
But the one with one, the switch will not appear... extract from homebridge -d:
{"label":"Beleuchtung Theke","deviceAddedDate":"/Date(1527532614833+0000)/","ControlPort":7,"contentProfileKey":XXX,"deviceProfileUri":"XXX=","manufacturer":"Paulmann","icon":"24","Capabilities":[],"deviceTypeDisplayName":"LightController","powerFeatures":[],"isManualPower":"false","controlGroup":[{"name":"Home","function":[{"action":"{\"command\":\"PowerToggle\",\"type\":\"IRCommand\",\"deviceId\":\"XXX\"}","name":"PowerToggle","label":"Power Toggle"}]},{"name":"Miscellaneous","function":[{"action":"{\"command\":\"Auto\",\"type\":\"IRCommand\",\"deviceId\":\"XXX\"}","name":"Auto","label":"Auto"},{"action":"{\"command\":\"B\",\"type\":\"IRCommand\",\"deviceId\":\"XXX\"}","name":"B","label":"B"},{"action":"{\"command\":\"BlueDown\",\"type\":\"IRCommand\",\"deviceId\":
And the config:
"Beleuchtung Theke;PowerToggle"
Last step ;)

Great, thanks! ...

@Phoenix-DH that is what I was about to say : you have the PowerToggle command ! Nice if it works as you want it to , enjoy !
Sent with GitHawk


Clarify Enterprise required for multiple instances

Update docs to match main readme
Please squash merge

Codecov Report

Merging #765 (f1894f7) into master (8aa6b62) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #765   +/-   ##
=======================================
  Coverage   32.38%   32.38%
=======================================
  Files          52       52
  Lines        5361     5361
=======================================
  Hits         1736     1736
  Misses       3423     3423
  Partials      202      202

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8aa6b62...f1894f7. Read the comment docs.

This PR has been automatically labelled "stale" because it hasn't had recent activity.
A core team member will check in on the status of the PR to help with questions.
Thank you for your contribution!
/cc @jasonblais @jfrerich @emilyacook


Add Segment Analytics to Showcase

Acceptance criteria
We want to see the activity on the showcase website, how many users, recurring users, more traffic after a demo, etc.

Let's determine what metrics we want to capture, then determine what tool to use.

following @christophe-f ideas, I am starting to list some ideas on metrics
MoSCoW List
Must

Number of Users/Visitors (identify times of more or less traffic, e.g., after demo, summit)
1.1. Number of Recurring Users/Visitors
Number of Sessions
2.1. duration of a session (interest and usability of the showcase)
2.2. pages during a session (finding funnels, often used pages, may not be possible within the Backstage framework?)

Should

Bounce rate (user only visiting one page could indicate technical, UI, or usability problems)
Conversions/Funnels (if we have paths that a user should follow/ we expect a user to follow? here this can be measured)

Could

Average time on page (depending on the page and its content; could identify too long or uneasy to read pages)

Won't
The above mentioned Must and Should could be archived with google analytics out of the box.

do we need more or others, please add and correct
is google analytics an option at all


feat(dev): Add type_name option for ObjectDescribedBy test matcher

Sometimes, it's easier to pass the name of the type you're checking against, rather than the type/class object itself, so this lets you do that. Using this option has the disadvantage of not being able to match on subclasses, so I added it rather than replacing the type check.

I would again have to see the context here before telling whether this is a good idea. I can imagine that due to lack of module name included this can lead to weak assertions. On the flipside I don't see where it would be a great inconvenience to pass the actual type.

This came up when writing a test for the AWS integration, where I was finding the inner workings to be a little more of a black box than normal, because we can only control things from afar. That said, I think I may be able to work around it a different way, so if you think this is too gross a hack, I'm happy to try to make that work first and maybe it'll be nicer.

I found a different workaround. Closing this.


Documentation does not load

Expected Behavior
Dev portal documentation should load
Actual Behavior
documentation page gets this error
Building a newer version of this documentation failed. Error: "Failed to generate docs from /var/folders/kk/g0ppfjfj52v62np_l0qc2flr0000gp/T/backstage-uJh8Mh into /private/var/folders/kk/g0ppfjfj52v62np_l0qc2flr0000gp/T/techdocs-tmp-CRObNC with error This operation requires Docker. Docker does not appear to be available.  Can you verify you have Docker installed and running? One option would be to run docker ps in your terminal and ensure it is not showing an error.

This operation requires Docker. Docker does not appear to be available

Hello,
This issue is also present in production.
On Tuesday, March 29, 2022, 09:58:02 AM GMT+3, Otto Sichert @.***> wrote:
Hey @bigmirc! Can you verify you have Docker installed and running? One option would be to run docker ps in your terminal and ensure it is not showing an error.
This operation requires Docker. Docker does not appear to be available
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: @.***>

Docker also needs to be installed and running in production. What is the output of following commands?
$ docker ps
$ echo $?

Hello,
Seems like no container is up, but I have Docker installed.
Thanks, Alex
On Tuesday, March 29, 2022, 11:06:34 AM GMT+3, Otto Sichert<EMAIL_ADDRESS>wrote:

Docker also needs to be installed and running in production. What is the output of following commands?
$ docker ps
$ echo $?
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: @.***>

it's returning 0.

 As you mentioned, no containers are running hence why the documentation does not load.
On this page we outline the building part for Docker containers: https://backstage.io/docs/deployment/docker
The setup usually is very different for each adopter of Backstage, so it's hard to give you a definite answer that applies to your setup. Just that you'd need to ensure the containers are up

@bigmirc are you unstuck now?

yeeep, thanks a lot
On Tuesday, April 26, 2022, 03:37:03 PM GMT+3, Gary Niemen @.***> wrote:
@bigmirc are you unstuck now?
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: @.***>

Closing then.

I have precisely the same issue.  I run backstage in Kubernetes with two pods - backstage (both FE and BE) and Postgres.  I also am getting the following error:

Building a newer version of this documentation failed. Error: "Failed to generate docs from /tmp/backstage-zO2TIA into /tmp/techdocs-tmp-JjUuox; caused by Error: This operation requires Docker. Docker does not appear to be available. Docker.ping() failed with; caused by Error: connect ENOENT /var/run/docker.sock"

Does it mean I need to run Docker inside of a Kubernetes pod?

@githoober I had the same issue but this document was helpful:
I had to
apt install python3-pip
pip3 install mkdocs-techdocs-core

and change my config to:
    techdocs:
        builder: 'local' # Alternatives - 'external'
        generator:
            runIn: 'local' # Alternatives - 'local'
        publisher:
            type: 'local' # Alternatives - 'googleGcs' or 'awsS3'. Read documentation for using alternatives.

And it works like a dream now


driver: add some optional interfaces

These interfaces offer an alterative way to handle file transfer and listing.
Using these interfaces instead of implementing Create/Open/OpenFile/Readdir inside a custom afero.Fs implementation seems simpler for me. I understand that this could be opinable. Anyway  since the code change is minimal and fully backward compatibile I hope these modifications can be merged, thank you!

Hi,
thank you for merging this pull request, very appreciated!
These interfaces make ftpserverlib integration very similar to pkg/sftp and this simplify my work, I have used them locally for my initial prof of concept and all the features available for SFTP seems to work for FTP too (I have some other local patches for ftpserverlib that I have to better test and clean up before submitting).
I'm a bit busy these days, when I'll resume FTP support I'll better explore afero fs integration too.
Please note that this pull request introduced the FileTransferError interface to notify for errors that happen inside an FTP transfer. I don't know if using the afero interface we can detect a read or write error on the network connection, anyway I'll try to understand if this is possibile and how easy/difficult is to support the current SFTPGo features using afero (for example configurable upload concurrency for S3, ability to interrumpt an S3/GCS transfer on demand or after the FTP/SFTP client disconnects for network issues etc...)


Find oversized element that's making a scroll bar appear on the map

Something on the page is overflowing beyond the bounds of the viewport, probably an element with an improper margin.

Resolved by #61.


Derivative of center of mass Jacobian

Hi, I am wondering if you could help direct me to the best way to use Pinocchio to compute the time derivative of the center of mass jacobian, i.e. for center of mass x, since x' = J v, then x'' = J v' + J' v and I want to compute J' (I am already computing the rest using jacobianCenterOfMass). Is there an analytical implementation I haven't been able to find, or could you offer a tip for where to look in the documentation to compute this via autodiff? Many thanks for your excellent library and helpful support.

computeCentroidalMapTimeVariation will compute dAg/dt, the time variation of the centroidal momemtum matrix. The 3 first rows do match the center of mass Jacobian time variation.
If you need a dedicated algorithm for computing this quantity, I may implement it.

I think the existing functionality fully satisfies my needs! thank you again.

@jakewelde @jcarpent Don't you need to divide dAg/dt by the total_mass of the robot to achieve a com jacobian dot?

@jakewelde @jcarpent Don't you need to divide dAg/dt by the total_mass of the robot to achieve a com jacobian dot?

Agreed, this is also a required detail

Agreed, this is also a required detail

@jakewelde @jcarpent Don't you need to divide dAg/dt by the total_mass of the robot to achieve a com jacobian dot?

Yes, you are correct.

@jakewelde @jcarpent Don't you need to divide dAg/dt by the total_mass of the robot to achieve a com jacobian dot?

Yes, you are correct.

@jcarpent @jakewelde  Thanks for verification!

@jcarpent @jakewelde  Thanks for verification!


chore(docs): Moved docs to GitHub Pages

Changes in this PR:
1st commit:

Updated directory structure inside /docs.
Added GitHub Actions CI to build docs site using mdbook (.github/workflows/github-pages.yaml)
Adding gh-pages branch to store built website files.

2nd commit:

Fixed typo in /docs/src/404.md
Removed file /netlify.toml as we are not using netlify.

3rd commit:

Fixed broken link on /docs/src/intro.md
Fixed broken link on /docs/src/features.md

Sample screenshot:

To test this build locally:

Install mdbook using the appropriate binary from https://github.com/rust-lang/mdBook/releases.
Example command for linux:

curl -fSsL https://github.com/rust-lang/mdBook/releases/download/v0.4.4/mdbook-v0.4.4-x86_64-unknown-linux-gnu.tar.gz | sudo tar -C /usr/local/bin -xzf -

Clone the repo and fetch the PR

git clone https://github.com/metacontroller/metacontroller.git
cd metacontroller
git fetch https://github.com/metacontroller/metacontroller.git refs/pull/45/head
git checkout -b new-branch FETCH_HEAD

Run the mdbook server locally and test the website on your browser

cd docs
mdbook serve

Use your web browser to navigate to http://<IP_ADDRESS>:3000 or http://localhost:3000
Signed-off-by: Niladri Halder<EMAIL_ADDRESS>

:tada: This PR is included in version 0.4.3 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Use ALCOM as the app link for vcc://vpm/add-repo

Summary / 概要
Make ALCOM capable of using vcc://vpm/add-repo to easily add VPM repositories to it.
Purpose / 目的
A lot of creators share links that use vcc://vpm/add-repo to allow users to add VPM repositories to VCC with one click. It'd be nice to allow ALCOM to open them instead of VCC for convenience.
Additional context / 補足
The only parameters I know of that can be passed to it are:

url -> The url of the listing.json
headers -> written in key:value format. Example: &headers=Authorization:Bearer github_pat_....
I think more headers could be passed to it by writing it again in the same format, but I haven't tested that.

I've planned for that but tauri doesn't support that in v1 so I'm considering migrate to v2 beta or use some workaround.
Duplicate of #391

Duplicate of https://github.com/vrc-get/vrc-get/issues/391


remove update dd device from execution

Description
Remove wipe execution
Type of change
What is it?

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Documentation update
[ ] Tests update
[X] Refactor

Checklist:

[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] New and existing unit tests pass locally with my changes
[ ] I run go fmt ./... to check that my code is properly formatted
[ ] I run go vet ./... to check that my code is free of common Go style mistakes

/retest

/retest

@mergify rebase

Codecov Report

Merging #709 (8876386) into main (7952c1d) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #709   +/-   ##
=======================================
  Coverage   31.37%   31.37%
=======================================
  Files          44       44
  Lines        4570     4570
=======================================
  Hits         1434     1434
  Misses       2938     2938
  Partials      198      198

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7952c1d...8876386. Read the comment docs.


Consider removing all the other layouts than Default from App.vue

Plus we probably don't need a dynamic importing layout. App.vue should become Deafult.vue and that's it: https://github.com/DivanteLtd/vsf-capybara/blob/544646edc78b278286d27d3a88d3c94a255394f4/App.vue#L3

Related: #104


Add uWrite implementation

ufmt and it's uWrite trait is supposed to be a smaller alternative to core::fmt. I use it in my project when I need to format strings, and now that I have a use case for slice-string I think it would be a nice addition (both slice-string to my project and ufmt support to it :) )

Thanks!


Many Barriers are assigned PostPass of 0 for Optipass input

While working on #149 I see I'm getting the following input file sent to optipass.

[ ] Why are most of the Postpass values 0?
[ ] When I override barrier values, the overrides do not appear in my report.

BARID,REGION,FOCUS,DSID,USHAB,PREPASS,NPROJ,ACTION,COST,POSTPASS
758651,San Juan Creek,1,759012,0.00875844701932682,0.0,0,0,0,0.0
758652,San Juan Creek,1,758651,0.0043621904373687,0.0,0,0,0,0.0
758653,San Juan Creek,1,758652,0.0242344971514639,0.0,0,0,0,0.0
758654,San Juan Creek,1,758653,0.0345562743683886,0.0,0,0,0,0.0
758656,San Juan Creek,1,758654,0.0172160289144989,0.0,0,0,0,0.0
758657,San Juan Creek,1,758656,0.275503599361259,0.0,0,0,0,0.0
758663,San Juan Creek,1,758657,1.71476599160097,0.0,0,0,0,0.0
758666,San Juan Creek,1,758675,0.398469976825666,0.0,0,0,0,0.0
758669,San Juan Creek,1,758666,3.11159260676829,0.0,0,0,0,0.0
758675,San Juan Creek,1,758676,0.181296164326927,0.0,0,0,0,0.0
758676,San Juan Creek,1,758678,0.0968557375997849,0.0,0,0,0,0.0
758678,San Juan Creek,1,758663,0.0835218787168919,0.0,0,0,0,0.0
758681,San Juan Creek,1,706807,0.778675614762319,0.0,0,0,0,0.0
758682,San Juan Creek,1,758681,0.782494057063785,0.0,0,0,0,0.0
758684,San Juan Creek,1,758682,2.65615909311841,0.0,0,0,0,0.0
758685,San Juan Creek,1,758684,0.205001528452521,0.0,0,0,0,0.0
758686,San Juan Creek,1,758704,0.00311888419325672,0.0,0,0,0,0.0
758704,San Juan Creek,1,758685,0.00817536793276367,0.0,0,0,0,0.0
758705,San Juan Creek,1,758686,0.00802352076798301,0.0,0,0,0,0.0
758706,San Juan Creek,1,758705,0.0801425595968527,0.0,0,0,0,0.0
758713,San Juan Creek,1,758706,0.0510315606006743,0.0,0,0,0,0.0
758715,San Juan Creek,1,758713,0.452031311285054,0.0,0,0,0,0.0
759012,San Juan Creek,1,706788,0.00432132382692525,0.5,0,0,0,0.5
706799,San Juan Creek,1,NA,3.62273335018664,0.5,0,0,0,0.5
706810,San Juan Creek,1,NA,0.493591707904111,0.0,1,0,1,1.0
706788,San Juan Creek,1,716932,13.8475167979629,0.5,0,0,0,0.5
706807,San Juan Creek,1,706809,14.2596630658949,0.0,1,0,1,1.0
706809,San Juan Creek,1,706810,0.0853056117277312,0.5,1,0,1,1.0
716932,San Juan Creek,1,706799,18.5193818312347,0.5,0,0,0,0.5

Fixed so postpass is not set to prepass if nproj set to 0 - we don't really care about postpass if the barrier cannot be remediated, but it was a bug nonetheless.

The bigger issue was why all of these were not being considered, which was because I didn't have cost data in my development env for these guys. They were all being left out of consideration.


Exporting part one

Most basic export functionality.

[ ] Export options alt E

[ ] Directory

[ ] Name outputs
[ ] Export hotkey ctrl E

Closing as duplicate of #32.


Updated examples application to take any json payload and schema to send messages to a topic using avro serializer.

This PR addresses issue #28

@harshach Updated yelp schema/payloads with multi level complex types.

Closing this PR as it is already merged into repo.


log location is wrong

set MO_PATH="/root/"
the log folder location is /root/logs, which is supposed to be in /root/matrixone/logs

fixed

fixed.


[v5] - Pass parent props into custom handleSubmit()

I'd really like to be able to do something like this:
function handleSubmit(values, dispatch, props) {
  return () => {
    const { policyType } = props;
    return createUserWithPolicy({
      policyType,
      user: { ...values }
    })
      .catch(error => Promise.reject({ _error: error.message }));
  };
}

I reckon it's possible to do this by making policyType a form field and managing its value at the level where it would change using redux-form field properties, but it would be worlds easier to just pass parent props into the handleSubmit() function. It'd be pretty easy to do; I've already made the change on my personal fork that I'm using in a project. So if nobody has any objections, I'd happily make the PR.
Let me know what you guys think.

Have you considered generating your handleSubmit function with a factory that receives policyType?
function handleSubmitFactory(policyType) {
  return (values) => {
    // createUser...
  }
}

<form onSubmit={handleSubmitFromReduxForm(handleSubmitFactory(props.policyType))}>

@naw There definitely are other ways to do it, and your suggestion seems like a fine one. Still, I think this could be an improvement to redux-form and make the API more consistent. For example, validate() and asyncValidate() functions receive the component's props as an argument, so it seems reasonable that the handleSubmit() function should as well.

Released @baronswindle's fix in v5.3.3.


Use restful methods to get the summary links instead of URL mangling with Addressable::URI

This is a follow-up PR for https://github.com/ManageIQ/manageiq/pull/4189
Also a follow-up action on the discussion <a href=https://gitter.im/ManageIQ/manageiq/ui?at=55f0998f24362d5253fe7711>here

/cc @matthewd


Samsung Tizen UE48JS8500 always OFF

Describe the bug
Integgration don't work on Smasung Tizen UE48JS8500. I see my TV but it always off. I cannot turn it on or don't see any other options for that TV. I try also official integration but have the same issue.
Expected behavior
Play, pause, stop, tunr off, turn on TV
Environment details:

Environment (HASSIO, Raspbian, etc): Home Assistant installed on Dietpi (Debian Bullseye) on HP-MiniPC, intel, x64.
Home Assistant version installed:

System Health

version
core-2021.11.3

installation_type
Unknown

dev
false

hassio
false

docker
false

virtualenv
false

python_version
3.9.7

os_name
Linux

os_version
5.10.0-9-amd64

arch
x86_64

timezone
Europe/Warsaw

Home Assistant Community Store

GitHub API
ok

Github API Calls Remaining
4786

Installed Version
1.17.0

Stage
running

Available Repositories
966

Installed Repositories
31

Home Assistant Cloud

logged_in
false

can_reach_cert_server
ok

can_reach_cloud_auth
ok

can_reach_cloud
ok

Lovelace

dashboards
2

resources
19

views
8

mode
storage

 - Component version installed: v0.4.8
 - Last know working version:?
 - TV model: UE48JS8500
It is supperted model?
Regards.

I have the same issue with my TV.  Works for a few days, then it stops working.  If I reload the integration, it works fine for a few minutes, then it goes back to a status off.     I am also using the SamsungTV Tizen hacs integration (which this is a spinoff of) and that works fine.


2023-08-18: 🎧UN Open GIS Monthly VTC

As announced by Kyoun-Soo on 2023-08-08T04:37+09:00:
Please be informed that Monthly VTC of UN Open GIS is scheduled on Friday, 18 August 2023 at 3:00 PM (Rome time) 9:00 AM (New York time).
Proposed agenda

Opening by Co-Chairs (Prof. Ki-Joune Li, Prof. Maria Brovelli, Mr. Erzen Ilijazi)
Messages from outgoing (Mr. Kyoung-Soo Eom) and incoming (Mr. Erzen Ilijazi, Chief Technology Operations Service, OICT) Co-Chair from United Nations
Introduction of new members
Open Q&As and discussions
Report, update & upcoming event
. UN-GGIM 13th Session in UNHQ, New York (31 July – 4 August 2023)
. UN Smart Maps meetup, online (9 Aust 2023)
. FOSS4G Asia 2023 in Seoul (28 Nov - 2 Dec 2023): UN Open GIS joint workshop
. Organizing QField introductory session (webinar) in September: Timur/Cung will send a calendar invite, please accept or decline when received invite.
Working Group report
. DWG-1 Hybrid GIS platform (Timur, Zeeshan, Epa/Rodwell, Akbar, #Cung);
.. Integration of Azure AD with GeoServer/GeoNode (Timur, Eddy)
.. Mobile GIS Solutions, including integration with Azure AD (Epa/Rodwell, Zeeshan, Cung)
.. UNMISS Story Map (Zahurul/Akbar/Debasish, Timur, Eddy):
. DWG-2 Capacity Building (Ivan, Susan, Diego);
. DWG-5 Geo-AI & Analytics (Zhongxin, Maria, Andrea, #Cung)
. DWG-6 OpenDroneMap & 3D (Catalina, Davide, Stephan Mather, #Cung)
. DWG-7 Smart Maps (Hidenori, Yuiseki, Diego, #Timur)
Any other business (AOB)

Sorry for a last-minute sharing, but I am preparing the slides at https://hackmd.io/@hfu/2023-08-18#/ this time.

Thanks. I mark this issue as successfully closed.


Turn ish into a 'real' project

Let's start with:

setup.py
bumpversion
upload to PyPI

Ish has been moved to its own repo


yarn vs npm

I kinda feel that npm is the "standard" and wonder if we should default a project like this to it.  npm@5 brings much of the same benefits, and in most cases would be one less dep to install.

Biased toward yarn because it works all the time for me 🥇. When npm5 released I often ran into issues where packages wouldn't correctly install if say you first install w/ yarn and then used npm.
Cool story, right? Yarn or npm whatever works for people. If we only use npm and not yarn, I'll attempt to break my yr command  habits

Hey all, we currently have a mix of npm and yarn running within the app (both lock files exist).  I'm gonna move to npm - it's pretty stable now since 5.3.0.  Yarn has the definite one up on npm if using monorepos but otherwise I don't think it brings that much more to the table that I wanna move away from what is essentially "standard".  Easier for people to initially onboard, and easy enough for those wanting to use yarn to simply switch over.


add ability to pass in extra resource attributes to traceprovider

Proposed Changes
Adding ability to pass in extra attributes. Specifically the use case is to set some kubernetes specific attributes.
https://opentelemetry.io/docs/reference/specification/resource/semantic_conventions/k8s/
Checklist

[ ] I have included #major/#minor/#patch tags in commit messages for commits that make major, minor or patch changes, so the autoreleaser can appropriately bump the package version

Testing Instructions

Looks good. I think the autoreleaser needs a reference in a commit to minor to tag the release version correctly, but approved. 👍