Package request: pytorch on darwin with GPU (MPS) support

Project description
Not sure if this is a "packaging request" but didn't seem like a bug report, just wanted a tracking issue, I might try to hack on this eventually.
Pytorch on aarch64-darwin now supports GPU:
https://pytorch.org/docs/stable/notes/mps.html
The currently packaged pytorch runs but doesn't use my Mac's GPU:
$ nix-shell -p 'python3.withPackages (ps: with ps; [ pytorch ])' --command "python -c 'import torch; print(torch.backends.mps.is_available())'"
False

In contrast, using the pytorch-provided wheel:
$ python3 -m venv .venv && source ./.venv/bin/activate
$ pip install https://files.pythonhosted.org/packages/85/68/f901437d3e3ef6fe97adb1f372479626d994185b8fa06803f5bdf3bb90fd/torch-2.0.1-cp311-none-macosx_11_0_arm64.whl
$ python -c 'import torch; print(torch.backends.mps.is_available())'
True

Metadata

homepage URL: https://pytorch.org/
source URL: https://github.com/pytorch/pytorch
license: bsd3 https://github.com/pytorch/pytorch/blob/main/LICENSE
platforms: darwin

cc @NixOS/darwin-maintainers

curious if python3Packages.torch-bin works?

@samuela torch-bin gives me unsupported system (cuda_nvtx-11.7.50)

hmmm that's odd. aarch64-darwin is a supported platform
https://github.com/NixOS/nixpkgs/blob/04f7903bb9cd886a0e2885c728ff735f9248de89/pkgs/development/python-modules/torch/bin.nix#L79C85-L79C85
and there are srcs available here:
https://github.com/NixOS/nixpkgs/blob/04f7903bb9cd886a0e2885c728ff735f9248de89/pkgs/development/python-modules/torch/binary-hashes.nix#L40-L54
what version of python are you using?

I made a nix and nix-gpu branch at https://github.com/n8henrie/whisper; nix-gpu uses the wheel (and I've confirmed it enables the GPU to run whisper).
However I'm not seeing any speed difference with a small test file (full of particularly witty content):
$ nix shell github:n8henrie/whisper/nix -c time whisper 20220922\ 084923.m4a
/nix/store/0fyl0qi21ljiswg05qz5p2bpnl016k0l-python3.10-whisper/lib/python3.10/site-packages/whisper/transcribe.py:114: UserWarning: FP16 is not supported on CPU; using FP32 instead
  warnings.warn("FP16 is not supported on CPU; using FP32 instead")
Detecting language using up to the first 30 seconds. Use `--language` to specify the language
Detected language: English
[00:00.000 --> 00:04.640]  This is an example of voice recording. Something might actually say if I
[00:04.640 --> 00:09.440]  recording something on my watch. I think this is how it would turn out.
19.93user 11.08system 0:14.23elapsed 217%CPU (0avgtext+0avgdata 2062800maxresident)k
0inputs+0outputs (3246major+212332minor)pagefaults 0swaps
$
$ nix shell github:n8henrie/whisper/nix-gpu -c time whisper 20220922\ 084923.m4a
/nix/store/fvpkcahvngmsssm3yfchvgw11hdic6sx-python3.10-whisper/lib/python3.10/site-packages/whisper/transcribe.py:114: UserWarning: FP16 is not supported on CPU; using FP32 instead
  warnings.warn("FP16 is not supported on CPU; using FP32 instead")
Detecting language using up to the first 30 seconds. Use `--language` to specify the language
Detected language: English
[00:00.000 --> 00:04.640]  This is an example of voice recording. Something might actually say if I
[00:04.640 --> 00:09.440]  recording something on my watch. I think this is how it would turn out.
19.81user 11.16system 0:14.02elapsed 220%CPU (0avgtext+0avgdata 2177600maxresident)k
0inputs+0outputs (3955major+212007minor)pagefaults 0swaps

Not sure if that's going to be a training vs inference issue (or more likely whisper isn't configured to use the GPU, it has a --device flag but seems to only look for cuda devices by default).

Huh, looks like I just need to pass --device mps (https://github.com/openai/whisper/pull/382) but no luck:
$ nix shell github:n8henrie/whisper/nix-gpu -c time whisper --device mps 20220922\ 084923.m4a
Traceback (most recent call last):
  File "/nix/store/fvpkcahvngmsssm3yfchvgw11hdic6sx-python3.10-whisper/bin/.whisper-wrapped", line 9, in <module>
    sys.exit(cli())
  File "/nix/store/fvpkcahvngmsssm3yfchvgw11hdic6sx-python3.10-whisper/lib/python3.10/site-packages/whisper/transcribe.py", line 444, in cli
    model = load_model(model_name, device=device, download_root=model_dir)
  File "/nix/store/fvpkcahvngmsssm3yfchvgw11hdic6sx-python3.10-whisper/lib/python3.10/site-packages/whisper/__init__.py", line 154, in load_model
    return model.to(device)
  File "/nix/store/0y2cidaighjvzwlw31k6fkcragba3jki-python3.10-torch-2.0.1/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1145, in to
    return self._apply(convert)
  File "/nix/store/0y2cidaighjvzwlw31k6fkcragba3jki-python3.10-torch-2.0.1/lib/python3.10/site-packages/torch/nn/modules/module.py", line 844, in _apply
    self._buffers[key] = fn(buf)
  File "/nix/store/0y2cidaighjvzwlw31k6fkcragba3jki-python3.10-torch-2.0.1/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1143, in convert
    return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
NotImplementedError: Could not run 'aten::empty.memory_format' with arguments from the 'SparseMPS' backend. This could be because the operator doesn't exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit https://fburl.com/ptmfixes for possible resolutions. 'aten::empty.memory_format' is only available for these backends: [CPU, MPS, Meta, QuantizedCPU, QuantizedMeta, MkldnnCPU, SparseCPU, SparseMeta, SparseCsrCPU, BackendSelect, Python, FuncTorchDynamicLayerBackMode, Functionalize, Named, Conjugate, Negative, ZeroTensor, ADInplaceOrView, AutogradOther, AutogradCPU, AutogradCUDA, AutogradHIP, AutogradXLA, AutogradMPS, AutogradIPU, AutogradXPU, AutogradHPU, AutogradVE, AutogradLazy, AutogradMeta, AutogradMTIA, AutogradPrivateUse1, AutogradPrivateUse2, AutogradPrivateUse3, AutogradNestedTensor, Tracer, AutocastCPU, AutocastCUDA, FuncTorchBatched, FuncTorchVmapMode, Batched, VmapMode, FuncTorchGradWrapper, PythonTLSSnapshot, FuncTorchDynamicLayerFrontMode, PythonDispatcher].

CPU: registered at /Users/runner/work/pytorch/pytorch/pytorch/build/aten/src/ATen/RegisterCPU.cpp:31034 [kernel]
MPS: registered at /Users/runner/work/pytorch/pytorch/pytorch/build/aten/src/ATen/RegisterMPS.cpp:22748 [kernel]
Meta: registered at /dev/null:241 [kernel]
QuantizedCPU: registered at /Users/runner/work/pytorch/pytorch/pytorch/build/aten/src/ATen/RegisterQuantizedCPU.cpp:929 [kernel]
QuantizedMeta: registered at /Users/runner/work/pytorch/pytorch/pytorch/build/aten/src/ATen/RegisterQuantizedMeta.cpp:105 [kernel]
MkldnnCPU: registered at /Users/runner/work/pytorch/pytorch/pytorch/build/aten/src/ATen/RegisterMkldnnCPU.cpp:507 [kernel]
SparseCPU: registered at /Users/runner/work/pytorch/pytorch/pytorch/build/aten/src/ATen/RegisterSparseCPU.cpp:1379 [kernel]
SparseMeta: registered at /Users/runner/work/pytorch/pytorch/pytorch/build/aten/src/ATen/RegisterSparseMeta.cpp:249 [kernel]
SparseCsrCPU: registered at /Users/runner/work/pytorch/pytorch/pytorch/build/aten/src/ATen/RegisterSparseCsrCPU.cpp:1128 [kernel]
BackendSelect: registered at /Users/runner/work/pytorch/pytorch/pytorch/build/aten/src/ATen/RegisterBackendSelect.cpp:726 [kernel]
Python: registered at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/core/PythonFallbackKernel.cpp:144 [backend fallback]
FuncTorchDynamicLayerBackMode: registered at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/functorch/DynamicLayer.cpp:491 [backend fallback]
Functionalize: registered at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/FunctionalizeFallbackKernel.cpp:280 [backend fallback]
Named: registered at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/core/NamedRegistrations.cpp:7 [backend fallback]
Conjugate: fallthrough registered at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/ConjugateFallback.cpp:21 [kernel]
Negative: fallthrough registered at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/native/NegateFallback.cpp:23 [kernel]
ZeroTensor: fallthrough registered at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/ZeroTensorFallback.cpp:90 [kernel]
ADInplaceOrView: fallthrough registered at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/core/VariableFallbackKernel.cpp:63 [backend fallback]
AutogradOther: registered at /Users/runner/work/pytorch/pytorch/pytorch/torch/csrc/autograd/generated/VariableType_2.cpp:17484 [autograd kernel]
AutogradCPU: registered at /Users/runner/work/pytorch/pytorch/pytorch/torch/csrc/autograd/generated/VariableType_2.cpp:17484 [autograd kernel]
AutogradCUDA: registered at /Users/runner/work/pytorch/pytorch/pytorch/torch/csrc/autograd/generated/VariableType_2.cpp:17484 [autograd kernel]
AutogradHIP: registered at /Users/runner/work/pytorch/pytorch/pytorch/torch/csrc/autograd/generated/VariableType_2.cpp:17484 [autograd kernel]
AutogradXLA: registered at /Users/runner/work/pytorch/pytorch/pytorch/torch/csrc/autograd/generated/VariableType_2.cpp:17484 [autograd kernel]
AutogradMPS: registered at /Users/runner/work/pytorch/pytorch/pytorch/torch/csrc/autograd/generated/VariableType_2.cpp:17484 [autograd kernel]
AutogradIPU: registered at /Users/runner/work/pytorch/pytorch/pytorch/torch/csrc/autograd/generated/VariableType_2.cpp:17484 [autograd kernel]
AutogradXPU: registered at /Users/runner/work/pytorch/pytorch/pytorch/torch/csrc/autograd/generated/VariableType_2.cpp:17484 [autograd kernel]
AutogradHPU: registered at /Users/runner/work/pytorch/pytorch/pytorch/torch/csrc/autograd/generated/VariableType_2.cpp:17484 [autograd kernel]
AutogradVE: registered at /Users/runner/work/pytorch/pytorch/pytorch/torch/csrc/autograd/generated/VariableType_2.cpp:17484 [autograd kernel]
AutogradLazy: registered at /Users/runner/work/pytorch/pytorch/pytorch/torch/csrc/autograd/generated/VariableType_2.cpp:17484 [autograd kernel]
AutogradMeta: registered at /Users/runner/work/pytorch/pytorch/pytorch/torch/csrc/autograd/generated/VariableType_2.cpp:17484 [autograd kernel]
AutogradMTIA: registered at /Users/runner/work/pytorch/pytorch/pytorch/torch/csrc/autograd/generated/VariableType_2.cpp:17484 [autograd kernel]
AutogradPrivateUse1: registered at /Users/runner/work/pytorch/pytorch/pytorch/torch/csrc/autograd/generated/VariableType_2.cpp:17484 [autograd kernel]
AutogradPrivateUse2: registered at /Users/runner/work/pytorch/pytorch/pytorch/torch/csrc/autograd/generated/VariableType_2.cpp:17484 [autograd kernel]
AutogradPrivateUse3: registered at /Users/runner/work/pytorch/pytorch/pytorch/torch/csrc/autograd/generated/VariableType_2.cpp:17484 [autograd kernel]
AutogradNestedTensor: registered at /Users/runner/work/pytorch/pytorch/pytorch/torch/csrc/autograd/generated/VariableType_2.cpp:17484 [autograd kernel]
Tracer: registered at /Users/runner/work/pytorch/pytorch/pytorch/torch/csrc/autograd/generated/TraceType_2.cpp:16726 [kernel]
AutocastCPU: fallthrough registered at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/autocast_mode.cpp:487 [backend fallback]
AutocastCUDA: fallthrough registered at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/autocast_mode.cpp:354 [backend fallback]
FuncTorchBatched: registered at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/functorch/LegacyBatchingRegistrations.cpp:815 [backend fallback]
FuncTorchVmapMode: fallthrough registered at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/functorch/VmapModeRegistrations.cpp:28 [backend fallback]
Batched: registered at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/LegacyBatchingRegistrations.cpp:1073 [backend fallback]
VmapMode: fallthrough registered at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/VmapModeRegistrations.cpp:33 [backend fallback]
FuncTorchGradWrapper: registered at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/functorch/TensorWrapper.cpp:210 [backend fallback]
PythonTLSSnapshot: registered at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/core/PythonFallbackKernel.cpp:152 [backend fallback]
FuncTorchDynamicLayerFrontMode: registered at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/functorch/DynamicLayer.cpp:487 [backend fallback]
PythonDispatcher: registered at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/core/PythonFallbackKernel.cpp:148 [backend fallback]

Command exited with non-zero status 1
3.87user 1.23system 0:03.39elapsed 150%CPU (0avgtext+0avgdata 1210224maxresident)k
0inputs+0outputs (19major+112713minor)pagefaults 0swaps

timing based benchmarks are tricky to debug... does import torch; print(torch.backends.mps.is_available()) work for you in this flake? also ooc what's the equivalent to nvtop for the MPS backend?

NotImplementedError: Could not run 'aten::empty.memory_format' with arguments from the 'SparseMPS' backend. This could be because the operator doesn't exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit https://fburl.com/ptmfixes for possible resolutions. 'aten::empty.memory_format' is only available for these backends: [CPU, MPS, Meta, QuantizedCPU, QuantizedMeta, MkldnnCPU, SparseCPU, SparseMeta, SparseCsrCPU, BackendSelect, Python, FuncTorchDynamicLayerBackMode, Functionalize, Named, Conjugate, Negative, ZeroTensor, ADInplaceOrView, AutogradOther, AutogradCPU, AutogradCUDA, AutogradHIP, AutogradXLA, AutogradMPS, AutogradIPU, AutogradXPU, AutogradHPU, AutogradVE, AutogradLazy, AutogradMeta, AutogradMTIA, AutogradPrivateUse1, AutogradPrivateUse2, AutogradPrivateUse3, AutogradNestedTensor, Tracer, AutocastCPU, AutocastCUDA, FuncTorchBatched, FuncTorchVmapMode, Batched, VmapMode, FuncTorchGradWrapper, PythonTLSSnapshot, FuncTorchDynamicLayerFrontMode, PythonDispatcher].

Yeah this error looks like it's an issue with whisper as opposed to nix's packaging of torch/torch-bin.
We can leave this open as a tracking issue for getting MPS support in our source build of torch however.

what version of python are you using?

Tried 39-311. I'm pinned to 23.05, I wonder if it's different in unstable?
Yup, that was it!
$ nix-shell -I nixpkgs=channel:nixpkgs-unstable -p 'python39.withPackages (ps: with ps; [ pytorch-bin ])' --command "python -c 'import torch; print(torch.backends.mps.is_available())'"
True

import torch; print(torch.backends.mps.is_available())

@samuela yes, I just patched this into the whisper code (in the cli() function) and confirmed my nix branch shows False and my nix-gpu branch shows the device as expected.

yup, that makes for two TODOs

[ ] Add MPS support to the source build python3Packages.torch
[ ] Add python 3.11, 3.12 support to torch-bin

I found that torchvision-bin support on aarch64-darwin effectively prevented MPS from being usable. Submitted https://github.com/NixOS/nixpkgs/pull/244716 to fix.

I can confirm that MPS support works with torch-bin and torchvision-bin now

I'm still trying to get it working. Tried adding the USE_MPS flag but it still results without MPS support; I think the build script uses xcrun to sort out some SDK path business, which fails to find the MPSGraph framework, and disables MPS:
bash: line 1: xcrun: command not found
-- MPS: unable to get MacOS sdk version
-- MPSGraph framework not found

https://github.com/pytorch/pytorch/blob/1da41157028ee8224e456f6fab18bc22fa2637fe/CMakeLists.txt#L99

Can any of the aarch64-darwin folks help me build this locally via nix develop?
Currently builds fine with nix build --offline (so I don't think it's a substituter issue) and nix shell.
Steps I'm taking / have tried:

nix develop .#python310Packages.pytorch
source $stdenv/setup
genericBuild

It looks like /usr/bin/xcrun bleeds into the environment, sets USE_MPS to true, and it fails to find the relevant SDK libraries (fixing this would hopefully help me close this issue, but I'd like to at least get it to build first).

nix develop -i .#python310Packages.pytorch --command bash --norc link
source $stdenv/setup
genericBuild

USE_MPS is set to false, but the build fails looking for #include <google/protobuf/implicit_weak_message.h>
After reading this issue, it looks like some of these are functions and some are variables, so I might need to e.g. eval "${buildPhase}" instead of just buildPhase.

nix develop -i .#python310Packages.pytorch --command bash --norc link
source $stdenv/setup
unpackPhase
eval $patchPhase
eval $preConfigurePhases
eval $configurePhase
cd source
eval $buildPhase

Here it fails with the same errors about protobuf.
I think these are relevant convos (for my future reference):

https://github.com/NixOS/nix/issues/6202
https://discourse.nixos.org/t/using-nix-develop-to-debug-builds/26024/4
https://discourse.nixos.org/t/differences-between-nix-build-and-development-shell/24060
https://discourse.nixos.org/t/unable-to-build-package-in-nix-shell-or-nix-develop-but-nix-build-or-nix-build-works-fine/15500

Can anyone point out what I'm getting wrong here? nix develop should be able to build it if nix build can, right? TIA for any help for a noob.

This issue has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/nix-develop-fails-with-command-bash-norc/31896/1

According to the pytorch cmakelists, it requires SDK >= 12.3 for MPS support, currently we only have:
$ nix eval --json --apply 'builtins.attrNames' github:nixos/nixpkgs/master#darwin | jq -r '.[] | select(contains("sdk"))'
apple_sdk
apple_sdk_10_12
apple_sdk_11_0

I've tried anyways, adding the MetalPerformanceShaders and MetalPerformanceShadersGraph to the build inputs and substituting xrun for xcbuild.xcrun with no luck. I suspect we might just be blocked on a newer SDK?

hmmm interesting... what's the process for adding a new apple_sdk version? cc @NixOS/darwin-maintainers

@samuela It's a long task with side-quests. See https://github.com/NixOS/nixpkgs/issues/242666 for some discussion - MoltenVK is also dependent on SDK 12. The work is ongoing on various aspects and the SDKs themselves have been proposed here: https://github.com/NixOS/nixpkgs/pull/229210

People looking for a work-around might find this flake helpful. It's based on @n8henrie's wheel approach.
nix run "github:david-r-cox/pytorch-darwin-env#verificationScript"

After https://github.com/NixOS/nixpkgs/pull/346043 lands, this should be doable. Just add apple-sdk_13 as a build input.

I tried to enable this with:
diff --git a/pkgs/development/python-modules/torch/default.nix b/pkgs/development/python-modules/torch/default.nix
index cd43c04d67d4..e7bbf4eb1b31 100644
--- a/pkgs/development/python-modules/torch/default.nix
+++ b/pkgs/development/python-modules/torch/default.nix
@@ -364,10 +364,14 @@ buildPythonPackage rec {
   # NB technical debt: building without NNPACK as workaround for missing `six`
   USE_NNPACK = 0;

+  # USE_PYTORCH_METAL = stdenv.hostPlatform.isDarwin;
+  USE_MPS = stdenv.hostPlatform.isDarwin;
+
   cmakeFlags =
     [
       # (lib.cmakeBool "CMAKE_FIND_DEBUG_MODE" true)
       (lib.cmakeFeature "CUDAToolkit_VERSION" cudaPackages.cudaVersion)
+      (lib.cmakeBool "USE_MPS" stdenv.hostPlatform.isDarwin)
     ]
     ++ lib.optionals cudaSupport [
       # Unbreaks version discovery in enable_language(CUDA) when wrapping nvcc with ccache
@@ -519,8 +523,12 @@ buildPythonPackage rec {
     ++ lib.optionals (cudaSupport || rocmSupport) [ effectiveMagma ]
     ++ lib.optionals stdenv.hostPlatform.isLinux [ numactl ]
     ++ lib.optionals stdenv.hostPlatform.isDarwin [
-      darwin.apple_sdk.frameworks.Accelerate
-      darwin.apple_sdk.frameworks.CoreServices
+      darwin.apple_sdk_12_3.frameworks.Accelerate
+      darwin.apple_sdk_12_3.frameworks.CoreServices
+      darwin.apple_sdk_12_3.frameworks.Metal
+      darwin.apple_sdk_12_3.frameworks.MetalKit
+      darwin.apple_sdk_12_3.frameworks.MetalPerformanceShaders
+      darwin.apple_sdk_12_3.frameworks.MetalPerformanceShadersGraph
       darwin.libobjc
     ]
     ++ lib.optionals tritonSupport [ _tritonEffective ]

I'm still getting the xcrun error:
python3.12-torch> bash: line 1: xcrun: command not found
python3.12-torch> -- MPS: unable to get MacOS sdk version
python3.12-torch> -- MPSGraph framework not found

I was waiting on @reckenrode's recent massive SDK refactor to land (at least in master, still staging as of this AM) and will retry then.

My mistake, didn't realize #346043 was not yet in the master.

A couple sites helpful for tracking -- the best known is https://nixpk.gs/pr-tracker.html?pr=346043
(I really wish we could get an RSS feed for when user-specific PRs land in a specific branch.)

It’s in staging-next, you can just target a PR to that. Many things have already been built.

I rebased my hack on top of staging-next, now I got different errors:
-- CLANG_VERSION_STRING:         clang version 16.0.6
Target: arm64-apple-darwin
Thread model: posix
InstalledDir: /nix/store/r0w268903vjlq1vrnajb3lvll4hs5z8g-clang-16.0.6/bin

warning: unhandled Target key DefaultVariant
warning: unhandled Target key SupportedTargets
warning: unhandled Target key VersionMap
warning: unhandled Target key Variants
warning: unhandled Target key DebuggerOptions
warning: unhandled Product key iOSSupportVersion
warning: unhandled Target key DefaultVariant
warning: unhandled Target key SupportedTargets
warning: unhandled Target key VersionMap
warning: unhandled Target key Variants
warning: unhandled Target key DebuggerOptions
warning: unhandled Product key iOSSupportVersion
warning: unhandled Target key DefaultVariant
warning: unhandled Target key SupportedTargets
warning: unhandled Target key VersionMap
warning: unhandled Target key Variants
warning: unhandled Target key DebuggerOptions
warning: unhandled Product key iOSSupportVersion
-- sdk version: 11.3, mps supported: OFF
warning: unhandled Target key DefaultVariant
warning: unhandled Target key SupportedTargets
warning: unhandled Target key VersionMap
warning: unhandled Target key Variants
warning: unhandled Target key DebuggerOptions
warning: unhandled Product key iOSSupportVersion
warning: unhandled Target key DefaultVariant
warning: unhandled Target key SupportedTargets
warning: unhandled Target key VersionMap
warning: unhandled Target key Variants
warning: unhandled Target key DebuggerOptions
warning: unhandled Product key iOSSupportVersion
warning: unhandled Target key DefaultVariant
warning: unhandled Target key SupportedTargets
warning: unhandled Target key VersionMap
warning: unhandled Target key Variants
warning: unhandled Target key DebuggerOptions
warning: unhandled Product key iOSSupportVersion
-- MPSGraph framework not found
-- Could not find ccache. Consider installing ccache to speed up compilation.

It now says SDK version is 11.3, maybe I need to change something else as well in that package, as I was trying to specify I want version 12.3

This was one of my test cases. Add apple-sdk_13 as a buildInput. Delete the ild framework junk. That should be it.
It has to be the 13.3 SDK because MPS isn’t available in older SDKs, and the version of pytorch in nixpkgs does not build with the 14.4 (and presumably 15.0) SDK.

It now says SDK version is 11.3, maybe I need to change something else as well in that package, as I was trying to specify I want version 12.3

After the update, the old SDK frameworks are stubs that do nothing on their own . If you used overrideSDK stdenv "12.3", you would get the 12.3 SDK, but using the old pattern is deprecated. After a tree-wide after 24.11, it will be moved to darwin-aliases.nix and warn on use.

Looks like success!
~/nixpkgs.git/ > nix-shell -p 'python3.withPackages (ps: with ps; [ torch ])' -I nixpkgs=.

[nix-shell:~/nixpkgs.git]$ python
Python 3.12.7 (main, Oct  1 2024, 02:05:46) [Clang 16.0.6 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> print(f"PyTorch version: {torch.__version__}")
PyTorch version: 2.5.0
>>> print(f"MPS available: {torch.backends.mps.is_available()}")
MPS available: True

Just for tests I also tried to use apple-sdk_12 but I got these errors:
python3.12-torch>   return MTLLanguageVersion3_0;
python3.12-torch>          ^~~~~~~~~~~~~~~~~~~~~
python3.12-torch>          MTLLanguageVersion2_0
python3.12-torch> /nix/store/2vs624s5vsdnhralhwlin5gnid2dl1qy-apple-sdk-12.3/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks/Metal.framework/Headers/MTLLibrary.h:185:5: note: 'MTLLanguageVersion2_0' declared here
python3.12-torch>     MTLLanguageVersion2_0 API_AVAILABLE(macos(10.13), ios(11.0)) = (2 << 16),
python3.12-torch>     ^
python3.12-torch> 1 error generated.
python3.12-torch> [2278/5844] Building CXX object caffe2/CMakeFiles/torch_cpu.dir/__/torch/csrc/api/src/serialize/output-archive.cpp.o
python3.12-torch> ninja: build stopped: subcommand failed.
python3.12-torch> ERROR Backend subprocess exited when trying to invoke build_wheel
error: builder for '/nix/store/n41ysm7qfk9azq8w09vs5l8iwvmwqvr3-python3.12-torch-2.5.0.drv' failed with exit code 1;
       last 10 log lines:
       >          ^~~~~~~~~~~~~~~~~~~~~
       >          MTLLanguageVersion2_0
       > /nix/store/2vs624s5vsdnhralhwlin5gnid2dl1qy-apple-sdk-12.3/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks/Metal.framework/Headers/MTLLibrary.h:185:5: note: 'MTLLanguageVersion2_0' declared here
       >     MTLLanguageVersion2_0 API_AVAILABLE(macos(10.13), ios(11.0)) = (2 << 16),
       >     ^
       > 1 error generated.
       > [2278/5844] Building CXX object caffe2/CMakeFiles/torch_cpu.dir/__/torch/csrc/api/src/serialize/output-archive.cpp.o
       > ninja: build stopped: subcommand failed.
       >
       > ERROR Backend subprocess exited when trying to invoke build_wheel
       For full logs, run 'nix log /nix/store/n41ysm7qfk9azq8w09vs5l8iwvmwqvr3-python3.12-torch-2.5.0.drv'.

So I guess 13 is way to go

I made a PR to staging-next now: #351778


typo: AppUsermodelExcludeFroeShowInNewInstall -> AppUserModelExcludeFromShowInNewInstall

Froe -> From
Usermodel -> UserModel

@Dansimp @lomayor @AndreaBarr - PR is ready for final review. This is just a typo issue. Thanks.

@Dansimp  @lomayor @AndreaBarr  - PR is ready for final review. This is just a typo issue. Thanks.


How to get playable duration outside of onProgress

How can I get access to playable duration (which is the buffer time) outside of onProgress? I need to be able to show how much a video has buffered even if its paused but I can only get playable duration in onProgress and onProgress is not triggered when the video is paused.

We don't currently have that capability. I think the best approach would be to see if we can generate buffering events at regular intervals while paused that contain that info. I don't have the time to tackle that at the moment, so you will have to look into it yourself or see if another contributor can help.

Hey, yeah I've come up with a similar solution and have got it working on my end. Thanks.

Hey, yeah I've come up with a similar solution and have got it working on my end. Thanks.

@msaad99 would you mind sharing your solution here?

@msaad99  I am also facing this issue, can you please share how did you implemented that

@msaad99 @FahadRS would you mind to share something about the above as i am also facing the same issue.


Fixed error being thrown if path was not provided while saving a new file

fixes #9

Thanks for the fix!


Create and use ome-tiff-tiler docker container

There are a lot of tricky details that may be hard to make platform independent: A docker container would be good for this. It probably has two mounts: The ome-tiff, and the directory where the tiles should be created. Everything else should be an environment variable. And maybe make a pythonic wrapper, so the end user doesn't need to think in terms of envvars.

Closed by #60.


JS error in Opencart's checkout - 'dateFormat' in bootstrap-datetimepicker.min.js

Hello, I am having this error in checkout page of my site. Input field is defined as

<input type="text" name="custom_field[<?php echo $custom_field['location']; ?>][<?php echo $custom_field['custom_field_id']; ?>]" value="<?php echo $custom_field['value']; ?>" placeholder="<?php echo $custom_field['name']; ?>" data-date-format="YYYY-MM-DD HH:mm" id="input-payment-custom-field<?php echo $custom_field['custom_field_id']; ?>" class="form-control" />

<button type="button" class="btn btn-default"></button>

And the jquery is written as

But still it shows error
"Uncaught TypeError: Cannot read property 'dateFormat' of undefined"
If you need any extra information, please let me know. Your quick response is highly appreciated. As the site is currently live and unable to process order.

Hi, not sure if you're using either:

A, really, older version of this picker
Using wrong picker

Why? Because the options pickTime pickDate are not used here.
data-date-format="YYYY-MM-DD HH:mm" is working for me as expected
You must check which datetimepicker you're using for this, moreover than that, this doesn't seems a bug but reither a missuing, if anymore problems enconter feel free to get back or find help on StackOverflow


Optimize several regexes from quadratic time to linear time

Part of the discussion in #798.

This looks good to me. Thank you.
@facelessuser do you have any input on this?

Generally looks good, I just want to check out the test change. I should be able to finish the review in a bit.

LGTM


Proposed: Add Vagrantfile for build

While trying to understand the build I wrote this Vagrantfile.
Might be useful
Tested with:
$ vagrant up
$ vagrant ssh

In the VM:
$ cd src/github.com/kubernetes/deployment-manager
$ make bootstrap
$ make build

Might be worth putting it in a folder, so that folks don't think it's required. I suppose they'd know that if they read the README, but given that it's not required, seems cleaner to move it down a level. What do you think?

Some documentation on what this is for would be good. I was expecting it to start a DM cluster inside of Kubernetes. But it looks like it's actually just a VM with the source code mounted into it. I'm not really sure what to do with it.
I think we could add it to hack/ if it's primarily for builds or developers.

@jackgr @technosophos putting it in hack is fine.
It's indeed just a build machine. I like to make sure I can reproduce environment and not make any local setup assumptions.

@runseb please address the comments and resubmit.

@jackgr how about I put a small README in /hack and write some small instructions there.
It might be confusing to put it in the main README.

Fine. I think the key is to put the Vagrantfile in hack, which it isn't in
this PR.
Jack Greenfield | Google
On Wed, Mar 9, 2016 at 7:44 AM, runseb<EMAIL_ADDRESS>wrote:

@jackgr https://github.com/jackgr how about I put a small README in
/hack and write some small instructions there.
It might be confusing to put it in the main README.
—
Reply to this email directly or view it on GitHub
https://github.com/kubernetes/deployment-manager/pull/329#issuecomment-194353463
.

@jackgr yes it is, I just forced pushed on the PR..so there is a README and the vagrant file under /hack

I see. thanks.
Jack Greenfield | Google
On Wed, Mar 9, 2016 at 10:03 AM, runseb<EMAIL_ADDRESS>wrote:

@jackgr https://github.com/jackgr yes it is, I just forced pushed on
the PR..so there is a README and the vagrant file under /hack
—
Reply to this email directly or view it on GitHub
https://github.com/kubernetes/deployment-manager/pull/329#issuecomment-194426750
.


RTL Languages text alignment

Hi
RTL Languages (like:arabic, persian...) are supported on twitter website and tweets shows in the correct alignment  right-to-left). In app RTL tweets are in the same text alignment as English (LTR). Can you fix this please?

This is a fair request, but it's been 3 years since your proposal. So I'd like to check if you're still interested in this. Please reopen this if you (or any other person) need one.
Also, I'd appreciate an example tweet or user with RTL languages for testing the feature when this is reopened. Thanks.


Windows Build issues

Windows CMake Build issues fixing.

The 2.16 branch as now reached and of life, so I'm closing this PR. Feel free to open a new PR based on one of the active branches (development and mbedtls-2.28) if the bug is also present there. Thanks for you contribution.


[CDMF-12] Need to validate the ownership of user before executing any action on a device

There should be a way to verify the ownership of the device before executing any action on a device. Intention of this functionality is to prevent users (not emm-admins) from executing actions on other users devices.Reference: https://wso2.org/jira/browse/CDMF-12

This is fixed, hence closing.  (This is for operation level)


Implement model circuit and run it

https://github.com/quantumlib/Cirq/blob/284499ddf3b50faea1ac9a15e28bb6b9fe306d58/examples/advanced/quantum_volume.py#L128

This should be fixed now, see contrib/quantum_volume. Can someone with permissions close it, please?

Note to anyone: Remove that TODO line when possible.

Thanks! That TODO should be gone: https://github.com/quantumlib/Cirq/blob/master/cirq/contrib/quantum_volume/quantum_volume.py


Replace fallthrough comment with C++17's attribute

Sorry for making 3 PRs for the same kind of issue, I was not able to verify this file earlier.
This time, however, I grepped the whole project for similar comments and this is the final one.

As of f3cd220:
$ git grep -in fall.\*thr
src/CalcManager/CEngine/scifunc.cpp:172:            // This case falls through to IDC_DMS case because in the old Win32 Calc,
src/CalcManager/CEngine/scifunc.cpp:176:            [[fallthrough]];
src/CalcManager/CalculatorManager.cpp:295:            [[fallthrough]];
src/CalcManager/Ratpack/conv.cpp:615:            [[fallthrough]];
src/CalcManager/Ratpack/conv.cpp:650:            [[fallthrough]];


Access the transaction prisma Client in the middleware

Problem
When adding a Prisma middleware for creating an Audit trail, there is no way to get the 'before' value when running via a transaction.
Suggested solution
We have access to the params.runInTransaction value (which I'm not sure of the purpose of this), but we don't have access to the actual Prisma transaction client. If this can be made available, my naive hope is that a lookup could be made before running next() to get the values of the object before updating and be able to save them to an audit table.  As it is now, the lookup that runs before next() is called returns the same values as after it is called.
Alternatives
It seems like I can have the safety of a transaction in case of failure at the loss of not having an audit trail. Or I can have the Audit trail and not use transactions.
Additional context
Here is my middleware:
import { Prisma } from "@prisma/client"
import _ from "lodash"
import { toCamelCase } from "./functions/stringFunctions"
import prisma from "./prisma"
import equal from "fast-deep-equal"

const modelsToIgnore: Prisma.ModelName[] = [
  Prisma.ModelName.Audit,
  Prisma.ModelName.Account,
  Prisma.ModelName.Session,
  Prisma.ModelName.VerificationToken,
]
const modelsWithoutLocationId: Prisma.ModelName[] = [
  Prisma.ModelName.Account,
  Prisma.ModelName.Session,
  Prisma.ModelName.VerificationToken,
  Prisma.ModelName.Address,
  Prisma.ModelName.UnionJar,
  Prisma.ModelName.TransferJar,
  Prisma.ModelName.TransferBarrel,
]

const uniqueActions: Prisma.PrismaAction[] = ["update", "delete", "upsert"]
const manyActions: Prisma.PrismaAction[] = ["updateMany", "deleteMany"]
const actionsToAudit: Prisma.PrismaAction[] = uniqueActions.concat(manyActions)

export const auditMiddleware = async (params: Prisma.MiddlewareParams, next: any) => {
  const { action, model } = params
  if (!actionsToAudit.includes(action) || !model || modelsToIgnore.includes(model)) {
    return next(params)
  }

  let oldValue = null
  let newValue = null
  let modelId: number | null = null
  let modelIds: number[] = []
  let locationId: number | null = null

  const beforeObject = await getOldValue(params)

  if (model === Prisma.ModelName.Location) {
    locationId = beforeObject.id
  } else if (model === Prisma.ModelName.Preparo) {
    locationId = beforeObject.destinationLocationId
  } else if (model === Prisma.ModelName.Transfer) {
    locationId = beforeObject.requestingLocationId
  } else if (!modelsWithoutLocationId.includes(model)) {
    locationId = beforeObject?.locationId
  }

  // Get id(s) for audit
  if (manyActions.includes(action)) {
    modelIds = beforeObject.map((item: any) => item.id)
  } else {
    modelId = beforeObject.id
  }

  // perform the actual db operation
  const result = await next(params)

  // No need to get data on delete it will be null
  if (action === "delete" || action === "deleteMany") {
    oldValue = beforeObject
  } else {
    const newObject = await getNewValue(params)

    // If the old and new values are the same we don't need to log it (update without changes)
    if (equal(newObject, beforeObject) && !params.runInTransaction) {
     // HERE IS WHERE IT IS ALWAYS EQUAL IN A TRANSACTION
      return result
    }
    if (action === "updateMany" && Array.isArray(beforeObject) && Array.isArray(newObject)) {
      // get the difference between first item in array, because the update will be the same for all items
      newValue = difference(beforeObject[0], newObject[0])
      oldValue = difference(newObject[0], beforeObject[0])
    } else {
      // get the difference between old and new values
      newValue = difference(beforeObject, newObject)
      oldValue = difference(newObject, beforeObject)
    }
  }

  try {
    await prisma.audit.create({
      data: {
        model,
        modelId,
        modelIds,
        action,
        locationId,
        oldValue: JSON.stringify(oldValue),
        newValue: JSON.stringify(newValue),
        // TODO: add userId, but Prisma does not know who the user is
        // userId: params.args.data.userId,
      },
    })
  } catch (e) {
    // Don't throw error if audit fails, only log it
    console.error("Failed to create Audit", e)
  }
  return result
}

const getOldValue = async (params: Prisma.MiddlewareParams) => {
  const { action, model, args } = params
  if (model) {
    const modelName = toCamelCase(model)
    if (uniqueActions.includes(action)) {
      //@ts-ignore model name will match up "Location" -> "location"
      return await prisma[modelName].findUnique({ where: args.where })
    } else if (manyActions.includes(action)) {
      // @ts-ignore model name will match up "Location" -> "location"
      return await prisma[modelName].findMany({ where: args.where })
    } else {
      console.warn("Unhandled action: ", action)
      return null
    }
  }
  console.warn("No model")
  return null
}

const getNewValue = async (params: Prisma.MiddlewareParams) => {
  const { action, model, args } = params
  if (model) {
    const modelName = toCamelCase(model)
    if (uniqueActions.includes(action)) {
      //@ts-ignore model name will match up "Location" -> "location"
      return await prisma[modelName].findUnique({ where: args.where })
    } else if (manyActions.includes(action)) {
      //@ts-ignore model name will match up "Location" -> "location"
      return await prisma[modelName].findMany({ where: args.where })
    } else {
      console.warn("Unhandled action: ", action)
      return null
    }
  }
  console.warn("No model")
  return null
}

/**
 * Find difference between two objects
 * @param  {object} origObj - Source object to compare newObj against
 * @param  {object} newObj  - New object with potential changes
 * @return {object} differences
 */
const difference = (origObj: any, newObj: any) => {
  function changes(newObj: any, origObj: any) {
    let arrayIndexCounter = 0
    return _.transform(newObj, function (result: any, value: any, key) {
      if (!_.isEqual(value, origObj[key])) {
        let resultKey = _.isArray(origObj) ? arrayIndexCounter++ : key
        result[resultKey] =
          _.isObject(value) && _.isObject(origObj[key]) ? changes(value, origObj[key]) : value
      }
    })
  }
  return changes(newObj, origObj)
}

Related to #1902, #4477, #6882 (because a user ID would be ideal in an audit log)

Here is what I have found that is working. By modifying the original code above:
...
  // perform the actual db operation
  const result = await next(params)

  // No need to get data on delete it will be null
  if (action === "delete" || action === "deleteMany") {
    oldValue = beforeObject
  } else {
    let newObject: any = null
    if (runInTransaction) {
      newObject = result
    } else {
      newObject = await getNewValue(params)
    }
...

It seems to me that I need to use the result value instead of the value that I look up. I would like to understand better how this is the case. Why would the getNewValue function still return the beforeObject instead of a new object when run in a transaction?

Are there any updates on this issue, at the moment I have removed all $transaction queries which may involve audit logging which is not desirable. Thanks!

It's unfortunate that middleware deprecated because extensions are not working out for us due to its type safety, ironically. So I guess there is no way to do this in Prisma.


Unable to import from from tensorflow.python.keras.engine

resnet50.py#L30 errors out with ModuleNotFoundError: No module named 'tensorflow.python.keras.engine'

Linux Mint 18
Python 3.6.2
Tensorflow 1.8.0 (CPU version)

Caused by 2bd7c979c5fc1ada90d38f6959839a99f8af4163

@bdwyer2 Yeah I think the tensorflow.python.keras._impl was removed in master recently, so I made a change.  Would be good to have a fix that works on both versions.


new(libsinsp): add concatenated process lineage filter fields + sinsp_filter_check_thread helpers cleanup

What type of PR is this?

Uncomment one (or more) /kind <> lines:

/kind bug

/kind cleanup

/kind design

/kind documentation

/kind failing-test

/kind feature

Any specific area of the project related to this PR?

Uncomment one (or more) /area <> lines:

/area API-version

/area build

/area CI

/area driver-kmod

/area driver-bpf

/area driver-modern-bpf

/area libscap-engine-bpf

/area libscap-engine-gvisor

/area libscap-engine-kmod

/area libscap-engine-modern-bpf

/area libscap-engine-nodriver

/area libscap-engine-noop

/area libscap-engine-source-plugin

/area libscap-engine-savefile

/area libscap

/area libpman

/area libsinsp

/area tests

/area proposals

Does this PR require a change in the driver versions?

/version driver-API-version-major

/version driver-API-version-minor

/version driver-API-version-patch

/version driver-SCHEMA-version-major

/version driver-SCHEMA-version-minor

/version driver-SCHEMA-version-patch

What this PR does / why we need it:

Less loops and more efficient if you want to export all process ancestors up to a certain level anyways
More convenient and intuitive display of information -> security analysts prefer an output like this /bin/java->/bin/bash->/bin/python->/bin/bash to quickly understand the process origins
Enhanced threat detection capabilities -> can string match exact lineage / sequence
Needed in the planned anomaly detection framework

Note: I don't think we need this for the other proc.a* fields aka only for the process names, exe, and exepaths ...
Which issue(s) this PR fixes:

Fixes #
Special notes for your reviewer:
Does this PR introduce a user-facing change?:

new(libsinsp): add concatenated process lineage fields as filter / display fields

CC @loresuso @darryk10

Yeah I had similar thoughts that we are getting to a point where there is lots of code duplication in this regard (not limited to the type of fields touched in this PR).
I'll check what makes sense to clean up in this PR!

@Andreagit97 you see I added 3 cleanup commits on top, performing significant consolidation with new helpers, beyond the initial request. WDYT and do you have additional ideas?

@incertum

Can you rebase pls? Then the sanitizer CI jobs should run.

It seems that while simply rebasing worked, there are updates needed because a few things changed I suspect ... will look into it soon.

@incertum we still have CI issues. I'll ping you once they are fixed.

While I was reviewing this, I opted for a slightly different methods organization, let me know if it works for you, otherwise feel free to revert it! In the meanwhile I've addressed my own review comments and the new changes should solve the visibility issues you faced after the rebase :)

LGTM, thanks!

Jumping in late because I lost sight of this, but I'm just wondering why a more general-purpose field with EPF_IS_LIST (e.g. fd.types) flag was not considered. It would allow the more performant in, intersectes, and pmatch operations, while still being printable as a comma-separated list (you'd lose the ->, that's true), and wouldn't add yet another edge case to the way we execute field comparisons. For clarity, I have nothing against these changes, I'm just wondering if we could achieve the same thing by using something we already support natively in our model, and if not what are the pros of introducing another "special" field.

yep I agree with Jason, reviewing this it was not clear to me what the use case of this -> concatenation...it's cool to see but I don't understand the difference with a simple list :thinking:

Jumping in late because I lost sight of this, but I'm just wondering why a more general-purpose field with EPF_IS_LIST (e.g. fd.types) flag was not considered. It would allow the more performant in, intersectes, and pmatch operations, while still being printable as a comma-separated list (you'd lose the ->, that's true), and wouldn't add yet another edge case to the way we execute field comparisons. For clarity, I have nothing against these changes, I'm just wondering if we could achieve the same thing by using something we already support natively in our model, and if not what are the pros of introducing another "special" field.

I totally agree with reusing EPF_IS_LIST, instead of reinventing the wheel and adding more edge cases.
Moreover, using EPF_IS_LIST would satisfy all points here:

Less loops and more efficient if you want to export all process ancestors up to a certain level anyways
More convenient and intuitive display of information -> security analysts prefer an output like this /bin/java->/bin/bash->/bin/python->/bin/bash to quickly understand the process origins
Enhanced threat detection capabilities -> can string match exact lineage / sequence
Needed in the planned anomaly detection framework

With the only difference of , instead of ->. Anyway, comma-separated convention is more consistent across all functionality offered by libs, so even in this case I don't see a compelling reason to change this.

Can we support both? The original motivation was that security analysts who triage these logs want the process lineage as human readable string specifically with the -> delimiter. It would complement and be consistent with the fd.name for ip:port tuples display. That's the output field display part.
Re the filtering yes I totally agree as well and I wasn't aware of EPF_IS_LIST.

Can we support both?

I would avoid doing this in the specific field implementation. In the long term, supporting multiple solutions for the same thing will create more trouble than benefits. We already have several similar situations in legacy fields, and honestly, it's a pain.

The original motivation was that security analysts who triage these logs want the process lineage as human readable string specifically with the -> delimiter. It would complement and be consistent with the fd.name for ip:port tuples display. That's the output field display part.

If the motivation that drives this change is primarily aesthetics, I would rather (a) use EPF_IS_LIST (in this PR) and (b) introduce a generic formatting option that allows the list separator to be specified (in another PR)

If the motivation that drives this change is primarily aesthetics, I would rather (a) use EPF_IS_LIST (in this PR) and (b) introduce a generic formatting option that allows the list separator to be specified (in another PR)

ok let's do this then, any preferences in field naming given these changes?

ok let's do this then, any preferences in field naming given these changes?

Not sure about this. Perhaps, maybe an implicit convention is to use plural nouns (eg. proc.aexes) for lists. But I'm not sure if we like this. cc @falcosecurity/libs-maintainers

or proc.aexe.lineage which doesn't disclose the data type as concat is usually for strings only.

Hey, actually the list type would truly defy the original purpose. We do not need the operators "in", "exists" or "intersects" -- at least not for the use case I am intending to introduce here. I would like to exactly match on a string sub-sequence, for example zsh->bash aka match on all logs that have that sub lineage in the process tree lineage of that nature -- with a list type I cannot easily do that during event filtering (EBPF_IS_LIST isn't meant for sublist matching and for fd.types for example we don't even care about the insertion order).  Also isn't the proc.aname in (zsh, bash) already doing what we need in cases where we just want to check if zsh and bash are anywhere in the lineage? Again I would like to match exactly on zsh->bash, which we cannot do with proc.aname today; it's a different use case.
Furthermore, I still don't get why -> is not ok for display, especially I wouldn't even want this to be associated with a list concept. It should be treated like fd.name for ips, e.g. we get a string like this <IP_ADDRESS>:22->127.0.01:2367 for fd.name. proc.env has also it's own string representation nature, so do the k8s pod labels have one, meaning I truly don't see what it should be an issue.
Question: Have you worked on incidents and used these type of fields? My feedback and recommendations stem form real-life and analysts feedback and is not my personal bias. The arrow gives clear indication about the direction of the lineage - analysts specifically need this to resolve incidents.
@jasondellaluce could you elaborate so that I understand your perspective better?

I would like to exactly match on a string sub-sequence, for example zsh->bash aka match on all logs that have that sub lineage in the process tree lineage of that nature -- with a list type I cannot easily do that during event filtering (EBPF_IS_LIST isn't meant for sublist matching and for fd.types for example we don't even care about the insertion order). Also isn't the proc.aname in (zsh, bash) already doing what we need in cases where we just want to check if zsh and bash are anywhere in the lineage? Again I would like to match exactly on zsh->bash, which we cannot do with proc.aname today; it's a different use case.

The use case is legit, but the proposed solution is sub-optimal. Performing a full-text search of zsh->bash into a very long string with all the lineage will create performance penalties. I would rather go for a new operator for lists to satisfy the specific requirement of this use case.

Furthermore, I still don't get why -> is not ok for display, especially I wouldn't even want this to be associated with a list concept. It should be treated like fd.name for ips, e.g. we get a string like this <IP_ADDRESS>:22->127.0.01:2367 for fd.name. proc.env has also it's own string representation format, so do the k8s pod labels have one, meaning I truly don't see why it should be an issue.

The separtor is a display/formatting issue indeed, not a way to represent the data. Thus, I'm not against -> perse, I'm just against implementing it at the field level for lists (or anything similar to a list).,  Moreover, <IP_ADDRESS>:22->127.0.01:2367 is not a list, and using the same separator by default would not help with consistency. Just think of a list of ``<IP_ADDRESS>:22->127.0.01:2367`... how would we separate them?

Considering what we have today for the proc name (it is the same for exepath, exe, ...), I would try to understand what would be the end goal here...
Today we have:
proc.name
proc.pname
proc.sname
proc.vpgid.name
proc.aname
proc.concat_aname // in this PR

A possible solution could be
proc.name
proc.aname[]
proc.lineage[]

Where:

proc.name is the same as today
proc.aname[] which always took an argument. It could be an index (0,1,2,...) or a specific key (SESSION, PARENT, GROUP). We cover proc.pname, proc.sname, proc.vpgid.name, proc.aname.
proc.lineage[] which is a list and could take an argument. if used without arguments it returns a list with the whole process lineage, if with an argument (an index 0,1,2,...) returns the lineage starting from the index.

Moreover, we need to implement other operators on the list, like =, startwith, ... to cover all the use cases we want with the aforementioned filter checks.
In this way, we can uniform our filter checks and solve the issues we have highlighted in this PR with ->. WDYT?

Moreover, we need to implement other operators on the list, like =, startwith, ... to cover all the use cases we want with the aforementioned filter checks.

:+1:

re https://github.com/falcosecurity/libs/pull/1625#issuecomment-1988173684

proc.aname[] ... proc.aname[] which always takes an argument

This would break the existing use of proc.aname in the filter expression where we traverse all levels up.
proc.name
proc.pname
proc.sname
proc.vpgid.name
proc.aname

Frankly I think the current variants are actually pretty clear, I don't see direct benefits of changing the naming. If you don't mind me asking, what are the benefits? It could also backfire and be less user friendly? WDYT?
From my perspective if anything we should create a super generic filtercheck class that let's folks access any thread property at any level etc. Something definitely beyond the more immediate improvement suggested in this PR, but likely worth it and it would be a bit in line with what @Andreagit97 suggested here.

solve the issues we have highlighted in this PR with ->

Forgive me, I still don't understand the issues with -> to display a lineage / linkage? It appears to be the best choice as a process lineage encodes the order and direction and is not just a list. Should we rather talk of EBPF_IS_LINKED_LIST or something that is closer to our internal concept of IP TUPLES -- anything that directly implies the direction. Now if we want an unordered list as well, that's ok, but we definitely need the new EBPF_IS_LINKED_LIST concept. Graphics or tutorials around linked lists always use -> to explain the links.
@leogr

The use case is legit, but the proposed solution is sub-optimal. Performing a full-text search of zsh->bash into a very long string with all the lineage will create performance penalties. I would rather go for a new operator for lists to satisfy the specific requirement of this use case.

I believe it comes down to the algorithm. Many rules search in proc.cmdline, perhaps we could up our game and make string searching way more performant across the board? What's our current Big O and algo?
Strings are extremely powerful, for example we could now also search for ->zsh->bash implying there was a parent to that sub process tree lineage or zsh->bash-> implying that there was a child ... sublists search couldn't support that, BUT ...
@Andreagit97

Moreover, we need to implement other operators on the list, like =, startwith, ... to cover all the use cases we want with the aforementioned filter checks.

This also opens up new searches as we could now search for sublists where the executable path, for example, ended with a string ... something the string representation cannot truly handle the way we would want to.

In summary:
Understanding that the filterchecks are the core of effectively using Falco, I rather would like to expose too many variants and empower end users. Comparatively they have been much easier to maintain. Historically we have had many many more issues in our parsers.
Let's support:

string representation of the lineage
new linked list concept
and why not an unordered list concept as well (internally) for more performant intersection searches (it could make proc.aname in (x, y, z) even more performant)?

re #1625 (comment)

proc.aname[] ... proc.aname[] which always takes an argument

This would break the existing use of proc.aname in the filter expression where we traverse all levels up.

According to its description in sinsp
When used without any arguments, proc.aname is applicable only in filters and matches any of the process ancestors. For instance, you can use `proc.aname=bash` to match any process ancestor whose name is `bash`

This behavior should be covered by the new proc.lineage, which will provide a list with the whole lineage and so you could do something like (proc.lineage) intersects bash

proc.name
proc.pname
proc.sname
proc.vpgid.name
proc.aname

Frankly I think the current variants are actually pretty clear, I don't see direct benefits of changing the naming. If you don't mind me asking, what are the benefits? It could also backfire and be less user friendly? WDYT? From my perspective if anything we should create a super generic filtercheck class that let's folks access any thread property at any level etc. Something definitely beyond the more immediate improvement suggested in this PR, but likely worth it and it would be a bit in line with what @Andreagit97 suggested here.

I just think that these filter checks are doing almost the same thing in the end, they are accessing a single ancestor in the lineage. Providing users with just one method for doing it, is more intuitive IMO. Of course, this is just my idea let's see what other thinks. Moreover having just one unique method will simplify our lives and will help us keep the code "bug-free". BTW yes this would be a long-term plan,

solve the issues we have highlighted in this PR with ->

Forgive me, I still don't understand the issues with -> to display a lineage / linkage? It appears to be the best choice as a process lineage encodes the order and direction and is not just a list. Should we rather talk of EBPF_IS_LINKED_LIST or something that is closer to our internal concept of IP TUPLES -- anything that directly implies the direction. Now if we want an unordered list as well, that's ok, but we definitely need the new EBPF_IS_LINKED_LIST concept. Graphics or tutorials around linked lists always use -> to explain the links.

Uhm this is not an issue, I used the wrong word sorry, it's just a matter of what Jason well explained here https://github.com/falcosecurity/libs/pull/1625#issuecomment-1985315625. Why do we need to support a new string representation of the lineage with -> when we can do a very similar thing with what we have today?
Let's consider again the new proposed filter check proc.lineage[] + the startwith operation between lists. Writing something like proc.lineage startwith (tail, bash) should do exactly what we are talking about... please note that proc.lineage will contain all the ancestors already in order, so for example proc.lineage -> tail, bash, containerd-shim, systemd. The only thing that changes here is the representation so (tail, bash) instead of (tail->bash), or maybe i'm missing something...
Moreover, as a side note, these days we are working on the possibility of comparing 2 filter checks so proc.name = proc.exe and on the possibility of adding new modifiers like toupper(evt.arg.name) = toupper(fd.path) so having less corner cases to handle would surely simplify things. Just to provide you with an example, at the moment we cannot implement the 2 aforementioned features on all the fields like proc.aexe or proc.aexepath  because they are using a custom comparing logic, and so we need to uniform them with others before supporting these new features. So I'm not saying we shouldn't add new approaches or new custom logic, I'm just saying that maybe we could obtain all we need from what we have already in place or that we will try to add in the next weeks.

Hey folks,
It looks like we have a lot of ideas here, but we haven't reached a consensus, so I will try to summarize the discussion, collect feedback, and come back with a design proposal.
/assign

Hey folks,
It looks like we have a lot of ideas here, but we haven't reached a consensus, so I will try to summarize the discussion, collect feedback, and come back with a design proposal.
/assign

SGTM Leo.

Also depending on the proposal we may end up using this PR just for some sinsp_filter_check_thread helpers cleanup and create a new one for the new fields? We will see, whichever makes the most sense.

Proposal [1/2]: introducing join(<list>, <sep>) for concatenating list with a custom separator
This first part of my proposal aims to address the displaying issue raised in this discussion. Later, I will post the [2/2] part of the proposal, which focuses on the filtering issue.
Requirements: This proposal depends on https://github.com/falcosecurity/libs/issues/1789, which needs to be implemented first.
Use case: string representation of the processes lineage (as reported by @incertum)
Solution:

introducing proc.lineage, which is a list (ie. EPF_IS_LIST) with the whole process lineage (similar to the @Andreagit97 proposal, we can discuss if this field should support arguments separately)
introducing the join(<list>, <sep>) transformer, as proposed by me in this comment

As a result, we can use join(proc.lineage, "->") in the output: field of a rule.
Note: This solution might also address the filtering part but with some performance penalty. For instance, one could use join(proc.lineage, "->") contains ->zsh->bash to match a sequence. At the functional level, this is equivalent to what is already implemented by this PR. However, I'd discourage users from doing so because performing a full-text search in a long string usually comes with severe performance penalties (we have seen this in the past). It might still a legit workaround until we have a specific implementation to deal with sequences (I will post my solution in the 2nd part of this proposal).
Action items:

If we reach a consensus on this part of the proposal, I'd recommend rescoping this PR to only add the required field(s) (e.g., proc.lineage).

:pray:

If we reach a consensus on this part of the proposal, I'd recommend rescoping this PR to only add the required field(s) (e.g., proc.lineage).

SGTM, only suggestion would be to perhaps follow existing more specific naming conventions, for instance:

Option A: proc.exe.lineage, proc.name.lineage or variants
Option B: proc.lineage[name], proc.lineage[exe], personally don't like that as it would be something entirely new, but if we intend to expand this new concept then I would be ok with it.
...

If we reach a consensus on this part of the proposal, I'd recommend rescoping this PR to only add the required field(s) (e.g., proc.lineage).

SGTM, only suggestion would be to perhaps follow existing more specific naming conventions, for instance:

Option A: proc.exe.lineage, proc.name.lineage or variants
Option B: proc.lineage[name], proc.lineage[exe], personally don't like that as it would be something entirely new, but if we intend to expand this new concept then I would be ok with it.
...

Option A is acceptable for me as an extension of my proposal.

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Provide feedback via https://github.com/falcosecurity/community.
/lifecycle stale

/remove-lifecycle stale

This PR is too mixed with cleanups and new features.
Will split this off into 2 PRs in the new dev cycle.

We can keep the cleanups here
and the new PR will add the new filtercheck fields (exposing the proc lineage as list in original order),
while the .join() operator will be introduced by someone else in yet another PR.

/milestone 0.19.0

Hey! I wanted to add something to this discussion.
Navigating the thread table from children to parents is not the only "direction" we are interested in. We might want to add another dimension: navigating a process sibling (or processes of the same group).
Shells use groups to put together a bunch of processes, so that they can wait for all the groups in the process or send the same signal to each of them. Also we can notice that all these processes are siblings to each other. So for instance, when we execute:
wget https://something | base64 -d | python3 ..

we might be able to navigate the wget and base64 -d when we see some python3 being spawned with it's proc.stdin.type=pipe.
Just an idea that we might want to take into account when we design this better!

/milestone 0.20.0


Performance improvement in UAParser

Describe the bug
The UAParser implementation seems to consume a lot of unnecessary memory, putting the ua yaml parse definition multiple times into memory.
What area is it related to
UAParser
To Reproduce
Steps to reproduce the behavior:

Run any instance of activelogin
Make sure you have a large amount of requests
Perform a memory dump in VS with Managed heap Summary
Observe many instances of 390.1 KB strings containing a YAML definition file for UAParser (problem)

Expected behavior
Single read of parse definition
Screenshots

NuGet package version
8.0.0
Runtime version
.NET Core 8.0
Desktop (please complete the following information):
N/A
Smartphone (please complete the following information):
N/A
Additional context
Parser uses Parser.GetDefault() for each detect which in turns reads manifest resource. Seems like this could easily be optimized by making the parser a static/singleton.
Activelogin
https://github.com/ActiveLogin/ActiveLogin.Authentication/blob/main/src/ActiveLogin.Authentication.BankId.UAParser/UAParserDeviceDetector.cs
UaParser.
https://github.com/ua-parser/uap-csharp/blob/master/UAParser/UAParser.cs

Thanks for reporting this! Will reproduce and have a look.


Ensure Terragrunt picks up changes to remote config

This PR fixes #15. Terragrunt now compares all settings of the remote configuration between what’s stored in the .tfstate file and in .terragrunt. If any of those settings don’t match up, it prompts the user whether it should re-run terraform remote config.
To make this code testable, I’ve also added a new --terragrunt-non-interactive flag that can be used to disable user prompts when running Terragrunt in an automated setting (e.g. in an automated test or script).

@josh-padnick I'm going to merge this one in to unblock myself for a few more PRs. Feedback welcome when you have time.


.RawContent includes unexpected frontmatter for empty content pages

What version of Hugo are you using (hugo version)?

$ hugo version
hugo v0.117.0-b2f0696cad918fb61420a6aff173eb36662b406e+extended darwin/arm64 BuildDate=2023-08-07T12:49:48Z VendorInfo=brew

Does this issue reproduce with the latest release?
Yes
Context
The page variable .RawContent gives access to the raw markdown content without the frontmatter. However, for empty content pages, the returned string incorrectly includes the frontmatter.
Steps to reproduce the error

Create a new site using the Hugo quick start

Modify content/my-first-post.md so it contains frontmatter only:
---
title: "RawContent test"
---

Copy single.html from themes/ananke/layouts/_default to layouts/_default

Insert the following statements to the top of layouts/_default/single.html:
{{ warnf "[%s]" .RelPermalink }}
{{ warnf "Content: '%s'" .Content }}
{{ warnf "Raw: '%s'" .RawContent }}

Start the local server with hugo server and observe the output on the terminal:

    WARN  [/posts/my-first-post/]
    WARN  Content: ''
    WARN  Raw: '---
    title: "RawContent test"
    ---'

The expected output for .RawContent would be an empty string, similar to .Content. If you add any content (including whitespace) to the post, .RawContent works as expected.

Thanks for the quick fix and tip!


a newer windows version?

hey!
i'm currently on windows, but the last version for this os is 1.2.1 :(

Oops sorry. I've forgotten to install wine on my Linux machine. I'm working on it now. Just a moment please.

@k0kubun thanks ^_^

Released Nocturn v1.2.4 with windows again. Could you try that?

using it :)


Use 'RequireAdminOnWindows' tag in Set-Date tests

PR Summary
Fix #3099
Currently Set-Date.Tests.ps1 dose not have RequireAdminOnWindows tag, so CI tests requiring elevated privilege are always skipped.
Using RequireAdminOnWindows tag instead of Test-Elevated function, I fix the problem.
PR Checklist
Note: Please mark anything not applicable to this PR NA.

[x] PR has a meaningful title

[x] Use the present tense and imperative mood when describing your changes

[x] Summarized changes
[ ] User facing Documentation needed

[x] Issue filed - Issue link: Set-Date don't work and admin-elevated test is silently skipped

[x] Change is not breaking
[x] Make sure you've added a new test if existing tests do not effectively test the code changed

[ ] Add [feature] if the change is significant or affects feature tests

[ ] This PR is ready to merge and is not Work in Progress.

If the PR is work in progress, please add the prefix WIP: to the beginning of the title and remove the prefix when the PR is ready.

@stknohg Thank you for your contribution.


image redering issue in contributing.md fixed

FIXES #166
- What I did
- How I did it
- How to verify it
- Description for the changelog

@nitesh2599 :)


Eavesarp frozen at start then stack dumps

Cloned project to kali box and installed requirements. Started eavsarp with the command;
sudo ./eavesarp.py c -i eth0
It hangs on the initialization screen
Capture interface: eth0
ARP resolution:    disabled
DNS resolution:    disabled
Requests analyzed: 0
And eventually stack traces;
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/engine/base.py", line 1244, in _execute_context
cursor, statement, parameters, context
File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/engine/default.py", line 552, in do_execute
cursor.execute(statement, parameters)
sqlite3.OperationalError: database is locked
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "./eavesarp.py", line 785, in
display_false=args.display_false)
File "./eavesarp.py", line 147, in get_output_table
transactions = get_transactions(db_session,order_by)
File "/home/me/tools/eavesarp/Eavesarp/sql.py", line 252, in get_transactions
.order_by(desc(Transaction.count))
File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/orm/query.py", line 3160, in all
return list(self)
File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/orm/query.py", line 3316, in iter
return self._execute_and_instances(context)
File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/orm/query.py", line 3341, in _execute_and_instances
result = conn.execute(querycontext.statement, self._params)
File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/engine/base.py", line 988, in execute
return meth(self, multiparams, params)
File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/sql/elements.py", line 287, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/engine/base.py", line 1107, in _execute_clauseelement
distilled_params,
File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/engine/base.py", line 1248, in _execute_context
e, statement, parameters, cursor, context
File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/engine/base.py", line 1466, in _handle_dbapi_exception
util.raise_from_cause(sqlalchemy_exception, exc_info)
File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/util/compat.py", line 383, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/util/compat.py", line 128, in reraise
raise value.with_traceback(tb)
File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/engine/base.py", line 1244, in _execute_context
cursor, statement, parameters, context
File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/engine/default.py", line 552, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) database is locked
[SQL: SELECT "transaction".id AS transaction_id, "transaction".sender_ip_id AS transaction_sender_ip_id, "transaction".target_ip_id AS transaction_target_ip_id, "transaction".count AS transaction_count
FROM "transaction" ORDER BY "transaction".count DESC]
(Background on this error at: http://sqlalche.me/e/e3q8)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "./eavesarp.py", line 902, in
if sniff_result:
NameError: name 'sniff_result' is not defined

Hmmm...interesting. When the requirements were installed, were they installed in context of root (sudo) and the following syntax was used?: python3.7 -m pip install -r requirements.txt. Also, we're sure that eth0 is the proper interface?

Running into a problem using this over an SSH Tunnel:
~/eavesarp$ sudo ./eavesarp.py --help
Unable to init server: Could not connect: Connection refused
Unable to init server: Could not connect: Connection refused

(eavesarp.py:6652): Gdk-CRITICAL **: 18:08:31.390: gdk_cursor_new_for_display: assertion 'GDK_IS_DISPLAY (display)' failed
Traceback (most recent call last):
  File "./eavesarp.py", line 6, in <module>
    from Eavesarp.color import ColorProfiles
  File "/home/air/eavesarp/Eavesarp/color.py", line 3, in <module>
    import colored
ModuleNotFoundError: No module named 'colored'


[OWLv2 / OWL-ViT] Thoughts about shift and scale

Hey there!
I was thinking about using owl-vit with vector databases, like milvus, but got stuck with the shift and scale. Since they happen after the dot product, it's hard to wrap my head around how I could use a vector database to cache all the embeddings for a dataset and then query all the objects in that datast with an object / text query. Any thoughts on whether one could get rid of the shift and scale if retraining was an option? Or any other ideas about how one could use owlvit within a vector database? Thanks!!

Closing as fixed for now, please re-open if you have questions.


Bump up the Lambda memory limit to increase performance?

Enabler
So that Lambda jobs run with an optimal price/performance ratio, we might want to increase their maximum memory allocation
Acceptance Criteria

[ ] ...
[ ] ...

Additional context
From official docs:

Lambda allocates CPU power in proportion to the amount of memory configured.

We might save time (and possibly money) by increasing the maximum RAM allocation for Lambda jobs, even though they are not reaching their current limit of 128 MB.
We could try the AWS Compute Optimizer to tweak the level automatically based on use.
Tasks

[ ] ...
[ ] ...

Definition of Ready

[ ] This story is ready to work on, according to the team's definition

Definition of Done

[ ] This story is done, according to the team's definition

Done in https://github.com/linz/geostore/commit/a5241fee707a8b3700b24b96182153b8dbdb4d8b


Pallas bug: Operand is null

Description
This script exposes a bug I've been hitting. Running this script will cause an "operand is null" error. But I think even if that was solved, the script would still crash due to the reasons reported here. To see that, just try uncommenting the line y = jnp.ones((b, d), dtype=x_ref.dtype)  and see that it also produces a segmentation fault.
from functools import partial
import jax
import jax.experimental.pallas as pl
import jax.numpy as jnp

def kernel(x_ref, O_ref, b):
    i = pl.program_id(0)
    t, d = x_ref.shape
    y = pl.load(x_ref, pl.ds(start=i*b, size=b))   # Raises Operand is null error
    # y = jnp.ones((b, d), dtype=x_ref.dtype)           # Causes segmentation fault
    z = jnp.ones((b, d), dtype=x_ref.dtype)
    acc = jnp.zeros((b, d), dtype=x_ref.dtype)
    for j in range(2):
        x = pl.load(x_ref, pl.ds(start=j*b, size=b))
        B = x @ y.T
        acc += B @ z
    pl.store(x_ref, pl.ds(start=i*b, size=b), acc)

def call_kernel():
    t, d = 126, 16
    b = 32
    grid = (t//b,)
    x = jnp.zeros((t, d), dtype=jnp.float32)
    out = pl.pallas_call(
        partial(kernel, b=b),
        grid=grid,
        out_shape = jax.ShapeDtypeStruct((t, d), jnp.float32),
    )(x)

call_kernel()

This raises
Operand is null
  %130 = call { float, float, float, float } asm sideeffect "mma.sync.aligned.m16n8k8.row.col.f32.tf32.tf32.f32 { $0, $1, $2, $3 }, { $8, $9, $10, $11 }, { $12, $13 }, { $4, $5, $6, $7 };", "=f,=f,=f,=f,0,1,2,3,r,r,r,r,r,r"(float 0.000000e+00, float 0.000000e+00, float 0.000000e+00, float 0.000000e+00, i32 %114, i32 %115, i32 %116, i32 %117, <null operand!>, <null operand!>)
Operand is null
  %135 = call { float, float, float, float } asm sideeffect "mma.sync.aligned.m16n8k8.row.col.f32.tf32.tf32.f32 { $0, $1, $2, $3 }, { $8, $9, $10, $11 }, { $12, $13 }, { $4, $5, $6, $7 };", "=f,=f,=f,=f,0,1,2,3,r,r,r,r,r,r"(float 0.000000e+00, float 0.000000e+00, float 0.000000e+00, float 0.000000e+00, i32 %114, i32 %115, i32 %116, i32 %117, <null operand!>, <null operand!>)
Operand is null
  %140 = call { float, float, float, float } asm sideeffect "mma.sync.aligned.m16n8k8.row.col.f32.tf32.tf32.f32 { $0, $1, $2, $3 }, { $8, $9, $10, $11 }, { $12, $13 }, { $4, $5, $6, $7 };", "=f,=f,=f,=f,0,1,2,3,r,r,r,r,r,r"(float 0.000000e+00, float 0.000000e+00, float 0.000000e+00, float 0.000000e+00, i32 %114, i32 %115, i32 %116, i32 %117, <null operand!>, <null operand!>)
Operand is null
  %145 = call { float, float, float, float } asm sideeffect "mma.sync.aligned.m16n8k8.row.col.f32.tf32.tf32.f32 { $0, $1, $2, $3 }, { $8, $9, $10, $11 }, { $12, $13 }, { $4, $5, $6, $7 };", "=f,=f,=f,=f,0,1,2,3,r,r,r,r,r,r"(float 0.000000e+00, float 0.000000e+00, float 0.000000e+00, float 0.000000e+00, i32 %114, i32 %115, i32 %116, i32 %117, <null operand!>, <null operand!>)
Operand is null
  %150 = call { float, float, float, float } asm sideeffect "mma.sync.aligned.m16n8k8.row.col.f32.tf32.tf32.f32 { $0, $1, $2, $3 }, { $8, $9, $10, $11 }, { $12, $13 }, { $4, $5, $6, $7 };", "=f,=f,=f,=f,0,1,2,3,r,r,r,r,r,r"(float %131, float %132, float %133, float %134, i32 %126, i32 %127, i32 %128, i32 %129, <null operand!>, <null operand!>)
Operand is null
  %155 = call { float, float, float, float } asm sideeffect "mma.sync.aligned.m16n8k8.row.col.f32.tf32.tf32.f32 { $0, $1, $2, $3 }, { $8, $9, $10, $11 }, { $12, $13 }, { $4, $5, $6, $7 };", "=f,=f,=f,=f,0,1,2,3,r,r,r,r,r,r"(float %136, float %137, float %138, float %139, i32 %126, i32 %127, i32 %128, i32 %129, <null operand!>, <null operand!>)
Operand is null
  %160 = call { float, float, float, float } asm sideeffect "mma.sync.aligned.m16n8k8.row.col.f32.tf32.tf32.f32 { $0, $1, $2, $3 }, { $8, $9, $10, $11 }, { $12, $13 }, { $4, $5, $6, $7 };", "=f,=f,=f,=f,0,1,2,3,r,r,r,r,r,r"(float %141, float %142, float %143, float %144, i32 %126, i32 %127, i32 %128, i32 %129, <null operand!>, <null operand!>)
Operand is null
  %165 = call { float, float, float, float } asm sideeffect "mma.sync.aligned.m16n8k8.row.col.f32.tf32.tf32.f32 { $0, $1, $2, $3 }, { $8, $9, $10, $11 }, { $12, $13 }, { $4, $5, $6, $7 };", "=f,=f,=f,=f,0,1,2,3,r,r,r,r,r,r"(float %146, float %147, float %148, float %149, i32 %126, i32 %127, i32 %128, i32 %129, <null operand!>, <null operand!>)
Traceback (most recent call last):
  File "/home/cgel/manifest2/core/kernels/bug.py", line 30, in <module>
    call_kernel()
  File "/home/cgel/manifest2/core/kernels/bug.py", line 24, in call_kernel
    out = pl.pallas_call(
          ^^^^^^^^^^^^^^^
jaxlib.xla_extension.XlaRuntimeError: INTERNAL: Failed to emit LLVM IR.
--------------------
For simplicity, JAX has removed its internal frames from the traceback of the following exception. Set JAX_TRACEBACK_FILTERING=off to include these.

System info (python version, jaxlib version, accelerator, etc.)
jax:    0.4.25
jaxlib: 0.4.25
numpy:  1.26.4
python: 3.11.4 (main, Dec  7 2023, 15:43:41) [GCC 12.3.0]
jax.devices (2 total, 2 local): [cuda(id=0) cuda(id=1)]
process_count: 1
platform: uname_result(system='Linux', node='jacob-manifestai', release='6.2.0-39-generic', version='#40-Ubuntu SMP PREEMPT_DYNAMIC Tue Nov 14 14:18:00 UTC 2023', machine='x86_64')

$ nvidia-smi
Sat Mar  9 09:44:04 2024
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.161.07             Driver Version: 535.161.07   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA RTX A6000               On  | 00000000:2D:00.0 Off |                  Off |
| 30%   26C    P2              25W / 300W |    269MiB / 49140MiB |      1%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   1  NVIDIA RTX A6000               On  | 00000000:41:00.0 Off |                  Off |
| 30%   31C    P2              29W / 300W |    269MiB / 49140MiB |      1%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A   1820886      C   ...2-pallas-YfL-FD7B-py3.11/bin/python      262MiB |
|    1   N/A  N/A   1820886      C   ...2-pallas-YfL-FD7B-py3.11/bin/python      262MiB |
+---------------------------------------------------------------------------------------+

I finally got to this issue, and unfortunately I cannot reproduce it on A100 with the latest nightly.
Please feel free to reopen if you encounter it again.


[Feature] Allow to choose profiling event type

Hi,
I would propose that we add an option to specify the profiling mode. The github page states some of these and especially the allocation mode would be interesting for us:

-e event - the profiling event: cpu, alloc, lock, cache-misses etc. Use list to see the complete list of available events.

As far as I can see, the profiling mode is currently hardcoded: https://github.com/VerizonMedia/kubectl-flame/blob/5cb73b39e20a9998308c6b1960add543da23558f/agent/profiler/jvm.go#L47
If you agree that this could be useful, I'd come up with a PR.
Cheers, u6f6o

Hi @u6f6o
Sounds like a great idea to me.
Looking forward to this PR.
Thanks


Constructor beside class name (Similar to Java Record, Similar to Kotlin class)

I not sure if anyone has suggested this before (at least I could not find in the issues section)
To make the class definition even more concise, can consider this
Current way
 class Horse{
      constructor(public name:string, private color:number){}
}

Suggested Enhance Way
class Horse (public name:string, private color:number) {}
In fact, for extra bonus, if the class body is empty, braces are optional
class Horse (public name:string, private color:number)
Of course, if needed additional constructors, can define in the class body, as per current way of doing
The suggestion to put constructor beside the class name is aspired by Java Record and Kotlin class definition.

You forgot to fill out the issue template. It contains a viability checklist, which your proposal violates. Your suggestion is not in line with the TypeScript Language Design goals.


Initial functionality

Implement initial functionality with checking multiple URLs in parallel and reporting results to Slack channel.

completed with #2


Readme update python

Have you read the Contributing Guidelines on Pull Requests?
yes
Description
updated readme files of python
Checklist

[x] I've read the contribution guidelines.
[x] I've checked the issue list before deciding what to submit.
[x] I've edited the README.md and link to my code.

Related Issues or Pull Requests
#1131

@HarshCasper kindly review and merge it


fix: resource naming condition regex

Description
According to Azure portal, the restriction is: "The resource name must contain between 1 to 255 characters inclusive. The name only allows alphanumeric characters, periods, underscores, hyphens and parenthesis and cannot end in a period."
This PR fixes this with updated regex.
Fixes #40
Closes #40
Type of Change

[ ] Non-module change (e.g. CI/CD, documentation, etc.)
[ X ] Azure Verified Module updates:

[ X ] Bugfix containing backwards compatible bug fixes, and I have NOT bumped the MAJOR or MINOR version in locals.version.tf.json:

[ ] Someone has opened a bug report issue, and I have included "Closes #{bug_report_issue_number}" in the PR description.
[ ] The bug was found by the module author, and no one has opened an issue to report it yet.

[ ] Feature update backwards compatible feature updates, and I have bumped the MINOR version in locals.version.tf.json.
[ ] Breaking changes and I have bumped the MAJOR version in locals.version.tf.json.
[ ] Update to documentation

Checklist

[ X ] I'm sure there are no other open Pull Requests for the same update/change
[ X ] My corresponding pipelines / checks run clean and green without any errors or warnings
[ X ] I did run all  pre-commit checks

@captainhook thanks for catching this!
Can you resync from main?
There were recently changes on switching how telemetry is being captured and a new linting rule enforcing the new provider. I just pushed a PR to implement the new telemetry module. This hasn't been fully documented yet in the AVM documentation; however, AVM terraform modules will no longer require the version or locals.telemetry files.
Version will be pulled from the registry, and I'll update the next minor version to include both your change and the telemetry updates.

Closing this as it was updated in: https://github.com/Azure/terraform-azurerm-avm-res-insights-component/pull/48


[WIP] Pushover

Without testing this I can't really know if it's working.

I'll test it for you :)

Thanks :)

Fixed somethings now it's done and working thanks :)


Update TimeSeriesAnomaly-ProcessExecutions.yaml

Changing rundll.exe to rundll32.exe, which is the actual process name of the dll loader.
Required items, please complete
Change(s):

Changing "rundll.exe" to "rundll32.exe"

Reason for Change(s):

rundll32.exe  is the actual name of the process used to load dll.

Version Updated:

Yes

Testing Completed:

Yes

Checked that the validations are passing and have addressed any issues that are present:

Yes

Hello @princemathew Please repackage the solution and update the version of the solution

Hello @princemathew Please repackage the solution and update the version of the solution


Add directory for mintlify

Current dependencies on/for this PR:

main

PR #14   👈

This stack of pull requests is managed by Graphite.


Webserver not working offline

Hi Luxionis Team,
I can not connect to the camera  when I'm offline.
The main.py shows:
======== Running on http://<IP_ADDRESS>:9999 ========
(Press CTRL+C to quit)

while the web-browser console shows:
client.mjs:31 [PC] ICE Gathering state:  new
client.mjs:35 [PC] ICE Connection state:  new
client.mjs:39 [PC] Signaling state:  stable
client.mjs:37 [PC] Signaling state:  have-local-offer

If I switch on the WiFi or Ethernet everything works correctly.
Do you have any idea why after have-local-offer I can't make the connection?
Thanks in advance!
Best regards
Juan

Hi @JuanFuriaz,
WebRTC used in this experiment requires an internet connection as it seeks for ICE and STUN/TURN servers for communication (internal WebRTC mechanism)
Since the app is offline, it stops at "have-local-offer" as it doesn't receive any ICE candidates and cannot send this offer to the server (and therefore establish the direct connection)
I would recommend either using the MJPEG experiment or the RTSP experiment

@VanDavv can we host these servers locally as well?

Hi @VanDavv and @themarpe,
I already made a lot of adaptation for my prototype so that I wouldn't like to start from scratch with other experiments. I'll first try to run locally  the STUN/TURN servers with https://openvidu.io/ if it doesn't work I'll change to your recommendations. Do you have any experience trying to run the servers locally?

Sorry Juan - haven't dealt with that from my side. Maybe check what openvidu uses, if they support "self-hosted" setups as well.

@themarpe @JuanFuriaz I have not set up a local server as well, so sadly cannot give any recommendations.
But I'm more than happy to extend this example and add support for offline mode if you manage to get it working this way @JuanFuriaz


Update README.md

I don't see a strong reason to change the wording, and there are typos in the changes.
Closed and thanks you for contributing.


[FEATURE] Follow official Go guidance on how to organize a module?

Requirements
Go released an official guidance on how to organize a module here: https://go.dev/doc/modules/layout. Do you think it is a good idea to follow this guidance for this repo? What it means here is to just move everything in pkg/openfeature to the top-level. IMHO it would make importing this module a bit nicer:
import "github.com/open-feature/go-sdk/pkg/openfeature"

vs
import "github.com/open-feature/go-sdk"

The mismatch of the package name and the import is a bit jarring though. I suppose one could also "import as".
Unfortunately, this would be a breaking change.

@Kavindu-Dodan @thisthat @bacherfl @RealAnna do you folks have any opinions on this?

github.com/open-feature/go-sdk/openfeature would be a nice import path, because it helps goimports to automatically import the package when it's used in a file.
You can probably do this without a breaking change using type aliases to make the types of the pkg/openfeature package aliases of the types in the openfeature package, and having functions in the pkg/openfeature package call their identically-named functions in the openfeature directory. Constants can likewise be aliases into the the openfeature package.
The memprovider package could take a similar approach.
I think that would preserve all identifiers and behaviors, while only maintaining a single implementation of the behaviors. Future identifiers wouldn't need to be added to pkg/openfeature. And because the types are considered the same types, there shouldn't be any compilation impact.

I think this is worth exploring, especially if we could make this a non-breaking change. @paddycarver, would you be willing to put together a proof of concept using type aliases?

I've opened #232 and #233 with proofs of concept of the github.com/open-feature/go-sdk/openfeature and github.com/open-feature/go-sdk import paths, respectively. I don't have time to do the harder work (validation), but the branches build and the tests pass. I don't have a consumer of the package at-hand to verify the compatibility, unfortunately.

Thank you @paddycarver for the POC PRs. I checked them and I'm in favor of the new packaging proposed through https://github.com/open-feature/go-sdk/pull/232  (root/pkg/openfeature -> root/openfeature). I think this packaging is ideal, solves the packaging issue as well as isolates logic instead of having it at root level (i.e- compared to @233)
I tried adapting go sdk providers and all looks fine. However, this make lint failures which means we have to update all references to the latest packaging.
@toddbaert  shall we proceed with this change?

I think this is a great idea. Thanks @paddycarver, for putting together the PoC!
@craigpastro are you happy with this solution?

Thank you @paddycarver for the POC PRs. I checked them and I'm in favor of the new packaging proposed through #232 (root/pkg/openfeature -> root/openfeature). I think this packaging is ideal, solves the packaging issue as well as isolates logic instead of having it at root level (i.e- compared to #233)
I tried adapting go sdk providers and all looks fine. However, this make lint failures which means we have to update all references to the latest packaging.
@toddbaert shall we proceed with this change?

I don't have a problem with correcting the lint errors. I think this is a very worthwhile change.
I'm very happy with this solution.

I think this is a great solution that avoids breakage. Thanks @paddycarver 🎉

Created https://github.com/open-feature/go-sdk/issues/234 to track follow-up tasks once this issue is resolved with the proposed fix at #232