high_speed_endoscopy build error

I have an Orin AGX 64GB Dev Kit and an Emergent camera with a Mellanox PCIe and rivermax. The camera works with the Emergent software. I am hoping to get the high_speed_endoscopy demo working, but am running into an error with the build.
It apparently is looking for the dlpack package. Should I update that in the cmake?
dev/infiniband/uverbs0 --group-add=29 --volume=/usr/lib/aarch64-linux-gnu/tegra:/usr/lib/aarch64-linux-gnu/tegra --group-add=44 --group-add=104 --group-add=993 --rm --device=/dev/nvgpu/igpu0/nvsched --device=/dev/nvhost-ctxsw-gpu --device=/dev/nvhost-nvsched-gpu --device=/dev/nvhost-sched-gpu --device=/dev/nvidia0 --device=/dev/nvidia-modeset -e CUPY_CACHE_DIR=/workspace/holohub/.cupy/kernel_cache -e XDG_SESSION_TYPE -e XDG_RUNTIME_DIR -v /run/user/1000:/run/user/1000 -v /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY -e PYTHONPATH=/opt/nvidia/holoscan/python/lib:/workspace/holohub/benchmarks/holoscan_flow_benchmarking --ipc=host --cap-add=CAP_SYS_PTRACE --ulimit=memlock=-1 --ulimit=stack=67108864 --entrypoint=bash holohub:ngc-v2.6.0-igpu -c ./run build high_speed_endoscopy
Building Holohub
Building high_speed_endoscopy application
[command] cmake -S . -B build/high_speed_endoscopy --no-warn-unused-cli -DPython3_EXECUTABLE=/usr/bin/python3 -DPython3_ROOT_DIR=/usr/lib/python3 -DHOLOHUB_DATA_DIR:PATH=/workspace/holohub/data -DCMAKE_BUILD_TYPE=release -DAPP_high_speed_endoscopy=1
Not searching for unused variables given on the command line.
-- The CXX compiler identification is GNU 11.4.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found CUDAToolkit: /usr/local/cuda/include (found version "12.6.20")
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- Found holoscan: /opt/nvidia/holoscan/lib/cmake/holoscan/holoscan-config.cmake (found suitable version "2.6.0", minimum required is "0.5")
-- The C compiler identification is GNU 11.4.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Found Python3: /usr/bin/python3 (found version "3.10.12") found components: Interpreter Development Development.Module Development.Embed
-- pybind11 v2.10.1
-- Performing Test HAS_FLTO
-- Performing Test HAS_FLTO - Success
-- Configuring done
-- Generating done
-- Build files have been written to: /workspace/holohub/build/high_speed_endoscopy
[command] cmake --build build/high_speed_endoscopy -j
[ 10%] Building CXX object gxf_extensions/emergent_source/CMakeFiles/gxf_emergent_source_lib.dir/emergent_source.cpp.o
[ 20%] Building CXX object operators/emergent_source/CMakeFiles/emergent_source.dir/emergent_source.cpp.o
[ 20%] Built target high_speed_endoscopy_yaml
In file included from /opt/nvidia/holoscan/include/gxf/multimedia/video.hpp:30,
from /workspace/holohub/gxf_extensions/emergent_source/emergent_source.cpp:22:
/opt/nvidia/holoscan/include/gxf/std/tensor.hpp:30:10: fatal error: dlpack/dlpack.h: No such file or directory
30 | #include "dlpack/dlpack.h"
|          ^~~~~~~~~~~~~~~~~
compilation terminated.
gmake[2]: *** [gxf_extensions/emergent_source/CMakeFiles/gxf_emergent_source_lib.dir/build.make:76: gxf_extensions/emergent_source/CMakeFiles/gxf_emergent_source_lib.dir/emergent_source.cpp.o] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:1328: gxf_extensions/emergent_source/CMakeFiles/gxf_emergent_source_lib.dir/all] Error 2
gmake[1]: *** Waiting for unfinished jobs....
[ 30%] Linking CXX shared library libemergent_source.so
[ 30%] Built target emergent_source
gmake: *** [Makefile:146: all] Error 2
ERROR: Error building application. (Exit code: 2)
Compute:
Architecture:            aarch64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                  12
On-line CPU(s) list:   0-11
Vendor ID:               ARM
Model name:            Cortex-A78AE
Model:               1
Thread(s) per core:  1
Core(s) per cluster: 4
Socket(s):           -
Cluster(s):          3
Stepping:            r0p1
CPU max MHz:         2201.6001
CPU min MHz:         115.2000
BogoMIPS:            62.50
Flags:               fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp uscat ilrcpc flagm paca pacg
Caches (sum of all):
L1d:                   768 KiB (12 instances)
L1i:                   768 KiB (12 instances)
L2:                    3 MiB (12 instances)
L3:                    6 MiB (3 instances)
NUMA:
NUMA node(s):          1
NUMA node0 CPU(s):     0-11
Vulnerabilities:
Gather data sampling:  Not affected
Itlb multihit:         Not affected
L1tf:                  Not affected
Mds:                   Not affected
Meltdown:              Not affected
Mmio stale data:       Not affected
Retbleed:              Not affected
Spec rstack overflow:  Not affected
Spec store bypass:     Mitigation; Speculative Store Bypass disabled via prctl
Spectre v1:            Mitigation; __user pointer sanitization
Spectre v2:            Mitigation; CSV2, but not BHB
Srbds:                 Not affected
Tsx async abort:       Not affected
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 540.4.0                Driver Version: 540.4.0      CUDA Version: 12.6     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Orin (nvgpu)                  N/A  | N/A              N/A |                  N/A |
| N/A   N/A  N/A               N/A /  N/A | Not Supported        |     N/A          N/A |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+

Hi there @tonyreina , thanks for the report. We've addressed the build issue in fc9db3e so that high_speed_endoscopy components link in Holoscan SDK's DLPack library distribution. Please try again with top-of-tree main and reach out if you continue to encounter issues.

Thanks @tbirdso That clears the error, but the Emergent SDK (eSDK) is not included in the Docker image. I've got it in /opt/EVT/eSDK on the system; should I try to mount that directory into the Docker when I run it?
Ideally, I'd like to run the app as a standalone outside of the Docker. I tried cmake but get an error linking the emergent_source library (which I think is the custom GXF component for the Emergent camera that wraps the eSDK C++ calls).
Any chance you could help me? We're in the NVidia Inception program so would love to get this as an example. I've built C++ and Python code that works with the Emergent, but just need some help in wrapping my head around the Holoscan components.

Hi @tonyreina
Here are the instructions to run it standalone.
After you have installed the EVT drivers as mentioned here and tested that the eCapture app can open and run the camera, do following in the Holohub repo:
sudo ./run setup
./run build <APP_NAME>
sudo ./run launch <APP_NAME>

We have the instructions for Native build here.
For EVT user manual, download it from here.

@anishag14 Thanks so much for the reply!
I'm able to build the high_speed_endoscopy but when I try to run it, then it doesn't seem to have the EmergentSource component registered.  Is there an extra LD path or something for me to include?
exthymic@orin-agx-001:~/holohub$ ./run build high_speed_endoscopy
Building Holohub
Building high_speed_endoscopy application
[command] cmake -S . -B build/high_speed_endoscopy --no-warn-unused-cli -DPython3_EXECUTABLE=/usr/bin/python3 -DPython3_ROOT_DIR=/usr/lib/python3 -DHOLOHUB_DATA_DIR:PATH=/home/exthymic/holohub/data -DCMAKE_BUILD_TYPE=release -DAPP_high_speed_endoscopy=1
Not searching for unused variables given on the command line.
-- The CXX compiler identification is GNU 13.1.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found CUDAToolkit: /usr/local/cuda/targets/aarch64-linux/include (found version "12.6.68")
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- Found holoscan: /opt/nvidia/holoscan/lib/cmake/holoscan/holoscan-config.cmake (found suitable version "2.6.0", minimum required is "0.5")
-- The C compiler identification is GNU 13.1.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Found Python3: /usr/bin/python3 (found version "3.10.12") found components: Interpreter Development Development.Module Development.Embed
CMake Deprecation Warning at build/high_speed_endoscopy/_deps/pybind11-src/CMakeLists.txt:8 (cmake_minimum_required):
  Compatibility with CMake < 3.10 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value.  Or, use the <min>...<max> syntax
  to tell CMake that the project requires at least <min> but has been updated
  to work with policies introduced by <max> or earlier.

-- pybind11 v2.10.1
-- Performing Test HAS_FLTO
-- Performing Test HAS_FLTO - Success
-- Configuring done (3.7s)
-- Generating done (0.1s)
-- Build files have been written to: /home/exthymic/holohub/build/high_speed_endoscopy
[command] cmake --build build/high_speed_endoscopy -j
[ 20%] Building CXX object operators/emergent_source/CMakeFiles/emergent_source.dir/emergent_source.cpp.o
[ 20%] Building CXX object gxf_extensions/emergent_source/CMakeFiles/gxf_emergent_source_lib.dir/emergent_source.cpp.o
[ 20%] Built target high_speed_endoscopy_yaml
[ 30%] Linking CXX shared library ../../lib/gxf_extensions/libgxf_emergent_source_lib.so
[ 30%] Built target gxf_emergent_source_lib
[ 40%] Building CXX object gxf_extensions/emergent_source/CMakeFiles/gxf_emergent_source.dir/emergent_ext.cpp.o
[ 50%] Linking CXX shared library ../../lib/gxf_extensions/libgxf_emergent_source.so
[ 50%] Built target gxf_emergent_source
[ 60%] Linking CXX shared library libemergent_source.so
[ 60%] Built target emergent_source
[ 70%] Building CXX object applications/high_speed_endoscopy/cpp/CMakeFiles/high_speed_endoscopy.dir/main.cpp.o
[ 80%] Building CXX object operators/emergent_source/python/CMakeFiles/emergent_source_python.dir/emergent_source.cpp.o
[ 90%] Linking CXX shared module ../../../python/lib/holohub/emergent_source/_emergent_source.cpython-310-aarch64-linux-gnu.so
lto-wrapper: warning: using serial compilation of 3 LTRANS jobs
lto-wrapper: note: see the ‘-flto’ option documentation for more information
[ 90%] Built target emergent_source_python
[100%] Linking CXX executable high_speed_endoscopy
[100%] Built target high_speed_endoscopy
Holohub build done.
exthymic@orin-agx-001:~/holohub$ sudo ./run launch high_speed_endoscopy
Default language for high_speed_endoscopy selected: cpp
Run environment: export PYTHONPATH=${PYTHONPATH}:/opt/nvidia/holoscan/lib/cmake/holoscan/../../../python/lib:/home/exthymic/holohub/build/high_speed_endoscopy/python/lib:/home/exthymic/holohub && export HOLOHUB_DATA_PATH=/home/exthymic/holohub/data && export HOLOSCAN_INPUT_PATH=/home/exthymic/holohub/data
Run workdir: cd /home/exthymic/holohub/build/high_speed_endoscopy
Run command: /home/exthymic/holohub/build/high_speed_endoscopy/applications/high_speed_endoscopy/cpp/high_speed_endoscopy
Run command args:
[command] export PYTHONPATH=${PYTHONPATH}:/opt/nvidia/holoscan/lib/cmake/holoscan/../../../python/lib:/home/exthymic/holohub/build/high_speed_endoscopy/python/lib:/home/exthymic/holohub && export HOLOHUB_DATA_PATH=/home/exthymic/holohub/data && export HOLOSCAN_INPUT_PATH=/home/exthymic/holohub/data
[command] cd /home/exthymic/holohub/build/high_speed_endoscopy
[command] /home/exthymic/holohub/build/high_speed_endoscopy/applications/high_speed_endoscopy/cpp/high_speed_endoscopy
[info] [fragment.cpp:585] Loading extensions from configs...
[info] [gxf_executor.cpp:255] Creating context
[warning] [type_registry.cpp:57] Unknown type: nvidia::holoscan::EmergentSource
[error] [graph_entity.hpp:646] Typename [nvidia::holoscan::EmergentSource] not found. Is this type registered?
[error] [graph_entity.cpp:159] Graph Entity failed to create codelet of type [nvidia::holoscan::EmergentSource]
[error] [gxf_executor.cpp:1438] Exception occurred during initialization of operator: 'emergent' - Failed to add codelet of type nvidia::holoscan::EmergentSource
terminate called after throwing an instance of 'std::runtime_error'
  what():  Failed to add codelet of type nvidia::holoscan::EmergentSource
./run: line 83: 300140 Aborted                 (core dumped) /home/exthymic/holohub/build/high_speed_endoscopy/applications/high_speed_endoscopy/cpp/high_speed_endoscopy
[command] export PYTHONPATH= && export HOLOHUB_DATA_PATH="" && export HOLOSCAN_INPUT_PATH="/home/exthymic/holohub/data"
exthymic@orin-agx-001:~/holohub$

Hey @tonyreina
It seems that you are either missed the step to install the Holoscan SDK or the path to Holoscan dir mentioned here.
Installing the Holoscan SDK is mentioned in the first line in the native build here.
If you have already installed the Holoscan SDK and this issue still persists, then I would recommend to check what <HOLOSCAN_INSTALL_DIR> points to.

@anishag14 Thanks.
Here are all of the environment variables that I have defined:
PWD=/home/exthymic/holohub
HOLOHUB_BUILD_DIR=/home/exthymic/holohub/build/
HOLOHUB_SOURCE_DIR=/home/exthymic/holohub/
PYTHONPATH=/opt/nvidia/holoscan/python/lib
HOLOSCAN_INSTALL_DIR=/opt/nvidia/holoscan
LD_LIBRARY_PATH=/opt/nvidia/holoscan/lib:/usr/local/cuda/lib64:/opt/EVT/eCapturePro/emergent3D/lib:/opt/EVT/eCapturePro/bin::/opt/nvidia/holoscan/lib
HOLOSCAN_INPUT_PATH=/opt/nvidia/holoscan/data

I re-installed sudo apt holoscan.
This is still giving me the same error.
Do the environment variables look correct?
Thanks again!
 Done
Building dependency tree... Done
Reading state information... Done
All packages are up to date.
W: https://nvidia.github.io/libnvidia-container/stable/ubuntu18.04/arm64/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
W: https://nvidia.github.io/nvidia-container-runtime/stable/ubuntu18.04/arm64/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
W: https://repos.influxdata.com/ubuntu/dists/jammy/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
holoscan is already the newest version (<IP_ADDRESS>-1).
The following packages were automatically installed and are no longer required:
  dh-elpa-helper libavfilter7 libavutil56 libpostproc55 librhash0 libswresample3 libswscale5 libva-x11-2 libvdpau1
Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
exthymic@orin-agx-001:~/holohub$   # if sudo is available
  sudo ./run setup
Hit:1 https://nvidia.github.io/libnvidia-container/stable/ubuntu18.04/arm64  InRelease
Hit:2 https://nvidia.github.io/nvidia-container-runtime/stable/ubuntu18.04/arm64  InRelease
Hit:3 https://nvidia.github.io/libnvidia-container/stable/deb/arm64  InRelease
Hit:4 https://download.docker.com/linux/ubuntu jammy InRelease
Hit:5 https://repo.download.nvidia.com/jetson/common r36.4 InRelease
Hit:6 https://repos.influxdata.com/ubuntu jammy InRelease
Hit:7 http://ports.ubuntu.com/ubuntu-ports jammy InRelease
Hit:8 https://repo.download.nvidia.com/jetson/t234 r36.4 InRelease
Hit:9 https://download.sublimetext.com apt/stable/ InRelease
Hit:10 https://apt.kitware.com/ubuntu jammy InRelease
Hit:11 http://ports.ubuntu.com/ubuntu-ports jammy-updates InRelease
Hit:12 https://packages.microsoft.com/repos/code stable InRelease
Hit:13 https://repo.download.nvidia.com/jetson/ffmpeg r36.4 InRelease
Hit:14 http://ports.ubuntu.com/ubuntu-ports jammy-backports InRelease
Hit:15 https://ppa.launchpadcontent.net/ubuntu-toolchain-r/test/ubuntu jammy InRelease
Hit:16 http://ports.ubuntu.com/ubuntu-ports jammy-security InRelease
Reading package lists... Done
W: https://nvidia.github.io/libnvidia-container/stable/ubuntu18.04/arm64/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
W: https://nvidia.github.io/nvidia-container-runtime/stable/ubuntu18.04/arm64/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
W: https://repos.influxdata.com/ubuntu/dists/jammy/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
Cuda version found: 12.6
Package libcudnn9-cuda-12 found with version libcudnn9-cuda-12/stable,now <IP_ADDRESS>-1 arm64 [installed,automatic]
Package libcudnn9-dev-cuda-12 found with version libcudnn9-dev-cuda-12/stable,now <IP_ADDRESS>-1 arm64 [installed,automatic]
Package libnvinfer-headers-dev found with version libnvinfer-headers-dev/stable,now <IP_ADDRESS>-1+cuda12.5 arm64 [installed]
Package libnvinfer-dev found with version libnvinfer-dev/stable,now <IP_ADDRESS>-1+cuda12.5 arm64 [installed]
Package libnvinfer-plugin-dev found with version libnvinfer-plugin-dev/stable,now <IP_ADDRESS>-1+cuda12.5 arm64 [installed]
Package libnvonnxparsers-dev found with version libnvonnxparsers-dev/stable,now <IP_ADDRESS>-1+cuda12.5 arm64 [installed]
Installing autocomplete
Setup for HoloHub is ready. Happy Holocoding!

exthymic@orin-agx-001:~/holohub$ export HOLOSCAN_INSTALL_DIR=/opt/nvidia/holoscan
exthymic@orin-agx-001:~/holohub$ ./run build high_speed_endoscopy
Building Holohub
Building high_speed_endoscopy application
[command] cmake -S . -B build/high_speed_endoscopy --no-warn-unused-cli -DPython3_EXECUTABLE=/usr/bin/python3 -DPython3_ROOT_DIR=/usr/lib/python3 -DHOLOHUB_DATA_DIR:PATH=/home/exthymic/holohub/data -DCMAKE_BUILD_TYPE=release -DAPP_high_speed_endoscopy=1
Not searching for unused variables given on the command line.
CMake Deprecation Warning at build/high_speed_endoscopy/_deps/pybind11-src/CMakeLists.txt:8 (cmake_minimum_required):
  Compatibility with CMake < 3.10 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value.  Or, use the <min>...<max> syntax
  to tell CMake that the project requires at least <min> but has been updated
  to work with policies introduced by <max> or earlier.

-- pybind11 v2.10.1
-- Configuring done (0.7s)
-- Generating done (0.1s)
-- Build files have been written to: /home/exthymic/holohub/build/high_speed_endoscopy
[command] cmake --build build/high_speed_endoscopy -j
[  0%] Built target high_speed_endoscopy_yaml
[ 20%] Built target gxf_emergent_source_lib
[ 40%] Built target emergent_source
[ 60%] Built target gxf_emergent_source
[ 80%] Built target emergent_source_python
[100%] Built target high_speed_endoscopy
Holohub build done.
exthymic@orin-agx-001:~/holohub$ sudo ./run launch high_speed_endoscopy
Default language for high_speed_endoscopy selected: cpp
Run environment: export PYTHONPATH=${PYTHONPATH}:/opt/nvidia/holoscan/lib/cmake/holoscan/../../../python/lib:/home/exthymic/holohub/build/high_speed_endoscopy/python/lib:/home/exthymic/holohub && export HOLOHUB_DATA_PATH=/home/exthymic/holohub/data && export HOLOSCAN_INPUT_PATH=/home/exthymic/holohub/data
Run workdir: cd /home/exthymic/holohub/build/high_speed_endoscopy
Run command: /home/exthymic/holohub/build/high_speed_endoscopy/applications/high_speed_endoscopy/cpp/high_speed_endoscopy
Run command args:
[command] export PYTHONPATH=${PYTHONPATH}:/opt/nvidia/holoscan/lib/cmake/holoscan/../../../python/lib:/home/exthymic/holohub/build/high_speed_endoscopy/python/lib:/home/exthymic/holohub && export HOLOHUB_DATA_PATH=/home/exthymic/holohub/data && export HOLOSCAN_INPUT_PATH=/home/exthymic/holohub/data
[command] cd /home/exthymic/holohub/build/high_speed_endoscopy
[command] /home/exthymic/holohub/build/high_speed_endoscopy/applications/high_speed_endoscopy/cpp/high_speed_endoscopy
[info] [fragment.cpp:585] Loading extensions from configs...
[info] [gxf_executor.cpp:255] Creating context
[warning] [type_registry.cpp:57] Unknown type: nvidia::holoscan::EmergentSource
[error] [graph_entity.hpp:646] Typename [nvidia::holoscan::EmergentSource] not found. Is this type registered?
[error] [graph_entity.cpp:159] Graph Entity failed to create codelet of type [nvidia::holoscan::EmergentSource]
[error] [gxf_executor.cpp:1438] Exception occurred during initialization of operator: 'emergent' - Failed to add codelet of type nvidia::holoscan::EmergentSource
terminate called after throwing an instance of 'std::runtime_error'
  what():  Failed to add codelet of type nvidia::holoscan::EmergentSource
./run: line 83: 312572 Aborted                 (core dumped) /home/exthymic/holohub/build/high_speed_endoscopy/applications/high_speed_endoscopy/cpp/high_speed_endoscopy
[command] export PYTHONPATH= && export HOLOHUB_DATA_PATH="" && export HOLOSCAN_INPUT_PATH="/home/exthymic/holohub/data"


Basic tests for authentication with Wiremock in compatTests

In #251 we hit a regression related to authentication which was detected by our e2e test (nice!). However, some of the problems with authentication (e.g. that we "forget" to pass credentials at all) could be probably spotted also in CompatTests with Wiremock. Maybe it would be worth to add some test(s?) with authentication enabled?

Could the regression be this one from another PR?
https://github.com/gradle-nexus/publish-plugin/pull/245/commits/e37d8476fc76f1009fd40d608d96a649fe92cfc5


update on unhealthly cluster

When I performed key value update on the  unhealthy cluster (2 of 3 nodes down) I've got.
{"message": "Internal Server Error"}

Which is expected,
but then after adding back one node to the cluster - previously failed updates are hiddenly applied (values changed index moved)
Is this expected behaviour?

Internal Server Error does not imply the proposal you sent must be failed. etcd is trying its best to get it committed.

Is there any flag (e.g. consistent=true) to control this behaviour?
e.g. consistent=true + fail -> means failed (no quorum) and forgotten (no auto retry)
This will simplify error handling scenarios.

This error is certainly misleading, I was under the impression it meant the value was not updated. Is it possible to change the error to reflect that it was committed but not updated until cluster health is restored?

@grzechum No. etcd cannot give you any guarantee at the moment.
If it is a follower (or even a leader), it might send out the proposal. It cannot know if the proposal will succeed or not until it gets the ack from leader (or get the ack from other followers). Any error happened during the process is Server Internal Error right now. We can improve the error message.
To support a stronger guarantee require significant amount of work. It is not a very simple problem.

@durzo It might be committed or might not be. We can improve the error message.


Only bind route callable state if DIC is specified

Otherwise let $this inside of closure fail.

I must have missed this...
Can I get some clarification? :)

The container is becoming 100% optional in v4. Currently in v3, there is always a container, and we bind all route callable closures' state to the container (i.e. $this inside of a route callable closure is the container). In v4, if no container is provided, we cannot bind the closure state.

Would it make more sense to completely remove that? Callables would be bound to a container whose API is undefined.

I'm assuming that we'll continue to require that the container is must conform to the container-interop interface.

@akrabat so $this->service should be removed in favor of $this->get('...') only?

@akrabat so $this->service should be removed in favor of $this->get('...') only?

Now that's a good question that we'll have to think about.

We're doing away with Pimple. And if we're going to support third-party containers, they should adhere to container-interop. And if that means only supporting $this->get('...') inside of route callable closures, that's the way I would go.

Related to #2100

Done. See https://github.com/slimphp/Slim/blob/4.x/Slim/App.php#L539-L541


this.fields does not seem to be immediately available

Versions:

VueJs: 2.3.1
Vee-Validate:2.0.0

Description:
In my computed properties, I seem to have to do this to prevent errors:
const field = this.fields[this.name]
const required = field && field.required

Steps to reproduce
If don't do the existence check, I get a Type Error:
this.fields[this.name].required

Error in render function: "TypeError: Cannot read property 'required' of undefined"

Am I missing something?

Yep, fields are not available at the rendering phase, because the directive is not bound then. That is why you have to always add an existence check. There is a helper called mapFields which helps reduce the verbosity of those checks.
http://vee-validate.logaretm.com/examples#flags-example
I'm thinking of using proxies to provide fake object until the actual flags are added which should remove the need for those excessive checks.

Ah, interesting. I didn't know about that limitation with directives.
What about adding a helper function to the Observable?
this.fields.has(prop)

It won't reduce the amount of checks tho, you would still need to do:
this.fields.has(prop) && this.fields[prop].valid

// instead of
this.fields[prop] && this.fields[prop].valid;

It won't reduce the amount of checks tho, you would still need to do:

I was suggesting more of a getter:
const required = fields.get('required') // null or true

well adding methods certainly would work, it used to be like this before the RCs I will see what I can do about this.


There is no protoc 3.0.0+ yet make installs claims to need it

I successfully did the following (v3.0.0-beta2 is the highest version):
git clone https://github.com/google/protobuf.git
cd ./protobuf/
git checkout v3.0.0-beta-2
./autogen.sh
./configure
make
make check
sudo make install # installs in /usr/local/include/google/, /usr/local/bin/protoc
sudo ldconfig # refresh shared library cache
cd ..

and
git clone --recursive https://github.com/grpc/grpc.git
cd ./grpc/
git checkout release-0_13_0
git submodule update --init
make
make test

but
sudo make install
returns
We couldn't find protoc 3.0.0+ installed on your system. While this
won't prevent grpc from working, you won't be able to compile
and run any meaningful code with it.

Please download and install protobuf 3.0.0+ from:

   https://github.com/google/protobuf/releases

Once you've done so, or if you think this message is in error,
you can re-run this check by doing:

   make verify-install

There is no protoc 3.0.0+

Protocol Buffers v3.0.0-beta-2 is what you want.
If you grab our submodules it's in third_party/protobuf.
On Sat, Feb 27, 2016, 8:43 AM Haisam Ido<EMAIL_ADDRESS>wrote:

I successfully did the following:
git clone https://github.com/google/protobuf.git
cd ./protobuf/
git checkout v3.0.0-beta-2
./autogen.sh
./configure
make
make check
sudo make install # installs in /usr/local/include/google/, /usr/local/bin/protoc
sudo ldconfig # refresh shared library cache.
and
git clone --recursive https://github.com/grpc/grpc.git
cd ./grpc/
git checkout release-0_13_0
git submodule update --init
make
make test
but
sudo make install
returns
We couldn't find protoc 3.0.0+ installed on your system. While this
won't prevent grpc from working, you won't be able to compile
and run any meaningful code with it.
Please download and install protobuf 3.0.0+ from:
https://github.com/google/protobuf/releases
Once you've done so, or if you think this message is in error,
you can re-run this check by doing:
make verify-install
There is no protoc 3.0.0+
—
Reply to this email directly or view it on GitHub
https://github.com/grpc/grpc/issues/5469.

So actually, I have seen that if you install protobuf 3, but still have
proto 2 on the system. Which will obviously create issues.
Please uninstall proto 2 from your system.
On Feb 27, 2016 13:09, "Craig Tiller"<EMAIL_ADDRESS>wrote:

Protocol Buffers v3.0.0-beta-2 is what you want.
If you grab our submodules it's in third_party/protobuf.
On Sat, Feb 27, 2016, 8:43 AM Haisam Ido<EMAIL_ADDRESS>wrote:

I successfully did the following:
git clone https://github.com/google/protobuf.git
cd ./protobuf/
git checkout v3.0.0-beta-2
./autogen.sh
./configure
make
make check
sudo make install # installs in /usr/local/include/google/,
/usr/local/bin/protoc
sudo ldconfig # refresh shared library cache.
and
git clone --recursive https://github.com/grpc/grpc.git
cd ./grpc/
git checkout release-0_13_0
git submodule update --init
make
make test
but
sudo make install
returns
We couldn't find protoc 3.0.0+ installed on your system. While this
won't prevent grpc from working, you won't be able to compile
and run any meaningful code with it.
Please download and install protobuf 3.0.0+ from:
https://github.com/google/protobuf/releases
Once you've done so, or if you think this message is in error,
you can re-run this check by doing:
make verify-install
There is no protoc 3.0.0+
—
Reply to this email directly or view it on GitHub
https://github.com/grpc/grpc/issues/5469.

—
Reply to this email directly or view it on GitHub
https://github.com/grpc/grpc/issues/5469#issuecomment-189726351.

I don't have proto 2.
How do I "grab" the submodules? I did git submodule update --init.  Do I need  to do something else.?

@haisamido Yes, you grab submodules by git submodule update --init which contains third_party/protobuf which you need to compile yourself and install it (i.e. protobuf 3) since the official release is only at 2.6.1. You can read the instructions to install it in the Readme (third_party/protobuf/README.MD)

Have all questions in this issue been answered? Should this issue be closed? What remains here?

Apparently nothing further is needed here; closing.


define jtable custom form

Larger
always blocking key fields
Showing only manual alerts in case of creation
and blocking portfolio and commodity for automatic alerts

See my gist for customization tip

To be tested.


Inline Google fonts

Minimal dependencies is valuable from a security standpoint.
Each external dependency could be used to attack the page potentially.
We can inline the google font to remove one dependency.

It's a few years old now, but a few years ago, I used webfont-dl to perform this task for https://github.com/18F/18f.gsa.gov/pull/672:
https://github.com/mmastrac/webfont-dl


Clarification on Azure RBAC for Azure AD integration

These docs show how we can add Azure AD integration, but also how to integrate with Azure RBAC. It seems like only the latter one is in preview, so simple Azure AD integration is GA?
I assume with Azure AD integration, a cluster admin would create Roles/RoleBindings and ClusterRoles/ClusterRoleBindings for the various groups in the Azure AD with permissions that they have, but can we get an example of what that might look like? Is it possible today to have custom roles for Azure AD directories/users with things like "can get/create Deployments but not list/patch/etc"? Or is it limited to a set of security roles defined in Azure RBAC (which we need preview for)?
I see the docs here: https://docs.microsoft.com/en-us/azure/aks/managed-aad
But is there anything special I need to set in the K8s RBAC resources? I'm familiar with Kubernetes RBAC resources, just not how they tie in to Azure AD.
Also, is my understanding correct, that we can use Azure AD right now (without preview) to limit access to the cluster using regular Kubernetes RBAC resources (not Azure RBAC)?
Thanks.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 92b18eba-07a5-dbbf-b697-0264e326f4d4
Version Independent ID: 8c025ef6-38a6-eca5-7729-8f040fbbbe10
Content: Concepts - Access and identity in Azure Kubernetes Services (AKS) - Azure Kubernetes Service
Content Source: articles/aks/concepts-identity.md
Service: container-service
GitHub Login: @palma21
Microsoft Alias: jpalma

Hello @m1o1 , thanks for the question! We are taking a look into this and will get back to you soon.

Thanks for the feedback! I have assigned the issue to the content author to investigate further and update the document as appropriate.

I think this documentation is what I was looking for:
https://docs.microsoft.com/en-us/azure/aks/azure-ad-rbac
But please correct me if I'm wrong:

Using Azure AD as described in this document does not require any preview features.
Azure RBAC for Kubernetes Authorization which is in preview, is only if we want to use Azure Roles (instead of Kubernetes resources like ClusterRoles and ClusterRoleBindings) for making RBAC decisions in AKS.

Feel free to close, but I would suggest clarifying that distinction (using Azure Roles vs using Kubernetes RBAC roles) a bit more, assuming my assumptions are correct. Thanks!

I think this documentation is what I was looking for:
https://docs.microsoft.com/en-us/azure/aks/azure-ad-rbac
But please correct me if I'm wrong:

Using Azure AD as described in this document does not require any preview features.
Azure RBAC for Kubernetes Authorization which is in preview, is only if we want to use Azure Roles (instead of Kubernetes resources like ClusterRoles and ClusterRoleBindings) for making RBAC decisions in AKS.

Feel free to close, but I would suggest clarifying that distinction (using Azure Roles vs using Kubernetes RBAC roles) a bit more, assuming my assumptions are correct. Thanks!

Hey @m1o1, that's right. I think this specific section is what you're looking for: https://docs.microsoft.com/en-us/azure/aks/azure-ad-rbac#interact-with-cluster-resources-using-azure-ad-identities
Thanks for the feedback!


Query for substances can include experiences

When using the substance query, I observed a record that appears to be an experience.
I'd expect only substances to be returned from the substance query.
You can find the experience I observed with this query:
 substances(query: "Experience:3-MeO-PCP, LSD, Clonazolam, and Amphetamine - Excessive Amounts and Excessive Confusion") {
        name
    	url
    }

Are they also included in the limit query?

@19h yes - that's how I noticed it. The substances(limit: 999) query included that experience as one of the results.


Added dragend to list of rootElementEvents

Looks like 90% of the work to add dragend was done here: https://github.com/facebook/lexical/pull/1986
Only missing to add 'dragend' to the list of events.

Looks right, thanks!


I can't  import usmbus SMBus

Hi,
I need transpose this project:https://github.com/neuberfran/OpenElectrons_i2c/blob/master/OpenElectrons_i2c.py
from python to micropython, but I have issue:

Hi there,
It looks like your python interpreter just can't find the usmbus module: how did you go about installing it?
You should be able to see the list of places python is looking for modules with:
import sys
print(sys.path)

I don't have micropython handy but having the usmbus directory in the same directory as the main script, or in a subdirectory called lib usually works.

I did not give any installation commands, I thought they were already part of the micropython kernel. Do you look at the OpenEletronics_i2c.py module? Will I have to make many changes to the code to stop using your smbus? Do you think I'd better adapt OpenEletronics_i2c.py with the micropython library machine?

Hi,
There are different ways you could tackle this, but it would help to know what you're trying to do: is there a particular sensor you're trying to use?

first of all, thanks for the help.
this is the sensor in case: http://www.mindsensors.com/rpi/76-smartdrive-high-current-motor-controller
the original driver is ### python.
I made a driver for it in java:
https://github.com/neuberfran/SmartDrive/blob/master/SmartDrive/src/main/java/com/neuberfran/androidthings/driver/SmartDrive/SmartDrive.java
for android things,
and thought of doing in ### c ++:
https://forum.arduino.cc/index.php?topic=619218.new#new
would be specifically pro esp32, but I prefer micropython

Can you help me more

I think this module can help you here - all you need to do is install it alongside the OpenElectrons_i2c.py on your micropython board.
If you call it smbus rather than usmbus the code should run more or less without modification, I think (I'm by no means an expert). Depending on which pins you're using, you may need to modify the line that initialises the SMBus object:
https://github.com/neuberfran/OpenElectrons_i2c/blob/e993f3d05ff94a0b0cdb27564348a1933a3917ec/OpenElectrons_i2c.py#L60
I'm also not sure whether the ctypes module will behave exactly as expected, but we can cross that bridge when we come to it :-)

the main issue is still at the beginning. how to install your module so I can import it?

I’m not sure how your environment is set up, so unsure what to suggest. Are you using a python board of some kind? How are you getting the openelectrons code onto it? As a brute force method you could just copy the SMBus class into your file above the openelectrons class

I'm trying to use usmbus SMBus on an esp32 with 30 pin espressif running micropython.
I already rode OpenElectrons_i2c on (and run motors with i2c smartdrive-drive) a raspberry pi 3 and an intel edison, but these two platforms ran normal python and not micropython, so I could use the modules smbus (python normal) and ctypes.
You said:"As a brute force method you could just copy the SMBus class into your file above the openelectrons class"
How can I do this? (pls)

How are you getting the openelectrons file onto your esp32 board at the moment? Do you have it connected via USB? When I'm developing on an esp8266 I have it connected via USB, and use the micropython IDE extension: https://marketplace.visualstudio.com/items?itemName=dphans.micropython-ide-vscode
This allows you to copy files onto the board, so to use usmbus.SMBus you git clone this project, and copy the usmbus directory from this project into the /lib directory on your board.

You said:"As a brute force method you could just copy the SMBus class into your file above the openelectrons class"
How can I do this? (pls)

Something like this is what I was imagining: https://gist.github.com/gkluoe/83b11673484c1a8a4e555607ad52c362

@gkluoe Hi,
https://github.com/neuberfran/SmartMPython
I would appreciate it if you continued to help me solve this project and make motor move. I created the file main.py, but it does not cause error or perform anything

@gkluoe
hi
https://github.com/neuberfran/SmartMPython
I would appreciate it if you continued to help me solve this project and make engines move. I created the file main.py, but it does one only error

Hi - the problem in this picture, I think, is that you're using python 2 syntax, and micropython is (roughly) Python 3. In python 3, you need to use the print() function instead:
print("Batt: " + str(SmartDrive.getBattVoltage())
Should work. If you fix up all the print statements, hopefully that will help!

@gkluoe Tks
I am evolving. While I bring more screenshots with updated issues:
### Look:
(base) iMac:SmartMPython neuberfran$ ampy  --port /dev/tty.SLAB_USBtoUART ls
/OpenElectrons_i2c.py
/SmartDrive.py
/boot.py

Hmm, OK, now I think your issue is here:
https://github.com/neuberfran/SmartMPython/blob/c1fcb3629c27d172440e82ddabfc3004a1555523/OpenElectrons_i2c.py#L3
The ctypes module doesn't exist on micropython. There is a [uctypes](https://docs.micropython.org/en/latest/library/uctypes.html) module, but as far as I can tell, it doesn't offer the .c_int() and .c_long() functions that this code is using.
You could re-implement these functions - in theory that's quite simple - all they do is convert a python int or long into c-style signed number, but this is getting a bit over my head!
To be honest, at this point it's probably going to be easier to just rewrite the OpenElectrons_i2c.py module to work natively in micropython. I'd love to help with that but without your hardware to test on, I don't think I can.

https://github.com/neuberfran/SmartMPython/invitations
any PR will be welcome


Add gen2-color-isp-raw experiment

Example app for IMX378 ColorCamera streaming at full resolution / 12MP (4056x3040):

isp - YUV420p uncompressed frames, processed by ISP
raw - BayerRG (R_Gr_Gb_B), as read from sensor, 10-bit packed

One or both streams can be enabled. By default only isp is enabled.
TODO: raw unpacking isn't implemented yet, so the display doesn't look good.
Install the depthai library: python3 -m pip install -r requirements.txt
then run python3 main.py
User interaction:

press c to capture a set of frames from the enabled streams. Will be saved as .png images.
press q to quit the app.

Also works with IMX477 on boards with FFC connector.

Also works with IMX477 on boards with FFC connector.

Implemented RAW10 unpacking and updated PR description.

Implemented RAW10 unpacking and updated PR description.


How to convert calendar to Islamic "Hijri" date and display it in different language "Arabic and English"?

When I use locale prop. with value "ar-SA" prop it only displayed Hijri Months' name only and it is not the name of the current Hijri Month Name.
How to convert calendar to Islamic "Hijri" preferply Hijri calendar based on Um AlQura date and display it in different language "Arabic and English"  with the correct Hijri Date value? if possible ...

This relates to #517

@berTOTAry I added a comment on the linked issue, this will not be worked on, however if you need to modify the display values, you can use slots and display what is needed


Use null instead of undefined

Could we get the following to return null instead of undefined?
const a = null;
return a?.b;

I'm not saying that null on LHS => null on RHS, I'd just prefer null on RHS in general, regardless of null or undefined on LHS. null is shorter to type and is a language keyword as opposed to undefined. It would also make it easier to distinguish properties that are not defined such as this:
const a = { foo: 5 };
const b = null;
const c = void 0;

a?.foo; // 5
a?.bar; // undefined

b?.foo; // null
b?.bar; // null

c?.foo; // null
c?.bar; // null

I think the reason undefined was used instead of null is that that is the value that's returned when one accesses an undefined property. I'd say that staying consistent with other language features is more important, though doing comparisons with null would be more ergonomic

Repeating what I've said in https://github.com/claudepache/es-optional-chaining/issues/8#issuecomment-275190327

Neither a.b nor a?.b is intended to preserve arbitrary information on the base object a, but only to give information about the property "b" of that object. If a property "b" is absent from a, this is reflected by a.b === undefined.
In the particular case, null and undefined are both treated as immutable ”objects” with no properties. Therefore, we have a?.b === undefined for both a === null and a === undefined, because the property "b" is absent from both null and undefined.

Given that this question was raised more than once, I'll write it down explicitly as part of the justification of the semantics. Noted in #8.

I'd still like to argue the fact that we didn't lookup b on null (in your quoted text), because given that a is null or undefined no lookup happens at all. Therefore, the end result of the expression a?.b is in my opinion totally up in the air and to be defined by the operator itself because it's not the same as looking up b on null (which is not at all undefined, but a TypeError). My point is that this is new behaviour that is to be defined, and defining it as null would be the better choice in my opinion.

@Alxandr
var a = {}
a?.bar // undefined

a?.bar.foo // null or undefined?

@hax last one is a typeerror, no? Cause it looks up bar on a (cause a exists), and the next one is not a "safe" lookup. Thus you get no foo on undefined error. Same as
var a = {bar: null};
a?.bar; // null

a?.bar.foo; // TypeError: cannot lookup foo on null
a?.bar?.foo; // safe: with current proposal will be undefined, I think it should be null.

Null is a value. Undefined is undefined. Otherwise it ambiguous again, and non-intuitive.

I think always using undefined will cause unexpected problems with TypeScript when ...|null is used>
let node: Node|null = getRootNode();
while(node) {
    // ...
    node = node?.parentNode;
}

This will not compile as the null propagating operator will result in the type Node|undefined instead of Node|null. Of course, using undefineed like proposed here would just lead to the same problem with Node|undefined, and you have a valid point of why not to use null or undefined depending on the left hand side. So I don't have a solution, just want to point it out.


Add dynamic SQL support for r2dbc

I want to add dynamic SQL support for r2dbc, just like mybatis. Added the DynamicTemplateProvider interface, although it does not provide a default implementation, it provides a basic implementation. Users can fully implement this function using any third-party template framework, such as enjoy, freemarker, st, and so on.
@schauder do you have chance to take a look ?

Any SQL execution should be performed through the Template API instead of introducing another abstraction for query methods. It would make much more sense to explore how to encapsulate operations from the Repository to invoke a Template-based approach. Right now, operations use PreparedOperation where mapping happens to some extent in R2dbcQueryCreator.

I think many developers prefer to write SQL directly instead of using template APIs. In addition, currently there is still a lot of development space for template APIs

Using SQL queries is already possible via @Query so there's nothing that would prevent you from providing SQL yourself.

Yes, but I need to execute different SQL statements based on the conditions, such as adding mobile number filtering conditions when the parameter has a mobile number. I tried using spel, but it does not support control commands such as if else for. Moreover, the parameter binding of r2dbc is very strict, requiring strict consistency between SQL parameters and interface parameters, otherwise exceptions will occur. For example, if the SQL placeholder has a # {id}, and the interface method has an id and paging parameters, the call to the interface method will fail.


[DMS-13] example simplified failure response body

@CSR2017 Here's an example of what I mean about simplifying the failure response body. I understand that this doesn't do 100% of your implementation, but it's simple without being naïve and does everything necessary.


InvalidApiVersionParameter: Azure CLI 2.51.0 fails to execute Key Vault calls

Describe the bug
We are running a script with two CLI Key Vault calls. These worked until yesterday, and then started to fail. There have been no recent changes to this script or the parameters being passed in. The calls in question are az keyvault show and  'az keyvault set-policy'.
Running az version tells us we are using CLI version 2.51.0. Downgrading to Az CLI version 2.44.1 before running the calls solves the issue.
Related command
$query =  "properties.accessPolicies[?objectId == '$objectId']"
$policy = az keyvault show --name $vaultName --query $query | ConvertFrom-Json
az keyvault set-policy --name $vaultName --object-id $objectId --secret-permissions get list --only-show-errors
Errors
ERROR: cli.azure.cli.core.azclierror: (InvalidApiVersionParameter) The api-version '2023-02-01' is invalid. The supported versions are '2023-07-01,2023-07-01-preview,2023-03-01-preview,2022-12-01,2022-11-01-preview,2022-09-01,2022-06-01,2022-05-01,2022-03-01-preview,2022-01-01,2021-04-01,2021-01-01,2020-10-01,2020-09-01,2020-08-01,2020-07-01,2020-06-01,2020-05-01,2020-01-01,2019-11-01,2019-10-01,2019-09-01,2019-08-01,2019-07-01,2019-06-01,2019-05-10,2019-05-01,2019-03-01,2018-11-01,2018-09-01,2018-08-01,2018-07-01,2018-06-01,2018-05-01,2018-02-01,2018-01-01,2017-12-01,2017-08-01,2017-06-01,2017-05-10,2017-05-01,2017-03-01,2016-09-01,2016-07-01,2016-06-01,2016-02-01,2015-11-01,2015-01-01,2014-04-01-preview,2014-04-01,2014-01-01,2013-03-01,2014-02-26,2014-04'.
Code: InvalidApiVersionParameter
Issue script & Debug output
azCLIVersionKVErrors.txt
Expected behavior
The two Key Vault calls succeed, as in the older versions.
Environment Summary
{
"azure-cli": "2.51.0",
"azure-cli-core": "2.51.0",
"azure-cli-telemetry": "1.1.0",
"extensions": {
"azure-devops": "0.26.0"
}
}
Additional context
No response

Thank you for opening this issue, we will look into it.

Did you specify '--resource-group' parameter to avoid listing all vaults?

Including the –resource-group param fixed the issue. That wasn’t needed before, but we are unblocked at this point.
From: Yong Zhang @.>
Sent: Saturday, August 12, 2023 2:25 AM
To: Azure/azure-cli @.>
Cc: Clemons, Christopher @.>; Author @.>
Subject: Re: [Azure/azure-cli] InvalidApiVersionParameter: Azure CLI 2.51.0 fails to execute Key Vault calls (Issue #27159)
This mail has been sent from an external source. Do not reply to it, or open any links/attachments unless you are sure of the sender's identity.
Did you specify '--resource-group' parameter to avoid listing all vaults?
—
Reply to this email directly, view it on GitHubhttps://github.com/Azure/azure-cli/issues/27159#issuecomment-1675737613, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A4LKY6TENHYVEKKRIYXVHULXU4OU3ANCNFSM6AAAAAA3NOOPOI.
You are receiving this because you authored the thread.Message ID<EMAIL_ADDRESS>


Test MJCF descriptions with Pinocchio

Pull Request Test Coverage Report for Build<PHONE_NUMBER>7
Details

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 97.335%

Totals

Change from base Build<PHONE_NUMBER>3:
0.0%

Covered Lines:
913

Relevant Lines:
938

💛  - Coveralls


be more integrator-friendly

The functions in utils.py often implicitly do a sys.exit(1) (_error()...)
This is unkind to those that'd want to use these imports. Default to returning an empty string or false or something, then have the cli detect the return and exit for us...

Should also add a --version in the argparse...


Integration tests for file provisioning plugins

Add integration tests for the S3 and GS file provisioning plugins that literally exercise uploading and downloading files from Amazon S3 and Google Cloud Storage
These tests should also exercise error conditions, e.g. when an invalid bucket name is specified.
┆Issue is synchronized with this Jira Story
┆Issue Type: Story
┆Fix Versions: Dockstore 1.6
┆Issue Number: DOCK-177

@wshands punting, think this can be done off-cycle from Dockstore releases


Persian translation of chapters Interrupts and Memory Managements

Me and @MHBahrampour are working on translations of chapters Interrupts and Memory Managements.
the Interrupts chapter is almost done and will be soon a PR.
So anyone interested in helping, let me know so you don't do any duplicate.

Translation of chapter interrupts is done in #904.
Thanks to @MHBahrampour for helping in translations.

Translation of chapter interrupts is done in #904.
Thanks to @MHBahrampour for helping in translations.

Translation of post-08 is done in #913.

done / #908 will handle the rest


make electron app title customizable

Screenshots:

I've always run multiple devtools for two projects,
this feature made easier to identify each devtool for their project.

any ones?

Maybe it would be better to just take the title of the page or nativescript app being inspected?

@Akryum How about this?
CurrentComponent - CurrentPageTitle - Vue Dev Tools


CLI command for adding additional user accounts

We need a way to programmatically add additional users.  Furthermore, it must be possible to query the added users' information:

username
full name
email address
initial password

For our specific use case, we actually only need to be able to add exactly one administrative user account.  For this purpose, the method employed by OpenSlides 3 (see below) would still work perfectly well.  Since that way the user information is available in a secrets file, it remains available even while the instance is not running which is an advantage.
I imagine you might want to add a generic adduser command instead, however, e.g., openslides adduser --role=admin --name="John Doe"<EMAIL_ADDRESS>john or something like that.  That would generally be okay, too, of course.  The password should be generated; an option to interactively ask for the password wouldn't hurt but is not strictly necessary, in my opinion.
For your convenience, this is the OpenSlides 3 setup:

template:

https://github.com/OpenSlides/OpenSlides/blob/bbdbdd3c10199485afaeedc94f3911331c57a082/docker/docker-compose.yml.m4#L44
https://github.com/OpenSlides/OpenSlides/blob/bbdbdd3c10199485afaeedc94f3911331c57a082/docker/docker-compose.yml.m4#L161
https://github.com/OpenSlides/OpenSlides/blob/bbdbdd3c10199485afaeedc94f3911331c57a082/docker/docker-compose.yml.m4#L286-L287

back end (presumably replaced by the manage service in OS4):

https://github.com/OpenSlides/OpenSlides/blob/bbdbdd3c10199485afaeedc94f3911331c57a082/server/docker/entrypoint-db-setup#L57-L80

I don't like it to perform actions without an authentication.
You can do this to achieve your goal:

Prompt a user to enter their username and password
Send a request to the auth's login-route
Store the returning token and cookie

Then, the user can perform actions by your tool without any modification of the infrastructure of our application.

Internal route exists so this has to be done now.

Should be done.


--print-types realiases from global context

When the following program is compiled with elm --print-types it shows a weird type.
problem = ([3.1415], [1.618])

It shows the following:
problem : Automaton.Queue Float

The original program does not import Automaton and Queue should not be exported. This indicates that realiasing with --print-types is not taking the local imports or module exports into account.
Note: this is based on issue #287

This flag no longer exists.


dll support

Instead of an executable, I need an dll with the same functionalities. Because not every platform allows executables to be distributed.
Also a .Net Standard 2.0 dll wrapping the native dll would even be better.

Or never mind, I did miss the part that it can be found at %ProgramFiles(x86)%\Microsoft Visual Studio\Installer\vswhere.exe

See also https://www.nuget.org/packages/Microsoft.VisualStudio.Setup.Configuration.Interop, but it doesn't do all the queries for you like vswhere.exe.


ERROR: Failed to load qobj

Environment

Qiskit Terra version: 0.20.0
Python version: 3.7.12
Operating system: win10
{qiskit-terra': '', 'qiskit-aer': '0.10.4', 'qiskit-ignis': '0.7.0', 'qiskit-ibmq-provider': '0.19.0', 'qiskit-aqua': None, 'qiskit': '0.36.0', 'qiskit-nature': None, 'qiskit-finance': None, 'qiskit-optimization': None, 'qiskit-machine-learning': None}

What is happening?
I create a "Quantum Class" with qiskit. Then I test the implementation according to the https://learn.qiskit.org/course/ch-applications/hybrid-quantum-classical-neural-networks-with-pytorch-and-qiskit
with a QiskitError!!!
QiskitError: 'Data for experiment "circuit-3" could not be found.'

How can we reproduce the issue?
I don't known
What should happen?
I don't know why,but it has error
Any suggestions?
No response

Hello, sorry for the slow reply.  Please could you provide more details, for example the actual code you have run, and the full Python error message and traceback you see?
Your image link hasn't uploaded correctly, but images aren't a good way to show us code - it's much easier for us to help debug if you can write a minimal reproducing example directly into the GitHub issue.  You can copy several lines of code in if you put it between two pairs of three backticks, such as
```
<code goes here>
```


feat(project): add support for .env files

What does this PR do?

If you don't know what an .env file is, please read about dotenv.

It adds support for .env files that can be scoped to your targets. The files can load environment variables that projext will send to the Node runner and the browser targets configuration and the "definitions" build engines replace on the bundles.
You can read more about the feature on the updated projectConfiguration.md.
why is this PR breaking?
First, BuildNodeRunner and BuildTraspiler now require Utils: until now, they were using it from inside the Targets injection, with a @todo saying that I would eventually change it... well, I did it.
And Targets now requires dotEnvUtils, which is a new service that takes care of loading and parsing the .env files.
How should it be tested manually?
As for now, this can only be tested on Node targets that don't require bundling nor transpilation; The build engines will later add the rest of the support.
Now, create a file called .env.[target-name] and declare some variable, then, on your target code, log that variable.
When you run your target, you should have access to the variable as it was declared on the .env file.
And of course...
yarn test
# or
npm test

Pull Request Test Coverage Report for Build 286

56 of 56 (100.0%)  changed or added relevant lines in 6 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 100.0%

Totals

Change from base Build 283:
0.0%

Covered Lines:
2194

Relevant Lines:
2194

💛  - Coveralls


Change distributionManagmeent to release to Central Repository

Should change poms to release artifacts to Maven Central instead of source.sakaiproject.org. Relates to issue filed on https://code.google.com/p/reflectutils/issues/detail?id=15&can=1.

http://central.sonatype.org/pages/ossrh-guide.html

It would be good to get a snapshot released to the OSS Snapshot repository so we could test this without having to build locally, is there anything I can do to help with this?

I'm open to suggestions. I think I applied to try to get a space in there
but not sure what the next steps would be (just haven't had much time to
dig into it).
On Wed, Apr 1, 2015 at 6:27 AM, Matthew Buckett<EMAIL_ADDRESS>wrote:

It would be good to get a snapshot released to the OSS Snapshot repository
so we could test this without having to build locally, is there anything I
can do to help with this?
—
Reply to this email directly or view it on GitHub
https://github.com/azeckoski/reflectutils/issues/2#issuecomment-88431318
.

--
Aaron Zeckoski - Software Architect - http://tinyurl.com/azprofile

I agree, it looks like from https://issues.sonatype.org/browse/OSSRH-9134 that it's ready to go. I can submit a PR to have the pom updated then if you were going to deploy snapshots with maven that sounds like all that would be necessary.

Sounds like a plan
:-)
On Thu, Apr 2, 2015 at 1:29 PM, Code Hugger (Matthew Jones) <
<EMAIL_ADDRESS>wrote:

I agree, it looks like from https://issues.sonatype.org/browse/OSSRH-9134
that it's ready to go. I can submit a PR to have the pom updated then if
you were going to deploy snapshots with maven that sounds like all that
would be necessary.
—
Reply to this email directly or view it on GitHub
https://github.com/azeckoski/reflectutils/issues/2#issuecomment-88981390
.

--
Aaron Zeckoski - Software Architect - http://tinyurl.com/azprofile

Presumably you've got the credentials for Sonatype so are you able to deploy a snapshot there now @azeckoski ?
I think you'll need to edit your ~/.m2/settings.xml to include your Sonatype credentials: http://central.sonatype.org/pages/apache-maven.html

Should be done now.
https://oss.sonatype.org/content/repositories/releases/org/azeckoski/reflectutils/0.9.20/
Description:
org.azeckoski:reflectutils:0.9.20
Deployer properties:
"userAgent" = "Nexus-Client/2.7.2-01"
"userId" = "azeckoski"
"ip" = "<IP_ADDRESS>"
Details:
The following artifacts have been promoted to the "Releases" [id=releases] repository
/org/azeckoski/reflectutils/0.9.20/reflectutils-0.9.20-javadoc.jar.asc
(SHA1: ee21d493a43a0c28b0f3fd26bab047c2aa973a2c)
/org/azeckoski/reflectutils/0.9.20/reflectutils-0.9.20.jar.asc
(SHA1: b2868351e021b1334372d4cf615bc57147ea096a)
/org/azeckoski/reflectutils/0.9.20/reflectutils-0.9.20-javadoc.jar
(SHA1: b8c1ec21a6bdfb2086370c08e2ca3d76ac3412b4)
/org/azeckoski/reflectutils/0.9.20/reflectutils-0.9.20.jar
(SHA1: c4e36085243e8e3e004651d9583caf22d4738420)
/org/azeckoski/reflectutils/0.9.20/reflectutils-0.9.20.pom
(SHA1: 9f5fa40da0dd258cfdaa682e82614f1bbfc4001c)
/org/azeckoski/reflectutils/0.9.20/reflectutils-0.9.20-sources.jar.asc
(SHA1: 62f25cbb3aadb6f46a1039770bb27bd502e0d9be)
/org/azeckoski/reflectutils/0.9.20/reflectutils-0.9.20-sources.jar
(SHA1: 3ca0838cd90fc1987d48ab11950a856ce245dcc4)
/org/azeckoski/reflectutils/0.9.20/reflectutils-0.9.20.pom.asc
(SHA1: 0d9396069e9ba7faed6a0d1bd462d9bd9a370fa2)
Action performed by Aaron Zeckoski


v2 install error message

I'm getting the following error message during the installation:
Error: The header content contains invalid characters
I made sure to delete all previous installation data.
Below my console dump:
cordova plugin add https://github.com/mapsplugin/cordova-plugin-googlemaps#multiple_maps --variable API_KEY_FOR_ANDROID="DUMMY" --variable API_KEY_FOR_IOS="DUMMY" —save
Fetching plugin "https://github.com/mapsplugin/cordova-plugin-googlemaps" via git clone
Repository "https://github.com/mapsplugin/cordova-plugin-googlemaps" checked out to git ref "multiple_maps".
Installing "cordova-plugin-googlemaps" for android
ANDROID_HOME=/Users/marcel/Library/Android/sdk
JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_25.jdk/Contents/Home
Subproject Path: CordovaLib
NDK is missing a "platforms" directory.
If you are using NDK, verify the ndk.dir is set to a valid NDK directory.  It is currently set to /Users/marcel/Library/Android/sdk/ndk-bundle.
If you are not using NDK, unset the NDK variable from ANDROID_NDK_HOME or local.properties to remove this warning.

null
The Task.leftShift(Closure) method has been deprecated and is scheduled to be removed in Gradle 5.0. Please use Task.doLast(Action) instead.
        at build_6yzno43nv6o3850av0e3l8i66.run(/Users/marcel/Documents/xCode/lingusocial/platforms/android/build.gradle:150)
org.xwalk:xwalk_core_library:23+
NDK is missing a "platforms" directory.
If you are using NDK, verify the ndk.dir is set to a valid NDK directory.  It is currently set to /Users/marcel/Library/Android/sdk/ndk-bundle.
If you are not using NDK, unset the NDK variable from ANDROID_NDK_HOME or local.properties to remove this warning.

Incremental java compilation is an incubating feature.
:clean
:CordovaLib:clean

BUILD SUCCESSFUL

Total time: 1.013 secs
Subproject Path: CordovaLib

             In order to run this plugin, you need to obtain Google Maps API key for Android.
             Visit https://github.com/mapsplugin/cordova-plugin-googlemaps/wiki/Installation

             Check our release notes.
             https://github.com/mapsplugin/cordova-plugin-googlemaps/wiki/Release-Notes

Installing "cordova-plugin-googlemaps" for browser
Installing "cordova-plugin-googlemaps" for ios
Fetching plugin "https://github.com/mapsplugin/cordova-plugin-googlemaps-sdk" via git clone
Repository "https://github.com/mapsplugin/cordova-plugin-googlemaps-sdk" checked out to git ref "master".
Installing "com.googlemaps.ios" for ios

             In order to run this plugin, you need to obtain Google Maps API key for iOS.
             Visit https://github.com/mapsplugin/cordova-plugin-googlemaps/wiki/Installation

             Check our release notes.
             https://github.com/mapsplugin/cordova-plugin-googlemaps/wiki/Release-Notes

Error: The header content contains invalid characters

Ummm, I have no idea.

Great, so we have a challenge :) I forgot to mention that I tried to install v2. I'm going to create a dummy project tomorrow morning and I'll try installing the plugin from a blank project.

Any update for this? Although I reinstalled the plugin code many times during the last 5 days, I haven't faced this issue.

Nope, didn't have the time yet. So I'm going to close this one because it seems to be related to my local configuration.


fix(cli): support for Windows [publish]

Improves logic for publishing to support Windows

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


What is new in 5.0.0?

Hi there, I just upgraded to 5.0.0 and I am getting a ton of errors. it seems the library's API has changed. Is there any new documentation for 5.0?

:wave: The changes should be outlined in the CHANGELOG: https://github.com/braintree/framebus/blob/master/CHANGELOG.md#500
It's mostly that you can now scope your bus to only send and listen for events on the same channel.

Not sure how I missed that! Apologies for opening the issue, I swear I did check the docs.


Warning: app.component.scss exceeded maximum budget - DOES NOT give any clue why its exceeded

🚀 Feature request
Command (mark with an x)

[ ] new
[x] build
[ ] serve
[ ] test
[ ] e2e
[ ] generate
[ ] add
[ ] update
[ ] lint
[ ] extract-i18n
[ ] run
[ ] config
[ ] help
[ ] version
[ ] doc

Description
Ng build in my application produce following warning
Warning: frontend/apps/vo/src/app/app.component.scss exceeded maximum budget. Budget 6.00 kB was not met by 274 bytes with a total of 6.27 kB.
But visually this scss file is pretty small and actually I do not see any generated css exceeding this budget.
Describe the solution you'd like
Add possibility to view generated css

@Lonli-Lokli,
Thanks for this feature request.
Similar to other budgets we don't show the contents of the file when a budget is not met as the content can be rather large and at this stage the CSS is minified thus whitespaces and formatting are completely removed and hence dumping a bunch of compressed CSS in the terminal might not be very helpful, it would also cause a lot of chum when multiple budgets are not met.
If you are uncertain what failed the failure, the best course of action is to analyze the generated bundle itself. Similar to how you'd do if another type of budget is not met.

@alan-agius4 may I kindly ask you to explain how it can be analyzed? webpack-bundle-analyzer do not give content of css, while all other parts with JS are listed.
Honestly speaking this warning is meaningess for develper as there is no explanation how to fix it.


Fix linux relocation instructions

/usr/local/lib is the correct place, I can accept an update that hints some systems may require adding /usr/local/lib to /etc/ld.conf, but I don't want to recommend installation that isn't standard.

I think I covered this in the latest update.


[macOS] First click does nothing after maximize/unmaximize

Double-clicking the top of the application to maximize or unmaximize the window results in the first click not doing anything. I have to click twice in order for any of my buttons/UI elements to register the click.
I've tried hooking into resized, resize, maximize, unmaximize events and calling show(), restore(), etc, but nothing seems to work.
Any suggestions?

@Jethro87 did you find solution? I'm facing same issue

@13thdeus Unfortunately not

Can you provide a minimum reproducible example?

I've found adding window toolbar cause this issue. In my case I need it to move window control buttons (traffic lights) down a bit
Here are some details:

main.dart content
void main() async {
  WidgetsFlutterBinding.ensureInitialized();
  await windowManager.ensureInitialized();

  WindowOptions windowOptions = const WindowOptions(
    size: Size(800, 600),
    backgroundColor: Colors.transparent,
    skipTaskbar: false,
    titleBarStyle: TitleBarStyle.hidden,
  );
  windowManager.waitUntilReadyToShow(windowOptions);

  runApp(const MyApp());
}

MainFlutterWindow.swift content
import Cocoa
import FlutterMacOS

class MainFlutterWindow: NSWindow {
  override func awakeFromNib() {
    let flutterViewController = FlutterViewController()
    let windowFrame = self.frame
    self.contentViewController = flutterViewController
    self.setFrame(windowFrame, display: true)

    RegisterGeneratedPlugins(registry: flutterViewController)

    self.toolbar = NSToolbar()

    super.awakeFromNib()
  }
}

Looks like issue is not related to package since same issue can be reproduced without using package.
I would be grateful for any help in solving this problem

Finally I've managed how to achieve effect I've wanted. I've deleted NSToolbar and used DragToMoveArea from this package to replace toolbar features (double click and drag). Then I've reimplemented this electon pull request to move traffic lights as I want.

MainFlutterWindow.swift
import Cocoa
import FlutterMacOS

class MainFlutterWindow: NSWindow {
    private var inFullScreenTransition = false

    override func awakeFromNib() {
        let flutterViewController = FlutterViewController()
        let windowFrame = self.frame
        self.contentViewController = flutterViewController
        self.setFrame(windowFrame, display: true)

        RegisterGeneratedPlugins(registry: flutterViewController)

        super.awakeFromNib()

        // traffic lights position must be recalculated on any resize step
        NotificationCenter.default.addObserver(forName: NSWindow.didResizeNotification, object: self, queue: OperationQueue.main) { Notification in
            self.repositionTrafficLights(x: 17.5, y: 38)
        }

        // during exiting from full screen reposition works bad. This part fixes behavior
        NotificationCenter.default.addObserver(forName: NSWindow.willExitFullScreenNotification, object: self, queue: OperationQueue.main) { Notification in
            self.inFullScreenTransition = true
            self.repositionTrafficLights(x: 17.5, y: 38)
        }
        NotificationCenter.default.addObserver(forName: NSWindow.didExitFullScreenNotification, object: self, queue: OperationQueue.main) { Notification in
            self.inFullScreenTransition = false
            self.repositionTrafficLights(x: 17.5, y: 38)
        }

    }

    func repositionTrafficLights(x: CGFloat, y: CGFloat) {
        let window = self
        let close = self.standardWindowButton(NSWindow.ButtonType.closeButton)!
        let miniaturize = self.standardWindowButton(NSWindow.ButtonType.miniaturizeButton)!
        let zoom = self.standardWindowButton(NSWindow.ButtonType.zoomButton)!
        let titleBarContainerView = close.superview?.superview

        if (titleBarContainerView == nil) {
            return
        }

        if (inFullScreenTransition) {
            titleBarContainerView?.isHidden = true
            return
        }

        titleBarContainerView!.isHidden = false
        let buttonHeight = close.frame.size.height
        let titleBarFrameHeight = buttonHeight + y
        var titleBarRect = titleBarContainerView!.frame
        titleBarRect.size.height = titleBarFrameHeight;
        titleBarRect.origin.y = window.frame.size.height - titleBarFrameHeight;
        titleBarContainerView!.frame = titleBarRect;

        let windowButtons = [ close, miniaturize, zoom ];
        let space_between = miniaturize.frame.origin.x - close.frame.origin.x;
        for i in 0...2 {
            let view = windowButtons[i]
            var rect = view.frame
            rect.origin.x = x + (CGFloat(i) * space_between)
            rect.origin.y = (titleBarFrameHeight - rect.size.height) / 2
            view.setFrameOrigin(rect.origin)
        }
    }
}

Based on this code it is possible to create new cool feature for macOS (moving traffic lights is really painful)


Adds ability to split a pallet across multiple files

A POC with the ability to split all components, except Storage which I am looking at. Once fixed, will probably create another example in frame/examples instead of modifying the template pallet.

do we really need this?

When I did my substrate seminar last fall this was the main thing people
brought up in the Q&A as a desired feature, with all sorts of reasons
ranging from breaking up very large pallets into smaller more digestible
files to allowing for code sharing of identical sections that could be
shared between multiple pallets.
The outer macro pattern our pallets rely on is extremely powerful, but the
price we pay is everything has to be in one module in one file. This
removes that restriction without sacrificing any of the power or
capabilities the outer macro pattern brings us in FRAME.
On Wed, Apr 19, 2023 at 12:24 AM Xiliang Chen @.***>
wrote:

do we really need this?
—
Reply to this email directly, view it on GitHub
https://github.com/paritytech/substrate/pull/13950#issuecomment-1514112720,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAOE4LPSEYPQOFQ5DGM5ZVLXB5SGLANCNFSM6AAAAAAXDOSZWM
.
You are receiving this because your review was requested.Message ID:
@.***>

You are not talking with the right set of developers for feedbacks. Have you asked senior substrate developers? like the core devs in Parity and parachains, instead of students?
We have a lot of complex pallets in Substrate, staking for example is one, have you asked the core dev for pallet-staking to see if this is helpful?
At least for all the pallets I maintain, I don't need this. The normal ways of abstracting the logics is enough.

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2699889

You are not talking with the right set of developers for feedbacks. Have you asked senior substrate developers? like the core devs in Parity and parachains, instead of students?
We have a lot of complex pallets in Substrate, staking for example is one, have you asked the core dev for pallet-staking to see if this is helpful?
At least for all the pallets I maintain, I don't need this. The normal ways of abstracting the logics is enough.

As you would imagine, the current mechanism stays and you can continue to use that if you don't find this useful.

Maybe you are right but you need to show an example of how this is beneficial and cannot be done otherwise. Chances are there are easier way to break pallets into smaller files and code reusing without macro magic.
Again, this is another thing I don't like how some approach a problem. You need to show there is an indeed a problem, propose a solution, highlight how it fixes the problem, and then code it.
How is moving events to its own file enable code reuse? How are you going to reuse those events?

You are absolutely right here. That's why I mentioned that this is a POC and the PR is a draft. Once the approach makes sense, I will start to build examples where this would make sense.

Have you asked senior substrate developers?

This was discussed a good bit in the parity forum a while ago where @bkchr, Shawn, and @aaronbassett were in favor of it. People coming out of the academy consistently ask for this as well.
Rather than look at this from a perspective of "why should we add this", consider this: in Rust, it is normal to be able to split up a module into multiple smaller modules. This is just how Rust works, ad it enables better readability and a number of positive things commonly associated with modularity that I'm sure we all could agree on.
Pallets are modules, however since FRAME parses pallets using the outer macro pattern, we are stuck with this artificial, non-rusty limitation that the pallet module cannot be split into multiple external files like normal Rust modules can.
The burden of proof should then be on "why shouldn't we remove this non-Rusty artificial limitation?"

Well, you are not wrong, but again, there are ways to break pallets into multiple files and that worked well so far (least for myself). But yeah maybe those are bad workaround that deserve a better solution. A real life example will help me to change my mind.

Also yes I can simply not use this features if I don't want to use but I guess I am just getting a bit grumpy when I see people wasting time building something isn't important than fixing critical issues I raised years ago.

You are not talking with the right set of developers for feedbacks. Have you asked senior substrate developers? like the core devs in Parity and parachains, instead of students?

The desire to break pallets into smaller parts stems not from PBA or students, but rather from Parity engineers, and dates at least to 2021, with the introduction of the new pallet macro that forced everything into one file.
I agree with the rest of your comments @xlc and I see that @sam0x17 and @gupnik are also onboard, we need to now find use-cases for this.
I broadly think this can help us with 3 classes of issues:

Breaking big files into smaller ones. Developers generally like to do this for various reasons and I think that's a totally fair ask. My only worry is that if this breaks RA support even further, I would personally be hesitant to use it.

If you want to try one, pallet-staking is already trying to break itself into smaller parts and could use a new approach.
Also, you can just have a new pallet-example-splitted that has each pallet part in a new file, and an aggregating lib.rs that is only bringing them under one umbrella.

As noted, I think the real killer feature is not splitting, but reusing. Right now, we have a paradigm where if a pallet wants to use a set of features from another code component that needs its own storage/events/calls, it has to be two pallets.

I think this is perfectly fine for most use cases. Moreover, it forces us to link the two pallets together via a trait, which makes you think about the interface between the two components.
The downside is having too much boilerplate, and some limitations like the fact that someday, we simply will run out of pallets to add to the outer RuntimeCall enum.
One example that you can look into is pallet-bags-list and how it is being used in pallet-staking. It is basically a pallet that only provides the functionality of a linked-list to some other pallet. I can't say I was particularly annoyed by implementing and using it as a whole new pallet, but I am certainly curious to see how it could have been done otherwise.

Perhaps a subset of the latter, I another feature that I would personally like to see is an easier means of "privatizing" a group of storage items, and putting them behind a firewall that makes sure they are only mutated in a sensible way. A few examples of this that come to mind:

bags-list has some storage items that must never be mutated directly
A similar one in the offences pallet.
This example in election-provider-multi-phase. This is a good example for use case 1 as well. All of the logic for the "signed phase" in this pallet is in signed.rs, but the storage and calls are not stuck in the main.rs.

All in all, I think this is a very promising avenue. I hear the call for more validation, and it is fair, but I think enforcing it at an early stage of an experimental work like this will hinder innovation. Sometimes you don't know what you are missing unless if some tool is there to fix it for you.

It sounds like we don't really want to break pallet into files. This is not the goal.
For (1), I don't yet see a big pallet that is build in such way will be benefit from this. We can already move most of the logics to other files. I definitely prefer to have all the definitions in a single file.
For (2) and (3), they sound like the same thing to me and split pallet across multiple files is not the right solution. Let's use common programming design patterns when approaching such a common issue.
Let's use OO here. A class (pallet) is made of fields (storages) and methods (calls). The calls can return errors and emit events. What happen if we have a god class that's too big to maintain? We break it down into multiple smaller class. We don't break it down into multiple files and write fields in one file and method in another file and somehow allow the fields to be reuseable??? This simply makes zero sense.
So the right solution is being able to define a component that's made of storages, calls and supporting types (events, errors and related types) and then build a pallet from those components.
How about inheritance. Allow a pallet to inherent everything from another pallet. Then we can build a storage only pallet and that's exactly what you want here. It could even support multiple inheritance (and make your life harder).
We can use composition. Allow a pallet to embed child pallet. The child pallet's storage will be placed under parent pallet's storege prefix. The calls events errors can be aggregated and flattened.

Let's use OO here. A class (pallet) is made of fields (storages) and methods (calls). The calls can return errors and emit events. What happen if we have a god class that's too big to maintain? We break it down into multiple smaller class. We don't break it down into multiple files and write fields in one file and method in another file and somehow allow the fields to be reuseable??? This simply makes zero sense.

I agree with this sentiment, but we are already stuck with how FRAME was designed around the outer macro pattern encompasing one giant pallet module. We could completely re-design FRAME to be more "proper" from an OO perspective for example, but with FRAME the way it is, where a pallet is a giant module, we can't stop people from expecting to be able to use normal Rust mechanics like use some::path to bring in other modules containing #[pallet::whatever] stuff. Not being able to do this violates principle of least surprise for people used to how Rust modules are supposed to work, and this PR provides a completely optional path for people who want to organize their code in a way the existing FRAME syntax already strongly suggests you can.

we can't stop people from expecting to be able to use normal Rust mechanics like use some::path to bring in other modules containing #[pallet::whatever] stuff. Not being able to do this violates principle of least surprise for people used to how Rust modules are supposed to work

Maybe it is just me but I definitely will be surprised if import form other modules actually work. I just don't agree with you on this. But anyway, lets skip this point.
Let's get back to code reuse. You have to give me an example on how this can achieve code reuse. Also the goal is to achieve code reuse, not split files so you NEED to explore alternative solutions and syntax.

You are not talking with the right set of developers for feedbacks. Have you asked senior substrate developers? like the core devs in Parity and parachains, instead of students?

If you do not take students' opinions into account then where do you think the next generation of senior substrate developers will come from?

NO. You may find Rust is hard to learn/use and have some idea on how to improve Rust syntax but I would not trust your suggestions on how to improve Rust. Your suggestion maybe great but it MUST be reviewed by multiple Rust experts first and go through a lot of discussions before any implementation. Same thing apply here.

@xlc your comments here are all valid alternative ways to reach reuse and composition, and very well formulated. We could achieve code-reuse and splitting to some extent via different ways, and this PR is merely adding new way to explore such avenues. I don't think this is a big issue in itself, and it takes me back to the point that your main point of complaint here:

That you think this is not a high priority, more important issues are not addressed.
This was not communicated in advance.
You want to see use cases of this in action.

all 3 of which are valid, I personally asked and suggested showcases for the 3rd one as well, but the first two are off-topic related to this PR and I suggest we take the discussion to another forum.

@kianenigma @sam0x17 @xlc @ggwpez Added an example here using the bags-list. This is now moved to a separate crate as a common utility and encapsulates its storage types as well.
Todo: Move the tests inside the bags-list crate as well.

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-check-benches
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2756673

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-check-each-crate-macos
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2892309

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-check-each-crate-macos
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2892394

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2892436

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2892437

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2892438

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-check-benches
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2912994

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-check-benches
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2912995

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-check-benches
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2913089

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-check-benches
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2913090

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-check-benches
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2913157

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-check-benches
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2913158

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-check-benches
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2913248

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-check-benches
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2913247

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-check-each-crate-macos
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2913334

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-check-each-crate-macos
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2913387

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2913453

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2913455

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2913454

bot fmt

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-check-benches
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2914131

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-check-benches
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2914132

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-check-each-crate-macos
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2914308

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2914394

bot fmt

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2914811

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-check-each-crate-macos
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2917269

bot fmt

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2917911

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2919633

bot fmt

bot fmt

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2919997

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-check-benches
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2923976

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-check-benches
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2923975

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-check-benches
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2926626

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-check-benches
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2926625

bot fmt

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-check-benches
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2930432

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-check-benches
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2930433

When I heard about breaking pallet into multiple parts I thought it was more about having a pallet broke down into multiple sub-pallets. Like each sub-pallet would define and manage their own storage, events, initialization etc...
In polkadot the parachains pallets do split logic into multiple pallets and then there is initializer pallet which depends on all the other pallets and initialize them. https://github.com/paritytech/polkadot/blob/master/runtime/parachains/src/initializer.rs
To improve on this we would need some kind of sub-pallet definitions, or on the other way, overarching-pallet definitions.
I think this pattern could be improved. But this is not what the current PR is addressing.
About the current PR:

maybe spans are wrongly generated or it is not possible to properly generate them but my IDE cannot lookup to definitions easily. Looking up for an Event variant will lead to import_section macro attribute. Which is confusing especially for newcomers.
it is confusing that type are not where they are define: I cannot write: type Foo<T> = events::events::Event<T>; which is the expected path for the Event type in rust. instead I have to write type Foo<T> = pallet::Event<T>; which is unexpected considering no Event type is defined in the pallet module.

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-check-each-crate-macos
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2932175

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-check-benches
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2932158

When I heard about breaking pallet into multiple parts I thought it was more about having a pallet broke down into multiple sub-pallets. Like each sub-pallet would define and manage their own storage, events, initialization etc...

Indeed, the vision is to be able to split into multiple sub-sections as you mentioned. This PR is just the first step in being able to break down the components.

maybe spans are wrongly generated or it is not possible to properly generate them but my IDE cannot lookup to definitions easily.

Currently, the rust analyzer support is limited. Will try to create a follow-up that might resolve this.

It is confusing that type are not where they are define: I cannot write: type Foo = events::events::Event; which is the expected path for the Event type in rust.

@sam0x17 @kianenigma Any thoughts here?

Also sub-pallet/overarching-pallet mechanism seems orthogonal to this.

The eventual idea here is to create these sections which encapsulate storage, events and calls within them. However, I can see that it probably doesn't align with the way you imagined. Could you expand on your thoughts around sub-pallets to help me understand better?

I could see 2 things:
1 - allow to split storage into multiple modules so that logic, user could then in each modules expose only some API which handle their storage. This would avoid having all storage usable from the root the crate, and allow hide them behind some sound API.
so mod1 would have storage s1, s2. mod2 would have storage s3 and s4. in the pallet module everything would be gathered to generate the metadata. But user should not have access to the storage in the pallet module. only in mod1 and mod2. Also mod1 storage would not be accessible from mod2.
Allow to hide storage behind sound API could be helpful.
2- some pallet are defined as set of pallets, similar to initializer in polkadot.
This pallet (the overarching pallet) would define the initalization order for each pallet for instance.
But also I'm not sure how useful this would be but polkadot parachains would use such pattern.

I think what you outline @thiolliere is definitely what we would want, but the fact that the #[pallet] macro is an outer macro pattern dramatically complicates things. Without something sketchy like include_str! or macro_magic, we can't give the #[pallet] macro information about other modules at compile-time. If you have a workaround for that that would also allow the use of real rust modules I'm totally for it!!
In either case we should definitely hold off on merging this until this is done/merged:
https://github.com/sam0x17/macro_magic/issues/3

Maybe we should move discussion to an issue, but for idea 1, I think it should be doable with some macro stuff similar to macro_magic.
pallet::foreign_storage would be an attribute macro which does similar to pallet::storage but takes into argument the path to the Config trait, and generate a macro which gives information for generating storage information, genesis_storage stuff etc... This macro will later be called by pallet macro when notify through an attibute like [pallet::foreign_storage(path_to_the_foreign_storage)].

I would suggest being careful not crossing the line where the outcome "super pallet" is so hard to debug and reason about.

Yeah in this case we are still creating a regular pallet as far as all the compile-time machinery is concerned (this code simply pre-empts most of the parsing and expansion logic for #[pallet] by injecting the external code beforehand.
So this shouldn't step on the toes of anyone trying to do "super pallets" I don't think 👍🏻

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-check-each-crate-macos
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/3038102

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-check-each-crate-macos
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/3038170

bot fmt

@kianenigma @sam0x17 Could you take a look now?

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-check-benches
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/3077077

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-check-benches
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/3077169

bot merge