cannot build containers, python_install.sh fails (Orin AGX 64GB, JetPack 6.1)

I've got a brand new Orin AGX developer kit (64GB) and used the SDK Manager to download and flash the latest Ubuntu (22.04) and JetPack SDK (6.1).
I'm trying to build a container (I need diffusers and transformers for my application code), however when I run the build command from the README (actually a truncated one, just trying to build pytorch without transformers and ros) it bombs out in the "install python" step:

jane@ubuntu:~/Code/jetson-containers$ jetson-containers build --name=my_container pytorch
Namespace(packages=['pytorch'], name='my_container', base='', multiple=False, build_flags='', build_args='', package_dirs=[''], list_packages=False, show_packages=False, skip_packages=[''], skip_errors=False, skip_tests=[''], test_only=[''], simulate=False, push='', logs='', verbose=False, no_github_api=False)
-- L4T_VERSION=36.4.0
-- JETPACK_VERSION=5.1
-- CUDA_VERSION=12.6
-- PYTHON_VERSION=3.10
-- LSB_RELEASE=22.04 (jammy)
-- Building containers  ['build-essential', 'pip_cache:cu126', 'cuda:12.6', 'cudnn', 'python', 'numpy', 'cmake', 'onnx', 'pytorch']
-- Building container my_container:r36.4.0-build-essential
DOCKER_BUILDKIT=0 docker build --network=host --tag my_container:r36.4.0-build-essential
--file /home/jane/Code/jetson-containers/packages/build/build-essential/Dockerfile
--build-arg BASE_IMAGE=ubuntu:22.04
/home/jane/Code/jetson-containers/packages/build/build-essential
2>&1 | tee /home/jane/Code/jetson-containers/logs/20241002_212941/build/my_container_r36.4.0-build-essential.txt; exit ${PIPESTATUS[0]}
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
environment-variable.
Sending build context to Docker daemon  19.97kB
Step 1/5 : ARG BASE_IMAGE
Step 2/5 : FROM ${BASE_IMAGE}
---> 981912c48e9a
Step 3/5 : ENV DEBIAN_FRONTEND=noninteractive     LANGUAGE=en_US:en     LANG=en_US.UTF-8     LC_ALL=en_US.UTF-8
---> Using cache
---> bee7dec2ca35
Step 4/5 : RUN set -ex     && apt-get update     && apt-get install -y --no-install-recommends         locales         locales-all         tzdata     && locale-gen en_US $LANG     && update-locale LC_ALL=$LC_ALL LANG=$LANG     && locale         && apt-get install -y --no-install-recommends         build-essential         software-properties-common         apt-transport-https         ca-certificates         lsb-release         pkg-config         gnupg         git         gdb         wget         curl         nano         zip         unzip         time 	   sshpass 	   ssh-client     && apt-get clean     && rm -rf /var/lib/apt/lists/*         && gcc --version     && g++ --version
---> Using cache
---> 5a61c6947b48
Step 5/5 : COPY tarpack /usr/local/bin/
---> Using cache
---> 990e4037e0a8
Successfully built 990e4037e0a8
Successfully tagged my_container:r36.4.0-build-essential
-- Building container my_container:r36.4.0-pip_cache_cu126
DOCKER_BUILDKIT=0 docker build --network=host --tag my_container:r36.4.0-pip_cache_cu126
--file /home/jane/Code/jetson-containers/packages/cuda/cuda/Dockerfile.pip
--build-arg BASE_IMAGE=my_container:r36.4.0-build-essential
--build-arg TAR_INDEX_URL="http://jetson.webredirect.org:8000/jp5/cu126"
--build-arg PIP_INDEX_REPO="http://jetson.webredirect.org/jp5/cu126"
--build-arg PIP_TRUSTED_HOSTS="jetson.webredirect.org"
--build-arg PIP_UPLOAD_REPO="http://localhost/jp5/cu126"
--build-arg PIP_UPLOAD_USER="jp5"
--build-arg PIP_UPLOAD_PASS="none"
--build-arg SCP_UPLOAD_URL="localhost:/dist/jp5/cu126"
--build-arg SCP_UPLOAD_USER="None"
--build-arg SCP_UPLOAD_PASS="None"
/home/jane/Code/jetson-containers/packages/cuda/cuda
2>&1 | tee /home/jane/Code/jetson-containers/logs/20241002_212941/build/my_container_r36.4.0-pip_cache_cu126.txt; exit ${PIPESTATUS[0]}
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
environment-variable.
Sending build context to Docker daemon  48.64kB
Step 1/4 : ARG BASE_IMAGE
Step 2/4 : FROM ${BASE_IMAGE}
---> 990e4037e0a8
Step 3/4 : ARG PIP_INDEX_REPO     PIP_UPLOAD_REPO     PIP_UPLOAD_USER     PIP_UPLOAD_PASS     PIP_TRUSTED_HOSTS     TAR_INDEX_URL     SCP_UPLOAD_URL     SCP_UPLOAD_USER     SCP_UPLOAD_PASS
---> Using cache
---> c2efa4465977
Step 4/4 : ENV TAR_INDEX_URL=${TAR_INDEX_URL}     PIP_INDEX_URL=${PIP_INDEX_REPO}     PIP_TRUSTED_HOST=${PIP_TRUSTED_HOSTS}     TWINE_REPOSITORY_URL=${PIP_UPLOAD_REPO}     TWINE_USERNAME=${PIP_UPLOAD_USER}     TWINE_PASSWORD=${PIP_UPLOAD_PASS}     SCP_UPLOAD_URL=${SCP_UPLOAD_URL}     SCP_UPLOAD_USER=${SCP_UPLOAD_USER}     SCP_UPLOAD_PASS=${SCP_UPLOAD_PASS}
---> Using cache
---> 416cffa73abf
Successfully built 416cffa73abf
Successfully tagged my_container:r36.4.0-pip_cache_cu126
-- Building container my_container:r36.4.0-cuda_12.6
DOCKER_BUILDKIT=0 docker build --network=host --tag my_container:r36.4.0-cuda_12.6
--file /home/jane/Code/jetson-containers/packages/cuda/cuda/Dockerfile
--build-arg BASE_IMAGE=my_container:r36.4.0-pip_cache_cu126
--build-arg CUDA_URL="https://developer.download.nvidia.com/compute/cuda/12.6.1/local_installers/cuda-tegra-repo-ubuntu2204-12-6-local_12.6.1-1_arm64.deb"
--build-arg CUDA_DEB="cuda-tegra-repo-ubuntu2204-12-6-local"
--build-arg CUDA_PACKAGES="cuda-toolkit*"
--build-arg CUDA_ARCH_LIST="87"
--build-arg DISTRO="ubuntu2204"
/home/jane/Code/jetson-containers/packages/cuda/cuda
2>&1 | tee /home/jane/Code/jetson-containers/logs/20241002_212941/build/my_container_r36.4.0-cuda_12.6.txt; exit ${PIPESTATUS[0]}
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
environment-variable.

BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
environment-variable.
Sending build context to Docker daemon  26.62kB
Step 1/7 : ARG BASE_IMAGE
Step 2/7 : FROM ${BASE_IMAGE}
---> 981d4a00a8fa
Step 3/7 : ARG CUDNN_URL
---> Using cache
---> bb6912f89c14
Step 4/7 : ARG CUDNN_DEB
---> Using cache
---> e5627d330ee1
Step 5/7 : ARG CUDNN_PACKAGES
---> Using cache
---> 45da32715ccd
Step 6/7 : RUN echo "Downloading ${CUDNN_DEB}" &&     mkdir /tmp/cudnn && cd /tmp/cudnn &&     wget --quiet --show-progress --progress=bar:force:noscroll ${CUDNN_URL} &&     dpkg -i .deb &&     cp /var/cudnn-local-tegra-repo-/cudnn-local-tegra--keyring.gpg /usr/share/keyrings/ &&     apt-get update &&     apt-cache search cudnn &&     apt-get install -y --no-install-recommends ${CUDNN_PACKAGES} &&     rm -rf /var/lib/apt/lists/ &&     apt-get clean &&     dpkg --list | grep cudnn &&     dpkg -P ${CUDNN_DEB} &&     rm -rf /tmp/cudnn
---> Using cache
---> f7739b053a55
Step 7/7 : RUN cd /usr/src/cudnn_samples_v*/conv_sample/ &&     make -j$(nproc)
---> Using cache
---> 40dbaf57b268
Successfully built 40dbaf57b268
Successfully tagged my_container:r36.4.0-cudnn
-- Testing container my_container:r36.4.0-cudnn (cudnn:9.4/test.sh)
docker run -t --rm --runtime=nvidia --network=host
--volume /home/jane/Code/jetson-containers/packages/cuda/cudnn:/test
--volume /home/jane/Code/jetson-containers/data:/data
--workdir /test
my_container:r36.4.0-cudnn
/bin/bash -c '/bin/bash test.sh'
2>&1 | tee /home/jane/Code/jetson-containers/logs/20241002_212941/test/my_container_r36.4.0-cudnn_test.sh.txt; exit ${PIPESTATUS[0]}
#define CUDNN_MAJOR 9
#define CUDNN_MINOR 4
#define CUDNN_VERSION (CUDNN_MAJOR * 10000 + CUDNN_MINOR * 100 + CUDNN_PATCHLEVEL)
#define CUDNN_MAX_SM_MAJOR_NUMBER 9
#define CUDNN_MAX_SM_MINOR_NUMBER 0
#define CUDNN_MAX_DEVICE_VERSION (CUDNN_MAX_SM_MAJOR_NUMBER * 100 + CUDNN_MAX_SM_MINOR_NUMBER * 10)
Executing: conv_sample
Using format CUDNN_TENSOR_NCHW (for INT8x4 and INT8x32 tests use CUDNN_TENSOR_NCHW_VECT_C)
Testing single precision
====USER DIMENSIONS====
input dims are 1, 32, 4, 4
filter dims are 32, 32, 1, 1
output dims are 1, 32, 4, 4
====PADDING DIMENSIONS====
padded input dims are 1, 32, 4, 4
padded filter dims are 32, 32, 1, 1
padded output dims are 1, 32, 4, 4
Testing conv
^^^^ CUDA : elapsed = 0.000864046 sec,
Test PASSED
Testing half precision (math in single precision)
====USER DIMENSIONS====
input dims are 1, 32, 4, 4
filter dims are 32, 32, 1, 1
output dims are 1, 32, 4, 4
====PADDING DIMENSIONS====
padded input dims are 1, 32, 4, 4
padded filter dims are 32, 32, 1, 1
padded output dims are 1, 32, 4, 4
Testing conv
^^^^ CUDA : elapsed = 0.0177303 sec,
Test PASSED
-- Building container my_container:r36.4.0-python
DOCKER_BUILDKIT=0 docker build --network=host --tag my_container:r36.4.0-python
--file /home/jane/Code/jetson-containers/packages/build/python/Dockerfile
--build-arg BASE_IMAGE=my_container:r36.4.0-cudnn
--build-arg PYTHON_VERSION_ARG="3.10"
/home/jane/Code/jetson-containers/packages/build/python
2>&1 | tee /home/jane/Code/jetson-containers/logs/20241002_212941/build/my_container_r36.4.0-python.txt; exit ${PIPESTATUS[0]}
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
environment-variable.

Building dependency tree...
Reading state information...
python3.10 is already the newest version (3.10.12-1~22.04.6).
python3.10 set to manually installed.
The following additional packages will be installed:
libexpat1-dev libpython3.10-dev zlib1g-dev
The following NEW packages will be installed:
libexpat1-dev libpython3.10-dev python3.10-dev zlib1g-dev
0 upgraded, 4 newly installed, 0 to remove and 0 not upgraded.
Need to get 5,464 kB of archives.
After this operation, 21.0 MB of additional disk space will be used.
Get:1 http://ports.ubuntu.com/ubuntu-ports jammy-updates/main arm64 libexpat1-dev arm64 2.4.7-1ubuntu0.4 [129 kB]
Get:2 http://ports.ubuntu.com/ubuntu-ports jammy-updates/main arm64 zlib1g-dev arm64 1:1.2.11.dfsg-2ubuntu9.2 [163 kB]
Get:3 http://ports.ubuntu.com/ubuntu-ports jammy-updates/main arm64 libpython3.10-dev arm64 3.10.12-1~22.04.6 [4,664 kB]
Get:4 http://ports.ubuntu.com/ubuntu-ports jammy-updates/main arm64 python3.10-dev arm64 3.10.12-1~22.04.6 [508 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 5,464 kB in 3s (1,710 kB/s)
Selecting previously unselected package libexpat1-dev:arm64.
(Reading database ... 29364 files and directories currently installed.)
Preparing to unpack .../libexpat1-dev_2.4.7-1ubuntu0.4_arm64.deb ...
Unpacking libexpat1-dev:arm64 (2.4.7-1ubuntu0.4) ...
Selecting previously unselected package zlib1g-dev:arm64.
Preparing to unpack .../zlib1g-dev_1%3a1.2.11.dfsg-2ubuntu9.2_arm64.deb ...
Unpacking zlib1g-dev:arm64 (1:1.2.11.dfsg-2ubuntu9.2) ...
Selecting previously unselected package libpython3.10-dev:arm64.
Preparing to unpack .../libpython3.10-dev_3.10.12-1~22.04.6_arm64.deb ...
Unpacking libpython3.10-dev:arm64 (3.10.12-1~22.04.6) ...
Selecting previously unselected package python3.10-dev.
Preparing to unpack .../python3.10-dev_3.10.12-1~22.04.6_arm64.deb ...
Unpacking python3.10-dev (3.10.12-1~22.04.6) ...
Setting up libexpat1-dev:arm64 (2.4.7-1ubuntu0.4) ...
Setting up zlib1g-dev:arm64 (1:1.2.11.dfsg-2ubuntu9.2) ...
Setting up libpython3.10-dev:arm64 (3.10.12-1~22.04.6) ...
Setting up python3.10-dev (3.10.12-1~22.04.6) ...
which python3.10
/usr/bin/python3.10
return_code=0
set -e
'[' 0 '!=' 0 ']'
rm -rf /var/lib/apt/lists/auxfiles /var/lib/apt/lists/lock /var/lib/apt/lists/partial /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_jammy-backports_InRelease /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_jammy-backports_main_binary-arm64_Packages.lz4 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_jammy-backports_universe_binary-arm64_Packages.lz4 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_jammy_InRelease /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_jammy_main_binary-arm64_Packages.lz4 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_jammy_multiverse_binary-arm64_Packages.lz4 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_jammy_restricted_binary-arm64_Packages.lz4 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_jammy-security_InRelease /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_jammy-security_main_binary-arm64_Packages.lz4 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_jammy-security_multiverse_binary-arm64_Packages.lz4 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_jammy-security_restricted_binary-arm64_Packages.lz4 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_jammy-security_universe_binary-arm64_Packages.lz4 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_jammy_universe_binary-arm64_Packages.lz4 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_jammy-updates_InRelease /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_jammy-updates_main_binary-arm64_Packages.lz4 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_jammy-updates_multiverse_binary-arm64_Packages.lz4 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_jammy-updates_restricted_binary-arm64_Packages.lz4 /var/lib/apt/lists/ports.ubuntu.com_ubuntu-ports_dists_jammy-updates_universe_binary-arm64_Packages.lz4
apt-get clean
curl -sS https://bootstrap.pypa.io/get-pip.py
python3.10
Looking in indexes: http://jetson.webredirect.org/jp5/cu126
ERROR: Could not find a version that satisfies the requirement pip (from versions: none)
ERROR: No matching distribution found for pip
curl -sS https://bootstrap.pypa.io/pip/3.6/get-pip.py
python3.6
/tmp/install_python.sh: line 27: python3.6: command not found
curl: (23) Failure writing output to destination
The command '/bin/sh -c /tmp/install_python.sh' returned a non-zero code: 127
Traceback (most recent call last):
File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/home/jane/Code/jetson-containers/jetson_containers/build.py", line 112, in
build_container(args.name, args.packages, args.base, args.build_flags, args.build_args, args.simulate, args.skip_tests, args.test_only, args.push, args.no_github_api)
File "/home/jane/Code/jetson-containers/jetson_containers/container.py", line 147, in build_container
status = subprocess.run(cmd.replace(NEWLINE, ' '), executable='/bin/bash', shell=True, check=True)
File "/usr/lib/python3.10/subprocess.py", line 526, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command 'DOCKER_BUILDKIT=0 docker build --network=host --tag my_container:r36.4.0-python --file /home/jane/Code/jetson-containers/packages/build/python/Dockerfile --build-arg BASE_IMAGE=my_container:r36.4.0-cudnn --build-arg PYTHON_VERSION_ARG="3.10" /home/jane/Code/jetson-containers/packages/build/python 2>&1 | tee /home/jane/Code/jetson-containers/logs/20241002_212941/build/my_container_r36.4.0-python.txt; exit ${PIPESTATUS[0]}' returned non-zero exit status 127.

It's failing in packages/build/python/install.sh in the "install pip" part of the script. I tried hacking around and fixing the python issue, and got a bit further, but I don't know how the jetson+python setup works well enough to fix it for sure so I thought I'd ask here.
Is this jetson-containers tool still the recommended way to build this stuff? I'm using current-gen hardware, with the latest SDK (afaict), so it seems like it should work? Happy to be pointed in other directions, though - I must confess I found it a bit confusing to know what's the latest/official(ish) way to do this (pull images from nvcr.io? build them with this tool? follow these docs, or perhaps others on the NVIDIA website?).
One other thing I noticed: jtop (and apt show nvidia-jetpack)reports that the board is running JetPack 6.1, but the jetson-containers output detects it as JETPACK_VERSION=5.1. While that doesn't seem to be the issue that's derailing me, it doesn't seem good.
A bit more output from jtop if it's helpful.

--------------------- PLATFORM -------------------------
Machine: aarch64
System: Linux
Distribution: Ubuntu 22.04 Jammy Jellyfish
Release: 5.15.148-tegra
Python: 3.10.12
-------------------- JETSON RAW OUTPUT -----------------
------------------
Path: /etc/nv_tegra_release
# R36 (release), REVISION: 4.0, GCID: 37537400, BOARD: generic, EABI: aarch64, DATE: Fri Sep 13 04:36:44 UTC 2024
------------------
Path: /sys/firmware/devicetree/base/model
NVIDIA Jetson AGX Orin Developer Kit
------------------
Path: /proc/device-tree/nvidia,boardids
No such file or directory
------------------
Path: /proc/device-tree/compatible
nvidia,p3737-0000+p3701-0005nvidia,p3701-0005nvidia,tegra234
------------------
Path: /proc/device-tree/nvidia,dtsfilename
No such file or directory
------------------

Hi @benswift, I had updated jetson-containers for initial JetPack 6.1 support a few days ago - can you try doing a git pull in your jetson-containers repo?   Then when you should start seeing jetson-containers to start reporting your board correctly as JetPack 6.1, and the python build should go through.  It takes a while to rebuild everything here, but PyTorch and Transformers are fine as should be diffusers.

Thanks for the quick response. I am using the latest (53eb0e69d5a9931a21044a52299609bd16f0f6d5), yeah.

One more update, just in case others are playing along at home: I had to get pkginfo from pypi.org, not the Jetson PA, so you'll need to change line 43 of the python install.sh to:
python3 -m pip install --upgrade pip pkginfo --index-url https://pypi.org/simple

(note the extra pkginfo in the middle there)

Ok, by commenting out the optimum package I was able to build my jetson-container (which includes both diffusers and transformers). I pushed up the few small tweaks required here - @dusty-nv I'm happy to turn it into a proper PR if you want, just let me know.

Sorry @benswift, that change to l4t_version.py I thought I had merge into master, but it was still hiding in dev - that has now been merged, and you can see the recognition of JP 6.1 here:   https://github.com/dusty-nv/jetson-containers/blob/d1a10b66220a3322d39772027516e08995a919b1/jetson_containers/l4t_version.py#L87
@tokk-nv had reported a similar thing about optimum , although neither myself or others had seen that behavior.  Thanks for reporting it - will remove or fix it 👍

@benswift : thanks for this - i have been struggling with the same since yesterday. I am though running into issues even after making the changes that you have suggested.
@dusty-nv : i had built a container for vlm using ros iron (nano_llm:iron-r36.3.0-cu122) please refer https://github.com/dusty-nv/jetson-containers/issues/622. This has been running without issues on an OrinNX whch is still on jetpack version 6.0.
I tried to run the same container on an Orin AGX Developer Kit which is running on 36.4.0 and Jetpack 6.1, and I started getting errors such as these
`Fetching 13 files: 100%|█████████████████████| 13/13 [00:00<00:00, 53932.69it/s]
Fetching 17 files: 100%|█████████████████████| 17/17 [00:00<00:00, 44788.42it/s]
['/data/models/mlc/dist/VILA1.5-3b/ctx256/VILA1.5-3b-q4f16_ft/mlc-chat-config.json', '/data/models/mlc/dist/VILA1.5-3b/ctx256/VILA1.5-3b-q4f16_ft/params/mlc-chat-config.json']
Fatal Python error: Segmentation fault
Current thread 0x0000ffff99cc8da0 (most recent call first):
Garbage-collecting
File "/usr/local/lib/python3.10/dist-packages/torch/nn/parallel/distributed.py", line 1222 in DistributedDataParallel
File "/usr/local/lib/python3.10/dist-packages/torch/nn/parallel/distributed.py", line 321 in
File "", line 241 in _call_with_frames_removed
File "", line 883 in exec_module
File "", line 688 in _load_unlocked
File "", line 1006 in _find_and_load_unlocked
File "", line 1027 in _find_and_load
File "/usr/local/lib/python3.10/dist-packages/torch/nn/parallel/init.py",
`
I then tried to build a new container with cuda 12.6 but the build process stops when building the OpenCV container. I am getting the below errors.
It appears to be the same python related ones - look forward to your feedback
`traceback (most recent call last):
File "/tmp/pip-build-env-1ci8_g7q/overlay/local/lib/python3.10/dist-packages/skbuild/setuptools_wrap.py", line 668, in setup
cmkr.make(make_args, install_target=cmake_install_target, env=env)
File "/tmp/pip-build-env-1ci8_g7q/overlay/local/lib/python3.10/dist-packages/skbuild/cmaker.py", line 696, in make
self.make_impl(clargs=clargs, config=config, source_dir=source_dir, install_target=install_target, env=env)
File "/tmp/pip-build-env-1ci8_g7q/overlay/local/lib/python3.10/dist-packages/skbuild/cmaker.py", line 741, in make_impl
raise SKBuildError(msg)
An error occurred while building with CMake.
Command:
/tmp/pip-build-env-1ci8_g7q/overlay/local/lib/python3.10/dist-packages/cmake/data/bin/cmake --build . --target install --config RELEASE --
Install target:
install
Source directory:
/opt/opencv-python
Working directory:
/opt/opencv-python/_skbuild/linux-aarch64-3.10/cmake-build
Please check the install target is valid and see CMake's output for more information.
error: subprocess-exited-with-error
× Building wheel for opencv-contrib-python (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
full command: /usr/bin/python3.10 /usr/local/lib/python3.10/dist-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py build_wheel /tmp/tmpi9uk2_pz
cwd: /opt/opencv-python
Building wheel for opencv-contrib-python (pyproject.toml): finished with status 'error'
ERROR: Failed building wheel for opencv-contrib-python
Failed to build opencv-contrib-python
ERROR: Failed to build one or more wheels
The command '/bin/sh -c cd /tmp/opencv && ./install.sh || ./build.sh' returned a non-zero code: 1
Traceback (most recent call last):
File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
`

@benswift as of yesterday morning opencv was building, and the wheels are up on my pip server, which the containers should install instead of needing to build them (like it appears to have in your case).  I'm not sure what the actual compilation error is from that log, but will try again (I was also trying to build ROS for that)
MLC and NanoLLM I haven't gotten to yet, NanoLLM has a lot of dependencies that am in the process of working through.  Does PyTorch in the r36.3 containers still work for you on r36.4, or is it just MLC?

@dusty-nv ah, I didn't think to look in the dev branch. Do you have any guidance (or are there any docs somewhere) on which packages it might be ok to pull from "regular" pypi.org and which ones have to come from the jetson.webredirect.org one?
@Fibo27 I actually had the exact same error trying to build the opencv container. I ended up just pip installing opencv-python-headless in my container (which uses the jetson-containers-built container as a base), since that's all I need.

@dusty-nv whoops - we were writing comments at the same time.
I'll re-pull the latest and check the opencv thing ASAP - will be able to do that in a couple of hours - and report back.

@benswift if you browse the previously-built wheels for CUDA 12.2, those are the ones that end up getting specially-built with CUDA support.   That pip server blacklists those packages from pypi, but mirrors others (so for example you can pip install transformers from it, and it will pull transformers from PyPi but CUDA-enabled pytorch from mine)

Ok, I just tried to build opencv from the latest master (d1a10b66220a3322d39772027516e08995a919b1) and had the same twine/pkginfo version error I was getting before (and fixed in my branch by getting pkginfo from the main pypi. Log attached.
twine-error.log

testing:

try to do first:
sudo docker system prune -a

and rebuild it

sudo docker system prune -a

@benswift don't do that unless you actually want to remove all your docker images :rofl:
You can disable the build cache though, I think that's what Johnny means.  It's weird you get that twine error, will cherry-pick your patch 👍

sudo docker system prune -a

@benswift don't do that unless you actually want to remove all your docker images 🤣
You can disable the build cache though, I think that's what Johnny means. It's weird you get that twine error, will cherry-pick your patch 👍

yeap jajajaja sometimes I have conflicts, so I clean everything and rebuild it

sudo docker system prune -a

@benswift don't do that unless you actually want to remove all your docker images 🤣
You can disable the build cache though, I think that's what Johnny means. It's weird you get that twine error, will cherry-pick your patch 👍

only the "error" that I obtain is this one:
https://github.com/dusty-nv/jetson-containers/pull/656/files

@dusty-nv one more update in case you're wondering.
I can build the xformers container, but the torch_tensorrt (which I'd like to use, but isn't essential for me right now) one doesn't work. packages/pytorch/torch_tensorrt/Dockerfile:18 copies in a WORKSPACE.jp46 file which doesn't feel right given that I'm on jp61. And then it bombs out a bit later trying to stat a file that's not there. The intermediate tensorrt container does build, though. Logs below.
-- Building container stjet:r36.4.0-tensorrt

DOCKER_BUILDKIT=0 docker build --network=host --tag stjet:r36.4.0-tensorrt \
--file /home/jane/Code/jetson-containers/packages/tensorrt/Dockerfile.tar \
--build-arg BASE_IMAGE=stjet:r36.4.0-xformers \
--build-arg TENSORRT_URL="https://developer.nvidia.com/downloads/compute/machine-learning/tensorrt/10.4.0/tars/TensorRT-<IP_ADDRESS>.l4t.aarch64-gnu.cuda-12.6.tar.gz" \
/home/jane/Code/jetson-containers/packages/tensorrt \
2>&1 | tee /home/jane/Code/jetson-containers/logs/20241004_165713/build/stjet_r36.4.0-tensorrt.txt; exit ${PIPESTATUS[0]}

DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
            environment-variable.

Sending build context to Docker daemon  26.11kB
Step 1/5 : ARG BASE_IMAGE
Step 2/5 : FROM ${BASE_IMAGE}
 ---> a7717d6dc2f7
Step 3/5 : ARG TENSORRT_URL
 ---> Using cache
 ---> 20fdd3940836
Step 4/5 : RUN set -ex &&     echo "Downloading ${TENSORRT_URL}" &&     mkdir -p /tmp/tensorrt &&     cd /tmp/tensorrt &&     wget --quiet --show-progress --progress=bar:force:noscroll ${TENSORRT_URL} -O TensorRT.tar &&     tar -xvf TensorRT.tar -C /usr/src &&     mv /usr/src/TensorRT-* /usr/src/tensorrt
 ---> Using cache
 ---> ec80a2b129b7
Step 5/5 : RUN cd /tmp/tensorrt &&     cp -r /usr/src/tensorrt/lib/* /usr/lib/$(uname -m)-linux-gnu/ &&     cp -r /usr/src/tensorrt/include/* /usr/include/$(uname -m)-linux-gnu/ &&     PY_VERSION=$(python3 -c 'import sys; print(f"{sys.version_info.major}{sys.version_info.minor}")') &&     pip3 install --verbose --no-cache-dir /usr/src/tensorrt/python/tensorrt-*-cp${PY_VERSION}-*.whl &&     rm -rf /tmp/tensorrt
 ---> Using cache
 ---> 5db9ddd5e7d0
Successfully built 5db9ddd5e7d0
Successfully tagged stjet:r36.4.0-tensorrt
-- Testing container stjet:r36.4.0-tensorrt (tensorrt:10.4/test.sh)

docker run -t --rm --runtime=nvidia --network=host \
--volume /home/jane/Code/jetson-containers/packages/tensorrt:/test \
--volume /home/jane/Code/jetson-containers/data:/data \
--workdir /test \
stjet:r36.4.0-tensorrt \
/bin/bash -c '/bin/bash test.sh' \
2>&1 | tee /home/jane/Code/jetson-containers/logs/20241004_165713/test/stjet_r36.4.0-tensorrt_test.sh.txt; exit ${PIPESTATUS[0]}

&&&& RUNNING TensorRT.trtexec [TensorRT v100400] [b26] # /usr/src/tensorrt/bin/trtexec --help
=== Model Options ===
  --onnx=<file>               ONNX model

=== Build Options ===
  --minShapes=spec                   Build with dynamic shapes using a profile with the min shapes provided
  --optShapes=spec                   Build with dynamic shapes using a profile with the opt shapes provided
  --maxShapes=spec                   Build with dynamic shapes using a profile with the max shapes provided
  --minShapesCalib=spec              Calibrate with dynamic shapes using a profile with the min shapes provided
  --optShapesCalib=spec              Calibrate with dynamic shapes using a profile with the opt shapes provided
  --maxShapesCalib=spec              Calibrate with dynamic shapes using a profile with the max shapes provided
                                     Note: All three of min, opt and max shapes must be supplied.
                                           However, if only opt shapes is supplied then it will be expanded so
                                           that min shapes and max shapes are set to the same values as opt shapes.
                                           Input names can be wrapped with escaped single quotes (ex: 'Input:0').
                                     Example input shapes spec: input0:1x3x256x256,input1:1x3x128x128
                                     For scalars (0-D shapes), use input0:scalar or simply input0: with nothing after the colon.
                                     Each input shape is supplied as a key-value pair where key is the input name and
                                     value is the dimensions (including the batch dimension) to be used for that input.
                                     Each key-value pair has the key and value separated using a colon (:).
                                     Multiple input shapes can be provided via comma-separated key-value pairs, and each input name can
                                     contain at most one wildcard ('*') character.
  --inputIOFormats=spec              Type and format of each of the input tensors (default = all inputs in fp32:chw)
                                     See --outputIOFormats help for the grammar of type and format list.
                                     Note: If this option is specified, please set comma-separated types and formats for all
                                           inputs following the same order as network inputs ID (even if only one input
                                           needs specifying IO format) or set the type and format once for broadcasting.
  --outputIOFormats=spec             Type and format of each of the output tensors (default = all outputs in fp32:chw)
                                     Note: If this option is specified, please set comma-separated types and formats for all
                                           outputs following the same order as network outputs ID (even if only one output
                                           needs specifying IO format) or set the type and format once for broadcasting.
                                     IO Formats: spec  ::= IOfmt[","spec]
                                                 IOfmt ::= type:fmt
                                               type  ::= "fp32"|"fp16"|"bf16"|"int32"|"int64"|"int8"|"uint8"|"bool"
                                               fmt   ::= ("chw"|"chw2"|"chw4"|"hwc8"|"chw16"|"chw32"|"dhwc8"|
                                                          "cdhw32"|"hwc"|"dla_linear"|"dla_hwc4")["+"fmt]
  --memPoolSize=poolspec             Specify the size constraints of the designated memory pool(s)
                                     Supports the following base-2 suffixes: B (Bytes), G (Gibibytes), K (Kibibytes), M (Mebibytes).
                                     If none of suffixes is appended, the defualt unit is in MiB.
                                     Note: Also accepts decimal sizes, e.g. 0.25M. Will be rounded down to the nearest integer bytes.
                                     In particular, for dlaSRAM the bytes will be rounded down to the nearest power of 2.
                                   Pool constraint: poolspec ::= poolfmt[","poolspec]
                                                      poolfmt ::= pool:size
                                                    pool ::= "workspace"|"dlaSRAM"|"dlaLocalDRAM"|"dlaGlobalDRAM"|"tacticSharedMem"
  --profilingVerbosity=mode          Specify profiling verbosity. mode ::= layer_names_only|detailed|none (default = layer_names_only).
                                     Please only assign once.
  --avgTiming=M                      Set the number of times averaged in each iteration for kernel selection (default = 8)
  --refit                            Mark the engine as refittable. This will allow the inspection of refittable layers
                                     and weights within the engine.
  --stripWeights                     Strip weights from plan. This flag works with either refit or refit with identical weights. Default
                                     to latter, but you can switch to the former by enabling both --stripWeights and --refit at the same
                                     time.
  --stripAllWeights                  Alias for combining the --refit and --stripWeights options. It marks all weights as refittable,
                                     disregarding any performance impact. Additionally, it strips all refittable weights after the
                                     engine is built.
  --weightless                       [Deprecated] this knob has been deprecated. Please use --stripWeights
  --versionCompatible, --vc          Mark the engine as version compatible. This allows the engine to be used with newer versions
                                     of TensorRT on the same host OS, as well as TensorRT's dispatch and lean runtimes.
  --pluginInstanceNorm, --pi         Set `kNATIVE_INSTANCENORM` to false in the ONNX parser. This will cause the ONNX parser to use
                                     a plugin InstanceNorm implementation over the native implementation when parsing.
  --useRuntime=runtime               TensorRT runtime to execute engine. "lean" and "dispatch" require loading VC engine and do
                                     not support building an engine.
                                           runtime::= "full"|"lean"|"dispatch"
  --leanDLLPath=<file>               External lean runtime DLL to use in version compatiable mode.
  --excludeLeanRuntime               When --versionCompatible is enabled, this flag indicates that the generated engine should
                                     not include an embedded lean runtime. If this is set, the user must explicitly specify a
                                     valid lean runtime to use when loading the engine.
  --sparsity=spec                    Control sparsity (default = disabled).
                                   Sparsity: spec ::= "disable", "enable", "force"
                                     Note: Description about each of these options is as below
                                           disable = do not enable sparse tactics in the builder (this is the default)
                                           enable  = enable sparse tactics in the builder (but these tactics will only be
                                                     considered if the weights have the right sparsity pattern)
                                           force   = enable sparse tactics in the builder and force-overwrite the weights to have
                                                     a sparsity pattern (even if you loaded a model yourself)
                                                     [Deprecated] this knob has been deprecated.
                                                     Please use <polygraphy surgeon prune> to rewrite the weights.
  --noTF32                           Disable tf32 precision (default is to enable tf32, in addition to fp32)
  --fp16                             Enable fp16 precision, in addition to fp32 (default = disabled)
  --bf16                             Enable bf16 precision, in addition to fp32 (default = disabled)
  --int8                             Enable int8 precision, in addition to fp32 (default = disabled)
  --fp8                              Enable fp8 precision, in addition to fp32 (default = disabled)
  --int4                             Enable int4 precision, in addition to fp32 (default = disabled)
  --best                             Enable all precisions to achieve the best performance (default = disabled)
  --stronglyTyped                    Create a strongly typed network. (default = disabled)
  --directIO                         Avoid reformatting at network boundaries. (default = disabled)
  --precisionConstraints=spec        Control precision constraint setting. (default = none)
                                       Precision Constraints: spec ::= "none" | "obey" | "prefer"
                                         none = no constraints
                                         prefer = meet precision constraints set by --layerPrecisions/--layerOutputTypes if possible
                                         obey = meet precision constraints set by --layerPrecisions/--layerOutputTypes or fail
                                                otherwise
  --layerPrecisions=spec             Control per-layer precision constraints. Effective only when precisionConstraints is set to
                                   "obey" or "prefer". (default = none)
                                   The specs are read left-to-right, and later ones override earlier ones. Each layer name can
                                     contain at most one wildcard ('*') character.
                                   Per-layer precision spec ::= layerPrecision[","spec]
                                                       layerPrecision ::= layerName":"precision
                                                       precision ::= "fp32"|"fp16"|"bf16"|"int32"|"int8"
  --layerOutputTypes=spec            Control per-layer output type constraints. Effective only when precisionConstraints is set to
                                   "obey" or "prefer". (default = none
                                   The specs are read left-to-right, and later ones override earlier ones. Each layer name can
                                     contain at most one wildcard ('*') character. If a layer has more than
                                   one output, then multiple types separated by "+" can be provided for this layer.
                                   Per-layer output type spec ::= layerOutputTypes[","spec]
                                                         layerOutputTypes ::= layerName":"type
                                                         type ::= "fp32"|"fp16"|"bf16"|"int32"|"int8"["+"type]
  --layerDeviceTypes=spec            Specify layer-specific device type.
                                     The specs are read left-to-right, and later ones override earlier ones. If a layer does not have
                                     a device type specified, the layer will opt for the default device type.
                                   Per-layer device type spec ::= layerDeviceTypePair[","spec]
                                                         layerDeviceTypePair ::= layerName":"deviceType
                                                           deviceType ::= "GPU"|"DLA"
  --calib=<file>                     Read INT8 calibration cache file
  --safe                             Enable build safety certified engine, if DLA is enable, --buildDLAStandalone will be specified
                                     automatically (default = disabled)
  --buildDLAStandalone               Enable build DLA standalone loadable which can be loaded by cuDLA, when this option is enabled,
                                     --allowGPUFallback is disallowed and --skipInference is enabled by default. Additionally,
                                     specifying --inputIOFormats and --outputIOFormats restricts I/O data type and memory layout
                                     (default = disabled)
  --allowGPUFallback                 When DLA is enabled, allow GPU fallback for unsupported layers (default = disabled)
  --restricted                       Enable safety scope checking with kSAFETY_SCOPE build flag
  --saveEngine=<file>                Save the serialized engine
  --loadEngine=<file>                Load a serialized engine
  --getPlanVersionOnly               Print TensorRT version when loaded plan was created. Works without deserialization of the plan.
                                     Use together with --loadEngine. Supported only for engines created with 8.6 and forward.
  --tacticSources=tactics            Specify the tactics to be used by adding (+) or removing (-) tactics from the default
                                     tactic sources (default = all available tactics).
                                     Note: Currently only cuDNN, cuBLAS, cuBLAS-LT, and edge mask convolutions are listed as optional
                                           tactics.
                                   Tactic Sources: tactics ::= [","tactic]
                                                     tactic  ::= (+|-)lib
                                                   lib     ::= "CUBLAS"|"CUBLAS_LT"|"CUDNN"|"EDGE_MASK_CONVOLUTIONS"
                                                               |"JIT_CONVOLUTIONS"
                                     For example, to disable cudnn and enable cublas: --tacticSources=-CUDNN,+CUBLAS
  --noBuilderCache                   Disable timing cache in builder (default is to enable timing cache)
  --noCompilationCache               Disable Compilation cache in builder, and the cache is part of timing cache (default is to enable compilation cache)
  --errorOnTimingCacheMiss           Emit error when a tactic being timed is not present in the timing cache (default = false)
  --timingCacheFile=<file>           Save/load the serialized global timing cache
  --preview=features                 Specify preview feature to be used by adding (+) or removing (-) preview features from the default
                                   Preview Features: features ::= [","feature]
                                                       feature  ::= (+|-)flag
                                                     flag     ::= "aliasedPluginIO1003"
                                                                  |"profileSharing0806"
  --builderOptimizationLevel         Set the builder optimization level. (default is 3)
                                     Higher level allows TensorRT to spend more building time for more optimization options.
                                     Valid values include integers from 0 to the maximum optimization level, which is currently 5.
  --maxTactics                       Set the maximum number of tactics to time when there is a choice of tactics. (default is -1)
                                     Larger number of tactics allow TensorRT to spend more building time on evaluating tactics.
                                     Default value -1 means TensorRT can decide the number of tactics based on its own heuristic.
  --hardwareCompatibilityLevel=mode  Make the engine file compatible with other GPU architectures. (default = none)
                                   Hardware Compatibility Level: mode ::= "none" | "ampere+"
                                         none = no compatibility
                                         ampere+ = compatible with Ampere and newer GPUs
  --runtimePlatform=platform         Set the target platform for runtime execution. (default = SameAsBuild)
                                     When this option is enabled, --skipInference is enabled by default.
                                   RuntimePlatfrom: platform ::= "SameAsBuild" | "WindowsAMD64"
                                         SameAsBuild = no requirement for cross-platform compatibility.
                                         WindowsAMD64 = set the target platform for engine execution as Windows AMD64 system
  --tempdir=<dir>                    Overrides the default temporary directory TensorRT will use when creating temporary files.
                                     See IRuntime::setTemporaryDirectory API documentation for more information.
  --tempfileControls=controls        Controls what TensorRT is allowed to use when creating temporary executable files.
                                     Should be a comma-separated list with entries in the format (in_memory|temporary):(allow|deny).
                                     in_memory: Controls whether TensorRT is allowed to create temporary in-memory executable files.
                                     temporary: Controls whether TensorRT is allowed to create temporary executable files in the
                                                filesystem (in the directory given by --tempdir).
                                     For example, to allow in-memory files and disallow temporary files:
                                         --tempfileControls=in_memory:allow,temporary:deny
                                     If a flag is unspecified, the default behavior is "allow".
  --maxAuxStreams=N                  Set maximum number of auxiliary streams per inference stream that TRT is allowed to use to run
                                     kernels in parallel if the network contains ops that can run in parallel, with the cost of more
                                     memory usage. Set this to 0 for optimal memory usage. (default = using heuristics)
  --profile                          Build with dynamic shapes using a profile with the min/max/opt shapes provided. Can be specified
                                         multiple times to create multiple profiles with contiguous index.
                                     (ex: --profile=0 --minShapes=<spec> --optShapes=<spec> --maxShapes=<spec> --profile=1 ...)
  --calibProfile                     Select the optimization profile to calibrate by index. (default = 0)
  --allowWeightStreaming             Enable a weight streaming engine. Must be specified with --stronglyTyped. TensorRT will disable
                                     weight streaming at runtime unless --weightStreamingBudget is specified.
  --markDebug                        Specify list of names of tensors to be marked as debug tensors. Separate names with a comma

=== Inference Options ===
  --shapes=spec               Set input shapes for dynamic shapes inference inputs.
                              Note: Input names can be wrapped with escaped single quotes (ex: 'Input:0').
                              Example input shapes spec: input0:1x3x256x256, input1:1x3x128x128
                              For scalars (0-D shapes), use input0:scalar or simply input0: with nothing after the colon.
                              Each input shape is supplied as a key-value pair where key is the input name and
                              value is the dimensions (including the batch dimension) to be used for that input.
                              Each key-value pair has the key and value separated using a colon (:).
                              Multiple input shapes can be provided via comma-separated key-value pairs, and each input
                              name can contain at most one wildcard ('*') character.
  --loadInputs=spec           Load input values from files (default = generate random inputs). Input names can be wrapped with single quotes (ex: 'Input:0')
                            Input values spec ::= Ival[","spec]
                                         Ival ::= name":"file
                              Consult the README for more information on generating files for custom inputs.
  --iterations=N              Run at least N inference iterations (default = 10)
  --warmUp=N                  Run for N milliseconds to warmup before measuring performance (default = 200)
  --duration=N                Run performance measurements for at least N seconds wallclock time (default = 3)
                              If -1 is specified, inference will keep running unless stopped manually
  --sleepTime=N               Delay inference start with a gap of N milliseconds between launch and compute (default = 0)
  --idleTime=N                Sleep N milliseconds between two continuous iterations(default = 0)
  --infStreams=N              Instantiate N execution contexts to run inference concurrently (default = 1)
  --exposeDMA                 Serialize DMA transfers to and from device (default = disabled).
  --noDataTransfers           Disable DMA transfers to and from device (default = enabled).
  --useManagedMemory          Use managed memory instead of separate host and device allocations (default = disabled).
  --useSpinWait               Actively synchronize on GPU events. This option may decrease synchronization time but increase CPU usage and power (default = disabled)
  --threads                   Enable multithreading to drive engines with independent threads or speed up refitting (default = disabled)
  --useCudaGraph              Use CUDA graph to capture engine execution and then launch inference (default = disabled).
                              This flag may be ignored if the graph capture fails.
  --timeDeserialize           Time the amount of time it takes to deserialize the network and exit.
  --timeRefit                 Time the amount of time it takes to refit the engine before inference.
  --separateProfileRun        Do not attach the profiler in the benchmark run; if profiling is enabled, a second profile run will be executed (default = disabled)
  --skipInference             Exit after the engine has been built and skip inference perf measurement (default = disabled)
  --persistentCacheRatio      Set the persistentCacheLimit in ratio, 0.5 represent half of max persistent L2 size (default = 0)
  --useProfile                Set the optimization profile for the inference context (default = 0 ).
  --allocationStrategy=spec   Specify how the internal device memory for inference is allocated.
                            Strategy: spec ::= "static", "profile", "runtime"
                                  static = Allocate device memory based on max size across all profiles.
                                  profile = Allocate device memory based on max size of the current profile.
                                  runtime = Allocate device memory based on the actual input shapes.
  --saveDebugTensors          Specify list of names of tensors to turn on the debug state
                              and filename to save raw outputs to.
                              These tensors must be specified as debug tensors during build time.
                            Input values spec ::= Ival[","spec]
                                         Ival ::= name":"file
  --weightStreamingBudget     Set the maximum amount of GPU memory TensorRT is allowed to use for weights.
                              It can take on the following values:
                                -2: (default) Disable weight streaming at runtime.
                                -1: TensorRT will automatically decide the budget.
                                 0-100%: Percentage of streamable weights that reside on the GPU.
                                         0% saves the most memory but will have the worst performance.
                                         Requires the % character.
                                >=0B: The exact amount of streamable weights that reside on the GPU. Supports the
                                     following base-2 suffixes: B (Bytes), G (Gibibytes), K (Kibibytes), M (Mebibytes).

=== Reporting Options ===
  --verbose                   Use verbose logging (default = false)
  --avgRuns=N                 Report performance measurements averaged over N consecutive iterations (default = 10)
  --percentile=P1,P2,P3,...   Report performance for the P1,P2,P3,... percentages (0<=P_i<=100, 0 representing max perf, and 100 representing min perf; (default = 90,95,99%)
  --dumpRefit                 Print the refittable layers and weights from a refittable engine
  --dumpOutput                Print the output tensor(s) of the last inference iteration (default = disabled)
  --dumpRawBindingsToFile     Print the input/output tensor(s) of the last inference iteration to file(default = disabled)
  --dumpProfile               Print profile information per layer (default = disabled)
  --dumpLayerInfo             Print layer information of the engine to console (default = disabled)
  --dumpOptimizationProfile   Print the optimization profile(s) information (default = disabled)
  --exportTimes=<file>        Write the timing results in a json file (default = disabled)
  --exportOutput=<file>       Write the output tensors to a json file (default = disabled)
  --exportProfile=<file>      Write the profile information per layer in a json file (default = disabled)
  --exportLayerInfo=<file>    Write the layer information of the engine in a json file (default = disabled)

=== System Options ===
  --device=N                  Select cuda device N (default = 0)
  --useDLACore=N              Select DLA core N for layers that support DLA (default = none)
  --staticPlugins             Plugin library (.so) to load statically (can be specified multiple times)
  --dynamicPlugins            Plugin library (.so) to load dynamically and may be serialized with the engine if they are included in --setPluginsToSerialize (can be specified multiple times)
  --setPluginsToSerialize     Plugin library (.so) to be serialized with the engine (can be specified multiple times)
  --ignoreParsedPluginLibs    By default, when building a version-compatible engine, plugin libraries specified by the ONNX parser
                              are implicitly serialized with the engine (unless --excludeLeanRuntime is specified) and loaded dynamically.
                              Enable this flag to ignore these plugin libraries instead.

=== Help ===
  --help, -h                  Print this message
TensorRT version: 10.4.0
-- Building container stjet:r36.4.0-bazel

DOCKER_BUILDKIT=0 docker build --network=host --tag stjet:r36.4.0-bazel \
--file /home/jane/Code/jetson-containers/packages/build/bazel/Dockerfile \
--build-arg BASE_IMAGE=stjet:r36.4.0-tensorrt \
/home/jane/Code/jetson-containers/packages/build/bazel \
2>&1 | tee /home/jane/Code/jetson-containers/logs/20241004_165713/build/stjet_r36.4.0-bazel.txt; exit ${PIPESTATUS[0]}

DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
            environment-variable.

Sending build context to Docker daemon  9.216kB
Step 1/4 : ARG BASE_IMAGE
Step 2/4 : FROM ${BASE_IMAGE}
 ---> 5db9ddd5e7d0
Step 3/4 : RUN BAZELISK_RELEASE=$(wget -qO- https://api.github.com/repos/bazelbuild/bazelisk/releases/latest | grep -Po '"tag_name": "\K.*?(?=")') &&     BAZELISK_URL="https://github.com/bazelbuild/bazelisk/releases/download/$BAZELISK_RELEASE/bazelisk-linux-arm64" &&     echo "BAZELISK_RELEASE=$BAZELISK_RELEASE" && echo "BAZELISK_URL=$BAZELISK_URL" &&     wget --quiet --show-progress --progress=bar:force:noscroll --no-check-certificate $BAZELISK_URL -O /usr/local/bin/bazel &&     chmod +x /usr/local/bin/bazel
 ---> Using cache
 ---> 1a7d84778b05
Step 4/4 : RUN bazel --version
 ---> Using cache
 ---> fd849b026ff6
Successfully built fd849b026ff6
Successfully tagged stjet:r36.4.0-bazel
-- Testing container stjet:r36.4.0-bazel (bazel/test.sh)

docker run -t --rm --runtime=nvidia --network=host \
--volume /home/jane/Code/jetson-containers/packages/build/bazel:/test \
--volume /home/jane/Code/jetson-containers/data:/data \
--workdir /test \
stjet:r36.4.0-bazel \
/bin/bash -c '/bin/bash test.sh' \
2>&1 | tee /home/jane/Code/jetson-containers/logs/20241004_165713/test/stjet_r36.4.0-bazel_test.sh.txt; exit ${PIPESTATUS[0]}

testing bazel...
bazel 7.3.2
bazel OK
-- Building container stjet:r36.4.0-torch_tensorrt

DOCKER_BUILDKIT=0 docker build --network=host --tag stjet:r36.4.0-torch_tensorrt \
--file /home/jane/Code/jetson-containers/packages/pytorch/torch_tensorrt/Dockerfile \
--build-arg BASE_IMAGE=stjet:r36.4.0-bazel \
--build-arg PYTHON_VERSION="3.10" \
--build-arg JETPACK_MAJOR="6" \
--build-arg JETPACK_MINOR="0" \
--build-arg TORCH_TRT_VERSION="v1.4.0" \
/home/jane/Code/jetson-containers/packages/pytorch/torch_tensorrt \
2>&1 | tee /home/jane/Code/jetson-containers/logs/20241004_165713/build/stjet_r36.4.0-torch_tensorrt.txt; exit ${PIPESTATUS[0]}

DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
            environment-variable.

Sending build context to Docker daemon  20.99kB
Step 1/12 : ARG BASE_IMAGE
Step 2/12 : FROM ${BASE_IMAGE}
 ---> fd849b026ff6
Step 3/12 : ARG JETPACK_MAJOR
 ---> Using cache
 ---> fd0263c5ee55
Step 4/12 : ARG JETPACK_MINOR
 ---> Using cache
 ---> 6ce446a843ed
Step 5/12 : ARG PYTHON_VERSION
 ---> Using cache
 ---> 7e77ef79fac2
Step 6/12 : ARG TORCH_TRT_VERSION=main
 ---> Using cache
 ---> 35730885e9d8
Step 7/12 : COPY WORKSPACE.jp46 /tmp
 ---> Using cache
 ---> 918f357a4b67
Step 8/12 : RUN git clone --branch=${TORCH_TRT_VERSION} --depth=1 https://github.com/pytorch/TensorRT torch_tensorrt &&     cd torch_tensorrt &&     mkdir -p toolchains/jp_workspaces || echo "path already existed" &&     cp /tmp/WORKSPACE.jp46 toolchains/jp_workspaces &&     cp toolchains/jp_workspaces/WORKSPACE.jp${JETPACK_MAJOR}${JETPACK_MINOR} WORKSPACE &&     cd py &&     sed 's|^        \"torch >=.*|        \"torch\"|g' -i setup.py &&     sed "s|^        'torch>=.*|        'torch',|g" -i setup.py &&     sed 's|^    python_requires=.*|    python_requires=\">=3.6\",|g' -i setup.py &&     cat setup.py &&     python3 setup.py bdist_wheel --jetpack-version ${JETPACK_MAJOR}.${JETPACK_MINOR} --use-cxx11-abi &&     cp dist/torch_tensorrt*.whl /opt &&     cd ../../ &&     rm -rf torch_tensorrt
 ---> Running in 990da1077759
Cloning into 'torch_tensorrt'...
Note: switching to '7d1d807731c95ffbaa980a3742e289d4184b7871'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

cp: cannot stat 'toolchains/jp_workspaces/WORKSPACE.jp60': No such file or directory
The command '/bin/sh -c git clone --branch=${TORCH_TRT_VERSION} --depth=1 https://github.com/pytorch/TensorRT torch_tensorrt &&     cd torch_tensorrt &&     mkdir -p toolchains/jp_workspaces || echo "path already existed" &&     cp /tmp/WORKSPACE.jp46 toolchains/jp_workspaces &&     cp toolchains/jp_workspaces/WORKSPACE.jp${JETPACK_MAJOR}${JETPACK_MINOR} WORKSPACE &&     cd py &&     sed 's|^        \"torch >=.*|        \"torch\"|g' -i setup.py &&     sed "s|^        'torch>=.*|        'torch',|g" -i setup.py &&     sed 's|^    python_requires=.*|    python_requires=\">=3.6\",|g' -i setup.py &&     cat setup.py &&     python3 setup.py bdist_wheel --jetpack-version ${JETPACK_MAJOR}.${JETPACK_MINOR} --use-cxx11-abi &&     cp dist/torch_tensorrt*.whl /opt &&     cd ../../ &&     rm -rf torch_tensorrt' returned a non-zero code: 1
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/jane/Code/jetson-containers/jetson_containers/build.py", line 112, in <module>
    build_container(args.name, args.packages, args.base, args.build_flags, args.build_args, args.simulate, args.skip_tests, args.test_only, args.push, args.no_github_api)
  File "/home/jane/Code/jetson-containers/jetson_containers/container.py", line 147, in build_container
    status = subprocess.run(cmd.replace(_NEWLINE_, ' '), executable='/bin/bash', shell=True, check=True)
  File "/usr/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command 'DOCKER_BUILDKIT=0 docker build --network=host --tag stjet:r36.4.0-torch_tensorrt --file /home/jane/Code/jetson-containers/packages/pytorch/torch_tensorrt/Dockerfile --build-arg BASE_IMAGE=stjet:r36.4.0-bazel --build-arg PYTHON_VERSION="3.10" --build-arg JETPACK_MAJOR="6" --build-arg JETPACK_MINOR="0" --build-arg TORCH_TRT_VERSION="v1.4.0" /home/jane/Code/jetson-containers/packages/pytorch/torch_tensorrt 2>&1 | tee /home/jane/Code/jetson-containers/logs/20241004_165713/build/stjet_r36.4.0-torch_tensorrt.txt; exit ${PIPESTATUS[0]}' returned non-zero exit status 1.

OK, there was an additional fix needed for OpenCV (see https://github.com/dusty-nv/jetson-containers/commit/f1b346b95b5e5c9e696ff60e2ba79644bc1a94a7)
Also checked in the changes to pip pkginfo and optimum.   torch_tensorrt I haven't touched in a while as it was challenging to keep building, instead end up using torch2trt a lot (coincidentally which I believe also needs updated for TensorRT 10, because I also need to finish updating jetson-inference for TRT10)
Sorry, you picked good week to start out haha, a lot of JetPack updates don't need a whole rebuild but this one was a pretty big version bump from CUDA 12.2->12.6, cuDNN 9.3, and TRT 10.  It's a big stack and can take a couple weeks for it all to settle down.

All good mate, I have built and maintained enough software in my life to know that everything is a fragile house of cards and it’s a miracle anything ever works at all. Thanks for your support, and I will try those things that you suggested.

try now torch_tensorrt. It was pointing to very old version. Now is compatible with: PyTorch 2.4, CUDA 12.4, TensorRT 10.1, Python 3.12 https://github.com/dusty-nv/jetson-containers/pull/657
also torch2trt is compatible with tensorrt 10 in last commit

@dusty-nv you can see it: https://github.com/pytorch/TensorRT/compare/main...lluo/jp6.1

OK thanks yea, it's still not building, will wait until updated and merged for CUDA 12.6.   Indeed torch2trt built and is working though!   dustynv/torch2trt:r36.4.0

OK thanks yea, it's still not building, will wait until updated and merged for CUDA 12.6. Indeed torch2trt built and is working though! dustynv/torch2trt:r36.4.0

super mate!

@dusty-nv : i started building the nano_llm container with ros and i could build the following packages - this of course a big progress since yesterday.

However I get the following error
Installing collected packages: protobuf, onnx
changing mode of /usr/local/bin/backend-test-tools to 755
changing mode of /usr/local/bin/check-model to 755
changing mode of /usr/local/bin/check-node to 755
Successfully installed onnx-1.17.0 protobuf-5.28.2
Name: onnx
Version: 1.17.0
Summary: Open Neural Network Exchange
Home-page: https://onnx.ai/
Author:
Author-email: ONNX Contributors<EMAIL_ADDRESS>License: Apache License v2.0
Location: /usr/local/lib/python3.10/dist-packages
Requires: numpy, protobuf
Required-by:
1.17.0
---> Removed intermediate container ec62e5a2bc46
---> 79b28a3da5ae
Successfully built 79b28a3da5ae
Successfully tagged nano_llm:iron-r36.4.0-cu126-onnx
-- Testing container nano_llm:iron-r36.4.0-cu126-onnx (onnx/test.py)
sudo docker run -t --rm --runtime=nvidia --network=host
--volume /home/test/jetson-containers/packages/ml/onnx:/test
--volume /home/test/jetson-containers/data:/data
--workdir /test
nano_llm:iron-r36.4.0-cu126-onnx
/bin/bash -c 'python3 test.py'
2>&1 | tee /home/test/jetson-containers/logs/20241004_141018/test/nano_llm_iron-r36.4.0-cu126-onnx_test.py.txt; exit ${PIPESTATUS[0]}
exec /ros_entrypoint.sh: exec format error
Traceback (most recent call last):
File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/home/test/jetson-containers/jetson_containers/build.py", line 112, in
build_container(args.name, args.packages, args.base, args.build_flags, args.build_args, args.simulate, args.skip_tests, args.test_only, args.push, args.no_github_api)
File "/home/test/jetson-containers/jetson_containers/container.py", line 154, in build_container
test_container(container_name, pkg, simulate)
File "/home/test/jetson-containers/jetson_containers/container.py", line 327, in test_container
status = subprocess.run(cmd.replace(NEWLINE, ' '), executable='/bin/bash', shell=True, check=True)
File "/usr/lib/python3.10/subprocess.py", line 526, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command 'sudo docker run -t --rm --runtime=nvidia --network=host --volume /home/test/jetson-containers/packages/ml/onnx:/test --volume /home/test/jetson-containers/data:/data --workdir /test nano_llm:iron-r36.4.0-cu126-onnx /bin/bash -c 'python3 test.py' 2>&1 | tee /home/test/jetson-containers/logs/20241004_141018/test/nano_llm_iron-r36.4.0-cu126-onnx_test.py.txt; exit ${PIPESTATUS[0]}' returned non-zero exit status 1.

hey @dusty-nv I tried again today with the latest master (1f5dee99fee).
The pkginfo thing has been fixed, so my diffusers transformers container now builds cleanly on master (and I don't need my branch anymore).
The optimum one still fails, though:
 ---> 92e86450ef25
Step 5/7 : RUN pip3 install --no-cache-dir --verbose accelerate &&     pip3 install --no-cache-dir --verbose sentencepiece &&     pip3 install --no-cache-dir --verbose optimum &&         pip3 uninstall -y transformers &&     pip3 install --no-cache-dir --verbose ${TRANSFORMERS_PACKAGE} &&         PYTHON_ROOT=`pip3 show transformers | grep Location: | cut -d' ' -f2` &&     sed -i 's|torch.distributed.is_initialized|torch.distributed.is_available|g' -i ${PYTHON_ROOT}/transformers/modeling_utils.py
 ---> Running in b1962ec3aaa5
Using pip 24.2 from /usr/local/lib/python3.10/dist-packages/pip (python 3.10)
Looking in indexes: http://jetson.webredirect.org/jp6/cu126
Collecting accelerate
  Downloading http://jetson.webredirect.org/root/pypi/%2Bf/d69/159e2c4e4a473/accelerate-0.34.2-py3-none-any.whl (324 kB)
Requirement already satisfied: numpy<3.0.0,>=1.17 in /usr/local/lib/python3.10/dist-packages (from accelerate) (1.26.4)
Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from accelerate) (24.1)
Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from accelerate) (6.0.0)
Requirement already satisfied: pyyaml in /usr/local/lib/python3.10/dist-packages (from accelerate) (6.0.2)
Requirement already satisfied: torch>=1.10.0 in /usr/local/lib/python3.10/dist-packages (from accelerate) (2.4.0)
Requirement already satisfied: huggingface-hub>=0.21.0 in /usr/local/lib/python3.10/dist-packages (from accelerate) (0.25.1)
Collecting safetensors>=0.4.3 (from accelerate)
  Downloading http://jetson.webredirect.org/root/pypi/%2Bf/688/5016f34bef80e/safetensors-0.4.5-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (441 kB)
Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.21.0->accelerate) (3.16.1)
Requirement already satisfied: fsspec>=2023.5.0 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.21.0->accelerate) (2024.9.0)
Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.21.0->accelerate) (2.32.3)
Requirement already satisfied: tqdm>=4.42.1 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.21.0->accelerate) (4.66.5)
Requirement already satisfied: typing-extensions>=<IP_ADDRESS> in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.21.0->accelerate) (4.12.2)
Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate) (1.13.3)
Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate) (3.3)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate) (3.1.4)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch>=1.10.0->accelerate) (2.1.5)
Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->huggingface-hub>=0.21.0->accelerate) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->huggingface-hub>=0.21.0->accelerate) (3.10)
Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->huggingface-hub>=0.21.0->accelerate) (2.2.3)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->huggingface-hub>=0.21.0->accelerate) (2024.8.30)
Requirement already satisfied: mpmath<1.4,>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from sympy->torch>=1.10.0->accelerate) (1.3.0)
Installing collected packages: safetensors, accelerate
  changing mode of /usr/local/bin/accelerate to 755
  changing mode of /usr/local/bin/accelerate-config to 755
  changing mode of /usr/local/bin/accelerate-estimate-memory to 755
  changing mode of /usr/local/bin/accelerate-launch to 755
  changing mode of /usr/local/bin/accelerate-merge-weights to 755
Successfully installed accelerate-0.34.2 safetensors-0.4.5
Using pip 24.2 from /usr/local/lib/python3.10/dist-packages/pip (python 3.10)
Looking in indexes: http://jetson.webredirect.org/jp6/cu126
Collecting sentencepiece
  Downloading http://jetson.webredirect.org/root/pypi/%2Bf/2fd/e4b08cfe237be/sentencepiece-0.2.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.3/1.3 MB 324.7 kB/s eta 0:00:00
Installing collected packages: sentencepiece
Successfully installed sentencepiece-0.2.0
Using pip 24.2 from /usr/local/lib/python3.10/dist-packages/pip (python 3.10)
Looking in indexes: http://jetson.webredirect.org/jp6/cu126
ERROR: Could not find a version that satisfies the requirement optimum (from versions: none)
ERROR: No matching distribution found for optimum
The command '/bin/sh -c pip3 install --no-cache-dir --verbose accelerate &&     pip3 install --no-cache-dir --verbose sentencepiece &&     pip3 install --no-cache-dir --verbose optimum &&         pip3 uninstall -y transformers &&     pip3 install --no-cache-dir --verbose ${TRANSFORMERS_PACKAGE} &&         PYTHON_ROOT=`pip3 show transformers | grep Location: | cut -d' ' -f2` &&     sed -i 's|torch.distributed.is_initialized|torch.distributed.is_available|g' -i ${PYTHON_ROOT}/transformers/modeling_utils.py' returned a non-zero code: 1

Anyway, I can make do with what I've got for now. Re: optimum you mentioned above you checked in some changes - are they in master or only on dev?
Do you think that tensorrt (via optimum) would give me some easy speedups for a Stable Diffusion pipeline on the AGX Orin? This is all for an interactive art installation where higher throughput for the pipeline would be handy.

Hey @dusty-nv I actually spoke to soon above - I was still using my local change where I had commented out the optimus dep.
Again, my goal is to build a container with diffusers, transformers, and torch2trt (or torch_tensorrt, or some other way of using TensorRT-accelerated pytorch).
As a first step, I ensure I'm on a clean master:
jane@ubuntu:~/Code/jetson-containers$ git show HEAD
commit 0189a3455531341c79b5877f58e3b30e92fc10b2 (HEAD -> master, origin/master, origin/HEAD)
Merge: 1f5dee99 e11c43ca
Author: Dustin Franklin<EMAIL_ADDRESS>Date:   Mon Oct 7 03:33:25 2024 -0400

    Merge remote-tracking branch 'origin/dev'

jane@ubuntu:~/Code/jetson-containers$ git status
On branch master
Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean

and I try and build the transformers container:

jane@ubuntu:~/Code/jetson-containers$ jetson-containers build --name=my_container transformers
Namespace(packages=['transformers'], name='my_container', base='', multiple=False, build_flags='', build_args='', package_dirs=[''], list_packages=False, show_packages=False, skip_packages=[''], skip_errors=False, skip_tests=[''], test_only=[''], simulate=False, push='', logs='', verbose=False, no_github_api=False)
-- L4T_VERSION=36.4.0
-- JETPACK_VERSION=6.1
-- CUDA_VERSION=12.6
-- PYTHON_VERSION=3.10
-- LSB_RELEASE=22.04 (jammy)
-- Building containers  ['build-essential', 'pip_cache:cu126', 'cuda:12.6', 'cudnn', 'python', 'numpy', 'cmake', 'onnx', 'pytorch:2.4', 'torchvision', 'huggingface_hub', 'rust', 'transformers']
-- Building container my_container:r36.4.0-build-essential
DOCKER_BUILDKIT=0 docker build --network=host --tag my_container:r36.4.0-build-essential
--file /home/jane/Code/jetson-containers/packages/build/build-essential/Dockerfile
--build-arg BASE_IMAGE=ubuntu:22.04
/home/jane/Code/jetson-containers/packages/build/build-essential
2>&1 | tee /home/jane/Code/jetson-containers/logs/20241009_134747/build/my_container_r36.4.0-build-essential.txt; exit ${PIPESTATUS[0]}
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
environment-variable.
Sending build context to Docker daemon  19.97kB
Step 1/5 : ARG BASE_IMAGE
Step 2/5 : FROM ${BASE_IMAGE}
---> 981912c48e9a
Step 3/5 : ENV DEBIAN_FRONTEND=noninteractive     LANGUAGE=en_US:en     LANG=en_US.UTF-8     LC_ALL=en_US.UTF-8
---> Using cache
---> bee7dec2ca35
Step 4/5 : RUN set -ex     && apt-get update     && apt-get install -y --no-install-recommends         locales         locales-all         tzdata     && locale-gen en_US $LANG     && update-locale LC_ALL=$LC_ALL LANG=$LANG     && locale         && apt-get install -y --no-install-recommends         build-essential         software-properties-common         apt-transport-https         ca-certificates         lsb-release         pkg-config         gnupg         git         gdb         wget         curl         nano         zip         unzip         time 	   sshpass 	   ssh-client     && apt-get clean     && rm -rf /var/lib/apt/lists/*         && gcc --version     && g++ --version
---> Using cache
---> 5a61c6947b48
Step 5/5 : COPY tarpack /usr/local/bin/
---> Using cache
---> 990e4037e0a8
Successfully built 990e4037e0a8
Successfully tagged my_container:r36.4.0-build-essential
-- Building container my_container:r36.4.0-pip_cache_cu126
DOCKER_BUILDKIT=0 docker build --network=host --tag my_container:r36.4.0-pip_cache_cu126
--file /home/jane/Code/jetson-containers/packages/cuda/cuda/Dockerfile.pip
--build-arg BASE_IMAGE=my_container:r36.4.0-build-essential
--build-arg TAR_INDEX_URL="http://jetson.webredirect.org:8000/jp6/cu126"
--build-arg PIP_INDEX_REPO="http://jetson.webredirect.org/jp6/cu126"
--build-arg PIP_TRUSTED_HOSTS="jetson.webredirect.org"
--build-arg PIP_UPLOAD_REPO="http://localhost/jp6/cu126"
--build-arg PIP_UPLOAD_USER="jp6"
--build-arg PIP_UPLOAD_PASS="none"
--build-arg SCP_UPLOAD_URL="localhost:/dist/jp6/cu126"
--build-arg SCP_UPLOAD_USER="None"
--build-arg SCP_UPLOAD_PASS="None"
/home/jane/Code/jetson-containers/packages/cuda/cuda
2>&1 | tee /home/jane/Code/jetson-containers/logs/20241009_134747/build/my_container_r36.4.0-pip_cache_cu126.txt; exit ${PIPESTATUS[0]}
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
environment-variable.
Sending build context to Docker daemon  48.64kB
Step 1/4 : ARG BASE_IMAGE
Step 2/4 : FROM ${BASE_IMAGE}
---> 990e4037e0a8
Step 3/4 : ARG PIP_INDEX_REPO     PIP_UPLOAD_REPO     PIP_UPLOAD_USER     PIP_UPLOAD_PASS     PIP_TRUSTED_HOSTS     TAR_INDEX_URL     SCP_UPLOAD_URL     SCP_UPLOAD_USER     SCP_UPLOAD_PASS
---> Using cache
---> c2efa4465977
Step 4/4 : ENV TAR_INDEX_URL=${TAR_INDEX_URL}     PIP_INDEX_URL=${PIP_INDEX_REPO}     PIP_TRUSTED_HOST=${PIP_TRUSTED_HOSTS}     TWINE_REPOSITORY_URL=${PIP_UPLOAD_REPO}     TWINE_USERNAME=${PIP_UPLOAD_USER}     TWINE_PASSWORD=${PIP_UPLOAD_PASS}     SCP_UPLOAD_URL=${SCP_UPLOAD_URL}     SCP_UPLOAD_USER=${SCP_UPLOAD_USER}     SCP_UPLOAD_PASS=${SCP_UPLOAD_PASS}
---> Using cache
---> 4531f23aa406
Successfully built 4531f23aa406
Successfully tagged my_container:r36.4.0-pip_cache_cu126
-- Building container my_container:r36.4.0-cuda_12.6
DOCKER_BUILDKIT=0 docker build --network=host --tag my_container:r36.4.0-cuda_12.6
--file /home/jane/Code/jetson-containers/packages/cuda/cuda/Dockerfile
--build-arg BASE_IMAGE=my_container:r36.4.0-pip_cache_cu126
--build-arg CUDA_URL="https://developer.download.nvidia.com/compute/cuda/12.6.2/local_installers/cuda-tegra-repo-ubuntu2204-12-6-local_12.6.2-1_arm64.deb"
--build-arg CUDA_DEB="cuda-tegra-repo-ubuntu2204-12-6-local"
--build-arg CUDA_PACKAGES="cuda-toolkit*"
--build-arg CUDA_ARCH_LIST="87"
--build-arg DISTRO="ubuntu2204"
/home/jane/Code/jetson-containers/packages/cuda/cuda
2>&1 | tee /home/jane/Code/jetson-containers/logs/20241009_134747/build/my_container_r36.4.0-cuda_12.6.txt; exit ${PIPESTATUS[0]}
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
environment-variable.

BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
environment-variable.
Sending build context to Docker daemon  26.62kB
Step 1/7 : ARG BASE_IMAGE
Step 2/7 : FROM ${BASE_IMAGE}
---> eb9eacbef413
Step 3/7 : ARG CUDNN_URL
---> Using cache
---> 3b016ae64a80
Step 4/7 : ARG CUDNN_DEB
---> Using cache
---> 43f58ee8553c
Step 5/7 : ARG CUDNN_PACKAGES
---> Using cache
---> f1528a899211
Step 6/7 : RUN echo "Downloading ${CUDNN_DEB}" &&     mkdir /tmp/cudnn && cd /tmp/cudnn &&     wget --quiet --show-progress --progress=bar:force:noscroll ${CUDNN_URL} &&     dpkg -i .deb &&     cp /var/cudnn-local-tegra-repo-/cudnn-local-tegra--keyring.gpg /usr/share/keyrings/ &&     apt-get update &&     apt-cache search cudnn &&     apt-get install -y --no-install-recommends ${CUDNN_PACKAGES} &&     rm -rf /var/lib/apt/lists/ &&     apt-get clean &&     dpkg --list | grep cudnn &&     dpkg -P ${CUDNN_DEB} &&     rm -rf /tmp/cudnn
---> Using cache
---> 1b87b4cfc62e
Step 7/7 : RUN cd /usr/src/cudnn_samples_v*/conv_sample/ &&     make -j$(nproc)
---> Using cache
---> 0ea94b62cb3d
Successfully built 0ea94b62cb3d
Successfully tagged my_container:r36.4.0-cudnn
-- Testing container my_container:r36.4.0-cudnn (cudnn:9.4/test.sh)
docker run -t --rm --runtime=nvidia --network=host
--volume /home/jane/Code/jetson-containers/packages/cuda/cudnn:/test
--volume /home/jane/Code/jetson-containers/data:/data
--workdir /test
my_container:r36.4.0-cudnn
/bin/bash -c '/bin/bash test.sh'
2>&1 | tee /home/jane/Code/jetson-containers/logs/20241009_134747/test/my_container_r36.4.0-cudnn_test.sh.txt; exit ${PIPESTATUS[0]}
#define CUDNN_MAJOR 9
#define CUDNN_MINOR 4
#define CUDNN_VERSION (CUDNN_MAJOR * 10000 + CUDNN_MINOR * 100 + CUDNN_PATCHLEVEL)
#define CUDNN_MAX_SM_MAJOR_NUMBER 9
#define CUDNN_MAX_SM_MINOR_NUMBER 0
#define CUDNN_MAX_DEVICE_VERSION (CUDNN_MAX_SM_MAJOR_NUMBER * 100 + CUDNN_MAX_SM_MINOR_NUMBER * 10)
Executing: conv_sample
Using format CUDNN_TENSOR_NCHW (for INT8x4 and INT8x32 tests use CUDNN_TENSOR_NCHW_VECT_C)
Testing single precision
====USER DIMENSIONS====
input dims are 1, 32, 4, 4
filter dims are 32, 32, 1, 1
output dims are 1, 32, 4, 4
====PADDING DIMENSIONS====
padded input dims are 1, 32, 4, 4
padded filter dims are 32, 32, 1, 1
padded output dims are 1, 32, 4, 4
Testing conv
^^^^ CUDA : elapsed = 0.000510164 sec,
Test PASSED
Testing half precision (math in single precision)
====USER DIMENSIONS====
input dims are 1, 32, 4, 4
filter dims are 32, 32, 1, 1
output dims are 1, 32, 4, 4
====PADDING DIMENSIONS====
padded input dims are 1, 32, 4, 4
padded filter dims are 32, 32, 1, 1
padded output dims are 1, 32, 4, 4
Testing conv
^^^^ CUDA : elapsed = 0.0279498 sec,
Test PASSED
-- Building container my_container:r36.4.0-python
DOCKER_BUILDKIT=0 docker build --network=host --tag my_container:r36.4.0-python
--file /home/jane/Code/jetson-containers/packages/build/python/Dockerfile
--build-arg BASE_IMAGE=my_container:r36.4.0-cudnn
--build-arg PYTHON_VERSION_ARG="3.10"
/home/jane/Code/jetson-containers/packages/build/python
2>&1 | tee /home/jane/Code/jetson-containers/logs/20241009_134747/build/my_container_r36.4.0-python.txt; exit ${PIPESTATUS[0]}
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
environment-variable.
Sending build context to Docker daemon  23.55kB
Step 1/6 : ARG BASE_IMAGE
Step 2/6 : FROM ${BASE_IMAGE}
---> 0ea94b62cb3d
Step 3/6 : ARG PYTHON_VERSION_ARG
---> Using cache
---> c90075ddcd71
Step 4/6 : ENV PYTHON_VERSION=${PYTHON_VERSION_ARG}     PIP_DISABLE_PIP_VERSION_CHECK=on     PIP_DEFAULT_TIMEOUT=100     PYTHONFAULTHANDLER=1     PYTHONUNBUFFERED=1     PYTHONIOENCODING=utf-8     PYTHONHASHSEED=random     PIP_NO_CACHE_DIR=off     PIP_CACHE_PURGE=true     PIP_ROOT_USER_ACTION=ignore     TWINE_NON_INTERACTIVE=1     DEBIAN_FRONTEND=noninteractive
---> Using cache
---> 92e35d45f4ae
Step 5/6 : COPY install.sh /tmp/install_python.sh
---> Using cache
---> 71d392a77ceb
Step 6/6 : RUN /tmp/install_python.sh
---> Using cache
---> 10dfd3f392d5
Successfully built 10dfd3f392d5
Successfully tagged my_container:r36.4.0-python
-- Building container my_container:r36.4.0-numpy
DOCKER_BUILDKIT=0 docker build --network=host --tag my_container:r36.4.0-numpy
--file /home/jane/Code/jetson-containers/packages/numeric/numpy/Dockerfile
--build-arg BASE_IMAGE=my_container:r36.4.0-python
/home/jane/Code/jetson-containers/packages/numeric/numpy
2>&1 | tee /home/jane/Code/jetson-containers/logs/20241009_134747/build/my_container_r36.4.0-numpy.txt; exit ${PIPESTATUS[0]}
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
environment-variable.
Sending build context to Docker daemon  15.87kB
Step 1/4 : ARG BASE_IMAGE
Step 2/4 : FROM ${BASE_IMAGE}
---> 10dfd3f392d5
Step 3/4 : ENV OPENBLAS_CORETYPE=ARMV8
---> Using cache
---> b3be003dc6a5
Step 4/4 : RUN pip3 install --upgrade --force-reinstall --no-cache-dir --verbose 'numpy<2' &&     pip3 show numpy && python3 -c 'import numpy; print(numpy.version)'
---> Using cache
---> 629ee99e8ef2
Successfully built 629ee99e8ef2
Successfully tagged my_container:r36.4.0-numpy
-- Testing container my_container:r36.4.0-numpy (numpy/test.py)
docker run -t --rm --runtime=nvidia --network=host
--volume /home/jane/Code/jetson-containers/packages/numeric/numpy:/test
--volume /home/jane/Code/jetson-containers/data:/data
--workdir /test
my_container:r36.4.0-numpy
/bin/bash -c 'python3 test.py'
2>&1 | tee /home/jane/Code/jetson-containers/logs/20241009_134747/test/my_container_r36.4.0-numpy_test.py.txt; exit ${PIPESTATUS[0]}
testing numpy...
numpy version: 1.26.4
/usr/local/lib/python3.10/dist-packages/numpy/config.py:155: UserWarning: Install pyyaml for better output
warnings.warn("Install pyyaml for better output", stacklevel=1)
{
"Compilers": {
"c": {
"name": "gcc",
"linker": "ld.bfd",
"version": "10.2.1",
"commands": "cc",
"args": "-fno-strict-aliasing",
"linker args": "-Wl,--strip-debug, -fno-strict-aliasing"
},
"cython": {
"name": "cython",
"linker": "cython",
"version": "3.0.8",
"commands": "cython"
},
"c++": {
"name": "gcc",
"linker": "ld.bfd",
"version": "10.2.1",
"commands": "c++",
"linker args": "-Wl,--strip-debug"
}
},
"Machine Information": {
"host": {
"cpu": "aarch64",
"family": "aarch64",
"endian": "little",
"system": "linux"
},
"build": {
"cpu": "aarch64",
"family": "aarch64",
"endian": "little",
"system": "linux"
}
},
"Build Dependencies": {
"blas": {
"name": "openblas64",
"found": true,
"version": "0.3.23.dev",
"detection method": "pkgconfig",
"include directory": "/usr/local/include",
"lib directory": "/usr/local/lib",
"openblas configuration": "USE_64BITINT=1 DYNAMIC_ARCH=1 DYNAMIC_OLDER= NO_CBLAS= NO_LAPACK= NO_LAPACKE= NO_AFFINITY=1 USE_OPENMP= NEOVERSEN1 MAX_THREADS=80",
"pc file directory": "/usr/local/lib/pkgconfig"
},
"lapack": {
"name": "dep281473538264272",
"found": true,
"version": "1.26.4",
"detection method": "internal",
"include directory": "unknown",
"lib directory": "unknown",
"openblas configuration": "unknown",
"pc file directory": "unknown"
}
},
"Python Information": {
"path": "/opt/python/cp310-cp310/bin/python",
"version": "3.10"
},
"SIMD Extensions": {
"baseline": [
"NEON",
"NEON_FP16",
"NEON_VFPV4",
"ASIMD"
],
"found": [
"ASIMDHP"
],
"not found": [
"ASIMDFHM"
]
}
}
None
numpy OK
-- Building container my_container:r36.4.0-cmake
DOCKER_BUILDKIT=0 docker build --network=host --tag my_container:r36.4.0-cmake
--file /home/jane/Code/jetson-containers/packages/build/cmake/cmake_pip/Dockerfile
--build-arg BASE_IMAGE=my_container:r36.4.0-numpy
/home/jane/Code/jetson-containers/packages/build/cmake/cmake_pip
2>&1 | tee /home/jane/Code/jetson-containers/logs/20241009_134747/build/my_container_r36.4.0-cmake.txt; exit ${PIPESTATUS[0]}
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
environment-variable.
Sending build context to Docker daemon   12.8kB
Step 1/3 : ARG BASE_IMAGE
Step 2/3 : FROM ${BASE_IMAGE}
---> 629ee99e8ef2
Step 3/3 : RUN set -ex     && pip3 install --upgrade --force-reinstall --no-cache-dir --verbose cmake         && cmake --version     && which cmake
---> Using cache
---> 633c0bb7e737
Successfully built 633c0bb7e737
Successfully tagged my_container:r36.4.0-cmake
-- Building container my_container:r36.4.0-onnx
DOCKER_BUILDKIT=0 docker build --network=host --tag my_container:r36.4.0-onnx
--file /home/jane/Code/jetson-containers/packages/ml/onnx/Dockerfile
--build-arg BASE_IMAGE=my_container:r36.4.0-cmake
--build-arg ONNX_VERSION="main"
/home/jane/Code/jetson-containers/packages/ml/onnx
2>&1 | tee /home/jane/Code/jetson-containers/logs/20241009_134747/build/my_container_r36.4.0-onnx.txt; exit ${PIPESTATUS[0]}
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
environment-variable.
Sending build context to Docker daemon  18.43kB
Step 1/4 : ARG BASE_IMAGE
Step 2/4 : FROM ${BASE_IMAGE}
---> 633c0bb7e737
Step 3/4 : ARG ONNX_VERSION
---> Using cache
---> 25b3196aadb1
Step 4/4 : RUN pip3 install --no-cache-dir --verbose onnx ||     pip3 install --no-cache-dir --verbose git+https://github.com/onnx/onnx@${ONNX_VERSION} &&     pip3 show onnx &&     python3 -c 'import onnx; print(onnx.version)'
---> Using cache
---> c9a62fe3d3df
Successfully built c9a62fe3d3df
Successfully tagged my_container:r36.4.0-onnx
-- Testing container my_container:r36.4.0-onnx (onnx/test.py)
docker run -t --rm --runtime=nvidia --network=host
--volume /home/jane/Code/jetson-containers/packages/ml/onnx:/test
--volume /home/jane/Code/jetson-containers/data:/data
--workdir /test
my_container:r36.4.0-onnx
/bin/bash -c 'python3 test.py'
2>&1 | tee /home/jane/Code/jetson-containers/logs/20241009_134747/test/my_container_r36.4.0-onnx_test.py.txt; exit ${PIPESTATUS[0]}
testing onnx...
onnx version: 1.17.0
onnx OK
-- Building container my_container:r36.4.0-pytorch_2.4
DOCKER_BUILDKIT=0 docker build --network=host --tag my_container:r36.4.0-pytorch_2.4
--file /home/jane/Code/jetson-containers/packages/pytorch/Dockerfile.pip
--build-arg BASE_IMAGE=my_container:r36.4.0-onnx
--build-arg TORCH_CUDA_ARCH_ARGS="8.7"
--build-arg TORCH_VERSION="2.4"
--build-arg PYTORCH_BUILD_VERSION="2.4.0"
--build-arg USE_NCCL="1"
/home/jane/Code/jetson-containers/packages/pytorch
2>&1 | tee /home/jane/Code/jetson-containers/logs/20241009_134747/build/my_container_r36.4.0-pytorch_2.4.txt; exit ${PIPESTATUS[0]}
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
environment-variable.
Sending build context to Docker daemon  135.7kB
Step 1/6 : ARG BASE_IMAGE
Step 2/6 : FROM ${BASE_IMAGE}
---> c9a62fe3d3df
Step 3/6 : ARG TORCH_CUDA_ARCH_ARGS     TORCH_VERSION     PYTORCH_BUILD_VERSION     USE_NCCL=0     FORCE_BUILD=off
---> Using cache
---> 764159ea9fab
Step 4/6 : ENV TORCH_CUDA_ARCH_LIST=${TORCH_CUDA_ARCH_ARGS}     TORCH_HOME=/data/models/torch
---> Using cache
---> d8606173f920
Step 5/6 : COPY install.sh build.sh /tmp/pytorch/
---> Using cache
---> 31e3471796bf
Step 6/6 : RUN /tmp/pytorch/install.sh || /tmp/pytorch/build.sh
---> Using cache
---> 006147abf80e
Successfully built 006147abf80e
Successfully tagged my_container:r36.4.0-pytorch_2.4
-- Testing container my_container:r36.4.0-pytorch_2.4 (pytorch:2.4/test.py)
docker run -t --rm --runtime=nvidia --network=host
--volume /home/jane/Code/jetson-containers/packages/pytorch:/test
--volume /home/jane/Code/jetson-containers/data:/data
--workdir /test
my_container:r36.4.0-pytorch_2.4
/bin/bash -c 'python3 test.py'
2>&1 | tee /home/jane/Code/jetson-containers/logs/20241009_134747/test/my_container_r36.4.0-pytorch_2.4_test.py.txt; exit ${PIPESTATUS[0]}
testing PyTorch...
PyTorch version: 2.4.0
CUDA available:  True
cuDNN version:   90400
PyTorch built with:

GCC 11.4
C++ Version: 201703
OpenMP 201511 (a.k.a. OpenMP 4.5)
LAPACK is enabled (usually provided by MKL)
NNPACK is enabled
CPU capability usage: NO AVX
CUDA Runtime 12.6
NVCC architecture flags: -gencode;arch=compute_87,code=sm_87
CuDNN 90.4
Build settings: BLAS_INFO=open, BUILD_TYPE=Release, CUDA_VERSION=12.6, CUDNN_VERSION=9.4.0, CXX_COMPILER=/usr/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=range-loop-construct -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=open, TORCH_VERSION=2.4.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=OFF, USE_EIGEN_FOR_BLAS=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=OFF, USE_MKLDNN=OFF, USE_MPI=ON, USE_NCCL=1, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF,

PACKAGING_VERSION=2.4.0
TORCH_CUDA_ARCH_LIST=8.7
/test/test.py:23: UserWarning: The torch.cuda.DtypeTensor constructors are no longer recommended. It's best to use methods such as torch.tensor(data, dtype=, device='cuda') to create tensors. (Triggered internally at /opt/pytorch/torch/csrc/tensor/python_tensor.cpp:78.)
a = torch.cuda.FloatTensor(2).zero_()
Tensor a = tensor([0., 0.], device='cuda:0')
Tensor b = tensor([-0.1467, -1.2518], device='cuda:0')
Tensor c = tensor([-0.1467, -1.2518], device='cuda:0')
testing LAPACK (OpenBLAS)...
done testing LAPACK (OpenBLAS)
testing torch.nn (cuDNN)...
done testing torch.nn (cuDNN)
testing CPU tensor vector operations...
/test/test.py:62: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.
cpu_y = F.softmax(cpu_x)
Tensor cpu_x = tensor([12.3450])
Tensor softmax = tensor([1.])
Tensor exp (float32) = tensor([[2.7183, 2.7183, 2.7183],
[2.7183, 2.7183, 2.7183],
[2.7183, 2.7183, 2.7183]])
Tensor exp (float64) = tensor([[2.7183, 2.7183, 2.7183],
[2.7183, 2.7183, 2.7183],
[2.7183, 2.7183, 2.7183]], dtype=torch.float64)
Tensor exp (diff) = 7.429356050359104e-07
PyTorch OK
-- Building container my_container:r36.4.0-torchvision
DOCKER_BUILDKIT=0 docker build --network=host --tag my_container:r36.4.0-torchvision
--file /home/jane/Code/jetson-containers/packages/pytorch/torchvision/Dockerfile
--build-arg BASE_IMAGE=my_container:r36.4.0-pytorch_2.4
--build-arg TORCHVISION_VERSION="0.19.1"
/home/jane/Code/jetson-containers/packages/pytorch/torchvision
2>&1 | tee /home/jane/Code/jetson-containers/logs/20241009_134747/build/my_container_r36.4.0-torchvision.txt; exit ${PIPESTATUS[0]}
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
environment-variable.
Sending build context to Docker daemon  31.74kB
Step 1/6 : ARG BASE_IMAGE
Step 2/6 : FROM ${BASE_IMAGE}
---> 006147abf80e
Step 3/6 : ARG TORCHVISION_VERSION     FORCE_BUILD=off
---> Using cache
---> 6fbdc867b8d8
Step 4/6 : COPY build.sh install.sh /tmp/torchvision/
---> Using cache
---> 45bad51bdef4
Step 5/6 : RUN apt-get update &&     apt-get install -y --no-install-recommends         libjpeg-dev         libpng-dev         zlib1g-dev 	&& rm -rf /var/lib/apt/lists/* 	&& apt-get clean
---> Using cache
---> ac7d80fbbdba
Step 6/6 : RUN /tmp/torchvision/install.sh || /tmp/torchvision/build.sh
---> Using cache
---> eb4eb2a57e01
Successfully built eb4eb2a57e01
Successfully tagged my_container:r36.4.0-torchvision
-- Testing container my_container:r36.4.0-torchvision (torchvision:0.19.1/test.py)
docker run -t --rm --runtime=nvidia --network=host
--volume /home/jane/Code/jetson-containers/packages/pytorch/torchvision:/test
--volume /home/jane/Code/jetson-containers/data:/data
--workdir /test
my_container:r36.4.0-torchvision
/bin/bash -c 'python3 test.py'
2>&1 | tee /home/jane/Code/jetson-containers/logs/20241009_134747/test/my_container_r36.4.0-torchvision_test.py.txt; exit ${PIPESTATUS[0]}
testing torchvision...
torchvision version: 0.19.1a0+6194369
testing torchvision extensions...
torchvision classification models: alexnet | convnext_base | convnext_large | convnext_small | convnext_tiny | densenet121 | densenet161 | densenet169 | densenet201 | efficientnet_b0 | efficientnet_b1 | efficientnet_b2 | efficientnet_b3 | efficientnet_b4 | efficientnet_b5 | efficientnet_b6 | efficientnet_b7 | efficientnet_v2_l | efficientnet_v2_m | efficientnet_v2_s | get_model | get_model_builder | get_model_weights | get_weight | googlenet | inception_v3 | list_models | maxvit_t | mnasnet0_5 | mnasnet0_75 | mnasnet1_0 | mnasnet1_3 | mobilenet_v2 | mobilenet_v3_large | mobilenet_v3_small | regnet_x_16gf | regnet_x_1_6gf | regnet_x_32gf | regnet_x_3_2gf | regnet_x_400mf | regnet_x_800mf | regnet_x_8gf | regnet_y_128gf | regnet_y_16gf | regnet_y_1_6gf | regnet_y_32gf | regnet_y_3_2gf | regnet_y_400mf | regnet_y_800mf | regnet_y_8gf | resnet101 | resnet152 | resnet18 | resnet34 | resnet50 | resnext101_32x8d | resnext101_64x4d | resnext50_32x4d | shufflenet_v2_x0_5 | shufflenet_v2_x1_0 | shufflenet_v2_x1_5 | shufflenet_v2_x2_0 | squeezenet1_0 | squeezenet1_1 | swin_b | swin_s | swin_t | swin_v2_b | swin_v2_s | swin_v2_t | vgg11 | vgg11_bn | vgg13 | vgg13_bn | vgg16 | vgg16_bn | vgg19 | vgg19_bn | vit_b_16 | vit_b_32 | vit_h_14 | vit_l_16 | vit_l_32 | wide_resnet101_2 | wide_resnet50_2
Namespace(data_url='https://nvidia.box.com/shared/static/y1ygiahv8h75yiyh0pt50jqdqt7pohgx.gz', data_tar='ILSVRC2012_img_val_subset_5k.tar.gz', models=['resnet18'], resolution=224, workers=2, batch_size=8, print_freq=25, test_threshold=-10.0, use_cuda=True)
using CUDA
dataset classes: 1000
dataset images:  5000
batch size:      8

-- resnet18
loading model 'resnet18'
/usr/local/lib/python3.10/dist-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
warnings.warn(
/usr/local/lib/python3.10/dist-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or None for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing weights=ResNet18_Weights.IMAGENET1K_V1. You can also use weights=ResNet18_Weights.DEFAULT to get the most up-to-date weights.
warnings.warn(msg)
loaded model 'resnet18'
resnet18 [  0/625]  Time  0.561 ( 0.561)  Acc@1  75.00 ( 75.00)  Acc@5 100.00 (100.00)
resnet18 [ 25/625]  Time  0.029 ( 0.050)  Acc@1  62.50 ( 79.81)  Acc@5  87.50 ( 94.71)
resnet18 [ 50/625]  Time  0.042 ( 0.041)  Acc@1 100.00 ( 73.53)  Acc@5 100.00 ( 91.91)
resnet18 [ 75/625]  Time  0.008 ( 0.041)  Acc@1  62.50 ( 76.48)  Acc@5  87.50 ( 92.27)
resnet18 [100/625]  Time  0.057 ( 0.040)  Acc@1  87.50 ( 78.22)  Acc@5 100.00 ( 93.07)
resnet18 [125/625]  Time  0.009 ( 0.039)  Acc@1  87.50 ( 77.38)  Acc@5 100.00 ( 93.35)
resnet18 [150/625]  Time  0.031 ( 0.038)  Acc@1  12.50 ( 76.49)  Acc@5 100.00 ( 93.38)
resnet18 [175/625]  Time  0.008 ( 0.037)  Acc@1  62.50 ( 76.42)  Acc@5 100.00 ( 93.75)
resnet18 [200/625]  Time  0.008 ( 0.037)  Acc@1 100.00 ( 76.55)  Acc@5 100.00 ( 93.72)
resnet18 [225/625]  Time  0.057 ( 0.036)  Acc@1  87.50 ( 76.77)  Acc@5 100.00 ( 93.58)
resnet18 [250/625]  Time  0.044 ( 0.037)  Acc@1  37.50 ( 76.64)  Acc@5  62.50 ( 93.63)
resnet18 [275/625]  Time  0.009 ( 0.036)  Acc@1  87.50 ( 75.72)  Acc@5 100.00 ( 93.25)
resnet18 [300/625]  Time  0.051 ( 0.036)  Acc@1  50.00 ( 74.38)  Acc@5  87.50 ( 92.07)
resnet18 [325/625]  Time  0.010 ( 0.035)  Acc@1  75.00 ( 73.27)  Acc@5 100.00 ( 91.53)
resnet18 [350/625]  Time  0.053 ( 0.035)  Acc@1  62.50 ( 72.69)  Acc@5  87.50 ( 91.10)
resnet18 [375/625]  Time  0.019 ( 0.037)  Acc@1  25.00 ( 72.64)  Acc@5  62.50 ( 90.89)
resnet18 [400/625]  Time  0.051 ( 0.036)  Acc@1  75.00 ( 71.95)  Acc@5  87.50 ( 90.27)
resnet18 [425/625]  Time  0.009 ( 0.036)  Acc@1  62.50 ( 71.33)  Acc@5  87.50 ( 89.91)
resnet18 [450/625]  Time  0.057 ( 0.036)  Acc@1  75.00 ( 71.26)  Acc@5  87.50 ( 89.99)
resnet18 [475/625]  Time  0.019 ( 0.035)  Acc@1  37.50 ( 70.75)  Acc@5  75.00 ( 89.63)
resnet18 [500/625]  Time  0.053 ( 0.036)  Acc@1  75.00 ( 70.33)  Acc@5  87.50 ( 89.25)
resnet18 [525/625]  Time  0.009 ( 0.036)  Acc@1  62.50 ( 69.94)  Acc@5  87.50 ( 89.02)
resnet18 [550/625]  Time  0.046 ( 0.036)  Acc@1 100.00 ( 69.67)  Acc@5 100.00 ( 88.77)
resnet18 [575/625]  Time  0.052 ( 0.036)  Acc@1  75.00 ( 69.31)  Acc@5  87.50 ( 88.54)
resnet18 [600/625]  Time  0.017 ( 0.036)  Acc@1  37.50 ( 69.68)  Acc@5  87.50 ( 88.71)
resnet18

Acc@1 69.740  Expected 69.760   Delta -0.020
Acc@5 88.760  Expected 89.080   Delta -0.320
Images/sec  224.597
PASS

-- Summary
resnet18

Acc@1 69.740  Expected 69.760   Delta -0.020
Acc@5 88.760  Expected 89.080   Delta -0.320
Images/sec  224.597
PASS

Model tests passing:  1 / 1
torchvision OK
-- Building container my_container:r36.4.0-huggingface_hub
DOCKER_BUILDKIT=0 docker build --network=host --tag my_container:r36.4.0-huggingface_hub
--file /home/jane/Code/jetson-containers/packages/llm/huggingface_hub/Dockerfile
--build-arg BASE_IMAGE=my_container:r36.4.0-torchvision
/home/jane/Code/jetson-containers/packages/llm/huggingface_hub
2>&1 | tee /home/jane/Code/jetson-containers/logs/20241009_134747/build/my_container_r36.4.0-huggingface_hub.txt; exit ${PIPESTATUS[0]}
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
environment-variable.
Sending build context to Docker daemon   16.9kB
Step 1/6 : ARG BASE_IMAGE
Step 2/6 : FROM ${BASE_IMAGE}
---> eb4eb2a57e01
Step 3/6 : ENV TRANSFORMERS_CACHE=/data/models/huggingface     HUGGINGFACE_HUB_CACHE=/data/models/huggingface     HF_HOME=/data/models/huggingface
---> Using cache
---> 36f1bfbe2159
Step 4/6 : COPY huggingface-downloader /usr/local/bin/
---> Using cache
---> 06ff69a3f3b8
Step 5/6 : COPY huggingface-downloader.py /usr/local/bin/_huggingface-downloader.py
---> Using cache
---> 2e2bc3dc8d53
Step 6/6 : RUN set -ex     && pip3 install --no-cache-dir --verbose         huggingface_hub[cli]         dataclasses         && huggingface-cli --help     && huggingface-downloader --help     && pip3 show huggingface_hub     && python3 -c 'import huggingface_hub; print(huggingface_hub.version)'         && apt-get update     && rm -rf /var/lib/apt/lists/*     && apt-get clean
---> Using cache
---> d4ce2186c392
Successfully built d4ce2186c392
Successfully tagged my_container:r36.4.0-huggingface_hub
-- Testing container my_container:r36.4.0-huggingface_hub (huggingface_hub/test.py)
docker run -t --rm --runtime=nvidia --network=host
--volume /home/jane/Code/jetson-containers/packages/llm/huggingface_hub:/test
--volume /home/jane/Code/jetson-containers/data:/data
--workdir /test
my_container:r36.4.0-huggingface_hub
/bin/bash -c 'python3 test.py'
2>&1 | tee /home/jane/Code/jetson-containers/logs/20241009_134747/test/my_container_r36.4.0-huggingface_hub_test.py.txt; exit ${PIPESTATUS[0]}
testing huggingface_hub...
huggingface_hub version: 0.25.1
huggingface_hub OK
-- Building container my_container:r36.4.0-rust
DOCKER_BUILDKIT=0 docker build --network=host --tag my_container:r36.4.0-rust
--file /home/jane/Code/jetson-containers/packages/build/rust/Dockerfile
--build-arg BASE_IMAGE=my_container:r36.4.0-huggingface_hub
/home/jane/Code/jetson-containers/packages/build/rust
2>&1 | tee /home/jane/Code/jetson-containers/logs/20241009_134747/build/my_container_r36.4.0-rust.txt; exit ${PIPESTATUS[0]}
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
environment-variable.
Sending build context to Docker daemon  9.216kB
Step 1/5 : ARG BASE_IMAGE
Step 2/5 : FROM ${BASE_IMAGE}
---> d4ce2186c392
Step 3/5 : RUN curl https://sh.rustup.rs -sSf | sh -s -- -y
---> Using cache
---> e849e4d7e258
Step 4/5 : ENV PATH="/root/.cargo/bin:${PATH}"
---> Using cache
---> bf688ba3569b
Step 5/5 : RUN rustc --version &&     pip3 install --no-cache-dir --verbose setuptools-rust
---> Using cache
---> d55885dd8d41
Successfully built d55885dd8d41
Successfully tagged my_container:r36.4.0-rust
-- Building container my_container:r36.4.0-transformers
DOCKER_BUILDKIT=0 docker build --network=host --tag my_container:r36.4.0-transformers
--file /home/jane/Code/jetson-containers/packages/llm/transformers/Dockerfile
--build-arg BASE_IMAGE=my_container:r36.4.0-rust
/home/jane/Code/jetson-containers/packages/llm/transformers
2>&1 | tee /home/jane/Code/jetson-containers/logs/20241009_134747/build/my_container_r36.4.0-transformers.txt; exit ${PIPESTATUS[0]}
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
environment-variable.
Sending build context to Docker daemon  35.33kB
Step 1/7 : ARG BASE_IMAGE
Step 2/7 : FROM ${BASE_IMAGE}
---> d55885dd8d41
Step 3/7 : ARG TRANSFORMERS_PACKAGE=transformers     TRANSFORMERS_VERSION=https://pypi.org/pypi/transformers/json
---> Using cache
---> 9f536c90bdd4
Step 4/7 : ADD ${TRANSFORMERS_VERSION} /tmp/transformers_version.json
---> Using cache
---> ba934d0bb9c4
Step 5/7 : RUN pip3 install --no-cache-dir --verbose accelerate &&     pip3 install --no-cache-dir --verbose sentencepiece &&     pip3 install --no-cache-dir --verbose optimum &&         pip3 uninstall -y transformers &&     pip3 install --no-cache-dir --verbose ${TRANSFORMERS_PACKAGE} &&         PYTHON_ROOT=pip3 show transformers | grep Location: | cut -d' ' -f2 &&     sed -i 's|torch.distributed.is_initialized|torch.distributed.is_available|g' -i ${PYTHON_ROOT}/transformers/modeling_utils.py
---> Running in 3cc8b2f2661b
Using pip 24.2 from /usr/local/lib/python3.10/dist-packages/pip (python 3.10)
Looking in indexes: http://jetson.webredirect.org/jp6/cu126
Collecting accelerate
Downloading http://jetson.webredirect.org/root/pypi/%2Bf/6fc/65cd7a43c4153/accelerate-1.0.0-py3-none-any.whl (330 kB)
Requirement already satisfied: numpy<3.0.0,>=1.17 in /usr/local/lib/python3.10/dist-packages (from accelerate) (1.26.4)
Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from accelerate) (24.1)
Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from accelerate) (6.0.0)
Requirement already satisfied: pyyaml in /usr/local/lib/python3.10/dist-packages (from accelerate) (6.0.2)
Requirement already satisfied: torch>=1.10.0 in /usr/local/lib/python3.10/dist-packages (from accelerate) (2.4.0)
Requirement already satisfied: huggingface-hub>=0.21.0 in /usr/local/lib/python3.10/dist-packages (from accelerate) (0.25.1)
Collecting safetensors>=0.4.3 (from accelerate)
Downloading http://jetson.webredirect.org/root/pypi/%2Bf/688/5016f34bef80e/safetensors-0.4.5-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (441 kB)
Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.21.0->accelerate) (3.16.1)
Requirement already satisfied: fsspec>=2023.5.0 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.21.0->accelerate) (2024.9.0)
Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.21.0->accelerate) (2.32.3)
Requirement already satisfied: tqdm>=4.42.1 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.21.0->accelerate) (4.66.5)
Requirement already satisfied: typing-extensions>=<IP_ADDRESS> in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.21.0->accelerate) (4.12.2)
Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate) (1.13.3)
Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate) (3.3)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate) (3.1.4)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch>=1.10.0->accelerate) (2.1.5)
Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->huggingface-hub>=0.21.0->accelerate) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->huggingface-hub>=0.21.0->accelerate) (3.10)
Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->huggingface-hub>=0.21.0->accelerate) (2.2.3)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->huggingface-hub>=0.21.0->accelerate) (2024.8.30)
Requirement already satisfied: mpmath<1.4,>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from sympy->torch>=1.10.0->accelerate) (1.3.0)
Installing collected packages: safetensors, accelerate
changing mode of /usr/local/bin/accelerate to 755
changing mode of /usr/local/bin/accelerate-config to 755
changing mode of /usr/local/bin/accelerate-estimate-memory to 755
changing mode of /usr/local/bin/accelerate-launch to 755
changing mode of /usr/local/bin/accelerate-merge-weights to 755
Successfully installed accelerate-1.0.0 safetensors-0.4.5
Using pip 24.2 from /usr/local/lib/python3.10/dist-packages/pip (python 3.10)
Looking in indexes: http://jetson.webredirect.org/jp6/cu126
Collecting sentencepiece
Downloading http://jetson.webredirect.org/root/pypi/%2Bf/2fd/e4b08cfe237be/sentencepiece-0.2.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.3 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.3/1.3 MB 1.2 MB/s eta 0:00:00
Installing collected packages: sentencepiece
Successfully installed sentencepiece-0.2.0
Using pip 24.2 from /usr/local/lib/python3.10/dist-packages/pip (python 3.10)
Looking in indexes: http://jetson.webredirect.org/jp6/cu126
ERROR: Could not find a version that satisfies the requirement optimum (from versions: none)
ERROR: No matching distribution found for optimum
The command '/bin/sh -c pip3 install --no-cache-dir --verbose accelerate &&     pip3 install --no-cache-dir --verbose sentencepiece &&     pip3 install --no-cache-dir --verbose optimum &&         pip3 uninstall -y transformers &&     pip3 install --no-cache-dir --verbose ${TRANSFORMERS_PACKAGE} &&         PYTHON_ROOT=pip3 show transformers | grep Location: | cut -d' ' -f2 &&     sed -i 's|torch.distributed.is_initialized|torch.distributed.is_available|g' -i ${PYTHON_ROOT}/transformers/modeling_utils.py' returned a non-zero code: 1
Traceback (most recent call last):
File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/home/jane/Code/jetson-containers/jetson_containers/build.py", line 112, in
build_container(args.name, args.packages, args.base, args.build_flags, args.build_args, args.simulate, args.skip_tests, args.test_only, args.push, args.no_github_api)
File "/home/jane/Code/jetson-containers/jetson_containers/container.py", line 147, in build_container
status = subprocess.run(cmd.replace(NEWLINE, ' '), executable='/bin/bash', shell=True, check=True)
File "/usr/lib/python3.10/subprocess.py", line 526, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command 'DOCKER_BUILDKIT=0 docker build --network=host --tag my_container:r36.4.0-transformers --file /home/jane/Code/jetson-containers/packages/llm/transformers/Dockerfile --build-arg BASE_IMAGE=my_container:r36.4.0-rust /home/jane/Code/jetson-containers/packages/llm/transformers 2>&1 | tee /home/jane/Code/jetson-containers/logs/20241009_134747/build/my_container_r36.4.0-transformers.txt; exit ${PIPESTATUS[0]}' returned non-zero exit status 1.

So optimus is still the issue. Can you confirm that this should build (latest JetPack 6.1, Orin AGX 64GB, everything else up to date)? Because if I can't build the transformers container then I can't even get to the tensorrt part.

Hey @dusty-nv I actually spoke to soon above - I was still using my local change where I had commented out the optimus dep.
Again, my goal is to build a container with diffusers, transformers, and torch2trt (or torch_tensorrt, or some other way of using TensorRT-accelerated pytorch).
As a first step, I ensure I'm on a clean master:
jane@ubuntu:~/Code/jetson-containers$ git show HEAD
commit 0189a3455531341c79b5877f58e3b30e92fc10b2 (HEAD -> master, origin/master, origin/HEAD)
Merge: 1f5dee99 e11c43ca
Author: Dustin Franklin<EMAIL_ADDRESS>Date:   Mon Oct 7 03:33:25 2024 -0400

    Merge remote-tracking branch 'origin/dev'

jane@ubuntu:~/Code/jetson-containers$ git status
On branch master
Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean

and I try and build the transformers container:
So optimus is still the issue. Can you confirm that this should build (latest JetPack 6.1, Orin AGX 64GB, everything else up to date)? Because if I can't build the transformers container then I can't even get to the tensorrt part.

What is yout command/prompt?

Do you mean how am I invoking the jetson-containers tool to build the container? It's at the top of that log in my previous message (collapsed inside a <details> element.
jane@ubuntu:~/Code/jetson-containers$ jetson-containers build --name=my_container transformers

Or do you mean what's my shell environment or something? Here's the output of env:
jane@ubuntu:~/Code/jetson-containers$ env
SHELL=/bin/bash
SESSION_MANAGER=local/ubuntu:@/tmp/.ICE-unix/2743,unix/ubuntu:/tmp/.ICE-unix/2743
QT_ACCESSIBILITY=1
COLORTERM=truecolor
XDG_CONFIG_DIRS=/etc/xdg/xdg-ubuntu:/etc/xdg
SSH_AGENT_LAUNCHER=gnome-keyring
XDG_MENU_PREFIX=gnome-
GNOME_DESKTOP_SESSION_ID=this-is-deprecated
GTK_IM_MODULE=ibus
JETSON_CUDA_ARCH_BIN=8.7
GNOME_SHELL_SESSION_MODE=ubuntu
SSH_AUTH_SOCK=/run/user/1000/keyring/ssh
XMODIFIERS=@im=ibus
DESKTOP_SESSION=ubuntu
GTK_MODULES=gail:atk-bridge
PWD=/home/jane/Code/jetson-containers
LOGNAME=jane
XDG_SESSION_DESKTOP=ubuntu
XDG_SESSION_TYPE=x11
GPG_AGENT_INFO=/run/user/1000/gnupg/S.gpg-agent:0:1
SYSTEMD_EXEC_PID=2929
JETSON_MODULE=NVIDIA Jetson AGX Orin (64GB ram)
XAUTHORITY=/run/user/1000/gdm/Xauthority
WINDOWPATH=2
HOME=/home/jane
USERNAME=jane
IM_CONFIG_PHASE=1
LANG=en_AU.UTF-8
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:
XDG_CURRENT_DESKTOP=ubuntu:GNOME
VTE_VERSION=6800
GNOME_TERMINAL_SCREEN=/org/gnome/Terminal/screen/e7d5b141_7b4e_4cf1_be03_098c8ea617a8
JETSON_L4T=36.4.0
JETSON_MODEL=NVIDIA Jetson AGX Orin Developer Kit
JETSON_SERIAL_NUMBER=1421524282093
HF_HOME=/ssd/huggingface
LESSCLOSE=/usr/bin/lesspipe %s %s
JETSON_SOC=tegra234
XDG_SESSION_CLASS=user
TERM=xterm-256color
LESSOPEN=| /usr/bin/lesspipe %s
USER=jane
GNOME_TERMINAL_SERVICE=:1.284
DISPLAY=:1
SHLVL=1
QT_IM_MODULE=ibus
LD_LIBRARY_PATH=/usr/local/cuda-12.6/lib64:
XDG_RUNTIME_DIR=/run/user/1000
XDG_DATA_DIRS=/usr/share/ubuntu:/usr/share/gnome:/usr/local/share/:/usr/share/:/var/lib/snapd/desktop
PATH=/home/jane/.rye/shims:/home/jane/.local/bin:/usr/local/cuda-12.6/bin:/home/jane/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin
GDMSESSION=ubuntu
JETSON_P_NUMBER=p3701-0005
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus
JETSON_JETPACK=6.1
_=/usr/bin/env
OLDPWD=/home/jane/Code/storytellers

Do you mean how am I invoking the jetson-containers tool to build the container? It's at the top of that log in my previous message (collapsed inside a <details> element.
jane@ubuntu:~/Code/jetson-containers$ jetson-containers build --name=my_container transformers

Or do you mean what's my shell environment or something? Here's the output of env:
jane@ubuntu:~/Code/jetson-containers$ env
SHELL=/bin/bash
SESSION_MANAGER=local/ubuntu:@/tmp/.ICE-unix/2743,unix/ubuntu:/tmp/.ICE-unix/2743
QT_ACCESSIBILITY=1
COLORTERM=truecolor
XDG_CONFIG_DIRS=/etc/xdg/xdg-ubuntu:/etc/xdg
SSH_AGENT_LAUNCHER=gnome-keyring
XDG_MENU_PREFIX=gnome-
GNOME_DESKTOP_SESSION_ID=this-is-deprecated
GTK_IM_MODULE=ibus
JETSON_CUDA_ARCH_BIN=8.7
GNOME_SHELL_SESSION_MODE=ubuntu
SSH_AUTH_SOCK=/run/user/1000/keyring/ssh
XMODIFIERS=@im=ibus
DESKTOP_SESSION=ubuntu
GTK_MODULES=gail:atk-bridge
PWD=/home/jane/Code/jetson-containers
LOGNAME=jane
XDG_SESSION_DESKTOP=ubuntu
XDG_SESSION_TYPE=x11
GPG_AGENT_INFO=/run/user/1000/gnupg/S.gpg-agent:0:1
SYSTEMD_EXEC_PID=2929
JETSON_MODULE=NVIDIA Jetson AGX Orin (64GB ram)
XAUTHORITY=/run/user/1000/gdm/Xauthority
WINDOWPATH=2
HOME=/home/jane
USERNAME=jane
IM_CONFIG_PHASE=1
LANG=en_AU.UTF-8
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:
XDG_CURRENT_DESKTOP=ubuntu:GNOME
VTE_VERSION=6800
GNOME_TERMINAL_SCREEN=/org/gnome/Terminal/screen/e7d5b141_7b4e_4cf1_be03_098c8ea617a8
JETSON_L4T=36.4.0
JETSON_MODEL=NVIDIA Jetson AGX Orin Developer Kit
JETSON_SERIAL_NUMBER=1421524282093
HF_HOME=/ssd/huggingface
LESSCLOSE=/usr/bin/lesspipe %s %s
JETSON_SOC=tegra234
XDG_SESSION_CLASS=user
TERM=xterm-256color
LESSOPEN=| /usr/bin/lesspipe %s
USER=jane
GNOME_TERMINAL_SERVICE=:1.284
DISPLAY=:1
SHLVL=1
QT_IM_MODULE=ibus
LD_LIBRARY_PATH=/usr/local/cuda-12.6/lib64:
XDG_RUNTIME_DIR=/run/user/1000
XDG_DATA_DIRS=/usr/share/ubuntu:/usr/share/gnome:/usr/local/share/:/usr/share/:/var/lib/snapd/desktop
PATH=/home/jane/.rye/shims:/home/jane/.local/bin:/usr/local/cuda-12.6/bin:/home/jane/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin
GDMSESSION=ubuntu
JETSON_P_NUMBER=p3701-0005
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus
JETSON_JETPACK=6.1
_=/usr/bin/env
OLDPWD=/home/jane/Code/storytellers

in which branch are you?
CUDA_VERSION=12.6 PYTHON_VERSION=3.10 jetson-containers build transformers

I'm on master.
jane@ubuntu:~/Code/jetson-containers$ git status
On branch master
Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean
jane@ubuntu:~/Code/jetson-containers$ git remote show origin
* remote origin
  Fetch URL: https://github.com/dusty-nv/jetson-containers
  Push  URL: https://github.com/dusty-nv/jetson-containers
  HEAD branch: master
  Remote branches:
    L4T-R32.4.2 tracked
    L4T-R32.4.3 tracked
    L4T-R32.4.4 tracked
    L4T-R32.5.0 tracked
    L4T-R32.6.1 tracked
    L4T-R32.7.1 tracked
    L4T-R34.1.0 tracked
    L4T-R34.1.1 tracked
    L4T-R35.1.0 tracked
    L4T-R35.2.1 tracked
    L4T-R35.3.1 tracked
    L4T-R35.4.1 tracked
    L4T-R36.2.0 tracked
    L4T-R36.3.0 tracked
    crossformer tracked
    dev         tracked
    docs        tracked
    jp6         tracked
    legacy      tracked
    master      tracked
    trt-llm     tracked
  Local branch configured for 'git pull':
    master merges with remote master
  Local ref configured for 'git push':
    master pushes to master (up to date)

Thanks for your help, btw. I can confirm that I just tried literally the same command:
CUDA_VERSION=12.6 PYTHON_VERSION=3.10 jetson-containers build transformers

and I still get the error about optimum
ERROR: Could not find a version that satisfies the requirement optimum (from versions: none)
ERROR: No matching distribution found for optimum

Thanks for your help, btw. I can confirm that I just tried literally the same command:
CUDA_VERSION=12.6 PYTHON_VERSION=3.10 jetson-containers build transformers

and I still get the error about optimum
ERROR: Could not find a version that satisfies the requirement optimum (from versions: none)
ERROR: No matching distribution found for optimum

i'm on master too, wtf jajaj it is strange, optimum has arm wheels

Yes it is strange, I have still never gotten this error, and had changed how it was installed.  @benswift just comment it out in the transformers dockerfile

Yes it is strange, I have still never gotten this error, and had changed how it was installed. @benswift just comment it out in the transformers dockerfile

wow, you are here, hero

Thanks all. I'm open to building stuff manually if there's any doco (doesn't have to be super polished, just enough for me to fill-in-the-gaps). Or @dusty-nv is there anything else you think I should try (dev branch, run further diagnostics, etc)?

Thanks all. I'm open to building stuff manually if there's any doco (doesn't have to be super polished, just enough for me to fill-in-the-gaps). Or @dusty-nv is there anything else you think I should try (dev branch, run further diagnostics, etc)?

If you have nothing changed,
Could you start from inititial steps?

Clone the repo again
clean all your images : sudo system prune -a
CUDA_VERSION=12.6 PYTHON_VERSION=3.10 jetson-containers build transformers

yep, will do

@dusty-nv Now I understand the error, we use a newer version of transformer, 4.45 (https://github.com/huggingface/optimum/commit/049b00f61c9bb17bd2b20a3b77d04cc4c0f20d86) which optimum has not yet released to be compatible (tags/release), therefore, it does not install. Maybe, here we need your opinion whether to keep optimum until now and will update it, or create build to make transformer:4.44 & optimum:4.44 type. or another solution

Oh, that's good news. Do I have to wait for the jetson.webredirect.org pip cache to be updated? I'm not sure if that's a nightly cronjob or has to be triggered manually by an admin.

Oh, that's good news. Do I have to wait for the jetson.webredirect.org pip cache to be updated? I'm not sure if that's a nightly cronjob or has to be triggered manually by an admin.

it is updated manually if the index is cuda. For optimum is on root http://jetson.webredirect.org/root/pypi/optimum/stable

Yea, optimum and transformers don't need CUDA/NVCC compilation directly (all the CUDA stuff they use is in dependencies), so I don't need to have wheels for them.  The 'root index' is just a mirror of pypi.   That's interesting that even though we don't pin transformers version in the dockerfile, it was still failing to resolve optimum...

Holy molly, 3 months later and it's still happening.

This is straight from your tutorial pages for text-generation-webui (https://www.jetson-ai-lab.com/tutorial_text-generation.html). At least have the decency to remove that content if it's powered by the same corporate overlords.