TF 2.16.1 Fails to work with GPUs

Issue type
Bug
Have you reproduced the bug with TensorFlow Nightly?
No
Source
binary
TensorFlow version
TF 2.16.1
Custom code
No
OS platform and distribution
Linux Ubuntu 22.04.4 LTS
Mobile device
No response
Python version
3.10.12
Bazel version
No response
GCC/compiler version
No response
CUDA/cuDNN version
12.4
GPU model and memory
No response
Current behavior?
I created a python venv in which I installed TF 2.16.1 following your instructions: pip install tensorflow
When I run python, import tf,  and issue tf.config.list_physical_devices('GPU')
I get an empty list  [ ]
I created another python venv, installed TF 2.16.1, only this time with the instructions:
python3 -m pip install tensorflow[and-cuda]
When I run that version, import tensorflow as tf, and  issue
tf.config.list_physical_devices('GPU')
I also get an empty list.
BTW, I have no problems running  on my box TF 2.15.1 with GPUs. Julia also works just fine with GPUs and so does PyTorch.
the
Standalone code to reproduce the issue
Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
2024-03-09 19:15:45.018171: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-03-09 19:15:50.412646: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
>>> tf.__version__
'2.16.1'

tf.config.list_physical_devices('GPU')
2024-03-09 19:16:28.923792: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2024-03-09 19:16:29.078379: W tensorflow/core/common_runtime/gpu/gpu_device.cc:2251] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
[]
>>>

Relevant log output
No response

It does not work with python=3.12.2 either. Same error.

The same error on bare Ubuntu and WSL2 2.15 works without any problems with python 3.11

I have the same problem with Ubuntu 22.04.4 with the following environment:

tensorflow==2.16.1
Python 3.10.13 (main, Sep 11 2023, 13:44:35) [GCC 11.2.0] on linux
cuDNN <IP_ADDRESS>
gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0

nvcc --version output:
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Sep_21_10:33:58_PDT_2022
Cuda compilation tools, release 11.8, V11.8.89
Build cuda_11.8.r11.8/compiler.31833905_0

I'm not sure if this is the root cause, but I resolved my own issue which also surfaced as a "Cannot dlopen some GPU libraries." error when trying to run python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"
To resolve my issue, I followed the tested build versions here:
https://www.tensorflow.org/install/source#gpu
and I needed to update my existing installations from cuDNN 9 -> 8.9 and CUDA 12.4->12.3
When you're on an NVIDIA download page like this one for CUDA Toolkit, don't just download the latest version. See previous versions by hitting "Archive of Previous CUDA Releases"
@JuanVargas can you try uninstalling your existing CUDA installation to a tested build configuration for TF 2.16 by downgrading to CUDA 12.3?
I followed this post to uninstall my existing cuda installation:
https://askubuntu.com/questions/530043/removing-nvidia-cuda-toolkit-and-installing-new-one
@DiegoMont can you try upgrading your cuDNN to 8.9 and CUDA to 12.3?

I am having the same issue. Brand new Ubuntu 22.04 WSL2 image. Blank Conda environment with either python 3.12.* or 3.11.* fails to correctly setup tensorflow for GPU use when following the recommended:
pip install tensorflow[and cuda]
Trying to list the physical devices results in:
2024-03-11 02:00:00.294704: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-03-11 02:00:00.709325: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2024-03-11 02:00:01.180225: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:984] could not open file to read NUMA node: /sys/bus/pci/devices/0000:2d:00.0/numa_node
Your kernel may have been built without NUMA support.
2024-03-11 02:00:01.180445: W tensorflow/core/common_runtime/gpu/gpu_device.cc:2251] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
[]

cuDNN 8.9.*
Cuda 12.3
Tensorflow 2.16.1

Is this a new issue caused by the fact that it doesn't appear that any system cuda needs to be separately installed in WSL2 anymore. I certainly didn't install one manually and yet nvidia-smi is happily reporting cuda version 12.3. It probably comes down to some env paths not set correctly but playing around with $CUDA_PATH and guessing the location within the conda environment has not resolved anything. Pytorch GPU visibility works as expected.

Hi @JuanVargas ,
For GPU package you need to ensure the installation of CUDA driver which can be verified with nvidia-smi command. Then you need to install TF-cuda package with pip install tensorflow[and-cuda] which automatically installs required cuda/cudnn libraries.
I have checked in colab and able to detect GPU.Please refer attached gist.

pip install "tensorflow[and-cuda]==2.16.1"

Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already satisfied: tensorflow==2.16.1 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow[and-cuda]==2.16.1) (2.16.1)
Requirement already satisfied: absl-py>=1.0.0 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (2.1.0)
Requirement already satisfied: astunparse>=1.6.0 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (1.6.3)
Requirement already satisfied: flatbuffers>=23.5.26 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (24.3.7)
Requirement already satisfied: gast!=0.5.0,!=0.5.1,!=0.5.2,>=0.2.1 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (0.5.4)
Requirement already satisfied: google-pasta>=0.1.1 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (0.2.0)
Requirement already satisfied: h5py>=3.10.0 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (3.10.0)
Requirement already satisfied: libclang>=13.0.0 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (16.0.6)
Requirement already satisfied: ml-dtypes~=0.3.1 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (0.3.2)
Requirement already satisfied: opt-einsum>=2.3.2 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (3.3.0)
Requirement already satisfied: packaging in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (24.0)
Requirement already satisfied: protobuf!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.20.3 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (4.25.3)
Requirement already satisfied: requests<3,>=2.21.0 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (2.31.0)
Requirement already satisfied: setuptools in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (69.1.1)
Requirement already satisfied: six>=1.12.0 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (1.16.0)
Requirement already satisfied: termcolor>=1.1.0 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (2.4.0)
Requirement already satisfied: typing-extensions>=3.6.6 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (4.10.0)
Requirement already satisfied: wrapt>=1.11.0 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (1.16.0)
Requirement already satisfied: grpcio<2.0,>=1.24.3 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (1.62.1)
Requirement already satisfied: tensorboard<2.17,>=2.16 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (2.16.2)
Requirement already satisfied: keras>=3.0.0 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (3.0.5)
Requirement already satisfied: tensorflow-io-gcs-filesystem>=0.23.1 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (0.36.0)
Requirement already satisfied: numpy<2.0.0,>=1.23.5 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (1.26.4)
Requirement already satisfied: nvidia-cublas-cu12==<IP_ADDRESS> in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow[and-cuda]==2.16.1) (<IP_ADDRESS>)
Requirement already satisfied: nvidia-cuda-cupti-cu12==12.3.101 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow[and-cuda]==2.16.1) (12.3.101)
Requirement already satisfied: nvidia-cuda-nvcc-cu12==12.3.107 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow[and-cuda]==2.16.1) (12.3.107)
Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.3.107 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow[and-cuda]==2.16.1) (12.3.107)
Requirement already satisfied: nvidia-cuda-runtime-cu12==12.3.101 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow[and-cuda]==2.16.1) (12.3.101)
Requirement already satisfied: nvidia-cudnn-cu12==<IP_ADDRESS> in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow[and-cuda]==2.16.1) (<IP_ADDRESS>)
Requirement already satisfied: nvidia-cufft-cu12==<IP_ADDRESS> in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow[and-cuda]==2.16.1) (<IP_ADDRESS>)
Requirement already satisfied: nvidia-curand-cu12==<IP_ADDRESS> in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow[and-cuda]==2.16.1) (<IP_ADDRESS>)
Requirement already satisfied: nvidia-cusolver-cu12==<IP_ADDRESS> in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow[and-cuda]==2.16.1) (<IP_ADDRESS>)
Requirement already satisfied: nvidia-cusparse-cu12==<IP_ADDRESS> in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow[and-cuda]==2.16.1) (<IP_ADDRESS>)
Requirement already satisfied: nvidia-nccl-cu12==2.19.3 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow[and-cuda]==2.16.1) (2.19.3)
Requirement already satisfied: nvidia-nvjitlink-cu12==12.3.101 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorflow[and-cuda]==2.16.1) (12.3.101)
Requirement already satisfied: wheel<1.0,>=0.23.0 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from astunparse>=1.6.0->tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (0.42.0)
Requirement already satisfied: rich in ./miniconda3/envs/tf/lib/python3.11/site-packages (from keras>=3.0.0->tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (13.7.1)
Requirement already satisfied: namex in ./miniconda3/envs/tf/lib/python3.11/site-packages (from keras>=3.0.0->tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (0.0.7)
Requirement already satisfied: dm-tree in ./miniconda3/envs/tf/lib/python3.11/site-packages (from keras>=3.0.0->tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (0.1.8)
Requirement already satisfied: charset-normalizer<4,>=2 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from requests<3,>=2.21.0->tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from requests<3,>=2.21.0->tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (3.6)
Requirement already satisfied: urllib3<3,>=1.21.1 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from requests<3,>=2.21.0->tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (2.2.1)
Requirement already satisfied: certifi>=2017.4.17 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from requests<3,>=2.21.0->tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (2024.2.2)
Requirement already satisfied: markdown>=2.6.8 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorboard<2.17,>=2.16->tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (3.5.2)
Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorboard<2.17,>=2.16->tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (0.7.2)
Requirement already satisfied: werkzeug>=1.0.1 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from tensorboard<2.17,>=2.16->tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (3.0.1)
Requirement already satisfied: MarkupSafe>=2.1.1 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from werkzeug>=1.0.1->tensorboard<2.17,>=2.16->tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (2.1.5)
Requirement already satisfied: markdown-it-py>=2.2.0 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from rich->keras>=3.0.0->tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (3.0.0)
Requirement already satisfied: pygments<3.0.0,>=2.13.0 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from rich->keras>=3.0.0->tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (2.17.2)
Requirement already satisfied: mdurl~=0.1 in ./miniconda3/envs/tf/lib/python3.11/site-packages (from markdown-it-py>=2.2.0->rich->keras>=3.0.0->tensorflow==2.16.1->tensorflow[and-cuda]==2.16.1) (0.1.2)

nvidia-smi

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.60.01              Driver Version: 551.76         CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 4070 Ti     On  |   00000000:01:00.0  On |                  N/A |
|  0%   39C    P5             10W /  285W |    4334MiB /  12282MiB |     13%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A        41      G   /Xwayland                                   N/A      |
+-----------------------------------------------------------------------------------------+

python3

Python 3.11.7 (main, Dec 15 2023, 18:12:31) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
print(tf.config.list_physical_devices('GPU'))2024-03-11 09:36:29.601060: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-03-11 09:36:29.921637: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-03-11 09:36:30.793353: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
>>> print(tf.config.list_physical_devices('GPU'))
2024-03-11 09:36:33.878560: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:984] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2024-03-11 09:36:33.980099: W tensorflow/core/common_runtime/gpu/gpu_device.cc:2251] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
[]
>>>

nvcc -V

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Tue_Feb_27_16:19:38_PST_2024
Cuda compilation tools, release 12.4, V12.4.99
Build cuda_12.4.r12.4/compiler.33961263_0

got it work :) first
https://developer.nvidia.com/rdp/cudnn-archive?source=post_page-----bfbeb77e7c89--------------------------------
then download Local Installer for Ubuntu22.04 x86_64 (Deb)
unpack and install libcudnn8_<IP_ADDRESS>-1+cuda12.2_amd64.deb
` sudo dpkg -i libcudnn8_<IP_ADDRESS>-1+cuda12.2_amd64.deb
Selecting previously unselected package libcudnn8.
(Reading database ... 47318 files and directories currently installed.)
Preparing to unpack libcudnn8_<IP_ADDRESS>-1+cuda12.2_amd64.deb ...
Unpacking libcudnn8 (<IP_ADDRESS>-1+cuda12.2) ...
Setting up libcudnn8 (<IP_ADDRESS>-1+cuda12.2) ...
`
python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"

2024-03-11 10:27:47.879686: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-03-11 10:27:47.909157: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-03-11 10:27:48.316717: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2024-03-11 10:27:48.664469: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:984] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2024-03-11 10:27:48.688059: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:984] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2024-03-11 10:27:48.688111: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:984] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]

Hi Krzysztof
I visited the  site
https://developer.nvidia.com/rdp/cudnn-archive?source=post_page-----bfbeb77e7c89--------------------------------
where I found an entry listed as " Local Installer for UBuntu22.04
x86_64(Deb)" which I downloaded.
Unfortunately what I got is a package named
"cudnn-local-repo-ubuntu2204-<IP_ADDRESS>_1.0-1_amd64.deb"
which is not the same as the name you suggest in your message, which is "
libcudnn8_<IP_ADDRESS>-1+cuda12.2_amd64.deb"
I assume what you meant is to get the libcudnn8_<IP_ADDRESS>*amd64.deb and
the cuda12.2_amd64.deb separately and install both.
I have CUDA 12.4. I will not go back to trying to make TF 2.16.1 work with
older versions of CUDA (12.2 or 12.3) because sooner or later
the TF team will have to produce a version with the updated version of
CUDA. IMHO, rather than us wasting time going back in versions,
the TF beak should invest time going forward to update TF to the current
CUDA version.
Thank you, Juan
On Mon, Mar 11, 2024 at 5:30 AM Krzysztof Radzikowski <
@.***> wrote:

got it work :) first
https://developer.nvidia.com/rdp/cudnn-archive?source=post_page-----bfbeb77e7c89--------------------------------
then download Local Installer for Ubuntu22.04 x86_64 (Deb)
https://developer.nvidia.com/downloads/compute/cudnn/secure/8.9.7/local_installers/12.x/cudnn-local-repo-ubuntu2204-<IP_ADDRESS>_1.0-1_amd64.deb/
unpack and install libcudnn8_<IP_ADDRESS>-1+cuda12.2_amd64.deb
`
sudo dpkg -i libcudnn8_<IP_ADDRESS>-1+cuda12.2_amd64.deb
Selecting previously unselected package libcudnn8.
(Reading database ... 47318 files and directories currently installed.)
Preparing to unpack libcudnn8_<IP_ADDRESS>-1+cuda12.2_amd64.deb ...
Unpacking libcudnn8 (<IP_ADDRESS>-1+cuda12.2) ...
Setting up libcudnn8 (<IP_ADDRESS>-1+cuda12.2) ...
`
python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"
2024-03-11 10:27:47.879686: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable TF_ENABLE_ONEDNN_OPTS=0.
2024-03-11 10:27:47.909157: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-03-11 10:27:48.316717: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2024-03-11 10:27:48.664469: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:984] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2024-03-11 10:27:48.688059: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:984] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2024-03-11 10:27:48.688111: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:984] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]
—
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/63362#issuecomment-1987970615,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAGK34PV2IIF5FUZ73EPKOTYXV2SZAVCNFSM6AAAAABEOPWBC2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBXHE3TANRRGU
.
You are receiving this because you were mentioned.Message ID:
@.***>

It's just tensorflow can't see the Cuda libraries.
Instal tensorflow[and-cuda] and add this to your .bashrc or conda
activation script
NVIDIA_PACKAGE_DIR="$CONDA_PREFIX/lib/python3.12/site-packages/nvidia"
for dir in $NVIDIA_PACKAGE_DIR/*; do
if [ -d "$dir/lib" ]; then
export LD_LIBRARY_PATH="$dir/lib:$LD_LIBRARY_PATH"
fi
done
On Mon, Mar 11, 2024, 7:04 a.m. Juan E. Vargas @.***>
wrote:

Hi Krzysztof
I visited the site
https://developer.nvidia.com/rdp/cudnn-archive?source=post_page-----bfbeb77e7c89--------------------------------
where I found an entry listed as " Local Installer for UBuntu22.04
x86_64(Deb)" which I downloaded.
Unfortunately what I got is a package named
"cudnn-local-repo-ubuntu2204-<IP_ADDRESS>_1.0-1_amd64.deb"
which is not the same as the name you suggest in your message, which is "
libcudnn8_<IP_ADDRESS>-1+cuda12.2_amd64.deb"
I assume what you meant is to get the libcudnn8_<IP_ADDRESS>*amd64.deb and
the cuda12.2_amd64.deb separately and install both.
I have CUDA 12.4. I will not go back to trying to make TF 2.16.1 work with
older versions of CUDA (12.2 or 12.3) because sooner or later
the TF team will have to produce a version with the updated version of
CUDA. IMHO, rather than us wasting time going back in versions,
the TF beak should invest time going forward to update TF to the current
CUDA version.
Thank you, Juan
On Mon, Mar 11, 2024 at 5:30 AM Krzysztof Radzikowski <
@.***> wrote:

got it work :) first

https://developer.nvidia.com/rdp/cudnn-archive?source=post_page-----bfbeb77e7c89--------------------------------

then download Local Installer for Ubuntu22.04 x86_64 (Deb)
<
https://developer.nvidia.com/downloads/compute/cudnn/secure/8.9.7/local_installers/12.x/cudnn-local-repo-ubuntu2204-<IP_ADDRESS>_1.0-1_amd64.deb/>

unpack and install libcudnn8_<IP_ADDRESS>-1+cuda12.2_amd64.deb
`
sudo dpkg -i libcudnn8_<IP_ADDRESS>-1+cuda12.2_amd64.deb
Selecting previously unselected package libcudnn8.
(Reading database ... 47318 files and directories currently installed.)
Preparing to unpack libcudnn8_<IP_ADDRESS>-1+cuda12.2_amd64.deb ...
Unpacking libcudnn8 (<IP_ADDRESS>-1+cuda12.2) ...
Setting up libcudnn8 (<IP_ADDRESS>-1+cuda12.2) ...
`
python3 -c "import tensorflow as tf;
print(tf.config.list_physical_devices('GPU'))"
2024-03-11 10:27:47.879686: I tensorflow/core/util/port.cc:113] oneDNN
custom operations are on. You may see slightly different numerical results
due to floating-point round-off errors from different computation orders.
To turn them off, set the environment variable TF_ENABLE_ONEDNN_OPTS=0.
2024-03-11 10:27:47.909157: I
tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary
is optimized to use available CPU instructions in performance-critical
operations.
To enable the following instructions: AVX2 AVX_VNNI FMA, in other
operations, rebuild TensorFlow with the appropriate compiler flags.
2024-03-11 10:27:48.316717: W
tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could
not find TensorRT
2024-03-11 10:27:48.664469: I
external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:984] could not
open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2024-03-11 10:27:48.688059: I
external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:984] could not
open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2024-03-11 10:27:48.688111: I
external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:984] could not
open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]
—
Reply to this email directly, view it on GitHub
<
https://github.com/tensorflow/tensorflow/issues/63362#issuecomment-1987970615>,

or unsubscribe
<
https://github.com/notifications/unsubscribe-auth/AAGK34PV2IIF5FUZ73EPKOTYXV2SZAVCNFSM6AAAAABEOPWBC2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBXHE3TANRRGU>

.
You are receiving this because you were mentioned.Message ID:
@.***>

—
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/63362#issuecomment-1988394624,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AZRPJAGFJU5ZGBGHOSUK6DTYXWTU3AVCNFSM6AAAAABEOPWBC2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBYGM4TINRSGQ
.
You are receiving this because you commented.Message ID:
@.***>

will try that and will let you know. Thank you for the suggestion. Juan
On Mon, Mar 11, 2024 at 10:52 AM Shayan Shahrokhi @.***>
wrote:

It's just tensorflow can't see the Cuda libraries.
Instal tensorflow[and-cuda] and add this to your .bashrc or conda
activation script
NVIDIA_PACKAGE_DIR="$CONDA_PREFIX/lib/python3.12/site-packages/nvidia"
for dir in $NVIDIA_PACKAGE_DIR/*; do
if [ -d "$dir/lib" ]; then
export LD_LIBRARY_PATH="$dir/lib:$LD_LIBRARY_PATH"
fi
done
On Mon, Mar 11, 2024, 7:04 a.m. Juan E. Vargas @.***>
wrote:

Hi Krzysztof
I visited the site

https://developer.nvidia.com/rdp/cudnn-archive?source=post_page-----bfbeb77e7c89--------------------------------

where I found an entry listed as " Local Installer for UBuntu22.04
x86_64(Deb)" which I downloaded.
Unfortunately what I got is a package named
"cudnn-local-repo-ubuntu2204-<IP_ADDRESS>_1.0-1_amd64.deb"
which is not the same as the name you suggest in your message, which is
"
libcudnn8_<IP_ADDRESS>-1+cuda12.2_amd64.deb"
I assume what you meant is to get the libcudnn8_<IP_ADDRESS>*amd64.deb and
the cuda12.2_amd64.deb separately and install both.
I have CUDA 12.4. I will not go back to trying to make TF 2.16.1 work
with
older versions of CUDA (12.2 or 12.3) because sooner or later
the TF team will have to produce a version with the updated version of
CUDA. IMHO, rather than us wasting time going back in versions,
the TF beak should invest time going forward to update TF to the current
CUDA version.
Thank you, Juan
On Mon, Mar 11, 2024 at 5:30 AM Krzysztof Radzikowski <
@.***> wrote:

got it work :) first

https://developer.nvidia.com/rdp/cudnn-archive?source=post_page-----bfbeb77e7c89--------------------------------

then download Local Installer for Ubuntu22.04 x86_64 (Deb)
<

https://developer.nvidia.com/downloads/compute/cudnn/secure/8.9.7/local_installers/12.x/cudnn-local-repo-ubuntu2204-<IP_ADDRESS>_1.0-1_amd64.deb/>

unpack and install libcudnn8_<IP_ADDRESS>-1+cuda12.2_amd64.deb
`
sudo dpkg -i libcudnn8_<IP_ADDRESS>-1+cuda12.2_amd64.deb
Selecting previously unselected package libcudnn8.
(Reading database ... 47318 files and directories currently
installed.)
Preparing to unpack libcudnn8_<IP_ADDRESS>-1+cuda12.2_amd64.deb ...
Unpacking libcudnn8 (<IP_ADDRESS>-1+cuda12.2) ...
Setting up libcudnn8 (<IP_ADDRESS>-1+cuda12.2) ...
`
python3 -c "import tensorflow as tf;
print(tf.config.list_physical_devices('GPU'))"
2024-03-11 10:27:47.879686: I tensorflow/core/util/port.cc:113] oneDNN
custom operations are on. You may see slightly different numerical
results
due to floating-point round-off errors from different computation
orders.
To turn them off, set the environment variable
TF_ENABLE_ONEDNN_OPTS=0.
2024-03-11 10:27:47.909157: I
tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow
binary
is optimized to use available CPU instructions in performance-critical
operations.
To enable the following instructions: AVX2 AVX_VNNI FMA, in other
operations, rebuild TensorFlow with the appropriate compiler flags.
2024-03-11 10:27:48.316717: W
tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning:
Could
not find TensorRT
2024-03-11 10:27:48.664469: I
external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:984] could
not
open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2024-03-11 10:27:48.688059: I
external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:984] could
not
open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2024-03-11 10:27:48.688111: I
external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:984] could
not
open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]
—
Reply to this email directly, view it on GitHub
<

https://github.com/tensorflow/tensorflow/issues/63362#issuecomment-1987970615>,

or unsubscribe
<

https://github.com/notifications/unsubscribe-auth/AAGK34PV2IIF5FUZ73EPKOTYXV2SZAVCNFSM6AAAAABEOPWBC2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBXHE3TANRRGU>

.
You are receiving this because you were mentioned.Message ID:
@.***>

—
Reply to this email directly, view it on GitHub
<
https://github.com/tensorflow/tensorflow/issues/63362#issuecomment-1988394624>,

or unsubscribe
<
https://github.com/notifications/unsubscribe-auth/AZRPJAGFJU5ZGBGHOSUK6DTYXWTU3AVCNFSM6AAAAABEOPWBC2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBYGM4TINRSGQ>

.
You are receiving this because you commented.Message ID:
@.***>

—
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/63362#issuecomment-1988630226,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAGK34PLSBZFKQ5AQKIZ7X3YXXAMRAVCNFSM6AAAAABEOPWBC2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBYGYZTAMRSGY
.
You are receiving this because you were mentioned.Message ID:
@.***>

Hi Krzysztof I visited the site https://developer.nvidia.com/rdp/cudnn-archive?source=post_page-----bfbeb77e7c89-------------------------------- where I found an entry listed as " Local Installer for UBuntu22.04 x86_64(Deb)" which I downloaded. Unfortunately what I got is a package named "cudnn-local-repo-ubuntu2204-<IP_ADDRESS>_1.0-1_amd64.deb" which is not the same as the name you suggest in your message, which is " libcudnn8_<IP_ADDRESS>-1+cuda12.2_amd64.deb" I assume what you meant is to get the libcudnn8_<IP_ADDRESS>amd64.deb and the cuda12.2_amd64.deb separately and install both. I have CUDA 12.4. I will not go back to trying to make TF 2.16.1 work with older versions of CUDA (12.2 or 12.3) because sooner or later the TF team will have to produce a version with the updated version of CUDA. IMHO, rather than us wasting time going back in versions, the TF beak should invest time going forward to update TF to the current CUDA version. Thank you, Juan
…
On Mon, Mar 11, 2024 at 5:30 AM Krzysztof Radzikowski < @.> wrote: got it work :) first https://developer.nvidia.com/rdp/cudnn-archive?source=post_page-----bfbeb77e7c89-------------------------------- then download Local Installer for Ubuntu22.04 x86_64 (Deb) https://developer.nvidia.com/downloads/compute/cudnn/secure/8.9.7/local_installers/12.x/cudnn-local-repo-ubuntu2204-<IP_ADDRESS>_1.0-1_amd64.deb/ unpack and install libcudnn8_<IP_ADDRESS>-1+cuda12.2_amd64.deb sudo dpkg -i libcudnn8_<IP_ADDRESS>-1+cuda12.2_amd64.deb Selecting previously unselected package libcudnn8. (Reading database ... 47318 files and directories currently installed.) Preparing to unpack libcudnn8_<IP_ADDRESS>-1+cuda12.2_amd64.deb ... Unpacking libcudnn8 (<IP_ADDRESS>-1+cuda12.2) ... Setting up libcudnn8 (<IP_ADDRESS>-1+cuda12.2) ... python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))" 2024-03-11 10:27:47.879686: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable TF_ENABLE_ONEDNN_OPTS=0. 2024-03-11 10:27:47.909157: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. 2024-03-11 10:27:48.316717: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT 2024-03-11 10:27:48.664469: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:984] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node Your kernel may have been built without NUMA support. 2024-03-11 10:27:48.688059: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:984] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node Your kernel may have been built without NUMA support. 2024-03-11 10:27:48.688111: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:984] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node Your kernel may have been built without NUMA support. [PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')] — Reply to this email directly, view it on GitHub <#63362 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGK34PV2IIF5FUZ73EPKOTYXV2SZAVCNFSM6AAAAABEOPWBC2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBXHE3TANRRGU . You are receiving this because you were mentioned.Message ID: @.*>

https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/  you can get  .deb file there directrly

Thanks @sh-shahrokhi. I thought it was path related. Modified slightly to make it python version independent if you put it in your conda environment activation ([environment]/etc/activate.d/env_vars.sh).
NVIDIA_DIR=$(dirname $(dirname $(python -c "import nvidia.cudnn;print(nvidia.cudnn.__file__)")))
for dir in $NVIDIA_DIR/*; do
    if [ -d "$dir/lib" ]; then
        export LD_LIBRARY_PATH="$dir/lib:$LD_LIBRARY_PATH"
    fi
done

This is not a resolution as this post install step should not be necessary.
W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
I can't seem to do similar tricks to resolve the TensorRT issues when installed similarly into the conda environment. Any ideas?

You need

Thanks @sh-shahrokhi. I thought it was path related. Modified slightly to make it python version independent if you put it in your conda environment activation ([environment]/etc/activate.d/env_vars.sh).
NVIDIA_DIR=$(dirname $(dirname $(python -c "import nvidia.cudnn;print(nvidia.cudnn.__file__)")))
for dir in $NVIDIA_DIR/*; do
    if [ -d "$dir/lib" ]; then
        export LD_LIBRARY_PATH="$dir/lib:$LD_LIBRARY_PATH"
    fi
done

This is not a resolution as this post install step should not be necessary.
W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
I can't seem to do similar tricks to resolve the TensorRT issues when installed similarly into the conda environment. Any ideas?

I don't actually use TensorRT, but I  would check if the required .so file for it is visible to tensorflow. Maybe I would need to find the name of required file in tensorflow source code.
This actually doesn't change the fact that the new tensorflow veraion should be tested by google team before release, or the bugs should be fixed. It seems they only care about having a working docker image, not anything else.

I have given up on TensorRT. I guess I won't be using it either.

This actually doesn't change the fact that the new tensorflow version should be tested by google team before release, or the bugs should be fixed. It seems they only care about having a working docker image, not anything else.

Agreed. Installing TF has always been hit or miss and it seems that in the many years since I last used TF that hasn't changed one bit.

Well, I wasted 8hr of my Sunday on this setting up another pc from scratch. Before reverting to the old version. Now looking to move off tensor flow.

In general, we used to test RC versions before release. For example, we used to have RC0, RC1 and RC2 for TF 2.9. This gave people and downstream teams enough time to test and report issues.
It seems that 2.16.1 only had an RC0 (for 2.16.0).
The release process is (was?) like this:

cut the release branch (e.g., r2.17)
immediately trigger the release pipeline. This would create a few PRs to update version numbers, release notes, but after this step RC0 should be as close as possible to the version on master branch at the time the release branch has been cut. There should not be any code changes to the release branch at this point (except to maybe cherrypick fixes from master from hard bugs caused by cutting the branch at a wrong commit)
have at least a week of testing for downstream teams to test RC0
get fixes to discovered bugs landed on master, cherrypick them to release branch, after they are already tested on nightly releases
trigger RC1 pipeline. Again, no other code changes should occur now, except to fix bugs discovered during building
wait a week for downstream teams to test. If there are bugs, repeat the steps above for another RC, otherwise repeat the steps above for the final version.

Overall, this process would take number_of_RCs + 1 weeks with a possibility of a few more weeks of delay.
However, for 2.16 release, although the branch was cut on Feb 8th, in an entire month there has been only one RC.

I am closing this (unresolved issue) because I am told by the Keras/TF team that the issue is related to TF.

So where is the resolution to this quite annoying installation issue being tracked?It’s kind of ridiculous that a fresh install using the official method results in a system that can’t access the GPUs out of the box.As others have noted this sort of dealbreaker when setting up a new machine pushes people away from TF.On 12 Mar 2024, at 13:03, Juan E. Vargas @.***> wrote:
I am closing this (unresolved issue) because I am told by the Keras/TF team that the issue is related to TF.
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: @.***>

I closed the issue because I was asked to do so by a TF/Keras team member,
for the reasons that I stated in the closing comment. Just like you (Leigh)
I am disappointed, and Idecided not to waste more time. There are strong
reasons for some of us (users) to prefer using our own hardware. I hope the
TF/keras people get that message. Thanks, Juan

Message ID: @.***>

You were asked to close the issue on keras, because it is not a keras issue. This is tensorflow, where proper detection of the cuda libs is definitely still an issue. Please reopen.

I already opened a similar issue in TF.
On Tue, Mar 12, 2024 at 9:20 AM Sam Levang @.***> wrote:

You were asked to close the issue on keras, because it is not a keras
issue. This is tensorflow, where proper detection of the cuda libs is
definitely still an issue. Please reopen.
—
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/63362#issuecomment-1991637460,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAGK34OEUIJZCSKQOGZCGQTYX36J7AVCNFSM6AAAAABEOPWBC2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOJRGYZTONBWGA
.
You are receiving this because you modified the open/close state.Message
ID: @.***>

Oh, sorry about that!
On Tue, Mar 12, 2024 at 9:46 AM Sam Levang @.***> wrote:

Again, you are on the tensorflow repo. You closed the wrong issue.
—
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/63362#issuecomment-1991690836,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAGK34K46DAULWCUUNQHYKDYX4BLJAVCNFSM6AAAAABEOPWBC2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOJRGY4TAOBTGY
.
You are receiving this because you modified the open/close state.Message
ID: @.***>

Issue reopened

It is really frustrating indeed. The silly thing is that with JAX as the Keras backend GPU is utilized smoothly (still my RTX3060 GPU is not in the list of physical devices that TensorFlow's framework is aware of (!) ).. I guess we cannot do something else but wait until the next version release... Thank you Mr. Juan for raising the issue here.

The thing is, JAX builds against multiple CUDA versions, whereas TF always pinned to just one version

On a somewhat similar note, I have been using the tensorflow/tensorflow:2.15.0.post1-gpu-jupyter docker image as base image for the past months without issue and yesterday I rebuilt some local images and all of a sudden any model.fit will just kill the kernel. After some debugging I found out that TF 2.16.1 had been installed even when using the 2.15 base image. Where is the right place for me to report this?

Maybe the Tf at GitHub is the right place to report that
On Tue, Mar 12, 2024, 18:06 andres-zartab @.***> wrote:

On a somewhat similar note, I have been using the
tensorflow/tensorflow:2.15.0.post1-gpu-jupyter docker image as base image
for the past months without issue and yesterday I rebuilt some local images
and all of a sudden any model.fit will just kill the kernel. After some
debugging I found out that TF 2.16.1 had been installed even when using the
2.15 base image. Where is the right place for me to report this?
—
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/63362#issuecomment-1992662806,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAGK34J4BAPGR5R2WGLVSD3YX534VAVCNFSM6AAAAABEOPWBC2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOJSGY3DEOBQGY
.
You are receiving this because you modified the open/close state.Message
ID: @.***>

Thanks @sh-shahrokhi. I thought it was path related. Modified slightly to make it python version independent if you put it in your conda environment activation ([environment]/etc/activate.d/env_vars.sh).
NVIDIA_DIR=$(dirname $(dirname $(python -c "import nvidia.cudnn;print(nvidia.cudnn.__file__)")))
for dir in $NVIDIA_DIR/*; do
    if [ -d "$dir/lib" ]; then
        export LD_LIBRARY_PATH="$dir/lib:$LD_LIBRARY_PATH"
    fi
done

This is not a resolution as this post install step should not be necessary.

Thanks for the workaround, but the path is wrong (conda is missing). Also the file name is arbitrary: ${CONDA_PREFIX}/etc/conda/activate.d/nvidia-lib-dirs.sh. Also the first line can be simplified to
NVIDIA_DIR=$(dirname $(python -c 'import nvidia;print(nvidia.__file__)'))

and if you don't have spaces in your environment path, the rest can be simplified to:
export LD_LIBRARY_PATH=$(echo ${NVIDIA_DIR}/*/lib/ | sed -r 's/\s+/:/g')${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}

Maybe the Tf at GitHub is the right place to report that
That's where it was reported :)

On a somewhat similar note, I have been using the tensorflow/tensorflow:2.15.0.post1-gpu-jupyter docker image as base image for the past months without issue and yesterday I rebuilt some local images and all of a sudden any model.fit will just kill the kernel. After some debugging I found out that TF 2.16.1 had been installed even when using the 2.15 base image. Where is the right place for me to report this?

This could probably be because the container did not pin the version and just installed the last one (which now is 2.16)

Thanks @mihaimaruseac! I had it pinned, also tried with the SHA digest but then realized I had unpinned tensorflow-text in my requirements and that was upgrading TF as well :P

I had the same issue. Setting TF_CPP_MAX_VLOG_LEVEL=3 shows more information.
TF_CPP_MAX_VLOG_LEVEL=3 python -c "import tensorflow as tf;print(tf.config.list_physical_devices('GPU'))"

2024-03-13 17:53:09.566158: I external/local_tsl/tsl/platform/default/dso_loader.cc:59] Successfully opened dynamic library libcudart.so.12
2024-03-13 17:53:09.588990: I external/local_tsl/tsl/platform/default/dso_loader.cc:59] Successfully opened dynamic library libcublas.so.12
2024-03-13 17:53:09.589043: I external/local_tsl/tsl/platform/default/dso_loader.cc:59] Successfully opened dynamic library libcublasLt.so.12
2024-03-13 17:53:09.590693: I external/local_tsl/tsl/platform/default/dso_loader.cc:59] Successfully opened dynamic library libcufft.so.11
2024-03-13 17:53:09.593849: I external/local_tsl/tsl/platform/default/dso_loader.cc:59] Successfully opened dynamic library libcusolver.so.11
2024-03-13 17:53:09.593933: I external/local_tsl/tsl/platform/default/dso_loader.cc:59] Successfully opened dynamic library libcusparse.so.12
2024-03-13 17:53:09.594012: I external/local_tsl/tsl/platform/default/dso_loader.cc:70] Could not load dynamic library 'libcudnn.so.8'; dlerror: libcudnn.so.8: cannot open shared object file: No such file or directory
Skipping registering GPU devices...
[]

It seems that TF 2.16 stops loading cuDNN from the Python site-packages directory (and other directories?), but other CUDA libraries are correctly loaded.
However, it works after setting LD_LIBRARY_PATH manually
export LD_LIBRARY_PATH=~/anaconda3/lib/python3.10/site-packages/nvidia/cudnn/lib/:$LD_LIBRARY_PATH

[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU'), PhysicalDevice(name='/physical_device:GPU:1', device_type='GPU')]

I had the same issue. Setting TF_CPP_MAX_VLOG_LEVEL=3 shows more information.
TF_CPP_MAX_VLOG_LEVEL=3 python -c "import tensorflow as tf;print(tf.config.list_physical_devices('GPU'))"

2024-03-13 17:53:09.566158: I external/local_tsl/tsl/platform/default/dso_loader.cc:59] Successfully opened dynamic library libcudart.so.12
2024-03-13 17:53:09.588990: I external/local_tsl/tsl/platform/default/dso_loader.cc:59] Successfully opened dynamic library libcublas.so.12
2024-03-13 17:53:09.589043: I external/local_tsl/tsl/platform/default/dso_loader.cc:59] Successfully opened dynamic library libcublasLt.so.12
2024-03-13 17:53:09.590693: I external/local_tsl/tsl/platform/default/dso_loader.cc:59] Successfully opened dynamic library libcufft.so.11
2024-03-13 17:53:09.593849: I external/local_tsl/tsl/platform/default/dso_loader.cc:59] Successfully opened dynamic library libcusolver.so.11
2024-03-13 17:53:09.593933: I external/local_tsl/tsl/platform/default/dso_loader.cc:59] Successfully opened dynamic library libcusparse.so.12
2024-03-13 17:53:09.594012: I external/local_tsl/tsl/platform/default/dso_loader.cc:70] Could not load dynamic library 'libcudnn.so.8'; dlerror: libcudnn.so.8: cannot open shared object file: No such file or directory
Skipping registering GPU devices...
[]

It seems that TF 2.16 stops loading cuDNN from the Python site-packages directory (and other directories?), but other CUDA libraries are correctly loaded.
However, it works after setting LD_LIBRARY_PATH manually
export LD_LIBRARY_PATH=~/anaconda3/lib/python3.10/site-packages/nvidia/cudnn/lib/:$LD_LIBRARY_PATH

[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU'), PhysicalDevice(name='/physical_device:GPU:1', device_type='GPU')]

By following the work around my GPU was included in the list of physical devices indeed. However, I attempted to run a short deep learning model training script but failed with the following error message:
ptxas returned an error during compilation of ptx to sass: 'INTERNAL: ptxas 12.3.103 has a bug that we think can affect XLA. Please use a different version.'  If the error message indicates that a file could not be written, please verify that sufficient filesystem space is provided.

Same issue to me. Is there any official solution for it? I am using WSL2 just today, I don't want to find and add the PATH manually. I am giving up TF.

By following the work around my GPU was included in the list of physical devices indeed. However, I attempted to run a short deep learning model training script but failed with the following error message:
ptxas returned an error during compilation of ptx to sass: 'INTERNAL: ptxas 12.3.103 has a bug that we think can affect XLA. Please use a different version.' If the error message indicates that a file could not be written, please verify that sufficient filesystem space is provided.

Actually you should have version 12.3.107, since it is defined in the extra [and-cuda]. Or did you install differently? Can you check which version you have with pip show nvidia-cuda-nvcc-cu12 and with conda list nvidia-nvcc? You should be able to update.

I am having the same issue.
env: Ubuntu 22.04 + Python 3.10.13 + CUDA 12.4 + tensorflow 2.16.1
(dl-opt) zxiong@ws2:/mnt/work/dl-opt$ nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Tue_Feb_27_16:19:38_PST_2024
Cuda compilation tools, release 12.4, V12.4.99
Build cuda_12.4.r12.4/compiler.33961263_0

(dl-opt) zxiong@ws2:/mnt/work/dl-opt$ nvidia-smi
Fri Mar 15 10:53:01 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.54.14              Driver Version: 550.54.14      CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 4080        Off |   00000000:01:00.0 Off |                  N/A |
|  0%   35C    P8             32W /  320W |       1MiB /  16376MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

(dl-opt) zxiong@ws2:/mnt/work/dl-opt$ python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"
2024-03-15 10:39:18.693721: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-03-15 10:39:18.717258: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-03-15 10:39:19.866184: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2024-03-15 10:39:21.671783: W tensorflow/core/common_runtime/gpu/gpu_device.cc:2251] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
[]

Hi all,
If you have installed tensorflow[and-cuda] package it will install all the required & compatible cuda/cudnn related libraries. I am of opinion that installing tensorflow[and-cuda] package will take care of path setting also.If not please let us know.
Only manual task is you need to install compatible cuda driver for the CUDA library. Please find the compatible cuda/cudnn versions required for TF2.16 version below.

Version
Python version
Compiler
Build tools
cuDNN
CUDA

tensorflow-2.16.1
3.9-3.12
Clang 17.0.6
Bazel 6.5.0
8.9
12.3

tensorflow-2.15.0
3.9-3.11
Clang 16.0.0
Bazel 6.1.0
8.9
12.2

As Mihai already commented here that TF will be tested against only one RC before release and later it will go ahead for forward compatibility.

I believe that tensorflow[and-cuda] does not work as expected. For reproduction, I built a simple Docker image:
FROM python:3.11
RUN pip install tensorflow[and-cuda]

Build and run it:
docker build . -t test-tf216-cuda
docker run --gpus all test-tf216-cuda python -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"

The output shows the libraries cannot be found, as the same as the above.

2024-03-15 04:45:04.566105: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-03-15 04:45:05.342407: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2024-03-15 04:45:05.895918: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2024-03-15 04:45:05.896280: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2024-03-15 04:45:05.896898: W tensorflow/core/common_runtime/gpu/gpu_device.cc:2251] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
[]

Executing docker run --gpus all -e TF_CPP_MAX_VLOG_LEVEL=3 -e LD_DEBUG=libs test-tf216-cuda python -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))" will give more information, where I think the search path is entirely wrong.
run.log

It does not work unfortunately. It worked with TF<=2.15.1. You would need to test it on non-docker version, since that one actully installs cuda libraries in the container with apt, not pip. Colab is also not a good place to test, because if you do pip list, you can see that cuda libraries are not installed from pypi, but come from some other methods.
For this, please test  local linux or wsl installation that has only nvidia drivers, not anything else. run pip install tensorflow[and-cuda]==2.16.1in a conda / venv / ... environment, and try to list the gpus. It will not work. Then test with 2.15.1 in another environment and see that it works.

From: Surya @.>
Sent: Thursday, March 14, 2024 10:14 PM
To: tensorflow/tensorflow @.>
Cc: Shayan Shahrokhi @.>; Mention @.>
Subject: Re: [tensorflow/tensorflow] TF 2.16.1 Fails to work with GPUs (Issue #63362)
Hi all,
If you have installed tensorflow[and-cuda] package it will install all the required & compatible cuda/cudnn related libraries. I am of opinion that installing tensorflow[and-cuda] package will take care of path setting also.If not please let us know.
Only manual task is you need to install compatible cuda driver for the CUDA library. Please find the compatible cuda/cudnn versions required for TF2.16 version below.
Version Python version  Compiler        Build tools     cuDNN   CUDA
tensorflow-2.16.1       3.9-3.12        Clang 17.0.6    Bazel 6.5.0     8.9     12.3
tensorflow-2.15.0       3.9-3.11        Clang 16.0.0    Bazel 6.1.0     8.9     12.2
As Mihai already commented herehttps://github.com/tensorflow/tensorflow/issues/63362#issuecomment-1990133275 that TF will be tested against only one RC before release and later it will go ahead for forward compatibility.
—
Reply to this email directly, view it on GitHubhttps://github.com/tensorflow/tensorflow/issues/63362#issuecomment-1998906038, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AZRPJACHU65LNEEHCU54R4LYYJYSXAVCNFSM6AAAAABEOPWBC2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOJYHEYDMMBTHA.
You are receiving this because you were mentioned.Message ID: @.***>

Hi all,
If you have installed tensorflow[and-cuda] package it will install all the required & compatible cuda/cudnn related libraries. I am of opinion that installing tensorflow[and-cuda] package will take care of path setting also.If not please let us know. Only manual task is you need to install compatible cuda driver for the CUDA library. Please find the compatible cuda/cudnn versions required for TF2.16 version below.
Version	Python version	Compiler	Build tools	cuDNN	CUDA
tensorflow-2.16.1	3.9-3.12	Clang 17.0.6	Bazel 6.5.0	8.9	12.3
tensorflow-2.15.0	3.9-3.11	Clang 16.0.0	Bazel 6.1.0	8.9	12.2
As Mihai already commented here that TF will be tested against only one RC before release and later it will go ahead for forward compatibility.

It does not work unfortunately. It worked with TF<=2.15.1. You would need to test it on non-docker version, since that one actully installs cuda libraries in the container with apt, not pip. Colab is also not a good place to test, because if you do pip list, you can see that cuda libraries are not installed from pypi, but come from some other methods.
For this, please test  local linux or wsl installation that has only nvidia drivers, not anything else. run pip install tensorflow[and-cuda]==2.16.1in a conda / venv / ... environment, and try to list the gpus. It will not work. Then test with 2.15.1 in another environment and see that it works.

Hi all,
If you have installed tensorflow[and-cuda] package it will install all the required & compatible cuda/cudnn related libraries. I am of opinion that installing tensorflow[and-cuda] package will take care of path setting also.If not please let us know. Only manual task is you need to install compatible cuda driver for the CUDA library. Please find the compatible cuda/cudnn versions required for TF2.16 version below.
Version	Python version	Compiler	Build tools	cuDNN	CUDA
tensorflow-2.16.1	3.9-3.12	Clang 17.0.6	Bazel 6.5.0	8.9	12.3
tensorflow-2.15.0	3.9-3.11	Clang 16.0.0	Bazel 6.1.0	8.9	12.2
As Mihai already commented here that TF will be tested against only one RC before release and later it will go ahead for forward compatibility.

Hi @SuryanarayanaY
It is pretty obvious that installing tensorflow[and-cuda] package did not actually take care of path setting (I already mentioned the issue like everybody else did as well) !  As far for compatibility I am still trying to figure out how to deal with the error:
`ptxas returned an error during compilation of ptx to sass: 'INTERNAL: ptxas 12.3.103 has a bug that we think can affect XLA. Please use a different version.' If the error message indicates that a file could not be written, please verify that sufficient filesystem space is provided.'

Seriously... I followed the instructions exactly. Yes I
used tensorflow[and-cuda].  Under wsl and conda. I'm using vscode and it
makes the conda environment in ./.conda
Sticking with the previous pip release 2.15.1 then porting to plain pytorch
and transformer.
On Fri, 15 Mar 2024, 15:09 Sotiris Gkouzias, @.***>
wrote:

Hi all,
If you have installed tensorflow[and-cuda] package it will install all
the required & compatible cuda/cudnn related libraries. I am of opinion
that installing tensorflow[and-cuda] package will take care of path
setting also.If not please let us know. Only manual task is you need to
install compatible cuda driver for the CUDA library. Please find the
compatible cuda/cudnn versions required for TF2.16 version below.
Version Python version Compiler Build tools cuDNN CUDA
tensorflow-2.16.1 3.9-3.12 Clang 17.0.6 Bazel 6.5.0 8.9 12.3
tensorflow-2.15.0 3.9-3.11 Clang 16.0.0 Bazel 6.1.0 8.9 12.2
As Mihai already commented here
https://github.com/tensorflow/tensorflow/issues/63362#issuecomment-1990133275
that TF will be tested against only one RC before release and later it will
go ahead for forward compatibility.
Hi @SuryanarayanaY https://github.com/SuryanarayanaY
It is pretty obvious that installing tensorflow[and-cuda] package did not
actually take care of path setting (I already mentioned the issue like
everybody else did as well) ! As far for compatibility I am still trying to
figure out how to deal with the error:
ptxas returned an error during compilation of ptx to sass: 'INTERNAL:
ptxas 12.3.103 has a bug that we think can affect XLA. Please use a
different version.' If the error message indicates that a file could not be
written, please verify that sufficient filesystem space is provided.
—
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/63362#issuecomment-1999060539,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AFZCXIPPKUWULEUPXES5Z6TYYKNCJAVCNFSM6AAAAABEOPWBC2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOJZGA3DANJTHE
.
You are receiving this because you commented.Message ID:
@.***>

By following the work around my GPU was included in the list of physical devices indeed. However, I attempted to run a short deep learning model training script but failed with the following error message:
ptxas returned an error during compilation of ptx to sass: 'INTERNAL: ptxas 12.3.103 has a bug that we think can affect XLA. Please use a different version.' If the error message indicates that a file could not be written, please verify that sufficient filesystem space is provided.

Actually you should have version 12.3.107, since it is defined in the extra [and-cuda]. Or did you install differently? Can you check which version you have with pip show nvidia-cuda-nvcc-cu12 and with conda list nvidia-nvcc? You should be able to update.

@ChristofKaufmann
I ran pip show nvidia-cuda-nvcc-cu12 and the output was:
Name: nvidia-cuda-nvcc-cu12 Version: 12.3.107 Summary: CUDA nvcc Home-page: https://developer.nvidia.com/cuda-zone Author: Nvidia CUDA Installer Team Author-email<EMAIL_ADDRESS>License: NVIDIA Proprietary Software
I ran as well conda list nvidia-nvcc but with no output.

@mihaimaruseac The least that Tensorflow team can do is to test and acknowledge the problem, or say it is not planned, or anything. Just ignoring the problem without even testing is not a supportive act at all.

I am of the opinion that just doing RC0 and then final release is not good testing. I hope the 2.16 situation was just a one off situation, to save time (it took the same amount of time for 2.16 with just one RC and final as for older releases with 3 RCs, multiple vulnerability fixes, etc.). I am no longer in the TensorFlow team, just helping here and there on the GitHub issues and PRs.

I had the same issue. Setting TF_CPP_MAX_VLOG_LEVEL=3 shows more information.
TF_CPP_MAX_VLOG_LEVEL=3 python -c "import tensorflow as tf;print(tf.config.list_physical_devices('GPU'))"

2024-03-13 17:53:09.566158: I external/local_tsl/tsl/platform/default/dso_loader.cc:59] Successfully opened dynamic library libcudart.so.12
2024-03-13 17:53:09.588990: I external/local_tsl/tsl/platform/default/dso_loader.cc:59] Successfully opened dynamic library libcublas.so.12
2024-03-13 17:53:09.589043: I external/local_tsl/tsl/platform/default/dso_loader.cc:59] Successfully opened dynamic library libcublasLt.so.12
2024-03-13 17:53:09.590693: I external/local_tsl/tsl/platform/default/dso_loader.cc:59] Successfully opened dynamic library libcufft.so.11
2024-03-13 17:53:09.593849: I external/local_tsl/tsl/platform/default/dso_loader.cc:59] Successfully opened dynamic library libcusolver.so.11
2024-03-13 17:53:09.593933: I external/local_tsl/tsl/platform/default/dso_loader.cc:59] Successfully opened dynamic library libcusparse.so.12
2024-03-13 17:53:09.594012: I external/local_tsl/tsl/platform/default/dso_loader.cc:70] Could not load dynamic library 'libcudnn.so.8'; dlerror: libcudnn.so.8: cannot open shared object file: No such file or directory
Skipping registering GPU devices...
[]

It seems that TF 2.16 stops loading cuDNN from the Python site-packages directory (and other directories?), but other CUDA libraries are correctly loaded.
However, it works after setting LD_LIBRARY_PATH manually
export LD_LIBRARY_PATH=~/anaconda3/lib/python3.10/site-packages/nvidia/cudnn/lib/:$LD_LIBRARY_PATH

[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU'), PhysicalDevice(name='/physical_device:GPU:1', device_type='GPU')]

By following the work around my GPU was included in the list of physical devices indeed. However, I attempted to run a short deep learning model training script but failed with the following error message:
ptxas returned an error during compilation of ptx to sass: 'INTERNAL: ptxas 12.3.103 has a bug that we think can affect XLA. Please use a different version.' If the error message indicates that a file could not be written, please verify that sufficient filesystem space is provided.

Success!!  Ultimately, I located valid pxtas in ...lib/python3.10/site-packages/nvidia/cuda_nvcc/bin and manually added the specified path to my environment variables only within the Conda virtual environment I created for TensorFlow version 2.16.1  It works like a charm!

Not really a solution, but clarification and summary of what is happening so far. As per the investigations of @njzjz, we see that libcublas et al are loaded, whereas libcudnn is not. He also investigated inside a docker container and found that none of the libraries are loaded. He further updated the library path and found that the libraries were now loaded.
I have tried on my end and came to the same conclusion. I think what's happening is that I have installed libcublas, libcudart, etcetera, from the Nvidia Fedora repo (which does not include libcudnn), which is why these libraries load but not libcudnn when I run after pip install tensorflow[and-cuda]. Even though all these libs exist in my venv lib dir. It seems that the libraries in /usr/local/cuda/lib64 are searched, but not the ones in the venv lib dir.
@njzjz's and @sgkouzias's solutions further support this. Clearly the current workaround is to follow their path altering advice.
I hope that this is useful to somebody. Hopefully there is a fix soon.

Thanks for the update.
On Sat, Mar 16, 2024, 12:51 James Paul Turner @.***>
wrote:

Not really a solution, but clarification and summary of what is happening
so far. As per the investigations of @njzjz https://github.com/njzjz,
we see that libcublas et al are loaded, whereas libcudnn is not. He also
investigated inside a docker container and found that none of the
libraries are loaded. He further updated the library path and found that
the libraries were now loaded.
I have tried on my end and came to the same conclusion. I think what's
happening is that I have installed libcublas, libcudart, etcetera, from
the Nvidia Fedora repo (which does not include libcudnn), which is why
these libraries load but not libcudnn when I run after pip install
tensorflow[and-cuda]. Even though all these libs exist in my venv lib
dir. It seems that the libraries in /usr/local/cuda/lib64 are searched,
but not the ones in the venv lib dir.
@njzjz https://github.com/njzjz's and @sgkouzias
https://github.com/sgkouzias's solutions further support this. Clearly
the current workaround is to follow their path altering advice.
I hope that this is useful to somebody. Hopefully there is a fix soon.
—
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/63362#issuecomment-2002043884,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAGK34M2VPGJUFFDNEXDVRTYYR2A7AVCNFSM6AAAAABEOPWBC2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMBSGA2DGOBYGQ
.
You are receiving this because you modified the open/close state.Message
ID: @.***>

Yes try downgrading TensorFlow with pip install tensorflow==2.15.0.post1 . At least this works fine for my Ubuntu20.04.

Hello everyone,
I had the same problem here,
but I could manage to solve it.
Thanks to @njzjz , setting TF_CPP_MAX_VLOG_LEVEL=3 shows more information:
soheil@soheil:~$ TF_CPP_MAX_VLOG_LEVEL=3 python -c "import tensorflow as tf;print(tf.config.list_physical_devices('GPU'))"

2024-03-18 00:32:05.073521: I external/local_tsl/tsl/platform/default/dso_loader.cc:59] Successfully opened dynamic library libcudart.so.12
2024-03-18 00:32:05.073737: I external/local_tsl/tsl/platform/cloud/gcs_file_system.cc:855] GCS cache max size = 0 ; block size = 67108864 ; max staleness = 0
2024-03-18 00:32:05.073751: I external/local_tsl/tsl/platform/cloud/ram_file_block_cache.h:64] GCS file block cache is disabled
2024-03-18 00:32:05.073755: I external/local_tsl/tsl/platform/cloud/gcs_file_system.cc:895] GCS DNS cache is disabled, because GCS_RESOLVE_REFRESH_SECS = 0 (or is not set)
2024-03-18 00:32:05.073758: I external/local_tsl/tsl/platform/cloud/gcs_file_system.cc:925] GCS additional header DISABLED. No environment variable set.
2024-03-18 00:32:05.073762: I external/local_tsl/tsl/platform/cloud/gcs_file_system.cc:306] GCS RetryConfig: init_delay_time_us = 1000000 ; max_delay_time_us = 32000000 ; max_retries = 10
2024-03-18 00:32:05.073765: I external/local_tsl/tsl/platform/cloud/gcs_file_system.cc:306] GCS RetryConfig: init_delay_time_us = 1000000 ; max_delay_time_us = 32000000 ; max_retries = 10
2024-03-18 00:32:05.075498: I external/local_tsl/tsl/platform/default/dso_loader.cc:59] Successfully opened dynamic library libcudart.so.12
2024-03-18 00:32:05.103619: I external/local_tsl/tsl/platform/cloud/gcs_file_system.cc:855] GCS cache max size = 0 ; block size = 67108864 ; max staleness = 0
2024-03-18 00:32:05.103641: I external/local_tsl/tsl/platform/cloud/ram_file_block_cache.h:64] GCS file block cache is disabled
2024-03-18 00:32:05.103645: I external/local_tsl/tsl/platform/cloud/gcs_file_system.cc:895] GCS DNS cache is disabled, because GCS_RESOLVE_REFRESH_SECS = 0 (or is not set)
2024-03-18 00:32:05.103648: I external/local_tsl/tsl/platform/cloud/gcs_file_system.cc:925] GCS additional header DISABLED. No environment variable set.
2024-03-18 00:32:05.103652: I external/local_tsl/tsl/platform/cloud/gcs_file_system.cc:306] GCS RetryConfig: init_delay_time_us = 1000000 ; max_delay_time_us = 32000000 ; max_retries = 10
2024-03-18 00:32:05.103655: I external/local_tsl/tsl/platform/cloud/gcs_file_system.cc:306] GCS RetryConfig: init_delay_time_us = 1000000 ; max_delay_time_us = 32000000 ; max_retries = 10
2024-03-18 00:32:05.103933: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-03-18 00:32:05.438768: I external/local_tsl/tsl/platform/cloud/gcs_file_system.cc:855] GCS cache max size = 0 ; block size = 67108864 ; max staleness = 0
2024-03-18 00:32:05.438784: I external/local_tsl/tsl/platform/cloud/ram_file_block_cache.h:64] GCS file block cache is disabled
2024-03-18 00:32:05.438808: I external/local_tsl/tsl/platform/cloud/gcs_file_system.cc:895] GCS DNS cache is disabled, because GCS_RESOLVE_REFRESH_SECS = 0 (or is not set)
2024-03-18 00:32:05.438812: I external/local_tsl/tsl/platform/cloud/gcs_file_system.cc:925] GCS additional header DISABLED. No environment variable set.
2024-03-18 00:32:05.438839: I external/local_tsl/tsl/platform/cloud/gcs_file_system.cc:306] GCS RetryConfig: init_delay_time_us = 1000000 ; max_delay_time_us = 32000000 ; max_retries = 10
2024-03-18 00:32:05.438843: I external/local_tsl/tsl/platform/cloud/gcs_file_system.cc:306] GCS RetryConfig: init_delay_time_us = 1000000 ; max_delay_time_us = 32000000 ; max_retries = 10
2024-03-18 00:32:05.609776: I external/local_tsl/tsl/platform/default/dso_loader.cc:70] Could not load dynamic library 'libnvinfer.so.8.6.1'; dlerror: libnvinfer.so.8.6.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /var/lib:/usr/lib:/home/soheil/anaconda3/lib/python3.10/site-packages/nvidia/cudnn/lib/::/usr/local/cuda-12.4/lib64:/usr/local/cuda-12.4/lib64
2024-03-18 00:32:05.609875: I external/local_tsl/tsl/platform/default/dso_loader.cc:70] Could not load dynamic library 'libnvinfer_plugin.so.8.6.1'; dlerror: libnvinfer_plugin.so.8.6.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /var/lib:/usr/lib:/home/soheil/anaconda3/lib/python3.10/site-packages/nvidia/cudnn/lib/::/usr/local/cuda-12.4/lib64:/usr/local/cuda-12.4/lib64
2024-03-18 00:32:05.609882: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2024-03-18 00:32:05.937145: I external/local_xla/xla/parse_flags_from_env.cc:207] For env var TF_XLA_FLAGS found arguments:
2024-03-18 00:32:05.937182: I external/local_xla/xla/parse_flags_from_env.cc:209]   argv[0] = <argv[0]>
2024-03-18 00:32:05.937188: I external/local_xla/xla/parse_flags_from_env.cc:207] For env var TF_JITRT_FLAGS found arguments:
2024-03-18 00:32:05.937192: I external/local_xla/xla/parse_flags_from_env.cc:209]   argv[0] = <argv[0]>
2024-03-18 00:32:05.937215: I tensorflow/compiler/jit/xla_cpu_device.cc:46] Not creating XLA devices, tf_xla_enable_xla_devices not set and XLA device creation not requested
2024-03-18 00:32:05.937219: I tensorflow/compiler/jit/xla_gpu_device.cc:49] Not creating XLA devices, tf_xla_enable_xla_devices not set and XLA devices creation not required
2024-03-18 00:32:05.937662: I external/local_tsl/tsl/platform/default/dso_loader.cc:59] Successfully opened dynamic library libcuda.so.1
2024-03-18 00:32:05.983926: I external/local_xla/xla/stream_executor/cuda/cuda_diagnostics.cc:192] found DLL info with name: /lib/x86_64-linux-gnu/libcuda.so.1
2024-03-18 00:32:05.983958: I external/local_xla/xla/stream_executor/cuda/cuda_diagnostics.cc:197] found DLL info with resolved path: /usr/lib/x86_64-linux-gnu/libcuda.so.550.54.14
2024-03-18 00:32:05.983978: I external/local_xla/xla/stream_executor/cuda/cuda_diagnostics.cc:97] version string "550.54.14" made value 550.54.14
2024-03-18 00:32:05.983999: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:968] trying to read NUMA node for device ordinal: 0
2024-03-18 00:32:05.984031: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2024-03-18 00:32:05.984184: I tensorflow/core/common_runtime/gpu/gpu_device.cc:2219] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: NVIDIA GeForce RTX 2060 computeCapability: 7.5
coreClock: 1.2GHz coreCount: 30 deviceMemorySize: 5.77GiB deviceMemoryBandwidth: 244.97GiB/s
2024-03-18 00:32:05.984205: I external/local_tsl/tsl/platform/default/dso_loader.cc:59] Successfully opened dynamic library libcudart.so.12
2024-03-18 00:32:06.002289: I external/local_tsl/tsl/platform/default/dso_loader.cc:59] Successfully opened dynamic library libcublas.so.12
2024-03-18 00:32:06.002375: I external/local_tsl/tsl/platform/default/dso_loader.cc:59] Successfully opened dynamic library libcublasLt.so.12
2024-03-18 00:32:06.004118: I external/local_tsl/tsl/platform/default/dso_loader.cc:59] Successfully opened dynamic library libcufft.so.11
2024-03-18 00:32:06.006168: I external/local_tsl/tsl/platform/default/dso_loader.cc:59] Successfully opened dynamic library libcusolver.so.11
2024-03-18 00:32:06.006254: I external/local_tsl/tsl/platform/default/dso_loader.cc:59] Successfully opened dynamic library libcusparse.so.12
2024-03-18 00:32:06.006336: I external/local_tsl/tsl/platform/default/dso_loader.cc:70] Could not load dynamic library 'libcudnn.so.8'; dlerror: libcudnn.so.8: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /var/lib:/usr/lib:/home/soheil/anaconda3/lib/python3.10/site-packages/nvidia/cudnn/lib/::/usr/local/cuda-12.4/lib64:/usr/local/cuda-12.4/lib64
2024-03-18 00:32:06.006343: W tensorflow/core/common_runtime/gpu/gpu_device.cc:2251] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
[]

what I did, was searching for where cudnn is placed:
soheil@soheil:~$ sudo find / -name "*libcudnn.so*"

/home/soheil/.local/lib/python3.10/site-packages/nvidia/cudnn/lib/libcudnn.so.8
/usr/lib/x86_64-linux-gnu/libcudnn.so.9
/usr/lib/x86_64-linux-gnu/libcudnn.so
/usr/lib/x86_64-linux-gnu/libcudnn.so.9.0.0
find: ‘/run/user/1000/doc’: Permission denied
find: ‘/run/user/1000/gvfs’: Permission denied

adding /usr/lib/x86_64-linux-gnu/libcudnn.so.9 to LD_LIBRARY_PATH solved my problem :)
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib/x86_64-linux-gnu

soheil@soheil:~$ python -c "import tensorflow as tf;print(tf.config.list_physical_devices('GPU'))"

2024-03-18 00:35:24.292809: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-03-18 00:35:24.797787: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2024-03-18 00:35:25.127729: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2024-03-18 00:35:25.148460: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2024-03-18 00:35:25.148676: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]

Hope this helps

Yes try downgrading TensorFlow with pip install tensorflow==2.15.0.post1 . At least this works fine for my Ubuntu20.04.

@caizhuodi undoubtedly. However, IMHO if you need the new features introduced with Keras 3.0 it would be nice to install TensorFlow 2.16 and utilize a CUDA-enabled GPU to explore the new options. As I understand it there are workarounds but not a straightforward, officially tested working solution yet. Naturally, we are expecting the next patch release for the issue to be resolved.

Hi Soheil,
I tried your suggestion of having LD_LIBRARY_PATH added, and then install
tf.2.16.1. I am happy to report that your suggestion appears to work.
I say "appears" only because I tested if tf can detect the GPUs via the
command
  print(tf.config.list_physical_devices('GPU'))

which returns a non-empty list, so I assume this will work with commands too.
I hope the TF team fixes the issue soon.
Thank you !
Juan E. Vargas

Wanted to drop in and thank everyone for the sleuthing done up to this point. I had no idea about the environmental variable up there that allows you to debug the loading of the libraries so easily. That was neat.
Anyway, to amalgamate all of the suggestions in the thread, you can build an env_vars.sh script within envs/<your environment>/etc/conda/activate.d/ in an anaconda-like install that looks like the following:
`NVIDIA_DIR=$(dirname $(dirname $(python -c "import nvidia.cudnn;print(nvidia.cudnn.file)")))
export LD_LIBRARY_PATH=${CONDA_PREFIX}/lib/:${LD_LIBRARY_PATH}
for dir in $(ls -1d $NVIDIA_DIR/*/); do
if [ -d "${dir}lib" ]; then
export LD_LIBRARY_PATH="${dir}lib:$LD_LIBRARY_PATH"
if [[ $(basename $dir) == 'cuda_nvcc' ]] ; then
export PATH="${dir}bin:$PATH"
fi
fi
done
export XLA_FLAGS=--xla_gpu_cuda_data_dir=${CONDA_PREFIX}/lib
`
You do indeed need the two dirname commands for TF 2.16. Similarly, you need to put the cuda_nvcc bin directory in the path so ptxas can be found. Doing it this way also generalizes to your specific conda environment. I've confirmed that you can perform the intro TF tutorial (https://www.tensorflow.org/tutorials/quickstart/beginner) with this workaround.
Having said all that, the TF Linux pip install page badly needs an update. The CUDA Toolkit and CUDNN version numbers are out of date. Since tensorflow[and-cuda] is now the recommended way to install everything for the GPU, the recommendation to install the other libraries independently is now useless as far as I can tell.

With every year it becomes more and more complicated and difficult to setup tensorflow with nvidia-gpu support on windows. Following worked for me, which is a good example of very bad developer experience.

Install CUDA-Toolkit 12.4 on host machine

Setup Ubuntu 22.04.3 LTS from Windows Store and update it.

wsl --setdefault Ubuntu-22.04
wsl
sudo apt update
sudo apt upgrade
sudo apt install python3-pip
``

3. Install tensorflow
`python3 -m pip install tensorflow[and-cuda]==2.16.1`
...
Downloading tensorflow-2.16.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Collecting nvidia-cuda-nvcc-cu12==12.3.107
Collecting nvidia-cusparse-cu12==<IP_ADDRESS>
Collecting nvidia-cuda-runtime-cu12==12.3.101
Collecting nvidia-curand-cu12==<IP_ADDRESS>
Collecting nvidia-cudnn-cu12==<IP_ADDRESS>
Collecting nvidia-cufft-cu12==<IP_ADDRESS>
Collecting nvidia-cuda-cupti-cu12==12.3.101
Collecting nvidia-cublas-cu12==<IP_ADDRESS>
Collecting nvidia-nccl-cu12==2.19.3

Find the cuda libraries. Following doesn't necessarily work. I had to find them by tracking where the tensorflow did the installation.

sudo find / -name "*libcudnn.so*"
sudo find / -name "*libcublas.so*"
sudo find / -name "*libcufft.so*"
sudo find / -name "*libcusolver.so*"
sudo find / -name "*libcusparse.so*"

Use cuda libraries

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib/x86_64-linux-gnu;
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib/python3.10/dist-packages/nvidia/cublas/lib;
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib/python3.10/dist-packages/nvidia/cufft/lib;
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib/python3.10/dist-packages/nvidia/cusolver/lib;
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib/python3.10/dist-packages/nvidia/cusparse/lib;

Run tensorflow check
TF_CPP_MAX_VLOG_LEVEL=3 python3 -c "import tensorflow as tf;print(tf.config.list_physical_devices('GPU'))"

I am aware that this might not be the place to post a new issue I am
experiencing, but since I have spent so much time on the problems related
to TF Keras update versions, I hope the members of this forum will
understand and hopefully will not object to my posting this message here.
In the past I ran with no problem the code from the second ed of Cholllet's
Book, pp. 289, using TF 2.15 and Keras 2.15. For reference the code is
from tensorflow.keras import layers
inputs = keras.Input(shape=(sequence_length, raw_data.shape[-1]))
x = keras.layers.Flatten()(inputs)
x = keras.layers.Dense(16, activation="relu")(x)
outputs = keras.layers.Dense(1)(x)
model = keras.Model(inputs, outputs)
callbacks = [
keras.callbacks.ModelCheckpoint("jena_dense.keras.x.keras",
save_best_only=True)
]
model.compile(optimizer="rmsprop", loss="mse", metrics=["mae"])
history = model.fit(train_dataset,
epochs=10,
validation_data=val_dataset,
callbacks=callbacks)
model = keras.models.load_model("jena_dense.keras.x.keras")
print(f"Test MAE: {model.evaluate(test_dataset)[1]:.2f}")
s3 = f"Test MAE: {model.evaluate(test_dataset)[1]:.2f}"
track_error.append(s3)
I installed TF 2.16.1 this week using the hack suggested by  Soheil Khatibi of
having LD_LIBRARY_PATH added to detect the cudnn libs.
When launching TF/Keras under a virtual Python Env the system detects the
GPU, so that was OK.
HOWEVER
When I run the code above (from pp. 289 in Chollet's book) under TF 2.16.1
and Keras 3.1.0, I get the error msg listed below.
What puzzles me is that the same exact code runs fine under TF.2.15.1.
Is this an error related to the Layers.Flatten ()  not detected yet in  TF
2.16.1 ??
The error message is:
{
"name": "InvalidArgumentError",
"message": "Graph execution error:
Detected at node functional_1_1/flatten_1/Reshape defined at (most recent
call last):

only one input size may be -1, not both 0 and 1
Stack trace for op definition:
File "usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
File "usr/lib/python3.10/runpy.py", line 86, in _run_code
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/ipykernel_launcher.py",
line 18, in
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/traitlets/config/application.py",
line 1075, in launch_instance
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/ipykernel/kernelapp.py",
line 739, in start
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/tornado/platform/asyncio.py",
line 205, in start
File "usr/lib/python3.10/asyncio/base_events.py", line 603, in run_forever
File "usr/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once
File "usr/lib/python3.10/asyncio/events.py", line 80, in _run
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/ipykernel/kernelbase.py",
line 545, in dispatch_queue
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/ipykernel/kernelbase.py",
line 534, in process_one
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/ipykernel/kernelbase.py",
line 437, in dispatch_shell
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/ipykernel/ipkernel.py",
line 359, in execute_request
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/ipykernel/kernelbase.py",
line 778, in execute_request
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/ipykernel/ipkernel.py",
line 446, in do_execute
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/ipykernel/zmqshell.py",
line 549, in run_cell
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/IPython/core/interactiveshell.py",
line 3075, in run_cell
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/IPython/core/interactiveshell.py",
line 3130, in _run_cell
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/IPython/core/async_helpers.py",
line 129, in _pseudo_sync_runner
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/IPython/core/interactiveshell.py",
line 3334, in run_cell_async
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/IPython/core/interactiveshell.py",
line 3517, in run_ast_nodes
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/IPython/core/interactiveshell.py",
line 3577, in run_code
File "tmp/ipykernel_12551/2185219983.py", line 20, in
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py",
line 117, in error_handler
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/backend/tensorflow/trainer.py",
line 325, in fit
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/backend/tensorflow/trainer.py",
line 118, in one_step_on_iterator
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/backend/tensorflow/trainer.py",
line 106, in one_step_on_data
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/backend/tensorflow/trainer.py",
line 57, in train_step
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py",
line 117, in error_handler
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/layers/layer.py",
line 814, in call
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py",
line 117, in error_handler
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/ops/operation.py",
line 48, in call
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py",
line 156, in error_handler
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/models/functional.py",
line 194, in call
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/ops/function.py",
line 151, in _run_through_graph
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/models/functional.py",
line 578, in call
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py",
line 117, in error_handler
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/layers/layer.py",
line 814, in call
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py",
line 117, in error_handler
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/ops/operation.py",
line 48, in call
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py",
line 156, in error_handler
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/layers/reshaping/flatten.py",
line 54, in call
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/ops/numpy.py",
line 4507, in reshape
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/backend/tensorflow/numpy.py",
line 1545, in reshape
\t [[{{node functional_1_1/flatten_1/Reshape}}]]
\ttf2xla conversion failed while converting
__inference_one_step_on_data_19335[]. Run with
TF_DUMP_GRAPH_PREFIX=/path/to/dump/dir and --vmodule=xla_compiler=2 to
obtain a dump of the compiled functions.
\t [[StatefulPartitionedCall]] [Op:__inference_one_step_on_iterator_19368]",
"stack":
"---------------------------------------------------------------------------
InvalidArgumentError                      Traceback (most recent call last)
Cell In[16], line 20
12 callbacks = [
13    keras.callbacks.ModelCheckpoint("jena_dense.keras.x.keras",
14                                     save_best_only=True)
15 ]
17 model.compile(optimizer="rmsprop", loss="mse",
metrics=["mae"])
---> 20 history = model.fit(train_dataset,
21                     epochs=10,
22                     validation_data=val_dataset,
23                     callbacks=callbacks)
26 model = keras.models.load_model("jena_dense.keras.x.keras")
27 print(f"Test MAE: {model.evaluate(test_dataset)[1]:.2f}")
File
/drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py:122,
in filter_traceback..error_handler(*args, **kwargs)
119     filtered_tb = _process_traceback_frames(e.traceback)
120     # To get the full stack trace, call:
121     # keras.config.disable_traceback_filtering()
--> 122     raise e.with_traceback(filtered_tb) from None
123 finally:
124     del filtered_tb
File
/drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/tensorflow/python/eager/execute.py:53,
in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name)
51 try:
52   ctx.ensure_initialized()
---> 53   tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name,
op_name,
54                                       inputs, attrs, num_outputs)
55 except core._NotOkStatusException as e:
56   if name is not None:
InvalidArgumentError: Graph execution error:
Detected at node functional_1_1/flatten_1/Reshape defined at (most recent
call last):

only one input size may be -1, not both 0 and 1
Stack trace for op definition:
File "usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
File "usr/lib/python3.10/runpy.py", line 86, in _run_code
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/ipykernel_launcher.py",
line 18, in
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/traitlets/config/application.py",
line 1075, in launch_instance
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/ipykernel/kernelapp.py",
line 739, in start
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/tornado/platform/asyncio.py",
line 205, in start
File "usr/lib/python3.10/asyncio/base_events.py", line 603, in run_forever
File "usr/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once
File "usr/lib/python3.10/asyncio/events.py", line 80, in _run
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/ipykernel/kernelbase.py",
line 545, in dispatch_queue
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/ipykernel/kernelbase.py",
line 534, in process_one
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/ipykernel/kernelbase.py",
line 437, in dispatch_shell
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/ipykernel/ipkernel.py",
line 359, in execute_request
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/ipykernel/kernelbase.py",
line 778, in execute_request
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/ipykernel/ipkernel.py",
line 446, in do_execute
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/ipykernel/zmqshell.py",
line 549, in run_cell
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/IPython/core/interactiveshell.py",
line 3075, in run_cell
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/IPython/core/interactiveshell.py",
line 3130, in _run_cell
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/IPython/core/async_helpers.py",
line 129, in _pseudo_sync_runner
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/IPython/core/interactiveshell.py",
line 3334, in run_cell_async
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/IPython/core/interactiveshell.py",
line 3517, in run_ast_nodes
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/IPython/core/interactiveshell.py",
line 3577, in run_code
File "tmp/ipykernel_12551/2185219983.py", line 20, in
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py",
line 117, in error_handler
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/backend/tensorflow/trainer.py",
line 325, in fit
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/backend/tensorflow/trainer.py",
line 118, in one_step_on_iterator
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/backend/tensorflow/trainer.py",
line 106, in one_step_on_data
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/backend/tensorflow/trainer.py",
line 57, in train_step
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py",
line 117, in error_handler
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/layers/layer.py",
line 814, in call
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py",
line 117, in error_handler
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/ops/operation.py",
line 48, in call
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py",
line 156, in error_handler
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/models/functional.py",
line 194, in call
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/ops/function.py",
line 151, in _run_through_graph
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/models/functional.py",
line 578, in call
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py",
line 117, in error_handler
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/layers/layer.py",
line 814, in call
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py",
line 117, in error_handler
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/ops/operation.py",
line 48, in call
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py",
line 156, in error_handler
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/layers/reshaping/flatten.py",
line 54, in call
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/ops/numpy.py",
line 4507, in reshape
File
"drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/backend/tensorflow/numpy.py",
line 1545, in reshape
\t [[{{node functional_1_1/flatten_1/Reshape}}]]
\ttf2xla conversion failed while converting
__inference_one_step_on_data_19335[]. Run with
TF_DUMP_GRAPH_PREFIX=/path/to/dump/dir and --vmodule=xla_compiler=2 to
obtain a dump of the compiled functions.
\t [[StatefulPartitionedCall]] [Op:__inference_one_step_on_iterator_19368]"
}
On Thu, Mar 21, 2024 at 8:18 AM David R @.***> wrote:

Wanted to drop in and thank everyone for the sleuthing done up to this
point. I had no idea about the environmental variable up there that allows
you to debug the loading of the libraries so easily. That was neat.
Anyway, to amalgamate all of the suggestions in the thread, you can build
an env_vars.sh script within envs//etc/conda/activate.d/
in an anaconda-like install that looks like the following:
NVIDIA_DIR=$(dirname $(dirname $(python -c "import nvidia.cudnn;print(nvidia.cudnn.file)")))
export LD_LIBRARY_PATH=${CONDA_PREFIX}/lib/:${LD_LIBRARY_PATH}
for dir in $(ls -1d $NVIDIA_DIR/*/); do
if [ -d "${dir}lib" ]; then
export LD_LIBRARY_PATH="${dir}lib:$LD_LIBRARY_PATH"
if [[ $(basename $dir) == 'cuda_nvcc' ]] ; then
export PATH="${dir}bin:$PATH"
fi
fi
done
export XLA_FLAGS=--xla_gpu_cuda_data_dir=${CONDA_PREFIX}/lib
You do indeed need the two dirname commands for TF 2.16. Similarly, you
need to put the cuda_nvcc bin directory in the path so ptxas can be found.
Doing it this way also generalizes to your specific conda environment. I've
confirmed that you can perform the intro TF tutorial (
https://www.tensorflow.org/tutorials/quickstart/beginner) with this
workaround.
Having said all that, the TF Linux pip install page badly needs an update.
The CUDA Toolkit and CUDNN version numbers are out of date. Since
tensorflow[and-cuda] is now the recommended way to install everything for
the GPU, the recommendation to install the other libraries independently is
now useless as far as I can tell.
—
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/63362#issuecomment-2012132577,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAGK34OAGI7LCXG5FDA6WTTYZLFZ7AVCNFSM6AAAAABEOPWBC2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMJSGEZTENJXG4
.
You are receiving this because you modified the open/close state.Message
ID: @.***>

I am aware that this might not be the place to post a new issue I am experiencing, but since I have spent so much time on the problems related to TF Keras update versions, I hope the members of this forum will understand and hopefully will not object to my posting this message here. In the past I ran with no problem the code from the second ed of Cholllet's Book, pp. 289, using TF 2.15 and Keras 2.15. For reference the code is # from tensorflow.keras import layers inputs = keras.Input(shape=(sequence_length, raw_data.shape[-1])) x = keras.layers.Flatten()(inputs) x = keras.layers.Dense(16, activation="relu")(x) outputs = keras.layers.Dense(1)(x) model = keras.Model(inputs, outputs) callbacks = [ keras.callbacks.ModelCheckpoint("jena_dense.keras.x.keras", save_best_only=True) ] model.compile(optimizer="rmsprop", loss="mse", metrics=["mae"]) history = model.fit(train_dataset, epochs=10, validation_data=val_dataset, callbacks=callbacks) model = keras.models.load_model("jena_dense.keras.x.keras") print(f"Test MAE: {model.evaluate(test_dataset)[1]:.2f}") s3 = f"Test MAE: {model.evaluate(test_dataset)[1]:.2f}" track_error.append(s3) I installed TF 2.16.1 this week using the hack suggested by Soheil Khatibi of having LD_LIBRARY_PATH added to detect the cudnn libs. When launching TF/Keras under a virtual Python Env the system detects the GPU, so that was OK. HOWEVER When I run the code above (from pp. 289 in Chollet's book) under TF 2.16.1 and Keras 3.1.0, I get the error msg listed below. What puzzles me is that the same exact code runs fine under TF.2.15.1. Is this an error related to the Layers.Flatten () not detected yet in TF 2.16.1 ?? The error message is: { "name": "InvalidArgumentError", "message": "Graph execution error: Detected at node functional_1_1/flatten_1/Reshape defined at (most recent call last):  only one input size may be -1, not both 0 and 1 Stack trace for op definition: File "usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main File "usr/lib/python3.10/runpy.py", line 86, in _run_code File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/ipykernel_launcher.py", line 18, in  File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/traitlets/config/application.py", line 1075, in launch_instance File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 739, in start File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/tornado/platform/asyncio.py", line 205, in start File "usr/lib/python3.10/asyncio/base_events.py", line 603, in run_forever File "usr/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once File "usr/lib/python3.10/asyncio/events.py", line 80, in _run File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 545, in dispatch_queue File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 534, in process_one File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 437, in dispatch_shell File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 359, in execute_request File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 778, in execute_request File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 446, in do_execute File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/ipykernel/zmqshell.py", line 549, in run_cell File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3075, in run_cell File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3130, in _run_cell File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3334, in run_cell_async File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3517, in run_ast_nodes File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3577, in run_code File "tmp/ipykernel_12551/2185219983.py", line 20, in  File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 117, in error_handler File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/backend/tensorflow/trainer.py", line 325, in fit File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/backend/tensorflow/trainer.py", line 118, in one_step_on_iterator File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/backend/tensorflow/trainer.py", line 106, in one_step_on_data File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/backend/tensorflow/trainer.py", line 57, in train_step File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 117, in error_handler File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/layers/layer.py", line 814, in call File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 117, in error_handler File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/ops/operation.py", line 48, in call File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 156, in error_handler File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/models/functional.py", line 194, in call File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/ops/function.py", line 151, in _run_through_graph File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/models/functional.py", line 578, in call File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 117, in error_handler File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/layers/layer.py", line 814, in call File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 117, in error_handler File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/ops/operation.py", line 48, in call File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 156, in error_handler File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/layers/reshaping/flatten.py", line 54, in call File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/ops/numpy.py", line 4507, in reshape File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/backend/tensorflow/numpy.py", line 1545, in reshape \t [[{{node functional_1_1/flatten_1/Reshape}}]] \ttf2xla conversion failed while converting __inference_one_step_on_data_19335[]. Run with TF_DUMP_GRAPH_PREFIX=/path/to/dump/dir and --vmodule=xla_compiler=2 to obtain a dump of the compiled functions. \t [[StatefulPartitionedCall]] [Op:__inference_one_step_on_iterator_19368]", "stack": "--------------------------------------------------------------------------- InvalidArgumentError Traceback (most recent call last) Cell In[16], line 20 12 callbacks = [ 13 keras.callbacks.ModelCheckpoint("jena_dense.keras.x.keras", 14 save_best_only=True) 15 ] 17 model.compile(optimizer="rmsprop", loss="mse", metrics=["mae"]) ---> 20 history = model.fit(train_dataset, 21 epochs=10, 22 validation_data=val_dataset, 23 callbacks=callbacks) 26 model = keras.models.load_model("jena_dense.keras.x.keras") 27 print(f"Test MAE: {model.evaluate(test_dataset)[1]:.2f}") File /drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py:122, in filter_traceback..error_handler(args, kwargs) 119 filtered_tb = _process_traceback_frames(e.traceback) 120 # To get the full stack trace, call: 121 # keras.config.disable_traceback_filtering() --> 122 raise e.with_traceback(filtered_tb) from None 123 finally: 124 del filtered_tb File /drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/tensorflow/python/eager/execute.py:53, in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name) 51 try: 52 ctx.ensure_initialized() ---> 53 tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, 54 inputs, attrs, num_outputs) 55 except core._NotOkStatusException as e: 56 if name is not None: InvalidArgumentError: Graph execution error: Detected at node functional_1_1/flatten_1/Reshape defined at (most recent call last):  only one input size may be -1, not both 0 and 1 Stack trace for op definition: File "usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main File "usr/lib/python3.10/runpy.py", line 86, in _run_code File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/ipykernel_launcher.py", line 18, in  File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/traitlets/config/application.py", line 1075, in launch_instance File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 739, in start File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/tornado/platform/asyncio.py", line 205, in start File "usr/lib/python3.10/asyncio/base_events.py", line 603, in run_forever File "usr/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once File "usr/lib/python3.10/asyncio/events.py", line 80, in _run File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 545, in dispatch_queue File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 534, in process_one File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 437, in dispatch_shell File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 359, in execute_request File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 778, in execute_request File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 446, in do_execute File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/ipykernel/zmqshell.py", line 549, in run_cell File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3075, in run_cell File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3130, in _run_cell File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3334, in run_cell_async File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3517, in run_ast_nodes File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3577, in run_code File "tmp/ipykernel_12551/2185219983.py", line 20, in  File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 117, in error_handler File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/backend/tensorflow/trainer.py", line 325, in fit File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/backend/tensorflow/trainer.py", line 118, in one_step_on_iterator File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/backend/tensorflow/trainer.py", line 106, in one_step_on_data File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/backend/tensorflow/trainer.py", line 57, in train_step File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 117, in error_handler File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/layers/layer.py", line 814, in call File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 117, in error_handler File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/ops/operation.py", line 48, in call File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 156, in error_handler File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/models/functional.py", line 194, in call File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/ops/function.py", line 151, in _run_through_graph File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/models/functional.py", line 578, in call File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 117, in error_handler File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/layers/layer.py", line 814, in call File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 117, in error_handler File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/ops/operation.py", line 48, in call File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 156, in error_handler File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/layers/reshaping/flatten.py", line 54, in call File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/ops/numpy.py", line 4507, in reshape File "drv3/hm3/code/python/tf.2.16.1/.tf.2.16.1/lib/python3.10/site-packages/keras/src/backend/tensorflow/numpy.py", line 1545, in reshape \t [[{{node functional_1_1/flatten_1/Reshape}}]] \ttf2xla conversion failed while converting __inference_one_step_on_data_19335[]. Run with TF_DUMP_GRAPH_PREFIX=/path/to/dump/dir and --vmodule=xla_compiler=2 to obtain a dump of the compiled functions. \t [[StatefulPartitionedCall]] [Op:__inference_one_step_on_iterator_19368]" }
…
On Thu, Mar 21, 2024 at 8:18 AM David R @.> wrote: Wanted to drop in and thank everyone for the sleuthing done up to this point. I had no idea about the environmental variable up there that allows you to debug the loading of the libraries so easily. That was neat. Anyway, to amalgamate all of the suggestions in the thread, you can build an env_vars.sh script within envs//etc/conda/activate.d/ in an anaconda-like install that looks like the following: NVIDIA_DIR=$(dirname $(dirname $(python -c "import nvidia.cudnn;print(nvidia.cudnn.file)"))) export LD_LIBRARY_PATH=${CONDA_PREFIX}/lib/:${LD_LIBRARY_PATH} for dir in $(ls -1d $NVIDIA_DIR//); do if [ -d "${dir}lib" ]; then export LD_LIBRARY_PATH="${dir}lib:$LD_LIBRARY_PATH" if [[ $(basename $dir) == 'cuda_nvcc' ]] ; then export PATH="${dir}bin:$PATH" fi fi done export XLA_FLAGS=--xla_gpu_cuda_data_dir=${CONDA_PREFIX}/lib You do indeed need the two dirname commands for TF 2.16. Similarly, you need to put the cuda_nvcc bin directory in the path so ptxas can be found. Doing it this way also generalizes to your specific conda environment. I've confirmed that you can perform the intro TF tutorial ( https://www.tensorflow.org/tutorials/quickstart/beginner) with this workaround. Having said all that, the TF Linux pip install page badly needs an update. The CUDA Toolkit and CUDNN version numbers are out of date. Since tensorflow[and-cuda] is now the recommended way to install everything for the GPU, the recommendation to install the other libraries independently is now useless as far as I can tell. — Reply to this email directly, view it on GitHub <#63362 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGK34OAGI7LCXG5FDA6WTTYZLFZ7AVCNFSM6AAAAABEOPWBC2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMJSGEZTENJXG4 . You are receiving this because you modified the open/close state.Message ID: @.**>

Please open a new issue for this

Hi guys, weird solution but worked for me
pip install torch

Almost final and automated fix below

Where I found the resolution

TF 2.16.1 Fails to work with GPUs

Solution proposed by "sh-shahrokhi", improved by "ChristofKaufmann"
See specially Comment by COntributor

Related Issues

GPU not detected on WSL2, where I have post some comments
Tensorflow WSL GPU CUDA recognition issue RTX3090
Once gain: tf.2.16.1 fails to recognize GPUs

Other mention on social media

Reddit issue

Exact solution

Temporary fix (after activating environment in which Tensorflow 2.16.1 is installed)
  export NVIDIA_DIR=$(dirname $(dirname $(python -c "import nvidia.cudnn;print(nvidia.cudnn.__file__)")))
  export LD_LIBRARY_PATH=$(echo ${NVIDIA_DIR}/*/lib/ | sed -r 's/\s+/:/g')${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}

Automating the variable set/unset process with Anaconda (one-time setup)

Activate your environement in which TF 2.16.1 is installed
Two files to be created in "anaconda3/envs/<ENV_NAME>/etc/conda"

anaconda3/envs/<ENV_NAME>/etc/conda/activate.d/env_vars.sh

  #!/bin/sh
  export NVIDIA_DIR=$(dirname $(dirname $(python -c "import nvidia.cudnn;print(nvidia.cudnn.__file__)")))
  export LD_LIBRARY_PATH=$(echo ${NVIDIA_DIR}/*/lib/ | sed -r 's/\s+/:/g')${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}

anaconda3/envs/<ENV_NAME>/etc/conda/deactivate.d/env_vars.sh

  #!/bin/sh
  unset NVIDIA_DIR
  unset LD_LIBRARY_PATH

Official documentation to do this via conda.io
Stack-overflow question where I got this Set environment vars when activating conda env

What else helped me

How to list all variables names and their current values?
Conda documenation for "env_vars.sh" for activate/deactivate

Hello everyone, Had the same problem here, Managed to solve it! Thanks to @njzjz , setting TF_CPP_MAX_VLOG_LEVEL=3 shows more information:
soheil@soheil:~$ TF_CPP_MAX_VLOG_LEVEL=3 python -c "import tensorflow as tf;print(tf.config.list_physical_devices('GPU'))"

2024-03-18 00:32:05.073521: I external/local_tsl/tsl/platform/default/dso_loader.cc:59] Successfully opened dynamic library libcudart.so.12
2024-03-18 00:32:05.073737: I external/local_tsl/tsl/platform/cloud/gcs_file_system.cc:855] GCS cache max size = 0 ; block size = 67108864 ; max staleness = 0
2024-03-18 00:32:05.073751: I external/local_tsl/tsl/platform/cloud/ram_file_block_cache.h:64] GCS file block cache is disabled
2024-03-18 00:32:05.073755: I external/local_tsl/tsl/platform/cloud/gcs_file_system.cc:895] GCS DNS cache is disabled, because GCS_RESOLVE_REFRESH_SECS = 0 (or is not set)
2024-03-18 00:32:05.073758: I external/local_tsl/tsl/platform/cloud/gcs_file_system.cc:925] GCS additional header DISABLED. No environment variable set.
2024-03-18 00:32:05.073762: I external/local_tsl/tsl/platform/cloud/gcs_file_system.cc:306] GCS RetryConfig: init_delay_time_us = 1000000 ; max_delay_time_us = 32000000 ; max_retries = 10
2024-03-18 00:32:05.073765: I external/local_tsl/tsl/platform/cloud/gcs_file_system.cc:306] GCS RetryConfig: init_delay_time_us = 1000000 ; max_delay_time_us = 32000000 ; max_retries = 10
2024-03-18 00:32:05.075498: I external/local_tsl/tsl/platform/default/dso_loader.cc:59] Successfully opened dynamic library libcudart.so.12
2024-03-18 00:32:05.103619: I external/local_tsl/tsl/platform/cloud/gcs_file_system.cc:855] GCS cache max size = 0 ; block size = 67108864 ; max staleness = 0
2024-03-18 00:32:05.103641: I external/local_tsl/tsl/platform/cloud/ram_file_block_cache.h:64] GCS file block cache is disabled
2024-03-18 00:32:05.103645: I external/local_tsl/tsl/platform/cloud/gcs_file_system.cc:895] GCS DNS cache is disabled, because GCS_RESOLVE_REFRESH_SECS = 0 (or is not set)
2024-03-18 00:32:05.103648: I external/local_tsl/tsl/platform/cloud/gcs_file_system.cc:925] GCS additional header DISABLED. No environment variable set.
2024-03-18 00:32:05.103652: I external/local_tsl/tsl/platform/cloud/gcs_file_system.cc:306] GCS RetryConfig: init_delay_time_us = 1000000 ; max_delay_time_us = 32000000 ; max_retries = 10
2024-03-18 00:32:05.103655: I external/local_tsl/tsl/platform/cloud/gcs_file_system.cc:306] GCS RetryConfig: init_delay_time_us = 1000000 ; max_delay_time_us = 32000000 ; max_retries = 10
2024-03-18 00:32:05.103933: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-03-18 00:32:05.438768: I external/local_tsl/tsl/platform/cloud/gcs_file_system.cc:855] GCS cache max size = 0 ; block size = 67108864 ; max staleness = 0
2024-03-18 00:32:05.438784: I external/local_tsl/tsl/platform/cloud/ram_file_block_cache.h:64] GCS file block cache is disabled
2024-03-18 00:32:05.438808: I external/local_tsl/tsl/platform/cloud/gcs_file_system.cc:895] GCS DNS cache is disabled, because GCS_RESOLVE_REFRESH_SECS = 0 (or is not set)