2024-03-18 00:32:05.438812: I external/local_tsl/tsl/platform/cloud/gcs_file_system.cc:925] GCS additional header DISABLED. No environment variable set.
2024-03-18 00:32:05.438839: I external/local_tsl/tsl/platform/cloud/gcs_file_system.cc:306] GCS RetryConfig: init_delay_time_us = 1000000 ; max_delay_time_us = 32000000 ; max_retries = 10
2024-03-18 00:32:05.438843: I external/local_tsl/tsl/platform/cloud/gcs_file_system.cc:306] GCS RetryConfig: init_delay_time_us = 1000000 ; max_delay_time_us = 32000000 ; max_retries = 10
2024-03-18 00:32:05.609776: I external/local_tsl/tsl/platform/default/dso_loader.cc:70] Could not load dynamic library 'libnvinfer.so.8.6.1'; dlerror: libnvinfer.so.8.6.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /var/lib:/usr/lib:/home/soheil/anaconda3/lib/python3.10/site-packages/nvidia/cudnn/lib/::/usr/local/cuda-12.4/lib64:/usr/local/cuda-12.4/lib64
2024-03-18 00:32:05.609875: I external/local_tsl/tsl/platform/default/dso_loader.cc:70] Could not load dynamic library 'libnvinfer_plugin.so.8.6.1'; dlerror: libnvinfer_plugin.so.8.6.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /var/lib:/usr/lib:/home/soheil/anaconda3/lib/python3.10/site-packages/nvidia/cudnn/lib/::/usr/local/cuda-12.4/lib64:/usr/local/cuda-12.4/lib64
2024-03-18 00:32:05.609882: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2024-03-18 00:32:05.937145: I external/local_xla/xla/parse_flags_from_env.cc:207] For env var TF_XLA_FLAGS found arguments:
2024-03-18 00:32:05.937182: I external/local_xla/xla/parse_flags_from_env.cc:209]   argv[0] = <argv[0]>
2024-03-18 00:32:05.937188: I external/local_xla/xla/parse_flags_from_env.cc:207] For env var TF_JITRT_FLAGS found arguments:
2024-03-18 00:32:05.937192: I external/local_xla/xla/parse_flags_from_env.cc:209]   argv[0] = <argv[0]>
2024-03-18 00:32:05.937215: I tensorflow/compiler/jit/xla_cpu_device.cc:46] Not creating XLA devices, tf_xla_enable_xla_devices not set and XLA device creation not requested
2024-03-18 00:32:05.937219: I tensorflow/compiler/jit/xla_gpu_device.cc:49] Not creating XLA devices, tf_xla_enable_xla_devices not set and XLA devices creation not required
2024-03-18 00:32:05.937662: I external/local_tsl/tsl/platform/default/dso_loader.cc:59] Successfully opened dynamic library libcuda.so.1
2024-03-18 00:32:05.983926: I external/local_xla/xla/stream_executor/cuda/cuda_diagnostics.cc:192] found DLL info with name: /lib/x86_64-linux-gnu/libcuda.so.1
2024-03-18 00:32:05.983958: I external/local_xla/xla/stream_executor/cuda/cuda_diagnostics.cc:197] found DLL info with resolved path: /usr/lib/x86_64-linux-gnu/libcuda.so.550.54.14
2024-03-18 00:32:05.983978: I external/local_xla/xla/stream_executor/cuda/cuda_diagnostics.cc:97] version string "550.54.14" made value 550.54.14
2024-03-18 00:32:05.983999: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:968] trying to read NUMA node for device ordinal: 0
2024-03-18 00:32:05.984031: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2024-03-18 00:32:05.984184: I tensorflow/core/common_runtime/gpu/gpu_device.cc:2219] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: NVIDIA GeForce RTX 2060 computeCapability: 7.5
coreClock: 1.2GHz coreCount: 30 deviceMemorySize: 5.77GiB deviceMemoryBandwidth: 244.97GiB/s
2024-03-18 00:32:05.984205: I external/local_tsl/tsl/platform/default/dso_loader.cc:59] Successfully opened dynamic library libcudart.so.12
2024-03-18 00:32:06.002289: I external/local_tsl/tsl/platform/default/dso_loader.cc:59] Successfully opened dynamic library libcublas.so.12
2024-03-18 00:32:06.002375: I external/local_tsl/tsl/platform/default/dso_loader.cc:59] Successfully opened dynamic library libcublasLt.so.12
2024-03-18 00:32:06.004118: I external/local_tsl/tsl/platform/default/dso_loader.cc:59] Successfully opened dynamic library libcufft.so.11
2024-03-18 00:32:06.006168: I external/local_tsl/tsl/platform/default/dso_loader.cc:59] Successfully opened dynamic library libcusolver.so.11
2024-03-18 00:32:06.006254: I external/local_tsl/tsl/platform/default/dso_loader.cc:59] Successfully opened dynamic library libcusparse.so.12
2024-03-18 00:32:06.006336: I external/local_tsl/tsl/platform/default/dso_loader.cc:70] Could not load dynamic library 'libcudnn.so.8'; dlerror: libcudnn.so.8: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /var/lib:/usr/lib:/home/soheil/anaconda3/lib/python3.10/site-packages/nvidia/cudnn/lib/::/usr/local/cuda-12.4/lib64:/usr/local/cuda-12.4/lib64
2024-03-18 00:32:06.006343: W tensorflow/core/common_runtime/gpu/gpu_device.cc:2251] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
[]

what I did, was searching for where cudnn is placed:
soheil@soheil:~$ sudo find / -name "*libcudnn.so*"

/home/soheil/.local/lib/python3.10/site-packages/nvidia/cudnn/lib/libcudnn.so.8
/usr/lib/x86_64-linux-gnu/libcudnn.so.9
/usr/lib/x86_64-linux-gnu/libcudnn.so
/usr/lib/x86_64-linux-gnu/libcudnn.so.9.0.0
find: ‘/run/user/1000/doc’: Permission denied
find: ‘/run/user/1000/gvfs’: Permission denied

adding /usr/lib/x86_64-linux-gnu to LD_LIBRARY_PATH solved my problem :)
soheil@soheil:~$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib/x86_64-linux-gnu

soheil@soheil:~$ python -c "import tensorflow as tf;print(tf.config.list_physical_devices('GPU'))"

2024-03-18 00:35:24.292809: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-03-18 00:35:24.797787: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2024-03-18 00:35:25.127729: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2024-03-18 00:35:25.148460: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2024-03-18 00:35:25.148676: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]

Hope this helps

It does help! I had the same problem (Ubuntu 22.04, tensorflow 2.16.1): "Cannot dlopen some GPU libraries...", even though the NVIDIA drivers were properly installed.
Tensorflow could not find the libcudnn.so.8 shared library. Thanks to SoheilKhatibi for providing the solution that worked for me.
In my case, I found the shared library in my Python virtual environment (named "venv1), under /venv1/lib/python3.10/site-packages/nvidia/cudnn/lib/.
So adding a line in .bashrc  ("export LD_LIBRARY_PATH=" pointing to the proper path) solved the problem.
If you do this, don't forget to reload .bashr using: source ~/.bashrc
My next problem was that, even though my tensorflow Python code worked fine when run from the terminal, the same script executed within PyCharm still threw the "Cannot dlopen some GPU libraries" error.
To solve this, you need to go to PyCharm's  Run menu and select Edit configurations. In the left panel select the script for which you want to solve the GPU error, and open the Environment variables (right panel).
Add the user environment variable LD_LIBRARY_PATH and give it the value corresponding to the path you've put in .bashr. Save and run, and the error is gone in Pycharm too!

Thanks for all hints.
I have written all steps in my blog.
https://mobinshaterian.medium.com/use-gpu-in-tensorflow-on-ubuntu-22-04-f033e59cf5cb

CC: @learning-to-play

I have tested with fresh conda environment with TF2.16v and it is able to detect GPUs. I am attaching the logs below for reference.
Tf2.16.1_gpu_test.txt
The colab environment also works fine as per gist.
Could anyone please confirm the environment details on which GPU fails to detect ?

@SuryanarayanaY you're running it on python 3.10, please try on 3.11 or 3.12

I have tested with fresh conda environment with TF2.16v and it is able to detect GPUs. I am attaching the logs below for reference.
Tf2.16.1_gpu_test.txt
The colab environment also works fine as per gist.
Could anyone please confirm the environment details on which GPU fails to detect ?

Followed your logs. Performed the exact steps but TensorFlow still can't find my Nvidia RTX3060 (with Ubuntu OS).

Followed your logs. Performed the exact steps but TensorFlow still can't find my Nvidia RTX3060 (with Ubuntu OS).

Could you please submit the logs? Can you confirm the outputs of nvidia-smi and whereis cuda ? Also check the compatible CUDA driver for  Nvidia RTX3060 has been installed or not?

Followed your logs. Performed the exact steps but TensorFlow still can't find my Nvidia RTX3060 (with Ubuntu OS).

Could you please submit the logs? Can you confirm the outputs of nvidia-smi and whereis cuda ? Also check the compatible CUDA driver for Nvidia RTX3060 has been installed or not?

logs.txt
Here you are.

@SuryanarayanaY you're running it on python 3.10, please try on 3.11 or 3.12 gist

May be Colab has 3.10v.I tried but my GPU usage exceeded free tier limit. But in the VM it has Python3.12 environment.
There seems to be a problem but not sure with what environment or is it due to some specific GPU/CUDA driver version.
If anyone got below type logs during installation of CUDA driver/lib then probably they need to set the paths.
Please make sure that
 -   PATH includes /usr/local/cuda-12.0/bin
 -   LD_LIBRARY_PATH includes /usr/local/cuda-12.0/lib64, or, add /usr/local/cuda-12.0/lib64 to /etc/ld.so.conf and run ldconfig as root

@SuryanarayanaY you're running it on python 3.10, please try on 3.11 or 3.12 gist

May be Colab has 3.10v.I tried but my GPU usage exceeded free tier limit. But in the VM it has Python3.12 environment.
There seems to be a problem but not sure with what environment or is it due to some specific GPU/CUDA driver version.
If anyone got below type logs during installation of CUDA driver/lib then probably they need to set the paths.
Please make sure that
 -   PATH includes /usr/local/cuda-12.0/bin
 -   LD_LIBRARY_PATH includes /usr/local/cuda-12.0/lib64, or, add /usr/local/cuda-12.0/lib64 to /etc/ld.so.conf and run ldconfig as root

So, are we expecting a fix for the bug @SuryanarayanaY ? Is it reported?

As per this comment for TF2.16v Cuda driver version >= 545 is needed. Please note that installation of cuda driver is manual and left it under user scope.

So, are we expecting a fix for the bug @SuryanarayanaY ? Is it reported?

I think you are using CUDA driver 535 version. Can you upgrade it to higher as mentioned above.

As per this comment for TF2.16v Cuda driver version >= 545 is needed. Please note that installation of cuda driver is manual and left it under user scope.

So, are we expecting a fix for the bug @SuryanarayanaY ? Is it reported?

I think you are using CUDA driver 535 version. Can you upgrade it to higher as mentioned above.

@SuryanarayanaY it should be clearly stated on the TensorFlow installation procedure!! Here: https://www.tensorflow.org/install/pip

@SuryanarayanaY Steps to reproduce:
conda create -n tensorflow-test-new python=3.12 -y
conda activate tensorflow-test-new
pip install tensorflow[and-cuda]==2.16.1
python
import tensorflow as tf;print(tf.__version__);tf.config.list_physical_devices('GPU')

gives this message:
2024-04-05 13:54:50.483020: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-04-05 13:54:50.982460: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2.16.1
2024-04-05 13:54:51.320296: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2024-04-05 13:54:51.320653: W tensorflow/core/common_runtime/gpu/gpu_device.cc:2251] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
[]

I use rtx3080ti but I think it's issue for all gpus nvidia-smi:
+---------------------------------------------------------------------------------------+
NVIDIA-SMI 545.29.06              Driver Version: 545.29.06    CUDA Version: 12.3

But installing tensorflow==2.15.1 with python 3.10 works without additional steps

So, even with NVIDIA GPU driver version 545 still TF 2.16 could not find GPU devices. The bug should be reported and fixed.

Wanted to stop by with an update. Tried a fresh install this morning via miniforge (anaconda) with python 3.11.8. Followed the instructions on the TF website with the simple pip install tensorflow[and-cuda]. Same thing -- would not recognize the GPU. Adding in the env_vars.sh fix I noted above makes the GPU recognizable.
Output from nvidia-smi:
Fri Apr  5 09:27:21 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.67                 Driver Version: 550.67         CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  Quadro RTX 4000 with Max...    Off |   00000000:01:00.0 Off |                  N/A |
| N/A   49C    P8              7W /   90W |       4MiB /   8192MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A      2306      G   /usr/bin/gnome-shell                            2MiB |
+-----------------------------------------------------------------------------------------+

In colab and gpu VM and docker image, you have cuda cuda installed as a
system lib. So tensorflow looks into /usr/lib and finds it. But the
standard thing that has worked before 2.16, and the expexpeted thing from
pip install tensorflow[and-cuda] is that tensorflow should also look into
cuda libraries that were installed via pip. If it looks, it can find nvcc,
cufft, cublas,... in there. The problem is that it just doesn't consider
them. Pytorch and tensorflow 2.15 do. You sure can install cuda as a system
lib and 2.16 works, but it just would be unnecessary and impossible for
users without admin rights. The env-var fix posted above just adds those
pip installed cuda libraries to library path so tensorflow finds and uses
them.
The fix is simple, you just needs to modify the logic that searches for
cuda librariew. but it requires modifying cpp files and recompile
tensorflow to be tested, which exceeds resources of most users.
On Fri, Apr 5, 2024, 5:49 a.m. Surya @.***> wrote:

As per this comment
https://github.com/tensorflow/tensorflow/issues/62234#issuecomment-1997135428
for TF2.16v Cuda driver version >= 545 is needed. Please note that
installation of cuda driver is manual and left it under user scope.
So, are we expecting a fix for the bug @SuryanarayanaY
https://github.com/SuryanarayanaY ? Is it reported?
I think you are using CUDA driver 535 version. Can you upgrade it to
higher as mentioned above.
—
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/63362#issuecomment-2039602159,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AZRPJAGU5GVQ4HTMYNU6O2TY32FURAVCNFSM6AAAAABEOPWBC2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMZZGYYDEMJVHE
.
You are receiving this because you were mentioned.Message ID:
@.***>

@sh-shahrokhi Yep, that's what I think we've been dancing around here -- it's a path problem, not a library problem explicitly.

As I understand the issue it is clear from the discussion that users with Linux OS and CUDA-enabled GPUs in order to utilize their GPUs should manually perform some additional actions (namely: adjust the LD_LIBRARY_PATH environment variable to include the directory where cuDNN is located and locate a compatible version of ptxas in the site-packages directory of a Python installation, under a CUDA toolkit installation path virtual_environment/lib/python3.10/site-packages/nvidia/cuda_nvcc/bin and add this specific path to the environment variables). @SuryanarayanaY should that be officially communicated as part of the procedure to pip install tensorflow[and-cuda] for users with GPUs and Linux OS?
Should it be fixed in the next versions of TensorFlow? I rest my case.

Yes, please!
On Fri, Apr 5, 2024 at 12:35 PM Sotiris Gkouzias @.***>
wrote:

As I understand the issue it is clear from the discussion that users with
Linux OS and CUDA-enabled GPUs in order to utilize their GPUs should
manually perform some additional actions (namely: adjust the
LD_LIBRARY_PATH environment variable to include the directory where cuDNN
is located and locate a compatible version of ptxas in the site-packages
directory of a Python installation, under a CUDA toolkit installation path
virtual_environment/lib/python3.10/site-packages/nvidia/cuda_nvcc/bin and
add this specific path to the environment variables). @SuryanarayanaY
https://github.com/SuryanarayanaY should that be officially
communicated as part of the procedure to pip install tensorflow[and-cuda]
for users with GPUs and Linux OS?
Should it be fixed in the next versions of TensorFlow? I rest my case.
—
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/63362#issuecomment-2040228959,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAGK34I2SYXBSCBDZNAQ6ATY33HFPAVCNFSM6AAAAABEOPWBC2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANBQGIZDQOJVHE
.
You are receiving this because you modified the open/close state.Message
ID: @.***>

I thank everyone here for leading me to a simple 'fix' for this issue...copy/paste. Sorry, but you guys are leaps and bounds above my understanding of Linux, tensorflow, and wsl2...I just wanted to tinker with a little ML using my rtx 3090. Since Windows apparently isn't supported, here I am wondering what I've gotten myself into. If you are like me and can follow the most basic of instructions, here's what I did after reading this thread...I used this command TF_CPP_MAX_VLOG_LEVEL=3 python -c "import tensorflow as tf;print(tf.config.list_physical_devices('GPU'))" and the output from that told me what libraries weren't found. More importantly, it told me the one it did find. Because I use wsl2, Windows explorer was all I needed to find the unfindable library files, that were all there in there own nice little folders, including the one that was found. I simply copy/pasted all the files of the ones that weren't found, into the same location of the one that was found. Worked for me, but one thing's become abundantly clear...mileage may vary. Good luck. I can't believe I spent two days trying to figure out all this command line stuff and bash, zsh, echo this and that...how is this even a thing?

As I understand the issue it is clear from the discussion that users with Linux OS and CUDA-enabled GPUs in order to utilize their GPUs should manually perform some additional actions (namely: adjust the LD_LIBRARY_PATH environment variable to include the directory where cuDNN is located and locate a compatible version of ptxas in the site-packages directory of a Python installation, under a CUDA toolkit installation path virtual_environment/lib/python3.XX/site-packages/nvidia/cuda_nvcc/bin and add this specific path to the environment variables). @SuryanarayanaY should that be officially communicated as part of the procedure to pip install tensorflow[and-cuda] for users with GPUs and Linux OS? Should it be fixed in the next versions of TensorFlow? I rest my case.

Agree. We used to have the instructions for setting the cuda path to the environment variable LD_LIBRARY_PATH in earlier versions. I think either we need to add these to documentation or atleast has to add a note in pip install guide that user has to setup the path for his own environment.
May be same instructions may not work for all environments and hence it might have discorded. Anyways adding a note on same in the installation guide should be must that can avoid confusions.
If anyone from here willing to contribute to add the required notes please feel free to help. The changes can be proposed here at this doc source which will be reflected in this page.

I created a respective pull request and it is pending review.

There should also be instructions for venv users.
On Mon, Apr 8, 2024, 11:48 a.m. Sotiris Gkouzias @.***>
wrote:

As I understand the issue it is clear from the discussion that users with
Linux OS and CUDA-enabled GPUs in order to utilize their GPUs should
manually perform some additional actions (namely: adjust the
LD_LIBRARY_PATH environment variable to include the directory where cuDNN
is located and locate a compatible version of ptxas in the site-packages
directory of a Python installation, under a CUDA toolkit installation path
virtual_environment/lib/python3.XX/site-packages/nvidia/cuda_nvcc/bin and
add this specific path to the environment variables). @SuryanarayanaY
https://github.com/SuryanarayanaY should that be officially
communicated as part of the procedure to pip install tensorflow[and-cuda]
for users with GPUs and Linux OS? Should it be fixed in the next versions
of TensorFlow? I rest my case.
Agree. We used to have the instructions for setting the cuda path to the
environment variable LD_LIBRARY_PATH in earlier versions. I think either
we need to add these to documentation or atleast has to add a note in pip
install guide that user has to setup the path for his own environment.
May be same instructions may not work for all environments and hence it
might have discorded. Anyways adding a note on same in the installation
guide should be must that can avoid confusions.
If anyone from here willing to contribute to add the required notes please
feel free to help. The changes can be proposed here at this doc source
https://github.com/tensorflow/docs/blob/master/site/en/install/pip.md
which will be reflected in this page
https://www.tensorflow.org/install/pip.
I created a respective pull request and it is pending review.
—
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/63362#issuecomment-2043324498,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AZRPJAEWMOXSOWF2BUOCU3DY4LJ7ZAVCNFSM6AAAAABEOPWBC2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANBTGMZDINBZHA
.
You are receiving this because you were mentioned.Message ID:
@.***>

There should also be instructions for venv users.
On Mon, Apr 8, 2024, 11:48 a.m. Sotiris Gkouzias @.***>
wrote:

As I understand the issue it is clear from the discussion that users with
Linux OS and CUDA-enabled GPUs in order to utilize their GPUs should
manually perform some additional actions (namely: adjust the
LD_LIBRARY_PATH environment variable to include the directory where cuDNN
is located and locate a compatible version of ptxas in the site-packages
directory of a Python installation, under a CUDA toolkit installation path
virtual_environment/lib/python3.XX/site-packages/nvidia/cuda_nvcc/bin and
add this specific path to the environment variables). @SuryanarayanaY
https://github.com/SuryanarayanaY should that be officially
communicated as part of the procedure to pip install tensorflow[and-cuda]
for users with GPUs and Linux OS? Should it be fixed in the next versions
of TensorFlow? I rest my case.
Agree. We used to have the instructions for setting the cuda path to the
environment variable LD_LIBRARY_PATH in earlier versions. I think either
we need to add these to documentation or atleast has to add a note in pip
install guide that user has to setup the path for his own environment.
May be same instructions may not work for all environments and hence it
might have discorded. Anyways adding a note on same in the installation
guide should be must that can avoid confusions.
If anyone from here willing to contribute to add the required notes please
feel free to help. The changes can be proposed here at this doc source
https://github.com/tensorflow/docs/blob/master/site/en/install/pip.md
which will be reflected in this page
https://www.tensorflow.org/install/pip.
I created a respective pull request and it is pending review.
—
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/63362#issuecomment-2043324498,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AZRPJAEWMOXSOWF2BUOCU3DY4LJ7ZAVCNFSM6AAAAABEOPWBC2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANBTGMZDINBZHA
.
You are receiving this because you were mentioned.Message ID:
@.***>

I totally agree. I will update the pull request accordingly later on.

Following on from the post  by chaudharyachint08, I did the following to automate it on venv.
I editied bin/activate in the folder of my venv and added the two lines at the end of the file:
export NVIDIA_DIR=$(dirname $(dirname $(python -c "import nvidia.cudnn;print(nvidia.cudnn.__file__)")))
 export LD_LIBRARY_PATH=$(echo ${NVIDIA_DIR}/*/lib/ | sed -r 's/\s+/:/g')${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}

Then while editing the same file, added the 2 unset lines inside the deactivate function (before the } closed curly bracket):
 unset NVIDIA_DIR
 unset LD_LIBRARY_PATH

I had tested it by entering the 2 lines in the terminal and my GPU was detected, so this was just the automation when the venc is activated.

There should also be instructions for venv users.
On Mon, Apr 8, 2024, 11:48 a.m. Sotiris Gkouzias @.***>
wrote:

As I understand the issue it is clear from the discussion that users with
Linux OS and CUDA-enabled GPUs in order to utilize their GPUs should
manually perform some additional actions (namely: adjust the
LD_LIBRARY_PATH environment variable to include the directory where cuDNN
is located and locate a compatible version of ptxas in the site-packages
directory of a Python installation, under a CUDA toolkit installation path
virtual_environment/lib/python3.XX/site-packages/nvidia/cuda_nvcc/bin and
add this specific path to the environment variables). @SuryanarayanaY
https://github.com/SuryanarayanaY should that be officially
communicated as part of the procedure to pip install tensorflow[and-cuda]
for users with GPUs and Linux OS? Should it be fixed in the next versions
of TensorFlow? I rest my case.
Agree. We used to have the instructions for setting the cuda path to the
environment variable LD_LIBRARY_PATH in earlier versions. I think either
we need to add these to documentation or atleast has to add a note in pip
install guide that user has to setup the path for his own environment.
May be same instructions may not work for all environments and hence it
might have discorded. Anyways adding a note on same in the installation
guide should be must that can avoid confusions.
If anyone from here willing to contribute to add the required notes please
feel free to help. The changes can be proposed here at this doc source
https://github.com/tensorflow/docs/blob/master/site/en/install/pip.md
which will be reflected in this page
https://www.tensorflow.org/install/pip.
I created a respective pull request and it is pending review.
—
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/63362#issuecomment-2043324498,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AZRPJAEWMOXSOWF2BUOCU3DY4LJ7ZAVCNFSM6AAAAABEOPWBC2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANBTGMZDINBZHA
.
You are receiving this because you were mentioned.Message ID:
@.***>

I totally agree. Will try to update the pull request accordingly later on.

Updated the respective pull request (pending review) yesterday. The fix was successfully tested today by @weavermech as well.

There should also be instructions for venv users.
On Mon, Apr 8, 2024, 11:48 a.m. Sotiris Gkouzias @.***>
wrote:

As I understand the issue it is clear from the discussion that users with
Linux OS and CUDA-enabled GPUs in order to utilize their GPUs should
manually perform some additional actions (namely: adjust the
LD_LIBRARY_PATH environment variable to include the directory where cuDNN
is located and locate a compatible version of ptxas in the site-packages
directory of a Python installation, under a CUDA toolkit installation path
virtual_environment/lib/python3.XX/site-packages/nvidia/cuda_nvcc/bin and
add this specific path to the environment variables). @SuryanarayanaY
https://github.com/SuryanarayanaY should that be officially
communicated as part of the procedure to pip install tensorflow[and-cuda]
for users with GPUs and Linux OS? Should it be fixed in the next versions
of TensorFlow? I rest my case.
Agree. We used to have the instructions for setting the cuda path to the
environment variable LD_LIBRARY_PATH in earlier versions. I think either
we need to add these to documentation or atleast has to add a note in pip
install guide that user has to setup the path for his own environment.
May be same instructions may not work for all environments and hence it
might have discorded. Anyways adding a note on same in the installation
guide should be must that can avoid confusions.
If anyone from here willing to contribute to add the required notes please
feel free to help. The changes can be proposed here at this doc source
https://github.com/tensorflow/docs/blob/master/site/en/install/pip.md
which will be reflected in this page
https://www.tensorflow.org/install/pip.
I created a respective pull request and it is pending review.
—
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/63362#issuecomment-2043324498,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AZRPJAEWMOXSOWF2BUOCU3DY4LJ7ZAVCNFSM6AAAAABEOPWBC2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANBTGMZDINBZHA
.
You are receiving this because you were mentioned.Message ID:
@.***>

I totally agree. Will try to update the pull request accordingly later on.

Updated the respective pull request (pending review) yesterday. The fix was successfully tested today by @weavermech as well.
Added instructions needed to resolve the ptxas issue.

I think the expected solution would be a new release that fixes this issue, so setting LD_LIBRARY_PATH is not needed like it is done in 2.15.1, it would be downgrade for users to do such workarounds it should just work with: pip install tensorflow[and-cuda]

I think the expected solution would be a new release that fixes this issue, so setting LD_LIBRARY_PATH is not needed like it is done in 2.15.1, it would be downgrade for users to do such workarounds it should just work with: pip install tensorflow[and-cuda]

@niko247 undoubtedly true. It is crystal clear that TF 2.16.1 does not work with the simple pip install tensorflow[and-cuda] command to actually utilize CUDA locally and no relative guidelines where provided yet to resolve this.
It seems practically impossible for someone owning a PC with CUDA-enabled GPU to perform deep learning experiments with TensorFlow and utilize his GPU locally without manually performing some extra steps not included (until today) in the official TensorFlow documentation of the standard installation procedure of TensorFlow for Linux users with GPUs at least as a temporal fix! That's why I submitted the pull request in good faith and for the shake of all users as TensorFlow is "An Open Source Machine Learning Framework for Everyone".
Hope that the next patch version of TensorFlow will fix the bug as soon as possible!

I think the expected solution would be a new release that fixes this issue, so setting LD_LIBRARY_PATH is not needed like it is done in 2.15.1, it would be downgrade for users to do such workarounds it should just work with: pip install tensorflow[and-cuda]

Thanks! Downgrade to 2.15.1 works well for me too (with TF prob <=0.23).
conda create -n tmpenv python=3.11
conda activate tmpenv
pip install tensorflow[and-cuda]==2.15.1
python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"

pip install tensorflow-probability==0.23.0

got it work :) first https://developer.nvidia.com/rdp/cudnn-archive?source=post_page-----bfbeb77e7c89--------------------------------
then download Local Installer for Ubuntu22.04 x86_64 (Deb)
unpack and install libcudnn8_<IP_ADDRESS>-1+cuda12.2_amd64.deb
sudo dpkg -i libcudnn8_<IP_ADDRESS>-1+cuda12.2_amd64.deb

Selecting previously unselected package libcudnn8.
(Reading database ... 47318 files and directories currently installed.)
Preparing to unpack libcudnn8_<IP_ADDRESS>-1+cuda12.2_amd64.deb ...
Unpacking libcudnn8 (<IP_ADDRESS>-1+cuda12.2) ...
Setting up libcudnn8 (<IP_ADDRESS>-1+cuda12.2) ...

python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"

2024-03-11 10:27:47.879686: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-03-11 10:27:47.909157: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-03-11 10:27:48.316717: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2024-03-11 10:27:48.664469: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:984] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2024-03-11 10:27:48.688059: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:984] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2024-03-11 10:27:48.688111: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:984] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]

works like a charm, thank you. :)

got it work :) first https://developer.nvidia.com/rdp/cudnn-archive?source=post_page-----bfbeb77e7c89--------------------------------
then download Local Installer for Ubuntu22.04 x86_64 (Deb)
unpack and install libcudnn8_<IP_ADDRESS>-1+cuda12.2_amd64.deb
sudo dpkg -i libcudnn8_<IP_ADDRESS>-1+cuda12.2_amd64.deb

Selecting previously unselected package libcudnn8.
(Reading database ... 47318 files and directories currently installed.)
Preparing to unpack libcudnn8_<IP_ADDRESS>-1+cuda12.2_amd64.deb ...
Unpacking libcudnn8 (<IP_ADDRESS>-1+cuda12.2) ...
Setting up libcudnn8 (<IP_ADDRESS>-1+cuda12.2) ...

python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"

2024-03-11 10:27:47.879686: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-03-11 10:27:47.909157: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-03-11 10:27:48.316717: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2024-03-11 10:27:48.664469: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:984] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2024-03-11 10:27:48.688059: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:984] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2024-03-11 10:27:48.688111: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:984] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]

works like a charm, thank you. :)

@Sivabooshan congrats! Good for you :) . However note that:

@JuanVargas who raised the issue under discussion has a certain setup including CUDA version 12.4 which is not compatible with TensorFlow version 2.16.1 and that's why he might need to install TensorFlow in a virtual environment so as  to avoid downgrading CUDA and a potential systemic global pollution (if you install packages globally, they clutter your main Python installation and can potentially interfere with system processes. Virtual environments protect your system-wide environment from this),
It turns out that when you pip install tensorflow[and-cuda] all required nvidia libs are installed. You just need to configure manually the environment variables as appropriate in order to utilize them and run TensorFlow with GPU,
Until today the officially documented TensorFlow standard installation procedure for Linux users with GPUs does not include the additional steps required to perform deep learning experiments with TensorFlow version 2.16.1 and utilize GPU locally. That's why I submitted the pull request in good faith and for the shake of all users as TensorFlow is "An Open Source Machine Learning Framework for Everyone".

Hope that the next patch version of TensorFlow will fix the bug as soon as possible!

Thanks @sh-shahrokhi. I thought it was path related. Modified slightly to make it python version independent if you put it in your conda environment activation ([environment]/etc/activate.d/env_vars.sh).
NVIDIA_DIR=$(dirname $(dirname $(python -c "import nvidia.cudnn;print(nvidia.cudnn.__file__)")))
for dir in $NVIDIA_DIR/*; do
    if [ -d "$dir/lib" ]; then
        export LD_LIBRARY_PATH="$dir/lib:$LD_LIBRARY_PATH"
    fi
done

This is not a resolution as this post install step should not be necessary.
W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
I can't seem to do similar tricks to resolve the TensorRT issues when installed similarly into the conda environment. Any ideas?

It has been very helpful! However the path I had to insert the script in was: [environment]/etc/conda/activate.d/.  See this stack overflow question.

Hello!  I outlined this behavior in a duplicate ticket ( #65842 ).  Torch now installs its CUDA dependencies using the NVIDIA-managed pip packages.   It doesn't appear to require the LD_LIBRARY_PATH to be set for the linker, like TF still does.  I assume (perhaps incorrectly) this is because they're manually sourcing libs from the venv.  Is this functionality on the roadmap for TF?
Thank you!

Thanks @huangyh09 !! Downgrading TF to 2.15.1 did the trick :)

I installed TensorFlow following the instructions on https://www.tensorflow.org/install/pip. However, instead of using the default command, I specified the version as 2.15.1. After completing the installation, everything works fine with GPU support.
Through my testing, it seems that TensorFlow version 2.16.1 cannot operate properly on WSL, but it works fine on a Linux host (my Linux host version is Ubuntu 22.04).

i tried 2.15.1 though with python 3.11.9, it works flawlessly. I think not using 2.16.1 at the current stage is the best solution atm

Running latest CUDA & cuDNN. 2.15.1 finds them, 2.16 does not. And neither does tf-nightly (2.17.x). How can such a large company make such buggy releases?

Running latest CUDA & cuDNN. 2.15.1 finds them, 2.16 does not. And neither does tf-nightly (2.17.x). How can such a large company make such buggy releases?

Team change, priority change and https://github.com/tensorflow/tensorflow/issues/63362#issuecomment-1990133275

@tknuutila and @mihaimaruseac I agree with you totally! IMHO, at least the Tensorflow team should suggest a temporal solution and change accordingly the `pip install tensorflow[and-cuda]' guidelines.
It is surely a pity by all means! And when I read the comment below from mr. Álvaro Rodríguez I get really disappointed because I really love TensorFlow and Keras 3.0 :

I have to say, that the drop of GPU support for windows, the lack of documentation and support for cpp, the lack of support and documentation for TensorFlow lite, the lack of TFrecord multi-platform standalone libraries… and so on is simply a strategy that will kill the library long term. Except for very niche projects in large companies.
Other platforms like PyTorch are investing in easy to use multi-platform solutions. If (or when) someone actually puts a solution powerful, stable, easy to access and easy to import and export to other platforms and languages. Private enthusiasts, researchers and academics will drop TensorFlow. And don’t forget that industries rely on specialists who learned in academia and come from research.
In a time of AI revolution, where the technology is more popular than ever, and is being added to literally everything. In my opinion, TensorFlow is neglecting everything outside Python-Linux, dropping an already lacking support for interoperability, and not investing in accessibility.
I’m saying this as a researcher and professor working in a computer science lab in an university. I write this just after investing almost 100 hours trying to simply build TensorFlow-cc to add some basic capabilities to a research project for the European Union, I failed. Also the absolute lack of recent information anywhere about TensorFlow-cc, and the responses I have seen to old threads lets me know most people gave up the same way I’m ready to tell my whole team to abandon TensorFlow and try other solutions.
I have seen others in my lab commenting similar concerns and frustrations. Many of our researchers are already moving away from TensorFlow and soon the whole department will follow.
For context. The Computer Science department is the largest department in my university, and serves the most important IT faculty in the Northwest of Spain.
For us ML is thriving. In addition to the Bachelor degree in Computer Science, where ML is more than present, being the most requested in the entire university. We opened a new one in Data Science, and are opening a new one in Artificial Intelligence. Next year we will be adding new classes and teachers to be able to serve the increasing number of students in two of the three Machine Learning subjects I teach… As far as I know, none of them will learn TensorFlow, none of them are using it in their personal projects and none of them will use it their degrees. They instead will be using Julia, Matlab, OpenCv , PyTorch, Scikit-Learn and other solutions.
Which leaves me to the industry sector. I worked also as a researcher in a public hospital in a project about diabetes, and in a private research center dedicated to laser and manufacturing. They all used TensorFlow, the same my laboratory did. I have been told they are all moving away from it, currently opting for a Scikit-Learn+OpenCv and PyTorch based approach. The reason is in one case the drop of GPU support for windows, and in the other a perceived drop of support combined with lack of interoperability.
The thing is, nobody moves away from a technology they spent years using and learning unless the technology fails them. And once you move away from something because of a problem, if you find a solution somewhere else, you will probably never return.
That is what is happening in TensorFlow. Google has intentionally dropped the ball with support, documentation, accessibility, ease of use, interoperability across languages and interoperability across platforms… so others will raise to the occasion.
Simply put, TensorFlow is becoming the Bing search engine with regards to AI.

Until today the officially documented TensorFlow standard installation procedure for Linux users with GPUs does not include the additional steps required to perform deep learning experiments with TensorFlow version 2.16.1 and utilize GPU locally. That's why I submitted the pull request in good faith and for the shake of all users as TensorFlow is "An Open Source Machine Learning Framework for Everyone".
Hope that the next patch version of TensorFlow will fix the bug as soon as possible!

Agree completely with the comments regarding the frustrating  situation
with tf 2.16.x . No wonder why torch is gaining the race of adoption
On Wed, May 15, 2024, 1:18 AM Sotiris Gkouzias @.***>
wrote:

@tknuutila https://github.com/tknuutila and @mihaimaruseac
https://github.com/mihaimaruseac I agree with you totally! IMHO, at
least the Tensorflow team should suggest a temporal solution and change
accordingly the `pip install tensorflow[and-cuda]' guidelines.
It is surely a pity by all means! And when I read the comment
https://discuss.tensorflow.org/t/tensorflow-and-its-future/23869/3below
from mr. Álvaro Rodríguez I get really disappointed because I really love
TensorFlow and Keras 3.0 :
I have to say, that the drop of GPU support for windows, the lack of
documentation and support for cpp, the lack of support and documentation
for TensorFlow lite, the lack of TFrecord multi-platform standalone
libraries… and so on is simply a strategy that will kill the library long
term. Except for very niche projects in large companies.
Other platforms like PyTorch are investing in easy to use multi-platform
solutions. If (or when) someone actually puts a solution powerful, stable,
easy to access and easy to import and export to other platforms and
languages. Private enthusiasts, researchers and academics will drop
TensorFlow. And don’t forget that industries rely on specialists who
learned in academia and come from research.
In a time of AI revolution, where the technology is more popular than
ever, and is being added to literally everything. In my opinion, TensorFlow
is neglecting everything outside Python-Linux, dropping an already lacking
support for interoperability, and not investing in accessibility.
I’m saying this as a researcher and professor working in a computer
science lab in an university. I write this just after investing almost 100
hours trying to simply build TensorFlow-cc to add some basic capabilities
to a research project for the European Union, I failed. Also the absolute
lack of recent information anywhere about TensorFlow-cc, and the responses
I have seen to old threads lets me know most people gave up the same way
I’m ready to tell my whole team to abandon TensorFlow and try other
solutions.
I have seen others in my lab commenting similar concerns and frustrations.
Many of our researchers are already moving away from TensorFlow and soon
the whole department will follow.
For context. The Computer Science department is the largest department in
my university, and serves the most important IT faculty in the Northwest of
Spain.
For us ML is thriving. In addition to the Bachelor degree in Computer
Science, where ML is more than present, being the most requested in the
entire university. We opened a new one in Data Science, and are opening a
new one in Artificial Intelligence. Next year we will be adding new classes
and teachers to be able to serve the increasing number of students in two
of the three Machine Learning subjects I teach… As far as I know, none of
them will learn TensorFlow, none of them are using it in their personal
projects and none of them will use it their degrees. They instead will be
using Julia, Matlab, OpenCv , PyTorch, Scikit-Learn and other solutions.
Which leaves me to the industry sector. I worked also as a researcher in a
public hospital in a project about diabetes, and in a private research
center dedicated to laser and manufacturing. They all used TensorFlow, the
same my laboratory did. I have been told they are all moving away from it,
currently opting for a Scikit-Learn+OpenCv and PyTorch based approach. The
reason is in one case the drop of GPU support for windows, and in the other
a perceived drop of support combined with lack of interoperability.
The thing is, nobody moves away from a technology they spent years using
and learning unless the technology fails them. And once you move away from
something because of a problem, if you find a solution somewhere else, you
will probably never return.
That is what is happening in TensorFlow. Google has intentionally dropped
the ball with support, documentation, accessibility, ease of use,
interoperability across languages and interoperability across platforms… so
others will raise to the occasion.
Simply put, TensorFlow is becoming the Bing search engine with regards to
AI.
Until today the officially documented TensorFlow standard installation
procedure for Linux users with GPUs does not include the additional
steps required to perform deep learning experiments with TensorFlow version
2.16.1 and utilize GPU locally. That's why I submitted the pull request
https://github.com/tensorflow/docs/pull/2299 in good faith and for the
shake of all users as TensorFlow is "An Open Source Machine Learning
Framework for Everyone".
Hope that the next patch version of TensorFlow will fix the bug as soon as
possible!
—
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/63362#issuecomment-2111604818,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAGK34IMRGFJIR6CXWASBNLZCLVZFAVCNFSM6AAAAABEOPWBC2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMJRGYYDIOBRHA
.
You are receiving this because you were mentioned.Message ID:
@.***>

@joaomh If you want to use 2.16.1 and you use conda then the easiest fix is with activated env:
mkdir -p ${CONDA_PREFIX}/etc/conda/activate.d
create file in ${CONDA_PREFIX}/etc/conda/activate.d/fix_gpu.sh with content below:
NVIDIA_DIR=$(dirname $(dirname $(python -c "import nvidia.cudnn;print(nvidia.cudnn.__file__)")))
for dir in $NVIDIA_DIR/*; do
    if [ -d "$dir/lib" ]; then
        export LD_LIBRARY_PATH="$dir/lib:$LD_LIBRARY_PATH"
    fi
done

tf2, shit!

pip install tensorflow[and-cuda]
tensorflow==2.16.1 still failed (Cannot dlopen some GPU libraries)

I have the same problem. 4090, it doesnt work with 2.16, but it works with 2.15. No idea why. I am also installing with Conda

I have the same problem. 4090, it doesnt work with 2.16, but it works with 2.15. No idea why. I am also installing with Conda

@thephet it is a path issue. You must manually configure the environment variables.
You can try the following:

Create a fresh conda virtual environment and activate it,
pip install --upgrade pip,
pip install tensorflow[and-cuda],
Set environment variables:

Locate the directory for the conda environment in your terminal window by running in the terminal:
echo $CONDA_PREFIX
Enter that directory and create these subdirectories and files:
cd $CONDA_PREFIX
mkdir -p ./etc/conda/activate.d
mkdir -p ./etc/conda/deactivate.d
touch ./etc/conda/activate.d/env_vars.sh
touch ./etc/conda/deactivate.d/env_vars.sh

Edit ./etc/conda/activate.d/env_vars.sh as follows:
#!/bin/sh

# Store original LD_LIBRARY_PATH
export ORIGINAL_LD_LIBRARY_PATH="${LD_LIBRARY_PATH}"

# Get the CUDNN directory
CUDNN_DIR=$(dirname $(dirname $(python -c "import nvidia.cudnn; print(nvidia.cudnn.__file__)")))

# Set LD_LIBRARY_PATH to include CUDNN directory
export LD_LIBRARY_PATH=$(find ${CUDNN_DIR}/*/lib/ -type d -printf "%p:")${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}

# Get the ptxas directory
PTXAS_DIR=$(dirname $(dirname $(python -c "import nvidia.cuda_nvcc; print(nvidia.cuda_nvcc.__file__)")))

# Set PATH to include the directory containing ptxas
export PATH=$(find ${PTXAS_DIR}/*/bin/ -type d -printf "%p:")${PATH:+:${PATH}}

Edit ./etc/conda/deactivate.d/env_vars.sh as follows:
#!/bin/sh

# Restore original LD_LIBRARY_PATH
export LD_LIBRARY_PATH="${ORIGINAL_LD_LIBRARY_PATH}"

# Unset environment variables
unset CUDNN_DIR
unset PTXAS_DIR

Verify the GPU setup:
python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"
I have submitted the pull request to update the official TensorFlow installation guide.
I hope it helps!

I'm generally confused about this setup with WSL 2. Where, what exactly needs to be installed.
When attempting to install using the command pip install tensorflow[and-cuda]
The following error is displayed:
ERROR: Could not find a version that satisfies the requirement nvidia-nccl-cu12==2.19.3; extra == "and-cuda" (from tensorflow[and-cuda]) (from versions: 0.0.1.dev5)
ERROR: No matching distribution found for nvidia-nccl-cu12==2.19.3; extra == "and-cuda"

tenserflow 2.16.1
To support it, you need this:

I'm running the command nvidia-smi and this is what it gives me:
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 555.85                 Driver Version: 555.85         CUDA Version: 12.5     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                  Driver-Model | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce GTX 1050 Ti   WDDM  |   00000000:01:00.0  On |                  N/A |
|  0%   50C    P8             N/A /   90W |     863MiB /   4096MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
+-----------------------------------------------------------------------------------------+

I realize I have the wrong version of Cuda. I tried to install cuda toolkit 12.3 in windows
It installed, but it still shows me Cuda 12.5 (which must have been installed with the drivers).

I read in the nvidia manual that you need to install the cuda toolkit in linux. Installed version 12.3 and still I can't install tenserflow.
Please explain it to me. What do i need to install where.
What do i need to install inside wsl 2 and what do i need to install inside windows.
I am confused with toolkit and cuDNN. What should be installed where

I'm generally confused about this setup with WSL 2. Where, what exactly needs to be installed.
When attempting to install using the command pip install tensorflow[and-cuda]
The following error is displayed:
ERROR: Could not find a version that satisfies the requirement nvidia-nccl-cu12==2.19.3; extra == "and-cuda" (from tensorflow[and-cuda]) (from versions: 0.0.1.dev5)
ERROR: No matching distribution found for nvidia-nccl-cu12==2.19.3; extra == "and-cuda"

tenserflow 2.16.1 To support it, you need this:
python version:
Python 3.12.0

I'm running the command nvidia-smi and this is what it gives me:
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 555.85                 Driver Version: 555.85         CUDA Version: 12.5     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                  Driver-Model | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce GTX 1050 Ti   WDDM  |   00000000:01:00.0  On |                  N/A |
|  0%   50C    P8             N/A /   90W |     863MiB /   4096MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

I installed cuda and cuDNN
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Wed_Nov_22_10:30:42_Pacific_Standard_Time_2023
Cuda compilation tools, release 12.3, V12.3.107
Build cuda_12.3.r12.3/compiler.33567101_0

wsl 2 (uname -m && cat /etc/*release):
x86_64
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.3 LTS"
PRETTY_NAME="Ubuntu 22.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.3 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

I read in the nvidia manual that you need to install the cuda toolkit in linux. Installed version 12.3 and still I can't install tenserflow.
Please explain it to me. What do i need to install where.
What do i need to install inside wsl 2 and what do i need to install inside windows.
I am confused with toolkit and cuDNN. What should be installed where

@MrOxMasTer the purpose of installing WSL 2 is to "install TensorFlow inside" and as a process it is irrelevant to installing the CUDA Toolkit and cuDNN on Windows. Generally, installing WSL 2 allows you to run a full Linux environment within Windows, making it easier to develop and run applications that rely on Linux-based tools and libraries as TensorFlow version 2.16.1. More specifically, in order to run the latest version of TensorFlow and utilize your GPU in WSL 2 according to TensorFlow official documentation you must proceed with the TensorFlow installation procedure on WSL 2. Hopefully that makes more sense.

@sgkouzias  Thank you. The solution from @niko247 worked and it is what I am using

@MrOxMasTer the purpose of installing WSL 2 is to "install TensorFlow inside" and as a process it is irrelevant to installing the CUDA Toolkit and cuDNN on Windows. Generally, installing WSL 2 allows you to run a full Linux environment within Windows, making it easier to develop and run applications that rely on Linux-based tools and libraries as TensorFlow version 2.16.1. More specifically, in order to run the latest version of TensorFlow and utilize your GPU in WSL 2 according to TensorFlow official documentation you must proceed with the TensorFlow installation procedure on WSL 2. Hopefully that makes more sense.

I also have to guess what the installation problem is:

The most terrible setup I've ever seen in my entire life. It is not clear where I need to enter these commands, and why, if all commands need to be entered in wsl2, then why do you have a command in your sentence that is entered only on the side where the graphics driver is installed, because clearly

After all, it is clearly written in white in nvidia that the graphics driver does not need to be installed in wsl, and for some reason you offer me to enter a command that works only on the windows side, but at the same time you say that the entire installation in wsl takes place inside.
How am I supposed to know what to install inside wsl, if it's just from the point of view of a person who uses such functionality for the first time, it sounds like nonsense, because I don't understand how to use it and naturally I will try to run something in windows, because I work in windows. Even if I install it, how would I know that for example in vs code there is an extension like “wsl” that allows you to connect to wsl?

@MrOxMasTer you can do every installation (mostly python packages) "pip install" or "sudo apt install" in WSL ubuntu like you are using a PC running ubuntu OS , except CUDA, CuDNN installation for Windows.

@MrOxMasTer since you only know how to use Windows you should initially follow the TensorFlow official documentation to install TensorFlow with pip for Windows WSL2 and follow the provided link for the official CUDA on WSL User Guide.

Also kindly note that the current issue opened "TF 2.16.1 Fails to work with GPUs" relates to Linux Operating Systems and possibly the additional steps to be specified in the official TensorFlow documentation in order to utilize GPUs locally.
Until today the officially documented TensorFlow standard installation procedure for Linux users with GPUs does not include the additional steps required to perform deep learning experiments with TensorFlow version 2.16.1 and utilize GPU locally. That's why I submitted a pull request in good faith and for the shake of all users as TensorFlow is "An Open Source Machine Learning Framework for Everyone".
Hope that the next patch version of TensorFlow will fix the bug as soon as possible!

"Once a Windows NVIDIA GPU driver is installed on the system, CUDA becomes available within WSL 2. The CUDA driver installed on Windows host will be stubbed inside the WSL 2 as libcuda.so, therefore users must not install any NVIDIA GPU Linux driver within WSL 2...."

Yes, I say for this that you do not need to install graphics drivers in wsl, but installing tensorflow involves not only installing graphics drivers, but also cuDNN and cudaToolkit (possibly TensorRT) and it was not clear to me specifically where it needed to be installed. Then I saw that I needed to install everything inside wsl except the graphics driver

Also kindly note that the current issue opened "TF 2.16.1 Fails to work with GPUs" involves Linux Operating Systems and potentially the additional steps to be specified in the official TensorFlow documentation in order to utilize GPUs locally.

I started a not very pleasant acquaintance with tensorflow with this version. As I understand it, the specific reason is 2.16.1 and it does not work in wsl. Because nothing worked for me. And the question is which version can be installed so that it works normally in wsl.
Also, for the future, I will say that installing conda does not help either. You can install a maximum of 2.10 version on it

Also kindly note that the current issue opened "TF 2.16.1 Fails to work with GPUs" involves Linux Operating Systems and potentially the additional steps to be specified in the official TensorFlow documentation in order to utilize GPUs locally.

I started a not very pleasant acquaintance with tensorflow with this version. As I understand it, the specific reason is 2.16.1 and it does not work in wsl. Because nothing worked for me. And the question is which version can be installed so that it works normally in wsl.
Also, for the future, I will say that installing anaconda does not help either. You can install a maximum of 2.10 version on it

@MrOxMasTer I totally understand your frustration and I can reassure you that TensorFlow version 2.16.1 can actually work with your cuda-enabled GPU.
You can try the following:

Create a fresh conda virtual environment in WSL and activate it, like this:

conda create --name tf python=3.11
conda activate tf

Within the fresh conda virtual environment tf created at the previous step run the following commands sequentially:

pip install --upgrade pip
pip install tensorflow[and-cuda]

Set environment variables:

Note: This step is required in order to utilize your GPU but not yet included in the official TensorFlow documentation. All NVIDIA libs are installed with TensorFlow due to the fact you ran the command pip install tensorflow[and-cuda] in the previous step!

Locate the directory for the conda environment in your terminal window by running in the terminal:
echo $CONDA_PREFIX
Enter that directory and create these subdirectories and files:
cd $CONDA_PREFIX
mkdir -p ./etc/conda/activate.d
mkdir -p ./etc/conda/deactivate.d
touch ./etc/conda/activate.d/env_vars.sh
touch ./etc/conda/deactivate.d/env_vars.sh

Edit ./etc/conda/activate.d/env_vars.sh as follows:
#!/bin/sh

# Store original LD_LIBRARY_PATH
export ORIGINAL_LD_LIBRARY_PATH="${LD_LIBRARY_PATH}"

# Get the CUDNN directory
CUDNN_DIR=$(dirname $(dirname $(python -c "import nvidia.cudnn; print(nvidia.cudnn.__file__)")))

# Set LD_LIBRARY_PATH to include CUDNN directory
export LD_LIBRARY_PATH=$(find ${CUDNN_DIR}/*/lib/ -type d -printf "%p:")${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}

# Get the ptxas directory
PTXAS_DIR=$(dirname $(dirname $(python -c "import nvidia.cuda_nvcc; print(nvidia.cuda_nvcc.__file__)")))

# Set PATH to include the directory containing ptxas
export PATH=$(find ${PTXAS_DIR}/*/bin/ -type d -printf "%p:")${PATH:+:${PATH}}

Edit ./etc/conda/deactivate.d/env_vars.sh as follows:
#!/bin/sh

# Restore original LD_LIBRARY_PATH
export LD_LIBRARY_PATH="${ORIGINAL_LD_LIBRARY_PATH}"

# Unset environment variables
unset CUDNN_DIR
unset PTXAS_DIR

Verify the GPU setup:
python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"
Additionally as I was informed that the next version of TensorFlow will hopefully arrive within the next days!
I hope it helps!

Thanks, but it doesn't work for me @sgkouzias. It does at least find some files. However, despite the verify gpu setup saying that 1 gpu is available, no gpu activity... all cpu

Thanks, but it doesn't work for me @sgkouzias. It does at least find some files. However, despite the verify gpu setup saying that 1 gpu is available, no gpu activity... all cpu

@GorillaDaddy well in order to "work" your setup should fulfill some certain technical requirements (please first check the official TensorFlow documentation). As I am not aware of your setup I could not possibly guess (if I could provide some useful assistance given the fact that I am not a professional developer just a TensorFlow user) why it seems that your GPU is not properly utilized. However, here are some hints that I hope will help you:
a) check your OS compatibility,
b) Also check whether your GPU is compatible, and of course
c) check the Python version compatibility with the desired TensorFlow version.

Also kindly note that the current issue opened "TF 2.16.1 Fails to work with GPUs" involves Linux Operating Systems and potentially the additional steps to be specified in the official TensorFlow documentation in order to utilize GPUs locally.

Hooray, it worked, thanks, but I have a mistake with this NUMA. Is this normal? Could it be because I did not install Cuda and Cdn on behalf of the administrator?

Also kindly note that the current issue opened "TF 2.16.1 Fails to work with GPUs" involves Linux Operating Systems and potentially the additional steps to be specified in the official TensorFlow documentation in order to utilize GPUs locally.

Hooray, it worked, thanks, but I have a mistake with this NUMA. Is this normal? Could it be because I did not install Cuda and Cdn on behalf of the administrator?

@MrOxMasTer congratulations and thanks for the feedback.
The error "Your kernel may have been built without NUMA support" refers to the lack of NUMA (Non-Uniform Memory Access) support in the kernel you are using. NUMA is a memory architecture used in multiprocessor systems where the memory access time depends on the memory location relative to the processor.
NUMA support is important for optimizing memory access on systems with multiple CPUs or GPUs. It allows the operating system to allocate memory and schedule processes in a way that reduces memory access latency.
On the other hand, WSL (Windows Subsystem for Linux) provides a Linux-compatible kernel interface developed by Microsoft and allows you to run Linux binaries on Windows. However, WSL's kernel might lack certain features present in a full-fledged Linux kernel, including NUMA support.
The lack of NUMA support might lead to suboptimal performance on systems with multiple processors or GPUs because the memory allocation might not be as efficient.
Consequently you can safely ignore the warning.

https://stackoverflow.com/a/78599028/13086128

I'm also facing this issue as the op since I've upgraded to 2.16.1. After the downgrade to 2.15.1 everything runs smooth.
TensorFlow version
TF 2.16.1
OS platform and distribution
Linux Ubuntu 22.04.4 LTS
Python version
3.10.12
CUDA/cuDNN version
12.4

@rednag as I understand it you have two available options:
1) Keep TensorFlow version 2.15 and reinstall Keras 3 afterwards
According to the official Keras documentation you can simply:
pip install --upgrade keras after installing tensorflow version 2.15
2) Upgrade Tensorflow to version 2.16.1
You can upgrade to TensorFlow version 2.16.1 and utilize your GPU locally (Keras 3.0 will be installed as well) through following the steps below:

Create a fresh conda virtual environment and activate it,
pip install --upgrade pip,
pip install tensorflow[and-cuda],
Set environment variables:

Locate the directory for the conda environment in your terminal window by running in the terminal:
echo $CONDA_PREFIX
Enter that directory and create these subdirectories and files:
cd $CONDA_PREFIX
mkdir -p ./etc/conda/activate.d
mkdir -p ./etc/conda/deactivate.d
touch ./etc/conda/activate.d/env_vars.sh
touch ./etc/conda/deactivate.d/env_vars.sh

Edit ./etc/conda/activate.d/env_vars.sh as follows:
#!/bin/sh

# Store original LD_LIBRARY_PATH
export ORIGINAL_LD_LIBRARY_PATH="${LD_LIBRARY_PATH}"

# Get the CUDNN directory
CUDNN_DIR=$(dirname $(dirname $(python -c "import nvidia.cudnn; print(nvidia.cudnn.__file__)")))

# Set LD_LIBRARY_PATH to include CUDNN directory
export LD_LIBRARY_PATH=$(find ${CUDNN_DIR}/*/lib/ -type d -printf "%p:")${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}

# Get the ptxas directory
PTXAS_DIR=$(dirname $(dirname $(python -c "import nvidia.cuda_nvcc; print(nvidia.cuda_nvcc.__file__)")))

# Set PATH to include the directory containing ptxas
export PATH=$(find ${PTXAS_DIR}/*/bin/ -type d -printf "%p:")${PATH:+:${PATH}}

Edit ./etc/conda/deactivate.d/env_vars.sh as follows:
#!/bin/sh

# Restore original LD_LIBRARY_PATH
export LD_LIBRARY_PATH="${ORIGINAL_LD_LIBRARY_PATH}"

# Unset environment variables
unset CUDNN_DIR
unset PTXAS_DIR

Verify the GPU setup:
python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"
I have submitted the respective pull request to update the official TensorFlow installation guide a month ago.
Additionally, as I was informed the next version of TensorFlow will hopefully arrive within the next days!
I hope it helps!

Thank you for the fast reply. At the moment I'm using the old functions from the keras.src.utils and tf packages, but I'm looking forward to the new release.

Thank you for the fast reply. At the moment I'm using the old functions from the keras.src.utils and tf packages, but I'm looking forward to the new release.

@rednag great. Another option to consider for fast model training if you ultimately install Keras 3 (by one way or another) is to use JAX as keras backend.

Can someone care to explain why TF >2.10 cannot be run with GPU in native windows?
This totally makes no sense whatsoever, as all other HW, WSL, and Conda works with GPU. Including other python packages, such as Torch. So what is going on?
I.e. What is the problem and why is it not being addressed by the community?

Can someone care to explain why TF >2.10 cannot be run with GPU in native windows? This totally makes no sense whatsoever, as all other HW, WSL, and Conda works with GPU. Including other python packages, such as Torch. So what is going on?
I.e. What is the problem and why is it not being addressed by the community?

Google removed the native windows cuda build starting TF 2.11
There is nothing you can do about it, building from source with cuda will also fail.

Everyone that cared about full support of TF is no longer in the team. See above comments for more details and differences

@sh-shahrokhi

Google removed the native windows cuda build starting TF 2.11

Unfortunately that doesn't say anything. I don't see how you can "remove" any of that, apart from breaking the build scripts. Whatever you "remove" must still be present for all other nix builds. WSL is not that different from MSYS, MinGW, which (no longer) is too far from VS C/C++ builds.

@sh-shahrokhi

Google removed the native windows cuda build starting TF 2.11

Unfortunately that doesn't say anything. I don't see how you can "remove" any of that, apart from breaking the build scripts. Whatever you "remove" must still be present for all other nix builds. WSL is not that different from MSYS, MinGW, which (no longer) is too far from VS C/C++ builds.

https://github.com/tensorflow/tensorflow/issues/58629

Also kindly note that the current issue opened "TF 2.16.1 Fails to work with GPUs" involves Linux Operating Systems and potentially the additional steps to be specified in the official TensorFlow documentation in order to utilize GPUs locally.

I started a not very pleasant acquaintance with tensorflow with this version. As I understand it, the specific reason is 2.16.1 and it does not work in wsl. Because nothing worked for me. And the question is which version can be installed so that it works normally in wsl.
Also, for the future, I will say that installing anaconda does not help either. You can install a maximum of 2.10 version on it

@MrOxMasTer I totally understand your frustration but I reassure you that TensorFlow version 2.16.1 can actually work with your cuda-enabled GPU.
You can try the following:

Create a fresh conda virtual environment in WSL and activate it, like this:

conda create --name tf python=3.11
conda activate tf

Within the fresh conda virtual environment tf created in the previous step run the following commands sequentially:

pip install --upgrade pip
pip install tensorflow[and-cuda]

Set environment variables:

Note: This step is required in order to utilize your GPU but not yet included in the official TensorFlow documentation. All NVIDIA libs are installed with TensorFlow due to the fact you ran the command pip install tensorflow[and-cuda] in the previous step!

Locate the directory for the conda environment in your terminal window by running in the terminal:
echo $CONDA_PREFIX
Enter that directory and create these subdirectories and files:
cd $CONDA_PREFIX
mkdir -p ./etc/conda/activate.d
mkdir -p ./etc/conda/deactivate.d
touch ./etc/conda/activate.d/env_vars.sh
touch ./etc/conda/deactivate.d/env_vars.sh

Edit ./etc/conda/activate.d/env_vars.sh as follows:
#!/bin/sh

# Store original LD_LIBRARY_PATH
export ORIGINAL_LD_LIBRARY_PATH="${LD_LIBRARY_PATH}"

# Get the CUDNN directory
CUDNN_DIR=$(dirname $(dirname $(python -c "import nvidia.cudnn; print(nvidia.cudnn.__file__)")))

# Set LD_LIBRARY_PATH to include CUDNN directory
export LD_LIBRARY_PATH=$(find ${CUDNN_DIR}/*/lib/ -type d -printf "%p:")${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}

# Get the ptxas directory
PTXAS_DIR=$(dirname $(dirname $(python -c "import nvidia.cuda_nvcc; print(nvidia.cuda_nvcc.__file__)")))

# Set PATH to include the directory containing ptxas
export PATH=$(find ${PTXAS_DIR}/*/bin/ -type d -printf "%p:")${PATH:+:${PATH}}

Edit ./etc/conda/deactivate.d/env_vars.sh as follows:
#!/bin/sh

# Restore original LD_LIBRARY_PATH
export LD_LIBRARY_PATH="${ORIGINAL_LD_LIBRARY_PATH}"

# Unset environment variables
unset CUDNN_DIR
unset PTXAS_DIR

Verify the GPU setup: python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"
Additionally, as I was informed the next version of TensorFlow will hopefully arrive within the next days!
I hope it helps!

Doesn't work for me :/ I even reinstalled completely WSL, but I still get an empty list when showing the available devices... Should CUDA be unistalled on Windows side ? When I use "nvidia-smi", it is written that I have the 12.5 Cuda Version, even if I didn't install anything on WSL... Is that normal ?
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 555.52.01              Driver Version: 555.99         CUDA Version: 12.5     |
|-----------------------------------------+------------------------+----------------------+

Also kindly note that the current issue opened "TF 2.16.1 Fails to work with GPUs" involves Linux Operating Systems and potentially the additional steps to be specified in the official TensorFlow documentation in order to utilize GPUs locally.

I started a not very pleasant acquaintance with tensorflow with this version. As I understand it, the specific reason is 2.16.1 and it does not work in wsl. Because nothing worked for me. And the question is which version can be installed so that it works normally in wsl.
Also, for the future, I will say that installing anaconda does not help either. You can install a maximum of 2.10 version on it

@MrOxMasTer I totally understand your frustration but I reassure you that TensorFlow version 2.16.1 can actually work with your cuda-enabled GPU.
You can try the following:

Create a fresh conda virtual environment in WSL and activate it, like this:

conda create --name tf python=3.11
conda activate tf

Within the fresh conda virtual environment tf created in the previous step run the following commands sequentially:

pip install --upgrade pip
pip install tensorflow[and-cuda]

Set environment variables:

Note: This step is required in order to utilize your GPU but not yet included in the official TensorFlow documentation. All NVIDIA libs are installed with TensorFlow due to the fact you ran the command pip install tensorflow[and-cuda] in the previous step!

Locate the directory for the conda environment in your terminal window by running in the terminal:
echo $CONDA_PREFIX
Enter that directory and create these subdirectories and files:
cd $CONDA_PREFIX
mkdir -p ./etc/conda/activate.d
mkdir -p ./etc/conda/deactivate.d
touch ./etc/conda/activate.d/env_vars.sh
touch ./etc/conda/deactivate.d/env_vars.sh

Edit ./etc/conda/activate.d/env_vars.sh as follows:
#!/bin/sh

# Store original LD_LIBRARY_PATH
export ORIGINAL_LD_LIBRARY_PATH="${LD_LIBRARY_PATH}"

# Get the CUDNN directory
CUDNN_DIR=$(dirname $(dirname $(python -c "import nvidia.cudnn; print(nvidia.cudnn.__file__)")))

# Set LD_LIBRARY_PATH to include CUDNN directory
export LD_LIBRARY_PATH=$(find ${CUDNN_DIR}/*/lib/ -type d -printf "%p:")${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}

# Get the ptxas directory
PTXAS_DIR=$(dirname $(dirname $(python -c "import nvidia.cuda_nvcc; print(nvidia.cuda_nvcc.__file__)")))

# Set PATH to include the directory containing ptxas
export PATH=$(find ${PTXAS_DIR}/*/bin/ -type d -printf "%p:")${PATH:+:${PATH}}

Edit ./etc/conda/deactivate.d/env_vars.sh as follows:
#!/bin/sh

# Restore original LD_LIBRARY_PATH
export LD_LIBRARY_PATH="${ORIGINAL_LD_LIBRARY_PATH}"

# Unset environment variables
unset CUDNN_DIR
unset PTXAS_DIR

Verify the GPU setup: python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"
Additionally, as I was informed the next version of TensorFlow will hopefully arrive within the next days!
I hope it helps!

Doesn't work for me :/ I even reinstalled completely WSL, but I still get an empty list when showing the available devices... Should CUDA be unistalled on Windows side ? When I use "nvidia-smi", it is written that I have the 12.5 Cuda Version, even if I didn't install anything on WSL... Is that normal ?
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 555.52.01              Driver Version: 555.99         CUDA Version: 12.5     |
|-----------------------------------------+------------------------+----------------------+

@ben-jy frankly I have no clue.  Did you check the official documentation? Your setup meets the technical requirements? What's the Python version in WSL2? Is it compatible with TensorFlow 2.16.1? What's the name of your NVIDIA GPU? The output of the command nvidia-smi in WSL2 seems normal since your GPU driver is installed in Windows. However you could try reinstalling everything (compatible GPU driver,  afterwards WSL2 and then TensorFlow)...

Official documentation has not been updated fot TF 2.16 and still refers to cuda 11.8 not 12.

From: Sotiris Gkouzias @.>
Sent: Monday, June 17, 2024 9:54 AM
To: tensorflow/tensorflow @.>
Cc: Shayan Shahrokhi @.>; Mention @.>
Subject: Re: [tensorflow/tensorflow] TF 2.16.1 Fails to work with GPUs (Issue #63362)
Also kindly note that the current issue opened "TF 2.16.1 Fails to work with GPUs" involves Linux Operating Systems and potentially the additional steps to be specified in the official TensorFlow documentation in order to utilize GPUs locally.
I started a not very pleasant acquaintance with tensorflow with this version. As I understand it, the specific reason is 2.16.1 and it does not work in wsl. Because nothing worked for me. And the question is which version can be installed so that it works normally in wsl.
Also, for the future, I will say that installing anaconda does not help either. You can install a maximum of 2.10 version on it
@MrOxMasTerhttps://github.com/MrOxMasTer I totally understand your frustration but I reassure you that TensorFlow version 2.16.1 can actually work with your cuda-enabled GPU.
You can try the following:

Create a fresh conda virtual environment in WSL and activate it, like this:

conda create --name tf python=3.11
conda activate tf

Within the fresh conda virtual environment tf created in the previous step run the following commands sequentially:

pip install --upgrade pip
pip install tensorflow[and-cuda]

Set environment variables:

Note: This step is required in order to utilize your GPU but not yet included in the official TensorFlow documentation. All NVIDIA libs are installed with TensorFlow due to the fact you ran the command pip install tensorflow[and-cuda] in the previous step!
Locate the directory for the conda environment in your terminal window by running in the terminal:
echo $CONDA_PREFIX
Enter that directory and create these subdirectories and files:
cd $CONDA_PREFIX
mkdir -p ./etc/conda/activate.d
mkdir -p ./etc/conda/deactivate.d
touch ./etc/conda/activate.d/env_vars.sh
touch ./etc/conda/deactivate.d/env_vars.sh
Edit ./etc/conda/activate.d/env_vars.sh as follows:
#!/bin/sh
Store original LD_LIBRARY_PATH
export ORIGINAL_LD_LIBRARY_PATH="${LD_LIBRARY_PATH}"
Get the CUDNN directory
CUDNN_DIR=$(dirname $(dirname $(python -c "import nvidia.cudnn; print(nvidia.cudnn.file)")))
Set LD_LIBRARY_PATH to include CUDNN directory
export LD_LIBRARY_PATH=$(find ${CUDNN_DIR}/*/lib/ -type d -printf "%p:")${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
Get the ptxas directory
PTXAS_DIR=$(dirname $(dirname $(python -c "import nvidia.cuda_nvcc; print(nvidia.cuda_nvcc.file)")))
Set PATH to include the directory containing ptxas
export PATH=$(find ${PTXAS_DIR}/*/bin/ -type d -printf "%p:")${PATH:+:${PATH}}
Edit ./etc/conda/deactivate.d/env_vars.sh as follows:
#!/bin/sh
Restore original LD_LIBRARY_PATH
export LD_LIBRARY_PATH="${ORIGINAL_LD_LIBRARY_PATH}"
Unset environment variables
unset CUDNN_DIR
unset PTXAS_DIR
Verify the GPU setup: python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"
Additionally, as I was informed the next version of TensorFlow will hopefully arrive within the next days!
I hope it helps!
Doesn't work for me :/ I even reinstalled completely WSL, but I still get an empty list when showing the available devices... Should CUDA be unistalled on Windows side ? When I use "nvidia-smi", it is written that I have the 12.5 Cuda Version, even if I didn't install anything on WSL... Is that normal ?
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 555.52.01 Driver Version: 555.99 CUDA Version: 12.5 |
|-----------------------------------------+------------------------+----------------------+
@ben-jyhttps://github.com/ben-jy frankly I have no clue. Did you check the official documentationhttps://www.tensorflow.org/install/pip#windows-wsl2? Your setup meets the technical requirements? What's the Python version in WSL2? Is it compatible with TensorFlow 2.16.1? What's the name of your NVIDIA GPU? The output of the command nvidia-smi in WSL2 seems normal since your GPU driver is installed in Windows. However you could try reinstalling everything (compatible GPU driver, afterwards WSL2 and then TensorFlow)...
—
Reply to this email directly, view it on GitHubhttps://github.com/tensorflow/tensorflow/issues/63362#issuecomment-2173775825, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AZRPJAD4P5CMSQLZQOQO5CTZH4BEJAVCNFSM6AAAAABEOPWBC2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNZTG43TKOBSGU.
You are receiving this because you were mentioned.Message ID: @.***>

Official documentation has not been updated fot TF 2.16 and still refers to cuda 11.8 not 12.

From: Sotiris Gkouzias @.>
Sent: Monday, June 17, 2024 9:54 AM
To: tensorflow/tensorflow @.>
Cc: Shayan Shahrokhi @.>; Mention @.>
Subject: Re: [tensorflow/tensorflow] TF 2.16.1 Fails to work with GPUs (Issue #63362)
Also kindly note that the current issue opened "TF 2.16.1 Fails to work with GPUs" involves Linux Operating Systems and potentially the additional steps to be specified in the official TensorFlow documentation in order to utilize GPUs locally.
I started a not very pleasant acquaintance with tensorflow with this version. As I understand it, the specific reason is 2.16.1 and it does not work in wsl. Because nothing worked for me. And the question is which version can be installed so that it works normally in wsl.
Also, for the future, I will say that installing anaconda does not help either. You can install a maximum of 2.10 version on it
@MrOxMasTerhttps://github.com/MrOxMasTer I totally understand your frustration but I reassure you that TensorFlow version 2.16.1 can actually work with your cuda-enabled GPU.
You can try the following:

Create a fresh conda virtual environment in WSL and activate it, like this:

conda create --name tf python=3.11
conda activate tf

Within the fresh conda virtual environment tf created in the previous step run the following commands sequentially:

pip install --upgrade pip
pip install tensorflow[and-cuda]

Set environment variables:

Note: This step is required in order to utilize your GPU but not yet included in the official TensorFlow documentation. All NVIDIA libs are installed with TensorFlow due to the fact you ran the command pip install tensorflow[and-cuda] in the previous step!
Locate the directory for the conda environment in your terminal window by running in the terminal:
echo $CONDA_PREFIX
Enter that directory and create these subdirectories and files:
cd $CONDA_PREFIX
mkdir -p ./etc/conda/activate.d
mkdir -p ./etc/conda/deactivate.d
touch ./etc/conda/activate.d/env_vars.sh
touch ./etc/conda/deactivate.d/env_vars.sh
Edit ./etc/conda/activate.d/env_vars.sh as follows:
#!/bin/sh
Store original LD_LIBRARY_PATH
export ORIGINAL_LD_LIBRARY_PATH="${LD_LIBRARY_PATH}"
Get the CUDNN directory
CUDNN_DIR=$(dirname $(dirname $(python -c "import nvidia.cudnn; print(nvidia.cudnn.file)")))
Set LD_LIBRARY_PATH to include CUDNN directory
export LD_LIBRARY_PATH=$(find ${CUDNN_DIR}/*/lib/ -type d -printf "%p:")${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
Get the ptxas directory
PTXAS_DIR=$(dirname $(dirname $(python -c "import nvidia.cuda_nvcc; print(nvidia.cuda_nvcc.file)")))
Set PATH to include the directory containing ptxas
export PATH=$(find ${PTXAS_DIR}/*/bin/ -type d -printf "%p:")${PATH:+:${PATH}}
Edit ./etc/conda/deactivate.d/env_vars.sh as follows:
#!/bin/sh
Restore original LD_LIBRARY_PATH
export LD_LIBRARY_PATH="${ORIGINAL_LD_LIBRARY_PATH}"
Unset environment variables
unset CUDNN_DIR
unset PTXAS_DIR
Verify the GPU setup: python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"
Additionally, as I was informed the next version of TensorFlow will hopefully arrive within the next days!
I hope it helps!
Doesn't work for me :/ I even reinstalled completely WSL, but I still get an empty list when showing the available devices... Should CUDA be unistalled on Windows side ? When I use "nvidia-smi", it is written that I have the 12.5 Cuda Version, even if I didn't install anything on WSL... Is that normal ?
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 555.52.01 Driver Version: 555.99 CUDA Version: 12.5 |
|-----------------------------------------+------------------------+----------------------+
@ben-jyhttps://github.com/ben-jy frankly I have no clue. Did you check the official documentationhttps://www.tensorflow.org/install/pip#windows-wsl2? Your setup meets the technical requirements? What's the Python version in WSL2? Is it compatible with TensorFlow 2.16.1? What's the name of your NVIDIA GPU? The output of the command nvidia-smi in WSL2 seems normal since your GPU driver is installed in Windows. However you could try reinstalling everything (compatible GPU driver, afterwards WSL2 and then TensorFlow)...
—
Reply to this email directly, view it on GitHubhttps://github.com/tensorflow/tensorflow/issues/63362#issuecomment-2173775825, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AZRPJAD4P5CMSQLZQOQO5CTZH4BEJAVCNFSM6AAAAABEOPWBC2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNZTG43TKOBSGU.
You are receiving this because you were mentioned.Message ID: @.***>

@sh-shahrokhi correct. And I would add that this is the whole issue here. The official TensorFlow documentation should be updated (it is hilariously outdated). However in order to figure out why @ben-jy can't successfully utilize his GPU his setup should meet certain requirements. For example that his GPU is actually in this list.

@sh-shahrokhi true. And I would like to add that this is the whole issue here. The official TensorFlow documentation should be updated (it is hilariously outdated). However in order to figure out why @ben-jy can't successfully utilize his GPU his setup should meet certain requirements. For example that his GPU is actually in this list.

@sh-shahrokhi I checked the official documentation, but I find it not very clear, and seems a bit contradictory: the software requirements state that CUDA and cuDNN should be installed on the machine, but the pip package should install them automatically with Tensorflow right ? Besides, this medium tutorial explain that CUDA should not be installed on Windows side, neither on WSL side, and be installed using the pip package. Maybe I should try to uninstall all CUDA-related on Windows...
Concerning your other questions:

I have an RTX 3070 Ti, which is in the list of CUDA-enabled product.
I use conda and I tried the install with Python 3.10 and 3.11, which are in the software requirements of the officiel documentation. Those versions are said compatible with TensorFlow 2.16.1, according to the PyPi package tags

I will try to make a clean reinstall of my GPU driver, as well as unistalling CUDA on Windows side. If it doesn't work, I think it is better to install CUDA and cuDNN manually, along with an oldest TensorFlow version. It is still a shame that the official documentation of such a large and important library is so unclear.

@learning-to-play

Can you test the 2.17.0 RC0, please? It is too late to update 2.16, but if 2.17 RC0 doesn't work, maybe there will be a chance to fix by RC1/final

@learning-to-play maybe this can get fixed before final release? TF does not work with GPUs, started failing since TF 2.16 release.

@SeeForTwo @poulsbo Could you please take a look? If there is a fix that needs to be cherry picked to 2.16.2 or 2.17.0, please follow these steps:

Submit a fix to TensorFlow HEAD
Ensure nightly builds are green.
Create a cherry pick PR to the corresponding release branches r2.16 and r2.17 and assign to @rtg0795

@mihaimaruseac, @learning-to-play Yesterday TensorFlow version 2.16.2 was released. Tested it today on my PC (with an NVIDIA RTX3060 GPU and Ubuntu 20.04 as my OS). Executed a deep learning model training script but got the following message while training was interrupted:

ptxas returned an error during compilation of ptx to sass: 'INTERNAL: ptxas 12.3.103 has a bug that we think can affect XLA. Please use a different version.'  If the error message indicates that a file could not be written, please verify that sufficient filesystem space is provided.
Aborted (core dumped)

Thus, as I understand it, TF 2.16.2 Fails to work with GPUs as well....

In my environment, tensorflow v2.16.2 is able to use the GPU.
At least with the following script, I could confirm that the training works with GPUs.
https://gist.github.com/StephenSzwiec/d4f6f587441f54b9b6c5fed5efd8002d
The ptxas is included in cuda-nvcc-cu12 and tensorflow 2.16.2 requires cuda-nvcc-cu12==12.3.107.
@sgkouzias
Perhaps tensorflow is incorrectly reading ptxas 12.3.103, which is installed somewhere in your environment (ex. /usr/local/cuda).

In my environment, tensorflow v2.16.2 is able to use the GPU.
At least with the following script, I could confirm that the training works with GPUs.
https://gist.github.com/StephenSzwiec/d4f6f587441f54b9b6c5fed5efd8002d
The ptxas is included in cuda-nvcc-cu12 and tensorflow 2.16.2 requires cuda-nvcc-cu12==12.3.107(not 12.3.103).
@sgkouzias
Perhaps tensorflow is incorrectly reading ptxas 12.3.103, which is installed somewhere in your environment (ex. /usr/local/cuda/bin/ptxas).

@Di-Is well shouldn't TensorFlow read ptxas from the correct path from the beginning? After all, everything required for GPU support is installed with TensorFlow in a dedicated virtual environment. I should not have to change any file of my OS... Perhaps by coincidence you have installed the compatible version of ptxas in your local files but not all users may have the same luck...

In my environment, tensorflow v2.16.2 is able to use the GPU.
At least with the following script, I could confirm that the training works with GPUs.
https://gist.github.com/StephenSzwiec/d4f6f587441f54b9b6c5fed5efd8002d
The ptxas is included in cuda-nvcc-cu12 and tensorflow 2.16.2 requires cuda-nvcc-cu12==12.3.107(not 12.3.103).
@sgkouzias
Perhaps tensorflow is incorrectly reading ptxas 12.3.103, which is installed somewhere in your environment (ex. /usr/local/cuda/bin/ptxas).

@Di-Is well shouldn't TensorFlow read ptxas from the correct path at the first place? Afterall, everything required for GPU support is installed with TensorFlow in the dedicated virtual environment by running the command pip install tensorflow[and-cuda]. I should not have to change any file of my OS... Perhaps by coincidence you have installed the compatible version of ptxas in your local files but not all users will have the same luck...

Can you check if the issue with ptxas is still resolved in 2.17.0rc1?

In my environment, tensorflow v2.16.2 is able to use the GPU.
At least with the following script, I could confirm that the training works with GPUs.
https://gist.github.com/StephenSzwiec/d4f6f587441f54b9b6c5fed5efd8002d
The ptxas is included in cuda-nvcc-cu12 and tensorflow 2.16.2 requires cuda-nvcc-cu12==12.3.107(not 12.3.103).
@sgkouzias
Perhaps tensorflow is incorrectly reading ptxas 12.3.103, which is installed somewhere in your environment (ex. /usr/local/cuda/bin/ptxas).

@Di-Is well shouldn't TensorFlow read ptxas from the correct path at the first place? Afterall, everything required for GPU support is installed with TensorFlow in the dedicated virtual environment by running the command pip install tensorflow[and-cuda]. I should not have to change any file of my OS... Perhaps by coincidence you have installed the compatible version of ptxas in your local files but not all users will have the same luck...

Can you check if the issue with ptxas is still resolved in 2.17.0rc1?

@belitskiy I confirm that the issue with ptxas is resolved with TensorFlow 2.17.0.rc1 indeed. My test script was executed without errors and smoothly.

@ALL,
Tensorflow 2.17.0-rc1 has been released. Could everyone please try to test with the 2.17.0-rc1 and let us know if it is working as expected.
https://github.com/tensorflow/tensorflow/releases
Thank you!

@tilakrayal , @all tags additional people. Simply replying here is enough to notify everyone that subscribed, this is not Slack

I try to use the docker to validate 2.17.0-rc1:
Dockerfile:
FROM python:3.11
RUN pip install tensorflow[and-cuda]==2.17.0rc1

Run:
docker build . -t test-tf217-cuda
docker run --gpus all test-tf217-cuda python -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"

Results:
2024-07-04 21:50:15.832656: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-07-04 21:50:15.844634: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-07-04 21:50:15.848366: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-07-04 21:50:15.857164: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-07-04 21:50:16.661013: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1720129817.109888       1 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
I0000 00:00:1720129817.110190       1 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
I0000 00:00:1720129817.141962       1 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
I0000 00:00:1720129817.142148       1 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
I0000 00:00:1720129817.142350       1 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
I0000 00:00:1720129817.142768       1 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU'), PhysicalDevice(name='/physical_device:GPU:1', device_type='GPU')]

I think this issue has been resolved.

@njzjz
If you do "pip install tensorflow[and-cuda]==2.17.0rc1" which is pretty much what the official install instruction is telling us, you shouldn't have any problems with cudnn, cufft, cublas, tensorRT etc.
So this issue is very far from being resolved!
The funny thing is that I get the same problem with fails to register the cudnn, cufft, cublas from the docker images provided by Tensorflow so these were released without even a single test run :1st_place_medal:
The only version I have successfully got running locally with GPU at all is 2.17.0rc0 using @sgkouzias instructions above...
I do the same with 2.17.0rc1 and it fails to register the cudnn, cufft, cublas...
It is completely unacceptable that release after release are made without being tested from a clean install.

If RC1 fails to work, even though RC0 did, then this is a regression and final release should not proceed before the fix. CC @rtg0795

I've been stuck with similar problems the last couple of days. The main problem for me was that TF was able to see the GPU, but I couldn't get it to actually utilize it. I was getting some ptxas warnings etc. and when I ran the training, it ran on the CPU. I was trying all sorts of tricks that I found on the internet, but nothing solved my problem. Today, I finally tried just running the training in the same container, but for some other (simple) model and it works! So it seems that the only problem was that I had some unsupported operation in my code and I didn't understand it from the warnings/errors I was receiving. Hope this saves some headaches if anyone gets into a similar situation as I did and ends up here. 😁

@njzjz If you do "pip install tensorflow[and-cuda]==2.17.0rc1" which is pretty much what the official install instruction is telling us, you shouldn't have any problems with cudnn, cufft, cublas, tensorRT etc. So this issue is very far from being resolved! The funny thing is that I get the same problem with fails to register the cudnn, cufft, cublas from the docker images provided by Tensorflow so these were released without even a single test run 🥇 The only version I have successfully got running locally with GPU at all is 2.17.0rc0 using @sgkouzias instructions above... I do the same with 2.17.0rc1 and it fails to register the cudnn, cufft, cublas... It is completely unacceptable that release after release are made without being tested from a clean install.

@siamect Does 2.17.0rc1 fail to find GPUs, or utilize them for you, e.g.
import tensorflow as tf
print(tf.config.list_physical_devices('GPU')

Just tried it, I can confirm that 2.17.0rc1 is able to find and use the GPU.
There are new non-fatal errors logged, not sure if this is intentional:
2024-07-09 09:32:23.358062: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-07-09 09:32:23.370417: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-07-09 09:32:23.374234: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered

Also, it'd be great if everyone in the community wasn't subjected to whatever pettiness this is:
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1720531947.534219    3257 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
I0000 00:00:1720531947.565743    3257 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
I0000 00:00:1720531947.567091    3257 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355

So, there's no regression and TF 2.17 will have support for GPU.

The double registration of those CUDA plugins is brought up in https://github.com/tensorflow/tensorflow/issues/62075, the trial fix for which is what ended up causing this issue, with CUDA/plugins not being detected at all.
The immediate fix was to revert the double-registration fix change, as the lesser of two evils

Thanks @belitskiy for the context. Yes, it looks like this will get resolved for TF 2.17 final release now.

@njzjz If you do "pip install tensorflow[and-cuda]==2.17.0rc1" which is pretty much what the official install instruction is telling us, you shouldn't have any problems with cudnn, cufft, cublas, tensorRT etc. So this issue is very far from being resolved! The funny thing is that I get the same problem with fails to register the cudnn, cufft, cublas from the docker images provided by Tensorflow so these were released without even a single test run 🥇 The only version I have successfully got running locally with GPU at all is 2.17.0rc0 using @sgkouzias instructions above... I do the same with 2.17.0rc1 and it fails to register the cudnn, cufft, cublas... It is completely unacceptable that release after release are made without being tested from a clean install.

@siamect Does 2.17.0rc1 fail to find GPUs, or utilize them for you, e.g.
import tensorflow as tf
print(tf.config.list_physical_devices('GPU'))

As I said  with 2.17.0rc1 and it fails to register the cudnn, cufft, cublas...  2.17.0rc0 does not. Both have the NUMA node problem.
Now there are people who think this is acceptable because it is not a fatal error. Same with the NUMA node problem. But if those are considered acceptable for whatever reason then just mute them or at least write something in the install instructions like "any error of this nature should be ignored" so we don't have to spend a week for our first installation tensorflow.
I have been using and contributing too opensource/free software for well over 25 years and tensorflow with GPU is by far the most difficult software to install that I have experienced the last 15 years...
Following the official install instruction is just a trap and you get stuck. The only way to get a working install is obviously to go through forums like this one.

TensorFlow 2.17.0 is released ! Tested it right away on the Cats Vs Dogs dataset and YES, it works with GPUs. So, that's the good news. It turns out that it displays the aforementioned NUMA warning indeed and, in addition, an extra warning: [gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced (the nature of which I am completely unaware). However, noticed that in the certain notebook (used to test the latest version) accuracy was reduced compared to the previous session with TensorFlow version 2.16.2 (as the last warning warned perhaps).

If TF 2.17 works then I think this issue can be closed, right?
Thank you for checking

But you still have the same ridiculous error messages so this will still cause endless headaches for those who doesn't magically know what error messages are ok or not. It seems like no one is doing even the slightest attempt to make this work according to the install instructions!
The 2.17.0rc0 at least got rid of some of the messages but they are back again in 2.17.0rc1.
Version 2.17.0 should never have been released!
Successfully installed tensorflow-2.17.0
martin@martin-GF75-Thin-9SC:~$ python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"
2024-07-13 08:34:20.430969: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-07-13 08:34:20.444256: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-07-13 08:34:20.448273: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-07-13 08:34:20.458075: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1720834462.966799    5737 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
I0000 00:00:1720834463.081761    5737 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
I0000 00:00:1720834463.085250    5737 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]

If TF 2.17 works then I think this issue can be closed, right?
Thank you for checking

@mihaimaruseac yes IMHO the issue should be closed.
@siamect I strongy believe that documentation and framework performance are very significant issues indeed and should be opened for the shake of all users, researhers and the TensorFlow Community and resolved as well!

TensorFlow 2.17.0 is released ! Tested it right away on the Cats Vs Dogs dataset and YES, it works with GPUs. So, that's the good news. It turns out that it displays the aforementioned NUMA warning indeed and, in addition, an extra warning: [gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced (the nature of which I am completely unaware). However, noticed that in the certain notebook (used to test the latest version) accuracy was reduced compared to the previous session with TensorFlow version 2.16.2 (as the last warning warned perhaps).

I did some tests and the model's accuracy really dropped, but I think it's related to keras:

2.15.1: no extra messages and expected results;
2.16.2: PTX messages and worse results;
2.16.2 with TF_USE_LEGACY_KERAS: PTX messages and expected results;
2.17.0: numa and gpu_timer messages and worse results.

TensorFlow 2.17.0 is released ! Tested it right away on the Cats Vs Dogs dataset and YES, it works with GPUs. So, that's the good news. It turns out that it displays the aforementioned NUMA warning indeed and, in addition, an extra warning: [gpu_timer.cc:114] Skipping the delay kernel, measurement accuracy will be reduced (the nature of which I am completely unaware). However, noticed that in the certain notebook (used to test the latest version) accuracy was reduced compared to the previous session with TensorFlow version 2.16.2 (as the last warning warned perhaps).

I did some tests and the model's accuracy really dropped, but I think it's related to keras:

2.15.1: no extra messages and expected results;
2.16.2: PTX messages and worse results;
2.16.2 with TF_USE_LEGACY_KERAS: PTX messages and expected results;
2.17.0: numa and gpu_timer messages and worse results.

Really interesting findings indeed! Thank you @arthurflor23

This problem is still not fixed right? Since tensorflow probability also requires now >2.16 I'm really looking forward to a new version which work with GPU w/o any workarounds.

This problem is still not fixed right? Since tensorflow probability also requires now >2.16 I'm really looking forward to a new version which work with GPU w/o any workarounds.

@rednag the latest version of TensorFlow namely 2.17.0 works fine with GPUs. However, new discussions have commenced regarding performance issues

Yeah, this isue can be closed as the GPU setup issues have been resolved. Will close it to signal this.
Thank you everyone for resolving this and @sgkouzias for testing every RC and final release to check for the fix!

Assuming there are multiple CUDA libraries (e.g. installed via apt and pip install), then ldd is your friend to find out which one is actually loaded, for example:
ldd $(python -c "import torch; print(torch._C.__file__)") | grep libcu
(use a similar command for tensorflow)

Yeah, this isue can be closed as the GPU setup issues have been resolved. Will close it to signal this.
Thank you everyone for resolving this and @sgkouzias for testing every RC and final release to check for the fix!

I am working on a google colab... and by testing the GPU devices it shows nothing... i had checked the tf version, it is 2.17.0. also tried with tf-nightly but no gpu devices found.
I had selected the runtime type T4 GPU.

This issue has been fixed in Tensorflow==2.17.0 version. No need to set any custom environment variables.
Able to access GPU with tensorflow[and-cuda]==2.17.0

not it is not, and 2.18.0 does not work either