failed to query event: CUDA_ERROR_LAUNCH_FAILED: unspecified launch failure tensorflow/stream_executor/dnn.cc:613] CUDNN_STATUS_INTERNAL_ERROR

I am unable to train my model. I get the below error after around the 100th epoch or so (randomly). Sometimes it fails on the 500th or so epoch.
Python Version: 3.7
Tesorflow Version: 2.3
Tensorflow Installed: From binary.
OS: Windows 10
GPU Card: Nvidia Titan V
CUDA: 10.1
CUDNN: <IP_ADDRESS>
: failed to query event: CUDA_ERROR_LAUNCH_FAILED: unspecified launch failure
2020-07-07 17:15:37.551112: E tensorflow/stream_executor/dnn.cc:613] CUDNN_STATUS_INTERNAL_ERROR
in tensorflow/stream_executor/cuda/cuda_dnn.cc(1867): 'cudnnRNNForwardTraining( cudnn.handle(), rnn_desc.handle(), model_dims.max_seq_length, input_desc.handles(), input_data.opaque(), input_h_desc.handle(), input_h_data.opaque(), input_c_desc.handle(), input_c_data.opaque(), rnn_desc.params_handle(), params.opaque(), output_desc.handles(), output_data->opaque(), output_h_desc.handle(), output_h_data->opaque(), output_c_desc.handle(), output_c_data->opaque(), workspace.opaque(), workspace.size(), reserve_space.opaque(), reserve_space.size())'
2020-07-07 17:15:37.551176: F tensorflow/stream_executor/cuda/cuda_dnn.cc:189] Check failed: status == CUDNN_STATUS_SUCCESS (7 vs. 0)Failed to set cuDNN stream.
Please make sure that this is a bug. As per our
GitHub Policy,
we only address code/doc bugs, performance issues, feature requests and
build/installation issues on GitHub. tag:bug_template
System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow):
OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
TensorFlow installed from (source or binary):
TensorFlow version (use command below):
Python version:
Bazel version (if compiling from source):
GCC/Compiler version (if compiling from source):
CUDA/cuDNN version:
GPU model and memory:

You can collect some of this information using our environment capture
script
You can also obtain the TensorFlow version with:

TF 1.0: python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"
TF 2.0: python -c "import tensorflow as tf; print(tf.version.GIT_VERSION, tf.version.VERSION)"

Describe the current behavior
Describe the expected behavior
Standalone code to reproduce the issue
Provide a reproducible test case that is the bare minimum necessary to generate
the problem. If possible, please share a link to Colab/Jupyter/any notebook.
Other info / logs Include any logs or source code that would be helpful to
diagnose the problem. If including tracebacks, please include the full
traceback. Large logs and files should be attached.

@nectario,
Could you please provide the complete code to reproduce the issue reported here?
Also, please take a look at this comment from a similar issue and let us know if it helps. Thanks!

I can't post the code since it's a proprietary model. It seems this is related to NVIDA drivers. This happens RANDOMNLY. Last night it happened on epoch 2500 for example.

My model is a 6 layer LSTM.

I really do not want to downgrade my NVIDIA GPU in order for this issue to go away. It's becoming impossible to train a model till completion because of this issue.

I can't post the code since it's a proprietary model.

@nectario,
As an alternative, is it possible for you to provide a working example which can mimic the error? Thanks!

Yes, I will soon be posting sample code that recreates this issue.

This is an extremely serious issue. It prevents me from training a model till the end.

Hi,
Here is the code that recreates this issue. I was able to make the error happen twice, although it happens randomly.
First time it happened at epoch 1548
Second time it happened at epoch 524
To Download:
http://www.nektron.com/cudnn_error.zip
To Run:
python python CUDNN_Error.py
I have tensorflow 2.3
GPU is the Nvidia Titan V
I am running this on Windows 10
Note: you need to have it run for 3000 epochs and it will happen as some point.

Please note, in the code you will see a Callback that deliberately tries to load data "on_epoch_end"  just to add a bit of a delay that mimics my own callbacks.

@nectario,
I was able to run the code without any issues with TF v2.3. Please find the gist of it here.
Could you please try setting a hard limit on the total GPU memory as show in this guide and check if it works. Thanks!

@amahendrakar Thank you. I will try your suggestion. Looking at how you ran it, I don't believe this is repeatable in a server GPU like the V100 or any other ones. It really needs to run in a consumer GPU like the Titan V, and in PC desktop environment. As of now, I can never finish training so it reaches the last epoch. I think it's related to the Nvidia driver.
Also, this gets triggered quicker if I am watching a YouTube video as I am training. I started getting this only after Nvidia updated their drivers.
Here is a person who also experienced a similar issue. Check this link:
https://forums.developer.nvidia.com/t/crash-on-training-cuda-error-launch-failed/76055
But it seems it was hardware memory issue which needed a GPU replacement?

If someone can run the above in their local desktop environment with the latest Nvidia gpu drivers would be great.

I keep getting this issue, in different variation. Very rarely am I able to complete training. It's extremely frustrating...
2020-09-01 17:05:07.819801: E tensorflow/stream_executor/dnn.cc:616] CUDNN_STATUS_EXECUTION_FAILED
in tensorflow/stream_executor/cuda/cuda_dnn.cc(1892): 'cudnnRNNForwardTraining( cudnn.handle(), rnn_desc.handle(), model_dims.max_seq_length, input_desc.handles(), input_data.opaque(), input_h_desc.handle(), input_h_data.opaque(), input_c_desc.handle(), input_c_data.opaque(), rnn_desc.params_handle(), params.opaque(), output_desc.handles(), output_data->opaque(), output_h_desc.handle(), output_h_data->opaque(), output_c_desc.handle(), output_c_data->opaque(), workspace.opaque(), workspace.size(), reserve_space.opaque(), reserve_space.size())'
2020-09-01 17:05:07.819965: E tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_LAUNCH_FAILED: unspecified launch failure
2020-09-01 17:05:07.861316: W tensorflow/core/framework/op_kernel.cc:1767] OP_REQUIRES failed at cudnn_rnn_ops.cc:1517 : Internal: Failed to call ThenRnnForward with model config: [rnn_mode, rnn_input_mode, rnn_direction_mode]: 2, 0, 0 , [num_layers, input_size, num_units, dir_count, max_seq_length, batch_size, cell_num_units]: [1, 300, 150, 1, 20, 620, 150]
2020-09-01 17:05:07.873534: F tensorflow/core/common_runtime/gpu/gpu_event_mgr.cc:220] Unexpected Event status: 1

Is this some kind of hardware issue I am facing with my Titan V? Today I was only able to have one successful run.

I have made the following observation:
When my room temperature is 73 or below Celsius, the problem does not seem to happen. Generally, on average, my room temperature is 78 Celsius.
Could this be an overheating issue of the GPU?

I think I may have finally resolved this issue. It seems I had two CUDA versions installed: 10.1, and 10.2. My installation was pointing to the 10.2 version. After uninstalling all the other other versions and having only 10.1 I was able to finally train more than once without issues. I will run a couple more tests. Bottom line: After uninstalling all my previous CUDA/CUDNN version and reinstalling the correct ones, it seems to have resolved the issue.

I had this same issue and read all the comments here. I recognized that a pip install software I made updated my tensorflow to 2.3 which is not even a stable build that conda supports. But my tensorflow-gpu version stayed at 1.14.
I realized this and forces both my tensorflow and tensorflow-gpu versions to 2.1. This solved my issue.

Thank you. Apparently this issue is still not gone for me. It is extremely frustrating. I cannot train till completion. TensorFlow is becoming unusable for me... Even using Python 3.8 with a fresh install I still get this...

Thank you too, I have been running for 20 minutes without a crash so far. Try Python 3.7 with tensorflow 2.1 and tensorflow-gpu 2.1 :)

I hate downgrades, but I will give it a shot!

Good luck!

@nectario Confirming, got an error with CUDNN_Error.py at epoch 1683. Python 3.8, TF 2.3.1, CUDA 10.2, RTX 2070S.
Although a bit different error:
2020-10-28 11:31:23.136550: E tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_LAUNCH_FAILED: unspecified launch failure
2020-10-28 11:31:23.136714: F tensorflow/core/common_runtime/gpu/gpu_event_mgr.cc:220] Unexpected Event status: 1
2020-10-28 11:31:23.136712: E tensorflow/stream_executor/dnn.cc:616] CUDNN_STATUS_EXECUTION_FAILED
in tensorflow/stream_executor/cuda/cuda_dnn.cc(1892): 'cudnnRNNForwardTraining( cudnn.handle(), rnn_desc.handle(), model_dims.max_seq_length, input_desc.handles(), input_data.opaque(), input_h_desc.handle(), input_h_data.opaque(), input_c_desc.handle(), input_c_data.opaque(), rnn_desc.params_handle(), params.opaque(), output_desc.handles(), output_data->opaque(), output_h_desc.handle(), output_h_data->opaque(), output_c_desc.handle(), output_c_data->opaque(), workspace.opaque(), workspace.size(), reserve_space.opaque(), reserve_space.size())'

By  the way, switching between CUDA versions in Win is easy without uninstall&reinstall. NVIDIA drivers are backwards compatible with CUDA 10.x and 11.x, so one can always have the latest drivers installed. Just have to make sure to install CUDA & CuDNN separately. nvidia-smi reported CUDA version output is "irrelevant". To switch, I'm using env cmds:
SET CUDA_PATH=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1
SET PATH=%CUDA_PATH%\bin;%PATH%
SET PATH=%CUDA_PATH%\extras\CUPTI\libx64;%PATH%
SET PATH=C:\cudnn-7.6.5\10.1\bin;%PATH%;

Training the same example with CUDA 10.1 in progress.
If I get time, will try again the CUDA 10.2 run with SET CUDA_LAUNCH_BLOCKING=1 to see if there's any extra bits coming out.

Wow, yeap, the error message would also change a bit sometimes for me. A few times I am able to complete an entire training run out of pure luck, but 90% of the time it fails. It can fail on epochs 50, 300, 800, 1100 -- it can be anywhere really.
Let me know how the 10.1 training goes.
Thanks

CUDA 10.1 training succeeds. Might be worth waiting a few weeks for the TF 2.4 release with CUDA 11 support and would be fun to try again. Don't know if it's going to be CUDA 11.0 or 11.1.
To be fair, TensorFlow >=2.1.0 officially supports only CUDA 10.1, in that sense works as expected.
Btw, my GPU utilization was quite low with this training (batch size was optimal for max gpu memory use), so the error was not coming from power consumption and definitely not from overheating.

Thank you. For me it happens with CUDA 10.1. Oh well... At least I now feel confident it is not a hardware issue.

@nectario I did run the same model training on the latest TF 2.4 release candidate (r2.4 branch at 541fd26bd37) with CUDA 11.0 and CuDNN <IP_ADDRESS>.
Unfortunately failed again at epoch 2720.  It is interesting that your failures occur close to the 3000 epochs. In my case, it almost always happens below 1200.

This really needs to get fixed, LSTM's can't be run with Cuda now.

Absolutely. It is even preventing another issue I reported to be debugged because of this issue. It's this one #45676

This issue belongs in a range of issues related to LSTMs and CUDA/CUDNN. Each time the error is a bit different but same effect.

Probably NVIDIA needs to be involved also for this to be fixed.

When I train on AWS on their V100s, this never happens. Seems to be happening mainly with the consumer based graphics cards.

This issue belongs in a range of issues related to LSTMs and CUDA/CUDNN. Each time the error is a bit different but same effect.

Yeah, I get all of them, sometimes this Cuda error, sometimes the rnnbackward or rnnforward error. The strange thing is that it only occurred after I had swapped out my motherboard and CPU and reinstalled my windows.
I downloaded the same drivers as before, did a complete reinstall of my conda environment, but somehow I got this issue in the process.

I know!! I have been living with this for way too long. More than six months... Do you get this with 2.4?

With 2.4 it seems to have made it better, but training got slower.

I know!! I have been living with this for way too long. More than six months... Do you get this with 2.4?

Yep :( I had bought a couple of months ago a RTX 3080, was using TF 2.4 RC0 up to the final release with no issues. Could run LSTM's without a problem, but after swapping my cpu and motherboard and reinstall windows, I can't seem to get around this problem.
I even reinstalled windows another time, but still getting the issue. So maybe it's also hardware related...

Yeah. I suspect it's a combination of gfx driver/tensorflow/cuda/cudnn. Was the gfx driver updated around that time?

This issue needs tensorflow heavyweights to look into it! The credibility of the entire framework is in line! I am this close to completely switching frameworks!

Yeah. I suspect it's a combination of gfx driver/tensorflow/cuda/cudnn. Was the gfx driver updated around that time?

No, I simply reinstalled all the same drivers so that everything was in the same condition.

Ooh, I see...

It's surely seems there may be some hardware issue because it never happens with the V100s.

This requires TensorFlow and NVIDA to look into it...

It's surely seems there may be some hardware issue because it never happens with the V100s.
Probably a combination of multiple things. It is strange that it did not occur for me before though.

This requires TensorFlow and NVIDA to look into it...
Definitely, but have they have even acknowledged this issue yet?

I believe it's been acknowledged but this issue is unique as it's hard to reproduce. If they try to reproduce this on any cloud machine it will never happen. It requires these consumer based graphics cards. Like I have a Titan V or an RTX 3080 which you have. It's good that we are keeping this thread "warm"
I hope someone from TensorFlow can update us here if they are investigating it on what is causing it.
My resolution for now: I reduced the number of epochs. When it fails, I load the last saved weights and continue on.

I believe it's been acknowledged but this issue is unique as it's hard to reproduce. If they try to reproduce this on any cloud machine it will never happen. It requires these consumer based graphics cards. Like I have a Titan V or an RTX 3080 which you have. It's good that we are keeping this thread "warm"
I hope someone from TensorFlow can update us here if they are investigating it on what is causing it.
My resolution for now: I reduced the number of epochs. When it fails, I load the last saved weights and continue on.

Ah I see, let's hope they are looking into it and they are able to reproduce it. I'll try if just disabling the cudnn lstm kernel is enough. I'll give an update if it works!

@nectario So disabling eager execution helps, because it's not using the cudnn kernel for lstm anymore.

@nectario I'm curious, which cudnn versions did you test with when running with the TF 2.4 final? There's now 8.0.3, <IP_ADDRESS> and <IP_ADDRESS>, all having CUDA 11.0 support. I presume the same crash still happened with the 2.4 as well, not just the slowdown issue?

@nectario I'm curious, which cudnn versions did you test with when running with the TF 2.4 final? There's now 8.0.3, <IP_ADDRESS> and <IP_ADDRESS>, all having CUDA 11.0 support. I presume the same crash still happened with the 2.4 as well, not just the slowdown issue?

I tried all of those versions, but no avail. All of them showed the same behaviour.

@nectario I'm curious, which cudnn versions did you test with when running with the TF 2.4 final? There's now 8.0.3, <IP_ADDRESS> and <IP_ADDRESS>, all having CUDA 11.0 support. I presume the same crash still happened with the 2.4 as well, not just the slowdown issue?

I initially tested with the latest versions <IP_ADDRESS> and then went back to the advertised versions.￼

@nectario I ran your cudnn_error.zip through my setup and at least the 1st run worked fine. Although, I did do a couple of optimizations to speed it up.
Started with ~487ms/step, 3s epoch.
Added to the beginning of the CUDNN_Error.py to enable mixed precision and changed all LSTM units from 150 to 152 to be more friendly for the mixed precision (n%8=0)
from tensorflow.keras.mixed_precision import experimental as mixed_precision
policy = mixed_precision.Policy('mixed_float16')
mixed_precision.set_policy(policy)

Went down to ~280ms/step. After enabling mixed precision I could also double the batch size from 620 to 1240 and went further down to ~440ms/step (for previous batch size normalization: 220ms/step). Each epoch down to 1s.
There might be room for XLA-based optimizations, but I'm getting SubProcess ended with return code: 0 from XLA.
Graphics driver 460.89, CUDA 11.0.2, cudnn-<IP_ADDRESS>.

@ahtik Thank you! I will try those changes, except the batch change. The batch was carefully selected to yield better metrics. A bigger batch does not necessarily give me better metric performance.
I wonder if this explains the drop in performance in 2.4?

@nectario I ran your cudnn_error.zip through my setup and at least the 1st run worked fine. Although, I did do a couple of optimizations to speed it up.
Started with ~487ms/step, 3s epoch.
Added to the beginning of the CUDNN_Error.py to enable mixed precision and changed all LSTM units from 150 to 152 to be more friendly for the mixed precision (n%8=0)
from tensorflow.keras.mixed_precision import experimental as mixed_precision
policy = mixed_precision.Policy('mixed_float16')
mixed_precision.set_policy(policy)

Went down to ~280ms/step. After enabling mixed precision I could also double the batch size from 620 to 1240 and went further down to ~440ms/step (for previous batch size normalization: 220ms/step). Each epoch down to 1s.
There might be room for XLA-based optimizations, but I'm getting SubProcess ended with return code: 0 from XLA.
Win10, Graphics driver 460.89, CUDA 11.0.2, cudnn-<IP_ADDRESS>.

@nikitamaia I wonder if the above solution is what is needed to fix the slow performance for #45676?

@nikitamaia @ahtik
Adding your code @ahtik at the start, dropped my per epoch time to 1 second!
from tensorflow.keras.mixed_precision import experimental as mixed_precision
policy = mixed_precision.Policy('mixed_float16')
mixed_precision.set_policy(policy)

@nectario To be honest, I suspect the 2.3 vs 2.4 slowdown in #45676 difference is still somewhere else because mixed precision hasn't been enabled, so it's an extra improvement on top of whatever else is going on.

@nectario To be honest, I suspect the 2.3 vs 2.4 slowdown in #45676 difference is still somewhere else because mixed precision has never been enabled by default, so it's an extra improvement on top of whatever else is going on.

Got it.

@nectario TF2.3 and Cuda 10.1 and cudnn 7.6.5 don't cause any crashes with the cudnn accelerated LSTMs for me, but the only downside with an RTX 3080 is that the initial loading takes up to 15 min! 👎 So also not really a solution. I even tried to install older versions of windows, but also no success. Something within cudnn 8+ and cuda 11+ is causing this.

@ion-elgreco Could you share your minimim LSTM code that is crashing with
TF2.4? I was no longer able to reproduce the crash with the setup described
above with RTX2070S. I did enable mixed precision to speed it up, so maybe
cudnn didn't get enough stress load... How long it takes in your case to
crash it?

@ion-elgreco Could you share your minimim LSTM code that is crashing with TF2.4? I was no longer able to reproduce the crash with the setup described above with RTX2070S. I did enable mixed precision to speed it up, so maybe cudnn didn't get enough stress load... How long it takes in your case to crash it?

def build_base(model_name, units):
model = Sequential(name=model_name)
model.add(layers.InputLayer(input_shape=(X_train.shape[1], X_train.shape[2])))
model.add(layers.LSTM(units, return_sequences=True,))
model.add(layers.Dropout(0.4))
model.add(layers.LayerNormalization())
model.add(layers.Dense(7, activation="softmax", name="Dense_Output"))
model.compile(
optimizer="rmsprop",
loss=CategoricalCrossentropy(),
metrics=["accuracy"]
)
return model
history = NN.fit(
X_train,
y_train,
batch_size=batchsize,
sample_weight=train_samples_weights[:train_div],
validation_data=(X_val, y_val),
epochs=5,
verbose=1,
shuffle=True,
)
`
It will already crash with 5 units after 3 epochs...

@ion-elgreco Can you provide the full code together with batchsize etc hyper parameters, train & val datasets etc. Otherwise, it's a bit too much effort to reproduce.
Does it also crash after adding this to the beginning of the file?
from tensorflow.keras.mixed_precision import experimental as mixed_precision
policy = mixed_precision.Policy('mixed_float16')
mixed_precision.set_policy(policy)

@ion-elgreco Can you provide the full code together with batchsize etc hyper parameters, train & val datasets etc. Otherwise, it's a bit too much effort to reproduce.
Does it also crash after adding this to the beginning of the file?
from tensorflow.keras.mixed_precision import experimental as mixed_precision
policy = mixed_precision.Policy('mixed_float16')
mixed_precision.set_policy(policy)

@ion-elgreco Can you provide the full code together with batchsize etc hyper parameters, train & val datasets etc. Otherwise, it's a bit too much effort to reproduce.
Does it also crash after adding this to the beginning of the file?
from tensorflow.keras.mixed_precision import experimental as mixed_precision
policy = mixed_precision.Policy('mixed_float16')
mixed_precision.set_policy(policy)

This is the code: https://github.com/ion-elgreco/thesis-FER/blob/main/models.py
However, I cannot share the datasets because of a legal agreement. The thing is, I hadn't changed a bit to this code. It worked fine before on my i7 7700k + rtx 3080, but after upgrading to a 5800x processor and having to reinstall windows, I keep getting this issue. I even tried reinstalling windows, 5 times..

@ion-elgreco I'm not aware of a good reason how the training could become slower after enabling mixed precision...
Do I understand correctly that you're also running TF2.3 right now inside this new desktop with a Ryzen CPU, and just switching between active CUDA version? So the TF2.3 is not crashing and TF2.4 does with the exact same hw and software conf.
Couldn't rule out the HW issue as well, this one is from pytorch, but still relevant. https://discuss.pytorch.org/t/runtimeerror-cudnn-status-internal-error-when-l-run-the-program-for-a-second-time/2960/4
As there can be many causes for CUDNN_STATUS_INTERNAL_ERROR, the only pragmatic way to help here, is to also provide the dataset (processed enough to not cause the legal issues), that would be as small and compact as possible to reproduce the issue with TF2.4. Also include the affected version info etc. Might be worth opening as a separate issue, as the cause seems to be different from the one from @nectario.
I noticed a comment of using conda. To be 100% clean, I'd definitely try to reproduce this also with a clean venv install with pip. Smth like
cd projdir
\python38\python.exe -m venv .venv
.\venv\Scripts\activate or .\venv\Scripts\Activate.ps1 for powershell
pip install tensorflow
python -c "import tensorflow as tf;print(tf.reduce_sum(tf.random.normal([1000, 1000])));print(tf.config.list_physical_devices('GPU'))"
python yourtrainging.py

@ion-elgreco I'm not aware of a good reason how the training could become slower after enabling mixed precision... Initial graph building, yes.
Do I understand correctly that you're also running TF2.3 right now inside this new desktop with a Ryzen CPU, and just switching between active CUDA version? So the TF2.3 is not crashing and TF2.4 does with the exact same hw and software conf.
Couldn't rule out the HW issue as well, this one is from pytorch, but still relevant. https://discuss.pytorch.org/t/runtimeerror-cudnn-status-internal-error-when-l-run-the-program-for-a-second-time/2960/4
As there can be many causes for CUDNN_STATUS_INTERNAL_ERROR, the only pragmatic way to help here, is to also provide the dataset (processed enough to not cause the legal issues), that would be as small and compact as possible to reproduce the issue with TF2.4. Also include the affected version info etc. Might be worth opening as a separate issue, as the cause seems to be different from the one from @nectario.
I noticed a comment of using conda. To be 100% clean, I'd definitely try to reproduce this also with a clean venv install with pip. Smth like
cd projdir
\python38\python.exe -m venv .venv
.\venv\Scripts\activate or .\venv\Scripts\Activate.ps1 for powershell
pip install -U pip
pip install tensorflow
python -c "import tensorflow as tf;print(tf.reduce_sum(tf.random.normal([1000, 1000])));print(tf.config.list_physical_devices('GPU'))"
python yourtrainging.py

I have two environments, my main one with TF 2.4 with Cuda 11 and Cudnn 8.0.5, and TF 2.3 with Cuda 10.1 with Cudnn 7.6.5. After running TF 2.3 longer it also seems to crash so I was mistaken that it only was with Cuda 11+ and Cudnn 8+.
I am not only getting CUDNN_STATUS_INTERNAL_ERROR, I also get:
Failed to call ThenRnnBackward with model config: [rnn_mode, rnn_input_mode, rnn_direction_mode]: 2, 0, 0 , [num_layers, input_size, num_units, dir_count, max_seq_length, batch_size, cell_num_units]: [1, 4608, 100, 1, 60, 32, 100]
	 [[{{node gradients/CudnnRNN_grad/CudnnRNNBackprop}}]]
	 [[PartitionedCall]] [Op:__inference_train_function_1127558]

And some other cudnn crashes, they all happen at random and the error code is each time random.
I'll try in a clean venv environment later today.

@ion-elgreco I'm not aware of a good reason how the training could become slower after enabling mixed precision... Initial graph building, yes.
Do I understand correctly that you're also running TF2.3 right now inside this new desktop with a Ryzen CPU, and just switching between active CUDA version? So the TF2.3 is not crashing and TF2.4 does with the exact same hw and software conf.
Couldn't rule out the HW issue as well, this one is from pytorch, but still relevant. https://discuss.pytorch.org/t/runtimeerror-cudnn-status-internal-error-when-l-run-the-program-for-a-second-time/2960/4
As there can be many causes for CUDNN_STATUS_INTERNAL_ERROR, the only pragmatic way to help here, is to also provide the dataset (processed enough to not cause the legal issues), that would be as small and compact as possible to reproduce the issue with TF2.4. Also include the affected version info etc. Might be worth opening as a separate issue, as the cause seems to be different from the one from @nectario.
I noticed a comment of using conda. To be 100% clean, I'd definitely try to reproduce this also with a clean venv install with pip. Smth like
cd projdir
\python38\python.exe -m venv .venv
.\venv\Scripts\activate or .\venv\Scripts\Activate.ps1 for powershell
pip install -U pip
pip install tensorflow
python -c "import tensorflow as tf;print(tf.reduce_sum(tf.random.normal([1000, 1000])));print(tf.config.list_physical_devices('GPU'))"
python yourtrainging.py

That is interesting. It could be definitely the case that the 5800x is causing it..

@ion-elgreco It could also help if you have a non-Ryzen GPU setup somewhere that can be used for this private dataset training making sure it works there with both TF2.3 and TF2.4.

@ion-elgreco It could also help if you have a non-Ryzen GPU setup somewhere that can be used for this private dataset training making sure it works there with both TF2.3 and TF2.4.

I had an i7-7700k a week ago, with TF2.3 and TF2.4 my code worked. I cant switch back to Intel because then I would have to lend components from someone.

One interesting thing that is happening right now in regards to this issue:
After installing TensorFlow 2.4.0, and ugrading my CUDA to 11.0 and 8.02 then downgrading to TensorFlow 2.3.1 and downgrading my CUDA to the version supported by TensorFlow, the issue seems to have stopped happening.
I have trained twice already with 1500 epochs, and no failures so far....

My NVIDIA driver is also different. I am using the studio version rather than the gaming version.

It is a bit slower. I remember running this at 1sec per epoch.

It is a bit slower. I remember running this at 1sec per epoch.

Studio and game ready drivers are the same if the versions are the same

It is a bit slower. I remember running this at 1sec per epoch.

Studio and game ready drivers are the same if the versions are the same

I see. It's strange, issue not happening now.

It is a bit slower. I remember running this at 1sec per epoch.

Studio and game ready drivers are the same if the versions are the same

I see. It's strange, issue not happening now.

I have tried either version of studio and game-ready. Tried all of them which are available for the RTX 3080, but the issue persists.

This is my training output wen it starts:
2020-12-28 13:59:46.246094: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library nvcuda.dll 2020-12-28 13:59:46.275929: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:  pciBusID: 0000:83:00.0 name: TITAN V computeCapability: 7.0 coreClock: 1.455GHz coreCount: 80 deviceMemorySize: 12.00GiB deviceMemoryBandwidth: 607.97GiB/s 2020-12-28 13:59:46.276253: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll 2020-12-28 13:59:46.288071: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cublas64_10.dll 2020-12-28 13:59:46.294217: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cufft64_10.dll 2020-12-28 13:59:46.296723: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library curand64_10.dll 2020-12-28 13:59:46.305299: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusolver64_10.dll 2020-12-28 13:59:46.311567: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusparse64_10.dll 2020-12-28 13:59:46.346196: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudnn64_7.dll 2020-12-28 13:59:46.346475: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0 2020-12-28 13:59:46.362025: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x20e513b8d80 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2020-12-28 13:59:46.362291: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version 2020-12-28 13:59:46.362731: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:  pciBusID: 0000:83:00.0 name: TITAN V computeCapability: 7.0 coreClock: 1.455GHz coreCount: 80 deviceMemorySize: 12.00GiB deviceMemoryBandwidth: 607.97GiB/s 2020-12-28 13:59:46.363085: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll 2020-12-28 13:59:46.363276: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cublas64_10.dll 2020-12-28 13:59:46.363465: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cufft64_10.dll 2020-12-28 13:59:46.363648: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library curand64_10.dll 2020-12-28 13:59:46.363841: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusolver64_10.dll 2020-12-28 13:59:46.364033: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusparse64_10.dll 2020-12-28 13:59:46.364218: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudnn64_7.dll 2020-12-28 13:59:46.364439: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0 2020-12-28 13:59:47.265842: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix: 2020-12-28 13:59:47.266092: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263]      0  2020-12-28 13:59:47.266217: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0:   N  2020-12-28 13:59:47.266667: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10245 MB memory) -> physical GPU (device: 0, name: TITAN V, pci bus id: 0000:83:00.0, compute capability: 7.0) 2020-12-28 13:59:47.271449: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x20ea8d5c4c0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2020-12-28 13:59:47.271684: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): TITAN V, Compute Capability 7.0

This is my training output wen it starts:
2020-12-28 13:59:46.246094: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library nvcuda.dll 2020-12-28 13:59:46.275929: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties: pciBusID: 0000:83:00.0 name: TITAN V computeCapability: 7.0 coreClock: 1.455GHz coreCount: 80 deviceMemorySize: 12.00GiB deviceMemoryBandwidth: 607.97GiB/s 2020-12-28 13:59:46.276253: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll 2020-12-28 13:59:46.288071: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cublas64_10.dll 2020-12-28 13:59:46.294217: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cufft64_10.dll 2020-12-28 13:59:46.296723: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library curand64_10.dll 2020-12-28 13:59:46.305299: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusolver64_10.dll 2020-12-28 13:59:46.311567: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusparse64_10.dll 2020-12-28 13:59:46.346196: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudnn64_7.dll 2020-12-28 13:59:46.346475: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0 2020-12-28 13:59:46.362025: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x20e513b8d80 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2020-12-28 13:59:46.362291: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2020-12-28 13:59:46.362731: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties: pciBusID: 0000:83:00.0 name: TITAN V computeCapability: 7.0 coreClock: 1.455GHz coreCount: 80 deviceMemorySize: 12.00GiB deviceMemoryBandwidth: 607.97GiB/s 2020-12-28 13:59:46.363085: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll 2020-12-28 13:59:46.363276: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cublas64_10.dll 2020-12-28 13:59:46.363465: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cufft64_10.dll 2020-12-28 13:59:46.363648: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library curand64_10.dll 2020-12-28 13:59:46.363841: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusolver64_10.dll 2020-12-28 13:59:46.364033: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusparse64_10.dll 2020-12-28 13:59:46.364218: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudnn64_7.dll 2020-12-28 13:59:46.364439: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0 2020-12-28 13:59:47.265842: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix: 2020-12-28 13:59:47.266092: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263] 0 2020-12-28 13:59:47.266217: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0: N 2020-12-28 13:59:47.266667: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10245 MB memory) -> physical GPU (device: 0, name: TITAN V, pci bus id: 0000:83:00.0, compute capability: 7.0) 2020-12-28 13:59:47.271449: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x20ea8d5c4c0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2020-12-28 13:59:47.271684: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): TITAN V, Compute Capability 7.0

So it's working now with the exact same model and data? Because somehow I can run LSTM fine with some small text data set, but not my big image dataset.

This is my training output wen it starts:

So it's working now with the exact same model and data? Because somehow I can run LSTM fine with some small text data set, but not with my big image dataset.

It's the same large data although slightly different (some content is different).

The difference right now is that I can control training based to cutoff dates. My cutoff date right now is at Sept 30th.

The main difference is the NVIDIA driver. It was different before. I am now using: 460.89 (Studio Version)

My existing training would almost always fail. I am on the third run right now.

I found the cause of cuda crashes... it is the RMSprop optimizer. With all other leaners, I can run the LSTM fine, this is mad...

Wow! What about adam? That is what I have.

Wow! What about adam? That is what I have.

Oh never mind.. With Adam it will crash simply not as fast as with RMSprop.

Ok I have some good news :) I was messing around with WSL2 to simply run tensorflow-gpu in Linux. So I had to install the cuda capable driver 465.12 (developer) driver, which now does not cause anymore crashes in Windows.
PS
In case someone has experience with WSL2, I am struggling to get it to recognize my GPU.

Wow, awesome! So it's a driver issue!
Funny you are messing with WSL2 -- I was doing the same a couple of days back. Haven't gotten too deep into it. Maybe this will help:
https://ubuntu.com/blog/getting-started-with-cuda-on-ubuntu-on-wsl-2

Wow, awesome! So it's a driver issue!
Funny you are messing with WSL2 -- I was doing the same a couple of days back. Haven't gotten too deep into it. Maybe this will help:
https://ubuntu.com/blog/getting-started-with-cuda-on-ubuntu-on-wsl-2

Yeah I found that guide already, but Ubuntu with WSL2 won't recognize my gpu. Even though  I have installed the right driver.

Same for me. 2.4 was core dumping and I had to install 2.3.1 but it was CPU version. If I get it to work with the GPU I will let you know.
But man, after all these months, finally a solution with the above problem! Today is the first time I can train with no issue for hours!

Same for me. 2.4 was core dumping and I had to install 2.3.1 but it was CPU version. If I get it to work with the GPU I will let you know.
But man, after all these months, finally a solution with the above problem! Today is the first time I can train with no issue for hours!

Yeah it's nice to see TF working properly again! If you got GPU working in WSL Lett me know :)  I don't know where I read it but it seems like they want to push GPU support more towards WSL.

You got it! That would be great.

You got it! That would be great.

I think I have found the cause. I am not on a windows insider build. I am on:
Edition	Windows 10 Pro
Version	20H2
Installed on	20/12/2020
OS build	19042.685
Experience	Windows Feature Experience Pack 120.2212.551.0

But according to the cuda docs I should be on 20145 or higher and for tensorflow directml 20150 or higher..

Ah, great. I just initiated the enrollment on the Windows Insider Build. This is great.

I know you found a fix, but another fix i found is changing the settings so they do not align with the cudnn requirements:
https://www.tensorflow.org/api_docs/python/tf/keras/layers/LSTM i.e. adding recurrent dropout - which can be very useful.
If you change these settings it forces it to use the generic GPU kernel and seems to work even if at a performance cost.

I know you found a fix, but another fix i found is changing the settings so they do not align with the cudnn requirements:
https://www.tensorflow.org/api_docs/python/tf/keras/layers/LSTM i.e. adding recurrent dropout - which can be very useful.
If you change these settings it forces it to use the generic GPU kernel and seems to work even if at a performance cost.

I know you found a fix, but another fix i found is changing the settings so they do not align with the cudnn requirements:
https://www.tensorflow.org/api_docs/python/tf/keras/layers/LSTM i.e. adding recurrent dropout - which can be very useful.
If you change these settings it forces it to use the generic GPU kernel and seems to work even if at a performance cost.

That is not a fix though, because you are not using the Cudnn kernel. It's a nice trick though, I used it too in the mean while by disabling eager execution

I know you found a fix, but another fix i found is changing the settings so they do not align with the cudnn requirements:
https://www.tensorflow.org/api_docs/python/tf/keras/layers/LSTM i.e. adding recurrent dropout - which can be very useful.
If you change these settings it forces it to use the generic GPU kernel and seems to work even if at a performance cost.

That is not a fix though, because you are not using the Cudnn kernel. It's a nice trick though, I used it too in the mean while by disabling eager execution

Sorry, this solution will not work for me. I need the speed increase that the CUDNN optimization provides due to the large number of epochs.

Sorry, this solution will not work for me. I need the speed increase that the CUDNN optimization provides due to the large number of epochs.

Have you tried and compared the speed? From what you said you are using a lot of RNN -- my trick doesn't hurt performance too much for one RNN layer as the rest of the layers (CNN, Dense) still use CUDNN. With many RNN layers maybe not. Another option is to use ConvLSTM2D with appropriately resized inputs. That also worked for me.

Have you tried and compared the speed? From what you said you are using a lot of RNN -- my trick doesn't hurt performance too much for one RNN layer as the rest of the layers (CNN, Dense) still use CUDNN. With many RNN layers maybe not. Another option is to use ConvLSTM2D with appropriately resized inputs. That also worked for me.

You got it! That would be great.

I think I have found the cause. I am not on a windows insider build. I am on:
Edition	Windows 10 Pro
Version	20H2
Installed on	20/12/2020
OS build	19042.685
Experience	Windows Feature Experience Pack 120.2212.551.0

But according to the cuda docs I should be on 20145 or higher and for tensorflow directml 20150 or higher to be able to use cuda in wsl2.

@ion-elgreco
I finally got TensorFlow with CUDA/CUDNN to work WSL 2 (without using Docker containers). Just a few things:

I had to use TF 2.3.1 as 2.4 would core dump.
It's actually slower: It takes 6 seconds per epoch versus 2 seconds on Windows 10.

If the speed issue can be resolved hands down this would be the only environment I will use!

You got it! That would be great.

I think I have found the cause. I am not on a windows insider build. I am on:
Edition	Windows 10 Pro
Version	20H2
Installed on	20/12/2020
OS build	19042.685
Experience	Windows Feature Experience Pack 120.2212.551.0

But according to the cuda docs I should be on 20145 or higher and for tensorflow directml 20150 or higher to be able to use cuda in wsl2.

@ion-elgreco
I finally got TensorFlow with CUDA/CUDNN to work WSL 2 (without using Docker containers). Just a few things:

I had to use TF 2.3.1 as 2.4 would core dump.
It's actually slower: It takes 6 seconds per epoch versus 2 seconds on Windows 10.

If the speed issue can be resolved hands down this would be the only environment I will use!

core dump.

Could you maybe explain in detail, which OS build, driver versions etc you needed to get it to work?

core dump.

Could you maybe explain in detail, which OS build, driver versions etc you needed to get it to work?

core dump.

Could you maybe explain in detail, which OS build, driver versions etc you needed to get it to work?

Here is the info:
Build: 10.0.21277 Build 21277
Nvidia Driver Version: 460.89 (Studio)
As for the commands, I ran a bunch but I think these made it work:
wget http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu2004/x86_64/nvidia-machine-learning-repo-ubuntu2004_1.0.0-1_amd64.deb

sudo apt install ./nvidia-machine-learning-repo-ubuntu2004_1.0.0-1_amd64.deb

sudo apt install nvidia-cuda-toolkit

download cudnn 7.65 for ubuntu 18.04 (there is no 20.04 version and you need to login and manually download it):

https://developer.nvidia.com/compute/machine-learning/cudnn/secure/<IP_ADDRESS>/Production/10.1_20191031/Ubuntu18_04-x64/libcudnn7_<IP_ADDRESS>-1%2Bcuda10.1_amd64.deb

sudo apt install ./libcudnn7_<IP_ADDRESS>-1%2Bcuda10.1_amd64.deb

sudo apt install nvidia-driver-440

type nvidia-smi to verify

Make sure you install tensorflow 3.1

core dump.

Could you maybe explain in detail, which OS build, driver versions etc you needed to get it to work?

Here is the info:
Build: 10.0.21277 Build 21277
Nvidia Driver Version: 460.89 (Studio)
As for the commands, I ran a bunch but I think these made it work:
wget http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu2004/x86_64/nvidia-machine-learning-repo-ubuntu2004_1.0.0-1_amd64.deb

sudo apt install ./nvidia-machine-learning-repo-ubuntu2004_1.0.0-1_amd64.deb

sudo apt install nvidia-cuda-toolkit

download cudnn 7.65 for ubuntu 18.04 (there is no 20.04 version and you need to login and manually download it):

https://developer.nvidia.com/compute/machine-learning/cudnn/secure/<IP_ADDRESS>/Production/10.1_20191031/Ubuntu18_04-x64/libcudnn7_<IP_ADDRESS>-1%2Bcuda10.1_amd64.deb

sudo apt install ./libcudnn7_<IP_ADDRESS>-1%2Bcuda10.1_amd64.deb

sudo apt install nvidia-driver-440

type nvidia-smi to verify

Make sure you install tensorflow 3.1

@ion-elgreco
I was trying many things for a long time not realizing the issue was with TF 2.4.0

@ion-elgreco
I was trying many things for a long time not realizing the issue was with TF 2.4.0

core dump.

Could you maybe explain in detail, which OS build, driver versions etc you needed to get it to work?

Here is the info:
Build: 10.0.21277 Build 21277
Nvidia Driver Version: 460.89 (Studio)
As for the commands, I ran a bunch but I think these made it work:
wget http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu2004/x86_64/nvidia-machine-learning-repo-ubuntu2004_1.0.0-1_amd64.deb

sudo apt install ./nvidia-machine-learning-repo-ubuntu2004_1.0.0-1_amd64.deb

sudo apt install nvidia-cuda-toolkit

download cudnn 7.65 for ubuntu 18.04 (there is no 20.04 version and you need to login and manually download it):

https://developer.nvidia.com/compute/machine-learning/cudnn/secure/<IP_ADDRESS>/Production/10.1_20191031/Ubuntu18_04-x64/libcudnn7_<IP_ADDRESS>-1%2Bcuda10.1_amd64.deb

sudo apt install ./libcudnn7_<IP_ADDRESS>-1%2Bcuda10.1_amd64.deb

sudo apt install nvidia-driver-440

type nvidia-smi to verify

Make sure you install tensorflow 3.1

Hmm interesting, because you are installing the Nvidia driver inside the WSL ubuntu env. According to env to properly use your GPU is to install the driver only on your main OS and not inside the WSL ubuntu. Maybe that's why it's slower?
I'll give your commands a try later this week! Thanks.

core dump.

Could you maybe explain in detail, which OS build, driver versions etc you needed to get it to work?

Here is the info:
Build: 10.0.21277 Build 21277
Nvidia Driver Version: 460.89 (Studio)
As for the commands, I ran a bunch but I think these made it work:
wget http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu2004/x86_64/nvidia-machine-learning-repo-ubuntu2004_1.0.0-1_amd64.deb

sudo apt install ./nvidia-machine-learning-repo-ubuntu2004_1.0.0-1_amd64.deb

sudo apt install nvidia-cuda-toolkit

download cudnn 7.65 for ubuntu 18.04 (there is no 20.04 version and you need to login and manually download it):

https://developer.nvidia.com/compute/machine-learning/cudnn/secure/<IP_ADDRESS>/Production/10.1_20191031/Ubuntu18_04-x64/libcudnn7_<IP_ADDRESS>-1%2Bcuda10.1_amd64.deb

sudo apt install ./libcudnn7_<IP_ADDRESS>-1%2Bcuda10.1_amd64.deb

sudo apt install nvidia-driver-440

type nvidia-smi to verify

Make sure you install tensorflow 3.1

Hmm interesting, because you are installing the Nvidia driver inside the WSL ubuntu env. According to env to properly use your GPU is to install the driver only on your main OS and not inside the WSL ubuntu. Maybe that's why it's slower?
I'll give your commands a try later this week! Thanks.

Maybe you are right. I was having a ton of issues before that. I will try uninstalling it and see what happens.

Maybe you are right. I was having a ton of issues before that. I will try uninstalling it and see what happens.

@ion-elgreco

core dump.

Could you maybe explain in detail, which OS build, driver versions etc you needed to get it to work?

Here is the info:
Build: 10.0.21277 Build 21277
Nvidia Driver Version: 460.89 (Studio)
As for the commands, I ran a bunch but I think these made it work:
wget http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu2004/x86_64/nvidia-machine-learning-repo-ubuntu2004_1.0.0-1_amd64.deb

sudo apt install ./nvidia-machine-learning-repo-ubuntu2004_1.0.0-1_amd64.deb

sudo apt install nvidia-cuda-toolkit

download cudnn 7.65 for ubuntu 18.04 (there is no 20.04 version and you need to login and manually download it):

https://developer.nvidia.com/compute/machine-learning/cudnn/secure/<IP_ADDRESS>/Production/10.1_20191031/Ubuntu18_04-x64/libcudnn7_<IP_ADDRESS>-1%2Bcuda10.1_amd64.deb

sudo apt install ./libcudnn7_<IP_ADDRESS>-1%2Bcuda10.1_amd64.deb

sudo apt install nvidia-driver-440

type nvidia-smi to verify

Make sure you install tensorflow 3.1

Hmm interesting, because you are installing the Nvidia driver inside the WSL ubuntu env. According to env to properly use your GPU is to install the driver only on your main OS and not inside the WSL ubuntu. Maybe that's why it's slower?
I'll give your commands a try later this week! Thanks.

Maybe you are right. I was having a ton of issues before that. I will try uninstalling it and see what happens.

Only the developer driver version is supported according to Nvidia. https://developer.nvidia.com/cuda/wsl/download

@ion-elgreco

core dump.

Could you maybe explain in detail, which OS build, driver versions etc you needed to get it to work?

Here is the info:
Build: 10.0.21277 Build 21277
Nvidia Driver Version: 460.89 (Studio)
As for the commands, I ran a bunch but I think these made it work:
wget http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu2004/x86_64/nvidia-machine-learning-repo-ubuntu2004_1.0.0-1_amd64.deb

sudo apt install ./nvidia-machine-learning-repo-ubuntu2004_1.0.0-1_amd64.deb

sudo apt install nvidia-cuda-toolkit

download cudnn 7.65 for ubuntu 18.04 (there is no 20.04 version and you need to login and manually download it):

https://developer.nvidia.com/compute/machine-learning/cudnn/secure/<IP_ADDRESS>/Production/10.1_20191031/Ubuntu18_04-x64/libcudnn7_<IP_ADDRESS>-1%2Bcuda10.1_amd64.deb

sudo apt install ./libcudnn7_<IP_ADDRESS>-1%2Bcuda10.1_amd64.deb

sudo apt install nvidia-driver-440

type nvidia-smi to verify

Make sure you install tensorflow 3.1

Hmm interesting, because you are installing the Nvidia driver inside the WSL ubuntu env. According to env to properly use your GPU is to install the driver only on your main OS and not inside the WSL ubuntu. Maybe that's why it's slower?
I'll give your commands a try later this week! Thanks.

Maybe you are right. I was having a ton of issues before that. I will try uninstalling it and see what happens.

Only the developer driver version is supported according to Nvidia. https://developer.nvidia.com/cuda/wsl/download

@ion-elgreco

core dump.

Could you maybe explain in detail, which OS build, driver versions etc you needed to get it to work?

Here is the info:
Build: 10.0.21277 Build 21277
Nvidia Driver Version: 460.89 (Studio)
As for the commands, I ran a bunch but I think these made it work:
wget http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu2004/x86_64/nvidia-machine-learning-repo-ubuntu2004_1.0.0-1_amd64.deb

sudo apt install ./nvidia-machine-learning-repo-ubuntu2004_1.0.0-1_amd64.deb

sudo apt install nvidia-cuda-toolkit

download cudnn 7.65 for ubuntu 18.04 (there is no 20.04 version and you need to login and manually download it):

https://developer.nvidia.com/compute/machine-learning/cudnn/secure/<IP_ADDRESS>/Production/10.1_20191031/Ubuntu18_04-x64/libcudnn7_<IP_ADDRESS>-1%2Bcuda10.1_amd64.deb

sudo apt install ./libcudnn7_<IP_ADDRESS>-1%2Bcuda10.1_amd64.deb

sudo apt install nvidia-driver-440

type nvidia-smi to verify

Make sure you install tensorflow 3.1

Hmm interesting, because you are installing the Nvidia driver inside the WSL ubuntu env. According to env to properly use your GPU is to install the driver only on your main OS and not inside the WSL ubuntu. Maybe that's why it's slower?
I'll give your commands a try later this week! Thanks.

Maybe you are right. I was having a ton of issues before that. I will try uninstalling it and see what happens.

Only the developer driver version is supported according to Nvidia. https://developer.nvidia.com/cuda/wsl/download

Actually, I had the correct driver version installed (the one in the link you sent). I confused the new driver update message with the version installed. I am still at 6 seconds even after uninstalling nvidia-440...

@ion-elgreco

core dump.

Could you maybe explain in detail, which OS build, driver versions etc you needed to get it to work?

Here is the info:
Build: 10.0.21277 Build 21277
Nvidia Driver Version: 460.89 (Studio)
As for the commands, I ran a bunch but I think these made it work:
wget http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu2004/x86_64/nvidia-machine-learning-repo-ubuntu2004_1.0.0-1_amd64.deb

sudo apt install ./nvidia-machine-learning-repo-ubuntu2004_1.0.0-1_amd64.deb

sudo apt install nvidia-cuda-toolkit

download cudnn 7.65 for ubuntu 18.04 (there is no 20.04 version and you need to login and manually download it):

https://developer.nvidia.com/compute/machine-learning/cudnn/secure/<IP_ADDRESS>/Production/10.1_20191031/Ubuntu18_04-x64/libcudnn7_<IP_ADDRESS>-1%2Bcuda10.1_amd64.deb

sudo apt install ./libcudnn7_<IP_ADDRESS>-1%2Bcuda10.1_amd64.deb

sudo apt install nvidia-driver-440

type nvidia-smi to verify

Make sure you install tensorflow 3.1

Hmm interesting, because you are installing the Nvidia driver inside the WSL ubuntu env. According to env to properly use your GPU is to install the driver only on your main OS and not inside the WSL ubuntu. Maybe that's why it's slower?
I'll give your commands a try later this week! Thanks.

Maybe you are right. I was having a ton of issues before that. I will try uninstalling it and see what happens.

Only the developer driver version is supported according to Nvidia. https://developer.nvidia.com/cuda/wsl/download

Actually, I had the correct driver version installed (the one in the link you sent). I confused the new driver update message with the version installed. I am still at 6 seconds even after uninstalling nvidia-440...

@ion-elgreco

core dump.

Could you maybe explain in detail, which OS build, driver versions etc you needed to get it to work?

Here is the info:
Build: 10.0.21277 Build 21277
Nvidia Driver Version: 460.89 (Studio)
As for the commands, I ran a bunch but I think these made it work:
wget http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu2004/x86_64/nvidia-machine-learning-repo-ubuntu2004_1.0.0-1_amd64.deb

sudo apt install ./nvidia-machine-learning-repo-ubuntu2004_1.0.0-1_amd64.deb

sudo apt install nvidia-cuda-toolkit

download cudnn 7.65 for ubuntu 18.04 (there is no 20.04 version and you need to login and manually download it):

https://developer.nvidia.com/compute/machine-learning/cudnn/secure/<IP_ADDRESS>/Production/10.1_20191031/Ubuntu18_04-x64/libcudnn7_<IP_ADDRESS>-1%2Bcuda10.1_amd64.deb

sudo apt install ./libcudnn7_<IP_ADDRESS>-1%2Bcuda10.1_amd64.deb

sudo apt install nvidia-driver-440

type nvidia-smi to verify

Make sure you install tensorflow 3.1

Hmm interesting, because you are installing the Nvidia driver inside the WSL ubuntu env. According to env to properly use your GPU is to install the driver only on your main OS and not inside the WSL ubuntu. Maybe that's why it's slower?
I'll give your commands a try later this week! Thanks.

Maybe you are right. I was having a ton of issues before that. I will try uninstalling it and see what happens.

Only the developer driver version is supported according to Nvidia. https://developer.nvidia.com/cuda/wsl/download

Actually, I had the correct driver version installed (the one in the link you sent). I confused the new driver update message with the version installed. I am still at 6 seconds even after uninstalling nvidia-440...

It's probably the slow IO with WSL. Do you need to load a lot of files?

@ion-elgreco

core dump.

Could you maybe explain in detail, which OS build, driver versions etc you needed to get it to work?

Here is the info:
Build: 10.0.21277 Build 21277
Nvidia Driver Version: 460.89 (Studio)
As for the commands, I ran a bunch but I think these made it work:
wget http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu2004/x86_64/nvidia-machine-learning-repo-ubuntu2004_1.0.0-1_amd64.deb

sudo apt install ./nvidia-machine-learning-repo-ubuntu2004_1.0.0-1_amd64.deb

sudo apt install nvidia-cuda-toolkit

download cudnn 7.65 for ubuntu 18.04 (there is no 20.04 version and you need to login and manually download it):

https://developer.nvidia.com/compute/machine-learning/cudnn/secure/<IP_ADDRESS>/Production/10.1_20191031/Ubuntu18_04-x64/libcudnn7_<IP_ADDRESS>-1%2Bcuda10.1_amd64.deb

sudo apt install ./libcudnn7_<IP_ADDRESS>-1%2Bcuda10.1_amd64.deb

sudo apt install nvidia-driver-440

type nvidia-smi to verify

Make sure you install tensorflow 3.1

Hmm interesting, because you are installing the Nvidia driver inside the WSL ubuntu env. According to env to properly use your GPU is to install the driver only on your main OS and not inside the WSL ubuntu. Maybe that's why it's slower?
I'll give your commands a try later this week! Thanks.

Maybe you are right. I was having a ton of issues before that. I will try uninstalling it and see what happens.

Only the developer driver version is supported according to Nvidia. https://developer.nvidia.com/cuda/wsl/download

Actually, I had the correct driver version installed (the one in the link you sent). I confused the new driver update message with the version installed. I am still at 6 seconds even after uninstalling nvidia-440...

It's probably the slow IO with WSL. Do you need to load a lot of files?

@nectario There's an interesting observation and a fix regarding the TDR setting causing issues, MAYBE it's also relevant to your CUDA_ERROR_LAUNCH_FAILED case.

@nectario There's an interesting observation and a fix regarding the TDR setting causing issues, MAYBE it's also relevant to your CUDA_ERROR_LAUNCH_FAILED case. #45987 (comment)

@ahtik Thank you for bringing this to our attention! I just applied the fix. Let's see...!

were you able to resolve the error?  I am getting the same thing currently.

@ericvoots Yes. It has not happened ever since I upgraded my driver. I also applied the above fix. Although this issue was already resolved with the driver upgrade.

@ion-elgreco, upgrading to the driver version 465.12 (developer)  solved this problem for me as well with TF2.2/Cuda 10.1.  Wonder whether this version in off the master release path and when a supported release driver will work.  I did not try the latest suggested driver to check.  Just happy I'm running again. Many thanks

@ericvoots Yes. It has not happened ever since I upgraded my driver. I also applied the above fix. Although this issue was already resolved with the driver upgrade.

Do you mean upgrade the drive version not the cuda version? I think this Bcuda10.1_amd64.deb link is not working and make sure tensorflow 3.1 is installed is not correct.
download cudnn 7.65 for ubuntu 18.04 (there is no 20.04 version and you need to login and manually download it):

https://developer.nvidia.com/compute/machine-learning/cudnn/secure/<IP_ADDRESS>/Production/10.1_20191031/Ubuntu18_04-x64/libcudnn7_<IP_ADDRESS>-1%2Bcuda10.1_amd64.deb

sudo apt install ./libcudnn7_<IP_ADDRESS>-1%2Bcuda10.1_amd64.deb

sudo apt install nvidia-driver-440

type nvidia-smi to verify

Make sure you install tensorflow 3.1

I have the same problem, I think it's because the versions of cuda and cudnn have been messed up.
My tensorflow is 2.4.1
CUDA Version: 11.2
Driver Version: 460.39
I installed cudnn by sudo dpkg -i libcudnn8_<IP_ADDRESS>-1+cuda11.2_amd64.deb and sudo dpkg -i libcudnn8-dev_<IP_ADDRESS>-1+cuda11.2_amd64.deb
But I cannot find the correct output via
cat /usr/include/cudnn.h | grep CUDNN_MAJOR -A 2

Is this issue addressed with the 2.7 release?

sorry everyone, I don't know which comment should be responded. I find I upgraded requests using sudo pip install --upgrade requests and now I can't import it anymore.

HI @nectario, This issue is fixed in Tf v2.9. with Driver Version: 470.141.03  and  CUDA Version: 11.4 on Tesla T4. Could you test it on latest Tensorflow version and confirm the same. Thank you!


Tree-shacking with code splitting

For now the System-builder can:

Build fast (scope hoisting) & small (tree-shaking) modules with buildStatic & Rollup
Build less efficient System.register bundles enabling code splitting and lazy-loading

Webpack2 offers the best of both world, allowing code splitting while tree-shaking ES dependencies. Rollup also has plan to support code splitting (https://github.com/rollup/rollup/issues/372), as the current "externals" mechnism is very limited (single export per bundle).
Is there plans for the System-builder to offer a similar feature to Webpack2? My guess is that it could come with Rollup's support, but not sure of the details, especially as it looks more like an optimized version of bundle than the current behavior of buildStatic.
The reason I'm asking is that Angular 2 apps basically requires that: the whole NgModule architecture relies on tree-shaking to scale by eliminated the unused components of an NgModule (at least when using the AoT compiler), and the Router is centered around lazy-loading route bundles, which makes Webpack2 currently the only viable solution to use Angular 2 :/

There is certainly a middle ground between build and bundle, and better tooling in this space is needed.

This is the holy grail and a big missing link in all the different JS tooling system available. Vanilla System bundles support tree-shaking just as much as Webpack 2: an optimized bundle based off of specified entry points, but without scope hoisting. Without scope hoisting, Webpack 2 is still missing a lot of the performance benefits of Rollup (and is also the highest requested feature). Rollup, on the other hand, does not support multiple bundles and, thus prevents code splitting. This becomes painful on the larger end of the app spectrum (where dynamic code loading can lead to better performance).
It is unfortunate that, even without Rollup, System.js does not provide more optimizations/features around dynamically loaded bundles/modules. This would be its killer feature over Webpack.

Anyone feel like working on a new module compiler?

System bundles support tree-shaking just as much as Webpack 2: an optimized bundle based off of specified entry points, but without scope hoisting.

That's actually a good point. My main issue with using bundle isn't really scope hoisting actually, but the fact that you then have to load the bundles configuration, which in my case (> 100 bundles) weights ~500kB and takes almost a second to be parsed :/.
I could be wrong but it seems to me that Webpack2 solves that by rewriting the imports within the production bundles, removing the need for the bundles configuration to be loaded. You lose the point of the module registry, but I'd argue that in a production workflow you probably don't need it (or it should contain your production "bundles", not the individual source modules).

Wow 100 bundles seems like a lot. How many overall modules do you have?
From my limited experience, I found Webpack 2 surprisingly under-engineered for bundling and dynamic module loading. There do exist things like the Commons Chunk Plugin, but it also seems like it has some limitations. The way Webpack 2 rewrites System.import calls to magically infer bundles based on looking at the code itself is also extremely clunky coming from System.js.
I also agree that the system bundle config is incredibly verbose and a second to parse that seems unacceptable. It would be nice to be able to optimize that somehow with globs/wildcards.

Wow 100 bundles seems like a lot. How many overall modules do you have?

Around 1500 of our own, but well over 5000 including external libraries. It goes up pretty quickly as the JS community is increasingly adopting ES modules: for instance a small Angular 2 app with RxJS is >200 modules, only importing the .template() method from lodash-es is >50 modules.
The main reason we have so many bundles is because they are loaded on top of an old school server-side stack, so most pages will only load a couple of them, but the fact remains that the SystemJS bundles config doesn't scale well :/

The way Webpack 2 rewrites System.import calls to magically infer bundles based on looking at the code itself is also extremely clunky coming from System.js.

I think it depends where one see the future of JS bundling, or JS production workflow in general, especially now that the path for <script type="module"> and dynamic import() syntax is finally clear.
1/ Currently the System builder bundles mode is mostly about saving HTTP requests (through concatenation) & round trips (through the injected bundles config), and extra HTTP round trips can be saved for common bundles through depCache. But the basic assumption remains that the production tree/registry is the same than the source one, which is convenient for things like hot module replacement or dynamically loading any module.
One can imaging that long term, with ES6, HTTP/2.0, <script type="module"> and dynamic import() support, modules won't need to be bundled, and a "builder" output could be limited to generating modules dependency maps to avoid unnecessary network round trips. These maps won't scale either though, and handling them server side (e.g., h2 push + cache-digest) would require a custom assets server (expect if a standard comes up on that).
2/ The other scenario to me is that - unlike System builder buildStatic mode - a client-side Loader is used, however the production module tree is different that the source one. Imports are traced from user-specified entry points, and (optionally) through parsing dynamic import()s, to generate scope hoisted tree-shaken bundles: the Loader registry doesn't contain source modules anymore, which makes it impossible to load any random module on the fly, but the production bundles.
While it remains clunky, my understanding is that webpack2 enables / plans to enable that second workflow, as it allows for more optimized (smaller, faster to execute) code, at the cost of declaring all your possible dynamic import values at build time. Eventually they could even drop their custom client-side loader for the native import() (or a polyfill), but no matter what that scenario would require rewriting dynamic import() at build-time.
That scenario has drawbacks, e.g. heavily transformed code is harder to debug; loading source ES modules in dev would be nice, but to me the future of JS production workflow is likely to be more (advanced) transformations (unused object properties removal, binary intermediate format?) that less.

This is a good synopsis of the two primary directions things are headed. It doesn't feel like System is inherently in conflict with scenario 2, but simply needs a more efficient way to configure bundles. Requiring each module to be mapped to a bundle is pretty limiting– perhaps if dependencies between entire bundles could be declared this could be made much more compact.


FYPO:0003449

This comment doesn't look right?
ID 	FYPO:0003449
Ontology 	fission_yeast_phenotype
Term name 	abnormal cell cycle arrest at mitotic G1/S phase transition
Definition 	A cellular process phenotype in which progression through the mitotic cell cycle is arrested at the mitotic G1/S phase transition, under conditions where arrest is not a normal occurrence.
Comment 	Use this term if the cell cycle is abnormally arrested after completion of DNA replication (S phase) but before the G2/M transition.

copy/paste/fail-to-edit. will be easy to fix.

fixed
(spotted a howler of a typo from y'day so thought I might as well do the other quick fixes too)

grump. comitted edit file (edb509d097aea20072d5b32554403d7bc2505d31) but can't release because script connection to GO is timing out. will try tomorrow.

release working at last
7e9c22f11eb9b1e69f9ce8ea445140bebcec949b


Darc publishing is failing in official builds of main

I'm seeing the "sequence contains more than one element" error when publishing Razor builds again.
Razor build Pipelines - Run 20230302.6 logs (azure.com)
Maestro build promotion Pipelines - Run Promoting dotnet-razor build 20230302.6 (168991) to channel(s) '.NET Core Tooling Dev', 'VS 17.6' # logs (azure.com)
Teams thread from the last time an issue like this occurred: David Wengier: Is main insertable anywhere yet? Seems darc is complaining about not h…

We merged #8397 but it looks like official builds are still red. The Mac job is now what's failing instead of darc publishing.
https://dev.azure.com/dnceng/internal/_build/results?buildId=2128994&view=results
The Mac job issue occurred in the build from just before merging #8397 and it's not clear what introduced it.

The pipeline is once again green, and the arcade team confirmed this issue is resolved - https://github.com/dotnet/arcade/issues/12582


Issue in SSE with GoFiber when large data

https://github.com/valyala/fasthttp/issues/840

Fixed: no, the issue in meltcd was with Response Buffering
https://gist.github.com/KunalSin9h/4dce3a8ae17980a16eae0b9fc3c0e238?permalink_comment_id=5018258#gistcomment-5018258


General guidance: error finding address for actor

I've been using Dapr for a while now, and I enjoy it, for the most part. However, there is one error which makes my heart sink every time I see it:
error finding address for actor
Sometimes it's an easily understandable failure situation - other times it causes more of a headache. I find some relevant information on the subject, but I still have some existential angst over when it might pop up next.
One scenario where it popped up, was in a scenario where the Dapr service (for unrelated reasons) does setup in two stages, such that the port was opened for listening before /dapr/config was ready to receive calls. This resulted in the endpoint being called, 404 returned, and when, milliseconds later, the endpoint was ready, it was never called again.
Another situation (with a different service), with the actor service hosted as an Azure Container App (ACA), we saw an unexplained restart of the ACA. This, on its own, is OK - we can handle that. But, for 10-15 minutes afterwards, we see the "error finding address for actor" error, before it suddenly goes away. The call to /dapr/config returned 200, as expected. I assume, with the correct content.
Is there a way to detect and handle situations like this - where this actor related error occurs? Can Dapr be made to call /dapr/config again? This question is relevant for other scenarios for us, but mainly this one.
I feel this is an area of Dapr where I don't have the required control, and a reason for not recommending Dapr being used in other projects.

Hi @CRidge i think the two errors are possibly unrelated.

One scenario where it popped up, was in a scenario where the Dapr service (for unrelated reasons) does setup in two stages, such that the port was opened for listening before /dapr/config was ready to receive calls. This resulted in the endpoint being called, 404 returned, and when, milliseconds later, the endpoint was ready, it was never called again.

If the server is listening, Dapr will try to invoke that. A 404 is considered a permanent (non-retriable) error in HTTP, so Dapr does not retry invoking the method if the call fails. This I think is the correct behavior, as Dapr did get a response from the server here.
What's the reason for making the server start without offering /dapr/config? Is that a behavior you can change in your app?

Another situation (with a different service), with the actor service hosted as an Azure Container App (ACA), we saw an unexplained restart of the ACA. This, on its own, is OK - we can handle that. But, for 10-15 minutes afterwards, we see the "error finding address for actor" error, before it suddenly goes away. The call to /dapr/config returned 200, as expected. I assume, with the correct content.

Apps on ACA can be restarted at any time and this is normal. However, I am curious about the error "error finding address for actor". Do you have more logs that you can share? Was that error transient, appearing just once, or was it recurring?


Invalid TRAPI: logs should be a list of objects

For reference: https://github.com/NCATSTranslator/ReasonerAPI/blob/master/TranslatorReasonerAPI.yaml#L551
This is in the dev environment.
I'm seeing responses that have logs like this:
"logs": [
    "no workflow specified",
    "single hop query requested, supported",
    "query is LOOKUP",
    "Database version: 20240605-MCQa",
    "Code version: 20240611-MCQa",
    "LOOKUP web query with source: 0 and target: 1 return total edge: 4675 in time: 0.803574800491333s"
  ],

Any updates on this? I'm seeing the same issues in TEST, so I assume CI also has this, and maybe PROD, but I haven't checked those.

I just noticed that the result.analyses.edge_bindings are also incorrectly formatted. Each edge binding needs to be a list of dicts that each contain an id and attributes, but you have a single dict.
What you're returning:
{
  "analyses": [
    {
      "attributes": [],
      "edge_bindings": {
        "edge_0": {
          "attributes": [],
          "id": "uuid:1--biolink:genetic_association--NCBIGene:57084"
        }
      },
      "resource_id": "infores:genetics-data-provider",
      "score": 6.788339474806989,
      "scoring_method": "probability",
      "support_graphs": []
    }
  ]
}

What should be returned:
{
  "analyses": [
    {
      "attributes": [],
      "edge_bindings": {
        "edge_0": [
          {
            "attributes": [],
            "id": "uuid:1--biolink:genetic_association--NCBIGene:57084"
          }
        ]
      },
      "resource_id": "infores:genetics-data-provider",
      "score": 6.788339474806989,
      "scoring_method": "probability",
      "support_graphs": []
    }
  ]
}

Thanks @marcdubybroad! Confirmed that genetics-data-provider results are successfully being returned.


Cannot add tags to existing deployment bucket

Version: 1.5.1
Issue:
For existing deployment bucket, after configuring "tags" property with new tags, it is not possible to add new tags. This seem to only work when bucket is either being created or if the bucket has at least one tag.
Steps to reproduce:
Step 1:
Configure vanilla serverless.yml with serverless-deployment-bucket. Bucket configuration in custom section:
  deploymentBucket:
    enabled: true

Then deploy (no tags).
Step 2:
Change deploymentBucket to following:
  deploymentBucket:
    enabled: true
    tags:
      - Key: Environment
        Value: production

Then deploy - observe that bucket tags are empty.
Preliminary issue notes:
Best I can surmise is that issue itself comes from DeploymentBucketPlugin.hasChangedBucketTags in index.json where
const response = await this.provider.request('S3', 'getBucketTagging', { Bucket: name })
throws an exception:
{"code":"AWS_S3_GET_BUCKET_TAGGING_NO_SUCH_TAG_SET","providerError":{"message":"The TagSet does not exist","code":"NoSuchTagSet","region":null,"time":"2021-06-28T17:46:11.101Z","requestId":".....","extendedRequestId":".......","statusCode":404,"retryable":false,"retryDelay":94.99932161936133},"providerErrorCodeExtension":"NO_SUCH_TAG_SET"}
And has DeploymentBucketPlugin.hasChangedBucketTags returns false. This in turn causes applyDeploymentBucket  to not apply new tags

I can attest to this being the case. In fact I found a way through testing to get the tags to apply.

Add tags with Environment being first in the list.
Add Environment tag to the bucket manually with a matching value.
Add another random tag not already in your list with random value
sls deploy

All your tags will be created and will retain through deployments if no changes are made to the tags

+1 ran into the same issue.  Any update on a fix for this?

I just put up a PR to fix this issue; I've been hitting it as well and that change should fix it

@ROSeaboyer Thanks, merged and released in v1.5.2!


Resolve typescript warnings for Rancher Components

Summary
This resolves typescript warnings for Rancher Components.
Fixes #11977

Occurred changes and/or fixed issues

Update pkg/rancher-components/src/shim-vue.d.ts
Remove instances of @ts-nocheck
Resolve each error flagged by tsc

Checklist

[x] The PR is linked to an issue and the linked issue has a Milestone, or no issue is needed
[x] The PR has a Milestone
[x] The PR template has been filled out
[x] The PR has been self reviewed
[x] The PR has a reviewer assigned
[x] The PR has automated tests or clear instructions for manual tests and the linked issue has appropriate QA labels, or tests are not needed
[x] The PR has reviewed with UX and tested in light and dark mode, or there are no UX changes

@cnotv this is ready for another review - we finally pass CI 🙃

@cnotv this is ready for another review - we finally pass CI 🙃

I was taking my time anyway 😅


Ensure that every item is in some module's children list

Previously, if an item's parent was not a module (eg a nested class definition whose parent it the outer class definition) and the parent was not whitelisted but the item was transitively whitelisted, then we could generate uses of the item without emitting any definition for it. This could happen because we were relying on the outer type calling for code generation on its inner types, but that relies on us doing code generation for the outer type, which won't happen if the outer type is not whitelisted.
This commit avoids this gotcha by ensuring that all items end up in a module's children list, and so will be code generated even if their parent is not whitelisted.
This does have the downside of changing the relative order of some of the emitted code, and so this has a big diff (as will the next bindgen update for downstream dependencies) but I actually think the newer order makes more sense, for what that is worth.
Fixes #769
r? @emilio

 Warning

These commits modify unsafe code. Please review it carefully!

Looks like this uncovered a bug with static variables whose type is a template type parameter. Coming up with a fix for that too.

Huh, now I'm getting tripped up by the recent naming scheme change for the layout tests for template instantiations. It appears it isn't as unique as we'd hoped.

Going to use the overloaded function counter instead.

This seems to make the order of the generated bindings not match across libclang versions though, and thus CI is red.

Thanks for review! (Sorry it's such a huge diff)

This seems to make the order of the generated bindings not match across libclang versions though, and thus CI is red.

I'll make the children an ItemSet instead of a Vec. This has two advantages:

An ItemSet is ordered, so we should get deterministic iteration of a module's children (and therefore emit them in that deterministic order during codegen)

It disallows duplicate entries (which should be harmless, but its so easy to prevent that it seems crazy not to prevent it).

I expect CI will still be red, since I need to get the diffs for libclang 3.8 and 3.9

This might need some more updates for other libclang versions, but I'm not 100%.
In any case, it probably deserves a second glance, since a bunch of code changed from the version you reviewed before, @emilio.
r? @emilio

Ok, tests for other libclang versions should be good to go.

@bors-servo r+

:pushpin: Commit 5230565 has been approved by emilio

:hourglass: Testing commit 52305657b6eea5d9a0dea17f72f8865281ebbccc with merge 232a21ebc4a4ddc2468aed8e54d165c613037038...

Thanks!

:sunny: Test successful - status-travis
Approved by: emilio
Pushing 232a21ebc4a4ddc2468aed8e54d165c613037038 to master...


[BUG] Tuya Air Quality detector exposes only VOC level sensor

Hi, the Tuya air quality detector located here  zhaquirks/tuya/air/ts0601_air_quality.py  doesn't expose all supported sensors.
See attached screenshot.

{
  "node_descriptor": "NodeDescriptor(logical_type=<LogicalType.Router: 1>, complex_descriptor_available=0, user_descriptor_available=0, reserved=0, aps_flags=0, frequency_band=<FrequencyBand.Freq2400MHz: 8>, mac_capability_flags=<MACCapabilityFlags.AllocateAddress|RxOnWhenIdle|MainsPowered|FullFunctionDevice: 142>, manufacturer_code=4417, maximum_buffer_size=66, maximum_incoming_transfer_size=66, server_mask=10752, maximum_outgoing_transfer_size=66, descriptor_capability_field=<DescriptorCapability.NONE: 0>, *allocate_address=True, *is_alternate_pan_coordinator=False, *is_coordinator=False, *is_end_device=False, *is_full_function_device=True, *is_mains_powered=True, *is_receiver_on_when_idle=True, *is_router=True, *is_security_capable=False)",
  "endpoints": {
    "1": {
      "profile_id": 260,
      "device_type": "0x0100",
      "in_clusters": [
        "0x0000",
        "0x0004",
        "0x0005",
        "0x0402",
        "0x0405",
        "0x040d",
        "0x042b",
        "0x042e",
        "0xef00"
      ],
      "out_clusters": [
        "0x000a",
        "0x0019"
      ]
    },
    "242": {
      "profile_id": 41440,
      "device_type": "0x0061",
      "in_clusters": [],
      "out_clusters": [
        "0x0021"
      ]
    }
  },
  "manufacturer": "_TZE200_yvx5lh6k",
  "model": "TS0601",
  "class": "zhaquirks.tuya.air.ts0601_air_quality.TuyaCO2SensorGPP"
}

It seems like the problem was discussed here https://github.com/zigpy/zha-device-handlers/issues/1004#issuecomment-1002206182

Your device (already supported) is getting the correct quirks:
https://github.com/zigpy/zha-device-handlers/blob/23a166902e50d5c32019e0d4d32a968715041bb6/zhaquirks/tuya/air/ts0601_air_quality.py#L74-L86
Is your device reporting values?
Do you have any error in logs?
Can you enable and attach the debug logs for:
  zigpy: debug
  zigpy.zcl: debug
  custom_zha_quirks: debug
  zhaquirks.tuya: debug
  homeassistant.components.zha: debug

Regards.

@javicalle I know it is formally supported but I have this problem

home-assistant.log

@javicalle  I noticed that if you put the json node below  under  TuyaCO2Sensor.ENDPOINTS I am able to see all the sensors.
        242: {
              PROFILE_ID: 41440,
              DEVICE_TYPE: 0x0061,
              INPUT_CLUSTERS: [ ],
              OUTPUT_CLUSTERS: [33],
        }

The guy here is saying the same thing https://github.com/zigpy/zha-device-handlers/issues/1004#issuecomment-1003404368

Another workaround is to remove ("_TZE200_yvx5lh6k", "TS0601"),  from the TuyaCO2Sensor's signature MODELS_INFO node.

@javicalle   I think it is a big mistake to support a device by 2 different quirks.

Im sorry i was adding the GPP version and was not knowing if the same device ID is being used without GPP that was already in the quirk so i was leaving it then it cant being triggered if the device is having GPP endpoint and for not braking devices for user that  is having devices without it.
I have testing TS004F and if having more matching quirks the system is using the first class i the quirk (from start for the file).
But if putting the same device ID is 2 different quirk is being one other story that i thing its a bad one.

It is something here that I can't understand.
From the capture at the firts comment, the device is loading the TuyaCO2SensorGPP quirk.
But it seems that the problem is that device is using the TuyaCO2Sensor quirk. Then, the solution comes from adding the GreenPowerProxy cluster  or removing _TZE200_yvx5lh6k from MODELS_INFO, both in TuyaCO2Sensor quirk.
There's something weird here. I understand that the first quirk that matches will be loaded, but I don't understand why it indicates that it is loading the another one.

@javicalle  I made a pull request for another fix previously. I don't know why new commits to my fork are added to this pull request.

It is something here that I can't understand.
From the capture at the firts comment, the device is loading the TuyaCO2SensorGPP quirk. But it seems that the problem is that device is using the TuyaCO2Sensor quirk. Then, the solution comes from adding the GreenPowerProxy cluster or removing _TZE200_yvx5lh6k from MODELS_INFO, both in TuyaCO2Sensor quirk.
There's something weird here. I understand that the first quirk that matches will be loaded, but I don't understand why it indicates that it is loading the another one.

It is something here that I can't understand.
From the capture at the firts comment, the device is loading the TuyaCO2SensorGPP quirk. But it seems that the problem is that device is using the TuyaCO2Sensor quirk. Then, the solution comes from adding the GreenPowerProxy cluster or removing _TZE200_yvx5lh6k from MODELS_INFO, both in TuyaCO2Sensor quirk.
There's something weird here. I understand that the first quirk that matches will be loaded, but I don't understand why it indicates that it is loading the another one.

It is very strange indeed. I think this problem comes for the fact that more than 1 class is handling the same device. This is absolutely wrong and causes issues.

Once PR #1346 is merged, this issue can be closed.
Fix (probably) will come in the next HA version.

Has this regressed again? I also only see a single sensor on my _TZE200_yvx5lh6k and a few other people seem to have the same issue in https://github.com/zigpy/zha-device-handlers/pull/1410 and https://github.com/zigpy/zha-device-handlers/issues/2395
Reconfiguration sees all, but only one entity is being created.

Same here.

Seems to work fine again on 2023.12.3, detects 5 entities

_TZE200_c2fmom5z only detects VOC in 2024.3.0, havent tested with older versions as I just got the sensor today


[Bug]: CORS error every time

What happened?
So I created Vercel thing, connected to my worker and to backend.undi.rest (tests in Admin panel are OK). But every time I start any video it fails and console outputs:
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://api.whvx.net/status. (Reason: CORS header ‘Access-Control-Allow-Origin’ missing). Status code: 403.

and
Access to fetch at 'https://api.whvx.net/status' from origin 'https://*.vercel.app' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.

and so on for all the sites. What should I do?
What browsers are you seeing the problem on?
No response
Steps to reproduce?
No response
Other relevant information
No response

NSBX and WHVX apis are closed source


[Opaque pointers regression] CTS vec_align_packed_struct

When The following kernel:
typedef struct __attribute__ ((packed)) myPackedStruct {
    char8 vec;
} testStruct;
__kernel void test_vec_align_packed_struct(__constant char8 *source, __global uint *dest)
{
    __local testStruct test;
    int  tid = get_global_id(0);
    dest[tid] = (uint)((__local uchar *)&(test.vec) - (__local uchar *)&test);
}

compiled with the following options:
clspv source.cl -print-before-all   -cl-std=CL3.0 -cl-single-precision-constant  -cl-kernel-arg-info   -fp16=0  -rewrite-packed-structs  -std430-ubo-layout  -decorate-nonuniform    -arch=spir  -spv-version=1.6  -max-pushconstant-size=256  -max-ubo-size=65536  -global-offset  -long-vector  -module-constants-in-storage-buffer  -cl-arm-non-uniform-work-group-size  -enable-feature-macros=__opencl_c_images,__opencl_c_read_write_images,__opencl_c_3d_image_writes,__opencl_c_atomic_order_acq_rel,__opencl_c_atomic_scope_device,__opencl_c_subgroups,__opencl_c_int64,__opencl_c_fp64    -o compiled.spv

triggers the following assertion in the ThreeElementVectorLoweringPass:
While deleting: ptr addrspace(3) %
Use still stuck around after Def is destroyed:  %5 = ptrtoint ptr addrspace(3) <badref> to i32
Use still stuck around after Def is destroyed:  %4 = ptrtoint ptr addrspace(3) <badref> to i32
clspv: /mnt/work/repos/clvk/external/clspv/third_party/llvm/llvm/lib/IR/Value.cpp:102: llvm::Value::~Value(): Assertion `materialized_use_empty() && "Uses remain when a value is destroyed!"' failed.
Abandon

This may or may not be the same bug as https://github.com/google/clspv/issues/1045 and/or https://github.com/google/clspv/issues/1046.


Remove Dev Diagnostics Experimental Feature

Summary of the pull request
References and relevant issues
Detailed description of the pull request / Additional comments
Validation steps performed
PR checklist

[ ] Closes #xxx
[ ] Tests added/passed
[ ] Documentation updated

  </Extensions>

This should also go as well. It was a part of Dev Diagnostics

Refers to: src/Package-Dev.appxmanifest:399 in 0c4e347. [](commit_id = 0c4e347a99223bdd3e28429d81c83f11cae89de7, deletion_comment = False)

<ProjectReference Include="..\tools\Utilities\RegPreviewUtility\DevHome.RegistryPreview.csproj" />

this was a piece of devdiagnostics not utilties

Refers to: src/DevHome.csproj:108 in 0c4e347. [](commit_id = 0c4e347a99223bdd3e28429d81c83f11cae89de7, deletion_comment = False)

<ProjectReference Include="..\tools\Utilities\RegPreviewUtility\DevHome.RegistryPreview.csproj" />

IfeoTool?

In reply to:<PHONE_NUMBER>

Refers to: src/DevHome.csproj:108 in 0c4e347. [](commit_id = 0c4e347a99223bdd3e28429d81c83f11cae89de7, deletion_comment = False)

<ProjectReference Include="..\tools\Utilities\RegPreviewUtility\DevHome.RegistryPreview.csproj" />

yes ifeotool

In reply to:<PHONE_NUMBER>

Refers to: src/DevHome.csproj:108 in 0c4e347. [](commit_id = 0c4e347a99223bdd3e28429d81c83f11cae89de7, deletion_comment = False)

<ProjectReference Include="..\service\DevHome.Service\DevHome.Service.csproj" />

The NT service should go as well, it was only used by devdiagnostics

Refers to: src/DevHome.csproj:87 in 0c4e347. [](commit_id = 0c4e347a99223bdd3e28429d81c83f11cae89de7, deletion_comment = False)


No such file or directory

Hello,
I keep getting an error when running python3 answer_bot.py
I am positive I did everything correct and that it is cd'ed
Screenshot:

Hi,
Make sure your in the correct directory.
Type ls to list the contents of the directory your in or type pwd to check directory.
Make sure the repo is cloned

@sushant10 Yes, I am in the correct directory and my directory is fine. Yeah, the repo is fine.

Is answer_bot.py in the directory?

Yes @sushant10

I dont know why it wouldnt work if the anwer_bot.py is in the correct directory. Can you show a screenshot of what shows when ls or pwd is typed?

Hi,
the directory works now but now when I run it I am getting errors with the module, so I tried installing requirments.txt again and get this error:

You have any idea?

Sorry for that I spelt the package wrong change it from buetifulsoup4 to  beautifulsoup4

i too have same problem

Do this
pip3 install --upgrade setuptools
If it's already up to date, check that the module ez_setup is not missing. If it is, then
pip3 install ez_setup
Then try again


:images followed by "Ansible version and collections" does not work with an ansible 2.9 based EE

ANSIBLE-AUTOMATION-PLATFORM-21-EE-29-RHEL8:LATEST (PRIMARY) (INFORMATION ABOUT ANSIBLE AND ANSIBLE COLLECTIONS)
 0│---
 1│ansible:
 2│  collections:
 3│    details: {}
 4│    errors:
 5│    - |-
 6│      usage: ansible-galaxy collection [-h] COLLECTION_ACTION ...
 7│      ansible-galaxy collection: error: argument COLLECTION_ACTION: invalid choice: 'list' (choose from 'init', 'build', 'publish', 'install')
 8│  version:
 9│    details: .9.2

There's not really much to be done here other than maybe improve the message to say "try :collections, this won't work with 2.9

Note: The version is incorrectly parsed as well


Current configuration will not reload as not all conditions are met,please refer to documentation

I received a warning that says: Current configuration will not reload as not all conditions are met,please refer to documentation.
Which documentation is this referring to? what do I need to meet all the conditions?

I have never encountered this problem. Are you using debug mode of HibiAPI under Windows? If so, this issue may be caused by limitation of operating system.

No I have it installed in Ubuntu 20.04 and debug mode is disable

I have already located where the problem is. This warning will not affect normal production use.
I will fix it soon, so please leave this issue open.