Masking LSTM: OP_REQUIRES failed at cudnn_rnn_ops.cc:1498 : Unknown: CUDNN_STATUS_BAD_PARAM

System information

Have I written custom code: Yes
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 18.04.2 LTS
TensorFlow installed from (source or binary): Binary, pip
TensorFlow version (use command below): v2.0.0-rc2-26-g64c3d38 2.0.0
Python version: Python 3.7.3
CUDA/cuDNN version: CUDA=10.0, CUDNN=<IP_ADDRESS>-1
GPU model and memory: Quadro RTX 6000 major: 7 minor: 5 memoryClockRate(GHz): 1.77

Describe the problem
It seems there is an issue with the CuDNN LSTM implementation when using a tf.keras.layers.Masking layer.
batch_size = 256
num_tsteps = 144
num_features = 130
num_units = 88
model = tf.keras.Sequential([
    tf.keras.layers.InputLayer(input_shape=(num_tsteps, num_features), batch_size=batch_size),
    tf.keras.layers.Masking(mask_value=0.0, input_shape=(num_tsteps, num_features)),
    tf.keras.layers.LSTM(num_units,  batch_input_shape=(batch_size, num_tsteps, num_features), return_sequences=True, stateful=False),
    tf.keras.layers.TimeDistributed(tf.keras.layers.Dense(1)),
    tf.keras.layers.Activation('sigmoid'),
])

Similar to #33069 I receive this error during training and I have strictly right-padded data (I am doing trimming and right-padding manually). However, in contrast to this issue, I confirmed that I do not have any inputs containing only zeroes via the following snippet:
for i, e in enumerate(ds_train):
    res = []
    f, l = [x.numpy() for x in e]
    for j in range(f.shape[0]):
        if not (f[j] == 0.0).all():
            res.append(1)
        else:
            res.append(0)
    fin = [res[0]]
    for e in res[1:]:
        if e != fin[-1]:
            fin.append(e)
    print("i {}: {}".format(i, fin))

# Result:
i 0: [1, 0]
i 1: [1, 0]
i 2: [1, 0]
i 3: [1, 0]
i 4: [1]
i 5: [1, 0]
...

If I remove the Masking-layer, the error does not occur. I confirmed this by running a complete epoch (2324 batches), however, the training is probably pretty pointless when including the padded data.
Is there any other pitfall that I am missing that could cause this issue?
Source code / logs
Python output:
Epoch 1/1000
WARNING:tensorflow:Early stopping conditioned on metric `val_loss` which is not available. Available metrics are:

CancelledErrorTraceback (most recent call last)
<ipython-input-7-1c503c2dd55c> in <module>
----> 1 m.fit(train=True)

/ws/tf/vol_local/_model_lstm.py in fit(self, train, verbose)
    315             ]
    316             self.model.fit(ds_train, epochs=num_epochs, verbose=verbose, shuffle=False,
--> 317                                 validation_data=ds_val, validation_steps=None, callbacks=cbs)
    318             #self.model.save(sess_hdf5_path)
    319             self.model.save_weights(self.sess_h5_path.as_posix())

/ws/miniconda3/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_freq, max_queue_size, workers, use_multiprocessing, **kwargs)
    726         max_queue_size=max_queue_size,
    727         workers=workers,
--> 728         use_multiprocessing=use_multiprocessing)
    729
    730   def evaluate(self,

/ws/miniconda3/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_v2.py in fit(self, model, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_freq, **kwargs)
    322                 mode=ModeKeys.TRAIN,
    323                 training_context=training_context,
--> 324                 total_epochs=epochs)
    325             cbks.make_logs(model, epoch_logs, training_result, ModeKeys.TRAIN)
    326

/ws/miniconda3/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_v2.py in run_one_epoch(model, iterator, execution_function, dataset_size, batch_size, strategy, steps_per_epoch, num_samples, mode, training_context, total_epochs)
    121         step=step, mode=mode, size=current_batch_size) as batch_logs:
    122       try:
--> 123         batch_outs = execution_function(iterator)
    124       except (StopIteration, errors.OutOfRangeError):
    125         # TODO(kaftan): File bug about tf function and errors.OutOfRangeError?

/ws/miniconda3/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_v2_utils.py in execution_function(input_fn)
     84     # `numpy` translates Tensors to values in Eager mode.
     85     return nest.map_structure(_non_none_constant_value,
---> 86                               distributed_function(input_fn))
     87
     88   return execution_function

/ws/miniconda3/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py in __call__(self, *args, **kwds)
    455
    456     tracing_count = self._get_tracing_count()
--> 457     result = self._call(*args, **kwds)
    458     if tracing_count == self._get_tracing_count():
    459       self._call_counter.called_without_tracing()

/ws/miniconda3/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py in _call(self, *args, **kwds)
    518         # Lifting succeeded, so variables are initialized and we can run the
    519         # stateless function.
--> 520         return self._stateless_fn(*args, **kwds)
    521     else:
    522       canon_args, canon_kwds = \

/ws/miniconda3/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py in __call__(self, *args, **kwargs)
   1821     """Calls a graph function specialized to the inputs."""
   1822     graph_function, args, kwargs = self._maybe_define_function(args, kwargs)
-> 1823     return graph_function._filtered_call(args, kwargs)  # pylint: disable=protected-access
   1824
   1825   @property

/ws/miniconda3/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py in _filtered_call(self, args, kwargs)
   1139          if isinstance(t, (ops.Tensor,
   1140                            resource_variable_ops.BaseResourceVariable))),
-> 1141         self.captured_inputs)
   1142
   1143   def _call_flat(self, args, captured_inputs, cancellation_manager=None):

/ws/miniconda3/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py in _call_flat(self, args, captured_inputs, cancellation_manager)
   1222     if executing_eagerly:
   1223       flat_outputs = forward_function.call(
-> 1224           ctx, args, cancellation_manager=cancellation_manager)
   1225     else:
   1226       gradient_name = self._delayed_rewrite_functions.register()

/ws/miniconda3/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py in call(self, ctx, args, cancellation_manager)
    509               inputs=args,
    510               attrs=("executor_type", executor_type, "config_proto", config),
--> 511               ctx=ctx)
    512         else:
    513           outputs = execute.execute_with_cancellation(

/ws/miniconda3/lib/python3.7/site-packages/tensorflow_core/python/eager/execute.py in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name)
     65     else:
     66       message = e.message
---> 67     six.raise_from(core._status_to_exception(e.code, message), None)
     68   except TypeError as e:
     69     keras_symbolic_tensors = [

/ws/miniconda3/lib/python3.7/site-packages/six.py in raise_from(value, from_value)

CancelledError:  [_Derived_]RecvAsync is cancelled.
	 [[{{node metrics/accuracy/broadcast_weights/assert_broadcastable/AssertGuard/else/_36/Assert/data_2/_62}}]]
	 [[loss/activation_loss/weighted_loss/broadcast_weights/assert_broadcastable/is_valid_shape/else/_1/has_valid_nonscalar_shape/then/_106/has_invalid_dims/concat/_28]] [Op:__inference_distributed_function_172102]

Function call stack:
distributed_function

Command line log:
2019-10-08 14:38:27.367875: W tensorflow/core/grappler/optimizers/implementation_selector.cc:310] Skipping optimization due to error while loading function libraries: Invalid argument: Functions '__inference___backward_cudnn_lstm_with_fallback_169668_171093' and '__inference___backward_cudnn_lstm_with_fallback_169668_171093_specialized_for_StatefulPartitionedCall_at___inference_distributed_function_172102' both implement 'lstm_dce676f4-acdd-4bb5-88d9-e8dd57573aba' but their signatures do not match.
2019-10-08 14:38:27.536666: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2019-10-08 14:38:39.982582: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2019-10-08 14:38:41.215567: W tensorflow/core/framework/op_kernel.cc:1622] OP_REQUIRES failed at cudnn_rnn_ops.cc:1498 : Unknown: CUDNN_STATUS_BAD_PARAM
in tensorflow/stream_executor/cuda/cuda_dnn.cc(1424): 'cudnnSetRNNDataDescriptor( data_desc.get(), data_type, layout, max_seq_length, batch_size, data_size, seq_lengths_array, (void*)&padding_fill)'
2019-10-08 14:38:41.215616: W tensorflow/core/common_runtime/base_collective_executor.cc:216] BaseCollectiveExecutor::StartAbort Unknown: CUDNN_STATUS_BAD_PARAM
in tensorflow/stream_executor/cuda/cuda_dnn.cc(1424): 'cudnnSetRNNDataDescriptor( data_desc.get(), data_type, layout, max_seq_length, batch_size, data_size, seq_lengths_array, (void*)&padding_fill)'
	 [[{{node cond_64/then/_0/CudnnRNNV3}}]]
2019-10-08 14:38:41.215638: W tensorflow/core/common_runtime/base_collective_executor.cc:216] BaseCollectiveExecutor::StartAbort Cancelled: [_Derived_]RecvAsync is cancelled.
	 [[{{node metrics/accuracy/broadcast_weights/assert_broadcastable/AssertGuard/else/_36/Assert/data_2/_62}}]]
	 [[loss/activation_loss/weighted_loss/broadcast_weights/assert_broadcastable/is_valid_shape/else/_1/has_valid_nonscalar_shape/then/_106/has_invalid_dims/concat/_28]]
2019-10-08 14:38:41.215693: W tensorflow/core/common_runtime/base_collective_executor.cc:216] BaseCollectiveExecutor::StartAbort Cancelled: [_Derived_]RecvAsync is cancelled.
	 [[{{node metrics/accuracy/broadcast_weights/assert_broadcastable/AssertGuard/else/_36/Assert/data_2/_62}}]]

@mimxrt ,
Thanks for reporting the issue, can you please provide simple and standalone code to reproduce the issue?

Issue replicating with TF-2.0, kindly find the gist of colab.Thanks!

I found my mistake and updated the code accordingly. Please update the gist to represent the corrected and less verbose version below. The error happens in TF-2.0 and does not happen in TF-1.0 (1.14.0). Please also try to removing the Masking layer to confirm the issue only exists with masking.
import numpy as np
import tensorflow as tf

# NOTE: Comment the block below for testing with TF-1.0
gpus = tf.config.experimental.list_physical_devices('GPU')
for gpu in gpus:
    tf.config.experimental.set_memory_growth(gpu, True)

# NOTE: Uncomment the block below for testing with TF-1.0
# tf.compat.v1.enable_eager_execution()
# config = tf.compat.v1.ConfigProto()
# config.gpu_options.allow_growth = True
# sess = tf.compat.v1.Session(config=config)
# tf.compat.v1.keras.backend.set_session(sess)

assert tf.executing_eagerly()

batch_size = 256
num_tsteps = 144
num_features = 130
num_units = 88

#n_files = 3320
n_files = 10
num_epochs = 1000

seq_len_max_trunc = batch_size * num_tsteps
flen = 3728

### Create TFRecord

X = np.random.rand(flen + 1, num_features + 2)
n_label0 = int((flen + 1) * 0.2)
Y = np.concatenate((
    np.zeros((n_label0, 1)), # label 0
    np.ones((flen - n_label0 + 1, 1)), # label 1
), axis=0)
ds_out = tf.data.Dataset.from_tensor_slices((X, Y))
ds_ser = ds_out.map(lambda *x:
   tf.reshape(tf.py_function(lambda *v:
       tf.train.Example(features=tf.train.Features(feature={
           "features": tf.train.Feature(float_list=tf.train.FloatList(value=v[0].numpy())),
           "label": tf.train.Feature(float_list=tf.train.FloatList(value=v[1].numpy())),
       })).SerializeToString(), x, tf.string
   ), ()), num_parallel_calls=tf.data.experimental.AUTOTUNE
)
writer = tf.data.experimental.TFRecordWriter("temp.tfrecord")
writer.write(ds_ser)

### Read TFRecord and train

files = ["temp.tfrecord"] * n_files

model = tf.keras.Sequential([
    tf.keras.layers.InputLayer(input_shape=(num_tsteps, num_features), batch_size=batch_size),
    tf.keras.layers.Masking(mask_value=0.0, input_shape=(num_tsteps, num_features)),
    tf.keras.layers.LSTM(num_units,  batch_input_shape=(batch_size, num_tsteps, num_features), return_sequences=True, stateful=False),
    tf.keras.layers.TimeDistributed(tf.keras.layers.Dense(1)),
    tf.keras.layers.Activation('sigmoid'),
])
model.compile(optimizer='adam',
              loss='binary_crossentropy',
              metrics=['accuracy'])

def _prep_ds_file(file):
    _ds = tf.data.TFRecordDataset(file)
    _ds = _ds.map(lambda x: tf.io.parse_single_example(x, {
        "features": tf.io.FixedLenFeature([132], tf.float32),
        "label": tf.io.FixedLenFeature([1], tf.float32),
    }), num_parallel_calls=tf.data.experimental.AUTOTUNE)

    _ds = _ds.flat_map(lambda v: tf.data.Dataset.from_tensors((v["features"][2:], v["label"])))

    _trunc = min(seq_len_max_trunc, ((flen + 1) // num_tsteps) * num_tsteps)
    _ds = _ds.take(_trunc)

    _c_pad = (batch_size - ((flen + 1) // num_tsteps)) * num_tsteps
    if _c_pad >= 0:
        assert _c_pad + ((flen + 1) // num_tsteps * num_tsteps) == seq_len_max_trunc
        _ds_pad = tf.data.Dataset.from_tensors((
            tf.constant(0.0, shape=[num_features,]),
            tf.constant(0.0, shape=[1,])))
        _ds_pad = _ds_pad.repeat(_c_pad)
        _ds = _ds.concatenate(_ds_pad) # pad to correct size

    _ds = _ds.window(size=num_tsteps, shift=None, stride=1, drop_remainder=True)
    _ds = _ds.flat_map(lambda x, y: tf.data.Dataset.zip((x.batch(num_tsteps), y.batch(num_tsteps))))

    _ds = _ds.batch(batch_size, drop_remainder=True)

    return _ds

ds_fs = tf.data.Dataset.list_files(files, shuffle=True, seed=1)
fs_train = ds_fs.take(int(n_files * 0.7))
fs_val = ds_fs.skip(int(n_files * 0.7)).take(int(n_files * 0.1))

ds_train = [_prep_ds_file(f) for f in fs_train.take(1)][0]
for f in fs_train.skip(1):
    ds_train = ds_train.concatenate(_prep_ds_file(f))
ds_train = ds_train.prefetch(buffer_size=tf.data.experimental.AUTOTUNE)

cbs = [
    tf.keras.callbacks.EarlyStopping(monitor="val_loss", patience=10, restore_best_weights=True),
]
model.fit(ds_train, epochs=num_epochs, verbose=1, shuffle=False,
          validation_data=None, validation_steps=None, callbacks=cbs)

@mimxrt I could reproduce the issue with tf-nightly-gpu. However, if I use only cpu then there is no error (with and without masking) as shown in the original post. I think root-cause may be due to tf.config.experimental.set_memory_growth(gpu, True). Thanks!

Thank you for your reply. I'm not sure what you are saying though: did you try removing tf.config.experimental.set_memory_growth(gpu, True) at all? I tried and for me the same error happens. For me, the only difference is that TF occupies all available GPU memory.
Could you please try again if the error also happens without memory growth? If so, I guess the issue could be in the GPU LSTM implementation (cuDNN)?

@mimxrt I mentioned two things.

I could reproduce the issue with tf-nightly-gpu
Just tried to point some root-cause. I ran your code as it is in a cpu and I don't see any error. so I guess root-cause may be related to tf.config.experimental.set_memory_growth(gpu, True). I may be wrong.

Thanks!

I see, thank you for the clarification. So in summary we can say that this issue does not happen on the CPU and that it is related to the GPU implementation. As I tried running it without tf.config.experimental.set_memory_growth(gpu, True) and the error still occurs, it should probably be save to rule that out as the cause/issue.

same problem here. did you find any solution?

No sorry, still waiting for any help myself.

I changed my model from:
       self.model = Sequential([             Embedding(len(self.item_map), self.embed_dim, input_length = X.shape[1],mask_zeros=True),             LSTM(self.lstm_out),             Dense(len(self.item_map)-1),         ])
to:
       self.model = Sequential([             Embedding(len(self.item_map), self.embed_dim, input_length = X.shape[1]),             Masking(mask_value=0),             LSTM(self.lstm_out),             Dense(len(self.item_map)-1),         ])
And solved my isssue
I know @mimxrt's code has the same model and I dont know why it works for me,
but im adding this for anyone else comes here with the issue and maybe it can help with debugging

@ynsgnr if you are running on CPU then it works properly, the problem is when you run it on GPU. By the way, you do not use the Timedistirbuted layer in your code since this problem shows up when you use Timedistributed with LSTM.

@jvishnuvardhan, I tried your suggestion but it still produces the same error. I am new to GitHub , how to tell this problem to someone from TensorFlow? do they see our conversation ?

In my experience and example the issue stems from the Masking + LSTM combination, not from the TimeDistributed layer. Please try the example in https://github.com/tensorflow/tensorflow/issues/33148#issuecomment-540472342 without it, i.e.
model = tf.keras.Sequential([
    tf.keras.layers.InputLayer(input_shape=(num_tsteps, num_features), batch_size=batch_size),
    tf.keras.layers.Masking(mask_value=0.0, input_shape=(num_tsteps, num_features)),
    tf.keras.layers.LSTM(num_units,  batch_input_shape=(batch_size, num_tsteps, num_features), return_sequences=False, stateful=False),
    tf.keras.layers.Dense(1),
    tf.keras.layers.Activation('sigmoid'),
])

(change return_sequences to False and remove TimeDistributed).
You can also try to remove the Masking layer in the example above and the error will go away.

but I have this model:
       model=tf.keras.Sequential()         embeding_layer=layers.Embedding(self.vocab_size,self.word_vector_dim,weights=[word_embeding_matrix],trainable=True,mask_zero=True)         model.add(embeding_layer)         model.add(layers.LSTM(50))         model.add(layers.Dropout(0.5))         model.add(layers.Dense(3,activation='softmax'))         opt=tf.keras.optimizers.RMSprop(learning_rate=0.001)         model.compile(optimizer=opt,loss='categorical_crossentropy',metrics=['accuracy'])         self.model=model
which works fine and with no error. As you can see I used masking and LSTM at the same time. so in my experience the problem stem from TimeDistributed and masking.

Interesting. Can you provide a complete example like mine so I can try your code as well?

the above example is a simple classifier, you can make a random dataset and feed the model with it, so you can see that it will work with no problem.
this is where the model does not work and produce an error while training :
       model=tf.keras.Sequential()         embeding_layer=layers.Embedding(self.vocab_size,self.word_vector_dim,weights=[word_embeding_matrix],trainable=False,mask_zero=True)         model.add(TimeDistributed(embeding_layer))         model.add(TimeDistributed(tf.keras.layers.LSTM(50)))         model.add(tf.keras.layers.Bidirectional(costumized_lstm.Costumized_LSTM(50)))         # model.add(tf.keras.layers.Bidirectional(costumized_lstm.Costumized_LSTM(100)))         model.add(layers.Dense(3,activation='softmax'))         opt=tf.keras.optimizers.Adam(learning_rate=0.001)         model.compile(optimizer=opt,loss='categorical_crossentropy',metrics=['accuracy'])

I have a similar issue, just posted a question on SO: https://stackoverflow.com/questions/58895694/cudnn-status-bad-param-when-trying-to-perform-inference-on-a-lstm-seq2seq-with-m

Sorry for the very late reply, I will take a close look this week. This issue is kind of tricky since the error is raised from cudnn kernel, which contains less information. The troubleshooting for root cause might take me sometime.

@qlzh727 in case it helps, for me the problem goes away when I disable eager mode.

@houtoms, the error is raised from cudnn kernel and it only says there is a bad param. Could you provide more details about which param is bad? I am not familiar with the cudnn kernel detail at all.

Yes, I can repro the error. Turns out the cudnnSetRNNDataDescriptor gives the BAD PARAM error. From the log, I can see you want to process the input data as 256 (batch size) x 144 (time steps) x 130 (unit size). But, when using masks, I observe that some entire sequences are masked out. For example, there could be only 25 meaningful sequences (at least having one time step) out of the 256 sequences in one batch. In this case, it seems the cuDNN requires the batch size to be 25. @mimxrt Can you first confirm the scenario I mentioned is what you want to do?

Thank you for your reply. The reason for processing 256 x 144 x 130 data is that I make the assumption that my maximum sequence (in real data) has approximately 256 * 144 = 36864 time steps (each time step has 130 values/features -> unit size). The shortest sequence has actually 3728 time steps. Hence, the input dataset in my example is generated using this shortest length and during processing the input is zero-padded to the longest sequence in _prep_ds_file(). During training, the Masking layer should then allow ignoring the zero-valued batch elements. Did that answer your question?
Now, I understand that you are saying that if any of my input batches does have at least one batch element that is completely filled with zeros (i.e. 144 x 130 zeros), cuDNN requires a different batch size for that batch. From my understanding of the docs the CuDNN kernel is used if

Inputs are not masked or strictly right padded.

And that is what I am doing, am I not? I use masking but made sure the data is strictly right padded. Some (or in this example code all) of the batches do have zero padding in the end that is bigger than one batch element (144 time steps) and therefore the entire batch element should be ignored.
If this is the wrong way of doing this, then what is the right way? Is it possible to change the batch size mid-training (i.e. per batch)? Otherwise, how can I train sequences of different length? Batch size 1?

Yes, your understanding is correct. And cuDNN doesn't support zero samples in batch for now.
I am not sure if it is possible/how to change the batch size during training using tf.dataset (@qlzh727 some tf.dataset experts?). Batch size 1 could work, but it might significantly affect the performance.
Or you could try to change the way you split your sequence into the batch, like using seq_len/batch_size as the current 'num_tsteps' rather than the fixed 144. But still you need to make sure the minimum seq_len >= batch_size.

@houtoms, Thanks for providing the context.
From high level API's perspective, I would expect the kernel to just return zeros for any of the sequence that is fully masked, rather than asking user to remove those values from the batch. It will be quite complicate to ask user to handle this on the python side.
@houtoms, will it be complicated to add this support (fully masked sequence) in the cudnn kernel?

Yes, I agree with you. I am contacting the cuDNN about this issue. Will update when I get any feedback.

Got some feedback from cuDNN and it sounds not complicated to do that. Already filed a request to them. For now, users still have to make sure they don't have such fully masked sequences in batch.

Thanks for the update.

我将模型从以下位置更改：
        自我 .model =顺序（[
            嵌入（len（self .item_map），self .embed_dim，input_length  = X.shape [ 1 ]，mask_zeros = True），
            LSTM（self .lstm_out），
            密集（len个（自 .item_map）- 1）
        ]）

至：
        自我 .model =顺序（[
            嵌入（len（self .item_map），self .embed_dim，input_length  = X.shape [ 1 ]），
            遮罩（mask_value = 0），
            LSTM（self .lstm_out），
            密集（len个（自 .item_map）- 1）
        ]）

解决了我的问题
我知道@mimxrt的代码具有相同的模型，我不知道为什么它对我有用，
但是我在这里为其他人添加了这个问题，也许它可以帮助调试

I changed my model from:
        self.model = Sequential([
            Embedding(len(self.item_map), self.embed_dim, input_length = X.shape[1],mask_zeros=True),
            LSTM(self.lstm_out),
            Dense(len(self.item_map)-1),
        ])

to:
        self.model = Sequential([
            Embedding(len(self.item_map), self.embed_dim, input_length = X.shape[1]),
            Masking(mask_value=0),
            LSTM(self.lstm_out),
            Dense(len(self.item_map)-1),
        ])

And solved my isssue
I know @mimxrt's code has the same model and I dont know why it works for me,
but im adding this for anyone else comes here with the issue and maybe it can help with debugging

I think embedding(mask_zero=true) create this problem,there are two way I find to slove it
1"mask_zero=False",but it changes code process
2 your way
thanks a lot.

Thank you @qlzh727 and @houtoms, very much looking forward to the new cuDNN feature. In the meantime I will try to re-arrange my input dimensions.

@houtoms and @qlzh727 thank you for the help. I am trying to implement a text classification model where the shape of the input is (samples, sentences, words, word_embeding) and I used TimeDistributed to implement it. having a fully masked sequence in this kind of model is inevitable. therefore it would be great if cuDNN supports fully masked sequence.

as @michal-au said, I deactivate eager mode and remove callbacks from model.fit and now the model is working and training with no error! @houtoms is it possible to be an eager mode issue, not the cudnn issue?

I can reproduce that mask_zeros=True is causing the crash. Doesn't matter if eager is on or off, with or without callbacks.
LSTM on a masked sequence is extremely common in NLP models, so this is a major bug in terms of impact.

@houtoms so why when I deactivate eager mode it works with no error?

@jalilasadi , I repro the error using mimxrt's python script posted above on Oct 10. The script works fine in both eager or graph mode if Masking layer is removed. Also, the script breaks in both modes if Masking layer is added.

I was recently dealing with the same problem. Do you have any new information regarding this issue?

@jalilasadi, when eager mode is disabled, the code went to a different path that doesn't use fused cudnn kernel, hence no error will be raised.

@qlzh727 thank you for your answer. I do not know what fused cudnn is, I am new to TensorFlow .does it make a difference in the result of the model that we implement or it just affects the computational performance?

THe fused cudnn lstm is faster to run on GPU, and should have the same math result as the standard implementation.

Yes, your understanding is correct. And cuDNN doesn't support zero samples in batch for now.
I am not sure if it is possible/how to change the batch size during training using tf.dataset (@qlzh727 some tf.dataset experts?). Batch size 1 could work, but it might significantly affect the performance.
Or you could try to change the way you split your sequence into the batch, like using seq_len/batch_size as the current 'num_tsteps' rather than the fixed 144. But still you need to make sure the minimum seq_len >= batch_size.

@houtoms As it seems the implementation takes more time than expected I wanted to continue and try your suggestion of using a dynamic number of timesteps (instead of the fixed 144). Unfortunately I get an error when doing this in TF 2.1.0 (calling model.fit() with an input of <PrefetchDataset shapes: ((128, None, 128), (128, None, 1)), types: (tf.float32, tf.float32)>:
 InvalidArgumentError:  ValueError: Attempt to convert a value (<BatchDataset shapes: ((128, None, 128), (128, None, 1)), types: (tf.float32, tf.float32)>) with an unsupported type (<class 'tensorflow.python.data.ops.dataset_ops.BatchDataset'>) to a Tensor.
Could it be that I misunderstood your suggestion (note the None in the input dimensions)? All ideas are much appreciated.

@houtoms, I assume this should be fixed already with the latest CuDNN kernel?

@qlzh727 I still get the same error with CuDNN 7.6.5. did you mean this version of CuDNN? because it is the last version of it.  this is my code
` def BGRU_BGRU(self,metrics_file_path):
self.metrics_file_path = metrics_file_path
# tf.compat.v1.disable_eager_execution()
    model=tf.keras.Sequential()
    # embeding_layer=layers.Embedding(self.vocab_size,self.word_vector_dim,weights=[self.word_embeding_matrix],trainable=True,mask_zero=True)
    embeding_layer=layers.Embedding(self.vocab_size,self.word_vector_dim,mask_zero=True)
    model.add(TimeDistributed(embeding_layer))
    model.add(TimeDistributed(tf.keras.layers.Bidirectional(tf.keras.layers.GRU(50))))
    # model.add(TimeDistributed(tf.keras.layers.Bidirectional(tf.keras.layers.GRU(50,dropout=0.2,recurrent_dropout=0.2))))
    model.add(tf.keras.layers.Bidirectional(tf.keras.layers.GRU(50)))
    # model.add(tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(50,dropout=0.2,recurrent_dropout=0.2)))
    # model.add(tf.keras.layers.Dense(30,activation='sigmoid'))
    model.add(layers.Dense(self.config.number_of_classess,activation='softmax'))
    lr_schedule = tf.keras.optimizers.schedules.InverseTimeDecay(self.config.lr, decay_steps=self.config.decay_steps, decay_rate=1,staircase=False)
    opt = tf.keras.optimizers.RMSprop(learning_rate=lr_schedule)
    model.compile(optimizer=opt, loss='categorical_crossentropy', metrics=['accuracy', 'mse'])
    self.model = model
    return`

and this is the error that running this code produce:
`Train on 33171 samples, validate on 8293 samples
Epoch 1/8
2020-02-29 03:04:13.742695: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_10.dll
2020-02-29 03:04:14.303999: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2020-02-29 03:04:15.018554: W tensorflow/core/framework/op_kernel.cc:1655] OP_REQUIRES failed at cudnn_rnn_ops.cc:1517 : Unknown: CUDNN_STATUS_BAD_PARAM
in tensorflow/stream_executor/cuda/cuda_dnn.cc(1430): 'cudnnSetRNNDataDescriptor( data_desc.get(), data_type, layout, max_seq_length, batch_size, data_size, seq_lengths_array, (void*)&padding_fill)'
2020-02-29 03:04:15.018854: W tensorflow/core/common_runtime/base_collective_executor.cc:217] BaseCollectiveExecutor::StartAbort Unknown: CUDNN_STATUS_BAD_PARAM
in tensorflow/stream_executor/cuda/cuda_dnn.cc(1430): 'cudnnSetRNNDataDescriptor( data_desc.get(), data_type, layout, max_seq_length, batch_size, data_size, seq_lengths_array, (void*)&padding_fill)'
[[{{node cond_31/then/_0/CudnnRNNV3}}]]
2020-02-29 03:04:15.019228: W tensorflow/core/common_runtime/base_collective_executor.cc:217] BaseCollectiveExecutor::StartAbort Unknown: {{function_node __forward_cudnn_gru_with_fallback_681666_specialized_for_sequential_time_distributed_1_bidirectional_backward_gru_StatefulPartitionedCall_at___inference_distributed_function_687655}} {{function_node __forward_cudnn_gru_with_fallback_681666_specialized_for_sequential_time_distributed_1_bidirectional_backward_gru_StatefulPartitionedCall_at___inference_distributed_function_687655}} CUDNN_STATUS_BAD_PARAM
in tensorflow/stream_executor/cuda/cuda_dnn.cc(1430): 'cudnnSetRNNDataDescriptor( data_desc.get(), data_type, layout, max_seq_length, batch_size, data_size, seq_lengths_array, (void*)&padding_fill)'
[[{{node cond_31/then/_0/CudnnRNNV3}}]]
[[sequential/time_distributed_1/bidirectional/backward_gru/StatefulPartitionedCall]]
[[Reshape_10/_28]]
2020-02-29 03:04:15.019884: W tensorflow/core/common_runtime/base_collective_executor.cc:217] BaseCollectiveExecutor::StartAbort Unknown: {{function_node __forward_cudnn_gru_with_fallback_681666_specialized_for_sequential_time_distributed_1_bidirectional_backward_gru_StatefulPartitionedCall_at___inference_distributed_function_687655}} {{function_node __forward_cudnn_gru_with_fallback_681666_specialized_for_sequential_time_distributed_1_bidirectional_backward_gru_StatefulPartitionedCall_at___inference_distributed_function_687655}} CUDNN_STATUS_BAD_PARAM
in tensorflow/stream_executor/cuda/cuda_dnn.cc(1430): 'cudnnSetRNNDataDescriptor( data_desc.get(), data_type, layout, max_seq_length, batch_size, data_size, seq_lengths_array, (void*)&padding_fill)'
[[{{node cond_31/then/_0/CudnnRNNV3}}]]
[[sequential/time_distributed_1/bidirectional/backward_gru/StatefulPartitionedCall]]
2020-02-29 03:04:15.023940: W tensorflow/core/framework/op_kernel.cc:1655] OP_REQUIRES failed at cudnn_rnn_ops.cc:1517 : Unknown: CUDNN_STATUS_BAD_PARAM
in tensorflow/stream_executor/cuda/cuda_dnn.cc(1430): 'cudnnSetRNNDataDescriptor( data_desc.get(), data_type, layout, max_seq_length, batch_size, data_size, seq_lengths_array, (void*)&padding_fill)'
2020-02-29 03:04:15.024206: W tensorflow/core/common_runtime/base_collective_executor.cc:217] BaseCollectiveExecutor::StartAbort Unknown: CUDNN_STATUS_BAD_PARAM
in tensorflow/stream_executor/cuda/cuda_dnn.cc(1430): 'cudnnSetRNNDataDescriptor( data_desc.get(), data_type, layout, max_seq_length, batch_size, data_size, seq_lengths_array, (void*)&padding_fill)'
[[{{node cond_31/then/_0/CudnnRNNV3}}]]
300/33171 [..............................] - ETA: 26:472020-02-29 03:04:19.483701: W tensorflow/core/framework/op_kernel.cc:1655] OP_REQUIRES failed at cudnn_rnn_ops.cc:1517 : Unknown: CUDNN_STATUS_BAD_PARAM
in tensorflow/stream_executor/cuda/cuda_dnn.cc(1430): 'cudnnSetRNNDataDescriptor( data_desc.get(), data_type, layout, max_seq_length, batch_size, data_size, seq_lengths_array, (void*)&padding_fill)'
2020-02-29 03:04:19.484099: W tensorflow/core/common_runtime/base_collective_executor.cc:217] BaseCollectiveExecutor::StartAbort Unknown: CUDNN_STATUS_BAD_PARAM
in tensorflow/stream_executor/cuda/cuda_dnn.cc(1430): 'cudnnSetRNNDataDescriptor( data_desc.get(), data_type, layout, max_seq_length, batch_size, data_size, seq_lengths_array, (void*)&padding_fill)'
[[{{node cond/then/_0/CudnnRNNV3}}]]
2020-02-29 03:04:19.484494: W tensorflow/core/common_runtime/base_collective_executor.cc:217] BaseCollectiveExecutor::StartAbort Unknown: {{function_node __inference_cudnn_gru_with_fallback_689679_specialized_for_sequential_time_distributed_1_bidirectional_backward_gru_StatefulPartitionedCall_at___inference_distributed_function_693569}} {{function_node __inference_cudnn_gru_with_fallback_689679_specialized_for_sequential_time_distributed_1_bidirectional_backward_gru_StatefulPartitionedCall_at___inference_distributed_function_693569}} CUDNN_STATUS_BAD_PARAM
in tensorflow/stream_executor/cuda/cuda_dnn.cc(1430): 'cudnnSetRNNDataDescriptor( data_desc.get(), data_type, layout, max_seq_length, batch_size, data_size, seq_lengths_array, (void*)&padding_fill)'
[[{{node cond/then/0/CudnnRNNV3}}]]
[[sequential/time_distributed_1/bidirectional/backward_gru/StatefulPartitionedCall]]
Traceback (most recent call last):
File "C:\Users\jalil\AppData\Local\Programs\Python\Python37\lib\contextlib.py", line 130, in exit
self.gen.throw(type, value, traceback)
File "C:\Users\jalil\PycharmProjects\my_projects\venv\lib\site-packages\tensorflow_core\python\ops\variable_scope.py", line 2803, in variable_creator_scope
yield
File "C:\Users\jalil\PycharmProjects\my_projects\venv\lib\site-packages\tensorflow_core\python\keras\engine\training_v2.py", line 397, in fit
prefix='val')
File "C:\Users\jalil\AppData\Local\Programs\Python\Python37\lib\contextlib.py", line 130, in exit
self.gen.throw(type, value, traceback)
File "C:\Users\jalil\PycharmProjects\my_projects\venv\lib\site-packages\tensorflow_core\python\keras\engine\training_v2.py", line 771, in on_epoch
self.callbacks.on_epoch_end(epoch, epoch_logs)
File "C:\Users\jalil\PycharmProjects\my_projects\venv\lib\site-packages\tensorflow_core\python\keras\callbacks.py", line 302, in on_epoch_end
callback.on_epoch_end(epoch, logs)
File "C:\Users\jalil\PycharmProjects\my_projects\coding_paian_nameh\model\classification_models.py", line 783, in on_epoch_end
y_test_pred = self.model.predict(self.X_test)
File "C:\Users\jalil\PycharmProjects\my_projects\venv\lib\site-packages\tensorflow_core\python\keras\engine\training.py", line 1013, in predict
use_multiprocessing=use_multiprocessing)
File "C:\Users\jalil\PycharmProjects\my_projects\venv\lib\site-packages\tensorflow_core\python\keras\engine\training_v2.py", line 498, in predict
workers=workers, use_multiprocessing=use_multiprocessing, **kwargs)
File "C:\Users\jalil\PycharmProjects\my_projects\venv\lib\site-packages\tensorflow_core\python\keras\engine\training_v2.py", line 475, in _model_iteration
total_epochs=1)
File "C:\Users\jalil\PycharmProjects\my_projects\venv\lib\site-packages\tensorflow_core\python\keras\engine\training_v2.py", line 128, in run_one_epoch
batch_outs = execution_function(iterator)
File "C:\Users\jalil\PycharmProjects\my_projects\venv\lib\site-packages\tensorflow_core\python\keras\engine\training_v2_utils.py", line 98, in execution_function
distributed_function(input_fn))
File "C:\Users\jalil\PycharmProjects\my_projects\venv\lib\site-packages\tensorflow_core\python\eager\def_function.py", line 568, in call
result = self._call(*args, *kwds)
File "C:\Users\jalil\PycharmProjects\my_projects\venv\lib\site-packages\tensorflow_core\python\eager\def_function.py", line 638, in _call
return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds)  # pylint: disable=protected-access
File "C:\Users\jalil\PycharmProjects\my_projects\venv\lib\site-packages\tensorflow_core\python\eager\function.py", line 1611, in _filtered_call
self.captured_inputs)
File "C:\Users\jalil\PycharmProjects\my_projects\venv\lib\site-packages\tensorflow_core\python\eager\function.py", line 1692, in _call_flat
ctx, args, cancellation_manager=cancellation_manager))
File "C:\Users\jalil\PycharmProjects\my_projects\venv\lib\site-packages\tensorflow_core\python\eager\function.py", line 545, in call
ctx=ctx)
File "C:\Users\jalil\PycharmProjects\my_projects\venv\lib\site-packages\tensorflow_core\python\eager\execute.py", line 67, in quick_execute
six.raise_from(core._status_to_exception(e.code, message), None)
File "", line 3, in raise_from
tensorflow.python.framework.errors_impl.UnknownError:  [Derived]  CUDNN_STATUS_BAD_PARAM
in tensorflow/stream_executor/cuda/cuda_dnn.cc(1430): 'cudnnSetRNNDataDescriptor( data_desc.get(), data_type, layout, max_seq_length, batch_size, data_size, seq_lengths_array, (void)&padding_fill)'
[[{{node cond/then/_0/CudnnRNNV3}}]]
[[sequential/time_distributed_1/bidirectional/backward_gru/StatefulPartitionedCall]] [Op:__inference_distributed_function_693569]
Function call stack:
distributed_function -> distributed_function -> distributed_function`

@houtoms, do u know whether the fix has reach the public release? If that's the case, please share a script/colab that includes version and the working model. Thanks.

@qlzh727 Not yet. We might need to wait for the next major release for the fix.

@qlzh727 Not yet. We might need to wait for the next major release for the fix.

Do you have an estimate time for that?

import tensorflow as tf
tf.compat.v1.disable_eager_execution()
Dissable eager execution and everything is running fine without the fused rnn kernel. Thx for the help guys :)

I have a work around that seems to work:  force TF to use the non CuDNN implementation by selecting a sigmoid activation instead of TANH
layers.LSTM(..., activation='sigmoid').
This forces TF to a generic GPU kernel.  It's slower but a slower implementation is a lot faster than not working at all ;p

If you really want that as a workaround, you can find the requirements for the cuDNN implementation here.

The requirements to use the cuDNN implementation are:

activation == tanh
recurrent_activation == sigmoid
recurrent_dropout == 0
unroll is False
use_bias is True
Inputs are not masked or strictly right padded.

I guess changing any one of these will result in the CPU implementation being used.

That's right mimxrt.  I guess the interesting point here is that the inputs to my LSTM are always masked, but I have to force the non-cudnn implementation using the activation function.  Might be a clue for someone who can fix this.

I'd be nice to have a flag on LSTM layers which allows to disable the use of the CuDNN implementation instead of either disabling eager execution or using non-default activation functions.

Facing this same issue and reported here: https://github.com/tensorflow/tensorflow/issues/40982.

I'd be nice to have a flag on LSTM layers which allows to disable the use of the CuDNN implementation instead of either disabling eager execution or using non-default activation functions.

There is a private field you can set to disable the cudnn kernel once the layer is created.
layer = tf.keras.layers.LSTM(4)
layer. _could_use_gpu_kernel = False

This will focus the lstm layer to use the non-cudnn implementation, even landed on GPU.

@queirozfcom what would you suggest to use between this option and disabling eager execution as when eager execution is disabled it does work?

Disable eager will have larger side effects since it change the runtime to execute in graph/session context. We don't recommend user to fallback to graph unless they really need some feature in graph/session.

Thanks :)

I am still facing this issue using TF 2.2.0. I also found the same workaround of forcing the LSTM to not use the cuDNN implementation to work, however it is nearly prohibitively slow. I found the generic GPU implementation took ~30 times longer to train per epoch than the cuDNN version. I hope this can be fixed soon.

I have disable the cudnn code path if there is fully masked data in the batch. It will have some performance dip since it fallback to generic kernel, but won't error out.

facing this issue using TF 2.2.0 while evaluating

Please try with the latest nightly version.

@qlzh727 Not yet. We might need to wait for the next major release for the fix.

cudnn 8 is here, is there any info about this issue?

@geetachavan1 I am using `tensorflow==2.3.0 but for this network:
def call(self, inputs, training=None, memory_states=None, **kwargs):

    x = self.embedding(inputs, training=training)

    if memory_states is None:
        memory_states = self.get_initial_state(shape_list(x)[0])

    next_memory_states = []

    for i, (lstm, norm) in enumerate(self.lstm_stack):
        x = norm(x, training=training)
        outputs = lstm(x, training=training, initial_state=memory_states[i])
        x, state = outputs[0], outputs[1:]
        next_memory_states.append(state)

    return x, next_memory_states

where we have:
self.embedding = layers.Embedding(vocab_size, embedding_size, mask_zero=True)
self.lstm_stack = list()
for _ in range(num_layers):
    lstm = layers.LSTM(
        units=lstm_units,
        return_sequences=True,
        return_state=True,
        dropout=dropout
    )
    norm = layers.LayerNormalization()
    self.lstm_stack.append((lstm, norm))

I am getting
Traceback (most recent call last):
  File "/home/sfalk/tmp/speech-v2/asr/bin/eval_transducer.py", line 153, in <module>
    main()
  File "/home/sfalk/tmp/speech-v2/asr/bin/eval_transducer.py", line 85, in main
    y_greedy = model.greedy_decode(encoder_inputs)[0]
  File "/home/sfalk/tmp/speech-v2/asr/model/transducer.py", line 351, in greedy_decode
    init_predict_network_state = PredictNetworkState(*self.predict_network(init_yseq))
  File "/home/sfalk/miniconda3/envs/asr2/lib/python3.8/site-packages/tensorflow/python/keras/engine/base_layer.py", line 985, in __call__
    outputs = call_fn(inputs, *args, **kwargs)
  File "/home/sfalk/tmp/speech-v2/asr/model/transducer.py", line 118, in call
    outputs = lstm(x, training=training, initial_state=memory_states[i])
  File "/home/sfalk/miniconda3/envs/asr2/lib/python3.8/site-packages/tensorflow/python/keras/layers/recurrent.py", line 720, in __call__
    return super(RNN, self).__call__(inputs, **kwargs)
  File "/home/sfalk/miniconda3/envs/asr2/lib/python3.8/site-packages/tensorflow/python/keras/engine/base_layer.py", line 985, in __call__
    outputs = call_fn(inputs, *args, **kwargs)
  File "/home/sfalk/miniconda3/envs/asr2/lib/python3.8/site-packages/tensorflow/python/keras/layers/recurrent_v2.py", line 1176, in call
    last_output, outputs, new_h, new_c, runtime = gpu_lstm(
  File "/home/sfalk/miniconda3/envs/asr2/lib/python3.8/site-packages/tensorflow/python/keras/layers/recurrent_v2.py", line 1401, in gpu_lstm
    outputs, h, c, _, _ = gen_cudnn_rnn_ops.cudnn_rnnv3(
  File "/home/sfalk/miniconda3/envs/asr2/lib/python3.8/site-packages/tensorflow/python/ops/gen_cudnn_rnn_ops.py", line 1914, in cudnn_rnnv3
    return cudnn_rnnv3_eager_fallback(
  File "/home/sfalk/miniconda3/envs/asr2/lib/python3.8/site-packages/tensorflow/python/ops/gen_cudnn_rnn_ops.py", line 2011, in cudnn_rnnv3_eager_fallback
    _result = _execute.execute(b"CudnnRNNV3", 5, inputs=_inputs_flat,
  File "/home/sfalk/miniconda3/envs/asr2/lib/python3.8/site-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.UnknownError: CUDNN_STATUS_BAD_PARAM
in tensorflow/stream_executor/cuda/cuda_dnn.cc(1521): 'cudnnSetRNNDataDescriptor( data_desc.get(), data_type, layout, max_seq_length, batch_size, data_size, seq_lengths_array, (void*)&padding_fill)' [Op:CudnnRNNV3]

Shouldn't this be fixed?

Some update: the cudnn v8.0.5 (next release) should be able to fix the issue exposed by https://github.com/tensorflow/tensorflow/issues/33148#issuecomment-540472342.

When will this release happen .....looks like without CuDNN the implementation of LSTM runs very slow.

When will this release happen .....looks like without CuDNN the implementation of LSTM runs very slow.
cudnn 8.0.5 is already out https://docs.nvidia.com/deeplearning/cudnn/archives/index.html

Some update: the cudnn v8.0.5 (next release) should be able to fix the issue exposed by #33148 (comment).

@kaixih where did you read it?
I'd like to give it a try.

When will this release happen .....looks like without CuDNN the implementation of LSTM runs very slow.

cudnn 8.0.5 is already out https://docs.nvidia.com/deeplearning/cudnn/archives/index.html

I tried with cudnn=8.0.4 cudatoolkit=11.0.221 tensorflow-gpu=2.4.0, and it fixed my problem.
cudnn can be installed by:
conda install -c nvidia cudnn

When will this release happen .....looks like without CuDNN the implementation of LSTM runs very slow.

cudnn 8.0.5 is already out https://docs.nvidia.com/deeplearning/cudnn/archives/index.html

I tried with cudnn=8.0.4 cudatoolkit=11.0.221 tensorflow-gpu=2.4.0, and it fixed my problem.
cudnn can be installed by:
conda install -c nvidia cudnn

I have the same issue on Google Colab using a Tesla P100 (tf 2.7.0, cuda v11.1, cudnn v8), when training lstm-based seq2seq models with a preceding Masking Layer On CPU the exact same code runs fine.


iOS  -  App crashes when re-inviting the member

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Actions Performed:

Open expensify app in member iOS device keep it open.
Open expensify app in admin device(any device) and remove the member from the workspace
Invite the member again to the same workspace

Expected Result:
Member should be able to use the app properly and interact with workspace
Actual Result:
iOS app is crashing on member device. Crashes every time i try to open the app again. only fix seems to be re-install.
Workaround:
unknown
Platforms:

Which of our officially supported platforms is this issue occurring on?

[ ] iOS / native

Version Number: 1.2.46-0
Reproducible in staging?: y
Reproducible in production?: y
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Notes/Photos/Videos:
https://user-images.githubusercontent.com/43996225/210087275-6fd1d1ee-72bb-43e4-b585-e963b1a15786.mov
https://user-images.githubusercontent.com/43996225/210087296-8f5db2d9-567f-4bab-a06b-4671241dae70.mp4
Expensify/Expensify Issue URL:
Issue reported by: @abdulrahuman5196
Slack conversation: https://expensify.slack.com/archives/C049HHMV9SM/p1672258474181849
View all open jobs on GitHub
Upwork Automation - Do Not Edit

Upwork Job URL: https://www.upwork.com/jobs/~016b62be585203ffd9
Upwork Job ID:<PHONE_NUMBER>823237632
Last Price Increase: 2023-01-02

Gonna check this one out!

was able to reproduce on iOS dev, something is happening when we try to save the Onyx response.

The contributor who originally pointed out this issue and made this proposal DMd me with a solution, and it mostly makes sense to me, I'm going to open up this issue to External so that they can post their proposal here for further review. At first glance the proposal makes sense to me.
Based on what they told me, here's an outline of what's going on with Onyx:

Onyx's mergeCollection relies on AsyncStorage.multiMerge.
When we delete a member from a workspace, we call Onyx.mergeCollection, merging null values for each workspace member to effectively delete them. However, even though the values have been cleared, the keys are still present.
Later on, when we add deleted members back to the workspace, the key still exists, so we try to merge an object into null.
In AsyncStorage, this action is handled gracefully on all other platforms (null is overwritten by the object), but for whatever reason on iOS it throws an exception.

The solution is when we call mergeCollection with null values, we should also delete the keys from storage as well. This shouldn't cause any problems because the behavior of getting a non-existent keys is also to return null.

Proposal
I have already add the Root cause and solution as part of the original bug statement in slack here - https://expensify.slack.com/archives/C049HHMV9SM/p1672258474181849

Proposal
Background:
Whenever we receive a pusher event to delete data, it has data like for eg.
key = 'keyToDelete'
value = null

In Onyx, what we do is call set() function with key('keyToDelete') and value 'null'. Now we call AsyncStorage(backbone of Onyx) and OnyxCache with the same key and value through set() function.
When a member is added to the workspace, we get 'mergeCollection' event which ultimately ends up here https://github.com/Expensify/react-native-onyx/blob/main/lib/Onyx.js#L1123

Rootcause:
Now we assume as the value is set to 'null' during 1st action, the data is deleted in OnyxCache and AsyncStorage. But no, the key is still present in AsyncStorage with value as null. And OnyxCache also has the key in its cache.  This key should have been ideally deleted from AsyncStorage and OnyxCache as per pusher event/standard practice.Now when a mergeCollection(as part of any pusher event in this context  workspace case) request comes to Onyx, we try to find if we already have keys present in cache/storage to understand if we need to call multiMerge(called for existing keys) or multiSet(called for new keys)
https://github.com/Expensify/react-native-onyx/blob/main/lib/Onyx.js#L1141
https://github.com/Expensify/react-native-onyx/blob/main/lib/Onyx.js#L1156
Here even though the values are null for the keys from previous pusher notifications, when we call getKeys() we would still get the key(which should have been ideally not present). So we call multiMerge of AsyncStorage instead of multiSet as we think AsyncStorage already has some value for the key. AsyncStorage now tries to merge a value of 'null' and new object from request which results in crash.This would also be the root cause related to these inconsistent iOS crash bugs - https://github.com/Expensify/App/issues/10975, https://github.com/Expensify/App/issues/12330
Why is this happening only for iOS?
This might be a difference in AsyncStorage itself on how it handles exceptions. On other platforms even though multiMerge is called with wrong data AsyncStorage handles gracefully but not with iOS. But expensify should be responsible in this case as we are sending invalid data to multiMerge.
Solution 1 (Best Long term fix IMO):
Instead of setting the key value as null, call AsyncStorage.removeItem(key)  and remove the key from OnyxCache as well. This will remove the keys from AsyncStorage and OnyxCache. This way when you call mergeCollection it will properly find if the the key is new or existing with proper value. This will also remove any inconsistencies we have with keys present in Onyx cache and AsyncStorgage.We can also extend the same fix to multiSet as well as similar to set().
I have added short sample solution code for contributors to test if my analysis is correct. I can work on
full code with proper standard formats and provide the changes required if my proposal gets approved.
Solution 2(Not Preferred IMO):
Reach out to AsyncStorage package to handle this exception gracefully. This might not fix our issues with OnyxCache and won't fix underlying issue in expensify app.
Solution 3(Not best just proposing IMO):
Before merge check if the key has actual value in the storage instead of null value. (edited)
Code diff for fix
diff --git a/lib/Onyx.js b/lib/Onyx.js
index 4a4196d..1e33923 100644
--- a/lib/Onyx.js
+++ b/lib/Onyx.js
@@ -799,8 +799,7 @@ function notifyCollectionSubscribersOnNextTick(key, value) {
  * @return {Promise}
  */
 function remove(key) {
-    // Cache the fact that the value was removed
-    cache.set(key, null);
+    cache.drop(key)
     notifySubscribersOnNextTick(key, null);
     return Storage.removeItem(key);
 }
@@ -847,6 +846,9 @@ function evictStorageAndRetry(error, onyxMethod, ...args) {
  * @returns {Promise}
  */
 function set(key, value) {
+    if (!value) {
+        return remove(key)
+    }
     // Logging properties only since values could be sensitive things we don't want to log
     Logger.logInfo(`set() called for key: ${key}${_.isObject(value) ? ` properties: ${_.keys(value).join(',')}` : ''}`);

diff --git a/lib/OnyxCache.js b/lib/OnyxCache.js
index 4ac4222..1500e47 100644
--- a/lib/OnyxCache.js
+++ b/lib/OnyxCache.js
@@ -104,6 +104,8 @@ class OnyxCache {
      */
     drop(key) {
         delete this.storageMap[key];
+        this.storageKeys.delete(key);
+        this.recentKeys.delete(key);
     }

     /**

Working video after fix
https://user-images.githubusercontent.com/46707890/210214334-60a648a6-413c-4e68-8d23-7dbbb665452a.mov

@abdulrahuman5196 awesome write-up! definitely a lot more clear here than the first one you provided on Slack 👍
Everything sounds pretty good to me, but I'm gonna drop another note in #engineering-chat to get more pairs of eyes on this proposal just in case, since I'm sure there are a few more engineers who have opinions on any changes to the actual Onyx library.

Thank you @jasperhuangg for hearing me out. Hope this information is sufficient. If not kindly do let me know and I would be able to provide any information requested.

@jasperhuangg, @abdulrahuman5196 , I think AsyncStorage will be deprecated on next Onyx release , It will be replaced with quick-sqlite , check https://github.com/Expensify/react-native-onyx/pull/211 , hence I think it's wise to hold this issue until sqllite deployed to production

Thanks for the ping @fedirjh!
@abdulrahuman5196 Let's wait for the linked PR to go out and see if we still run into the same problems. Thank you for your patience.

Yeah, I agree that with AsyncStorage going away, this is less of a problem. Unfortunately, with the SQLite change already merged, we are stuck with fixing any bugs in Onyx until we are ready to release the SQLite version. This is one of the projects I will be working on this week with @marcaaron.
I do think that it would probably be a good idea to clean this up anyway (a key with a null value doesn't feel quite right), but I do have a question about what will happen to Onyx subscribers (both withOnyx as well as Onyx.connect()). What will their behavior be when a key is deleted (assuming we go with solution 1 above)?

OK, got it. Thanks for the links! I think that should work OK then.
It sounds like we are going to move forward with the SQLite migration (the update PR is being written right now), so this bug will automatically be going away since AsyncStorage is no longer used. That solves the crash and the bug, so the proposal from @abdulrahuman5196 is really more just a code quality and cleanup issue with low priority. I just wanted to point that out as we move forward with this.
Is everyone OK with that path?

Thank you @tgolen for notifying. I agree on the assessment if we are moving to SQLite sooner and proposal will be a cleanup action item.
So what is the process now for this cleanup action item which can be achieved with my proposal.
Are we going to keep the change aside for it to be taken on a future date or should we do a PR as the code change is already available?

I think it would be OK to work on it as long as you don't have any bugs that you're currently working on, and under the assumption that people reviewing your PR will review at a lower priority than any bugs.

@tgolen Understood!
@abdulrahuman5196 please confirm that you don't have anything else on your plate at the moment, and I'll be happy to assign you to this issue and review your PR.
Otherwise, please let me know when you have more time to deal with this low-priority issue.

Thank you. @jasperhuangg Currently I am not working on any other bugs. I can take this task up.

Taking the HOLD off based on Tim's comments here

I skipped a lot of this conversation, but I think we can just do a retest to see if react-native-quick-sqlite still has this problem and then close this issue? AFAIK there should be no issues with calling multiMerge() with objects that have null properties for SQLite.
Not to be a stickler. But I think I am not really OK with the path of this issue. I would prefer to have Onyx changes discussed publicly. Let's talk about what problem we are solving, what doesn't feel right about it, and then find some clear solution. Again, I didn't read through this issue (I might 100% agree with the proposal) so that is just a high level comment on converting an issue about a "crash" into a "code cleanup".

Hey @marcaaron I agree with what you're saying.. no reason to have this issue lying around if it was fixed by the react-native-quick-sqlite stuff.
We demoted the priority of this because it became a code-cleanup, but I get what you're saying–nothing is actually broken here. The PR is more there to make us feel "better safe than sorry".
I also feel that it's strange that the key for an item still exists even after it's been removed.
cc @tgolen what are your thoughts?

Yes, this issue is fixed by the release of SQLite and it can be closed now since it should no longer be reproducible.

Hi,
given this issue is closed.
Melvi bot commented to apply for a upwork job, but didn't have a upwork link.
I was the initial reporter of this issue as well and the null check PR is pending in review.
What is the compensation process for this issue?

The onyx version was updated 5 days back itself part of this commit - https://github.com/Expensify/App/commit/e7651685ba1f8b9e4ee54d023a3c917de5179b2b
I don't think we need to merge a separate PR
@jasperhuangg

Hey @Christinadobrzyn do we know if there is an Upwork job up for this issue? @abdulrahuman5196 needs to be compensated for discovering the issue and fixing it as well.

Gentle bump @Christinadobrzyn

@abdulrahuman5196 apologies for the delay.  @jasperhuangg @eVoloshchak is total compensation due to @abdulrahuman5196 $1250? $1000 for fix and $250 for reporting? @eVoloshchak are you also due C+ compensation?
Job is https://www.upwork.com/jobs/~016b62be585203ffd9
It's private but both y'all should be able to see.

@mallenexpensify
I am asking this question because I am not sure on the process of this case. The original PR raised was code complete and got all the required approvals within 2 days. But it was put on hold due to sqlite merging. Other the hold time the PR only took 2 days to complete and get all the required approvals. I think this should be eligible for 50% bonus as the PR was on hold due to different change. Kindly let me know otherwise.
Hold reference: https://github.com/Expensify/react-native-onyx/pull/223#issuecomment-1374196239

Ah... an edge case!  Let's see...

In general, if you were hired for a job, were actively working on it (or finished it) then we put the job on hold or closed it, you would be compensated. So.. That should be $1000 (I think)
If your PR was merged within 3 business days and it was before the issue was put on hold, you'd be eligible for the bonus too.

I'm going to wait for feedback from @eVoloshchak and/or @jasperhuangg as they're more familiar.  Thanks for the patience here @abdulrahuman5196 and for providing the details above.  We'll get this sorted soon (might not be til Monday cuz it's Friday though)

Thank you @mallenexpensify. I am fine with either though. Just wanted to check on eligibility. I have accepted the offer in upwork.

@abdulrahuman5196 apologies for the delay. @jasperhuangg @eVoloshchak is total compensation due to @abdulrahuman5196 $1250? $1000 for fix and $250 for reporting? @eVoloshchak are you also due C+ compensation?

@mallenexpensify, I wasn't C+ for this, so not eligible for any payment

Thanks @eVoloshchak.
@jasperhuangg , please review when you have a min

Hi guys, since this wasn't really a high priority fix (the bug itself was fixed by the sqlite migration) there was a good reason to hold the review for this for other, higher priority items. The PRs themselves were just code cleanup.
With that being said, I don't think it'd be fair to @abdulrahuman5196 if we didn't pay out the 50% bonus because they were able to get the PRs done quickly–it wasn't their fault that we weren't able to merge the PRs.
So I think we should still pay out the 50% bonus.
cc @mallenexpensify

Thanks @jasperhuangg , I agree.  I just paid @abdulrahuman5196  $1750 - $1000 for the job, 50% for the bonus and $250 for reporting.  Since this was fixed by sqlite, regression test steps shouldn't be needed so I'm closing.

Thank you @jasperhuangg and @mallenexpensify


MetamaskInpageProvider - lost connection to MetaMask RpcProvider

MetamaskInpageProvider - lost connection to MetaMask RpcProvider
Error: StreamMiddleware - Unknown response id<PHONE_NUMBER>
at chrome-extension://dlemoigadagaakbfkaflkknjpidifbji/inpage.js:1:276079
at h.write [as _write] (chrome-extension://dlemoigadagaakbfkaflkknjpidifbji/inpage.js:1:276195)
at v (chrome-extension://dlemoigadagaakbfkaflkknjpidifbji/inpage.js:1:371323)
at chrome-extension://dlemoigadagaakbfkaflkknjpidifbji/inpage.js:1:374439
at h.g.write (chrome-extension://dlemoigadagaakbfkaflkknjpidifbji/inpage.js:1:374466)
at a.b (chrome-extension://dlemoigadagaakbfkaflkknjpidifbji/inpage.js:1:363746)
at d (chrome-extension://dlemoigadagaakbfkaflkknjpidifbji/inpage.js:1:95984)
at a.o.emit (chrome-extension://dlemoigadagaakbfkaflkknjpidifbji/inpage.js:1:98970)
at _ (chrome-extension://dlemoigadagaakbfkaflkknjpidifbji/inpage.js:1:358585)
at w (chrome-extension://dlemoigadagaakbfkaflkknjpidifbji/inpage.js:1:358400)

Hard to tell, not something our snap should use. Did you build metamask yourself?
We stopped maintaining this snap until Metamask releases snaps in beta as it's quite difficult to use snaps now.

Hard to tell, not something our snap should use. Did you build metamask yourself?
We stopped maintaining this snap until Metamask releases snaps in beta as it's quite difficult to use snaps now.

stale? re-open if needed.


Attribute Error "NoneType" object has no attribute success

I am trying to run the dojo_ci_cd.py file in my local environment and it is prompting with the below error.  Please could you check?  ( or probably it could be a defectdojo bug)
Defectdojo=lastest master
Defectdojo_api 1.1.4
Command & Output:
python dojo_ci_cd.py --product=1 --file "~/Dev/misc/defectdojo_api/tests/scans/Bodgeit-burp.xml" --scanner="Burp Scan" --high=0 --host=http://<IP_ADDRESS>:8000 —api_key=xxxx --user=admin

Traceback (most recent call last):
  File "dojo_ci_cd.py", line 241, in <module>
    class Main:
  File "dojo_ci_cd.py", line 282, in Main
    test_ids = processFiles(dd, engagement_id, file, scanner=scanner)
  File "dojo_ci_cd.py", line 144, in processFiles
    if test_id.success == False:
AttributeError: 'NoneType' object has no attribute 'success'

Output of defectdojo logs:

[11/Jul/2018 05:29:06] "GET /api/v1/users/?username=admin&limit=20 HTTP/1.1" 200 246
Internal Server Error: /api/v1/engagements/
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/tastypie/resources.py", line 227, in wrapper
    response = callback(request, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/tastypie/resources.py", line 467, in dispatch_list
    return self.dispatch('list', request, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/tastypie/resources.py", line 499, in dispatch
    response = method(request, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/tastypie/resources.py", line 1408, in post_list
    updated_bundle = self.obj_create(bundle, **self.remove_api_resource_names(kwargs))
  File "/usr/local/lib/python2.7/dist-packages/tastypie/resources.py", line 2245, in obj_create
    return self.save(bundle)
  File "/usr/local/lib/python2.7/dist-packages/tastypie/resources.py", line 2380, in save
    self.is_valid(bundle)
  File "/usr/local/lib/python2.7/dist-packages/tastypie/resources.py", line 1320, in is_valid
    errors = self._meta.validation.is_valid(bundle, bundle.request)
  File "/usr/local/lib/python2.7/dist-packages/tastypie/validation.py", line 73, in is_valid
    form = self.form_class(**self.form_args(bundle))
  File "/vagrant/dojo/api.py", line 85, in form_args
    pk = self._get_pk_from_resource_uri(rel_field, resource_uri)
  File "/vagrant/dojo/api.py", line 70, in _get_pk_from_resource_uri
    resource_uri, base_resource_uri))
Exception: Couldn't match resource_uri //api/v1/users/1/ with /api/v1/users/
Internal Server Error: /api/v1/engagements/
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/tastypie/resources.py", line 227, in wrapper
    response = callback(request, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/tastypie/resources.py", line 467, in dispatch_list
    return self.dispatch('list', request, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/tastypie/resources.py", line 499, in dispatch
    response = method(request, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/tastypie/resources.py", line 1408, in post_list
    updated_bundle = self.obj_create(bundle, **self.remove_api_resource_names(kwargs))
  File "/usr/local/lib/python2.7/dist-packages/tastypie/resources.py", line 2245, in obj_create
    return self.save(bundle)
  File "/usr/local/lib/python2.7/dist-packages/tastypie/resources.py", line 2380, in save
    self.is_valid(bundle)
  File "/usr/local/lib/python2.7/dist-packages/tastypie/resources.py", line 1320, in is_valid
    errors = self._meta.validation.is_valid(bundle, bundle.request)
  File "/usr/local/lib/python2.7/dist-packages/tastypie/validation.py", line 73, in is_valid
    form = self.form_class(**self.form_args(bundle))
  File "/vagrant/dojo/api.py", line 85, in form_args
    pk = self._get_pk_from_resource_uri(rel_field, resource_uri)
  File "/vagrant/dojo/api.py", line 70, in _get_pk_from_resource_uri
    resource_uri, base_resource_uri))
Exception: Couldn't match resource_uri //api/v1/users/1/ with /api/v1/users/

I'm facing the same issue.. From what I see, I think it's referring to the API-v1. That might be the problem. However, I'm not sure.. Have you found a fix? If not, I'll be waiting for someone's response!

I am about to lose my mind
I am on master defectdojo, and master defectdojo-api. I have somehow managed to overcome the double slash issue that @sirferl has a workaround for. (My file looks nothing like https://github.com/aaronweaver/defectdojo_api/issues/18 by the way, but I will leave that for now)
I am left with the stack that is posted on the top of this issue by @raghunath24 . In the console I can see defectdojo logging and working like normal with no error messages. When I check the engagement in the user interface, there are ZERO findings from the engagement. However if I import the file manually, the findings are registered.
Has anyone managed to find a workaround for the AttributeError ? are the other reasons beside the AttributeError that make the import of the vulnerabilities fail ?
Output:
python dojo_ci_cd.py --product=9 --file "/dependency-check/xxx/dependency-check-report.json" --scanner="Dependency Check" --critical=0 --host=http://localhost:8000 --api_key=xxx --user=xxx
Traceback (most recent call last):
  File "dojo_ci_cd.py", line 242, in <module>
    class Main:
  File "dojo_ci_cd.py", line 283, in Main
    test_ids = processFiles(dd, engagement_id, file, scanner=scanner)
  File "dojo_ci_cd.py", line 145, in processFiles
    if test_id.success == False:
AttributeError: 'NoneType' object has no attribute 'success'

-----------

[18/Oct/2018 06:11:18] "GET /alerts/count HTTP/1.1" 200 13
[18/Oct/2018 06:11:20] "GET /alerts/json?limit=12 HTTP/1.1" 200 3211
[18/Oct/2018 06:11:22] "GET /product/9/engagements HTTP/1.1" 200 40963
[18/Oct/2018 06:11:25] "GET /alerts/json?limit=12 HTTP/1.1" 200 3211
[18/Oct/2018 06:11:30] "GET /product/9/engagements/cicd HTTP/1.1" 200 78469
[18/Oct/2018 06:11:41] "GET /alerts/count HTTP/1.1" 200 13
[18/Oct/2018 06:11:51] "GET /alerts/count HTTP/1.1" 200 13

Ok.. I just hardcoded scannerName = "Dependency Check Scan" on line 85 and that seems to work
Now if I can find away to hack my way around the TypeErrors :|


Search by type

What's New?

Point search endpoint does not allow filtering by pointer_type. Now it does.

Coverage remained the same at 91.449% when pulling b49b128c604c92c528c8c78a74abe93f0a2f41c0 on RickCarlino:search_by_type into 3bf9ea84160362bf53a03a2b038002eaa614f1bc on FarmBot:staging.


runCrossVal expand node names before extracting from samples matrix

runCrossVal formerly didn't expand node names, before using the node names to index the columns of a matrix of posterior samples.  This caused an error when extracting multivariate nodes, for example x[1:2], when indexing the columns as:
samples[, "x[1:2]" ]     ## error

@paciorek Pending testing and such, maybe this could also make it into the upcoming release.


Wrong configuration of vfat boot filesystems

Describe the bug
There's an evaluation error that prevents the nixos rebuild-switch command to succeed.
Steps To Reproduce
Steps to reproduce the behaviour:

Use the latest nixos-unstable branch of nixpkgs and have your boot partition in a vfat filesystem (or any other partition that needs to be loaded a initrd time)
Just do a nixos rebuild switch

Expected behavior
Configuration should build normally.
Error message
 error: A definition for option boot.initrd.systemd.extraBin is not of type attribute set of path. Definition values:
       - In /nix/store/35dcag44a0ymww0vy0s4jjgxwpv9g62d-source/nixos/modules/tasks/filesystems/vfat.nix:
           [
             <derivation dosfstools-4.2>
           ]

Additional context
The line that creates the problem was added in this commit: 18013cc71e8d00cc1297af29d519b420f1326b41.
Here is a link to the offeding file
I guess the author meant to write boot.initrd.systemd.initrdBin but I am not very knowledgeable about nix so I am not 100% sure of this.
Notify maintainers
@ElvishJerricco
Metadata
Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.
nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.5.7, NixOS, 23.11 (Tapir), 23.11.20231019.7c9cc5a`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.17.1`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

Fixed by: https://github.com/NixOS/nixpkgs/pull/263627


Allow chaining of workspaces

Though it is not recommended to chain workspaces it might be desirable to source a workspace that us not the currently sourced one. With the changes introduced in version 0.5.0 this possibility was removed. This commit re-adds that functionality. The .cur_env file is always read independent of whether the ROB_FOLDERS_ACTIVE_ENV variable is set or not.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 45.84%. Comparing base (077fd4d) to head (84ce0e1).

Additional details and impacted files
@@           Coverage Diff           @@
##             main      #24   +/-   ##
=======================================
  Coverage   45.84%   45.84%
=======================================
  Files          26       26
  Lines        1444     1444
=======================================
  Hits          662      662
  Misses        782      782

Flag
Coverage Δ

unittests
45.84% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

The current PR will cause the workspaces to become active at the same time:

Thank you @Oberacda! With the changes you provided, I think this is good to get merged.

I think when souring the virtual env after robot folders, or within the setup_local.sh file, one can still alter the env.

I think when souring the virtual env after robot folders, or within the setup_local.sh file, one can still alter the env.

That's probably a fair point. I've added some documentation on this in 8eb1526. @Oberacda, do you think that makes things clear?


Create an Account on the SAP Gateway Demo System

Tutorials: https://developers.sap.com/tutorials/gateway-demo-signup.html

I can't register the account, it always loading, I am the SAP APJ employee,
Write here what you need help with ...

The steps in the tutorial are working fine. When I go to the page it works correctly.  I'd suggest trying again.  Beyond that for technical support for this SAP Gateway Demo System, please email<EMAIL_ADDRESS>


Blank screen on iOS 3.1.2?

Even though the PhotoSwipe demos work correctly, I'm experiencing a blank screen (when clicking on a thumbnail to open PhotoSwipe) on an iPod Touch with iOS 3.1.2 installed. I'm wondering if anyone has experienced this and knows of a fix?
Example link is as follows: http://dev.hotheadgames.com/games/kard-combat/screenshots/

Nope sorry, just tested on an iphone 3g with iOS3 - seems ok


To which inference engine libraries must I link my code?

Hi everyone,
to do accelerated deep learning inference I need to use OpenVINO's inference engine from another C++ tool, called OpenFOAM. I programmed my inference code into my OpenFOAM executable, and compiled it. To compile within OpenFOAM I must use "wmake", which requires the header files and shared libraries. The headers I provided are here:
/opt/intel/openvino/inference_engine/include
The libraries I provided are here:
/opt/intel/openvino/inference_engine/lib/intel64/
My code compiled properly, however now if I run my application "tabulatedCombustionFoam" I get this error:
tabulatedCombustionFoam: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /home/elias/OpenFOAM/elias-4.1/platforms/linux64GccDPInt64Opt/lib/libinference_engine.so)
tabulatedCombustionFoam: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by /home/elias/OpenFOAM/elias-4.1/platforms/linux64GccDPInt64Opt/lib/libinference_engine.so)
tabulatedCombustionFoam: /lib64/libstdc++.so.6: version `CXXABI_1.3.8' not found (required by /home/elias/OpenFOAM/elias-4.1/platforms/linux64GccDPInt64Opt/lib/libinference_engine.so)
tabulatedCombustionFoam: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.22' not found (required by /home/elias/OpenFOAM/elias-4.1/platforms/linux64GccDPInt64Opt/lib/libinference_engine.so)
tabulatedCombustionFoam: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by /home/elias/OpenFOAM/elias-4.1/platforms/linux64GccDPInt64Opt/lib/libinference_engine.so)

I know the problem from another software, checking the versions gives me:
strings /lib64/libstdc++.so.6 | grep GLIBC

GLIBCXX_3.4
GLIBCXX_3.4.1
GLIBCXX_3.4.2
GLIBCXX_3.4.3
GLIBCXX_3.4.4
GLIBCXX_3.4.5
GLIBCXX_3.4.6
GLIBCXX_3.4.7
GLIBCXX_3.4.8
GLIBCXX_3.4.9
GLIBCXX_3.4.10
GLIBCXX_3.4.11
GLIBCXX_3.4.12
GLIBCXX_3.4.13
GLIBCXX_3.4.14
GLIBCXX_3.4.15
GLIBCXX_3.4.16
GLIBCXX_3.4.17
GLIBCXX_3.4.18
GLIBCXX_3.4.19
GLIBC_2.3
GLIBC_2.2.5
GLIBC_2.14
GLIBC_2.4
GLIBC_2.3.2
GLIBCXX_DEBUG_MESSAGE_LENGTH

strings /lib64/libstdc++.so.6 | grep CXXABI

CXXABI_1.3
CXXABI_1.3.1
CXXABI_1.3.2
CXXABI_1.3.3
CXXABI_1.3.4
CXXABI_1.3.5
CXXABI_1.3.6
CXXABI_1.3.7
CXXABI_TM_1

For the other software, the problem was that I must use gcc 4.8.5 on CentOS 7.6. However, there are two reasons why I do not understand OpenVINO's inference engine gives me this error here:

According to this page (https://software.intel.com/en-us/openvino-toolkit/choose-download/free-download-linux) OpenVINO should work with gcc >=3.4, and it is stated that it is compatible to CentOS
Most surprising: I did the entire installation with the same hard- and software. I was able to build all the Intel samples using cmake, I was also able to create my own code (a copy of hello_autoresize_classification) and run it. So obviously, there it works. I do not see why it did not require the new GLIBCXXs / GLIBCs and CXXABIs for my sample build, but now it does.

Thanks in advance for any help,
Elias

Environment:
CentOS Version: CentOS Linux release 7.6.1810 (Core)
gcc Version: gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-36)
OpenVINO version: 2019.1.144

Ok, now my errors have changed, so I decide to give an update directly. First of all, to show my inference engine code, here are the relevant parts of the class that I added it to:
Header: FPVANNModel.H
#ifndef FPVANNModel_H
#define FPVANNModel_H

//#include "OVInfer.H"
#include <iomanip>
#include <vector>
#include <memory>
#include <string>
#include <cstdlib>
#include <map>
#include <list>
#include <limits>
#include <random>
#include <cctype>
#include <functional>
#include <time.h>
#include <iostream>
#include <fstream>
#include <utility>
#include <algorithm>
#include <chrono>
#include <ie_plugin_dispatcher.hpp>
#include <ie_plugin_ptr.hpp>
#include <cpp/ie_cnn_net_reader.h>
#include <cpp/ie_infer_request.hpp>
#include <ie_device.hpp>
#include <ie_blob.h>
#include <opencv2/opencv.hpp>
#include <inference_engine.hpp>
[...]

CPP-file: FPVANNModel.C:
#include "FPVANNModel.H"
[...]
if(InferenceMode==3)  // OpenVINO
            {
             	auto t_start = std::chrono::high_resolution_clock::now();

                char cwd[256];
                getcwd(cwd,sizeof(cwd));
                const std::string case_directory(cwd);
                const std::string path_to_xml = "/data/FPV_ANN_Standard.xml";
                const std::string full_path = cwd + path_to_xml;
                //const char* binGraphName = full_path.c_str(); // if memory leak occurs maybe check this

                const size_t no_of_inputs = in_mean.size();
                const size_t no_of_outputs = out_mean.size();
                const size_t cellsAndPatches = (input_f_zeta_PVNorm.size())/no_of_inputs;

                InferenceEngine::InferencePlugin plugin = InferenceEngine::PluginDispatcher().getPluginByDevice("CPU");
                InferenceEngine::CNNNetReader network_reader;
                network_reader.ReadNetwork(full_path);
                network_reader.ReadWeights(full_path.substr(0,full_path.size()-4)+".bin");

                network_reader.getNetwork().setBatchSize(cellsAndPatches);
                InferenceEngine::CNNNetwork network = network_reader.getNetwork();

                InferenceEngine::InputInfo::Ptr input_info = network.getInputsInfo().begin()->second;
                std::string input_name = network.getInputsInfo().begin()->first;
                input_info->setLayout(InferenceEngine::Layout::NC);
                input_info->setPrecision(InferenceEngine::Precision::FP32);

InferenceEngine::DataPtr output_info = network.getOutputsInfo().begin()->second;
                std::string output_name = network.getOutputsInfo().begin()->first;
                output_info->setPrecision(InferenceEngine::Precision::FP32);

                InferenceEngine::ExecutableNetwork executable_network = plugin.LoadNetwork(network, {});
                InferenceEngine::InferRequest infer_request = executable_network.CreateInferRequest();

                cv::Mat combustionIn = cv::Mat(cellsAndPatches,no_of_inputs,CV_32FC1);
                for(size_t batchCounter=0;batchCounter<cellsAndPatches;batchCounter++)
                {
                    for(size_t inputCounter=0;inputCounter<no_of_inputs;inputCounter++)
                    {
                     	combustionIn.at<float>(batchCounter,inputCounter) = input_f_zeta_PVNorm.at(batchCounter*no_of_inputs+inputCounter);
                    }
                }

                InferenceEngine::TensorDesc tDescCombustion(InferenceEngine::Precision::FP32,{cellsAndPatches,no_of_inputs},InferenceEngine::Layout::NC);
                InferenceEngine::Blob::Ptr combustionBlob = InferenceEngine::make_shared_blob<float>(tDescCombustion,reinterpret_cast<float*>(combustionIn.data));
                infer_request.SetBlob(input_name,combustionBlob);

                infer_request.Infer();
                InferenceEngine::Blob::Ptr output = infer_request.GetBlob(output_name);
                InferenceEngine::Blob* pointerToOutput = output.get();

                float* decode_blob = pointerToOutput->buffer().as<float*>();

                output_real.clear();
                output_real.resize(cellsAndPatches*no_of_outputs);

for(size_t i=0;i<no_of_outputs;i++)
                {
                    output_real.at(i) = *decode_blob;
                    decode_blob++;
                }

                auto t_end = std::chrono::high_resolution_clock::now();
                auto total = std::chrono::duration<float, std::milli>(t_end - t_start).count();
                Info << "Total inference time ONNX Runtime: " << total << endl;
                float throughput = (cellsAndPatches/total)*1000;
                Info << "Total throughput OpenVINO Inference engine: " << throughput << endl;
}

As I mentioned I must compile the code using "wmake". This requires the headers and libraries (.so) that the code above needs. My CPUs support AVX2 and SSE4, therefore I would like to activate these.

Ok so the last error with the undefined symbol could be solved by not using OpenCV within the project. Now the code is running, but I am still unsure about the library usage... is it sufficient to link these libraries?
libinference_engine.so
libMKLDNNPlugin.so
libcpu_extension_avx2
libcpu_extension_sse4

Does that automatically activate avx2 and sse4 usage? And do I even have to install OpenVINO if I just want to use the inference engine? Or is it enough to include the required headers and link to the 4 libraries above?

Dear @el1995,
You can certainly build a standalone OpenVino application as long as you have all the required libraries.  Anything under inference-engine\bin is fair game for libs which need to be included in a standalone project.
But to make sure you got everything you need, please use a tool like binscan or Elf Library Viewer or in the case of Windows Dependency Walker
I'm sure you realize that these tools don't lie. If you have the required OpenVino libraries and their dependencies in your standalone project then there is no reason to install OpenVino on a your client's machine.
Hope it helps,
Thanks,
Shubha

Hello Shubha,
thanks for your reply, I will check asap with the recommended tools. I have two remaining questions regarding the linked so libraries:

During compilation and runtime, the inference engine asks for the library libcpu_extension. I did not find it in the inference engine libraries folder, insted it was located here:
/inference_engine_samples_build/intel64/Release/lib
I assume it is built together with the samples. Does it already activate the available instruction set architectures (AVX2 etc) or not? What is it required for?
I so far linked the libraries for avx2, sse4 etc to my code. Do they activate the ISAS or is it done with the library mentioned in 1)? How can I make sure I get the maximum ISAs my machine offers?

Best regards and thanks in advance!

Dear @el1995
Yeah you definitely need libcpu_extension also. And yes, it's built together with the samples. It's required for  all the extension code you see here (see below) - these are layer (wrapping  an operation) implementations on the CPU hardware.
https://github.com/opencv/dldt/tree/2019/inference-engine/src/extension
If your machine supports AVX512, an avx512 extension should be built automatically. The build system uses CPUID to figure out which ISAs your system supports. For instance please look at at this file:

inference-engine\src\extension\cmake\CPUID.cmake

So the burden is not upon you to determine whether the right ISAs are included. Rather it's on the Inference Engine and Extension code build system.
Hope it helps !
Shubha

Helped, thank you very much!


[FEATURE] Start of Tadukoo Look & Feel

Describe the solution you'd like
Create a new Look & Feel extending Metal Look & Feel, and override the default and required methods.
Is your feature request related to a problem? Please describe.
No
Describe alternatives you've considered
None
Additional context
Part of #16
Changes TO DO

[ ] Create TadukooLookAndFeel
[ ] Override String getName() to return "Tadukoo"
[ ] Override String getID() to return "Tadukoo"
[ ] Override String getDescription() to return "The Tadukoo Look and Feel"
[ ] Override boolean isNativeLookAndFeel() to return false
[ ] Override boolean isSupportedLookAndFeel() to return true (we're intending to make it cross-platform)

For the purposes of completing this now (since this was mostly done in Alpha v.0.1), this will just depend on Javadocing


lxc image copy :  fails with metadata.yaml error

I tried to do the following, it fails with a metadata.yaml error.
$ lxc image copy srv01:ubuntu/trusty/amd64 srv02: --debug
2015/07/14 10:07:52 fingering the daemon
2015/07/14 10:07:53 raw response: {"type":"sync","status":"Success","status_code":200,"metadata":{"api_compat":1,"auth":"trusted","config":{"core.trust_password":true},"environment":{"backing_fs":"btrfs","driver":"lxc","kernel_version":"3.19.0-21-generic","lxc_version":"1.1.2","lxd_version":"0.13"}}}
2015/07/14 10:07:53 pong received
2015/07/14 10:07:53 fingering the daemon
2015/07/14 10:07:54 raw response: {"type":"sync","status":"Success","status_code":200,"metadata":{"api_compat":1,"auth":"trusted","config":{"core.trust_password":true},"environment":{"backing_fs":"btrfs","driver":"lxc","kernel_version":"3.19.0-22-generic","lxc_version":"1.1.2","lxd_version":"0.13"}}}
2015/07/14 10:07:54 pong received
2015/07/14 10:07:54 raw response: {"type":"sync","status":"Success","status_code":200,"metadata":{"target":"451c1b479aa516a5d9106353cf9dd9fc6ed9220a5684da0a192950315caf00bb","description":"ubuntu/trusty/amd64"}}
2015/07/14 10:07:54 raw response: {"type":"sync","status":"Success","status_code":200,"metadata":{"aliases":[{"target":"ubuntu/trusty","description":"ubuntu/trusty"},{"target":"ubuntu/trusty/amd64","description":"ubuntu/trusty/amd64"}],"architecture":0,"fingerprint":"451c1b479aa516a5d9106353cf9dd9fc6ed9220a5684da0a192950315caf00bb","filename":"","properties":{},"public":0,"size":137758073,"created_at":0,"expires_at":0,"uploaded_at":1436618208}}
2015/07/14 10:08:07 raw response: {"error":"Could not extract image metadata metadata.yaml from tar: exit status 2 (tar: metadata.yaml: Nicht im Archiv gefunden.)","error_code":500,"type":"error"}

2015/07/14 10:08:07 Error caught in main: *errors.errorString
error: Could not extract image metadata metadata.yaml from tar: exit status 2 (tar: metadata.yaml: Nicht im Archiv gefunden.)

lxc and both lxd are on latest master.

Same with lxc image import, image.tar contains a metadata.yaml
pcdummy@UbuntuPro:~/Downloads$ lxc image import image.tar srv02: --debug
2015/07/14 10:22:33 fingering the daemon
2015/07/14 10:22:34 raw response: {"type":"sync","status":"Success","status_code":200,"metadata":{"api_compat":1,"auth":"trusted","config":{"core.trust_password":true},"environment":{"backing_fs":"btrfs","driver":"lxc","kernel_version":"3.19.0-22-generic","lxc_version":"1.1.2","lxd_version":"0.13"}}}
2015/07/14 10:22:34 pong received
2015/07/14 10:22:43 raw response: {"error":"Could not extract image metadata metadata.yaml from tar: exit status 2 (tar: metadata.yaml: Nicht im Archiv gefunden.)","error_code":500,"type":"error"}

2015/07/14 10:22:43 Error caught in main: *errors.errorString
error: Could not extract image metadata metadata.yaml from tar: exit status 2 (tar: metadata.yaml: Nicht im Archiv gefunden.)

This fixes that error:
diff --git a/lxd/images.go b/lxd/images.go
index 2169683..6000be7 100644
--- a/lxd/images.go
+++ b/lxd/images.go
@@ -572,7 +572,7 @@ func xzReader(r io.Reader) io.ReadCloser {
 }

 func getImageMetadata(fname string) (*imageMetadata, error) {
-       metadataName := "metadata.yaml"
+       metadataName := "/metadata.yaml"

        compression, _, err := detectCompression(fname)

But metadata.yaml is json not yaml ?

Yeah scripts/lxd-image
    # Add the metadata file
    metadata_yaml = json.dumps(metadata, sort_keys=True,
                               indent=4, separators=(',', ': '),
                               ensure_ascii=False).encode('utf-8') + b"\n"

    metadata_file = tarfile.TarInfo()
    metadata_file.size = len(metadata_yaml)
    metadata_file.name = "metadata.yaml"
    target_tarball.addfile(metadata_file,
                           io.BytesIO(metadata_yaml))

json is conveniently a subset of yaml so a json document will be successfully parsed by a yaml parser.
As python doesn't come with a yaml parser by default and I didn't want to depend on external module, I've simply been using the json parser/writer instead.
Anyway, with that branch merged, I think we can close this issue now.
Thanks for the fix!


Autocomplete does not work at times

Does this issue occur when all extensions are disabled?: Yes/No

VS Code Version:
OS Version:

Steps to Reproduce:

Attached video to demonstrate the steps to replicate:

https://user-images.githubusercontent.com/3875065/135715659-0c3c437b-5e3d-43f5-a098-8bef3a1940ae.mov

VS Code Version: 1.60.2
OS: macOS 11.6 20G165 x86_64

I think it's a normal behaviour that autocomplete doesn't work when you are using a snippet?

@jaideeprana I had a same problem... It's because of builtin typescript & JavaScript language features extension
solution #134321

Try setting "editor.suggest.snippetsPreventQuickSuggestions": false

@mjbvz that doesn't fix the issue, the cause of issue is built-in vs code typescript & JavaScript language features extension #134321

@ullaskunder3 The video shows snippets blocking completions. You're talking about a different issue with a different cause

Thanks @mjbvz and @ullaskunder3 . I appreciate both of you reaching out to help me and responding to the issue so fast.
I can confirm that the fix proposed by @mjbvz has fixed the issue for me.
However I have one question @mjbvz why is this setting on the disabled or false by default?


Search

What is the feature?
Search place nearby by google autocomplete js API
Linked place list to place detail
Caching API data
Download image from each place that has an image so we won't call google map image API again.
What is the solution?
I use the google JS autocomplete in search.
I have a problem that google Maps API is costly so I have to find a caching system that can cache data and an image but I couldn't find one, so I created it by myself.
What areas of the site does it impact?
place_detail: cache place detail data
Reviewer Guide
I recommend you to start it in the beginning and follow the readme.md that we have
Screen shot

First time load place_list page

Second time load place_list page

First time load place_detail page

Second-time load place_detail page

Codecov Report

Merging #46 (378e0dd) into beta (4484c58) will decrease coverage by 15.89%.
The diff coverage is 49.86%.

@@             Coverage Diff             @@
##             beta      #46       +/-   ##
===========================================
- Coverage   81.27%   65.37%   -15.90%
===========================================
  Files          18       24        +6
  Lines         486      699      +213
  Branches       27       65       +38
===========================================
+ Hits          395      457       +62
- Misses         73      219      +146
- Partials       18       23        +5

Impacted Files
Coverage Δ

threpose/settings.py
100.00% <ø> (ø)

trip/urls.py
100.00% <ø> (ø)

search/views.py
20.00% <20.00%> (ø)

src/caching/caching_gmap.py
36.36% <36.36%> (ø)

trip/views.py
53.47% <43.10%> (-14.20%)
:arrow_down:

search/api.py
52.38% <52.38%> (ø)

users/views.py
60.00% <60.00%> (ø)

users/utils.py
78.94% <78.94%> (ø)

users/tests.py
94.59% <94.59%> (ø)

search/admin.py
100.00% <100.00%> (ø)

... and 6 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4484c58...378e0dd. Read the comment docs.

I think when place don't have some information it cause this error

Also this too


Learning paths

First draft. To take a look at learning paths.
Two questions:

Is it worth having?
Does it look ok if it is?

Moving to review, still in draft. Any further feedback?

Overall, LGMT. But I don't think is necessary to have the location in the tables. The type should be enough. As the table has the links, I do not see the purpose to have the link path in the table. I'll never copy it if I have the link in the same row.

Yeah, still messing around with the available data and how to present it best. I think I need to work on a more tree like presentation.


Agdroid 469 instance

@matzew @danielpassos :eyes: ?
What has changed

I've changed from using gcm.register to using InstanceId.  See: AGDROID-469
I've made senderId a single value instead of a set.  See: AG ML Thread
As a "bonus" this PR also also handles AGDROID-351.

How to test

Make sure you have the latest Android SDK installed and is fully up to date
Make sure you have run the latest maven-android-sdk-deployer
Make sure you have the latest aerogear-android-core, and -pipe
Run the build while an android device or emulator running the Google Play APIs is connected
You can also update the various helloworld/push quickstarts to use 3.0.0-SNAPSHOT and they should work as is.

What to do if you have trouble

Bother me on email
IRC
This issue :)

cool stuff, summers - I will take a look on Friday or Monday (PTO today)

:eyes:

:heavy_check_mark: Build
:heavy_check_mark: Tests
:heavy_minus_sign: Receive message using Unified Push Helloworld
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] AeroGear Android Push Parent ...................... SUCCESS [2.950s]
[INFO] AeroGear Android Push Library ..................... SUCCESS [12.168s]
[INFO] AeroGear Android Push Test ........................ SUCCESS [1:03.805s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1:42.419s
[INFO] Finished at: Tue Jul 14 11:31:58 BRT 2015
[INFO] Final Memory: 48M/141M
[INFO] ------------------------------------------------------------------------

Seems my device was already registered in the app I used to test. Let me test it again using a clean UPS/Wildfly installation.

@danielpassos I did have a problem using the version on my OpenShift container.  It was getting a Hibernate validation error when it tried to save the registration token.  This went away when I used the latest version of UPS locally.

I did a new test using a clean UPS/Wildfly installation but got it:
[12:17:27,703 INFO  [NotificationRouter] (default task-13) Processing send request with '[alert=Test - instanceID API, criteria=Criteria{aliases=null, deviceTypes=null, categories=null, variants=null}, time-to-live=Config{timeToLive=-1}]' payload
12:17:28,096 INFO  [PushNotificationSenderEndpoint] (default task-13) Message submitted to PushNetworks for further processing
12:17:28,322 INFO  [GCMPushNotificationSender] (Thread-53 (HornetQ-client-global-threads-376723932)) Sending payload for [1] devices to GCM
12:17:29,169 SEVERE [GCMPushNotificationSender] (Thread-53 (HornetQ-client-global-threads-376723932)) Error sending payload to GCM server
12:17:29,174 WARNING [NotificationDispatcher] (Thread-53 (HornetQ-client-global-threads-376723932)) Error on 'android' delivery

@matzew Any thoughts?

I've created the new app in Google console and the server error stopped. But I can't receive the message in the app
12:25:39,677 INFO  [NotificationRouter] (default task-60) Processing send request with '[alert=Test 2 - InstanceID, criteria=Criteria{aliases=null, deviceTypes=null, categories=null, variants=null}, time-to-live=Config{timeToLive=-1}]' payload
12:25:39,760 INFO  [PushNotificationSenderEndpoint] (default task-60) Message submitted to PushNetworks for further processing
12:25:39,802 INFO  [GCMPushNotificationSender] (Thread-55 (HornetQ-client-global-threads-376723932)) Sending payload for [1] devices to GCM
12:25:40,156 INFO  [GCMPushNotificationSender] (Thread-55 (HornetQ-client-global-threads-376723932)) Message batch to GCM has been submitted

It was my fault, I forgot update the senderID in the app :D

:+1:

@matzew Says feel free to merge and he will file bugs later ;)


Implement return / break / continue loop fold operators for f*

#988 introduced new fold operators (a _cf and _return version for fold_enumerated_chunked_slice, fold_enumerated_slice, fold_range_step_by, fold_range, Core__iter__traits__iterator__Iterator__fold, while_loop) but they are not provided by the fstar core lib. We should provide a definition for each of them (it can even be an assume val for the beginning).

How is this related to #1171? The description here sounds like this should be working for while loops.

How is this related to #1171? The description here sounds like this should be working for while loops.

I don't think they are related.