Dataset Preparation and Training on GPU

CL_PATH = 'gs://bucket-name/VGGFace2_classlabels.txt'
DATASET_PATH = 'gs://bucket-name/VGGFace2_tfrecs'
TB_PATH = 'gs://bucket-name/vggface2_deepface_tensorboard'

What locations are above files available online ? I tried getting them from http://www.robots.ox.ac.uk/~vgg/data/vgg_face2/data_infor.html
but I am getting confused by your terms  VGGFace2_classlabels, VGGFace2_tfrecs, vggface2_deepface_tensorboard. what do these terms mean ?

hi
any hints ?

The VGGFace2 dataset (train_data_v1) have been used to train the DeepFace model. The license for the dataset does not allow any form of redistribution of this dataset although it is a publicly available dataset.
The DeepFace model was trained on Cloud TPUs using tf.keras which required us to prepare TFRecords from the face crops available from the dataset. 2D alignment had been used to align the faces in each of the images of the dataset and tight crops were used. More information regarding the alignment and face detection step (using dlib) is mentioned here. In the process of detection and alignment many of the images in the VGGFace2 train dataset were dropped as they lacked frontal posed faces yet, we could retain approx. 2.3 million images for training the model.
Pretrained weights for DeepFace model are available [here].(https://github.com/swghosh/DeepFace/releases)
If you are interested in retraining the DeepFace model using VGGFace2 dataset maybe on Cloud TPUs or on GPUs you can prepare TFRecords using this script. And, then those TFRecords can be uploaded to GCS bucket for model training. (Cloud TPUs require that the data either fits on memory of the Compute Engine VM or alternatively data be loaded from TFRecords based tf.data pipelines from data stored in GCS buckets of your bucket; not necessary for tf.keras on GPUs)
Please refer to this Google Codelabs tutorial for more information. https://codelabs.developers.google.com/codelabs/keras-flowers-data/
As per your actual query,
CL_PATH refers to a simple text file placed on a GCS bucket that contains list of class labels for the VGGFace2 (train_data_v1) dataset. One class label per line for easy parsing in the data pipeline.
DATASET_PATH  refers to a directory in the GCS bucket where TFRecords are stored. Each TFRecords contains multiple images and are used in the data pipeline.
TB_PATH is a directory that you specify on a GCS bucket which will be used to store TensorBoard logs during model training on Cloud TPUs.
If you are planning to train your own network, you have to set these paths on the basis of buckets created on your own GCP project.

I am astound by your knowledge on this topic. I use 1080ti as a training machine and its my personal machine so I can leave training on it for weeks. How do make a python script train.py which is not reliant on google colab ? any pointers would really help

Thanks @ajinkya933!
Yes, definitely you can use the same training script (train_on_tpu.py) with some minor modifications to train a DeepFace network using GPUs on your personal machine.
I looked up the following:
NVIDIA 1080Ti has approx. 11.34 FP32 TFLOPs while Cloud TPU v3 (hardware accelerator on which I trained the model) has 420 FP32 TFLOPs which tells me that on your personal system, a few weeks should be more than fine. Model training for 15 epochs (as per paper) took me around
6.5 hours. Epoch-wise training details are available here.
In order to train the model on GPUs, you need to remove the following lines of code (which is related to TPUStrategy):
https://github.com/swghosh/DeepFace/blob/94558ee5790dd6efc4c0bdc5a5605d55ef81fb8f/train_on_tpu.py#L36-L38
The strategy.scope() context manager should be removed and deepface.create_deepface() should be called directly in global scope.
https://github.com/swghosh/DeepFace/blob/94558ee5790dd6efc4c0bdc5a5605d55ef81fb8f/train_on_tpu.py#L73-L74
For ideal training results, please make sure to use the exact same hyper-parameters as given in the script because they have been carefully chosen from the training process given in the paper.Using 2D-aligned face crops (using detection - dlib.get_frontal_face_detector, alignment - dlib.get_face_chip), 90.34% accuracy was achievable on the validation set. (which is reasonably good)
Since, tf.keras APIs are being used, a TFRecords based pipeline ensures maximum performance for the training pipeline. It is advisable that you create TFRecords (in that case, you can set DATASET_PATH = ./local-folder/tfrecs', and CL_PATH = './local-folder/classlabels.txt') out of the images in the dataset for effective training. Alternatively, you can also use the tf.keras.preprocessing.image.ImageDataGenerator with model.fit_generator method for training the model.
Let me know if you are facing any issues regarding the setting up of data pipelines for model training or face detection-alignment step.

Good luck for your model training.
Let me know if you are facing any issues regarding the setting up of data pipelines for model training or face detection-alignment step.
Optionally, it'll be great if you fork this repo and send a PR for our upstream master branch based on the training script (same can be added to contrib/ folder) you use to train the DeepFace network on GPUs.

@swghosh I have converted TPU code to GPU and have created a pull request for the same. I have a different question on my PC my GPU has 11 GB RAM and system has 60 GB RAM. However the batch size of 1024 in your code is too large for me, heck even 400 batch size is large for me. So I tried with batch size 200. and I get this output:
WARNING:tensorflow:From /home/serveradmin/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/keras/initializers.py:143: calling RandomNormal.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
Model: "DeepFace"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
C1 (Conv2D)                  (None, 142, 142, 32)      11648
_________________________________________________________________
M2 (MaxPooling2D)            (None, 71, 71, 32)        0
_________________________________________________________________
C3 (Conv2D)                  (None, 63, 63, 16)        41488
_________________________________________________________________
L4 (LocallyConnected2D)      (None, 55, 55, 16)        62774800
_________________________________________________________________
L5 (LocallyConnected2D)      (None, 25, 25, 16)        7850000
_________________________________________________________________
L6 (LocallyConnected2D)      (None, 21, 21, 16)        2829456
_________________________________________________________________
F0 (Flatten)                 (None, 7056)              0
_________________________________________________________________
F7 (Dense)                   (None, 4096)              28905472
_________________________________________________________________
D0 (Dropout)                 (None, 4096)              0
_________________________________________________________________
F8 (Dense)                   (None, 8631)              35361207
=================================================================
Total params: 137,774,071
Trainable params: 137,774,071
Non-trainable params: 0
_________________________________________________________________
2019-12-28 14:00:02.204872: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:01:00.0
2019-12-28 14:00:02.204975: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2019-12-28 14:00:02.205000: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
2019-12-28 14:00:02.205021: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10.0
2019-12-28 14:00:02.205040: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10.0
2019-12-28 14:00:02.205057: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10.0
2019-12-28 14:00:02.205076: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10.0
2019-12-28 14:00:02.205098: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-12-28 14:00:02.205763: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2019-12-28 14:00:02.205826: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-12-28 14:00:02.205837: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0
2019-12-28 14:00:02.205845: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N
2019-12-28 14:00:02.206572: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10286 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:01:00.0, compute capability: 7.5)
Epoch 1/15
2019-12-28 14:01:57.878154: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
2019-12-28 14:01:58.427627: W ./tensorflow/core/framework/model.h:213] Encountered a stop event that was not preceded by a start event.
2019-12-28 14:02:06.919931: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
Traceback (most recent call last):
  File "train_on_tpu.py", line 84, in <module>
    callbacks=cbs, epochs=EPOCHS)
  File "/home/serveradmin/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py", line 780, in fit
    steps_name='steps_per_epoch')
  File "/home/serveradmin/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/keras/engine/training_arrays.py", line 274, in model_iteration
    batch_outs = f(actual_inputs)
  File "/home/serveradmin/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/keras/backend.py", line 3292, in __call__
    run_metadata=self.run_metadata)
  File "/home/serveradmin/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1458, in __call__
    run_metadata_ptr)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
  (0) Invalid argument: logits and labels must be broadcastable: logits_size=[200,8631] labels_size=[200,3141890]
	 [[{{node loss/F8_loss/softmax_cross_entropy_with_logits}}]]
	 [[metrics/acc/Identity/_115]]
  (1) Invalid argument: logits and labels must be broadcastable: logits_size=[200,8631] labels_size=[200,3141890]
	 [[{{node loss/F8_loss/softmax_cross_entropy_with_logits}}]]
0 successful operations.
0 derived errors ignored.

What other parameters should I change along with the batch size to make it more compatible on my machine ?

You can use BATCH_SIZE = 128 (the training process in the paper uses batch size as 128 itself) if there are any GPU memory issues with your model. The SGD learning rate in the training script automatically adjusts itself on the basis of larger/smaller batch sizes. (https://stackoverflow.com/a/53046624)

WARNING:tensorflow:From /home/serveradmin/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/keras/initializers.py:143: calling RandomNormal.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
Model: "DeepFace"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
C1 (Conv2D)                  (None, 142, 142, 32)      11648
_________________________________________________________________
M2 (MaxPooling2D)            (None, 71, 71, 32)        0
_________________________________________________________________
C3 (Conv2D)                  (None, 63, 63, 16)        41488
_________________________________________________________________
L4 (LocallyConnected2D)      (None, 55, 55, 16)        62774800
_________________________________________________________________
L5 (LocallyConnected2D)      (None, 25, 25, 16)        7850000
_________________________________________________________________
L6 (LocallyConnected2D)      (None, 21, 21, 16)        2829456
_________________________________________________________________
F0 (Flatten)                 (None, 7056)              0
_________________________________________________________________
F7 (Dense)                   (None, 4096)              28905472
_________________________________________________________________
D0 (Dropout)                 (None, 4096)              0
_________________________________________________________________
F8 (Dense)                   (None, 8631)              35361207
=================================================================
Total params: 137,774,071
Trainable params: 137,774,071
Non-trainable params: 0
_________________________________________________________________
2019-12-28 14:11:55.720011: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:01:00.0
2019-12-28 14:11:55.720114: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2019-12-28 14:11:55.720138: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
2019-12-28 14:11:55.720161: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10.0
2019-12-28 14:11:55.720180: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10.0
2019-12-28 14:11:55.720198: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10.0
2019-12-28 14:11:55.720218: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10.0
2019-12-28 14:11:55.720238: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-12-28 14:11:55.720933: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2019-12-28 14:11:55.720996: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-12-28 14:11:55.721007: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0
2019-12-28 14:11:55.721015: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N
2019-12-28 14:11:55.721831: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10286 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:01:00.0, compute capability: 7.5)
Epoch 1/15
2019-12-28 14:13:50.959745: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
2019-12-28 14:13:51.293027: W ./tensorflow/core/framework/model.h:213] Encountered a stop event that was not preceded by a start event.
2019-12-28 14:14:00.278514: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
Traceback (most recent call last):
  File "train_on_tpu.py", line 84, in <module>
    callbacks=cbs, epochs=EPOCHS)
  File "/home/serveradmin/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py", line 780, in fit
    steps_name='steps_per_epoch')
  File "/home/serveradmin/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/keras/engine/training_arrays.py", line 274, in model_iteration
    batch_outs = f(actual_inputs)
  File "/home/serveradmin/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/keras/backend.py", line 3292, in __call__
    run_metadata=self.run_metadata)
  File "/home/serveradmin/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1458, in __call__
    run_metadata_ptr)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
  (0) Invalid argument: logits and labels must be broadcastable: logits_size=[200,8631] labels_size=[200,3141890]
	 [[{{node loss/F8_loss/softmax_cross_entropy_with_logits}}]]
	 [[loss/mul/_113]]
  (1) Invalid argument: logits and labels must be broadcastable: logits_size=[200,8631] labels_size=[200,3141890]
	 [[{{node loss/F8_loss/softmax_cross_entropy_with_logits}}]]
0 successful operations.
0 derived errors ignored.

I'm looking into the error that you have shared regarding training on a GPU. Actually, I'm trying to replicate the same error by running the script on a system with NVIDIA GPU.
Will help you with the same in sometime.
Meanwhile, can you please share the script which you used to create TFRecords for your dataset? I suspect that the error is related to mismatch of train labels size with the number of units in output dense layer.

For tf records I use https://github.com/swghosh/tfrecords-faster/blob/master/tfrecsfaster/prepare_tfrecords.py

CL_PATH refers to a simple text file placed on a GCS bucket that contains list of class labels for the VGGFace2 (train_data_v1) dataset. One class label per line for easy parsing in the data pipeline.

I think there is some problem related to the data pipeline itself which is causing such an error. Can you please share the class_labels file with me?
https://github.com/swghosh/DeepFace/blob/72b194edc9a0bd288db108770e74bd6018dac15c/contrib/train_on_gpu.py#L25
The class labels file that I used for training, can be generated using a simple script like this:
import glob
class_labels = glob.glob('/path/to/VGGFace2_train_v1/train')

with open('VGGFace2-class_labels.txt', 'w') as txt_file:
     for cl in class_labels:
          print(cl, file=txt_file)

Meanwhile, you can look at this notebook (try if you can find something lurking inside) where I did try to train the model on a Colab GPU. https://colab.research.google.com/drive/1ewJTfQLlxYXmeszRs4KllgRhTaBZ455H

@ajinkya933 I was not able to reproduce the error that you have shared, instead on a Colab GPU, OOM (out of GPU memory) issues are reported even when using a batch size of 128 images. (although after a very long wait time)
https://colab.research.google.com/drive/1ewJTfQLlxYXmeszRs4KllgRhTaBZ455H
The Colab environment provided a NVIDIA Tesla P4 GPU with approx. 8GB of GPU memory.

https://drive.google.com/file/d/1JP9TXKj1OUpACecGn2Q6zHRpk-GWBPzk/view?usp=sharing   link to train_images_v1.txt

CL_PATH refers to a simple text file placed on a GCS bucket that contains list of class labels for the VGGFace2 (train_data_v1) dataset. One class label per line for easy parsing in the data pipeline.

The class labels file that I used for training, can be generated using a simple script like this:
import glob
class_labels = glob.glob('/path/to/VGGFace2_train_v1/train')

with open('VGGFace2-class_labels.txt', 'w') as txt_file:
     for cl in class_labels:
          print(cl, file=txt_file)

You have to use the path (please set CL_PATH accordingly) to a simple text file that has one class label per line (without repetitions or file names). In your case you're trying to use the VGGFace2_train_v1 meta data file which is why it is causing the error.
A text file like this should work perfectly fine with the data pipeline:
n00002
n00003
n00004
...

I'm sharing the class labels file (specifically, for VGGFace2 train_v1 dataset) with you so that you can directly use it with your training script.
https://storage.googleapis.com/shrill-anstett-us/VGGFace2_test_classslabels.txt

I am able to initiate the script, but then its abruptly stopped with the following error:
2019-12-30 11:04:21.558273: I tensorflow/core/common_runtime/bfc_allocator.cc:816] Sum Total of in-use chunks: 10.04GiB
2019-12-30 11:04:21.558284: I tensorflow/core/common_runtime/bfc_allocator.cc:818] total_region_allocated_bytes_:<PHONE_NUMBER>2 memory_limit_:<PHONE_NUMBER>7 available bytes: 205 curr_region_allocation_bytes_:<PHONE_NUMBER>6
2019-12-30 11:04:21.558298: I tensorflow/core/common_runtime/bfc_allocator.cc:824] Stats:
Limit:<PHONE_NUMBER>7
InUse:<PHONE_NUMBER>2
MaxInUse:<PHONE_NUMBER>2
NumAllocs:                    1169
MaxAllocSize:<PHONE_NUMBER>

2019-12-30 11:04:21.558360: W tensorflow/core/common_runtime/bfc_allocator.cc:319] ****************************************************************************************************
2019-12-30 11:04:21.558383: W tensorflow/core/framework/op_kernel.cc:1502] OP_REQUIRES failed at strided_slice_op.cc:137 : Resource exhausted: OOM when allocating tensor with shape[1024,16,9,9] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc

nvidia-smi

I think batch size is the issue here, I am testing with multiple batch sizes which are (multiples of 8). Also when I stop the code the GPU memory is not getting freed as a result I need to restart my system each time I stop the training script.

Please try with batch size 128 or lesser (multiples of 8, as you said), once the GPU memory is freed. (With tf.keras APIs, sometimes GPU memory is not released when script terminates, I've faced it somewhere else)

Its a bit convoluted issue not related to batch size I tried batch size 128 and 16. However I get this message:
for batchsize 128
(tensorflow_p36) serveradmin@weng:~/Documents/ajinkya/output/DeepFace$ python train_on_tpu.py
/home/serveradmin/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/serveradmin/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/serveradmin/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/serveradmin/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/serveradmin/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/serveradmin/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
/home/serveradmin/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/serveradmin/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/serveradmin/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/serveradmin/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/serveradmin/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/serveradmin/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
WARNING:tensorflow:From /home/serveradmin/Documents/ajinkya/output/DeepFace/deepface/dataset.py:34: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

2019-12-30 11:22:05.569776: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2019-12-30 11:22:05.618271: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:01:00.0
2019-12-30 11:22:05.620973: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2019-12-30 11:22:05.657401: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
2019-12-30 11:22:05.675288: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10.0
2019-12-30 11:22:05.681490: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10.0
2019-12-30 11:22:05.722576: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10.0
2019-12-30 11:22:05.749454: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10.0
2019-12-30 11:22:05.824545: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-12-30 11:22:05.826274: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2019-12-30 11:22:05.826790: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-12-30 11:22:06.197532: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x564c3d9ffed0 executing computations on platform CUDA. Devices:
2019-12-30 11:22:06.197581: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): GeForce RTX 2080 Ti, Compute Capability 7.5
2019-12-30 11:22:06.230851: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency:<PHONE_NUMBER> Hz
2019-12-30 11:22:06.233911: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x564c3c680f00 executing computations on platform Host. Devices:
2019-12-30 11:22:06.233953: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
2019-12-30 11:22:06.235353: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:01:00.0
2019-12-30 11:22:06.235438: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2019-12-30 11:22:06.235475: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
2019-12-30 11:22:06.235505: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10.0
2019-12-30 11:22:06.235537: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10.0
2019-12-30 11:22:06.235567: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10.0
2019-12-30 11:22:06.235597: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10.0
2019-12-30 11:22:06.235628: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-12-30 11:22:06.237791: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2019-12-30 11:22:06.238414: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2019-12-30 11:22:06.239770: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-12-30 11:22:06.239788: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0
2019-12-30 11:22:06.239795: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N
2019-12-30 11:22:06.241568: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10286 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:01:00.0, compute capability: 7.5)
2019-12-30 11:22:06.285261: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
WARNING:tensorflow:From /home/serveradmin/Documents/ajinkya/output/DeepFace/deepface/dataset.py:56: The name tf.parse_single_example is deprecated. Please use tf.io.parse_single_example instead.

WARNING:tensorflow:From /home/serveradmin/Documents/ajinkya/output/DeepFace/deepface/dataset.py:45: The name tf.image.resize_image_with_pad is deprecated. Please use tf.compat.v1.image.resize_image_with_pad instead.

2019-12-30 11:22:06.706375: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:01:00.0
2019-12-30 11:22:06.706450: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2019-12-30 11:22:06.706484: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
2019-12-30 11:22:06.706498: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10.0
2019-12-30 11:22:06.706513: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10.0
2019-12-30 11:22:06.706526: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10.0
2019-12-30 11:22:06.706540: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10.0
2019-12-30 11:22:06.706555: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-12-30 11:22:06.707181: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2019-12-30 11:22:06.707226: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-12-30 11:22:06.707236: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0
2019-12-30 11:22:06.707245: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N
2019-12-30 11:22:06.707907: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10286 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:01:00.0, compute capability: 7.5)
train.num_classes ==  501
validate.num_classes ==  501
WARNING:tensorflow:From /home/serveradmin/Documents/ajinkya/output/DeepFace/deepface/deepface.py:19: The name tf.keras.initializers.RandomNormal is deprecated. Please use tf.compat.v1.keras.initializers.RandomNormal instead.

WARNING:tensorflow:From /home/serveradmin/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/keras/initializers.py:143: calling RandomNormal.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
Model: "DeepFace"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
C1 (Conv2D)                  (None, 142, 142, 32)      11648
_________________________________________________________________
M2 (MaxPooling2D)            (None, 71, 71, 32)        0
_________________________________________________________________
C3 (Conv2D)                  (None, 63, 63, 16)        41488
_________________________________________________________________
L4 (LocallyConnected2D)      (None, 55, 55, 16)        62774800
_________________________________________________________________
L5 (LocallyConnected2D)      (None, 25, 25, 16)        7850000
_________________________________________________________________
L6 (LocallyConnected2D)      (None, 21, 21, 16)        2829456
_________________________________________________________________
F0 (Flatten)                 (None, 7056)              0
_________________________________________________________________
F7 (Dense)                   (None, 4096)              28905472
_________________________________________________________________
D0 (Dropout)                 (None, 4096)              0
_________________________________________________________________
F8 (Dense)                   (None, 8631)              35361207
=================================================================
Total params: 137,774,071
Trainable params: 137,774,071
Non-trainable params: 0
_________________________________________________________________
2019-12-30 11:22:20.594258: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:01:00.0
2019-12-30 11:22:20.594335: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2019-12-30 11:22:20.594372: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
2019-12-30 11:22:20.594389: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10.0
2019-12-30 11:22:20.594406: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10.0
2019-12-30 11:22:20.594421: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10.0
2019-12-30 11:22:20.594446: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10.0
2019-12-30 11:22:20.594466: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-12-30 11:22:20.595099: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2019-12-30 11:22:20.595152: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-12-30 11:22:20.595162: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0
2019-12-30 11:22:20.595170: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N
2019-12-30 11:22:20.595843: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10286 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:01:00.0, compute capability: 7.5)
Epoch 1/15
2019-12-30 11:24:04.661212: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
2019-12-30 11:24:05.193815: W ./tensorflow/core/framework/model.h:213] Encountered a stop event that was not preceded by a start event.
2019-12-30 11:24:05.359139: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
Traceback (most recent call last):
  File "train_on_tpu.py", line 84, in <module>
    callbacks=cbs, epochs=EPOCHS)
  File "/home/serveradmin/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py", line 780, in fit
    steps_name='steps_per_epoch')
  File "/home/serveradmin/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/keras/engine/training_arrays.py", line 274, in model_iteration
    batch_outs = f(actual_inputs)
  File "/home/serveradmin/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/keras/backend.py", line 3292, in __call__
    run_metadata=self.run_metadata)
  File "/home/serveradmin/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1458, in __call__
    run_metadata_ptr)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
  (0) Invalid argument: logits and labels must be broadcastable: logits_size=[128,8631] labels_size=[128,501]
	 [[{{node loss/F8_loss/softmax_cross_entropy_with_logits}}]]
	 [[loss/mul/_113]]
  (1) Invalid argument: logits and labels must be broadcastable: logits_size=[128,8631] labels_size=[128,501]
	 [[{{node loss/F8_loss/softmax_cross_entropy_with_logits}}]]
0 successful operations.
0 derived errors ignored.

for batch size 16:
same error but with changed logits_size
  (0) Invalid argument: logits and labels must be broadcastable: logits_size=[16,8631] labels_size=[16,501]
	 [[{{node loss/F8_loss/softmax_cross_entropy_with_logits}}]]
	 [[loss/mul/_113]]
  (1) Invalid argument: logits and labels must be broadcastable: logits_size=[16,8631] labels_size=[16,501]
	 [[{{node loss/F8_loss/softmax_cross_entropy_with_logits}}]]
0 successful operations.
0 derived errors ignored.

To clear memory use  rmmod nvidia

The problem is still there with your data pipeline and not with the model. (for some reason)
The last dense layer of the DeepFace model has 8631 units. (equal to the set NUM_CLASSES)
In your case, it is 501.

for batchsize 128
train.num_classes ==  501
validate.num_classes ==  501
Traceback (most recent call last):
  File "train_on_tpu.py", line 84, in <module>
    callbacks=cbs, epochs=EPOCHS)
  File "/home/serveradmin/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py", line 780, in fit
    steps_name='steps_per_epoch')
  File "/home/serveradmin/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/keras/engine/training_arrays.py", line 274, in model_iteration
    batch_outs = f(actual_inputs)
  File "/home/serveradmin/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/keras/backend.py", line 3292, in __call__
    run_metadata=self.run_metadata)
  File "/home/serveradmin/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1458, in __call__
    run_metadata_ptr)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
  (0) Invalid argument: logits and labels must be broadcastable: logits_size=[128,8631] labels_size=[128,501]
	 [[{{node loss/F8_loss/softmax_cross_entropy_with_logits}}]]
	 [[loss/mul/_113]]
  (1) Invalid argument: logits and labels must be broadcastable: logits_size=[128,8631] labels_size=[128,501]
	 [[{{node loss/F8_loss/softmax_cross_entropy_with_logits}}]]
0 successful operations.
0 derived errors ignored.

Please try using this file, https://storage.googleapis.com/shrill-anstett-us/VGGFace2_test_classslabels.txt for the class labels and set CL_PATH = './VGGFace2_test_classslabels.txt'.

Please use this file for VGGFace2_train_data_v1 class labels.
https://storage.googleapis.com/shrill-anstett-us/VGGFace2_classlabels.txt
This file has 8631 class labels total and should resolve the errors.

It starts training with batch size=1 :grinning:
with batch size=32 it fails with OOM.
I guess my costly GPU is of little to no use  :hushed:
I have to find a golden mean between batch size 1 and 32, for my gpu 2080TI and another systems gpu 1080TI
Also I have modified your script to generate training labels slightly on ubuntu 16:
import glob
class_labels = glob.glob('/home/serveradmin/Documents/ajinkya/output/DeepFace/train/*')

with open('VGGFace2-class_labels_train.txt', 'w') as txt_file:
     for cl in class_labels:
          print(cl[58:], file=txt_file)

Ill post the optimum batch size for training without running into OOM. But, does having low batch size affect the accuracy of this model ?

The DeepFace network proposed by Taigman et al. (CVPR 2014) is a very computationally expensive deep neural net. The high accuracy cited in the paper is cleanly evident from the FLOPS vs Accuracy tradeoff. It is due to the use of 3 locally connected layers (LocallyConnected2D L4, L5, L6) in the network which does not share weights unlike Conv2D layers.
The training process mentioned in the paper uses a mini batch size of 128 images with a stochastic gradient descent optimizer on a cluster of GPUs that took days of training. The learning rate was initially set to 0.01 and momentum as 0.9.
I went through a few StackOverflow posts and related papers which tell me that whenever the batch size is increased or decreased k times, the learning rate should also be updated k times when using Gradient Descent. I guess using this concept, consistent / approximately similar accuracy can be achieved using smaller / larger batch sizes. In my case, when I used 1024 as the batch size instead of 128, similar accuracy was achievable.
I am hoping that in your case with a batch size set to 8 should not affect the accuracy of this model much, provided learning rate is adjusted. (which is there in the training itself)
https://github.com/swghosh/DeepFace/blob/72b194edc9a0bd288db108770e74bd6018dac15c/contrib/train_on_gpu.py#L21
References:
https://stackoverflow.com/a/53046624
https://research.fb.com/publications/deepface-closing-the-gap-to-human-level-performance-in-face-verification/

Thank you and I really appretiate your diligence, I am closing this issue, As of now the training is bring progressed without any error.


Bug on install

Hi, there maybe something you're using that is not in my project.
This is the error when I first install Vortex:
Assets\Vortex\Vortex\Code\Runtime\Tasks\PlaySequence.cs(10,31): error CS0246: The type or namespace name 'SerializeReferenceButton' could not be found (are you missing a using directive or an assembly reference?)
Assets\Vortex\Vortex\Code\Runtime\Tasks\PlaySequence.cs(10,31): error CS0246: The type or namespace name 'SerializeReferenceButtonAttribute' could not be found (are you missing a using directive or an assembly reference?)

Also I test build the project and all the editor scripts in Vortex's Editor folder throw errors.
Normally Editor scripts will be excluded on build while they're in Editor folder, which is this case. But why is that ?
I have to go in and use #if Unity_Editor on them to able to build.

Hi, the library has third-party dependencies(also my own libraries) which will be included later via "package.json". Right now Vortex is highly experimental and contains known bugs. The vortex is undergoing major architectural changes in the "dev_rumman" branch.
Reasons:

Current architecture is overly complex to maintain
I will implement features such as Unreal Engine style Anim notify, Anim montage, Anim composite, etc. And this will be best done on this new arch.
Current solution has several bugs which will be better eliminated in the new architectural approach.

All current features should work fine with this new architecture, except events(Anim Notify will replace them). And current API will be changed, so do not hold current method names. So I highly suggest you not use this library for production purposes(as stated in my GitHub bio as well). And I would recommend waiting for the next upcoming major release that will use new architecture.

But in case you insist on using this now on the master branch, you need this in your project.
If you want to check how development is going on "dev_rumman" branch, you need

Playable graph visualizer
UnityExt
AttributeExt
And of course the first library I mentioned earlier


update dependencies

ok to test

any concerns here with maintaining B.C. when supporting consumer option pass through?

actually, doesn't look like an issue, as the only dependency updated was lodash. which is not something we expose to consumers.


T06.08-Solution-PreferenceSummary

Not really an issue, just a question and a note.
Either ToDo-Text is incomplete or maybe easier solution is possible

   // Go through all of the preferences, and set up their preference summary.

   for (int i = 0; i < count; i++) {

       Preference p = prefScreen.getPreference(i);

       // You don't need to set up preference summaries for checkbox preferences because

       // they are already set up in xml using summaryOff and summary On

       if (!(p instanceof CheckBoxPreference)) {

           String value = sharedPreferences.getString(p.getKey(), "");

           setPreferenceSummary(p, value);

       }

   }

Why don't you use just p.getValue()?

        if (p instanceof ListPreference) {
            String value = ((ListPreference) p).getValue();
        }


Event loop stops working after 21 days

We have several users having our Windows product using aws-crt-cpp stopped working after a while. I found out that the reason is incorrect handling of timestamp values which makes them invalid after ~21 days of uptime.
The problematic code is in aws_high_res_clock_get_ticks():
int aws_high_res_clock_get_ticks(uint64_t *timestamp) {
    LARGE_INTEGER ticks, frequency;
    /* QPC runs on sub-microsecond precision, convert to nanoseconds */
    if (QueryPerformanceFrequency(&frequency) && QueryPerformanceCounter(&ticks)) {
        *timestamp = aws_timestamp_convert((uint64_t)ticks.QuadPart, AWS_TIMESTAMP_SECS, AWS_TIMESTAMP_MICROS, NULL) /
                     (uint64_t)frequency.QuadPart;

        *timestamp = aws_timestamp_convert(*timestamp, AWS_TIMESTAMP_MICROS, AWS_TIMESTAMP_NANOS, NULL);
        return AWS_OP_SUCCESS;
    }

    return aws_raise_error(AWS_ERROR_CLOCK_FAILURE);
}

The problem is the line
        *timestamp = aws_timestamp_convert((uint64_t)ticks.QuadPart, AWS_TIMESTAMP_SECS, AWS_TIMESTAMP_MICROS, NULL) /
                     (uint64_t)frequency.QuadPart;

which starts by multiplying ticks by 1,000,000. This will saturate uint64_t when ticks is > UINT64_MAX / 1000000 =<PHONE_NUMBER>3709. With a typical frequency of 10,000,000 this amounts to 21.3 days. The visible effect of this is that the time stops at<PHONE_NUMBER>370000 ns and the event loop starts repeating the following message without ever reaching the task's timestamp:
detected more scheduled tasks with the next occurring at<PHONE_NUMBER>370000, using timeout of 1000.

The problematic behavior stops only by rebooting the computer.
As an example of a correct way to handle the conversion, @EeroKurimo found out that QT5 function ticksToNanoseconds() (LGPL license) seems to handle the same issue in a way which doesn't have the risk of overflow.

PR #793 and release v0.5.7 fixed the problem for us (tested successfully on one client).

Excellent.


Support Dapr for .NET 6 minimal API

Using Dapr for minimal API. I tried and the service require to add controllers services.AddControllers().AddDapr()
I was wondering if we can just do services.AddDapr()  So no need to add controllers.

Transfered to correct repo.
cc @rynowak

@rizamarhaban - What are you trying to add into your services? There is a separate method for the client if that's all you need. I believe this method is on top of controllers because we need controllers to setup the various Dapr endpoints.

I'm going to mark this as closed since it's getting to be an older ticket, but please feel free to re-open it if you have any follow-up questions.


Change contained styles. Remove headers

Change of the contained versions for the NDC partnership site: https://ndcpartnership.org/
Designs here: https://bluewatermedia.invisionapp.com/share/JQTO84KKNDP#/screens/381291453

Having a look!


xx-sss - Screen Space Shadows

https://github.com/bkaradzic/bgfx/issues/2195
Implement screen space shadows. Requires deferred rendering or a depth prepass. Convert rendered depth to linear depth to skip reconstructing multiple times when doing shadow test.
Project light into screen space to find direction from each pixel to the light. Walk through screen space texture towards light. Sample depth to reconstruct position represented by this sample pixel and compare to position along interpolated ray from pixel to light. If position represented by depth is closer to the eye than the light ray, an initial pixel is in shadow.
Specify distance of shadow ray via world units or pixels in screen space.
Optionally offset the initial sample position by noise to reduce banding.
Demonstrate other ways to reduce hard edge of screen space shadow.

Wow you've been busy! :+1: Thanks!
There is missing vs_sss_screenquad.sc shader. Also since you're going to add this, can you rename xx-sss to 44-sss?

Wow you've been busy! :+1: Thanks!
There is missing vs_sss_screenquad.sc shader. Also since you're going to add this, can you rename xx-sss to 44-sss?

oops!

oops!


Inline date range

Is it currently possible to do date ranges with an inline datepicker?  An example or 2 of the range datepicker on the demo page would be really nice too.

Isn't working I think.

This is not working for me

It's working for me, now I'll make the arrengements for the configurations I need, but it has been very usefull, thanks a lot for the appointment!
PD: I find it strange, but not all the properties work inline, as in an single-datepicker implementation; in my case, for example, data-date-format="dd/mm/yyyy" data-date-week-start="1" data-date-start-date="+1d" are working inline, but maxViewMode or languege have to be on the .datepicker( options_datepicker ) declaration.


How to JSON encode geometry and properties in the layer description in TileSet metadata

In JSON encodings proposed by Jerome

We should probably have an enumeration type for geometry type,
suggesting:

points
lines
polygons

For the layer properties, as previously discussed we should
coordinate this with the general question of queryables / presentables /
schema, and perhaps additional things specified in that layer metadata
could also be things specified by OGC API - Common Part 2 either at the
extended /collections/{collectionId} end-point, or at a sub-end-point
like /collections/{collectionId}/{something}.

Our current implementation of queryables uses "id" rather than "name",

In my opinion it should be better to have a
"schemas" : {
}
(exactly as OpenAPI)
The content of "schemas" is under discussion. One alternative could be to have a a full GeoJSON schema personalized to the feature types. These mean, defining a geometry and a property sections and setting the geometry to a single geomentry (e.g. LineString) and the properties to the actual property names and types.

This is what we agreed to use.
"propertiesSchema" :
{
"type": "object",
"properties" :
{
"F_CODE" :
{
"title" : "Feature Code",
"type" : "string",
"enum" : [ "EA010", "EA040", "AM020" ]
},
"FFN" : { "type" : "integer" },
"FCSUBTYPE" :
{
"title" : "Feature subtype",
"type" : "integer",
"enum" : [ 100380, 100384 ]
},
"ZI005_FNA" : { "type" : "string"  }
}
}

This is what we agreed to use.
"propertiesSchema" :
{
"type": "object",
"properties" :
{
"F_CODE" :
{
"title" : "Feature Code",
"type" : "string",
"enum" : [ "EA010", "EA040", "AM020" ]
},
"FFN" : { "type" : "integer" },
"FCSUBTYPE" :
{
"title" : "Feature subtype",
"type" : "integer",
"enum" : [ 100380, 100384 ]
},
"ZI005_FNA" : { "type" : "string"  }
}
}

Solution has been applied to the JSON schemas using the meta-schema for JSON and in section 9 (JSON encoding) there is a table mapping the UML class with the JSON schema tags.. Since this was presented yesterday (2021-02-11) and there was no objection, I'm closing the issue


[Not working - need help] autocomplete on types not working

The autocomplete doesn't seem to work when the language is set to Typescript but works when TypescriptReact.
This is my configs which i think maybe relevant (let me know if you need more:

Core (Atom v1.26.0):

[x] Use Tree Sitter Parser

Autocomplete-plus (v2.40.5):

Maximum Visible Suggestions - 10
[x] Use Core Movement Commands

Language-typescript (v0.3.3)
Atom-typescript (v12.4.0):

Autocomplete Suggestion Priority - 5

When as Typescript grammar:
I've noticed the classes aren't TS

When as TypescriptReact grammar:

Changing it back to Typescript still results in the same problem.
Any ideas?
I've tried toggling tree sitter off - same problem.

Most likely a third-party TypeScript grammar interfering. Make sure you don't have other typescript-related packages installed?

Just FYI, editor:log-cursor-scope command (accessible from command palette) will show you the list of grammar scopes the text cursor is currently in. The top one is always the document scope. It should either be source.ts or typescript (when using tree-sitter)

editor:log-cursor-scope command grammar scopes the text cursor is currently in.

Looks like it is typescript. Is the scopes correct?
typescript
program
expression_statement
call_expression
arguments
arrow_function
statement_block
expression_statement
call_expression
arguments
call_expression
arguments
arrow_function
statement_block
expression_statement
assignment_expression
call_expression
member_expression
property_identifier

Make sure you don't have other typescript-related packages installed?

Suggestions on the quickest way to do this? ...i suppose safemode..will have to try that

Sorry, my bad. I think I know where the problem is. Although this remark:

I've tried toggling tree sitter off - same problem.

kinda threw me off.

 There is no way to enable autocomplete for tree-sitter TypeScript without also enabling it for JavaScript.
So yeah... don't use tree-sitter for now I guess. I don't have another solution at the moment.

Nevermind, figured it out... somewhat. v12.4.1 should work. Sorry for giving you the runaround.

All good! Much ❤
On Fri., 27 Apr. 2018, 1:13 pm Nikolay Yakimov<EMAIL_ADDRESS>wrote:

Nevermind, figured it out... somewhat. v12.4.1 should work. Sorry for
giving you the runaround.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/TypeStrong/atom-typescript/issues/1422#issuecomment-384851395,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AKXAhr2qEHbVV-y7yDI0KV1k3LvPl92_ks5tsozkgaJpZM4TkYhO
.

Confirming: it works now!


Preserve weak types in array slices

Depends on #5157; will rebase after that is merged.

By the way, if you set the PR to merge into another PR/branch rather than to merge into master, github will show a more useful diffbase!

Rebased and ready for review!


Adjustment for incorrect card types

If you can incorporate  information about individual card selections, then add:
• If the initial random selection has a card type that is anything other than “Legendary Creature” or “Legendary Planeswalker” that also includes “[Planeswalker’s name] can be your commander” in its oracle text,  reselect.

Should be fixed with https://github.com/ericjorgensen/magic-card-fetcher/commit/1a4c44ed3334f6120afda514a203c0c16171346e


Extensions with inheritance

The current schema for installing script extensions relies on an importance score for each mod, where the score is equal to how many times a mod appears as a dependency inside the current mod list.
That score is used to let more important mods load before others, and thus let their extensions load before those of other mods.
This layout poses one problem in situations where parent and child scripts in vanilla are supposed to be extended within the same mod list : a mod with bigger importance can look to extend a script that is child to a script extended by mod(s) of lower importance.
This results in the child script not taking into account the changes made by the extension(s) to the parent script.
This is also not a problem that can be solved through mod load order alone, since regardless of the importance score, 2 mods may be faced with this situation twice, with their roles reversed in the second situation.
This PR looks to take into account vanilla's inheritance chain to decide the order in which extensions are installed. The mods' importance scores are still used to load mods following their dependencies.
But in situations where a more important mod would extend a child before their parent, the extension(s) of the parent script is given priority.
Also, even when extending in that order, vanilla parents still get loaded as base scripts when extending a child script, instead of their extended version.
To make sure the extended parents are used when extending child scripts, reload() is called everytime a script is extended

I went ahead and applied your suggestions, code should look better now !

closed in favor of #124


fix(.circleci): bintray secret and user for creating stable versions

Signed-off-by: Leonardo Di Donato<EMAIL_ADDRESS>What type of PR is this?
/kind bug
Any specific area of the project related to this PR?
/area build
What this PR does / why we need it:
Which issue(s) this PR fixes:
Special notes for your reviewer:
Does this PR introduce a user-facing change?:

NONE

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by:
To complete the pull request process, please assign leodido
You can assign the PR to them by writing /assign @leodido in a comment when ready.
The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

/cc @fntlnz

/milestone 0.21.0

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: fntlnz
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [fntlnz]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

LGTM label has been added.  Git tree hash: ccabcaac4e86b83cb9b6ca699eb37c4c79a34763


Keep getting 422 error code.

Hello,
I have followed everything you said on your installation guide. So I tested by buying a minecraft server off my panel and I enabled debug logger so I can see errors so I did buy it and this error came up:
Array
(
[errors] => Array
(
[0] => Array
(
[code] => required
[detail] => The Server Jar File variable field is required.
[source] => Array
(
[field] => environment.SERVER_JARFILE
)
            )

        [1] => Array
            (
                [code] => required
                [detail] => The Build Number variable field is required.
                [source] => Array
                    (
                        [field] => environment.BUILD_NUMBER
                    )

            )

    )

[status_code] => 422

)
Build is set to default on my panel and server jar file is server.jar?
Any answers would help me!
Thanks - Connorcivi

Please use the Pterodactyl Discord for configuration related support instead of GitHub issues.


Cannot create instance of Gnosis Safe when creating a safe

What is this bug about?
Few users reported a bug when they got stuck on safe creation screen, for majority of them the reason was the error "Cannot create instance of Gnosis Safe". So, after the tx is mined, we get safe address from logs, but it fails to create an instance saying no code is available on this address. Probably there is some delay with updating it.
Why is it needed? What is the value? For whom do we build it?
So users don't get stuck on safe creation screen
High-level overview of the bugfix
We can try to wrap getGnosisSafeInstance in try/catch, and if we get this error we may try to do polling for 30s-1m

this can be addressed in #242


Export Protocol and ConnectionLike

I need these to be able to do:
extern crate mysql_async as my;
use my::{QueryResult, Protocol, ConnectionLike};
use my::prelude::FromRow;
use tokio_async_await::compat::forward::IntoAwaitable;

pub async fn get_all_results<TupleType, T, P>(result : QueryResult<T, P>) -> Result<Vec<TupleType>, my::error::Error>
    where TupleType: FromRow + Send + 'static,
          P: Protocol + Send + 'static,
          T: ConnectionLike + Sized + Send + 'static,  {
    let mut fut = result.collect();
    let (_, data) :(_, Vec<TupleType>) = match await!(fut.into_awaitable()) {
        Ok(r) => r,
        Err(e) => {return Err(e);}
    };
    Ok(data)
}

pub async fn get_single_result<TupleType, T, P>(result : QueryResult<T, P>) -> Result<TupleType, my::error::Error>
    where TupleType: FromRow + Send + 'static,
          P: Protocol + Send + 'static,
          T: ConnectionLike + Sized + Send + 'static,  {
    let mut data : Vec<TupleType> = await!(get_all_results(result))?;
    if data.len() != 1 {
        Err(my::error::Error::Io(std::io::Error::from(std::io::ErrorKind::InvalidData)))
    }else {
        Ok(data.remove(0))
    }
}```
right now I need to use modified version to to make them public exported.

Hi. Pipes/sockets support is in progress atm. I'll look into your issue after.

I changed line 195 in lib.rs to:
pub use self::queryable::{BinaryProtocol, TextProtocol, Protocol};
and added new line pub use self::connection_like::ConnectionLike;

Of course it would be even better if more functions like these would be available directly on the QueryResult :) but if my code will work with standard crate version it will be enough for now :)

Fixed in v0.19.0
See also this test.

thanks


It would be nice to have a pre-defined location where config maps containing MQSC could be mounted and executed

It doesn't appear to be possible to add a file into an existing directory using a config map. So if you mount a config map (in my case based on an mqsc file) to /etc/mqm the rest of the /etc/mqm directory is shadowed and 10-dev.mqsc can't be found when the container starts.
One way to fix this could be to have a predefined path (or paths) that don't currently exist, but that users can mount config maps to. At runtime we could check for them and run any mqsc files we find, separately to the dev scripts we already ship.
There might be other/better ways of doing it though.

I originally raise this issue because it seemed like it wasn't possible to mount a config map into an existing directory inside the container. However, the following stateful set config does work:
volumes:
- name: mq-config-volume
configMap:
name: mq-config
volumeMounts:
- mountPath: "/mnt/mqm"
name: "{{ template "fullname" . }}-{{ .Values.dataPVC.name }}"
- mountPath: "/etc/mqm/qmconfig.mqsc"
name: mq-config-volume
subPath: qmconfig.mqsc
Specifically the subPath attribute on the volumeMount adds the config map file into /etc/mqm/qmconfig.mqsc rather than replacing/shadowing the entire /etc/mqm directory.
The limitation of subPath is that it doesn't get replaced when the config map is updated, so a pod restart is required. Without subPath, if you change the config map the volume is re-mounted and the container can see the changes to the file without being restarted.


Support hashmap_get_by_hash, and hashmap_set_by_hash

As part of a project we're working on, we are not creating a key based on the item stored in the hash map.  We have a pointer to the item which we need to lookup within the hash map at periodic intervals.

Maybe I'm missing something, but isn't this the same as defining a hash function such as
uint64_t my_hash_fn (
        const void *item, uint64_t seed0, uint64_t seed1)
{ return (uint64_t) item; }

for your hash map?

Considering there is the function hashmap_set_with_hash now, this PR could be closed...


Testnet PR

@arunkumargdr thank you for your PR!
Please update the root level README.md before we can merge it.

Just up @arunkumargdr , you can update PR?


Adds .editorconfig

This commit consists solely of whitespace cleanup in the files to match
the editorconfig spec. Having the entire repo’s worth of whitespace changes in a separate commit will keep future commits cleaner. If you prefer not to use editorconfig that’s totally cool.

Oops I see master is ahead of this, I'll update.

Sure, this is fine 😄 Do you need me to merge or can you do it?

Thanks for asking, looks great!


clang_check/clang_tidy options for clang passed without -extra-arg

The clang_check checker overrides the configuration that is usually made in the CMake output file compile_commands.json, because it uses the -- variant. See docs.
I have a fix that is probably incomplete and you would need to check it, but to give you an idea how additional options needs to be passed down to clang. The patch refers to this line.
    let makeprg = self.makeprgBuild({
        \ 'post_args':
        \   '-extra-arg=-fshow-column ' .
        \   '-extra-arg=-fshow-source-location ' .
        \   '-extra-arg=-fno-caret-diagnostics ' .
        \   '-extra-arg=-fno-color-diagnostics ' .
        \   '-extra-arg=-fdiagnostics-format=clang '.
        \   syntastic#c#ReadConfig(g:syntastic_clang_check_config_file)})

Note that I've removed the -- entirely and syntastic#c#ReadConfig() was adding options after --, so this might break for everyone, but actually makes more sense, in my opinion.
Please also check clang_tidy and eventually other LibTooling checkers (if there are more of them).

From the manual, assuming C++:

By default you can set "ClangCheck" parameters in g:syntastic_cpp_clang_check_args, and compilation parameters (defines, optimisation flags, etc.) in the configuration file.
However, if you want "ClangCheck" to use compilation databases (http://clang.llvm.org/docs/JSONCompilationDatabase.html) -- perhaps generated by "CMake" (https://github.com/rizsotto/Bear) or "Build EAR" (https://github.com/rizsotto/Bear) -- rather than pass compilation flags explicitly, set g:syntastic_cpp_clang_check_post_args to an empty string:
let g:syntastic_cpp_clang_check_post_args = ""

Configuration files pointed to by g:syntastic_clang_check_config_file are then ignored.

There is a similar mechanism for C.

Ok, because it looked like, you would like to strip colors and force showing columns etc to avoid getting escape codes and wrong error message formats. With
let g:syntastic_cpp_clang_check_post_args = ""

these options are not passed down to clang_check. The documentation does not mention, if the removed -f options are important. I assumed they are.

They are mostly defaults.  Also, if you force colours on in the compilation database you deserve what you get. :smirk:

I don't see any of these -f options set/forced. Ok, I understand that the -f options are redundant, because they are default. Thank you for the information.

I don't see any of these -f options set/forced.

There were people who did that in the past.


Topic recognition

Project
This pull request provides a VQVAE implementation on the ADNI dataset (Task 7). In addition to the VQVAE, a GAN network is provided to act as a prior model to the VQVAE. The GAN network is trained to generate latent space representations of the training data, which are then used as input to the VQVAE network to generate images from provided dataset.
Structure

README.md provides an overview of the project, the report, usage and complete documentation of all classes
dataset.py contains class-based implementations for loading and preprocessing the dataset.
modules.py contains class-based implementations of all networks used in the project.
train.py contains the training loop and evaluation code.
predict.py contains the code for generating images from both GAN and VQVAE model, as well as assessing the performance of the end-models
utils.py contains a single source of truth for defining constants and parameters across all models
main.py is the test driver for the codebase. It is used to run the training and evaluation code.
images/ is a top-level directory containing images for the results and the README.md report

Results
The produced VQVAE was able to reconstruct the training set with no obvious differences to the original data when viewed manually. A GAN network was used to assist in generating images from noise by learning the probabilities of latent space representation as an input to the VQVAE network. This produced images of similar structure to the dataset, but with a noticeable blurring of portions and lack of fine details. The average SSIM was 0.62, as the structure of the image did resemble that of the test set data.
Better results may be achievable with data augmentation to improve the representation of details in the encoded indices, and by changing the prior network used for generation as the generator diverged.

This is an initial inspection, no action is required at this point
Difficulty: Hard
Readme Overall:
Good

Project Overview: Good
Model: Good
Data + Preprocessing: Good
Usage: Good
Training / Loss Curve(s): Good
Result Demonstration: Good
References: Good

Functionality:

Reconstruction: Good
Generation: Good

Code:

Consistent with Results: Yes
File Structure: Good
Commenting: Good
Commit frequency: Good
Commit messages: Good

Other Comments:
N/A

Marking
Good Practice (Design/Commenting, TF/Torch Usage)
Adequate design and implementation
Good spacing and comments
Header blocks
Recognition Problem
Solves problem
Driver Script present
File structure present
Shows Usage & Demo & Visualisation & Data usage
Module present
Commenting
No Data leakage
Difficulty: Hard
Commit Log
Meaningful commit messages
Progressive commits used
Documentation
ReadMe good
Model/technical explanation
Good Description and Comments
Markdown used and PDF submitted
Pull Request
Successful Pull Request (Working Algorithm Delivered on Time in Correct Branch)
No Feedback required
Request Description good


Deserializing with type name handling on assemblies loaded with Assembly.LoadFrom

Consider this situation.
Library.dll
namespace LibraryNS
{
    public abstract class BaseClass
    {
    }

    public class DerivedClass : BaseClass
    {
    }
}

Program.exe
using System.Reflection;
using Newtonsoft.Json;

namespace ProgramNS
{
    class Program
    {
        static void Main()
        {
            var assembly = Assembly.LoadFrom("../../../Library/bin/Debug/Library.dll");
            var b = JsonConvert.DeserializeObject("{\"$type\":\"LibraryNS.DerivedClass, Library\"}", new JsonSerializerSettings { TypeNameHandling = TypeNameHandling.Auto });
        }
    }
}

DeserializeObject will fail with a JsonSerializationException with message of "Could not load assembly 'Library'." with a stacktrace of
at Newtonsoft.Json.Serialization.DefaultSerializationBinder.GetTypeFromTypeNameKey(TypeNameKey typeNameKey) in C:\Users\brinkleyt\Documents\GitHubVisualStudio\Newtonsoft.Json\Src\Newtonsoft.Json\Serialization\DefaultSerializationBinder.cs:line 83
To fix this the partial name should be compared instead of FullName.

I don't make changes without unit tests.

I'd like to create a unit test for this but due to the nature of the bug it requires a separate dll that is loaded using the Assembly.LoadFrom method and it can't be a system dll. I could add a project for this simple dll to the main solution, create a separate solution for the generation of the dll, or simply include a dll in the repository. Which would be most preferable?


Can ICMP be stateful or not?

Can you add stateful ACL rules (on a Hyper-V Virtual Switch) on the ICMP protocol?
If not, this leaves you to either open ICMP to everyone or close ICMP to everyone including the VM itself.
Neither is secure or practical for such an important and basic functionality (ping).

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 69cab761-7568-ea87-a69c-c6798b7baa84
Version Independent ID: 77dcb2b9-1688-b399-7019-8c72f9908563
Content: Create Security Policies with Extended Port Access Control Lists
Content Source: WindowsServerDocs/virtualization/hyper-v-virtual-switch/Create-Security-Policies-with-Extended-Port-Access-Control-Lists.md
Product: windows-server
Technology: networking-hv-switch
GitHub Login: @shortpatti
Microsoft Alias: pashort

Still without an answer...


Add missing addresses field to broker (#3261)

Signed-off-by: Calum Murray<EMAIL_ADDRESS>

OCF Webhook is merging this PR


Add Julia plotting notebook

Fix #182

One comment actually, should we add a particle_da.h5 file to the repository so this would plot something immediately after cloning the repo?

And another one, the name of both this and the python notebook should not mention tdac 😅


Make function/script that returns json file of patterns and occurrences

Something in these lines:
{
    "GENE-TAG-MARKER 32": ["ase1-GFP-mCherry", ]
}

Reopened as it would need to work with the new code.

This should be easy once you have the json from #22

I think this is done, right?

Yes, this is done


Radio: ariaLabel property

Info
Currently the ariaLabel property is assigned to the Radio Group component, not the Radio Item component. It then gets sent to each child item, so every single item in the group gets the exact same arialabel. So for a screen reader right now reading our Radio components, the Form Item label isn't read, it's the arialabel, or if not used, the name property which is likely a generated name not meant for human consumption. And then if arialabel is used, every single item in the group is read the exact same.
Acceptance Criteria:

ariaLabel when applied to the Radio Group should be read for the whole group, basically what the Form Item label is supposed to do

It should not be applied to the Radio Items in the Group
It should also not fallback to the name property

Radio Item should have it's own ariaLabel property that should fallback to the label property if it's not given
Documentation is updated for both RadioGroup and RadioItem properties

Description of ariaLabel for Radio Group
New property for Radio Item, ariaLabel

:tada: This issue has been resolved in version 1.17.0-alpha.75 :tada:
The release is available on:

npm package (@alpha dist-tag)
GitHub release

Your semantic-release bot :package::rocket:

:tada: This issue has been resolved in version 4.17.0-alpha.32 :tada:
The release is available on:

npm package (@alpha dist-tag)
GitHub release

Your semantic-release bot :package::rocket:

:tada: This issue has been resolved in version 1.23.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:

:tada: This issue has been resolved in version 4.22.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:

:tada: This issue has been resolved in version 3.0.0-alpha.5 :tada:
The release is available on:

npm package (@alpha dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


How to enable SSL pinning on mobile

Summary
Enable SSL Pinning on the mobile apps when you build your own.
Do not merge until this functionality is released.
Related PR's:
https://github.com/mattermost/react-native-network-client/pull/129
https://github.com/mattermost/mattermost-mobile/pull/8055
Ticket Link
https://mattermost.atlassian.net/browse/MM-59055

@cwarnermm any idea how to achieve Colton's suggestion ?

@enahum @coltoneshaw - My latest commit adds an Important Note directly after the disadvantages list:


how to obtain the visual result

hello thank you for your great job!
I want to know how to obtain the visual result as you shown.

hello thank you for your great job!
I want to know how to obtain the visual result as you shown.

Thanks for your attention. We visual result by coco api.


How to show that notification that appears on top in iOS?

For example, when we use Whatsapp in iOS, and we are inside a chat screen, if another person sends a message to me, a top notification appear.
How to send a notification like that?

That's something that you would need to do yourself when onNotificationReceivedForeground triggers.


add control inputs

You know, CRF node potentials! Like u[t].

Bumping this since we continue to get requests for this feature.  Here's a bit more info from Murphy pgs 641--644.  Let
z_t ~ N(A_t z_{t-1} + B_t u_t, Q_t)
x_t ~ N(C_t z_t + D_t u_t, R_t)

Inputs factor into the prediction step,
\mu_{t|t-1} = A_t \mu_{t-1} + B_t u_t

and into the measurement (condition) step,
\hat{y}_t = C_t \mu_{t|t-1} + D_t u_t

Corresponding changes to the info form shouldn't be hard to derive.

Yeah! So for message passing in distribution form, clearly we just need to adjust some means. In particular, as you said, predict needs to learn how to add B_t u_t by using a dcopy or dgemv into mu_predict before the dgemv and condition_on needs to replace y_t with y_t - D_t u_t by using a daxpy instead of dcopy.
For information form, this seems like a good place to write out some notes. We want to write out the joint distribution as a product of local directed factors, suppressing conditioning on parameters,

and write those local terms as pair and node potentials on the state sequence x,

where I've dropped some base density terms involving \pi and also dropped time indices for parameters on the last line of each equation.
The only really new thing there is the linear term on the pairwise log potential; previously it was just a quadratic form in x_t. The node potential has the same form as before, just with y_t replaced with y_t - D_t u_t, as expected. So in the information form message passing code we need to add linear terms in the pairwise potentials, since with my lack of foresight I assumed there would only be J_pair's and no h_pair's for the pairwise potentials. The change to the node potentials would happen in Python, where we set up J_node and h_node, and we just need to replace data with data - np.einsum('tj,tij->ti', control_inputs, self.D) there.
Disclaimer: those formulas need to be checked and the variables may not correspond precisely to the variable names in the code (e.g. sign errors may be present).

For documentation purposes, the other pieces to the puzzle are the formulas for conditioning and marginalizing in information form, plus the RTS smoother logic. Those are currently best represented in the SVAE repo: conditioning and marginalizing and RTS update.

This has been incorporated into master with https://github.com/mattjj/pylds/pull/6
See notes/info_form.tex for more detailed derivations.


feat: node-fetch's natural exports should be re-exported by mock

The node-fetch package exports helpers like Headers and Response by default, allowing Node projects to use them in their code, usually like so:
const { default: fetch, Headers } = require('node-fetch')
const headers = new Headers(...)
fetch('...', {headers})

If your code relies on these utilities, mocking node-fetch causes errors, such as: TypeError: Headers is not a constructor. It would be nice if fetch-mock (and fetch-mock-jest by association) re-exported these useful utilities, so as not to break production code.
Repl.it demonstrating issue

My current work-around:
// __mocks__/node-fetch.js
'use strict'

const nodeFetch = require('node-fetch')
const fetchMock = require('fetch-mock-jest')
const fetch = fetchMock.sandbox()

module.exports = exports = fetch
module.exports.default = exports
module.exports.Headers = nodeFetch.Headers
module.exports.Request = nodeFetch.Request
module.exports.Response = nodeFetch.Response

I think this is a good idea. Thanks for raising it.
There is however some nuance:fetch-mock is configurable to use any implementation of fetch, and node-fetch is not the only one available (although it is the most popular), so I don't want to rush into this. Specifically, I wonder if I should export using proxies that can be used as getters for properties on fetchMock.config
Will think about it for a bit

Reviewing your code, I see that the default behavior is to pull Headers, Request, and Response directly from node-fetch into your config, and allowing the user to overload each of them with their own implementation. A good compromise that leaves this behavior intact, would be to expose each of those classes either on the main fetchMock class or instance class provided by FetchMock.createInstance, pulling them from the config. Then, re-expose the sandbox class on a circular default property (sandbox.default = sandbox) to prevent breaking code that implements node-fetch's export pattern. This adds useful functionality while preventing any breaking changes to your existing module.

I followed your reasoning there apart from the bit about sandbox. Could you put together a draft PR to demonstrate what you mean?


README.md style change

Remove bold styling from the world builds in the README.md.

Fixed in commit: https://github.com/seabearDEV/nodejs-typescript-web-server/commit/674aaead095ccd516dc629b0cfe341390b87cd72

Fixed in commit: https://github.com/seabearDEV/nodejs-typescript-web-server/commit/674aaead095ccd516dc629b0cfe341390b87cd72

Fixed in commit: https://github.com/seabearDEV/nodejs-typescript-web-server/commit/674aaead095ccd516dc629b0cfe341390b87cd72


Fix #3102 - Add an AddMvcOptions to builders

This makes it easier/possible for any third party code that extends
IMvcBuilder to set options.

:shipit:

b95dcd5f1ce17f18aee7b75640f5e80fbff2c4fa


Update documentation

[x] Filter https://github.com/orgs/jdi-testing/projects/6?card_filter_query=label%3A"filter+feature"
[x] #1063
[x] Epics table
[x] US table
[ ] "Requirements gathering" label: https://github.com/orgs/jdi-testing/projects/6?card_filter_query=label%3A"requirements+gathering"
[x] #155
[x] https://github.com/jdi-testing/jdn-ai/issues/760
[x] NR https://github.com/jdi-testing/jdn-ai/issues/567
[x] #1067
[ ] Menu (3 types) :
https://github.com/jdi-testing/jdn-ai/issues/916
https://github.com/jdi-testing/jdn-ai/issues/863
Page Object Actions menu
Locator's action menu
Quick Actions menu

#1068


Created 383.RansomNote.cpp file

Pull Request Template
What have you Changed(must)
I have created the file 383.RansomNote.cpp to the Leetcode Problem no 383.
Write here what you have changed in the codebase
in the CPP folder
Issue no.(must) - #1122
Pr will be close and marked as spam. If issue number not found or issue was assigned to someone else.
Marking as spam can block your account from HacktoberFest.
Self Check(Tick After Making pull Request)

[x] This issue was assigned to me if not it will be marked as spam.
[x] My file is in the proper folder
[x] I am following clean code and Documentation.
[x] I have added the title and what program will do first in the file
[x] I have named the file as (Problem #). (Problem Title(in Camel Case)).(extension)
ex. 2. AddToNumbers.py

README - How to Contribute

@SSKale1 @Yashica7 @swaingotnochill @JOKUE2002 feel free to merge and check it.


Add extra vars at service and operation level

Allow to define a json of extra var that are automatically added when executing a service or an operation. (Like already done in global hook feature)
If a var is defined at both level, the operation override the value set at service level.

Done and merged in dev.


can't run triangle example

tried to run the triangle example, but got the error:
render_pipeline = device.create_render_pipeline(
TypeError: create_render_pipeline() got an unexpected keyword argument 'vertex'
Don't know how to make it works.

Did you also try to run triangle_glfw.py or any of the other triangle example scripts? The code in triangle.py is not meant to run independently and is instead imported and utilised in the other triangle_*.py scripts.

The vertex keyword was recently renamed; are you running on a git checkout or with the installed wheel?

I run triangle_glfw.py that calls the main method from triangle.py.
I just following the instruction to install wgpu:

pip install wgpu (this should also install Rust binary?)
pip install pyshader
copy the example files: triangle.py and triangle.glfw.py
then run triangle_glfw.py
the error message returned.

also:

pip install glfw

It seems the wgpu package (installed using pip) version is inconsistent with examples.

Ah! You are using rewritten versions of the examples with an older version of the library, because we have not released our latest changes here yet.
Try using these versions of the example scripts, that should match with the version you have installed: https://github.com/pygfx/wgpu-py/tree/v0.3.0/examples

That makes sense.
Thank you for excellent work.

I'm in the same position right now,
python 3.9.5
pip install wgpu pip install pyshader pip install glfw
copied the old examples
tons of errors in triangle.py, cube_glfw.py.
Can't run triangle_glfw.py
What's even more astounding is I tried doing a minimal installation

Windows Sandbox
Install Python 3.9.5 (newest version) and add it to path using the python installer
Install VS Code and the python extensions recommended and suggested inside of VS Code
install wgpu, pyshader, glfw using pip
download both sets of examples (old and new)
got errors stating that wgpu_native-release.dll failed to load with error 0x7e

To me it sounds like the general developer has something that isn't included with the recommended install method.

WGPU is changing fast, so the used examples must match the installed version. If you installed yesterday, you got version 0.4.0. If by old examples you mean the ones references in the link by Korijn, these match 0.3.0, not 0.4.0 (you can see it in the link).
It may add a bit to the confusion that even though 0.4.0 was released only a few days ago, it represents work from somewhat longer ago, and the examples in main don't work with it.
Last week I finished updating to some big changes in wgpu-native. I was planning on a release early next week, but might as well do it earlier :)  if you pip install -U wgpu now, you get the latest version, and the examples in main should work (I hope). PyShader is no longer needed; shaders are now written in WGSL.
If they don't work, please include the error message.


Update the maximum capacity for OS disk size from 2,048 GiB to 4095 GiB

The value mentioned for the maximum OS disk capacity/size is incorrect and needs to updated from 2048 GiB to 4095 GiB.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: bfaec879-f3e0-afba-fbe8-85e6e28937df
Version Independent ID: b21d00cd-0323-26d0-34b4-f0a76ee8aaea
Content: Azure Disk Storage overview - Azure Windows Virtual Machines
Content Source: articles/virtual-machines/windows/managed-disks-overview.md
Service: virtual-machines
Sub-service: disks
GitHub Login: @roygara
Microsoft Alias: rogarana

@smotwani We are checking this and will get back to you.

@smotwani Hello, thanks for reaching out. Would you clarify why you think this value is incorrect? We just checked with the product team, they say it's still 2048 GiB.

@smotwani As mentioned in the previous comment, there is no update on the document required now. Hope the provided information is helpful.
We will now close this issue. If there are further questions regarding this, please tag me in a comment. I will reopen it and we will continue the discussion.


Bug fix for images.toblocks()

When first dimension of the images object is > 1e5 the chunk size is 0 same as #347

Absolutely, nice catch @boazmohar!

@jwittenbach No problem, thank you for #355 now images.map_as_series is super fast :)

@boazmohar glad to hear that it's running faster!


Add the ability to pass if at least one expected output found and fix acting expected stderr as failure

fix acting expected stderr as an error in logs and results, no need for ignore_exit_code to be present for expected stderr.

Add the ability to pass if at least one expected output found by introducing the 'partial: true'

Example:
    - name: I can see all connectors
      command: >-
        lenses-cli connectors --clusterName=* --names --no-json
      stdout:
        - match: [ "logs-broker", "nullsink", "or that"] # at least one found.
          partial: true

And the best, the below will fail because partial:true is per match(es):
    - name: I can see all connectors
      command: >-
        lenses-cli connectors --clusterName=* --names --no-json
      stdout:
           # at least one found but test failed because of the second match.
        - match: [ "logs-broker", "nullsink", "or that"]
          partial: true
        - match: ["this should be there but it's not"]

Use case: we want to pass the test if error can be failed with status code: 403 or user has no access to do that action, we can do:
stderr:
 - match: [ "failed with", "no access"]
   partial: true

or failed with status code 500 and failed because other reason [ this is just one - match value which should be contained to the output in order to pass, like the above ]
stderr:
  - match: ["failed"]
    partial: true

This change is

Great work!


Bug: xss vulnerability within links?

Lexical version: lexicon playground
Steps To Reproduce

type a word
highlight word
click add link icon
type anything in there, something like 'javacript:alert('xss')'

seems you can enter anything inside the link prompt

Link to code example: https://playground.lexical.dev/

Yea, we should probably have some basic, but configurable, validation here. Also might be nice to set the rel attribute to head off reverse tabnabbing.
Seems like a good first issue :)

I would love to work on this issue! Could you please assign this to me if not assigned to anybody yet? thanks!


Option not to pluralize feature name on generation

When generating a feature with rails g rspec:feature its name is always pluralized. Sometimes this leads to undesired results, such as user_manages_account being pluralized into accounts. The description within the spec file will be pluralized, too. This requires renaming the file plus editing the description.
I would expect the generator not to pluralize the name.
However, it'd be also nice if a singular or no-pluralize option could be passed.
I could try to implement this if it's accepted.

Did you manually generate the associated controller / models or use a generator?

For features, I always generate them independently, so I run g rspec:feature.

@carlesjove yes, I see what you are saying. I agree I do not think pluralizing the name is generally what you want for a feature spec. Additionally, I think it would be nice to be able to provide it a string description which it would properly handle in the file name as well:
$ bin/rails g rspec:feature "User manages account"
      create  spec/features/user_manages_account_spec.rb

I'm sure how useful additional options would be considering the generated file is very minimal:
require 'rails_helper'

RSpec.feature "User manages account", type: :feature do
  pending "add some scenarios (or delete) #{__FILE__}"
end

Though I am always open to the community stating otherwise.


Automatic uid registration

Implemented generated automatic registration and querying of all CPACS objects with uids at the CTiglUIDManager

Hi @dixxi . Can you please rebase onto current branch cpacs_3? I fixed some travis related problems.

sure thing

ready when you are ;)

Great! Unfortunately, the template stuff makes it increasingly difficult, to wrap the API to python with SWIG though. These python bindings aren't currently build in CI, and I am building them occasionally by my own.
It would be good, that we keep template methods at a minimum if it is possible.


Function tests for ETOS Client

Description
Create function tests for ETOS Client
Motivation
Detect regressions from new code contributions or intermittent bugs
Exemplification
A nightly/daily regression test suite could typically run these tests.
Benefits
Better code and service quality, detecting regression service quickly
Possible Drawbacks
None, if the tests are written properly and test anomalies such as flakiness is dealt with.

Moved due to lack of time for testing.


Adds a management command to create discount code(s) via command line

Pre-Flight checklist

[X] Testing

[X] Code is tested
[X] Changes have been manually tested

What are the relevant tickets?
#1139
What's this PR do?
Adds a command to create discount/enrollment codes via the command line. Optionally, it can create batches of codes.
How should this be manually tested?
Run the management command generate_discount_code --help to see the options, then test with various permutations:

The defaults are to create a percent-off code with unlimited redemptions for the specified amount with the code specified. (You can specify multiple codes on the command line to generate several codes.)
You can add an expiration date by specifying --expires, or make the code a one-time redemption code by using --one-time
The discount type can be specified with --discount-type
Use --prefix and --count to generate a batch of codes. In this mode, the code specified on the command line will be ignored, and it will instead use your specified prefix and a UUID as the code.

The command will write its output to generated-codes.csv. This file will be overwritten on each run.

Codecov Report

Merging #1142 (2730b7b) into main (94ce681) will decrease coverage by 2.72%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##             main    #1142      +/-   ##
==========================================
- Coverage   85.03%   82.30%   -2.73%
==========================================
  Files         287      147     -140
  Lines       11019     7247    -3772
  Branches      730      740      +10
==========================================
- Hits         9370     5965    -3405
+ Misses       1439     1070     -369
- Partials      210      212       +2

Impacted Files
Coverage Δ

...erce/management/commands/generate_discount_code.py
0.00% <0.00%> (ø)

ecommerce/views/v0/__init__.py
90.41% <0.00%> (-0.42%)
:arrow_down:

ecommerce/models.py
88.79% <0.00%> (ø)

...ublic/src/components/forms/elements/inputs_test.js

frontend/public/src/components/Loader_test.js

frontend/public/src/components/Markdown_test.js

frontend/public/src/components/EnrolledItemCard.js

frontend/public/src/lib/notificationsApi_test.js

frontend/public/src/lib/test_utils.js

frontend/public/src/lib/selectors.js

... and 134 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Also FYI @jkachel, I am not sure about that 2 years thing mentioned in slack and in the ticket that expire in two years as well.

Addressed comments in https://github.com/mitodl/mitxonline/pull/1142/commits/3272508579256d3ba7d9c2dc3facd261ae3ea77c . The expiration date can be set with --expires, so to complete the ticket I would just specify the expiration date on the command line.


Using redux-persist from mattermost-redux

@jespino you need to rebase this

Yes, sorry, now must be rebased.

@jespino once the PR in mattermost-redux gets merged you'll need to update yarn so this build passes

Discarded this approach, we will try to use peerDependencies.


Fixed root directory resource rename bug

Resolves https://github.com/Azure/azure-sdk-for-java/issues/19491
Based off https://github.com/Azure/azure-sdk-for-java/pull/19498 but adds tests and also fixes case for subdirectories
An extra slash is being appended when getting a resource from a root directory

/check-enforcer evaluate


change php-cs-fixer to phpcs

I have exclude SuperfluousSuffix for now, we can refactor the exceptions in another PR.

I have exclude SuperfluousSuffix for now, we can refactor the exceptions in another PR.


fix(trajectory): do not allow reading frames in append mode

Fixes #272.

You will want to also update the documentation in Trajectory.hpp, and perhaps in the doc/ directory too.

Sorry, good point, am not used to having such elaborate comments/documentation. There seem to be no occurrences in doc/

And this should be a CHANGELOG entry (sorry for not thinking about this earlier ...)

Sure, I also forgot to add the TRR stuff the last time.


Account module

Adds external methods totalCollateralValue and totalAccountOpenInterest to a new Account Module
Also adds view modifiers to methods that are views

run prettier prior to merging

run prettier prior to merging

Ops, I think I have it configured to ts/js files only...