[E360-66] Package creation reusable workflow for AWS ECR

Description:

Require reusable workflow for publishing image to ECR

As an improvement, maybe we can add in some Docker caching actions.
@niroz89 can you link @Dineshk77 up with your existing work on ECR image building perhaps?

for buildx and caching, can take reference here.
https://github.com/SPHTech/bt-drupal9/blob/master/.github/workflows/deploy.yml
https://github.com/SPHTech-Platform/reusable-workflows/blob/main/.github/workflows/prisma.yaml
For BT, I have both cicd in same file now, will split it up and reuse it, have asked @pkailasu on this, he will add another reusable workflow for k8s-deploy.
https://github.com/SPHTech-Platform/docker-base-images/pull/1 needs update as well based on recent release ( https://github.com/aws-actions/amazon-ecr-login/pull/306 )
Eventually I will use the reusable workflow.

bro, one more thing, we may need to provide option to disable the latest tag, when we are creating ECR repo with immutable tags setting.

Hey @niroz89 , Can I get more info on this. Trying to understand the scenario. Is this more like deleting an image with tags?

bro, one more thing, we may need to provide option to disable the latest tag, when we are creating ECR repo with immutable tags setting.

Hey @niroz89 , Can I get more info on this. Trying to understand the scenario. Is this more like deleting an image with tags?

When we create ECR repo, it's recommended to specify the tags as immutable ( https://aquasecurity.github.io/tfsec/v0.63.1/checks/aws/ecr/enforce-immutable-repository/ ), means we won't be able to push with the same tag value, sha will make it dynamic, but when we hardcode latest tag, push will fail next time, if the repo set as immutable tag.

bro, one more thing, we may need to provide option to disable the latest tag, when we are creating ECR repo with immutable tags setting.

Hey @niroz89 , Can I get more info on this. Trying to understand the scenario. Is this more like deleting an image with tags?

bro, one more thing, we may need to provide option to disable the latest tag, when we are creating ECR repo with immutable tags setting.

Hey @niroz89 , Can I get more info on this. Trying to understand the scenario. Is this more like deleting an image with tags?

I see the latest tag is added by default in the workflow. Will check  on this

bro, one more thing, we may need to provide option to disable the latest tag, when we are creating ECR repo with immutable tags setting.

Hey @niroz89 , Can I get more info on this. Trying to understand the scenario. Is this more like deleting an image with tags?

bro, one more thing, we may need to provide option to disable the latest tag, when we are creating ECR repo with immutable tags setting.

Hey @niroz89 , Can I get more info on this. Trying to understand the scenario. Is this more like deleting an image with tags?

I see I hardcoded the latest tag in the workflow. Will check on this

Added an input to set the tag mutability.


Review comments on TR547 v3.2 draft

Hi @amazzini ,
TR-547-TAPI Reference Implementation Agreement_v3.2-RJcomments.docx
Attaching the file here with review comments because I could not send it as email attachment.
Please note that I had reduce picture resolutions in this file so that I could reduce the size to 25MB. Files larger than 25MB cannot be uploaded, apparently...

Added my comments at https://github.com/Open-Network-Models-and-Interfaces-ONMI/TAPI-Activities/blob/main/ContributionsForDiscussions/TR-547-TAPI.RIA_v3.2-RJcomments_AM.docx

Thanks @amazzini .
I have noted another comment in page 242 of the attached file. This is related to termination-state.
TR-547-TAPI.RIA_v3.2-RJcomments_AM_updated.docx


Proposed changes:

o.Debug print not only received, but also sent data.
Using standard log. (So each log entry synchronized, and log can be reassigned to file)
changed startTLSIfRequired switch.
Can be used connection without TLS:
xmpp.Options {NoTLS: true, StartTLS: false, InsecureAllowUnencryptedAuth: true}
clientIQ used pointer to bindBind (Because bindBind is optional)
init(): session use different id (not the same as bind).
added waiting iq session response (I found it important to Tigase XMPP server,
otherwise it receive presence and did not answer to iq session)

Opps. I want update my clone of the library, but by mistake update also the pull request
(that seems already is too complicated).
So I revert it.

I don't make sure what you mean the change. Is that issue of this repository? Then, could you please send me pull-request?

I'm very happy with your repository. Thanks.
I don't think you should add to your repository new change (that I already reverted)
It's not fix of issues, but example that your library is extendable, and programmer can easy add new custom XMPP extensions.
I added to XMPP message parser (ClientMessage), the XMPP extensions:

 // Custom XMPP extension of my firm.
       // It's XEP-0184

I'm also made changes in Debug options that can skip XMPP messages during session opening.
It  help me in debug, but add too much complexity to the library.

@mdosch: What do you think?
@ikq: Why have you closed your PR?

Hi, passed 8 years. ;-)
At time of writing I suppose it was a  relevant  PR.
I'm off topic now (not use XMPP and rarely use Go).


Downloaded PDF doesn't open correctly

It's probably the case that the PDF isn't being encoded correctly by the tool. When I tried to download the writing in PDF form and opened it, it failed to open.

And this is what the file  originally looks on the website:

I have tried opening it more than one time, the same error persists.

It's probably the case that the PDF isn't being encoded correctly by the tool. When I tried to download the writing in PDF form and opened it, it failed to open.

And this is what the file originally looks on the website:

Yes, there's an error in the encoding.
Would like to rewrite the function for Export as PDF?

@sj-on will you like to work on this or you want anyone else to take this up?

@sj-on will you like to work on this or you want anyone else to take this up?

You can assign it to someone else. 😅  @anushbhatia

You can assign it to someone else. 😅  @anushbhatia

Hey, I would like to work on this issue, can you please assign it to me?

Hey, I would like to work on this issue, can you please assign it to me?

can you assign it to me @anushbhatia

can you assign it to me @anushbhatia


Add index.js file for RN 0.49

Issue:
Since RN 0.49, projects now have a single index.js entry point rather than the platform-specific index.android.js and index.ios.js.
What I did
I have added an index.js file to serve as the entry point for RN 0.49 apps. As there is an existing PR for RN 0.49 ( #1971 ) to update the dependencies, I have only made the change to add the file to the generator templates.
How to test
Link the cli project and run getstorybook.

I felt this was the simplest solution rather than cater for different versions of RN in the generator script. However, I'd be happy to consider alternatives if you prefer.

@ximenean thanks for this!
I'm not sure how to proceed with this PR. I think merging this to master directly would help newcomers. This doesn't benefit an existing user who upgraded to 0.49 so the user has to be knowledgeable enough to figure out to add that index file.
Maybe that's not our responsibility if they upgraded their own dependencies.

@danielduan - I think that just adding something to the docs for RN 0.49 would suffice. I'm not sure I like any alternatives I can think of to automatically handle the upgrade from the storybook side.
If you think there's a good solution to this, I'm happy to have a go at it.


Changing value directly takes more time than clearing the text first

I have an editor instance with the value "abc".
There are two buttons. Button 1 will set the value to "xyz", button 2 clears the value.
Just clicking button 1 takes more time (loading indicator is shown) than quickly clicking button 2 and than button 1. Any hints why that could be?

Hey Sirbris,
Ahn doesn't have time to work on this project. I saw your comment on directory project. I updated the project with a new repo. If you want to move this issue to my repo, I'll work on it.
Thanks.

Will do, thank you!


Following the Bert-finetuning tutorial results in ImportError or IsADirectoryError: for run_squad_baseline.sh

I followed the getting started directions here
https://www.deepspeed.ai/tutorials/bert-finetuning/
I pulled the docker image and started a container.
I ran the following commands in a Jupyter notebook (server running in the container)
%set_env CUDA_VISIBLE_DEVICES=0,1,2
%cd /home/santosh/Projects/MsZeroTS
!git clone https://github.com/microsoft/DeepSpeed
!mkdir tfModel
!mkdir hfModel

#Save a HF version of the model
!pip install transformers
from transformers import BertModel
model = BertModel.from_pretrained('bert-base-cased')
model.save_pretrained('/home/santosh/Projects/MsZeroTS/hfModel')

#Save a tf version of the model
%cd /home/santosh/Projects/MsZeroTS/tfModel
!wget https://storage.googleapis.com/bert_models/2018_10_18/cased_L-12_H-768_A-12.zip
!unzip -q cased_L-12_H-768_A-12.zip
%cd ..

%cd DeepSpeed
!git submodule update --init --recursive
%cd DeepSpeedExamples/BingBertSquad

#Download data
!mkdir Data
%cd Data
!wget https://rajpurkar.github.io/SQuAD-explorer/dataset/train-v1.1.json
!wget https://rajpurkar.github.io/SQuAD-explorer/dataset/train-v1.1.json
%cd ..

#try tf version
!bash run_squad_baseline.sh 3 /home/santosh/Projects/MsZeroTS/TestModel/cased_L-12_H-768_A-12 /home/santosh/Projects/MsZeroTS/DeepSpeed/DeepSpeedExamples/BingBertSquad/Data /home/santosh/Projects/MsZeroTS/output1

#try hf version
!bash run_squad_baseline.sh 3 /home/santosh/Projects/MsZeroTS/hfModel /home/santosh/Projects/MsZeroTS/DeepSpeed/DeepSpeedExamples/BingBertSquad/Data /home/santosh/Projects/MsZeroTS/output2

Neither the tf or hf versions of the models are working. This is a sample output from the baselines
lr is 0.00003
seed is 12345
master port is 29500
dropout is 0.1
deepspeed --num_nodes 1 --num_gpus 3 --master_port=29500 --hostfile /dev/null nvidia_run_squad_deepspeed.py --bert_model bert-large-uncased --do_train --do_lower_case --predict_batch_size 3 --do_predict --train_file /home/santosh/Projects/MicrosoftZero/DeepSpeed/DeepSpeedExamples/BingBertSquad/Data/train-v1.1.json --predict_file /home/santosh/Projects/MicrosoftZero/DeepSpeed/DeepSpeedExamples/BingBertSquad/Data/dev-v1.1.json --train_batch_size 8 --learning_rate 0.00003 --num_train_epochs 2.0 --max_seq_length 384 --doc_stride 128 --output_dir /home/santosh/Projects/MicrosoftZero/DeepSpeed/output --job_name deepspeed_3GPUs_24batch_size --gradient_accumulation_steps 2 --fp16 --deepspeed --deepspeed_config onebit_deepspeed_bsz24_config.json --dropout 0.1 --model_file /home/santosh/Projects/MicrosoftZero/TestModel/cased_L-12_H-768_A-12 --seed 12345 --preln
[2020-10-16 04:30:19,447] [WARNING] [stage2.py:32:<module>] apex was installed without --cpp_ext.  Falling back to Python flatten and unflatten.
[2020-10-16 04:30:19,465] [WARNING] [engine.py:48:<module>] Warning:  apex was installed without --cpp_ext.  Falling back to Python flatten and unflatten.
[2020-10-16 04:30:19,469] [WARNING] [runner.py:117:fetch_hostfile] Unable to find hostfile, will proceed with training with local resources only.
[2020-10-16 04:30:19,533] [INFO] [runner.py:355:main] cmd = /usr/bin/python -u -m deepspeed.launcher.launch --world_info=eyJsb2NhbGhvc3QiOiBbMCwgMSwgMl19 --master_addr=<IP_ADDRESS> --master_port=29500 nvidia_run_squad_deepspeed.py --bert_model bert-large-uncased --do_train --do_lower_case --predict_batch_size 3 --do_predict --train_file /home/santosh/Projects/MicrosoftZero/DeepSpeed/DeepSpeedExamples/BingBertSquad/Data/train-v1.1.json --predict_file /home/santosh/Projects/MicrosoftZero/DeepSpeed/DeepSpeedExamples/BingBertSquad/Data/dev-v1.1.json --train_batch_size 8 --learning_rate 0.00003 --num_train_epochs 2.0 --max_seq_length 384 --doc_stride 128 --output_dir /home/santosh/Projects/MicrosoftZero/DeepSpeed/output --job_name deepspeed_3GPUs_24batch_size --gradient_accumulation_steps 2 --fp16 --deepspeed --deepspeed_config onebit_deepspeed_bsz24_config.json --dropout 0.1 --model_file /home/santosh/Projects/MicrosoftZero/TestModel/cased_L-12_H-768_A-12 --seed 12345 --preln
[2020-10-16 04:30:20,172] [WARNING] [stage2.py:32:<module>] apex was installed without --cpp_ext.  Falling back to Python flatten and unflatten.
[2020-10-16 04:30:20,190] [WARNING] [engine.py:48:<module>] Warning:  apex was installed without --cpp_ext.  Falling back to Python flatten and unflatten.
[2020-10-16 04:30:20,193] [INFO] [launch.py:71:main] 0 NCCL_VERSION 2.6.4
[2020-10-16 04:30:20,193] [INFO] [launch.py:78:main] WORLD INFO DICT: {'localhost': [0, 1, 2]}
[2020-10-16 04:30:20,194] [INFO] [launch.py:87:main] nnodes=1, num_local_procs=3, node_rank=0
[2020-10-16 04:30:20,194] [INFO] [launch.py:99:main] global_rank_mapping=defaultdict(<class 'list'>, {'localhost': [0, 1, 2]})
[2020-10-16 04:30:20,194] [INFO] [launch.py:100:main] dist_world_size=3
[2020-10-16 04:30:20,194] [INFO] [launch.py:103:main] Setting CUDA_VISIBLE_DEVICES=0,1,2
[2020-10-16 04:30:20,899] [WARNING] [stage2.py:32:<module>] apex was installed without --cpp_ext.  Falling back to Python flatten and unflatten.
[2020-10-16 04:30:20,917] [WARNING] [engine.py:48:<module>] Warning:  apex was installed without --cpp_ext.  Falling back to Python flatten and unflatten.
[2020-10-16 04:30:20,927] [WARNING] [stage2.py:32:<module>] apex was installed without --cpp_ext.  Falling back to Python flatten and unflatten.
[2020-10-16 04:30:20,945] [WARNING] [engine.py:48:<module>] Warning:  apex was installed without --cpp_ext.  Falling back to Python flatten and unflatten.
[2020-10-16 04:30:20,985] [WARNING] [stage2.py:32:<module>] apex was installed without --cpp_ext.  Falling back to Python flatten and unflatten.
[2020-10-16 04:30:21,003] [WARNING] [engine.py:48:<module>] Warning:  apex was installed without --cpp_ext.  Falling back to Python flatten and unflatten.
10/16/2020 04:30:21 - INFO - pytorch_pretrained_bert.tokenization -   loading vocabulary file https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-uncased-vocab.txt from cache at /home/santosh/.pytorch_pretrained_bert/9b3c03a36e83b13d5ba95ac965c9f9074a99e14340c523ab405703179e79fc46.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084
10/16/2020 04:30:21 - INFO - pytorch_pretrained_bert.tokenization -   loading vocabulary file https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-uncased-vocab.txt from cache at /home/santosh/.pytorch_pretrained_bert/9b3c03a36e83b13d5ba95ac965c9f9074a99e14340c523ab405703179e79fc46.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084
10/16/2020 04:30:21 - INFO - pytorch_pretrained_bert.tokenization -   loading vocabulary file https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-uncased-vocab.txt from cache at /home/santosh/.pytorch_pretrained_bert/9b3c03a36e83b13d5ba95ac965c9f9074a99e14340c523ab405703179e79fc46.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084
Traceback (most recent call last):
  File "nvidia_run_squad_deepspeed.py", line 1143, in <module>
    main()
  File "nvidia_run_squad_deepspeed.py", line 816, in main
    model = BertForQuestionAnsweringPreLN(bert_config, args)
  File "/home/santosh/Projects/MicrosoftZero/DeepSpeed/DeepSpeedExamples/BingBertSquad/turing/nvidia_modelingpreln.py", line 1500, in __init__
    self.bert = BertModel(config, args)
  File "/home/santosh/Projects/MicrosoftZero/DeepSpeed/DeepSpeedExamples/BingBertSquad/turing/nvidia_modelingpreln.py", line 927, in __init__
    self.embeddings = BertEmbeddings(config)
  File "/home/santosh/Projects/MicrosoftZero/DeepSpeed/DeepSpeedExamples/BingBertSquad/turing/nvidia_modelingpreln.py", line 354, in __init__
    self.LayerNorm = BertLayerNorm(config.hidden_size, eps=1e-12)
  File "/usr/local/lib/python3.6/dist-packages/apex/normalization/fused_layer_norm.py", line 133, in __init__
    fused_layer_norm_cuda = importlib.import_module("fused_layer_norm_cuda")
  File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 658, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 571, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 922, in create_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
ImportError: /usr/local/lib/python3.6/dist-packages/fused_layer_norm_cuda.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN6caffe26detail37_typeMetaDataInstance_preallocated_32E
Traceback (most recent call last):
  File "nvidia_run_squad_deepspeed.py", line 1143, in <module>
    main()
  File "nvidia_run_squad_deepspeed.py", line 816, in main
    model = BertForQuestionAnsweringPreLN(bert_config, args)
  File "/home/santosh/Projects/MicrosoftZero/DeepSpeed/DeepSpeedExamples/BingBertSquad/turing/nvidia_modelingpreln.py", line 1500, in __init__
    self.bert = BertModel(config, args)
  File "/home/santosh/Projects/MicrosoftZero/DeepSpeed/DeepSpeedExamples/BingBertSquad/turing/nvidia_modelingpreln.py", line 927, in __init__
    self.embeddings = BertEmbeddings(config)
  File "/home/santosh/Projects/MicrosoftZero/DeepSpeed/DeepSpeedExamples/BingBertSquad/turing/nvidia_modelingpreln.py", line 354, in __init__
    self.LayerNorm = BertLayerNorm(config.hidden_size, eps=1e-12)
  File "/usr/local/lib/python3.6/dist-packages/apex/normalization/fused_layer_norm.py", line 133, in __init__
    fused_layer_norm_cuda = importlib.import_module("fused_layer_norm_cuda")
  File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 658, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 571, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 922, in create_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
ImportError: /usr/local/lib/python3.6/dist-packages/fused_layer_norm_cuda.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN6caffe26detail37_typeMetaDataInstance_preallocated_32E
Traceback (most recent call last):
  File "nvidia_run_squad_deepspeed.py", line 1143, in <module>
    main()
  File "nvidia_run_squad_deepspeed.py", line 816, in main
    model = BertForQuestionAnsweringPreLN(bert_config, args)
  File "/home/santosh/Projects/MicrosoftZero/DeepSpeed/DeepSpeedExamples/BingBertSquad/turing/nvidia_modelingpreln.py", line 1500, in __init__
    self.bert = BertModel(config, args)
  File "/home/santosh/Projects/MicrosoftZero/DeepSpeed/DeepSpeedExamples/BingBertSquad/turing/nvidia_modelingpreln.py", line 927, in __init__
    self.embeddings = BertEmbeddings(config)
  File "/home/santosh/Projects/MicrosoftZero/DeepSpeed/DeepSpeedExamples/BingBertSquad/turing/nvidia_modelingpreln.py", line 354, in __init__
    self.LayerNorm = BertLayerNorm(config.hidden_size, eps=1e-12)
  File "/usr/local/lib/python3.6/dist-packages/apex/normalization/fused_layer_norm.py", line 133, in __init__
    fused_layer_norm_cuda = importlib.import_module("fused_layer_norm_cuda")
  File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 658, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 571, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 922, in create_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
ImportError: /usr/local/lib/python3.6/dist-packages/fused_layer_norm_cuda.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN6caffe26detail37_typeMetaDataInstance_preallocated_32E

I tried both the hf and tf versions of the model because it looked like the error was related to the model initialization.
This info might be helpful; in the same notebook I ran another pytorch training script without any errors.
I tried running run_squad_baseline.sh outside the jupyter notebook, directly in terminal. For both the hf and tf versions, I get a different error; it looks like it's not able to load the model from the directory. Here is a sample output.
/home/santosh/Projects/MicrosoftZero/DeepSpeed/DeepSpeedExamples/BingBertSquad$ bash run_squad_baseline.sh 3 /home/santosh/Projects/MsZeroTS/hfModel/ /home/santosh/Projects/MsZeroTS/DeepSpeed/DeepSpeedExamples/BingBertSquad/Data /home/santosh/Projects/MsZeroTS/output12
deepspeed --num_nodes 1 --num_gpus 3 nvidia_run_squad_baseline.py --bert_model bert-large-uncased --do_train --do_lower_case --do_predict --train_file /home/santosh/Projects/MsZeroTS/DeepSpeed/DeepSpeedExamples/BingBertSquad/Data/train-v1.1.json --predict_file /home/santosh/Projects/MsZeroTS/DeepSpeed/DeepSpeedExamples/BingBertSquad/Data/dev-v1.1.json --train_batch_size 8 --learning_rate 3e-5 --num_train_epochs 2.0 --max_seq_length 384 --doc_stride 128 --output_dir /home/santosh/Projects/MsZeroTS/output12 --job_name baseline_3GPUs_24batch_size --gradient_accumulation_steps 1 --fp16 --model_file /home/santosh/Projects/MsZeroTS/hfModel/
[2020-10-16 05:23:17,877] [WARNING] [runner.py:117:fetch_hostfile] Unable to find hostfile, will proceed with training with local resources only.
[2020-10-16 05:23:17,910] [INFO] [runner.py:355:main] cmd = /usr/bin/python -u -m deepspeed.launcher.launch --world_info=eyJsb2NhbGhvc3QiOiBbMCwgMSwgMl19 --master_addr=<IP_ADDRESS> --master_port=29500 nvidia_run_squad_baseline.py --bert_model bert-large-uncased --do_train --do_lower_case --do_predict --train_file /home/santosh/Projects/MsZeroTS/DeepSpeed/DeepSpeedExamples/BingBertSquad/Data/train-v1.1.json --predict_file /home/santosh/Projects/MsZeroTS/DeepSpeed/DeepSpeedExamples/BingBertSquad/Data/dev-v1.1.json --train_batch_size 8 --learning_rate 3e-5 --num_train_epochs 2.0 --max_seq_length 384 --doc_stride 128 --output_dir /home/santosh/Projects/MsZeroTS/output12 --job_name baseline_3GPUs_24batch_size --gradient_accumulation_steps 1 --fp16 --model_file /home/santosh/Projects/MsZeroTS/hfModel/
[2020-10-16 05:23:18,468] [INFO] [launch.py:71:main] 0 NCCL_VERSION 2.6.4
[2020-10-16 05:23:18,469] [INFO] [launch.py:78:main] WORLD INFO DICT: {'localhost': [0, 1, 2]}
[2020-10-16 05:23:18,469] [INFO] [launch.py:87:main] nnodes=1, num_local_procs=3, node_rank=0
[2020-10-16 05:23:18,469] [INFO] [launch.py:99:main] global_rank_mapping=defaultdict(<class 'list'>, {'localhost': [0, 1, 2]})
[2020-10-16 05:23:18,469] [INFO] [launch.py:100:main] dist_world_size=3
[2020-10-16 05:23:18,469] [INFO] [launch.py:103:main] Setting CUDA_VISIBLE_DEVICES=0,1,2
10/16/2020 05:23:19 - INFO - __main__ -   device: cuda:1 n_gpu: 1, distributed training: True, 16-bits training: True
10/16/2020 05:23:19 - INFO - __main__ -   device: cuda:2 n_gpu: 1, distributed training: True, 16-bits training: True
10/16/2020 05:23:19 - INFO - __main__ -   device: cuda:0 n_gpu: 1, distributed training: True, 16-bits training: True
10/16/2020 05:23:19 - INFO - pytorch_pretrained_bert.tokenization -   loading vocabulary file https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-uncased-vocab.txt from cache at /home/santosh/Projects/MicrosoftZero/DeepSpeed/cache/9b3c03a36e83b13d5ba95ac965c9f9074a99e14340c523ab405703179e79fc46.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084
10/16/2020 05:23:19 - INFO - pytorch_pretrained_bert.tokenization -   loading vocabulary file https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-uncased-vocab.txt from cache at /home/santosh/Projects/MicrosoftZero/DeepSpeed/cache/9b3c03a36e83b13d5ba95ac965c9f9074a99e14340c523ab405703179e79fc46.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084
10/16/2020 05:23:19 - INFO - pytorch_pretrained_bert.tokenization -   loading vocabulary file https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-uncased-vocab.txt from cache at /home/santosh/Projects/MicrosoftZero/DeepSpeed/cache/9b3c03a36e83b13d5ba95ac965c9f9074a99e14340c523ab405703179e79fc46.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084
VOCAB SIZE: 30528
10/16/2020 05:23:32 - INFO - __main__ -   Loading Pretrained Bert Encoder from: /home/santosh/Projects/MsZeroTS/hfModel/
Traceback (most recent call last):
  File "nvidia_run_squad_baseline.py", line 1158, in <module>
    main()
  File "nvidia_run_squad_baseline.py", line 872, in main
    map_location=torch.device("cpu"))
  File "/usr/local/lib/python3.6/dist-packages/torch/serialization.py", line 381, in load
    f = open(f, 'rb')
IsADirectoryError: [Errno 21] Is a directory: '/home/santosh/Projects/MsZeroTS/hfModel/'
VOCAB SIZE: 30528
10/16/2020 05:23:32 - INFO - __main__ -   Loading Pretrained Bert Encoder from: /home/santosh/Projects/MsZeroTS/hfModel/
Traceback (most recent call last):
  File "nvidia_run_squad_baseline.py", line 1158, in <module>
    main()
  File "nvidia_run_squad_baseline.py", line 872, in main
    map_location=torch.device("cpu"))
  File "/usr/local/lib/python3.6/dist-packages/torch/serialization.py", line 381, in load
    f = open(f, 'rb')
IsADirectoryError: [Errno 21] Is a directory: '/home/santosh/Projects/MsZeroTS/hfModel/'
VOCAB SIZE: 30528
10/16/2020 05:23:32 - INFO - __main__ -   Loading Pretrained Bert Encoder from: /home/santosh/Projects/MsZeroTS/hfModel/
Traceback (most recent call last):
  File "nvidia_run_squad_baseline.py", line 1158, in <module>
    main()
  File "nvidia_run_squad_baseline.py", line 872, in main
    map_location=torch.device("cpu"))
  File "/usr/local/lib/python3.6/dist-packages/torch/serialization.py", line 381, in load
    f = open(f, 'rb')
IsADirectoryError: [Errno 21] Is a directory: '/home/santosh/Projects/MsZeroTS/hfModel/'

@Santosh-Gupta, thanks for using DeepSpeed.
The second argument to script is the model file itself rather than a folder. Please see here and here for details.

Thanks for the info. It looks like I need to point to the checkpoint file in particular. So for a Tensorflow model, point to the model.ckpt.index (or is it the model.ckpt.meta ? ), and for a huggingface model, just point to the model.bin.
It seems that some of the model types need more than one file to be fully defined, I'm guessing the library will search the containing folder to search for any other files it needs, such as the config files. Is that what is going on, or is it somehow just using the checkpoint file?

I tried pointing to the model files itself. For huggingface, the .bin file, and for tensorflow, the .ckpt.index , ckpt.data-00000-of-00001 and .ckpt.meta file.
For each I end up with an AssertionError: Default process group is not initialized error.
Here is a full output for the huggingface .bin file
lr is 0.00003
seed is 12345
master port is 29500
dropout is 0.1
deepspeed --num_nodes 1 --num_gpus 3 --master_port=29500 --hostfile /dev/null nvidia_run_squad_deepspeed.py --bert_model bert-large-uncased --do_train --do_lower_case --predict_batch_size 3 --do_predict --train_file /home/santosh/projects/deepSpeed/testData/train-v1.1.json --predict_file /home/santosh/projects/deepSpeed/testData/dev-v1.1.json --train_batch_size 8 --learning_rate 0.00003 --num_train_epochs 2.0 --max_seq_length 384 --doc_stride 128 --output_dir /home/santosh/projects/deepSpeed/outputs/a1 --job_name deepspeed_3GPUs_24batch_size --gradient_accumulation_steps 2 --fp16 --deepspeed --deepspeed_config onebit_deepspeed_bsz24_config.json --dropout 0.1 --model_file /home/santosh/projects/deepSpeed/testModels/hf/pytorch_model.bin --seed 12345 --preln
[2020-10-22 19:06:30,721] [WARNING] [runner.py:117:fetch_hostfile] Unable to find hostfile, will proceed with training with local resources only.
[2020-10-22 19:06:30,737] [INFO] [runner.py:355:main] cmd = /usr/bin/python -u -m deepspeed.launcher.launch --world_info=eyJsb2NhbGhvc3QiOiBbMCwgMSwgMl19 --master_addr=<IP_ADDRESS> --master_port=29500 nvidia_run_squad_deepspeed.py --bert_model bert-large-uncased --do_train --do_lower_case --predict_batch_size 3 --do_predict --train_file /home/santosh/projects/deepSpeed/testData/train-v1.1.json --predict_file /home/santosh/projects/deepSpeed/testData/dev-v1.1.json --train_batch_size 8 --learning_rate 0.00003 --num_train_epochs 2.0 --max_seq_length 384 --doc_stride 128 --output_dir /home/santosh/projects/deepSpeed/outputs/a1 --job_name deepspeed_3GPUs_24batch_size --gradient_accumulation_steps 2 --fp16 --deepspeed --deepspeed_config onebit_deepspeed_bsz24_config.json --dropout 0.1 --model_file /home/santosh/projects/deepSpeed/testModels/hf/pytorch_model.bin --seed 12345 --preln
[2020-10-22 19:06:31,286] [INFO] [launch.py:71:main] 0 NCCL_VERSION 2.6.4
[2020-10-22 19:06:31,286] [INFO] [launch.py:78:main] WORLD INFO DICT: {'localhost': [0, 1, 2]}
[2020-10-22 19:06:31,286] [INFO] [launch.py:87:main] nnodes=1, num_local_procs=3, node_rank=0
[2020-10-22 19:06:31,286] [INFO] [launch.py:99:main] global_rank_mapping=defaultdict(<class 'list'>, {'localhost': [0, 1, 2]})
[2020-10-22 19:06:31,287] [INFO] [launch.py:100:main] dist_world_size=3
[2020-10-22 19:06:31,287] [INFO] [launch.py:103:main] Setting CUDA_VISIBLE_DEVICES=0,1,2
10/22/2020 19:06:32 - INFO - pytorch_pretrained_bert.tokenization -   loading vocabulary file https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-uncased-vocab.txt from cache at /home/santosh/.pytorch_pretrained_bert/9b3c03a36e83b13d5ba95ac965c9f9074a99e14340c523ab405703179e79fc46.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084
10/22/2020 19:06:32 - INFO - pytorch_pretrained_bert.tokenization -   loading vocabulary file https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-uncased-vocab.txt from cache at /home/santosh/.pytorch_pretrained_bert/9b3c03a36e83b13d5ba95ac965c9f9074a99e14340c523ab405703179e79fc46.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084
10/22/2020 19:06:32 - INFO - pytorch_pretrained_bert.tokenization -   loading vocabulary file https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-uncased-vocab.txt from cache at /home/santosh/.pytorch_pretrained_bert/9b3c03a36e83b13d5ba95ac965c9f9074a99e14340c523ab405703179e79fc46.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084
Traceback (most recent call last):
  File "nvidia_run_squad_deepspeed.py", line 1143, in <module>
    main()
  File "nvidia_run_squad_deepspeed.py", line 816, in main
    model = BertForQuestionAnsweringPreLN(bert_config, args)
  File "/home/santosh/projects/deepSpeed/DeepSpeed/DeepSpeedExamples/BingBertSquad/turing/nvidia_modelingpreln.py", line 1500, in __init__
    self.bert = BertModel(config, args)
  File "/home/santosh/projects/deepSpeed/DeepSpeed/DeepSpeedExamples/BingBertSquad/turing/nvidia_modelingpreln.py", line 930, in __init__
    self.apply(self.init_bert_weights)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 293, in apply
    module.apply(fn)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 293, in apply
    module.apply(fn)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 293, in apply
    module.apply(fn)
  [Previous line repeated 3 more times]
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 294, in apply
    fn(self)
  File "/home/santosh/projects/deepSpeed/DeepSpeed/DeepSpeedExamples/BingBertSquad/turing/nvidia_modelingpreln.py", line 730, in init_bert_weights
    if torch.distributed.get_rank() == 0:
  File "/usr/local/lib/python3.6/dist-packages/torch/distributed/distributed_c10d.py", line 562, in get_rank
    _check_default_pg()
  File "/usr/local/lib/python3.6/dist-packages/torch/distributed/distributed_c10d.py", line 191, in _check_default_pg
    "Default process group is not initialized"
AssertionError: Default process group is not initialized
Traceback (most recent call last):
  File "nvidia_run_squad_deepspeed.py", line 1143, in <module>
    main()
  File "nvidia_run_squad_deepspeed.py", line 816, in main
    model = BertForQuestionAnsweringPreLN(bert_config, args)
  File "/home/santosh/projects/deepSpeed/DeepSpeed/DeepSpeedExamples/BingBertSquad/turing/nvidia_modelingpreln.py", line 1500, in __init__
    self.bert = BertModel(config, args)
  File "/home/santosh/projects/deepSpeed/DeepSpeed/DeepSpeedExamples/BingBertSquad/turing/nvidia_modelingpreln.py", line 930, in __init__
    self.apply(self.init_bert_weights)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 293, in apply
    module.apply(fn)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 293, in apply
    module.apply(fn)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 293, in apply
    module.apply(fn)
  [Previous line repeated 3 more times]
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 294, in apply
    fn(self)
  File "/home/santosh/projects/deepSpeed/DeepSpeed/DeepSpeedExamples/BingBertSquad/turing/nvidia_modelingpreln.py", line 730, in init_bert_weights
    if torch.distributed.get_rank() == 0:
  File "/usr/local/lib/python3.6/dist-packages/torch/distributed/distributed_c10d.py", line 562, in get_rank
    _check_default_pg()
  File "/usr/local/lib/python3.6/dist-packages/torch/distributed/distributed_c10d.py", line 191, in _check_default_pg
    "Default process group is not initialized"
AssertionError: Default process group is not initialized
Traceback (most recent call last):
  File "nvidia_run_squad_deepspeed.py", line 1143, in <module>
    main()
  File "nvidia_run_squad_deepspeed.py", line 816, in main
    model = BertForQuestionAnsweringPreLN(bert_config, args)
  File "/home/santosh/projects/deepSpeed/DeepSpeed/DeepSpeedExamples/BingBertSquad/turing/nvidia_modelingpreln.py", line 1500, in __init__
    self.bert = BertModel(config, args)
  File "/home/santosh/projects/deepSpeed/DeepSpeed/DeepSpeedExamples/BingBertSquad/turing/nvidia_modelingpreln.py", line 930, in __init__
    self.apply(self.init_bert_weights)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 293, in apply
    module.apply(fn)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 293, in apply
    module.apply(fn)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 293, in apply
    module.apply(fn)
  [Previous line repeated 3 more times]
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 294, in apply
    fn(self)
  File "/home/santosh/projects/deepSpeed/DeepSpeed/DeepSpeedExamples/BingBertSquad/turing/nvidia_modelingpreln.py", line 730, in init_bert_weights
    if torch.distributed.get_rank() == 0:
  File "/usr/local/lib/python3.6/dist-packages/torch/distributed/distributed_c10d.py", line 562, in get_rank
    _check_default_pg()
  File "/usr/local/lib/python3.6/dist-packages/torch/distributed/distributed_c10d.py", line 191, in _check_default_pg
    "Default process group is not initialized"
AssertionError: Default process group is not initialized

@Santosh-Gupta Did you report a Default process group is not initialized error?

@Santosh-Gupta Did you report a Default process group is not initialized error?

Yes, sorry I noticed in the code that the model used was bert-large-cased where I was using bert-base-uncased so I wanted to see if switching the model made a difference, but I'm still getting errors.
For the following, I pointed the model file path to the .bin huggingface file.
I first tried running the code in a jupyter notebook, the server running on the deepspeed container. This was the full output
lr is 0.00003
seed is 12345
master port is 29500
dropout is 0.1
deepspeed --num_nodes 1 --num_gpus 3 --master_port=29500 --hostfile /dev/null nvidia_run_squad_deepspeed.py --bert_model bert-large-uncased --do_train --do_lower_case --predict_batch_size 3 --do_predict --train_file /home/santosh/projects/deepSpeed/testData/train-v1.1.json --predict_file /home/santosh/projects/deepSpeed/testData/dev-v1.1.json --train_batch_size 8 --learning_rate 0.00003 --num_train_epochs 2.0 --max_seq_length 384 --doc_stride 128 --output_dir /home/santosh/projects/deepSpeed/outputs/a1 --job_name deepspeed_3GPUs_24batch_size --gradient_accumulation_steps 2 --fp16 --deepspeed --deepspeed_config onebit_deepspeed_bsz24_config.json --dropout 0.1 --model_file /home/santosh/projects/deepSpeed/testModels/hf/pytorch_model.bin --seed 12345 --preln
[2020-10-23 08:33:23,956] [WARNING] [runner.py:117:fetch_hostfile] Unable to find hostfile, will proceed with training with local resources only.
[2020-10-23 08:33:24,011] [INFO] [runner.py:355:main] cmd = /usr/bin/python -u -m deepspeed.launcher.launch --world_info=eyJsb2NhbGhvc3QiOiBbMCwgMSwgMl19 --master_addr=<IP_ADDRESS> --master_port=29500 nvidia_run_squad_deepspeed.py --bert_model bert-large-uncased --do_train --do_lower_case --predict_batch_size 3 --do_predict --train_file /home/santosh/projects/deepSpeed/testData/train-v1.1.json --predict_file /home/santosh/projects/deepSpeed/testData/dev-v1.1.json --train_batch_size 8 --learning_rate 0.00003 --num_train_epochs 2.0 --max_seq_length 384 --doc_stride 128 --output_dir /home/santosh/projects/deepSpeed/outputs/a1 --job_name deepspeed_3GPUs_24batch_size --gradient_accumulation_steps 2 --fp16 --deepspeed --deepspeed_config onebit_deepspeed_bsz24_config.json --dropout 0.1 --model_file /home/santosh/projects/deepSpeed/testModels/hf/pytorch_model.bin --seed 12345 --preln
[2020-10-23 08:33:24,576] [INFO] [launch.py:71:main] 0 NCCL_VERSION 2.6.4
[2020-10-23 08:33:24,576] [INFO] [launch.py:78:main] WORLD INFO DICT: {'localhost': [0, 1, 2]}
[2020-10-23 08:33:24,576] [INFO] [launch.py:87:main] nnodes=1, num_local_procs=3, node_rank=0
[2020-10-23 08:33:24,576] [INFO] [launch.py:99:main] global_rank_mapping=defaultdict(<class 'list'>, {'localhost': [0, 1, 2]})
[2020-10-23 08:33:24,576] [INFO] [launch.py:100:main] dist_world_size=3
[2020-10-23 08:33:24,576] [INFO] [launch.py:103:main] Setting CUDA_VISIBLE_DEVICES=0,1,2
10/23/2020 08:33:25 - INFO - pytorch_pretrained_bert.tokenization -   loading vocabulary file https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-uncased-vocab.txt from cache at "/home/santosh"/.pytorch_pretrained_bert/9b3c03a36e83b13d5ba95ac965c9f9074a99e14340c523ab405703179e79fc46.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084
10/23/2020 08:33:25 - INFO - pytorch_pretrained_bert.tokenization -   loading vocabulary file https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-uncased-vocab.txt from cache at "/home/santosh"/.pytorch_pretrained_bert/9b3c03a36e83b13d5ba95ac965c9f9074a99e14340c523ab405703179e79fc46.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084
10/23/2020 08:33:25 - INFO - pytorch_pretrained_bert.tokenization -   loading vocabulary file https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-uncased-vocab.txt from cache at "/home/santosh"/.pytorch_pretrained_bert/9b3c03a36e83b13d5ba95ac965c9f9074a99e14340c523ab405703179e79fc46.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084
Traceback (most recent call last):
  File "nvidia_run_squad_deepspeed.py", line 1143, in <module>
    main()
  File "nvidia_run_squad_deepspeed.py", line 816, in main
    model = BertForQuestionAnsweringPreLN(bert_config, args)
  File "/home/santosh/projects/deepSpeed/DeepSpeed/DeepSpeedExamples/BingBertSquad/turing/nvidia_modelingpreln.py", line 1500, in __init__
    self.bert = BertModel(config, args)
  File "/home/santosh/projects/deepSpeed/DeepSpeed/DeepSpeedExamples/BingBertSquad/turing/nvidia_modelingpreln.py", line 930, in __init__
    self.apply(self.init_bert_weights)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 293, in apply
    module.apply(fn)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 293, in apply
    module.apply(fn)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 293, in apply
    module.apply(fn)
  [Previous line repeated 3 more times]
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 294, in apply
    fn(self)
  File "/home/santosh/projects/deepSpeed/DeepSpeed/DeepSpeedExamples/BingBertSquad/turing/nvidia_modelingpreln.py", line 730, in init_bert_weights
    if torch.distributed.get_rank() == 0:
  File "/usr/local/lib/python3.6/dist-packages/torch/distributed/distributed_c10d.py", line 562, in get_rank
    _check_default_pg()
  File "/usr/local/lib/python3.6/dist-packages/torch/distributed/distributed_c10d.py", line 191, in _check_default_pg
    "Default process group is not initialized"
AssertionError: Default process group is not initialized
Traceback (most recent call last):
  File "nvidia_run_squad_deepspeed.py", line 1143, in <module>
    main()
  File "nvidia_run_squad_deepspeed.py", line 816, in main
    model = BertForQuestionAnsweringPreLN(bert_config, args)
  File "/home/santosh/projects/deepSpeed/DeepSpeed/DeepSpeedExamples/BingBertSquad/turing/nvidia_modelingpreln.py", line 1500, in __init__
    self.bert = BertModel(config, args)
  File "/home/santosh/projects/deepSpeed/DeepSpeed/DeepSpeedExamples/BingBertSquad/turing/nvidia_modelingpreln.py", line 930, in __init__
    self.apply(self.init_bert_weights)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 293, in apply
    module.apply(fn)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 293, in apply
    module.apply(fn)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 293, in apply
    module.apply(fn)
  [Previous line repeated 3 more times]
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 294, in apply
    fn(self)
  File "/home/santosh/projects/deepSpeed/DeepSpeed/DeepSpeedExamples/BingBertSquad/turing/nvidia_modelingpreln.py", line 730, in init_bert_weights
    if torch.distributed.get_rank() == 0:
  File "/usr/local/lib/python3.6/dist-packages/torch/distributed/distributed_c10d.py", line 562, in get_rank
    _check_default_pg()
  File "/usr/local/lib/python3.6/dist-packages/torch/distributed/distributed_c10d.py", line 191, in _check_default_pg
    "Default process group is not initialized"
AssertionError: Default process group is not initialized
Traceback (most recent call last):
  File "nvidia_run_squad_deepspeed.py", line 1143, in <module>
    main()
  File "nvidia_run_squad_deepspeed.py", line 816, in main
    model = BertForQuestionAnsweringPreLN(bert_config, args)
  File "/home/santosh/projects/deepSpeed/DeepSpeed/DeepSpeedExamples/BingBertSquad/turing/nvidia_modelingpreln.py", line 1500, in __init__
    self.bert = BertModel(config, args)
  File "/home/santosh/projects/deepSpeed/DeepSpeed/DeepSpeedExamples/BingBertSquad/turing/nvidia_modelingpreln.py", line 930, in __init__
    self.apply(self.init_bert_weights)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 293, in apply
    module.apply(fn)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 293, in apply
    module.apply(fn)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 293, in apply
    module.apply(fn)
  [Previous line repeated 3 more times]
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 294, in apply
    fn(self)
  File "/home/santosh/projects/deepSpeed/DeepSpeed/DeepSpeedExamples/BingBertSquad/turing/nvidia_modelingpreln.py", line 730, in init_bert_weights
    if torch.distributed.get_rank() == 0:
  File "/usr/local/lib/python3.6/dist-packages/torch/distributed/distributed_c10d.py", line 562, in get_rank
    _check_default_pg()
  File "/usr/local/lib/python3.6/dist-packages/torch/distributed/distributed_c10d.py", line 191, in _check_default_pg
    "Default process group is not initialized"
AssertionError: Default process group is not initialized

I then tried running it directly in the deepspeed docker container terminal, in case there was an issue with jupyter, since there seems to be a different error.
lr is 0.00003
seed is 12345
master port is 29500
dropout is 0.1
deepspeed --num_nodes 1 --num_gpus 3 --master_port=29500 --hostfile /dev/null nvidia_run_squad_deepspeed.py --bert_model bert-large-uncased --do_train --do_lower_case --predict_batch_size 3 --do_predict --train_file /home/santosh/projects/deepSpeed/testData/train-v1.1.json --predict_file /home/santosh/projects/deepSpeed/testData/dev-v1.1.json --train_batch_size 8 --learning_rate 0.00003 --num_train_epochs 2.0 --max_seq_length 384 --doc_stride 128 --output_dir /home/santosh/projects/deepSpeed/outputs/a1ab --job_name deepspeed_3GPUs_24batch_size --gradient_accumulation_steps 2 --fp16 --deepspeed --deepspeed_config onebit_deepspeed_bsz24_config.json --dropout 0.1 --model_file /home/santosh/projects/deepSpeed/testModels/hf/pytorch_model.bin --seed 12345 --preln
[2020-10-23 09:11:34,436] [WARNING] [stage2.py:32:<module>] apex was installed without --cpp_ext.  Falling back to Python flatten and unflatten.
[2020-10-23 09:11:34,454] [WARNING] [engine.py:48:<module>] Warning:  apex was installed without --cpp_ext.  Falling back to Python flatten and unflatten.
[2020-10-23 09:11:34,458] [WARNING] [runner.py:117:fetch_hostfile] Unable to find hostfile, will proceed with training with local resources only.
[2020-10-23 09:11:34,514] [INFO] [runner.py:355:main] cmd = /usr/bin/python -u -m deepspeed.launcher.launch --world_info=eyJsb2NhbGhvc3QiOiBbMCwgMSwgMl19 --master_addr=<IP_ADDRESS> --master_port=29500 nvidia_run_squad_deepspeed.py --bert_model bert-large-uncased --do_train --do_lower_case --predict_batch_size 3 --do_predict --train_file /home/santosh/projects/deepSpeed/testData/train-v1.1.json --predict_file /home/santosh/projects/deepSpeed/testData/dev-v1.1.json --train_batch_size 8 --learning_rate 0.00003 --num_train_epochs 2.0 --max_seq_length 384 --doc_stride 128 --output_dir /home/santosh/projects/deepSpeed/outputs/a1ab --job_name deepspeed_3GPUs_24batch_size --gradient_accumulation_steps 2 --fp16 --deepspeed --deepspeed_config onebit_deepspeed_bsz24_config.json --dropout 0.1 --model_file /home/santosh/projects/deepSpeed/testModels/hf/pytorch_model.bin --seed 12345 --preln
[2020-10-23 09:11:35,131] [WARNING] [stage2.py:32:<module>] apex was installed without --cpp_ext.  Falling back to Python flatten and unflatten.
[2020-10-23 09:11:35,150] [WARNING] [engine.py:48:<module>] Warning:  apex was installed without --cpp_ext.  Falling back to Python flatten and unflatten.
[2020-10-23 09:11:35,153] [INFO] [launch.py:71:main] 0 NCCL_VERSION 2.6.4
[2020-10-23 09:11:35,154] [INFO] [launch.py:78:main] WORLD INFO DICT: {'localhost': [0, 1, 2]}
[2020-10-23 09:11:35,154] [INFO] [launch.py:87:main] nnodes=1, num_local_procs=3, node_rank=0
[2020-10-23 09:11:35,154] [INFO] [launch.py:99:main] global_rank_mapping=defaultdict(<class 'list'>, {'localhost': [0, 1, 2]})
[2020-10-23 09:11:35,154] [INFO] [launch.py:100:main] dist_world_size=3
[2020-10-23 09:11:35,154] [INFO] [launch.py:103:main] Setting CUDA_VISIBLE_DEVICES=0,1,2
[2020-10-23 09:11:35,886] [WARNING] [stage2.py:32:<module>] apex was installed without --cpp_ext.  Falling back to Python flatten and unflatten.
[2020-10-23 09:11:35,897] [WARNING] [stage2.py:32:<module>] apex was installed without --cpp_ext.  Falling back to Python flatten and unflatten.
[2020-10-23 09:11:35,905] [WARNING] [engine.py:48:<module>] Warning:  apex was installed without --cpp_ext.  Falling back to Python flatten and unflatten.
[2020-10-23 09:11:35,916] [WARNING] [engine.py:48:<module>] Warning:  apex was installed without --cpp_ext.  Falling back to Python flatten and unflatten.
[2020-10-23 09:11:35,953] [WARNING] [stage2.py:32:<module>] apex was installed without --cpp_ext.  Falling back to Python flatten and unflatten.
[2020-10-23 09:11:35,971] [WARNING] [engine.py:48:<module>] Warning:  apex was installed without --cpp_ext.  Falling back to Python flatten and unflatten.
10/23/2020 09:11:36 - INFO - pytorch_pretrained_bert.tokenization -   loading vocabulary file https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-uncased-vocab.txt from cache at /home/santosh/.pytorch_pretrained_bert/9b3c03a36e83b13d5ba95ac965c9f9074a99e14340c523ab405703179e79fc46.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084
10/23/2020 09:11:36 - INFO - pytorch_pretrained_bert.tokenization -   loading vocabulary file https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-uncased-vocab.txt from cache at /home/santosh/.pytorch_pretrained_bert/9b3c03a36e83b13d5ba95ac965c9f9074a99e14340c523ab405703179e79fc46.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084
10/23/2020 09:11:36 - INFO - pytorch_pretrained_bert.tokenization -   loading vocabulary file https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-uncased-vocab.txt from cache at /home/santosh/.pytorch_pretrained_bert/9b3c03a36e83b13d5ba95ac965c9f9074a99e14340c523ab405703179e79fc46.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084
Traceback (most recent call last):
  File "nvidia_run_squad_deepspeed.py", line 1143, in <module>
    main()
  File "nvidia_run_squad_deepspeed.py", line 816, in main
    model = BertForQuestionAnsweringPreLN(bert_config, args)
  File "/home/santosh/projects/deepSpeed/DeepSpeed/DeepSpeedExamples/BingBertSquad/turing/nvidia_modelingpreln.py", line 1500, in __init__
    self.bert = BertModel(config, args)
  File "/home/santosh/projects/deepSpeed/DeepSpeed/DeepSpeedExamples/BingBertSquad/turing/nvidia_modelingpreln.py", line 927, in __init__
    self.embeddings = BertEmbeddings(config)
  File "/home/santosh/projects/deepSpeed/DeepSpeed/DeepSpeedExamples/BingBertSquad/turing/nvidia_modelingpreln.py", line 354, in __init__
    self.LayerNorm = BertLayerNorm(config.hidden_size, eps=1e-12)
  File "/usr/local/lib/python3.6/dist-packages/apex/normalization/fused_layer_norm.py", line 133, in __init__
    fused_layer_norm_cuda = importlib.import_module("fused_layer_norm_cuda")
  File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 658, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 571, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 922, in create_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
ImportError: /usr/local/lib/python3.6/dist-packages/fused_layer_norm_cuda.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN6caffe26detail37_typeMetaDataInstance_preallocated_32E
Traceback (most recent call last):
  File "nvidia_run_squad_deepspeed.py", line 1143, in <module>
    main()
  File "nvidia_run_squad_deepspeed.py", line 816, in main
    model = BertForQuestionAnsweringPreLN(bert_config, args)
  File "/home/santosh/projects/deepSpeed/DeepSpeed/DeepSpeedExamples/BingBertSquad/turing/nvidia_modelingpreln.py", line 1500, in __init__
    self.bert = BertModel(config, args)
  File "/home/santosh/projects/deepSpeed/DeepSpeed/DeepSpeedExamples/BingBertSquad/turing/nvidia_modelingpreln.py", line 927, in __init__
    self.embeddings = BertEmbeddings(config)
  File "/home/santosh/projects/deepSpeed/DeepSpeed/DeepSpeedExamples/BingBertSquad/turing/nvidia_modelingpreln.py", line 354, in __init__
    self.LayerNorm = BertLayerNorm(config.hidden_size, eps=1e-12)
  File "/usr/local/lib/python3.6/dist-packages/apex/normalization/fused_layer_norm.py", line 133, in __init__
    fused_layer_norm_cuda = importlib.import_module("fused_layer_norm_cuda")
  File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 658, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 571, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 922, in create_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
ImportError: /usr/local/lib/python3.6/dist-packages/fused_layer_norm_cuda.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN6caffe26detail37_typeMetaDataInstance_preallocated_32E
Traceback (most recent call last):
  File "nvidia_run_squad_deepspeed.py", line 1143, in <module>
    main()
  File "nvidia_run_squad_deepspeed.py", line 816, in main
    model = BertForQuestionAnsweringPreLN(bert_config, args)
  File "/home/santosh/projects/deepSpeed/DeepSpeed/DeepSpeedExamples/BingBertSquad/turing/nvidia_modelingpreln.py", line 1500, in __init__
    self.bert = BertModel(config, args)
  File "/home/santosh/projects/deepSpeed/DeepSpeed/DeepSpeedExamples/BingBertSquad/turing/nvidia_modelingpreln.py", line 927, in __init__
    self.embeddings = BertEmbeddings(config)
  File "/home/santosh/projects/deepSpeed/DeepSpeed/DeepSpeedExamples/BingBertSquad/turing/nvidia_modelingpreln.py", line 354, in __init__
    self.LayerNorm = BertLayerNorm(config.hidden_size, eps=1e-12)
  File "/usr/local/lib/python3.6/dist-packages/apex/normalization/fused_layer_norm.py", line 133, in __init__
    fused_layer_norm_cuda = importlib.import_module("fused_layer_norm_cuda")
  File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 658, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 571, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 922, in create_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
ImportError: /usr/local/lib/python3.6/dist-packages/fused_layer_norm_cuda.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN6caffe26detail37_typeMetaDataInstance_preallocated_32E

I get the same errors when pointing to a tensorflow .ckpt.index file

These new import errors suggest a mismatch in cuda, apex, or torch versions. Can you double check those?
ImportError: /usr/local/lib/python3.6/dist-packages/fused_layer_norm_cuda.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN6caffe26detail37_typeMetaDataInstance_preallocated_32E

These new import errors suggest a mismatch in cuda, apex, or torch versions. Can you double check those?
ImportError: /usr/local/lib/python3.6/dist-packages/fused_layer_norm_cuda.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN6caffe26detail37_typeMetaDataInstance_preallocated_32E

torch is version 1.6.0
apex is Version: 0.1
Cuda is Version 10.0.130
I see that the latest Cuda version is 11.1, I'll upgrade it and check if that solves the issue.

Actually can you try out these sequence of commands in python to test compatibility of cuda, torch, and apex fusedlayernorm?
>>> import torch
>>> import apex
>>> input = torch.randn(20, 5, 10, 10)
>>> m = apex.normalization.FusedLayerNorm(input.size()[1:])
>>> output = m(input)

import torch
import apex
input = torch.randn(20, 5, 10, 10)
m = apex.normalization.FusedLayerNorm(input.size()[1:])
output = m(input)

Running this resulted in an error for the 4th line, here's the output
>>> import torch
>>> import apex
>>> input = torch.randn(20, 5, 10, 10)
>>> m = apex.normalization.FusedLayerNorm(input.size()[1:])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/dist-packages/apex/normalization/fused_layer_norm.py", line 133, in __init__
    fused_layer_norm_cuda = importlib.import_module("fused_layer_norm_cuda")
  File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 658, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 571, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 922, in create_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
ImportError: /usr/local/lib/python3.6/dist-packages/fused_layer_norm_cuda.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN6caffe26detail37_typeMetaDataInstance_preallocated_32E

This confirms an incompatibility issue independent of deepspeed. I vaguely recall that either torch 1.6.0 or apex 0.1 requires cuda 10.1, and so upgrading cuda should fix the problem. For reference my cuda/torch/apex versions are
cuda 10.1
torch 1.6.0
apex 0.1

Great, thanks tjruwase, I'll upgrade it and report back the results.

This confirms an incompatibility issue independent of deepspeed. I vaguely recall that either torch 1.6.0 or apex 0.1 requires cuda 10.1, and so upgrading cuda should fix the problem. For reference my cuda/torch/apex versions are
cuda 10.1
torch 1.6.0
apex 0.1

I am wondering if the deepspeed docker image has an outdated version of cuda, that's what it seems like here
https://github.com/microsoft/DeepSpeed/blob/master/docker/Dockerfile#L1
Currently nvcc -V in the deepspeed container is showing
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2018 NVIDIA Corporation
Built on Sat_Aug_25_21:08:01_CDT_2018
Cuda compilation tools, release 10.0, V10.0.130

Even though we have recently installed 11.1 on our machine. I created a fresh docker container from the original image, and it is still showing V10.0.130.

Yes, the deepspeed docker image is cuda 10.0, which is a bit confusing since it does not work with torch 1.6.0. However, it does work with torch 1.5.0 which was the deepspeed release was tested against. So it seems the options are (1) Downgrade to torch 1.5.0 to use cuda 10.0, or (2) Upgrade docker file to cuda 10.1 to use torch 1.6.0. Do either of these options work for you?

Yes, the deepspeed docker image is cuda 10.0, which is a bit confusing since it does not work with torch 1.6.0. However, it does work with torch 1.5.0 which the deepspeed release was tested against. So it seems the options are (1) Downgrade to torch 1.5.0 to use cuda 10.0, or (2) Upgrade docker file to cuda 10.1 to use torch 1.6.0. Do either of these options work for you?

Ahh I see. Yeah downgrading python should work; Cuda seems to be very tricky to work with on our machines. I'll downgrade python and report back the results.

Yes, the deepspeed docker image is cuda 10.0, which is a bit confusing since it does not work with torch 1.6.0. However, it does work with torch 1.5.0 which the deepspeed release was tested against. So it seems the options are (1) Downgrade to torch 1.5.0 to use cuda 10.0, or (2) Upgrade docker file to cuda 10.1 to use torch 1.6.0. Do either of these options work for you?

I downgraded my torch version to 1.5.0 to work with the official docker image, but I am still getting an error for that code snippet to test the compatibility against.
nvcc -V gives

release 10.0, V10.0.130

and 'torch.version' gives 1.5.0
but
import torch
import apex
input = torch.randn(20, 5, 10, 10)
m = apex.normalization.FusedLayerNorm(input.size()[1:])
output = m(input)

gives
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-8-ff19223b4d5c> in <module>
      4 import apex
      5 input = torch.randn(20, 5, 10, 10)
----> 6 m = apex.normalization.FusedLayerNorm(input.size()[1:])
      7 output = m(input)

/usr/local/lib/python3.6/dist-packages/apex/normalization/fused_layer_norm.py in __init__(self, normalized_shape, eps, elementwise_affine)
    131
    132         global fused_layer_norm_cuda
--> 133         fused_layer_norm_cuda = importlib.import_module("fused_layer_norm_cuda")
    134
    135         if isinstance(normalized_shape, numbers.Integral):

/usr/lib/python3.6/importlib/__init__.py in import_module(name, package)
    124                 break
    125             level += 1
--> 126     return _bootstrap._gcd_import(name[level:], package, level)
    127
    128

/usr/lib/python3.6/importlib/_bootstrap.py in _gcd_import(name, package, level)

/usr/lib/python3.6/importlib/_bootstrap.py in _find_and_load(name, import_)

/usr/lib/python3.6/importlib/_bootstrap.py in _find_and_load_unlocked(name, import_)

/usr/lib/python3.6/importlib/_bootstrap.py in _load_unlocked(spec)

/usr/lib/python3.6/importlib/_bootstrap.py in module_from_spec(spec)

/usr/lib/python3.6/importlib/_bootstrap_external.py in create_module(self, spec)

/usr/lib/python3.6/importlib/_bootstrap.py in _call_with_frames_removed(f, *args, **kwds)

ImportError: /usr/local/lib/python3.6/dist-packages/fused_layer_norm_cuda.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN6caffe26detail37_typeMetaDataInstance_preallocated_32E


mirrors/return_type_test is flaky on Safari

lib/mirrors/return_type_test is flaky on Safari. It just timed out (whereas previous runs didn't show up in the slow-tests list).
http://build.chromium.org/p/client.dart/builders/dart2js-safari-mac10.7-3-3/builds/6227/steps/dart2js tests failures/logs/stdio
Marking as flaky.

Mirrors support on the web/dart2js is removed. Announcement.


Fixes #66 now pressing on active button doesn't toggle the class

Fixes #66 removed the line that makes the button inactive.
@enigmaeth @mariobehling review.

Please fix typo in the commit message :smile: . Doesn't look good!

@enigmaeth changed the commit message :smile:

your branch is 1 commit behind, a rebase and it should be ready for merge

@rhemon done.

cool. one more thing, i dont think its good practice to have so manny commits in one simple pr where change serves only one purpose. usually comits should be for each meaningful change. some good github tutorial links are available, incase you dont know of squashing, rebase and amend, here . try to keep those practices in your prs :)

Squash the commits and rebase it on top of master. Rest looks good.

Squash and rebase please^^


refactor: removing msg-key

Please read contributing guidelines and remove this line.

Current dependencies on/for this PR:

#4856  👈
master

This stack of pull requests is managed by Graphite.

Docs Preview
Hey there! 👋 You can check your preview at https://65e0fe566dd3ca3347a02464--aztec-docs-dev.netlify.app

Benchmark results
No metrics with a significant change found.

Detailed results
All benchmarks are run on txs on the Benchmarking contract on the repository. Each tx consists of a batch call  to create_note and increment_balance, which guarantees that each tx has a private call, a nested private call, a public call, and a nested public call, as well as an emitted private note, an unencrypted log, and public storage read and write.
This benchmark source data is available in JSON format on S3 here.
Values are compared against data from master at commit 40adc5cd and shown if the difference exceeds 1%.
L2 block published to L1
Each column represents the number of txs on an L2 block published to L1.

Metric
8 txs
32 txs
64 txs

l1_rollup_calldata_size_in_bytes
4,932
17,348
33,892

l1_rollup_calldata_gas
62,856
232,436
458,080

l1_rollup_execution_gas
190,756
493,694
897,290

l2_block_processing_time_in_ms
1,210 (+1%)
4,584 (-7%)
9,087 (+1%)

note_successful_decrypting_time_in_ms
199
546 (-1%)
999 (+1%)

note_trial_decrypting_time_in_ms
8.64 (+12%)
71.0 (-6%)
132 (+3%)

l2_block_building_time_in_ms
16,297 (+2%)
64,372 (+1%)
129,065 (+1%)

l2_block_rollup_simulation_time_in_ms
12,449 (+2%)
49,159 (+1%)
98,744 (+1%)

l2_block_public_tx_process_time_in_ms
3,817 (+1%)
15,123 (+1%)
30,169 (+1%)

L2 chain processing
Each column represents the number of blocks on the L2 chain where each block has 16 txs.

Metric
5 blocks
10 blocks

node_history_sync_time_in_ms
15,448 (+2%)
29,006 (+1%)

note_history_successful_decrypting_time_in_ms
1,249 (-8%)
2,459 (+1%)

note_history_trial_decrypting_time_in_ms
89.8 (-5%)
171 (+20%)

node_database_size_in_bytes
18,600,016
35,287,120

pxe_database_size_in_bytes
29,923
59,478

Circuits stats
Stats on running time and I/O sizes collected for every circuit run across all benchmarks.

Circuit
circuit_simulation_time_in_ms
circuit_input_size_in_bytes
circuit_output_size_in_bytes

private-kernel-init
249
44,736
28,001

private-kernel-ordering
178
52,625
14,627

base-rollup
1,308
175,343
933

root-rollup
71.3 (+1%)
4,192
825

private-kernel-inner
315
73,715
28,001

public-kernel-app-logic
191 (-1%)
32,254
25,379

merge-rollup
5.73
2,712
933

Tree insertion stats
The duration to insert a fixed batch of leaves into each tree type.

Metric
0 deployed contracts

tx_size_in_bytes
19,147

Transaction processing duration by data writes.

Metric
0 new note hashes
1 new note hashes

tx_pxe_processing_time_ms
2,599 (+2%)
1,336

Metric
0 public data writes
1 public data writes

tx_sequencer_processing_time_ms
0.0302 (-4%)
468


DotOrg: New Suport Dashboard TM Script

There doesnt seem to be any changes to code here yet, just docblocks. Ping me when there are code changes and I will look again.

@bordoni pinged you in slack.


ImportError: cannot import name region

ImportError: cannot import name region @

Duplicate of #16. Please refer to INSTALL.md

thanks,when it offer the training code?
@lb1100


after install and create account. Then webpage show error.

Describe the bug
Here is the report:
Version: 0.25.1
Path: /project/bf/stories
Trace: TypeError: NetworkError when attempting to fetch resource.
    in
    in Unknown
    in Suspense
    in c
    in Unknown
    in div
    in Unknown
    in div
    in t
    in div
    in t
    in div
    in div
    in W
    in t
    in Connect(t)
    in c
    in Unknown
    in RouterContext
    in Router
    in o
    in c
    in t
    in DocumentTitle
    in SideEffect(DocumentTitle)

To Reproduce
Steps to reproduce the behavior:

Go to '...'
Click on '....'
Scroll down to '....'
See error

Desktop (please complete the following information):

OS: [e.g. iOS]
Browser [e.g. chrome, safari]
Version [e.g. 22]

Smartphone (please complete the following information):

Device: [e.g. iPhone6]
OS: [e.g. iOS8.1]
Browser [e.g. stock browser, safari]
Version [e.g. 22]

Additional context
Add any other context about the problem here.

Are you running Botfront on your local machine or a remote server? See here: https://botfront.io/docs/deployment/configuration/

remote server, after changing to public ip. now fixed. thanks


Support for turbo's morphing with prefilled_display_value

I'm using this gem together with turbo's morphing turned on. Everything seems to be working perfectly except there is one problem with prefilled_display_value.
Although turbo did morph the component (the <fieldset>) the value in prefilled_display_value does not show up as prefilled.
Steps to reproduce the behavior:

Set up combobox with associated object. E.g Post with belongs_to :category
Turn on turbo morph with %meta{name: "turbo-refresh-method", content: "morph"} (haml)
In posts#new set up the post with a default category - @post = Post.new category_id: params[:category_id]
Verify that prefilled_display_value exist in returned HTML in posts#new and that the combobox shows the prefilled category's name in combobox <input>
Submit the form to posts#create with some validation errors in it. Perhaps an empty title.
Verify that fieldset has prefilled_display_value is the same category name as before.

Expected behavior
prefilled_display_value should show up in combobox <input>
Screenshots

Sidenote: It looks as though there is no actual input[value] attribute for both cases (with or without morph). I wonder how the prefilled value was used to display it.

It looks like the value is set here, via connect(). I believe connect() is never recalled for turbo morphing. Not sure how one would go about resolving this bug yet.
https://github.com/josefarias/hotwire_combobox/blob/068cbca5bc7ccfe1e841b7258a165cfb3a7120ca/app/assets/javascripts/hw_combobox/models/combobox/selection.js#L10-L15

Great catch! Thanks for the report. Fixed in https://github.com/josefarias/hotwire_combobox/pull/132


Make repo name truncation configurable

The repo name is truncated at a hard 18 chars
https://github.com/dlvhdr/gh-dash/blob/d89be39ea4a8d930ee6f8e33eaa0cd982e758006/ui/components/pr/pr.go#LL143C64-L143C66
I do have a lot of repo starting with ansible-.. or terraform-.... To properly understand which repo we are talking about I would need more chars. Could we make this configurable?

@dlvhdr Can I ask if there is value in continuing to support truncation on repository names. As someone who often interacts with repositories that have long names, I actually prefer to increase column width and disable truncation (by modifying the code and rebuilding). And I can't find an easy way to make this configurable (see details below).
I have to assume that most users who interact with repository names longer than the hard-coded limit of 18 characters would prefer to remove truncation altogether. And users with repository names less than the hard-coded limit are unaffected either way.
I am happy to submit a PR that removes truncation altogether, if that's the direction you are willing to go in.
Configuring truncation length
If there is value in truncating the repo name, then I think it's actually quite tricky to make this configurable. Taking PRs as an example (though the same logic applies to issues too) - the truncation happens when creating the contents here - before the TextStyle is applied and the cell is rendered. But there is currently no mechanism to provide the column width at this point. The column width is not calculated until the entire table is rendered here - after the row is created and the TextStyle applied.
I explored truncating the text at this point that the table is rendered but it does not work as expected due to the text styling adding extra characters. And there is no obvious way to remove the text styling, truncate the contents, and re-add the text styling. The number of characters added by styling isn't consistent, so we can't just add an offset when truncating. And we can't "undo" the styling, because it's just a string.
Commit history for truncation
It looks like truncation used to be present for other fields - specifically title and author. But truncation for those fields was removed in this commit - leaving only truncation for repo name. It's not clear why truncation was removed for these other fields but kept for repo name.
Additional commit history:

This commit added truncation was for title, author, and repository name.
This commit removed truncation for title but kept it for author and repo name.
This commit re-added truncation for title.

Hi @robdimsdale, appreciate the discussion.
I'm not quite sure why I decided on the 18 character limit, I think it just made sense prior to adding configuration to allow columns to stretch etc. It probably what worked for the repos I worked with.
Anyway this isn't the ideal solution of course and I'd very much appreciate a contribution that gets rid of this :)
Possible solutions are:

Just delete the limit
Delete the limit and add a grow: true by default on repo names headers (which users can override with their config.yml.
In addition, add the repo owner + name to the preview pane which has more vertical space to fully display these values.

The calculation for how to distribute the available horizontal space is here
If I'm not mistaken, it first calculates how many terminal columns the non growing columns take, and then distributes the rest among the growing columns.

This makes sense to me. I think option 2 (which includes option 1) is an easy win and I'd like to start there. Just wanted to make sure I wasn't missing some context about truncating names.
I'm happy to explore the third idea as well. I think it's valuable. If it's easy, I'm happy to include it in the PR that removes the truncation. If it's more complex then I think I'll open a separate issue.

Cool, so without grow=true, how does that look? Can you add a screenshot?

Given that #224 was merged, can we close this issue?


How To read the error messages like error code, cause, retriable etc.. parameters from kafka error objects.

For Eg: consider below error,
KafkaJSConnectionError: Connection error: connect ECONNREFUSED <IP_ADDRESS>:9092
    at Socket.onError (/home/kafkacluster/consumer/node_modules/kafkajs/src/network/connection.js:210:23)
    at Socket.emit (node:events:527:28)
    at Socket.emit (node:domain:475:12)
    at emitErrorNT (node:internal/streams/destroy:157:8)
    at emitErrorCloseNT (node:internal/streams/destroy:122:3)
    at processTicksAndRejections (node:internal/process/task_queues:83:21) {
  retriable: true,
  helpUrl: undefined,
  broker: '<IP_ADDRESS>:9092',
  code: 'ECONNREFUSED',
  [cause]: undefined
}

How can I read the  values like retriable, broker, code, etc...and as well as Connection error: connect ECONNREFUSED <IP_ADDRESS>:9092 and also kafka error name KafkaJSConnectionError

My bad, refer to the code, https://github.com/tulios/kafkajs/blob/master/src/errors.js and accordingly attributes from error object can be read.


Fix getCollectionKey() and splitCollectionMemberKey() logic to consider more possibilities of keys / collection keys

Details
This PR:

Fixes both getCollectionKey() and splitCollectionMemberKey() logic to consider some keys / collection keys edge cases with underscore that weren't being considered before e.g. report__FAKE and cards_-1_Expensify Card, thus fixing https://github.com/Expensify/App/issues/48777 and https://github.com/Expensify/App/issues/49147.
Fixes canEvict tests in useOnyxTest.ts as they were actually wrong and were only working because of the faulty logic this PR is fixing.

Related Issues
https://github.com/Expensify/App/issues/48777
https://github.com/Expensify/App/issues/49147
Automated Tests

Manual Tests

Author Checklist

[x] I linked the correct issue in the ### Related Issues section above
[x] I wrote clear testing steps that cover the changes made in this PR

[x] I added steps for local testing in the Tests section
[x] I tested this PR with a High Traffic account against the staging or production API to ensure there are no regressions (e.g. long loading states that impact usability).

[x] I included screenshots or videos for tests on all platforms
[x] I ran the tests on all platforms & verified they passed on:

[x] Android / native
[x] Android / Chrome
[x] iOS / native
[x] iOS / Safari
[x] MacOS / Chrome / Safari
[x] MacOS / Desktop

[x] I verified there are no console errors (if there's a console error not related to the PR, report it or open an issue for it to be fixed)
[x] I followed proper code patterns (see Reviewing the code)

[x] I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick)
[x] I verified that the left part of a conditional rendering a React component is a boolean and NOT a string, e.g. myBool && <MyComponent />.
[x] I verified that comments were added to code that is not self explanatory
[x] I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
[x] I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
[x] I verified the JSDocs style guidelines (in STYLE.md) were followed

[x] If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
[x] I followed the guidelines as stated in the Review Guidelines
[x] I tested other components that can be impacted by my changes (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar are working as expected)
[x] I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
[x] I verified any variables that can be defined as constants (ie. in CONST.js or at the top of the file that uses the constant) are defined as such
[x] I verified that if a function's arguments changed that all usages have also been updated correctly
[x] If a new component is created I verified that:

[x] A similar component doesn't exist in the codebase
[x] All props are defined accurately and each prop has a /** comment above it */
[x] The file is named correctly
[x] The component has a clear name that is non-ambiguous and the purpose of the component can be inferred from the name alone
[x] The only data being stored in the state is data necessary for rendering and nothing else
[x] If we are not using the full Onyx data that we loaded, I've added the proper selector in order to ensure the component only re-renders when the data it is using changes
[x] For Class Components, any internal methods passed to components event handlers are bound to this properly so there are no scoping issues (i.e. for onClick={this.submit} the method this.submit should be bound to this in the constructor)
[x] Any internal methods bound to this are necessary to be bound (i.e. avoid this.submit = this.submit.bind(this); if this.submit is never passed to a component event handler like onClick)
[x] All JSX used for rendering exists in the render method
[x] The component has the minimum amount of code necessary for its purpose, and it is broken down into smaller components in order to separate concerns and functions

[x] If any new file was added I verified that:

[x] The file has a description of what it does and/or why is needed at the top of the file if the code is not self explanatory

[x] If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
[x] If the main branch was merged into this PR after a review, I tested again and verified the outcome was still expected according to the Test steps.
[x] I have checked off every checkbox in the PR author checklist, including those that don't apply to this PR.

Screenshots/Videos

Android: Native

Android: mWeb Chrome

iOS: Native

iOS: mWeb Safari

MacOS: Chrome / Safari

MacOS: Desktop

@s77rt please test this in App using steps from https://github.com/Expensify/App/issues/48777. Even if this is still marked as draft. Thank you!

Tested with https://github.com/Expensify/App/issues/49147 and I confirm it works


Series Label on Custom Series not working properly

Version
5.0.1
Reproduction link
https://echarts.apache.org/next/examples/en/editor.html?c=custom-profit
Steps to reproduce
1 - Open https://echarts.apache.org/next/examples/en/editor.html?c=custom-profit
       1.1 - First impression: the color of the label dont follow the color of the bars, like on echarts 4.x - (https://echarts.apache.org/examples/en/editor.html?c=custom-profit)

2 - add fontWeight: "900" or      textBorderColor: 'auto' or fontSize: 30
What is expected?
Changes on the label
What is actually happening?
Nothing

Work properly on https://echarts.apache.org/examples/en/editor.html?c=custom-profit (echarts 4.x)

This is not working on 5.1.0 =\


Union types cause problems as link targets.

Several problems occur with union types as link targets.
abstract type R {
    required property name -> str {
        delegated constraint exclusive;
    }
}
type S extending R {
    required property s -> str;
}
type T extending R {
    required property t -> str;
}
abstract type U {
    required property u -> str;
}
type V extending U, S, T;
type W {
    required property name -> str {
        constraint exclusive;
    }
    link w -> W;
}

# link target as a union type
type Z {
    # have 'name' in common
    link stw0 -> S | T | W;
    multi link stw1 -> S | T | W;
}

Inserting data seems to go without error:
INSERT S {name:= 'name0', s := 's0'};
INSERT T {name:= 'name0', t := 't0'};
INSERT V {name:= 'name1', s := 's1', t := 't1', u := 'u1'};
INSERT W {name := 'w0'};

INSERT Z {
    stw0 := (
        SELECT V FILTER .name = 'name1'
        LIMIT 1
    ),
    stw1 := (
        SELECT V FILTER .name = 'name1'
    ),
};

The problem arises with SELECT:
db> SELECT V {name};
{Object { name: 'name1' }}
db> SELECT Z {stw1: {name}};
{Object { stw1: {Object { name: 'name1' }, Object { name: 'name1' }} }}
db> SELECT Z {stw0: {name}};
InternalServerError: more than one row returned by a subquery used as an expression

The underlying issue is similar to that we found earlier for link unions.


Overlapping submenus

Hi,
I currently have two submenus that are above each other (import/export). If you quickly mouse over both of them, both submenus end up appearing at the same time and overlapping.

@codedreamin FIXED, update to v0.2


updated auth flow

Signed-off-by: Kial Jinnah<EMAIL_ADDRESS>Issue #: /bcgov/entity#12428
Description of changes:
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of the PPR license (Apache 2.0).

/gcbrun


Incompatible with PyZen

It appears that the modifications that django-nose makes to the test suite render it incompatible with PyZen.
https://github.com/coderanger/pyzen/issues/3

Probably still broken.  Lots of details in https://github.com/coderanger/pyzen/issues/3, would require some deep changes in test suite construction.


Wrong information in error message for overridden accessor

class Base {
    set data(value: number) {
        console.log("data changed to " + value);
    }
}

class Derived extends Base {
    data = 10;
}

This gives the error message:
Class 'Base' defines instance member accessor 'data', but extended class 'Derived' defines it as instance member property.

But there is clearly no

instance member accessor 'data'

because data is an accessor on the prototype.
What about the following?

'data' is defined as an accessor in class 'Base', but is overridden here in 'Derived' as an instance property.

Hmm, I read that differently.  I see it the distinction as being one of “instance property” vs. “static property” instead of “instance” vs “prototype”.
Your proposed message is a definite improvement in any case - it fits with the idea that TS doesn’t distinguish between own keys and prototype keys at the type level.
BUT... isn’t it now legal to do this override, per the class fields proposal?

BUT... isn’t it now legal to do this override, per the class fields proposal?

legal but questionable, especially with the different behavior in emitDefineForClassFields.


Deprecate node selector type - 1913

[Auto generated]
Number: [#1913]

openshift-python-wrapper[pull_request][35db54a0-4735-11ef-8cff-120a96232bc6][PR 1913]: Closing issue for PR: Deprecate node selector type.
PR was closed.


Add Block Producer genesis transaction

Please make sure your node is up and running. Join #validators Discord for help: https://discord.gg/QeJWCrE.


Option to disable check for valid folder in side-list

Description of the new feature / enhancement
Give option to disable check for valid folder in side-list
Scenario when this would be used?

prevent lag when chapter (as folder) count is high

Supporting information
No response

Its no related.
In manga reader, side-list's items are checked one time if they are valid as "chapter" (if they have images or are archive formats).
Checking if they have images can be slow and cause lag. And this feature is meant to disable that.


How to decouple a Component from it's own data fetching needs.

When a component requires a certain set of data to be fetched before rendering, I usually do something along those lines:
class Comp extends Component {
  componentWillMount() {
    this.props.requestData() // triggeres in action in the wrapper
  }
  render() {
    const { data } = this.props.
    return (
       <div>
         {/* do something with data */}
       </div>
    )
  }
}

export default connect(mapStateToProps, mapDispatchToProps)(Comp)

Now I have two React SPA's in which I want to use this component. In one App the needed data needs to be fetched before the component is mounted, in another the data already exists.
So I was wondering, if there is any best practice regarding the issue of fetching data that is a prerequisite for a component.
Is there a way for the wrapper to listen to Comp's lifecycle? Should I just rename  requestData to onComponentWillMount and listen to that for action triggering?

The data has three states with regard to loading:

The data is missing.
The data is loading. (optional)
The data is present.

Regardless of having or not having the data, dispatch a "data requested" action from componentWillMount or componentDidMount.
If you're using redux-thunk for actions' side effects, check for the current state of the data via getState (the second arg of the thunk).
If you're using redux-saga, use the select effect to check the state.
Handle all the states of the data in your render.
In the first app, the app will initialize in the state 1 and transition to (optionally) 2 and then to 3.
In the second one, the app will initialize in the state 3.
The flow is generally the same, just some of the states get skipped.

BTW, please don't think about the render function as "do something with the data".
In render, you should just describe how the data should be transformed into the markup (the HTML markup and other React components).
That's how the declarative render is different from imperative functions that call a set of DOM manipulations to display the data or to change the UI state.

This is a usage question, and should be asked on Stack Overflow instead.  You'll get more attention and better responses over there.