PPO example not working with DeepSpeed Stage 3 or FSDP

I've been trying to get a PPO trainer to work with fully sharded training using either DeepSpeed stage 3 or FSDP. However, no matter what exact configuration options I try, I cannot get even the example in the documentation to work. It seems the problems are with calling trainer.generate() when sampling a rollout. With FSDP, it usually crashes, with the exact error message depending on exact accelerate config (e.g. https://github.com/pytorch/pytorch/issues/82461 ) With DeepSpeed, the script seems to just hang and time out, without an error message.
Is this known behavior, and is there a working example or documentation of PPO + Deepspeed/FSDP anywhere?
To reproduce, inside examples:
accelerate launch --config_file accelerate_configs/deepspeed_zero3.yaml scripts/ppo.py
or even accelerate launch --config_file accelerate_configs/deepspeed_zero3.yaml helloworld.py

OK, so I've been doing more digging, and here's what I've found so far. The following is all with the scripts/ppo.py example. @kangqiyue

DeepSpeed: Example does work, just very slow.
It turns out, the example does work with Deepspeed - it's just that it's so slow, I thought it was stuck.  But if I turn down the batch and minibatch size to 4 or 16, it does run okay - still pretty slow though. A few things I've noticed:

Deepspeed is much slower than running the script without accelerate, even with a single GPU (roughly 5x as slow as running the script without accelerate launch). With 8x GPU, it takes another 4x hit, roughly. Running it in a debugger ats another 2x slowdown. Not sure if all of this is expected.
Deepspeed is especially slow on the second iteration, for some reason, and it seems to be in the trainer.generate() call. This happens only in the second iteration for some reason, and it's order of magnitude slower - think one hour (!) with the original batch size of 128, with 8xA100 GPUs. Hence my previous thinking that it was just frozen completely. The first iteration is fine, and any subsequent iterations seem fine.
I am getting Deepspeed warnings along the lines of Invalidate trace cache @ step 0: expected module 0, but got module 1 - possibly this is related to the slowdown?
If I try to modify the script to run it with a different model (llama-2-7b), Deepspeed crashes with an error, although now the problem seems to be in the backward step. The error is

AssertionError    assert param.ds_status == ZeroParamStatus.AVAILABLE, param.ds_summary()
: {'id': 331, 'status': 'INFLIGHT', 'numel': 0, 'ds_numel': 16777216, 'shape': (0,), 'ds_shape': (4096, 4096), 'requires_grad': True, 'grad_shape': None, 'persist': False, 'active_sub_modules': {66}, 'ds_tensor.shape': torch.Size([2097152])}
Possibly relevant discussion [here] (https://github.com/microsoft/DeepSpeed/issues/3068) - what's interesting is that for them it also seems to happen when training after a forward call.

* With FSDP, I get the error
```return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: The tensor has a non-zero number of elements, but its data is not allocated yet. Caffe2 uses a lazy allocation, so you will need to call mutable_data() or raw_mutable_data() to actually allocate memory.```
That's also discussed e.g. [here](https://github.com/pytorch/pytorch/issues/82461) or [here](https://discuss.huggingface.co/t/how-to-do-model-generate-in-evaluation-steps-with-trainer-fsdp/35141). Some comments in that discussion mention that running a forward pass through the model first solved the problem for them, but I haven't been able to make that work (yet).

That's about as far as I got so far.

OK, so I've been doing more digging, and here's what I've found so far. The following is all with the scripts/ppo.py example. @kangqiyue

DeepSpeed: Example does work, just very slow.
It turns out, the example does work with Deepspeed - it's just that it's so slow, I thought it was stuck.  But if I turn down the batch and minibatch size to 4 or 16, it does run okay - still pretty slow though. A few things I've noticed:

Deepspeed is much slower than running the script without accelerate, even with a single GPU (roughly 5x as slow as running the script without accelerate launch). With 8x GPU, it takes another 4x hit, roughly. Running it in a debugger ats another 2x slowdown. Not sure if all of this is expected.
Deepspeed is especially slow on the second iteration, for some reason, and it seems to be in the trainer.generate() call. This happens only in the second iteration for some reason, and it's order of magnitude slower - think one hour (!) with the original batch size of 128, with 8xA100 GPUs. Hence my previous thinking that it was just frozen completely. The first iteration is fine, and any subsequent iterations seem fine.
I am getting Deepspeed warnings along the lines of Invalidate trace cache @ step 0: expected module 0, but got module 1 - possibly this is related to the slowdown?
If I try to modify the script to run it with a different model (llama-2-7b), Deepspeed crashes with an error, although now the problem seems to be in the backward step. The error is

AssertionError    assert param.ds_status == ZeroParamStatus.AVAILABLE, param.ds_summary()
: {'id': 331, 'status': 'INFLIGHT', 'numel': 0, 'ds_numel': 16777216, 'shape': (0,), 'ds_shape': (4096, 4096), 'requires_grad': True, 'grad_shape': None, 'persist': False, 'active_sub_modules': {66}, 'ds_tensor.shape': torch.Size([2097152])}

Possibly relevant discussion here - what's interesting is that for them it also seems to happen when training after a forward call.

With FSDP, I get the error

return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: The tensor has a non-zero number of elements, but its data is not allocated yet. Caffe2 uses a lazy allocation, so you will need to call mutable_data() or raw_mutable_data() to actually allocate memory.

That's also discussed e.g. here or here. Some comments in that discussion mention that running a forward pass through the model first solved the problem for them, but I haven't been able to make that work (yet).
That's about as far as I got so far.

@mgerstgrasser

I run the script with accelerate, and I tried Deepspeed stage 2 and stage 3. For Deepspeed stage 3, I met the same question: for the zero iteration took too laong time and the error is :

RuntimeError: NCCL communicator was aborted on rank 0.  Original reason for failure was: [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1092, OpType=ALLREDUCE, Timeout(ms)=1800000) ran for 1805633 milliseconds before timing out. [E ProcessGroupNCCL.cpp:455] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. [E ProcessGroupNCCL.cpp:460] To avoid data inconsistency, we are taking the entire process down.

Therefore I tried Deepspeed stage 2, and I met OOM. So, I have to change from full finetue to lora finetune. For this, setting, I can run the script on 1A100. But when I tried to use 8A100, then it meet device error:

   return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse) RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0! (when checking argument for argument index in method wrapper_CUDA__index_select) Traceback (most recent call last):

Finally, I changed to a much light reward model, and I run the script with Deepspeed stage 2. The iteration is fine, except the generation process is much slow. If I generate 256 tokens, the iteration, the time is 280.36s/it. However, I run this script with Deepspeed stage 2. It will not stuck at the first iteration. Try Deepspeed stage 2 for finetuning! (note: need to freeze share layers)

Invalidate trace cache @ step 0: expected module 0, but got module 1.  I alse got this message error, and I don't know how to do with it. I think we should dig more on this warning.

For FSDP, I did not use it, therefore I have no idea how to run FDSP.

@mgerstgrasser

In addition, when I start training, the rewards decrease, as follows:

I tested my model on MT-bench, it seems the model is getting worse, when compared to my base model. I wonder is there any idea, that can increase the mean rewards, instead of decreasing the mean rewards? I have increased the batch size to 128

@mgerstgrasser

I run the script with accelerate, and I tried Deepspeed stage 2 and stage 3. For Deepspeed stage 3, I met the same question: for the zero iteration took too a long time and the error is shown as below:
RuntimeError: NCCL communicator was aborted on rank 0. Original reason for failure was: [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1092, OpType=ALLREDUCE, Timeout(ms)=1800000) ran for 1805633 milliseconds before timing out. [E ProcessGroupNCCL.cpp:455] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. [E ProcessGroupNCCL.cpp:460] To avoid data inconsistency, we are taking the entire process down.

Hm, that could be that it's the same issue I encountered, just for you it takes so long it triggers a timeout. Maybe there's a way to increase timeouts? Or try stage 3 with a smaller batch size, e.g. 4?
I think it is the same question that we have met. It very wired for the first iteration which take a very long time. I think what you observed could be the problem that I met. As you have pointed:

DeepSpeed: Example does work, just very slow.
It turns out, the example does work with Deepspeed - it's just that it's so slow, I thought it was stuck. But if I turn down the batch and minibatch size to 4 or 16, it does run okay - still pretty slow though.

Interesting, I haven't run into that, but it's curious that the error happens at the exact same line as the FSDP error! Do you maybe need a somewhere, e.g. on the inputs?.to(ppo_trainer.accelerator.device)

For lora tuning, I just used the default code in ppo.py, as follows:
` query_tensors = batch["input_ids"]
# Get response from gpt2
response_tensors, ref_response_tensors = ppo_trainer.generate(
    query_tensors, return_prompt=False, generate_ref_response=True, **generation_kwargs
)`

I think the  ppo_trainer.generate may be could automatically process the input tensors, so I did not handle them manually.  As you pointed, maybe I could try: .to(ppo_trainer.accelerator.device)

Yes, agree. In my own testing, it seems 99% of the time each iteration is spent generating, that's clearly not good. I imagine there must be some bug with sharded generation, but I don't know much more than that right now.

In addition, I have a question about the training model and the ref_model. I have not figure out if the parameters of the ref_model is updated or not in the training process. As far as I'm concerned, the training model's parameters are updated in the training. But for the ref_model, I do not know if the ref_model's parameters are updated or not in the training. Can you tell me your opinion on this question? I think if the ref_model is frozen, we can speed up the ref_model's generation.

Shared layers between what? Actor and critic? Or are you somehow sharing layers between the actor-critic model and the reward model?

My origin reward model is a 7B model, therefore I change it to a very very small model (very very small) that I won't meet OOM errors. For the training, the training model (Actor) and the critic model (ref model, if I understand right, I usually use model and ref_model) used share layers, using the create_reference_model function. Try it at Deepspeed stage 2, if you met OOM, then freeze all layers except the last layer. Then I think you can run your script.

@vwxyzjn I've been using your script for a specific use case, and it proceeded to the training phase —thank you for providing it.
However, I cannot find ways to safely save the model at certain checkpoints/ after some epochs/ at the end using ppo.py.
The file does not provide a direct way to save models after certain epochs, so I manually try to save the model.
When I use either ppo_trainer._save_pretrained(path_name) or ppo_trainer.save_pretrained(path_name) to save the model, it seems to save only a subset of layers. For instance, the output indicates that shared tensors like 'model.layers.9.mlp.up_proj.weight', 'model.layers.10.post_attention_layernorm.weight', and several others are removed during the save process. The message displayed is:
Removed shared tensor {'model.layers.9.mlp.up_proj.weight' .... q_proj.weight', 'model.layers.10.post_attention_layernorm.weight', 'model.layers.24.mlp.down_proj.weight', 'model.layers.1.self_attn.k_proj.weight', 'model.layers.4.self_attn.q_proj.weight', 'model.layers.13.mlp.up_proj.weight'} while saving. This should be OK, but check by verifying that you don't receive any warning while reloading

Subsequently, when I attempt to reload that saved model using AutoModelForCausalLM.from_pretrained, it results in a warning message:
Some weights of MistralForCausalLM were not initialized from the model checkpoint at my_save_path and are newly initialized [those layers above]

This suggests that the model is not being saved correctly. Could you provide any guidance on how to ensure the entire model is saved and can be properly reloaded? My code for saving the model to a local directory is as straightforward as the commands mentioned above.
The accelerate script I ran is exactly like the above, and I modified examples/accelerate_configs/deepspeed_zero2.yaml to be compatible with 4 GPUs instead of the default 8. I'm running on 4xA100 80GB. I modified ppo.py to save ppo_traianer after X epochs.  I tried both saving at every GPU or conditioning to only save ppo_trainer if device==0, and neither of them worked.
Thank you for your guidance.
transformers.__version__, accelerate.__version__, deepspeed.__version__
>> ('4.35.1', '0.24.1', '0.12.5')

We have some preliminary tests working with 7B models using PPO.
Can you try running accelerate launch --config_file examples/accelerate_configs/deepspeed_zero2.yaml examples/scripts/ppo.py --ppo_config.exp_name ppo_Cerebras-GPT-6.7B_grad_accu_deepspeed_stage2 --ppo_config.batch_size 32 --ppo_config.mini_batch_size 32 --ppo_config.log_with wandb --ppo_config.model_name cerebras/Cerebras-GPT-6.7B --ppo_config.reward_model sentiment-analysis:cerebras/Cerebras-GPT-6.7B see at least if your can run this script with 8 GPUs?

Hi @vwxyzjn - thanks so much for looking into this, and sorry it's taken me a while to reply. I've finally had time to try this out, and unfortunately, this still fails for me! In fact, the script fails even with the default model when using deepspeed for me (i.e. running the command you suggested, but without setting either ppo_config.model_name or ppo_config.reward_model).
The error seems to be a timeout somewhere during the generate() call; the exact error message varies between runs, but it seems to always be in that method. It also seems that iteration 0 runs fine, but then iteration 1 throws the error. The seems to match with some of my earlier experiments where I could get it to run with much smaller batch sizes, but generate() in iteration 1 took a very, very long time - possibly that's still the issue here, but with a bigger batch size it leads to a timeout?
Interesting that it seems to work for others though. Could I ask either or both of you (@vwxyzjn and @viethoangtranduong ) for your pip freeze, CUDA version, and maybe even linux kernel version to be safe? I'm also copyig in mine below, in case one of you sees something obvious that could be an issue.
Thanks so much!
error message:
[2024-01-01 17:25:19,192] [INFO] [real_accelerator.py:161:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-01-01 17:25:19,192] [INFO] [real_accelerator.py:161:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-01-01 17:25:19,192] [INFO] [real_accelerator.py:161:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-01-01 17:25:19,192] [INFO] [real_accelerator.py:161:get_accelerator] Setting ds_accelerator to cuda (auto detect)
/n/home04/mgerstgrasser/.conda/envs/tortoise/lib/python3.11/site-packages/tyro/_resolver.py:313: UserWarning: <class 'int'> does not match any type in Union: [<class 'float'>, <class 'NoneType'>]
  warnings.warn(
/n/home04/mgerstgrasser/.conda/envs/tortoise/lib/python3.11/site-packages/tyro/_resolver.py:313: UserWarning: <class 'int'> does not match any type in Union: [<class 'float'>, <class 'NoneType'>]
  warnings.warn(
/n/home04/mgerstgrasser/.conda/envs/tortoise/lib/python3.11/site-packages/tyro/_resolver.py:313: UserWarning: <class 'int'> does not match any type in Union: [<class 'float'>, <class 'NoneType'>]
  warnings.warn(
/n/home04/mgerstgrasser/.conda/envs/tortoise/lib/python3.11/site-packages/tyro/_resolver.py:313: UserWarning: <class 'int'> does not match any type in Union: [<class 'float'>, <class 'NoneType'>]
  warnings.warn(
/n/home04/mgerstgrasser/.conda/envs/tortoise/lib/python3.11/site-packages/tyro/_fields.py:307: UserWarning: The field world_size is annotated with type <class 'int'>, but the default value None has type <class 'NoneType'>. We'll try to handle this gracefully, but it may cause unexpected behavior.
  warnings.warn(
/n/home04/mgerstgrasser/.conda/envs/tortoise/lib/python3.11/site-packages/tyro/_fields.py:307: UserWarning: The field world_size is annotated with type <class 'int'>, but the default value None has type <class 'NoneType'>. We'll try to handle this gracefully, but it may cause unexpected behavior.
  warnings.warn(
/n/home04/mgerstgrasser/.conda/envs/tortoise/lib/python3.11/site-packages/tyro/_fields.py:307: UserWarning: The field world_size is annotated with type <class 'int'>, but the default value None has type <class 'NoneType'>. We'll try to handle this gracefully, but it may cause unexpected behavior.
  warnings.warn(
/n/home04/mgerstgrasser/.conda/envs/tortoise/lib/python3.11/site-packages/tyro/_fields.py:307: UserWarning: The field world_size is annotated with type <class 'int'>, but the default value None has type <class 'NoneType'>. We'll try to handle this gracefully, but it may cause unexpected behavior.
  warnings.warn(
/n/home04/mgerstgrasser/.conda/envs/tortoise/lib/python3.11/site-packages/tyro/_fields.py:307: UserWarning: The field global_backward_batch_size is annotated with type <class 'int'>, but the default value None has type <class 'NoneType'>. We'll try to handle this gracefully, but it may cause unexpected behavior.
  warnings.warn(
/n/home04/mgerstgrasser/.conda/envs/tortoise/lib/python3.11/site-packages/tyro/_fields.py:307: UserWarning: The field global_backward_batch_size is annotated with type <class 'int'>, but the default value None has type <class 'NoneType'>. We'll try to handle this gracefully, but it may cause unexpected behavior.
  warnings.warn(
/n/home04/mgerstgrasser/.conda/envs/tortoise/lib/python3.11/site-packages/tyro/_fields.py:307: UserWarning: The field global_batch_size is annotated with type <class 'int'>, but the default value None has type <class 'NoneType'>. We'll try to handle this gracefully, but it may cause unexpected behavior.
  warnings.warn(
/n/home04/mgerstgrasser/.conda/envs/tortoise/lib/python3.11/site-packages/tyro/_fields.py:307: UserWarning: The field global_batch_size is annotated with type <class 'int'>, but the default value None has type <class 'NoneType'>. We'll try to handle this gracefully, but it may cause unexpected behavior.
  warnings.warn(
/n/home04/mgerstgrasser/.conda/envs/tortoise/lib/python3.11/site-packages/tyro/_fields.py:307: UserWarning: The field global_backward_batch_size is annotated with type <class 'int'>, but the default value None has type <class 'NoneType'>. We'll try to handle this gracefully, but it may cause unexpected behavior.
  warnings.warn(
/n/home04/mgerstgrasser/.conda/envs/tortoise/lib/python3.11/site-packages/tyro/_fields.py:307: UserWarning: The field global_backward_batch_size is annotated with type <class 'int'>, but the default value None has type <class 'NoneType'>. We'll try to handle this gracefully, but it may cause unexpected behavior.
  warnings.warn(
/n/home04/mgerstgrasser/.conda/envs/tortoise/lib/python3.11/site-packages/tyro/_fields.py:307: UserWarning: The field global_batch_size is annotated with type <class 'int'>, but the default value None has type <class 'NoneType'>. We'll try to handle this gracefully, but it may cause unexpected behavior.
  warnings.warn(
/n/home04/mgerstgrasser/.conda/envs/tortoise/lib/python3.11/site-packages/tyro/_fields.py:307: UserWarning: The field global_batch_size is annotated with type <class 'int'>, but the default value None has type <class 'NoneType'>. We'll try to handle this gracefully, but it may cause unexpected behavior.
  warnings.warn(
/n/home04/mgerstgrasser/.conda/envs/tortoise/lib/python3.11/site-packages/tyro/_fields.py:307: UserWarning: The field base_model_name_or_path is annotated with type <class 'str'>, but the default value None has type <class 'NoneType'>. We'll try to handle this gracefully, but it may cause unexpected behavior.
  warnings.warn(
/n/home04/mgerstgrasser/.conda/envs/tortoise/lib/python3.11/site-packages/tyro/_fields.py:307: UserWarning: The field base_model_name_or_path is annotated with type <class 'str'>, but the default value None has type <class 'NoneType'>. We'll try to handle this gracefully, but it may cause unexpected behavior.
  warnings.warn(
/n/home04/mgerstgrasser/.conda/envs/tortoise/lib/python3.11/site-packages/tyro/_fields.py:307: UserWarning: The field base_model_name_or_path is annotated with type <class 'str'>, but the default value None has type <class 'NoneType'>. We'll try to handle this gracefully, but it may cause unexpected behavior.
  warnings.warn(
/n/home04/mgerstgrasser/.conda/envs/tortoise/lib/python3.11/site-packages/tyro/_fields.py:307: UserWarning: The field base_model_name_or_path is annotated with type <class 'str'>, but the default value None has type <class 'NoneType'>. We'll try to handle this gracefully, but it may cause unexpected behavior.
  warnings.warn(
/n/home04/mgerstgrasser/.conda/envs/tortoise/lib/python3.11/site-packages/tyro/_fields.py:307: UserWarning: The field revision is annotated with type <class 'str'>, but the default value None has type <class 'NoneType'>. We'll try to handle this gracefully, but it may cause unexpected behavior.
  warnings.warn(
/n/home04/mgerstgrasser/.conda/envs/tortoise/lib/python3.11/site-packages/tyro/_fields.py:307: UserWarning: The field revision is annotated with type <class 'str'>, but the default value None has type <class 'NoneType'>. We'll try to handle this gracefully, but it may cause unexpected behavior.
  warnings.warn(
/n/home04/mgerstgrasser/.conda/envs/tortoise/lib/python3.11/site-packages/tyro/_fields.py:307: UserWarning: The field revision is annotated with type <class 'str'>, but the default value None has type <class 'NoneType'>. We'll try to handle this gracefully, but it may cause unexpected behavior.
  warnings.warn(
/n/home04/mgerstgrasser/.conda/envs/tortoise/lib/python3.11/site-packages/tyro/_fields.py:307: UserWarning: The field revision is annotated with type <class 'str'>, but the default value None has type <class 'NoneType'>. We'll try to handle this gracefully, but it may cause unexpected behavior.
  warnings.warn(
/n/home04/mgerstgrasser/.conda/envs/tortoise/lib/python3.11/site-packages/tyro/_resolver.py:313: UserWarning: <class 'NoneType'> does not match any type in Union: [<class 'str'>, <enum 'PeftType'>]
  warnings.warn(
/n/home04/mgerstgrasser/.conda/envs/tortoise/lib/python3.11/site-packages/tyro/_resolver.py:313: UserWarning: <class 'NoneType'> does not match any type in Union: [<class 'str'>, <enum 'PeftType'>]
  warnings.warn(
/n/home04/mgerstgrasser/.conda/envs/tortoise/lib/python3.11/site-packages/tyro/_resolver.py:313: UserWarning: <class 'NoneType'> does not match any type in Union: [<class 'str'>, <enum 'PeftType'>]
  warnings.warn(
/n/home04/mgerstgrasser/.conda/envs/tortoise/lib/python3.11/site-packages/tyro/_resolver.py:313: UserWarning: <class 'NoneType'> does not match any type in Union: [<class 'str'>, <enum 'PeftType'>]
  warnings.warn(
/n/home04/mgerstgrasser/.conda/envs/tortoise/lib/python3.11/site-packages/tyro/_resolver.py:313: UserWarning: <class 'NoneType'> does not match any type in Union: [<class 'str'>, <enum 'TaskType'>]
  warnings.warn(
/n/home04/mgerstgrasser/.conda/envs/tortoise/lib/python3.11/site-packages/tyro/_resolver.py:313: UserWarning: <class 'NoneType'> does not match any type in Union: [<class 'str'>, <enum 'TaskType'>]
  warnings.warn(
/n/home04/mgerstgrasser/.conda/envs/tortoise/lib/python3.11/site-packages/tyro/_resolver.py:313: UserWarning: <class 'NoneType'> does not match any type in Union: [<class 'str'>, <enum 'TaskType'>]
  warnings.warn(
/n/home04/mgerstgrasser/.conda/envs/tortoise/lib/python3.11/site-packages/tyro/_resolver.py:313: UserWarning: <class 'NoneType'> does not match any type in Union: [<class 'str'>, <enum 'TaskType'>]
  warnings.warn(
[2024-01-01 17:25:27,294] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-01-01 17:25:27,295] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-01-01 17:25:27,295] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-01-01 17:25:27,296] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-01-01 17:25:27,296] [INFO] [comm.py:668:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl
Detected kernel version 4.18.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
wandb: Currently logged in as: mgerstgrasser. Use `wandb login --relogin` to force relogin
wandb: wandb version 0.16.1 is available!  To upgrade, please run:
wandb:  $ pip install wandb --upgrade
wandb: Tracking run with wandb version 0.16.0
wandb: Run data is saved locally in /n/home04/mgerstgrasser/trl/wandb/run-20240101_172539-cds4utft
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run eager-sponge-48
wandb: ⭐ View project at https://wandb.ai/mgerstgrasser/trl
wandb: 🚀 View run at https://wandb.ai/mgerstgrasser/trl/runs/cds4utft
[2024-01-01 17:25:45,831] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed info: version=0.12.6, git-hash=unknown, git-branch=unknown
[2024-01-01 17:25:56,815] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False
[2024-01-01 17:25:56,818] [INFO] [logging.py:96:log_dist] [Rank 0] Using client Optimizer as basic optimizer
[2024-01-01 17:25:56,819] [INFO] [logging.py:96:log_dist] [Rank 0] Removing param_group that has no 'params' in the basic Optimizer
[2024-01-01 17:25:56,821] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Basic Optimizer = Adam
[2024-01-01 17:25:56,821] [INFO] [utils.py:56:is_zero_supported_optimizer] Checking ZeRO support for optimizer=Adam type=<class 'torch.optim.adam.Adam'>
[2024-01-01 17:25:56,821] [INFO] [logging.py:96:log_dist] [Rank 0] Creating torch.bfloat16 ZeRO stage 2 optimizer
[2024-01-01 17:25:56,821] [INFO] [stage_1_and_2.py:148:__init__] Reduce bucket size 500,000,000
[2024-01-01 17:25:56,821] [INFO] [stage_1_and_2.py:149:__init__] Allgather bucket size 500,000,000
[2024-01-01 17:25:56,822] [INFO] [stage_1_and_2.py:150:__init__] CPU Offload: False
[2024-01-01 17:25:56,822] [INFO] [stage_1_and_2.py:151:__init__] Round robin gradient partitioning: False
[2024-01-01 17:25:57,199] [INFO] [utils.py:791:see_memory_usage] Before initializing optimizer states
[2024-01-01 17:25:57,200] [INFO] [utils.py:792:see_memory_usage] MA 0.36 GB         Max_MA 0.42 GB         CA 0.42 GB         Max_CA 0 GB
[2024-01-01 17:25:57,200] [INFO] [utils.py:799:see_memory_usage] CPU Virtual Memory:  used = 16.47 GB, percent = 3.3%
[2024-01-01 17:25:57,498] [INFO] [utils.py:791:see_memory_usage] After initializing optimizer states
[2024-01-01 17:25:57,498] [INFO] [utils.py:792:see_memory_usage] MA 0.59 GB         Max_MA 0.82 GB         CA 0.89 GB         Max_CA 1 GB
[2024-01-01 17:25:57,499] [INFO] [utils.py:799:see_memory_usage] CPU Virtual Memory:  used = 15.16 GB, percent = 3.0%
[2024-01-01 17:25:57,499] [INFO] [stage_1_and_2.py:516:__init__] optimizer state initialized
[2024-01-01 17:25:57,593] [INFO] [utils.py:791:see_memory_usage] After initializing ZeRO optimizer
[2024-01-01 17:25:57,594] [INFO] [utils.py:792:see_memory_usage] MA 0.59 GB         Max_MA 0.59 GB         CA 0.89 GB         Max_CA 1 GB
[2024-01-01 17:25:57,594] [INFO] [utils.py:799:see_memory_usage] CPU Virtual Memory:  used = 15.16 GB, percent = 3.0%
[2024-01-01 17:25:57,595] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Final Optimizer = Adam
[2024-01-01 17:25:57,595] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed using client LR scheduler
[2024-01-01 17:25:57,595] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed LR Scheduler = None
[2024-01-01 17:25:57,595] [INFO] [logging.py:96:log_dist] [Rank 0] step=0, skipped=0, lr=[1.41e-05], mom=[(0.9, 0.999)]
[2024-01-01 17:25:57,595] [INFO] [config.py:984:print] DeepSpeedEngine configuration:
[2024-01-01 17:25:57,595] [INFO] [config.py:988:print]   activation_checkpointing_config  {
    "partition_activations": false,
    "contiguous_memory_optimization": false,
    "cpu_checkpointing": false,
    "number_checkpoints": null,
    "synchronize_checkpoint_boundary": false,
    "profile": false
}
[2024-01-01 17:25:57,595] [INFO] [config.py:988:print]   aio_config ................... {'block_size': 1048576, 'queue_depth': 8, 'thread_count': 1, 'single_submit': False, 'overlap_events': True}
[2024-01-01 17:25:57,596] [INFO] [config.py:988:print]   amp_enabled .................. False
[2024-01-01 17:25:57,596] [INFO] [config.py:988:print]   amp_params ................... False
[2024-01-01 17:25:57,596] [INFO] [config.py:988:print]   autotuning_config ............ {
    "enabled": false,
    "start_step": null,
    "end_step": null,
    "metric_path": null,
    "arg_mappings": null,
    "metric": "throughput",
    "model_info": null,
    "results_dir": "autotuning_results",
    "exps_dir": "autotuning_exps",
    "overwrite": true,
    "fast": true,
    "start_profile_step": 3,
    "end_profile_step": 5,
    "tuner_type": "gridsearch",
    "tuner_early_stopping": 5,
    "tuner_num_trials": 50,
    "model_info_path": null,
    "mp_size": 1,
    "max_train_batch_size": null,
    "min_train_batch_size": 1,
    "max_train_micro_batch_size_per_gpu": 1.024000e+03,
    "min_train_micro_batch_size_per_gpu": 1,
    "num_tuning_micro_batch_sizes": 3
}
[2024-01-01 17:25:57,596] [INFO] [config.py:988:print]   bfloat16_enabled ............. True
[2024-01-01 17:25:57,596] [INFO] [config.py:988:print]   checkpoint_parallel_write_pipeline  False
[2024-01-01 17:25:57,596] [INFO] [config.py:988:print]   checkpoint_tag_validation_enabled  True
[2024-01-01 17:25:57,596] [INFO] [config.py:988:print]   checkpoint_tag_validation_fail  False
[2024-01-01 17:25:57,596] [INFO] [config.py:988:print]   comms_config ................. <deepspeed.comm.config.DeepSpeedCommsConfig object at 0x14e1c1eb8610>
[2024-01-01 17:25:57,596] [INFO] [config.py:988:print]   communication_data_type ...... None
[2024-01-01 17:25:57,596] [INFO] [config.py:988:print]   compression_config ........... {'weight_quantization': {'shared_parameters': {'enabled': False, 'quantizer_kernel': False, 'schedule_offset': 0, 'quantize_groups': 1, 'quantize_verbose': False, 'quantization_type': 'symmetric', 'quantize_weight_in_forward': False, 'rounding': 'nearest', 'fp16_mixed_quantize': False, 'quantize_change_ratio': 0.001}, 'different_groups': {}}, 'activation_quantization': {'shared_parameters': {'enabled': False, 'quantization_type': 'symmetric', 'range_calibration': 'dynamic', 'schedule_offset': 1000}, 'different_groups': {}}, 'sparse_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'row_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'head_pruning': {'shared_parameters': {'enabled': False, 'method': 'topk', 'schedule_offset': 1000}, 'different_groups': {}}, 'channel_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'layer_reduction': {'enabled': False}}
[2024-01-01 17:25:57,596] [INFO] [config.py:988:print]   curriculum_enabled_legacy .... False
[2024-01-01 17:25:57,596] [INFO] [config.py:988:print]   curriculum_params_legacy ..... False
[2024-01-01 17:25:57,596] [INFO] [config.py:988:print]   data_efficiency_config ....... {'enabled': False, 'seed': 1234, 'data_sampling': {'enabled': False, 'num_epochs': 1000, 'num_workers': 0, 'curriculum_learning': {'enabled': False}}, 'data_routing': {'enabled': False, 'random_ltd': {'enabled': False, 'layer_token_lr_schedule': {'enabled': False}}}}
[2024-01-01 17:25:57,596] [INFO] [config.py:988:print]   data_efficiency_enabled ...... False
[2024-01-01 17:25:57,596] [INFO] [config.py:988:print]   dataloader_drop_last ......... False
[2024-01-01 17:25:57,596] [INFO] [config.py:988:print]   disable_allgather ............ False
[2024-01-01 17:25:57,596] [INFO] [config.py:988:print]   dump_state ................... False
[2024-01-01 17:25:57,596] [INFO] [config.py:988:print]   dynamic_loss_scale_args ...... None
[2024-01-01 17:25:57,596] [INFO] [config.py:988:print]   eigenvalue_enabled ........... False
[2024-01-01 17:25:57,596] [INFO] [config.py:988:print]   eigenvalue_gas_boundary_resolution  1
[2024-01-01 17:25:57,597] [INFO] [config.py:988:print]   eigenvalue_layer_name ........ bert.encoder.layer
[2024-01-01 17:25:57,597] [INFO] [config.py:988:print]   eigenvalue_layer_num ......... 0
[2024-01-01 17:25:57,597] [INFO] [config.py:988:print]   eigenvalue_max_iter .......... 100
[2024-01-01 17:25:57,597] [INFO] [config.py:988:print]   eigenvalue_stability ......... 1e-06
[2024-01-01 17:25:57,597] [INFO] [config.py:988:print]   eigenvalue_tol ............... 0.01
[2024-01-01 17:25:57,597] [INFO] [config.py:988:print]   eigenvalue_verbose ........... False
[2024-01-01 17:25:57,597] [INFO] [config.py:988:print]   elasticity_enabled ........... False
[2024-01-01 17:25:57,597] [INFO] [config.py:988:print]   flops_profiler_config ........ {
    "enabled": false,
    "recompute_fwd_factor": 0.0,
    "profile_step": 1,
    "module_depth": -1,
    "top_modules": 1,
    "detailed": true,
    "output_file": null
}
[2024-01-01 17:25:57,597] [INFO] [config.py:988:print]   fp16_auto_cast ............... None
[2024-01-01 17:25:57,597] [INFO] [config.py:988:print]   fp16_enabled ................. False
[2024-01-01 17:25:57,597] [INFO] [config.py:988:print]   fp16_master_weights_and_gradients  False
[2024-01-01 17:25:57,597] [INFO] [config.py:988:print]   global_rank .................. 0
[2024-01-01 17:25:57,597] [INFO] [config.py:988:print]   grad_accum_dtype ............. None
[2024-01-01 17:25:57,597] [INFO] [config.py:988:print]   gradient_accumulation_steps .. 1
[2024-01-01 17:25:57,597] [INFO] [config.py:988:print]   gradient_clipping ............ 0.0
[2024-01-01 17:25:57,597] [INFO] [config.py:988:print]   gradient_predivide_factor .... 1.0
[2024-01-01 17:25:57,597] [INFO] [config.py:988:print]   graph_harvesting ............. False
[2024-01-01 17:25:57,597] [INFO] [config.py:988:print]   hybrid_engine ................ enabled=False max_out_tokens=512 inference_tp_size=1 release_inference_cache=False pin_parameters=True tp_gather_partition_size=8
[2024-01-01 17:25:57,597] [INFO] [config.py:988:print]   initial_dynamic_scale ........ 1
[2024-01-01 17:25:57,597] [INFO] [config.py:988:print]   load_universal_checkpoint .... False
[2024-01-01 17:25:57,597] [INFO] [config.py:988:print]   loss_scale ................... 1.0
[2024-01-01 17:25:57,597] [INFO] [config.py:988:print]   memory_breakdown ............. False
[2024-01-01 17:25:57,597] [INFO] [config.py:988:print]   mics_hierarchial_params_gather  False
[2024-01-01 17:25:57,597] [INFO] [config.py:988:print]   mics_shard_size .............. -1
[2024-01-01 17:25:57,597] [INFO] [config.py:988:print]   monitor_config ............... tensorboard=TensorBoardConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') wandb=WandbConfig(enabled=False, group=None, team=None, project='deepspeed') csv_monitor=CSVConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') enabled=False
[2024-01-01 17:25:57,598] [INFO] [config.py:988:print]   nebula_config ................ {
    "enabled": false,
    "persistent_storage_path": null,
    "persistent_time_interval": 100,
    "num_of_version_in_retention": 2,
    "enable_nebula_load": true,
    "load_path": null
}
[2024-01-01 17:25:57,598] [INFO] [config.py:988:print]   optimizer_legacy_fusion ...... False
[2024-01-01 17:25:57,598] [INFO] [config.py:988:print]   optimizer_name ............... None
[2024-01-01 17:25:57,598] [INFO] [config.py:988:print]   optimizer_params ............. None
[2024-01-01 17:25:57,598] [INFO] [config.py:988:print]   pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0, 'pipe_partitioned': True, 'grad_partitioned': True}
[2024-01-01 17:25:57,598] [INFO] [config.py:988:print]   pld_enabled .................. False
[2024-01-01 17:25:57,598] [INFO] [config.py:988:print]   pld_params ................... False
[2024-01-01 17:25:57,598] [INFO] [config.py:988:print]   prescale_gradients ........... False
[2024-01-01 17:25:57,598] [INFO] [config.py:988:print]   scheduler_name ............... None
[2024-01-01 17:25:57,598] [INFO] [config.py:988:print]   scheduler_params ............. None
[2024-01-01 17:25:57,598] [INFO] [config.py:988:print]   seq_parallel_communication_data_type  torch.float32
[2024-01-01 17:25:57,598] [INFO] [config.py:988:print]   sparse_attention ............. None
[2024-01-01 17:25:57,598] [INFO] [config.py:988:print]   sparse_gradients_enabled ..... False
[2024-01-01 17:25:57,598] [INFO] [config.py:988:print]   steps_per_print .............. inf
[2024-01-01 17:25:57,598] [INFO] [config.py:988:print]   train_batch_size ............. 512
[2024-01-01 17:25:57,598] [INFO] [config.py:988:print]   train_micro_batch_size_per_gpu  128
[2024-01-01 17:25:57,598] [INFO] [config.py:988:print]   use_data_before_expert_parallel_  False
[2024-01-01 17:25:57,598] [INFO] [config.py:988:print]   use_node_local_storage ....... False
[2024-01-01 17:25:57,598] [INFO] [config.py:988:print]   wall_clock_breakdown ......... False
[2024-01-01 17:25:57,598] [INFO] [config.py:988:print]   weight_quantization_config ... None
[2024-01-01 17:25:57,598] [INFO] [config.py:988:print]   world_size ................... 4
[2024-01-01 17:25:57,598] [INFO] [config.py:988:print]   zero_allow_untested_optimizer  True
[2024-01-01 17:25:57,598] [INFO] [config.py:988:print]   zero_config .................. stage=2 contiguous_gradients=True reduce_scatter=True reduce_bucket_size=500,000,000 use_multi_rank_bucket_allreduce=True allgather_partitions=True allgather_bucket_size=500,000,000 overlap_comm=False load_from_fp32_weights=True elastic_checkpoint=False offload_param=DeepSpeedZeroOffloadParamConfig(device='none', nvme_path=None, buffer_count=5, buffer_size=100,000,000, max_in_cpu=1,000,000,000, pin_memory=False) offload_optimizer=DeepSpeedZeroOffloadOptimizerConfig(device='none', nvme_path=None, buffer_count=4, pin_memory=False, pipeline=False, pipeline_read=False, pipeline_write=False, fast_init=False, ratio=1.0) sub_group_size=1,000,000,000 cpu_offload_param=None cpu_offload_use_pin_memory=None cpu_offload=None prefetch_bucket_size=50,000,000 param_persistence_threshold=100,000 model_persistence_threshold=sys.maxsize max_live_parameters=1,000,000,000 max_reuse_distance=1,000,000,000 gather_16bit_weights_on_model_save=False stage3_gather_fp16_weights_on_model_save=False ignore_unused_parameters=True legacy_stage1=False round_robin_gradients=False zero_hpz_partition_size=1 zero_quantized_weights=False zero_quantized_nontrainable_weights=False zero_quantized_gradients=False mics_shard_size=-1 mics_hierarchical_params_gather=False memory_efficient_linear=True pipeline_loading_checkpoint=False override_module_apply=True
[2024-01-01 17:25:57,598] [INFO] [config.py:988:print]   zero_enabled ................. True
[2024-01-01 17:25:57,598] [INFO] [config.py:988:print]   zero_force_ds_cpu_optimizer .. True
[2024-01-01 17:25:57,599] [INFO] [config.py:988:print]   zero_optimization_stage ...... 2
[2024-01-01 17:25:57,599] [INFO] [config.py:974:print_user_config]   json = {
    "train_batch_size": 512,
    "train_micro_batch_size_per_gpu": 128,
    "gradient_accumulation_steps": 1,
    "zero_optimization": {
        "stage": 2,
        "offload_optimizer": {
            "device": "none",
            "nvme_path": null
        },
        "offload_param": {
            "device": "none",
            "nvme_path": null
        },
        "stage3_gather_16bit_weights_on_model_save": false
    },
    "steps_per_print": inf,
    "bf16": {
        "enabled": true
    },
    "fp16": {
        "enabled": false
    },
    "zero_allow_untested_optimizer": true
}
[2024-01-01 17:25:57,599] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed info: version=0.12.6, git-hash=unknown, git-branch=unknown
[2024-01-01 17:25:57,733] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False
[2024-01-01 17:25:57,733] [INFO] [logging.py:96:log_dist] [Rank 0] Creating BF16 optimizer
[2024-01-01 17:25:57,821] [INFO] [utils.py:791:see_memory_usage] begin bf16_optimizer
[2024-01-01 17:25:57,822] [INFO] [utils.py:792:see_memory_usage] MA 0.84 GB         Max_MA 0.84 GB         CA 0.9 GB         Max_CA 1 GB
[2024-01-01 17:25:57,822] [INFO] [utils.py:799:see_memory_usage] CPU Virtual Memory:  used = 14.73 GB, percent = 2.9%
[2024-01-01 17:25:57,899] [INFO] [utils.py:791:see_memory_usage] end bf16_optimizer
[2024-01-01 17:25:57,899] [INFO] [utils.py:792:see_memory_usage] MA 0.84 GB         Max_MA 0.84 GB         CA 0.9 GB         Max_CA 1 GB
[2024-01-01 17:25:57,900] [INFO] [utils.py:799:see_memory_usage] CPU Virtual Memory:  used = 14.73 GB, percent = 2.9%
[2024-01-01 17:25:57,900] [INFO] [config.py:984:print] DeepSpeedEngine configuration:
[2024-01-01 17:25:57,900] [INFO] [config.py:988:print]   activation_checkpointing_config  {
    "partition_activations": false,
    "contiguous_memory_optimization": false,
    "cpu_checkpointing": false,
    "number_checkpoints": null,
    "synchronize_checkpoint_boundary": false,
    "profile": false
}
[2024-01-01 17:25:57,900] [INFO] [config.py:988:print]   aio_config ................... {'block_size': 1048576, 'queue_depth': 8, 'thread_count': 1, 'single_submit': False, 'overlap_events': True}
[2024-01-01 17:25:57,900] [INFO] [config.py:988:print]   amp_enabled .................. False
[2024-01-01 17:25:57,900] [INFO] [config.py:988:print]   amp_params ................... False
[2024-01-01 17:25:57,900] [INFO] [config.py:988:print]   autotuning_config ............ {
    "enabled": false,
    "start_step": null,
    "end_step": null,
    "metric_path": null,
    "arg_mappings": null,
    "metric": "throughput",
    "model_info": null,
    "results_dir": "autotuning_results",
    "exps_dir": "autotuning_exps",
    "overwrite": true,
    "fast": true,
    "start_profile_step": 3,
    "end_profile_step": 5,
    "tuner_type": "gridsearch",
    "tuner_early_stopping": 5,
    "tuner_num_trials": 50,
    "model_info_path": null,
    "mp_size": 1,
    "max_train_batch_size": null,
    "min_train_batch_size": 1,
    "max_train_micro_batch_size_per_gpu": 1.024000e+03,
    "min_train_micro_batch_size_per_gpu": 1,
    "num_tuning_micro_batch_sizes": 3
}
[2024-01-01 17:25:57,900] [INFO] [config.py:988:print]   bfloat16_enabled ............. True
[2024-01-01 17:25:57,901] [INFO] [config.py:988:print]   checkpoint_parallel_write_pipeline  False
[2024-01-01 17:25:57,901] [INFO] [config.py:988:print]   checkpoint_tag_validation_enabled  True
[2024-01-01 17:25:57,901] [INFO] [config.py:988:print]   checkpoint_tag_validation_fail  False
[2024-01-01 17:25:57,901] [INFO] [config.py:988:print]   comms_config ................. <deepspeed.comm.config.DeepSpeedCommsConfig object at 0x14e1c1f5e790>
[2024-01-01 17:25:57,901] [INFO] [config.py:988:print]   communication_data_type ...... None
[2024-01-01 17:25:57,901] [INFO] [config.py:988:print]   compression_config ........... {'weight_quantization': {'shared_parameters': {'enabled': False, 'quantizer_kernel': False, 'schedule_offset': 0, 'quantize_groups': 1, 'quantize_verbose': False, 'quantization_type': 'symmetric', 'quantize_weight_in_forward': False, 'rounding': 'nearest', 'fp16_mixed_quantize': False, 'quantize_change_ratio': 0.001}, 'different_groups': {}}, 'activation_quantization': {'shared_parameters': {'enabled': False, 'quantization_type': 'symmetric', 'range_calibration': 'dynamic', 'schedule_offset': 1000}, 'different_groups': {}}, 'sparse_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'row_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'head_pruning': {'shared_parameters': {'enabled': False, 'method': 'topk', 'schedule_offset': 1000}, 'different_groups': {}}, 'channel_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'layer_reduction': {'enabled': False}}
[2024-01-01 17:25:57,901] [INFO] [config.py:988:print]   curriculum_enabled_legacy .... False
[2024-01-01 17:25:57,901] [INFO] [config.py:988:print]   curriculum_params_legacy ..... False
[2024-01-01 17:25:57,901] [INFO] [config.py:988:print]   data_efficiency_config ....... {'enabled': False, 'seed': 1234, 'data_sampling': {'enabled': False, 'num_epochs': 1000, 'num_workers': 0, 'curriculum_learning': {'enabled': False}}, 'data_routing': {'enabled': False, 'random_ltd': {'enabled': False, 'layer_token_lr_schedule': {'enabled': False}}}}
[2024-01-01 17:25:57,901] [INFO] [config.py:988:print]   data_efficiency_enabled ...... False
[2024-01-01 17:25:57,901] [INFO] [config.py:988:print]   dataloader_drop_last ......... False
[2024-01-01 17:25:57,901] [INFO] [config.py:988:print]   disable_allgather ............ False
[2024-01-01 17:25:57,901] [INFO] [config.py:988:print]   dump_state ................... False
[2024-01-01 17:25:57,901] [INFO] [config.py:988:print]   dynamic_loss_scale_args ...... None
[2024-01-01 17:25:57,901] [INFO] [config.py:988:print]   eigenvalue_enabled ........... False
[2024-01-01 17:25:57,901] [INFO] [config.py:988:print]   eigenvalue_gas_boundary_resolution  1
[2024-01-01 17:25:57,901] [INFO] [config.py:988:print]   eigenvalue_layer_name ........ bert.encoder.layer
[2024-01-01 17:25:57,901] [INFO] [config.py:988:print]   eigenvalue_layer_num ......... 0
[2024-01-01 17:25:57,901] [INFO] [config.py:988:print]   eigenvalue_max_iter .......... 100
[2024-01-01 17:25:57,901] [INFO] [config.py:988:print]   eigenvalue_stability ......... 1e-06
[2024-01-01 17:25:57,901] [INFO] [config.py:988:print]   eigenvalue_tol ............... 0.01
[2024-01-01 17:25:57,901] [INFO] [config.py:988:print]   eigenvalue_verbose ........... False
[2024-01-01 17:25:57,901] [INFO] [config.py:988:print]   elasticity_enabled ........... False
[2024-01-01 17:25:57,901] [INFO] [config.py:988:print]   flops_profiler_config ........ {
    "enabled": false,
    "recompute_fwd_factor": 0.0,
    "profile_step": 1,
    "module_depth": -1,
    "top_modules": 1,
    "detailed": true,
    "output_file": null
}
[2024-01-01 17:25:57,901] [INFO] [config.py:988:print]   fp16_auto_cast ............... None
[2024-01-01 17:25:57,901] [INFO] [config.py:988:print]   fp16_enabled ................. False
[2024-01-01 17:25:57,901] [INFO] [config.py:988:print]   fp16_master_weights_and_gradients  False
[2024-01-01 17:25:57,902] [INFO] [config.py:988:print]   global_rank .................. 0
[2024-01-01 17:25:57,902] [INFO] [config.py:988:print]   grad_accum_dtype ............. None
[2024-01-01 17:25:57,902] [INFO] [config.py:988:print]   gradient_accumulation_steps .. 1
[2024-01-01 17:25:57,902] [INFO] [config.py:988:print]   gradient_clipping ............ 0.0
[2024-01-01 17:25:57,902] [INFO] [config.py:988:print]   gradient_predivide_factor .... 1.0
[2024-01-01 17:25:57,902] [INFO] [config.py:988:print]   graph_harvesting ............. False
[2024-01-01 17:25:57,902] [INFO] [config.py:988:print]   hybrid_engine ................ enabled=False max_out_tokens=512 inference_tp_size=1 release_inference_cache=False pin_parameters=True tp_gather_partition_size=8
[2024-01-01 17:25:57,902] [INFO] [config.py:988:print]   initial_dynamic_scale ........ 1
[2024-01-01 17:25:57,902] [INFO] [config.py:988:print]   load_universal_checkpoint .... False
[2024-01-01 17:25:57,902] [INFO] [config.py:988:print]   loss_scale ................... 1.0
[2024-01-01 17:25:57,902] [INFO] [config.py:988:print]   memory_breakdown ............. False
[2024-01-01 17:25:57,902] [INFO] [config.py:988:print]   mics_hierarchial_params_gather  False
[2024-01-01 17:25:57,902] [INFO] [config.py:988:print]   mics_shard_size .............. -1
[2024-01-01 17:25:57,902] [INFO] [config.py:988:print]   monitor_config ............... tensorboard=TensorBoardConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') wandb=WandbConfig(enabled=False, group=None, team=None, project='deepspeed') csv_monitor=CSVConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') enabled=False
[2024-01-01 17:25:57,902] [INFO] [config.py:988:print]   nebula_config ................ {
    "enabled": false,
    "persistent_storage_path": null,
    "persistent_time_interval": 100,
    "num_of_version_in_retention": 2,
    "enable_nebula_load": true,
    "load_path": null
}
[2024-01-01 17:25:57,902] [INFO] [config.py:988:print]   optimizer_legacy_fusion ...... False
[2024-01-01 17:25:57,902] [INFO] [config.py:988:print]   optimizer_name ............... None
[2024-01-01 17:25:57,902] [INFO] [config.py:988:print]   optimizer_params ............. None
[2024-01-01 17:25:57,902] [INFO] [config.py:988:print]   pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0, 'pipe_partitioned': True, 'grad_partitioned': True}
[2024-01-01 17:25:57,902] [INFO] [config.py:988:print]   pld_enabled .................. False
[2024-01-01 17:25:57,902] [INFO] [config.py:988:print]   pld_params ................... False
[2024-01-01 17:25:57,902] [INFO] [config.py:988:print]   prescale_gradients ........... False
[2024-01-01 17:25:57,902] [INFO] [config.py:988:print]   scheduler_name ............... None
[2024-01-01 17:25:57,902] [INFO] [config.py:988:print]   scheduler_params ............. None
[2024-01-01 17:25:57,902] [INFO] [config.py:988:print]   seq_parallel_communication_data_type  torch.float32
[2024-01-01 17:25:57,903] [INFO] [config.py:988:print]   sparse_attention ............. None
[2024-01-01 17:25:57,903] [INFO] [config.py:988:print]   sparse_gradients_enabled ..... False
[2024-01-01 17:25:57,903] [INFO] [config.py:988:print]   steps_per_print .............. inf
[2024-01-01 17:25:57,903] [INFO] [config.py:988:print]   train_batch_size ............. 512
[2024-01-01 17:25:57,903] [INFO] [config.py:988:print]   train_micro_batch_size_per_gpu  128
[2024-01-01 17:25:57,903] [INFO] [config.py:988:print]   use_data_before_expert_parallel_  False
[2024-01-01 17:25:57,903] [INFO] [config.py:988:print]   use_node_local_storage ....... False
[2024-01-01 17:25:57,903] [INFO] [config.py:988:print]   wall_clock_breakdown ......... False
[2024-01-01 17:25:57,903] [INFO] [config.py:988:print]   weight_quantization_config ... None
[2024-01-01 17:25:57,903] [INFO] [config.py:988:print]   world_size ................... 4
[2024-01-01 17:25:57,903] [INFO] [config.py:988:print]   zero_allow_untested_optimizer  True
[2024-01-01 17:25:57,903] [INFO] [config.py:988:print]   zero_config .................. stage=0 contiguous_gradients=True reduce_scatter=True reduce_bucket_size=500,000,000 use_multi_rank_bucket_allreduce=True allgather_partitions=True allgather_bucket_size=500,000,000 overlap_comm=False load_from_fp32_weights=True elastic_checkpoint=False offload_param=DeepSpeedZeroOffloadParamConfig(device='none', nvme_path=None, buffer_count=5, buffer_size=100,000,000, max_in_cpu=1,000,000,000, pin_memory=False) offload_optimizer=DeepSpeedZeroOffloadOptimizerConfig(device='none', nvme_path=None, buffer_count=4, pin_memory=False, pipeline=False, pipeline_read=False, pipeline_write=False, fast_init=False, ratio=1.0) sub_group_size=1,000,000,000 cpu_offload_param=None cpu_offload_use_pin_memory=None cpu_offload=None prefetch_bucket_size=50,000,000 param_persistence_threshold=100,000 model_persistence_threshold=sys.maxsize max_live_parameters=1,000,000,000 max_reuse_distance=1,000,000,000 gather_16bit_weights_on_model_save=False stage3_gather_fp16_weights_on_model_save=False ignore_unused_parameters=True legacy_stage1=False round_robin_gradients=False zero_hpz_partition_size=1 zero_quantized_weights=False zero_quantized_nontrainable_weights=False zero_quantized_gradients=False mics_shard_size=-1 mics_hierarchical_params_gather=False memory_efficient_linear=True pipeline_loading_checkpoint=False override_module_apply=True
[2024-01-01 17:25:57,903] [INFO] [config.py:988:print]   zero_enabled ................. False
[2024-01-01 17:25:57,903] [INFO] [config.py:988:print]   zero_force_ds_cpu_optimizer .. True
[2024-01-01 17:25:57,903] [INFO] [config.py:988:print]   zero_optimization_stage ...... 0
[2024-01-01 17:25:57,903] [INFO] [config.py:974:print_user_config]   json = {
    "train_batch_size": 512,
    "train_micro_batch_size_per_gpu": 128,
    "gradient_accumulation_steps": 1,
    "zero_optimization": {
        "stage": 0,
        "offload_optimizer": {
            "device": "none",
            "nvme_path": null
        },
        "offload_param": {
            "device": "none",
            "nvme_path": null
        },
        "stage3_gather_16bit_weights_on_model_save": false
    },
    "steps_per_print": inf,
    "bf16": {
        "enabled": true
    },
    "fp16": {
        "enabled": false
    },
    "zero_allow_untested_optimizer": true
}
0it [00:00, ?it/s]You're using a GPT2TokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
You're using a GPT2TokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
You're using a GPT2TokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
You're using a GPT2TokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
/n/home04/mgerstgrasser/.conda/envs/tortoise/lib/python3.11/site-packages/transformers/pipelines/text_classification.py:105: UserWarning: `return_all_scores` is now deprecated,  if want a similar functionality use `top_k=None` instead of `return_all_scores=True` or `top_k=1` instead of `return_all_scores=False`.
  warnings.warn(
/n/home04/mgerstgrasser/.conda/envs/tortoise/lib/python3.11/site-packages/transformers/pipelines/text_classification.py:105: UserWarning: `return_all_scores` is now deprecated,  if want a similar functionality use `top_k=None` instead of `return_all_scores=True` or `top_k=1` instead of `return_all_scores=False`.
  warnings.warn(
/n/home04/mgerstgrasser/.conda/envs/tortoise/lib/python3.11/site-packages/transformers/pipelines/text_classification.py:105: UserWarning: `return_all_scores` is now deprecated,  if want a similar functionality use `top_k=None` instead of `return_all_scores=True` or `top_k=1` instead of `return_all_scores=False`.
  warnings.warn(
/n/home04/mgerstgrasser/.conda/envs/tortoise/lib/python3.11/site-packages/transformers/pipelines/text_classification.py:105: UserWarning: `return_all_scores` is now deprecated,  if want a similar functionality use `top_k=None` instead of `return_all_scores=True` or `top_k=1` instead of `return_all_scores=False`.
  warnings.warn(
/n/home04/mgerstgrasser/.conda/envs/tortoise/lib/python3.11/site-packages/deepspeed/runtime/zero/stage_1_and_2.py:1652: UserWarning: The torch.cuda.*DtypeTensor constructors are no longer recommended. It's best to use methods such as torch.tensor(data, dtype=*, device='cuda') to create tensors. (Triggered internally at ../torch/csrc/tensor/python_tensor.cpp:83.)
  total_norm_cuda = get_accelerator().FloatTensor([float(total_norm)])
/n/home04/mgerstgrasser/.conda/envs/tortoise/lib/python3.11/site-packages/deepspeed/runtime/zero/stage_1_and_2.py:1652: UserWarning: The torch.cuda.*DtypeTensor constructors are no longer recommended. It's best to use methods such as torch.tensor(data, dtype=*, device='cuda') to create tensors. (Triggered internally at ../torch/csrc/tensor/python_tensor.cpp:83.)
  total_norm_cuda = get_accelerator().FloatTensor([float(total_norm)])
/n/home04/mgerstgrasser/.conda/envs/tortoise/lib/python3.11/site-packages/deepspeed/runtime/zero/stage_1_and_2.py:1652: UserWarning: The torch.cuda.*DtypeTensor constructors are no longer recommended. It's best to use methods such as torch.tensor(data, dtype=*, device='cuda') to create tensors. (Triggered internally at ../torch/csrc/tensor/python_tensor.cpp:83.)
  total_norm_cuda = get_accelerator().FloatTensor([float(total_norm)])
/n/home04/mgerstgrasser/.conda/envs/tortoise/lib/python3.11/site-packages/deepspeed/runtime/zero/stage_1_and_2.py:1652: UserWarning: The torch.cuda.*DtypeTensor constructors are no longer recommended. It's best to use methods such as torch.tensor(data, dtype=*, device='cuda') to create tensors. (Triggered internally at ../torch/csrc/tensor/python_tensor.cpp:83.)
  total_norm_cuda = get_accelerator().FloatTensor([float(total_norm)])
1it [00:29, 29.37s/it][E ProcessGroupNCCL.cpp:475] [Rank 3] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=31, OpType=_ALLGATHER_BASE, NumelIn=2, NumelOut=8, Timeout(ms)=1800000) ran for 1800035 milliseconds before timing out.
1it [30:45, 1845.05s/it]
Traceback (most recent call last):
  File "/n/home04/mgerstgrasser/trl/examples/scripts/ppo.py", line 211, in <module>
    stats = ppo_trainer.step(query_tensors, response_tensors, rewards)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/n/home04/mgerstgrasser/.conda/envs/tortoise/lib/python3.11/contextlib.py", line 81, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/n/home04/mgerstgrasser/trl/trl/trainer/ppo_trainer.py", line 683, in step
    model_inputs["input_ids"] = self.accelerator.pad_across_processes(
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/n/home04/mgerstgrasser/.conda/envs/tortoise/lib/python3.11/site-packages/accelerate/accelerator.py", line 2317, in pad_across_processes
    return pad_across_processes(tensor, dim=dim, pad_index=pad_index, pad_first=pad_first)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/n/home04/mgerstgrasser/.conda/envs/tortoise/lib/python3.11/site-packages/accelerate/utils/operations.py", line 368, in wrapper
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/n/home04/mgerstgrasser/.conda/envs/tortoise/lib/python3.11/site-packages/accelerate/utils/operations.py", line 567, in pad_across_processes
    return recursively_apply(
           ^^^^^^^^^^^^^^^^^^
  File "/n/home04/mgerstgrasser/.conda/envs/tortoise/lib/python3.11/site-packages/accelerate/utils/operations.py", line 128, in recursively_apply
    return func(data, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/n/home04/mgerstgrasser/.conda/envs/tortoise/lib/python3.11/site-packages/accelerate/utils/operations.py", line 557, in _pad_across_processes
    new_tensor = tensor.new_zeros(tuple(new_size)) + pad_index
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Storage size calculation overflowed with sizes=[128,<PHONE_NUMBER>785319098]
[E ProcessGroupNCCL.cpp:489] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[E ProcessGroupNCCL.cpp:495] To avoid data inconsistency, we are taking the entire process down.
[E ProcessGroupNCCL.cpp:916] [Rank 3] NCCL watchdog thread terminated with exception: [Rank 3] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=31, OpType=_ALLGATHER_BASE, NumelIn=2, NumelOut=8, Timeout(ms)=1800000) ran for 1800035 milliseconds before timing out.
terminate called after throwing an instance of 'std::runtime_error'
  what():  [Rank 3] NCCL watchdog thread terminated with exception: [Rank 3] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=31, OpType=_ALLGATHER_BASE, NumelIn=2, NumelOut=8, Timeout(ms)=1800000) ran for 1800035 milliseconds before timing out.
[E ProcessGroupNCCL.cpp:475] [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=31, OpType=_ALLGATHER_BASE, NumelIn=2, NumelOut=8, Timeout(ms)=1800000) ran for 1800334 milliseconds before timing out.
[E ProcessGroupNCCL.cpp:475] [Rank 2] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=31, OpType=_ALLGATHER_BASE, NumelIn=2, NumelOut=8, Timeout(ms)=1800000) ran for 1800693 milliseconds before timing out.
1it [30:45, 1845.98s/it]
Traceback (most recent call last):
  File "/n/home04/mgerstgrasser/trl/examples/scripts/ppo.py", line 211, in <module>
1it [30:45, 1845.98s/it]
Traceback (most recent call last):
  File "/n/home04/mgerstgrasser/trl/examples/scripts/ppo.py", line 211, in <module>
    stats = ppo_trainer.step(query_tensors, response_tensors, rewards)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/n/home04/mgerstgrasser/.conda/envs/tortoise/lib/python3.11/contextlib.py", line 81, in inner
    stats = ppo_trainer.step(query_tensors, response_tensors, rewards)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/n/home04/mgerstgrasser/.conda/envs/tortoise/lib/python3.11/contextlib.py", line 81, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/n/home04/mgerstgrasser/trl/trl/trainer/ppo_trainer.py", line 683, in step
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/n/home04/mgerstgrasser/trl/trl/trainer/ppo_trainer.py", line 683, in step
    model_inputs["input_ids"] = self.accelerator.pad_across_processes(
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/n/home04/mgerstgrasser/.conda/envs/tortoise/lib/python3.11/site-packages/accelerate/accelerator.py", line 2317, in pad_across_processes
    model_inputs["input_ids"] = self.accelerator.pad_across_processes(
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/n/home04/mgerstgrasser/.conda/envs/tortoise/lib/python3.11/site-packages/accelerate/accelerator.py", line 2317, in pad_across_processes
    return pad_across_processes(tensor, dim=dim, pad_index=pad_index, pad_first=pad_first)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/n/home04/mgerstgrasser/.conda/envs/tortoise/lib/python3.11/site-packages/accelerate/utils/operations.py", line 368, in wrapper
    return pad_across_processes(tensor, dim=dim, pad_index=pad_index, pad_first=pad_first)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/n/home04/mgerstgrasser/.conda/envs/tortoise/lib/python3.11/site-packages/accelerate/utils/operations.py", line 368, in wrapper
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/n/home04/mgerstgrasser/.conda/envs/tortoise/lib/python3.11/site-packages/accelerate/utils/operations.py", line 567, in pad_across_processes
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/n/home04/mgerstgrasser/.conda/envs/tortoise/lib/python3.11/site-packages/accelerate/utils/operations.py", line 567, in pad_across_processes
    return recursively_apply(
           ^^^^^^^^^^^^^^^^^^
  File "/n/home04/mgerstgrasser/.conda/envs/tortoise/lib/python3.11/site-packages/accelerate/utils/operations.py", line 128, in recursively_apply
[E ProcessGroupNCCL.cpp:489] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[E ProcessGroupNCCL.cpp:495] To avoid data inconsistency, we are taking the entire process down.
[E ProcessGroupNCCL.cpp:916] [Rank 2] NCCL watchdog thread terminated with exception: [Rank 2] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=31, OpType=_ALLGATHER_BASE, NumelIn=2, NumelOut=8, Timeout(ms)=1800000) ran for 1800693 milliseconds before timing out.
terminate called after throwing an instance of 'std::runtime_error'
  what():  [Rank 2] NCCL watchdog thread terminated with exception: [Rank 2] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=31, OpType=_ALLGATHER_BASE, NumelIn=2, NumelOut=8, Timeout(ms)=1800000) ran for 1800693 milliseconds before timing out.
    return recursively_apply(
           ^^^^^^^^^^^^^^^^^^
  File "/n/home04/mgerstgrasser/.conda/envs/tortoise/lib/python3.11/site-packages/accelerate/utils/operations.py", line 128, in recursively_apply
[E ProcessGroupNCCL.cpp:489] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[E ProcessGroupNCCL.cpp:495] To avoid data inconsistency, we are taking the entire process down.
[E ProcessGroupNCCL.cpp:916] [Rank 1] NCCL watchdog thread terminated with exception: [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=31, OpType=_ALLGATHER_BASE, NumelIn=2, NumelOut=8, Timeout(ms)=1800000) ran for 1800334 milliseconds before timing out.
terminate called after throwing an instance of 'std::runtime_error'
  what():  [Rank 1] NCCL watchdog thread terminated with exception: [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=31, OpType=_ALLGATHER_BASE, NumelIn=2, NumelOut=8, Timeout(ms)=1800000) ran for 1800334 milliseconds before timing out.
[2024-01-01 17:56:49,647] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 1375726 closing signal SIGTERM
[2024-01-01 17:56:50,311] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: -6) local_rank: 1 (pid: 1375727) of binary: /n/home04/mgerstgrasser/.conda/envs/tortoise/bin/python3.11
Traceback (most recent call last):
  File "/n/home04/mgerstgrasser/.conda/envs/tortoise/bin/accelerate", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/n/home04/mgerstgrasser/.conda/envs/tortoise/lib/python3.11/site-packages/accelerate/commands/accelerate_cli.py", line 47, in main
    args.func(args)
  File "/n/home04/mgerstgrasser/.conda/envs/tortoise/lib/python3.11/site-packages/accelerate/commands/launch.py", line 979, in launch_command
    deepspeed_launcher(args)
  File "/n/home04/mgerstgrasser/.conda/envs/tortoise/lib/python3.11/site-packages/accelerate/commands/launch.py", line 695, in deepspeed_launcher
    distrib_run.run(args)
  File "/n/home04/mgerstgrasser/.conda/envs/tortoise/lib/python3.11/site-packages/torch/distributed/run.py", line 797, in run
    elastic_launch(
  File "/n/home04/mgerstgrasser/.conda/envs/tortoise/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/n/home04/mgerstgrasser/.conda/envs/tortoise/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
========================================================
examples/scripts/ppo.py FAILED
--------------------------------------------------------
Failures:
[1]:
  time      : 2024-01-01_17:56:49
  host      :
  rank      : 2 (local_rank: 2)
  exitcode  : -6 (pid: 1375728)
  error_file: <N/A>
  traceback : Signal 6 (SIGABRT) received by PID 1375728
[2]:
  time      : 2024-01-01_17:56:49
  host      :
  rank      : 3 (local_rank: 3)
  exitcode  : -6 (pid: 1375729)
  error_file: <N/A>
  traceback : Signal 6 (SIGABRT) received by PID 1375729
--------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2024-01-01_17:56:49
  host      :
  rank      : 1 (local_rank: 1)
  exitcode  : -6 (pid: 1375727)
  error_file: <N/A>
  traceback : Signal 6 (SIGABRT) received by PID 1375727
========================================================

pip freeze:
accelerate==0.24.1
aiohttp==3.9.1
aiosignal==1.3.1
alpaca-eval==0.3.6
alpaca-farm==0.1.9
anyio==3.7.1
appdirs==1.4.4
asttokens @ file:///home/conda/feedstock_root/build_artifacts/asttokens_1698341106958/work
attrs==23.1.0
bitsandbytes==0.41.2.post2
certifi==2023.11.17
charset-normalizer==3.3.2
click==8.1.7
comm @ file:///home/conda/feedstock_root/build_artifacts/comm_1691044910542/work
datasets==2.15.0
debugpy @ file:///home/conda/feedstock_root/build_artifacts/debugpy_1695534290440/work
decorator @ file:///home/conda/feedstock_root/build_artifacts/decorator_1641555617451/work
deepspeed==0.12.6
dill==0.3.7
docker-pycreds==0.4.0
docstring-parser==0.15
einops==0.7.0
exceptiongroup @ file:///home/conda/feedstock_root/build_artifacts/exceptiongroup_1700579780973/work
executing @ file:///home/conda/feedstock_root/build_artifacts/executing_1698579936712/work
fastapi==0.104.1
filelock==3.13.1
fire==0.5.0
flash-attn==2.3.3
frozenlist==1.4.0
fschat==0.2.33
fsspec==2023.10.0
gitdb==4.0.11
GitPython==3.1.40
h11==0.14.0
hjson==3.1.0
httpcore==1.0.2
httptools==0.6.1
httpx==0.25.2
huggingface-hub==0.19.4
idna==3.6
importlib-metadata @ file:///home/conda/feedstock_root/build_artifacts/importlib-metadata_1688754491823/work
ipykernel @ file:///home/conda/feedstock_root/build_artifacts/ipykernel_1698244021190/work
ipython @ file:///home/conda/feedstock_root/build_artifacts/ipython_1701092366260/work
jedi @ file:///home/conda/feedstock_root/build_artifacts/jedi_1696326070614/work
Jinja2==3.1.2
joblib==1.3.2
jsonschema==4.20.0
jsonschema-specifications==2023.11.1
jupyter_client @ file:///home/conda/feedstock_root/build_artifacts/jupyter_client_1699283905679/work
jupyter_core @ file:///home/conda/feedstock_root/build_artifacts/jupyter_core_1698673643471/work
Markdown==3.5.1
markdown-it-py==3.0.0
markdown2==2.4.10
MarkupSafe==2.1.3
matplotlib-inline @ file:///home/conda/feedstock_root/build_artifacts/matplotlib-inline_1660814786464/work
mdurl==0.1.2
mpmath==1.3.0
msgpack==1.0.7
multidict==6.0.4
multiprocess==0.70.15
nest-asyncio @ file:///home/conda/feedstock_root/build_artifacts/nest-asyncio_1697083700168/work
networkx==3.2.1
nh3==0.2.14
ninja==<IP_ADDRESS>
nltk==3.8.1
numpy==1.26.2
nvidia-cublas-cu12==<IP_ADDRESS>
nvidia-cuda-cupti-cu12==12.1.105
nvidia-cuda-nvrtc-cu12==12.1.105
nvidia-cuda-runtime-cu12==12.1.105
nvidia-cudnn-cu12==<IP_ADDRESS>
nvidia-cufft-cu12==<IP_ADDRESS>
nvidia-curand-cu12==<IP_ADDRESS>
nvidia-cusolver-cu12==<IP_ADDRESS>
nvidia-cusparse-cu12==<IP_ADDRESS>
nvidia-nccl-cu12==2.18.1
nvidia-nvjitlink-cu12==12.3.101
nvidia-nvtx-cu12==12.1.105
openai==0.28.0
packaging @ file:///home/conda/feedstock_root/build_artifacts/packaging_1696202382185/work
pandas==2.1.3
parso @ file:///home/conda/feedstock_root/build_artifacts/parso_1638334955874/work
patsy==0.5.3
peft==0.6.2
pexpect @ file:///home/conda/feedstock_root/build_artifacts/pexpect_1667297516076/work
pickleshare @ file:///home/conda/feedstock_root/build_artifacts/pickleshare_1602536217715/work
platformdirs @ file:///home/conda/feedstock_root/build_artifacts/platformdirs_1699715570510/work
prompt-toolkit @ file:///home/conda/feedstock_root/build_artifacts/prompt-toolkit_1699963054032/work
protobuf==4.25.1
psutil==5.9.6
ptyprocess @ file:///home/conda/feedstock_root/build_artifacts/ptyprocess_1609419310487/work/dist/ptyprocess-0.7.0-py2.py3-none-any.whl
pure-eval @ file:///home/conda/feedstock_root/build_artifacts/pure_eval_1642875951954/work
py-cpuinfo==9.0.0
pyarrow==14.0.1
pyarrow-hotfix==0.6
pydantic==1.10.13
Pygments @ file:///home/conda/feedstock_root/build_artifacts/pygments_1700607939962/work
pynvml==11.5.0
python-dateutil @ file:///home/conda/feedstock_root/build_artifacts/python-dateutil_1626286286081/work
python-dotenv==1.0.0
pytz==2023.3.post1
PyYAML==6.0.1
pyzmq @ file:///home/conda/feedstock_root/build_artifacts/pyzmq_1698062432081/work
ray==2.8.0
referencing==0.31.0
regex==2023.10.3
requests==2.31.0
rich==13.7.0
rpds-py==0.13.1
safetensors==0.4.0
scikit-learn==1.3.2
scipy==1.11.4
sentencepiece==0.1.99
sentry-sdk==1.37.1
setproctitle==1.3.3
shortuuid==1.0.11
shtab==1.6.4
six @ file:///home/conda/feedstock_root/build_artifacts/six_1620240208055/work
smmap==5.0.1
sniffio==1.3.0
stack-data @ file:///home/conda/feedstock_root/build_artifacts/stack_data_1669632077133/work
starlette==0.27.0
statsmodels==0.14.0
svgwrite==1.4.3
sympy==1.12
tabulate==0.9.0
termcolor==2.3.0
threadpoolctl==3.2.0
tiktoken==0.5.1
tokenizers==0.15.0
torch==2.1.1
tornado @ file:///home/conda/feedstock_root/build_artifacts/tornado_1695373450800/work
-e git+ssh://git@github.com/mgerstgrasser/multistep_rlhf.git@41ce61e93a430079fbadf0af6542d65792efa9a0#egg=tortoise
tqdm==4.66.1
traitlets @ file:///home/conda/feedstock_root/build_artifacts/traitlets_1701095650114/work
transformers==4.35.2
triton==2.1.0
trl==0.7.7
typing_extensions @ file:///home/conda/feedstock_root/build_artifacts/typing_extensions_1695040754690/work
tyro==0.5.18
tzdata==2023.3
urllib3==2.1.0
uvicorn==0.24.0.post1
uvloop==0.19.0
vllm==0.2.2
wandb==0.16.0
watchfiles==0.21.0
wavedrom==2.0.3.post3
wcwidth @ file:///home/conda/feedstock_root/build_artifacts/wcwidth_1700607916581/work
websockets==12.0
xformers==0.0.22.post7
xxhash==3.4.1
yarl==1.9.3
zipp @ file:///home/conda/feedstock_root/build_artifacts/zipp_1695255097490/work

nvidia driver and cuda versions (from nvidia-smi):
Driver Version: 535.104.12   CUDA Version: 12.2
kernel version (uname -r)
4.18.0-425.10.1.el8_7.x86_64

Further update: After a lot more digging, it turns out the NCCL timeout in Deepspeed Stage 2 was the same issue as #1103, and is fixed by #1177.
I'm still seeing issues with Deepspeed Stage 3 though, e.g. the Invalidate trace cache @ step 0: expected module 0, but got module 1 warning above, performance seems much lower than I would expect, and sometimes it crashes after a few iterations with the params INFLIGHT error I mentioned earlier.
Any chance you have any advice on that? @vwxyzjn

Just to confirm, does deepspeed stage 3 work for you at the moment? I had weird experiences with stage 3 as well, saw Invalidate trace cache @ step 0: expected module 0, but got module 1.
stage 2 should be sufficient, as https://github.com/OpenLMLab/MOSS-RLHF can fit in a 6.9B critic, reward, policy, and ref policy model.
Also, what's your hardware settings?

Just to confirm, does deepspeed stage 3 work for you at the moment? I had weird experiences with stage 3 as well, saw Invalidate trace cache @ step 0: expected module 0, but got module 1.
stage 2 should be sufficient, as https://github.com/OpenLMLab/MOSS-RLHF can fit in a 6.9B critic, reward, policy, and ref policy model.
Also, what's your hardware settings?

Ah, ideally I'd like to scale up to 13B and maybe even 70B models though.
And good question whether it's working or not - I thought it always crashed after a few steps with that params INFLIGHT error, but I just had one run later last night that seemed to work, and just now I managed to get several iterations into another one, so looks like it is working now. I wonder if that's a coincidence, or if this was also fixed by #1177 maybe.
The remaining issue is that it's slow - around 6-8 times as long per iteration compared to stage 2. That's just with the default model in the ppo example script, and 2 or 4 NVLinked GPUs. I realise of course that it will be somewhat slower than stage 2, but is that big of a performance drop expected?
Re hardware: Varies, but the main configuration I've been using has 4x A100 80GB SXM4 per node, and 200Gbps Infiniband between nodes. For the 7B model I've tested on a node with 8x A100 80GB SXM4 GPUs.

Hello, I am facing the same error in deepspeed zero3, AssertionError    assert param.ds_status == ZeroParamStatus.AVAILABLE, in self.accelerator.backward(loss) of PPOTrainer.train_minibatch. I suspect the error is related to the issues reported in https://github.com/microsoft/DeepSpeed/issues/4194 and https://github.com/microsoft/DeepSpeed/issues/4194, where using transformers==4.31.0 is reported to work. But for me downgrading transformers caused other issues, so I really hope to know other solutions if any.


How to capture chart events?

eg. plotly_click()?

Two variants:

Get ref of chart and manually add event with this.$refs.chart.on('plotly_click', () => {});
Apply patch  from PR #4  to use vue-like events


Add support for range validation

Add a range annotation:
Range[NUMBER_TYPE, FROM, TO]

Implemented with switch to pydantic


Night CI 2023-06-30: Failed

action url: https://github.com/spidernet-io/spiderpool/actions/runs/5419403009

https://github.com/spidernet-io/spiderpool/pull/1996


Function type parameters with explicit Object bound are instantiated to dynamic

The default type for a type parameter with no specified bound is dynamic. The default type for a type parameter with bound Object is Object.
For function types, CFE instantiates dynamic in both cases and thus does not report the error in this program:
Q hest<Q>(dynamic x) {
  if (x is Q) return x;
  return null;
}

Q Function<Q>(dynamic) pony = hest;
Q Function<Q extends Object>(dynamic) zebra = hest;

main() {
  hest(42).fisk(); //# 01: ok
  pony(42).fisk(); //# 02: ok
  zebra(42).fisk(); //# 03: compile-time error
}

I think there's a fix for it that's going to be landed soon: https://dart-review.googlesource.com/c/sdk/+/81405.


You CAN use conditional expressions in if

From 7.4 Templates: This is not true.

You CANNOT use conditional expressions in if, for instance<EMAIL_ADDRESS>Only boolean values are acceptable.

You can use eq built-in function to check equality in if.
Strictly speaking, this is not a conditional expression but it provides the same function.
package main

import (
	"os"
	"text/template"
)

func main() {
	tIfElse := template.New("template test")
	tIfElse = template.Must(tIfElse.Parse("if-else demo: {{if eq .Mail<EMAIL_ADDRESS>if part {{else}} else part.{{end}}\n"))
	tIfElse.Execute(os.Stdout, map[string]string{"Mail"<EMAIL_ADDRESS>}

I think eq is add in Go1.3. when I write this book it still not works


cmd/go2go: Panic on unresolved multi-parameter generic calls

What version of Go are you using (go version)?
go version 2021-03-22 linux/amd64

Does this issue reproduce with the latest release?
yes
What operating system and processor architecture are you using (go env)?
go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN="/home/rvolosatovs/.local/bin.go"
GOCACHE="/home/rvolosatovs/.local/cache/go-build"
GOENV="/home/rvolosatovs/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/rvolosatovs/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/rvolosatovs"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/nix/store/jfzfymdd8pdia5jlvnc96nm5znjh9x6j-go2-unstable-2021-03-22/share/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/nix/store/jfzfymdd8pdia5jlvnc96nm5znjh9x6j-go2-unstable-2021-03-22/share/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="2021-03-22"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/rvolosatovs/src/github.com/rvolosatovs/collections/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/run/user/1000/go-build2470782358=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Panics(two or more parameters): https://go2goplay.golang.org/p/gc3cZ-qe5bG
Works: https://go2goplay.golang.org/p/AQ_JWojgRic
Prints correct error(single parameter): https://go2goplay.golang.org/p/bW2i30OWP6F
What did you expect to see?
The first example should print an error same as last one
Note, that calls to identifiers defined in imported packages with multiple parameters fail with the same error
What did you see instead?
Panic in parser

This is a panic in the go2go type checker.  The current type checker reports an error (undefined: some).  I don't think it's worth updating the go2go type checker to better handle erroneous cases, but leaving for @griesemer .

Well
panic: prog.go2:4:7: unknown expression type *ast.ListExpr [recovered]
	panic: prog.go2:4:7: unknown expression type *ast.ListExp

is quite cryptic for the end user in my opinion, so I think it's definitely worth improving. Note, again, that it works fine with single parameter identifiers.
What's more important is that this seems to also break references to identifiers in different packages.
Locally, the go2go tool panics when instantiating multi-parameter imported types/functions.

@rvolosatovs This is working as expected on tip (types2, and compiler with -G=3 option). It doesn't produce any error message with go/types at the moment, and that seems like a bug. What you are seeing is likely due to running a version that is not fully up-to-date. The go2go playground will be updated eventually, but currently is a bit behind.

@findleyr This program:
package main

func main() {
	some[int, int]()
}

doesn't seem to produce an error message with go/types at the moment.


ci: add startup-script to local node repo

Lack of this image is reason for gke tests to fail, if it's available
node init should run properly in gke.

test-gke K8sDemos

test-gke K8sDemos

test-gke K8sDemos

Coverage increased (+0.02%) to 36.941% when pulling 7fa3c4796fe04bbcbbf4c89db7996721e22e801d on pr/gke-fix-node-init-backoff into ea8b727d7b663c4482f26348cbf613c29811879f on master.


[ZEPPELIN-552] Shadow on focused paragraph in default looknfeel

What is this PR for?
Add visual indication of focused paragraph by adding shadow in 'default' looknfeel.
What type of PR is it?
Improvement
Is there a relevant Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-552
How should this be tested?
focus paragraph by click editor or output of paragraph
Screenshots (if appropriate)

Questions:

Does the licenses files need update? no
Is there breaking changes for older versions? no
Does this needs documentation? no

I have added shadow on focused paragraph in 'default' looknfeel, just like 'simple' looknfeel does.
Any opinion and feedback on this change is very appreciated

Looks good, However when we focus by clicking on the output, shortcut are not working.

Tested, works well, LGTM.

@corneadoug While this PR only adds shadow by editing css, if keyboard shortcut does not work, i think it must be different problem. Please create jira issue when you find problem on keyboard shortcut and let me handle it.
I'm merging this PR if adding shadow to focused paragraph looks fine and there're no more discussions.

LGTM


Provide a SLES like boot structure in ISOs to provision from SUMA/Uyuni

Describe the solution you'd like:
For customers with traditional deployments that are using SUSE Manager the configured PXE boot mechanism could be used to deploy elemental generated ISO images.
Currently, with SLES, the ISO image is mounted as a loopback device on a directory and server from there without any modification.
Anything else you would like to add:
The boot structure for the ISO image to work directly to serve via PXE boot with SUMA / Uyuni is as follows:
/boot/<arch>/loader/linux <-- Kernel
/boot/<arch>/loader/initrd <-- Init image
Example
/boot/x86_64/loader/linux
/boot/x86_64/loader/initrd

Environment:

Elemental release version (use cat /etc/os-release): N/A
Rancher version: N/A
Kubernetes version (use kubectl version): N/A
Cloud provider or hardware configuration: N/A

Thanks to @mbologna, we were able to test the new .iso boot structure against SUSE Manager and the image is recognized as valid:

We also counter tested with an old image and that indeed fail.
So this should be resolved as soon the PR is merged.


ERROR: Couldn't initialize windows connection: 无法在设置线程模式后对其加以更改。 (0x80010106) [0921/024011.459:ERROR:window_impl.cc(121)] Failed to unregister class Chrome_WidgetWin_0. Error = 1412 error Command failed with exit code<PHONE_NUMBER>. info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command. error Command failed with exit code 1. info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

ERROR: Couldn't initialize windows connection: 无法在设置线程模式后对其加以更改。 (0x80010106)
[0921/024011.459:ERROR:window_impl.cc(121)] Failed to unregister class Chrome_WidgetWin_0. Error = 1412
error Command failed with exit code<PHONE_NUMBER>.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
mycode >>>>
#[command]
fn get_system_volume() -> Result<f32, String> {
unsafe {
let mut controller = AudioController::init(None);
controller.GetSessions();
controller.GetDefaultAudioEnpointVolumeControl();
controller.GetAllProcessSessions();
let test = controller.get_all_session_names();
let master_session = controller.get_session_by_name("master".to_string());
println!("{:?}",master_session.unwrap().getVolume());
Ok(master_session.unwrap().getVolume())
}
}

Hey, sorry for the late reply... i don't think this is an issue of this library.  It doesn't support wasm.


Install Chrome

brew cask install google-chrome

Also, it should set Chrome as the default browser and install related extensions

Extensions can be synced by signing into Chrome and turning on sync. Can this be done in the dot files?


Bug in midiToInput & 1500+ unit tests

Hello,
There is a bug in QLCMIDIProtocol::midiToInput(): It expects 2 data bytes but according to the MIDI spec only 1 byte is provided. Note that the reverse function (feedbackToMidi) correctly uses only the first byte.
I also produced some unit tests...
M.

I didn't try to recompile the whole project… and CHANNEL_OFFSET_PROGRAM_CHANGE_MAX is used by InputChannelEditor::numberToMidi :-(
I'll update my code with something that wholly compiles.

Ok, let's take a step back then.
What is the reason why this patch is needed ? Cause last time I checked, program changes were working.
Please indicate step by step what you did, and the equipment you used. Thanks.

I have a problem with a Behringer X-Touch Compact -- but this isn't directly related to this patch.
However, I wanted to learn a bit the code before diving into my problem; so I started writing some unit tests for the midiprotocol.cpp functions. And this is how I discovered there is an inconsistency between both functions (midiToInput & feedbackToMidi).
Note that I just grep-ed through all the profile files in resources/inputprofiles for any value between 256 and 513 as the channel number (i.e. Note aftertouch, Program Change, Channel aftertouch & Pitch Wheel). Only Pitch Wheel is used, by Behringer-BCF2000inMackieControlMode.qxi & Zoom-R16.qxi.
So I believe no manufacturers use PC within their DAW… So there is no real reason -- so far -- to apply this patch. But there is a bug in midiToInput because data2 isn't defined in the case of PC. Or this is feedbackToMidi that is wrong (because it uses data1) -- but I think midi.org is accurate.
My 2 cents.


here is a Chinese translation

http://www.cnblogs.com/jonkee/p/5092484.html
this is a Chinese translation for promise/A+

great!


Make documentation link to C++ code

System information

TensorFlow version: All
Doc Link: All the Python API documentation, for example: https://www.tensorflow.org/api_docs/python/tf/nn/conv2d_transpose

Describe the documentation issue
The Python API documentation often points to the Python code (on github) where the operation is defined. For the example, for tf.nn.conv2d_transpose(), it links to this code.
Unfortunately, most operations are fairly thin wrappers around C++ operations, and since the link from Python to C++ is automatically generated (in this example, it's gen_nn_ops.conv2d_backprop_input()), it is not trivial to find the corresponding C++ code (the mapping is in Bazel code, really hard to find). Many people have been bothered by this problem, as you can see by searching for gen_nn_ops on StackOverflow, for example this question: https://stackoverflow.com/questions/41147734/looking-for-source-code-of-from-gen-nn-ops-in-tensorflow
It would be great if the documentation could point to both the Python function and the C++ operation. In this case, it would be https://www.tensorflow.org/versions/r2.0/api_docs/cc/class/tensorflow/ops/conv2-d-backprop-input and the source code is in tensorflow/core/kernels/conv_grad_input_ops.cc.
To find it, I had to search locally on my computer to find the gen_nn_ops.py file, and I found that gen_nn_ops.conv2d_backprop_input() just called the Conv2DBackpropInput operation. But then I had to go back to github to search for its C++ source code (since the TensorFlow binary does not include it), and it was tricky to find, since the C++ operation is also dynamically registered, so the actual name of the function is Conv2DCustomBackpropInputOp. Searching for "REGISTER_KERNEL_BUILDER  Conv2DBackpropInput" helps.
We welcome contributions by users. Will you be able to update submit a PR (use the doc style guide) to fix the doc Issue?
I'm not sure where I could contribute this fix.

Hi,
Thank you for opening this issue. Since this issue has been open for a long time, the code/debug information for this issue may not be relevant with the current state of the code base.
The Tensorflow team is constantly improving the framework by fixing bugs and adding new features. We suggest you try the latest TensorFlow version with the latest compatible hardware configuration which could potentially resolve the issue. If you are still facing the issue, please create a new GitHub issue with your latest findings, with all the debugging information which could help us investigate.
Please follow the release notes to stay up to date with the latest developments which are happening in the Tensorflow space.
Thank you!

This feature is a good idea, but I' never going to have tie to come back and fix this.


docker export and load problems

Hi everyone,
I got an error when doing a export and load. In the host A I do export
sudo  docker export content-web-kr | gzip - > content-web-kr.tgz
When it is done I scp the tar ball and load it in machine B
sudo docker load <  content-web-kr.tgz
In B it runs for a while and then print the error:
FATA[0159] Error: open /var/lib/docker/tmp/docker-import-937914018/repo/bin/json: no such file or directory
I did extract the tar ball in verify that there is no such file a directory.
A is a centos 7 64 bit machine with docker Docker version 1.3.2, build 39fa2fa/1.3.2
B is a centos 6 64 bit - same version of docker
I tried when B is ubuntu 14.04 as well same errors. (Docker version 1.4.1, build 5bc2ff8).

I think while load failed but import works fine with exported tar ball. So for some reason the doco is not clear on that point - if you export a CT to a tar abll you need to use import rather than load. if you save an image to a tar ball then use load to restore.
So it might not be a bug at all

tar cvf image.tar image/
check the image  path.
it should be:
cd image/
tar cvf image.tar  ＊


dev-website

Minimal updates, no functional changes:

Added animated Marie video
Updated link to Universal Digital Twin paper.

Tested in a locally spun up stack, no issues found.


Move action required message to the spec

WHY are these changes introduced?
Move the responsibility of the "Action Required" message to the specification instead of hardcoding the message in the dev-session

WHAT is this pull request doing?
Add a new getDevSessionActionUpdateMessage to the extension-instance and the extension-specification. And define it for the app access module

How to test your changes?
Run consistent-dev, make a change in the scopes, you should see the action required message.

Post-release steps

Measuring impact
How do we know this change was effective? Please choose one:

[ ] n/a - this doesn't need measurement, e.g. a linting rule or a bug-fix
[ ] Existing analytics will cater for this addition
[ ] PR includes analytics changes to measure impact

Checklist

[ ] I've considered possible cross-platform impacts (Mac, Linux, Windows)
[ ] I've considered possible documentation changes

Merge activity

Nov 22, 6:01 PM EST: A user started a stack merge that includes this pull request via Graphite.