[BUG] Error during LoRA-merge in HF upload for Llama 3.1 70B model

🐛 Bug
Today when attempting to upload a LoRA-trained Llama 3.1 70B model (first time I've trained Llama 3.1), I hit the following during the eLoRA merge. Note I used the cpu_shard method to upload. I've tried it twice now with the same error.
2024-07-24 17:22:58,705 - INFO: Stop token ids: [tensor([  27,   91, 9125,   91,   29])]
2024-07-24 17:22:59,686 - INFO: Stop token ids: [tensor([  27,   91, 9125,   91,   29])]
2024-07-24 17:22:59,701 - WARNING: PAD token id not matching between config and tokenizer. Overwriting with tokenizer id 128001.
2024-07-24 17:22:59,701 - INFO: Setting pretraining_tp of model config to 1.
2024-07-24 17:22:59,723 - INFO: Using bfloat16 for backbone
2024/07/24 17:23:07 # {"client":"3f76ec33-3e3f-4837-9673-cda3f39f377f","state":"DISCONNECT","t":"ws_disconnect"}
2024/07/24 17:23:07 # {"addr":"<IP_ADDRESS>:49420","client_id":"3f76ec33-3e3f-4837-9673-cda3f39f377f","t":"client_reconnect"}
2024-07-24 18:04:05,704 - INFO: Attention implementation: sdpa
2024-07-24 18:04:05,713 - INFO: Lora module names: ['q_proj', 'k_proj', 'v_proj', 'o_proj', 'gate_proj', 'up_proj', 'down_proj']
2024-07-24 18:06:03,026 - INFO: Trainable parameters count:<PHONE_NUMBER>
2024-07-24 18:06:03,027 - INFO: Total parameters count:<PHONE_NUMBER>6
2024-07-24 18:06:03,027 - INFO: Trainable %: 8.5863%
2024-07-24 18:08:56,811 - INFO: Weights loaded from: /home/ubuntu/h2o-llmstudio/output/user/heavyiq-llama-3-1-70b-combo-v61-5-no-cte-judge-3584-tokens-lora-r-512-a-1024-lr-1-1e-5.1/checkpoint.pth
2024-07-24 18:10:15,356 - INFO: Merging LORA layers with base model.
2024-07-24 18:10:15,561 - ERROR: Unknown exception
Traceback (most recent call last):
File "/home/ubuntu/h2o-llmstudio/./llm_studio/app_utils/handlers.py", line 358, in handle
await experiment_push_to_huggingface_dialog(q)
File "/home/ubuntu/h2o-llmstudio/./llm_studio/app_utils/sections/experiment.py", line 2015, in experiment_push_to_huggingface_dialog
publish_model_to_hugging_face(
File "/home/ubuntu/h2o-llmstudio/./llm_studio/app_utils/hugging_face_utils.py", line 216, in publish_model_to_hugging_face
cfg, model, tokenizer = load_cfg_model_tokenizer(
File "/home/ubuntu/h2o-llmstudio/./llm_studio/app_utils/sections/chat.py", line 241, in load_cfg_model_tokenizer
model.backbone = model.backbone.merge_and_unload()
File "/home/ubuntu/miniconda3/envs/h2o_llm_studio_jul_24/lib/python3.10/site-packages/peft/tuners/lora/model.py", line 838, in merge_and_unload
return self._unload_and_optionally_merge(
File "/home/ubuntu/miniconda3/envs/h2o_llm_studio_jul_24/lib/python3.10/site-packages/peft/tuners/lora/model.py", line 457, in _unload_and_optionally_merge
target.merge(safe_merge=safe_merge, adapter_names=adapter_names)
File "/home/ubuntu/miniconda3/envs/h2o_llm_studio_jul_24/lib/python3.10/site-packages/peft/tuners/lora/layer.py", line 470, in merge
delta_weight = self.get_delta_weight(active_adapter)
File "/home/ubuntu/miniconda3/envs/h2o_llm_studio_jul_24/lib/python3.10/site-packages/peft/tuners/lora/layer.py", line 533, in get_delta_weight
output_tensor = transpose(weight_B @ weight_A, self.fan_in_fan_out) * self.scaling[adapter]
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0! (when checking argument for argument mat2 in method wrapper_CUDA_mm)
To Reproduce
cfg.yaml
Last login: Wed Jul 24 09:21:21 on ttys003
The default interactive shell is now zsh.
To update your account to use zsh, please run chsh -s /bin/zsh.
                                  GPU CLOUD

Documentation:  https://help.ubuntu.com
Management:     https://landscape.canonical.com
Support:        https://ubuntu.com/advantage

System information as of Wed Jul 24 18:59:10 UTC 2024
System load:  0.04296875         Processes:                2188
Usage of /:   16.2% of 18.93TB   Users logged in:          1
Memory usage: 3%                 IPv4 address for docker0: <IP_ADDRESS>
Swap usage:   0%                 IPv4 address for eno1:    <IP_ADDRESS>
architecture:
backbone_dtype: bfloat16
gradient_checkpointing: true
intermediate_dropout: 0.0
pretrained: true
pretrained_weights: ''
augmentation:
neftune_noise_alpha: 0.0
random_parent_probability: 0.0
skip_parent_probability: 0.0
token_mask_probability: 0.0
dataset:
add_bos_token_to_answer: false
add_bos_token_to_prompt: false
add_bos_token_to_system: false
add_eos_token_to_answer: true
add_eos_token_to_prompt: false
add_eos_token_to_system: true
answer_column: answer
chatbot_author: H2O.ai
chatbot_name: h2oGPT
data_sample: 1.0
data_sample_choice:
- Train
- Validation
limit_chained_samples: false
mask_prompt_labels: true
parent_id_column: None
personalize: false
prompt_column:
- prompt
prompt_column_separator: ''
system_column: None
text_answer_separator: ''
text_prompt_start: ''
text_system_start: <|system|>
train_dataframe: /home/ubuntu/h2o-llmstudio/data/user/heavyiq_combo_v61_5_no_cte_judgements_3584_tokens_gen1/heavyiq_combo_v61_5_no_cte_judgements_3584_tokens_gen1_train.csv
validation_dataframe: /home/ubuntu/h2o-llmstudio/data/user/heavyiq_combo_v61_5_no_cte_judgements_3584_tokens_gen1/heavyiq_combo_v61_5_no_cte_judgements_3584_tokens_gen1_eval.csv
validation_size: 0.01
validation_strategy: custom
environment:
compile_model: false
deepspeed_allgather_bucket_size: 1000000
deepspeed_method: ZeRO3
deepspeed_reduce_bucket_size: 1000000
deepspeed_stage3_param_persistence_threshold: 1000000
deepspeed_stage3_prefetch_bucket_size: 1000000
find_unused_parameters: false
gpus:
- '0'
- '1'
- '2'
- '3'
- '4'
- '5'
- '6'
- '7'
huggingface_branch: main
mixed_precision: false
mixed_precision_dtype: bfloat16
number_of_workers: 8
seed: 2
trust_remote_code: true
use_deepspeed: true
experiment_name: heavyiq-llama-3-1-70b-combo-v61-5-no-cte-judge-3584-tokens-lora-r-512-a-1024-lr-1-1e-5.1
llm_backbone: meta-llama/Meta-Llama-3.1-70B
logging:
logger: Neptune
neptune_project: tmostak/heavyiq
"cfg.yaml" 119L, 3565B                                                                                1,1           Top
architecture:
backbone_dtype: bfloat16
gradient_checkpointing: true
intermediate_dropout: 0.0
pretrained: true
pretrained_weights: ''
augmentation:
neftune_noise_alpha: 0.0
random_parent_probability: 0.0
skip_parent_probability: 0.0
token_mask_probability: 0.0
dataset:
add_bos_token_to_answer: false
add_bos_token_to_prompt: false
add_bos_token_to_system: false
add_eos_token_to_answer: true
add_eos_token_to_prompt: false
add_eos_token_to_system: true
answer_column: answer
chatbot_author: H2O.ai
chatbot_name: h2oGPT
data_sample: 1.0
data_sample_choice:
- Train
- Validation
limit_chained_samples: false
mask_prompt_labels: true
parent_id_column: None
personalize: false
prompt_column:
- prompt
prompt_column_separator: ''
system_column: None
text_answer_separator: ''
text_prompt_start: ''
text_system_start: <|system|>
train_dataframe: /home/ubuntu/h2o-llmstudio/data/user/heavyiq_combo_v61_5_no_cte_judgements_3584_tokens_gen1/heavyiq_combo_v61_5_no_cte_judgements_3584_tokens_gen1_train.csv
validation_dataframe: /home/ubuntu/h2o-llmstudio/data/user/heavyiq_combo_v61_5_no_cte_judgements_3584_tokens_gen1/heavyiq_combo_v61_5_no_cte_judgements_3584_tokens_gen1_eval.csv
validation_size: 0.01
validation_strategy: custom
environment:
compile_model: false
deepspeed_allgather_bucket_size: 1000000
deepspeed_method: ZeRO3
deepspeed_reduce_bucket_size: 1000000
deepspeed_stage3_param_persistence_threshold: 1000000
deepspeed_stage3_prefetch_bucket_size: 1000000
find_unused_parameters: false
gpus:
- '0'
- '1'
- '2'
- '3'
- '4'
- '5'
- '6'
- '7'
huggingface_branch: main
mixed_precision: false
mixed_precision_dtype: bfloat16
number_of_workers: 8
seed: 2
trust_remote_code: true
use_deepspeed: true
experiment_name: heavyiq-llama-3-1-70b-combo-v61-5-no-cte-judge-3584-tokens-lora-r-512-a-1024-lr-1-1e-5.1
llm_backbone: meta-llama/Meta-Llama-3.1-70B
logging:
logger: Neptune
neptune_project: tmostak/heavyiq
"cfg.yaml" 119L, 3565B                                                                                1,1           Top
LLM Studio version
a1b2923a1ad29571e4fcdf4e9fdfd505c949bd16 (tip of main)

One thing I found that might relate
https://discuss.huggingface.co/t/runtimeerror-expected-all-tensors-to-be-on-the-same-device-but-found-at-least-two-devices-cuda-2-and-cuda-0-when-checking-argument-for-argument-index-in-method-wrapper-cuda-index-select/48991/4
They basically hit the same issue, and someone noted that i have a similar error on the other model(minicpm),i change the version of deepspeed from 0.14.0 to 0.13.2. and it works
Going to try downgrading Deepspeed to see if this helps.

I tried again with deepspeed 0.13.2 and hit the same issue
2024-07-25 15:14:39,542 - INFO: Weights loaded from: /home/ubuntu/h2o-llmstudio/output/user/heavyiq-llama-3-1-70b-combo-v61-5-no-cte-judge-3584-tokens-lora-r-512-a-1024-lr-1-1e-5.1/checkpoint.pth
2024-07-25 15:16:00,567 - INFO: Merging LORA layers with base model.
2024-07-25 15:16:00,771 - ERROR: Unknown exception
Traceback (most recent call last):
  File "/home/ubuntu/h2o-llmstudio/./llm_studio/app_utils/handlers.py", line 358, in handle
    await experiment_push_to_huggingface_dialog(q)
  File "/home/ubuntu/h2o-llmstudio/./llm_studio/app_utils/sections/experiment.py", line 2012, in experiment_push_to_huggingface_dialog
    publish_model_to_hugging_face(
  File "/home/ubuntu/h2o-llmstudio/./llm_studio/app_utils/hugging_face_utils.py", line 216, in publish_model_to_hugging_face
    cfg, model, tokenizer = load_cfg_model_tokenizer(
  File "/home/ubuntu/h2o-llmstudio/./llm_studio/app_utils/sections/chat.py", line 241, in load_cfg_model_tokenizer
    model.backbone = model.backbone.merge_and_unload()
  File "/home/ubuntu/miniconda3/envs/h2o_llm_studio/lib/python3.10/site-packages/peft/tuners/lora/model.py", line 838, in merge_and_unload
    return self._unload_and_optionally_merge(
  File "/home/ubuntu/miniconda3/envs/h2o_llm_studio/lib/python3.10/site-packages/peft/tuners/lora/model.py", line 457, in _unload_and_optionally_merge
    target.merge(safe_merge=safe_merge, adapter_names=adapter_names)
  File "/home/ubuntu/miniconda3/envs/h2o_llm_studio/lib/python3.10/site-packages/peft/tuners/lora/layer.py", line 470, in merge
    delta_weight = self.get_delta_weight(active_adapter)
  File "/home/ubuntu/miniconda3/envs/h2o_llm_studio/lib/python3.10/site-packages/peft/tuners/lora/layer.py", line 533, in get_delta_weight
    output_tensor = transpose(weight_B @ weight_A, self.fan_in_fan_out) * self.scaling[adapter]
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0! (when checking argument for argument mat2 in method wrapper_CUDA_mm)
2024-07-25 15:16:00,773 - INFO: {'home/gpu_stats', 'experiment/display/footer', 'experiment/display/charts/train_loss', 'experiment/display/tab', 'experiment/display/charts/validation_Perplexity', 'experiment/display/charts/validation_loss', 'home/experiments_stats', 'dataset/list', 'experiment/list', 'init_app', 'home/disk_usage', 'home/compute_stats', 'dataset/display/footer', 'experiment/display/charts/meta_lr'}

Thank you for the details. We recently upgraded deepspeed, so could indeed be an issue caused by this. I'll look into it.

@pascal-pfeiffer I wrote a quick Python script to write out the layer names per GPU, and it seems the issue might be how the LoRA layers for layer 8 are split between GPU 0 and GPU 1. Also why are there only the LoRA layers for layer 8 and not the other layers?
GPU 0:
  backbone.base_model.model.model.embed_tokens
  backbone.base_model.model.model.layers.0
  backbone.base_model.model.model.layers.1
  backbone.base_model.model.model.layers.2
  backbone.base_model.model.model.layers.3
  backbone.base_model.model.model.layers.4
  backbone.base_model.model.model.layers.5
  backbone.base_model.model.model.layers.6
  backbone.base_model.model.model.layers.7
  backbone.base_model.model.model.layers.8.self_attn.k_proj.base_layer
  backbone.base_model.model.model.layers.8.self_attn.k_proj.lora_A
  backbone.base_model.model.model.layers.8.self_attn.k_proj.lora_dropout
  backbone.base_model.model.model.layers.8.self_attn.q_proj
GPU 1:
  backbone.base_model.model.model.layers.10
  backbone.base_model.model.model.layers.11
  backbone.base_model.model.model.layers.12
  backbone.base_model.model.model.layers.13
  backbone.base_model.model.model.layers.14
  backbone.base_model.model.model.layers.15
  backbone.base_model.model.model.layers.16
  backbone.base_model.model.model.layers.17
  backbone.base_model.model.model.layers.18.self_attn
  backbone.base_model.model.model.layers.8.input_layernorm
  backbone.base_model.model.model.layers.8.mlp
  backbone.base_model.model.model.layers.8.post_attention_layernorm
  backbone.base_model.model.model.layers.8.self_attn.k_proj.lora_B.default
  backbone.base_model.model.model.layers.8.self_attn.k_proj.lora_embedding_A
  backbone.base_model.model.model.layers.8.self_attn.k_proj.lora_embedding_B
  backbone.base_model.model.model.layers.8.self_attn.o_proj
  backbone.base_model.model.model.layers.8.self_attn.rotary_emb
  backbone.base_model.model.model.layers.8.self_attn.v_proj
  backbone.base_model.model.model.layers.9
GPU 2:
  backbone.base_model.model.model.layers.18.input_layernorm
  backbone.base_model.model.model.layers.18.mlp
  backbone.base_model.model.model.layers.18.post_attention_layernorm
  backbone.base_model.model.model.layers.19
  backbone.base_model.model.model.layers.20
  backbone.base_model.model.model.layers.21
  backbone.base_model.model.model.layers.22
  backbone.base_model.model.model.layers.23
  backbone.base_model.model.model.layers.24
  backbone.base_model.model.model.layers.25
  backbone.base_model.model.model.layers.26
  backbone.base_model.model.model.layers.27
  backbone.base_model.model.model.layers.28.mlp.gate_proj
  backbone.base_model.model.model.layers.28.self_attn
GPU 3:
  backbone.base_model.model.model.layers.28.input_layernorm
  backbone.base_model.model.model.layers.28.mlp.act_fn
  backbone.base_model.model.model.layers.28.mlp.down_proj
  backbone.base_model.model.model.layers.28.mlp.up_proj
  backbone.base_model.model.model.layers.28.post_attention_layernorm
  backbone.base_model.model.model.layers.29
  backbone.base_model.model.model.layers.30
  backbone.base_model.model.model.layers.31
  backbone.base_model.model.model.layers.32
  backbone.base_model.model.model.layers.33
  backbone.base_model.model.model.layers.34
  backbone.base_model.model.model.layers.35
  backbone.base_model.model.model.layers.36
  backbone.base_model.model.model.layers.37
  backbone.base_model.model.model.layers.38.mlp.gate_proj
  backbone.base_model.model.model.layers.38.mlp.up_proj
  backbone.base_model.model.model.layers.38.self_attn
GPU 4:
  backbone.base_model.model.model.layers.38.input_layernorm
  backbone.base_model.model.model.layers.38.mlp.act_fn
  backbone.base_model.model.model.layers.38.mlp.down_proj
  backbone.base_model.model.model.layers.38.post_attention_layernorm
  backbone.base_model.model.model.layers.39
  backbone.base_model.model.model.layers.40
  backbone.base_model.model.model.layers.41
  backbone.base_model.model.model.layers.42
  backbone.base_model.model.model.layers.43
  backbone.base_model.model.model.layers.44
  backbone.base_model.model.model.layers.45
  backbone.base_model.model.model.layers.46
  backbone.base_model.model.model.layers.47
  backbone.base_model.model.model.layers.48
GPU 5:
  backbone.base_model.model.model.layers.49
  backbone.base_model.model.model.layers.50
  backbone.base_model.model.model.layers.51
  backbone.base_model.model.model.layers.52
  backbone.base_model.model.model.layers.53
  backbone.base_model.model.model.layers.54
  backbone.base_model.model.model.layers.55
  backbone.base_model.model.model.layers.56
  backbone.base_model.model.model.layers.57
  backbone.base_model.model.model.layers.58
  backbone.base_model.model.model.layers.59.self_attn
GPU 6:
  backbone.base_model.model.model.layers.59.input_layernorm
  backbone.base_model.model.model.layers.59.mlp
  backbone.base_model.model.model.layers.59.post_attention_layernorm
  backbone.base_model.model.model.layers.60
  backbone.base_model.model.model.layers.61
  backbone.base_model.model.model.layers.62
  backbone.base_model.model.model.layers.63
  backbone.base_model.model.model.layers.64
  backbone.base_model.model.model.layers.65
  backbone.base_model.model.model.layers.66
  backbone.base_model.model.model.layers.67
  backbone.base_model.model.model.layers.68
  backbone.base_model.model.model.layers.69.mlp.gate_proj
  backbone.base_model.model.model.layers.69.self_attn
GPU 7:
  backbone.base_model.model.lm_head
  backbone.base_model.model.model.layers.69.input_layernorm
  backbone.base_model.model.model.layers.69.mlp.act_fn
  backbone.base_model.model.model.layers.69.mlp.down_proj
  backbone.base_model.model.model.layers.69.mlp.up_proj
  backbone.base_model.model.model.layers.69.post_attention_layernorm
  backbone.base_model.model.model.layers.70
  backbone.base_model.model.model.layers.71
  backbone.base_model.model.model.layers.72
  backbone.base_model.model.model.layers.73
  backbone.base_model.model.model.layers.74
  backbone.base_model.model.model.layers.75
  backbone.base_model.model.model.layers.76
  backbone.base_model.model.model.layers.77
  backbone.base_model.model.model.layers.78
  backbone.base_model.model.model.layers.79
  backbone.base_model.model.model.norm
  backbone.base_model.model.model.rotary_emb
  loss_fn
  perplexity

Another update, I tried to upload to HuggingFace the Llama 3 (not 3.1) model that I had previously successfully uploaded, and got the same Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0! (when checking argument for argument mat2 in method wrapper_CUDA_mm) error as with 3.1, suggesting it was the requirements upgrade that has changed things?

Ok so I rolled back to b4d04c057c7b0a4894d57264d2df7e219e234db2 (fix prompt separator) and re-installed the deps (although had to use transformers 4.43.2 to accomodate the llama 3.1 rope change but had deepspeed 0.13.2), and got the same error exporting.
I then wrote a script to dump the layer names from the model... note how all the LoRA layers are there unlike in the output of device_map
Just showing the first 4 layers but you get the idea
model.embed_tokens.weight
model.layers.0.self_attn.q_proj.base_layer.weight
model.layers.0.self_attn.q_proj.lora_A.default.weight
model.layers.0.self_attn.q_proj.lora_B.default.weight
model.layers.0.self_attn.k_proj.base_layer.weight
model.layers.0.self_attn.k_proj.lora_A.default.weight
model.layers.0.self_attn.k_proj.lora_B.default.weight
model.layers.0.self_attn.v_proj.base_layer.weight
model.layers.0.self_attn.v_proj.lora_A.default.weight
model.layers.0.self_attn.v_proj.lora_B.default.weight
model.layers.0.self_attn.o_proj.base_layer.weight
model.layers.0.self_attn.o_proj.lora_A.default.weight
model.layers.0.self_attn.o_proj.lora_B.default.weight
model.layers.0.mlp.gate_proj.base_layer.weight
model.layers.0.mlp.gate_proj.lora_A.default.weight
model.layers.0.mlp.gate_proj.lora_B.default.weight
model.layers.0.mlp.up_proj.base_layer.weight
model.layers.0.mlp.up_proj.lora_A.default.weight
model.layers.0.mlp.up_proj.lora_B.default.weight
model.layers.0.mlp.down_proj.base_layer.weight
model.layers.0.mlp.down_proj.lora_A.default.weight
model.layers.0.mlp.down_proj.lora_B.default.weight
model.layers.0.input_layernorm.weight
model.layers.0.post_attention_layernorm.weight
model.layers.1.self_attn.q_proj.base_layer.weight
model.layers.1.self_attn.q_proj.lora_A.default.weight
model.layers.1.self_attn.q_proj.lora_B.default.weight
model.layers.1.self_attn.k_proj.base_layer.weight
model.layers.1.self_attn.k_proj.lora_A.default.weight
model.layers.1.self_attn.k_proj.lora_B.default.weight
model.layers.1.self_attn.v_proj.base_layer.weight
model.layers.1.self_attn.v_proj.lora_A.default.weight
model.layers.1.self_attn.v_proj.lora_B.default.weight
model.layers.1.self_attn.o_proj.base_layer.weight
model.layers.1.self_attn.o_proj.lora_A.default.weight
model.layers.1.self_attn.o_proj.lora_B.default.weight
model.layers.1.mlp.gate_proj.base_layer.weight
model.layers.1.mlp.gate_proj.lora_A.default.weight
model.layers.1.mlp.gate_proj.lora_B.default.weight
model.layers.1.mlp.up_proj.base_layer.weight
model.layers.1.mlp.up_proj.lora_A.default.weight
model.layers.1.mlp.up_proj.lora_B.default.weight
model.layers.1.mlp.down_proj.base_layer.weight
model.layers.1.mlp.down_proj.lora_A.default.weight
model.layers.1.mlp.down_proj.lora_B.default.weight
model.layers.1.input_layernorm.weight
model.layers.1.post_attention_layernorm.weight
model.layers.2.self_attn.q_proj.base_layer.weight
model.layers.2.self_attn.q_proj.lora_A.default.weight
model.layers.2.self_attn.q_proj.lora_B.default.weight
model.layers.2.self_attn.k_proj.base_layer.weight
model.layers.2.self_attn.k_proj.lora_A.default.weight
model.layers.2.self_attn.k_proj.lora_B.default.weight
model.layers.2.self_attn.v_proj.base_layer.weight
model.layers.2.self_attn.v_proj.lora_A.default.weight
model.layers.2.self_attn.v_proj.lora_B.default.weight
model.layers.2.self_attn.o_proj.base_layer.weight
model.layers.2.self_attn.o_proj.lora_A.default.weight
model.layers.2.self_attn.o_proj.lora_B.default.weight
model.layers.2.mlp.gate_proj.base_layer.weight
model.layers.2.mlp.gate_proj.lora_A.default.weight
model.layers.2.mlp.gate_proj.lora_B.default.weight
model.layers.2.mlp.up_proj.base_layer.weight
model.layers.2.mlp.up_proj.lora_A.default.weight
model.layers.2.mlp.up_proj.lora_B.default.weight
model.layers.2.mlp.down_proj.base_layer.weight
model.layers.2.mlp.down_proj.lora_A.default.weight
model.layers.2.mlp.down_proj.lora_B.default.weight
model.layers.2.input_layernorm.weight
model.layers.2.post_attention_layernorm.weight
model.layers.3.self_attn.q_proj.base_layer.weight
model.layers.3.self_attn.q_proj.lora_A.default.weight
model.layers.3.self_attn.q_proj.lora_B.default.weight
model.layers.3.self_attn.k_proj.base_layer.weight
model.layers.3.self_attn.k_proj.lora_A.default.weight
model.layers.3.self_attn.k_proj.lora_B.default.weight
model.layers.3.self_attn.v_proj.base_layer.weight
model.layers.3.self_attn.v_proj.lora_A.default.weight
model.layers.3.self_attn.v_proj.lora_B.default.weight
model.layers.3.self_attn.o_proj.base_layer.weight
model.layers.3.self_attn.o_proj.lora_A.default.weight
model.layers.3.self_attn.o_proj.lora_B.default.weight
model.layers.3.mlp.gate_proj.base_layer.weight
model.layers.3.mlp.gate_proj.lora_A.default.weight
model.layers.3.mlp.gate_proj.lora_B.default.weight
model.layers.3.mlp.up_proj.base_layer.weight
model.layers.3.mlp.up_proj.lora_A.default.weight
model.layers.3.mlp.up_proj.lora_B.default.weight
model.layers.3.mlp.down_proj.base_layer.weight
model.layers.3.mlp.down_proj.lora_A.default.weight
model.layers.3.mlp.down_proj.lora_B.default.weight
model.layers.3.input_layernorm.weight
model.layers.3.post_attention_layernorm.weight
model.layers.4.self_attn.q_proj.base_layer.weight
model.layers.4.self_attn.q_proj.lora_A.default.weight
model.layers.4.self_attn.q_proj.lora_B.default.weight
model.layers.4.self_attn.k_proj.base_layer.weight
model.layers.4.self_attn.k_proj.lora_A.default.weight
model.layers.4.self_attn.k_proj.lora_B.default.weight
model.layers.4.self_attn.v_proj.base_layer.weight
model.layers.4.self_attn.v_proj.lora_A.default.weight
model.layers.4.self_attn.v_proj.lora_B.default.weight
model.layers.4.self_attn.o_proj.base_layer.weight
model.layers.4.self_attn.o_proj.lora_A.default.weight
model.layers.4.self_attn.o_proj.lora_B.default.weight
model.layers.4.mlp.gate_proj.base_layer.weight
model.layers.4.mlp.gate_proj.lora_A.default.weight
model.layers.4.mlp.gate_proj.lora_B.default.weight
model.layers.4.mlp.up_proj.base_layer.weight
model.layers.4.mlp.up_proj.lora_A.default.weight
model.layers.4.mlp.up_proj.lora_B.default.weight
model.layers.4.mlp.down_proj.base_layer.weight
model.layers.4.mlp.down_proj.lora_A.default.weight
model.layers.4.mlp.down_proj.lora_B.default.weight
model.layers.4.input_layernorm.weight
model.layers.4.post_attention_layernorm.weight

Thank you for all the further investigations @tmostak. I am trying to reproduce the issue starting with default parameters and mostly aligning with the ones you used and the default dataset.
Using the cfg below, I ran a successful training experiment and upload to Hugging Face Hub.
Everything ran on commit 87c2978698545c758b639fb83e0ceef7e43e91e5, so basically what we have in v1.9.0 release.
Could you by chance upload a reproducable config using the default dataset where you are facing the issue? Your config above for example doesn't include LoRA settings.
architecture:
    backbone_dtype: bfloat16
    gradient_checkpointing: true
    intermediate_dropout: 0.0
    pretrained: true
    pretrained_weights: ''
augmentation:
    neftune_noise_alpha: 0.0
    random_parent_probability: 0.0
    skip_parent_probability: 0.0
    token_mask_probability: 0.0
dataset:
    add_eos_token_to_answer: true
    add_eos_token_to_prompt: true
    add_eos_token_to_system: true
    answer_column: output
    chatbot_author: H2O.ai
    chatbot_name: h2oGPT
    data_sample: 0.2
    data_sample_choice:
    - Train
    limit_chained_samples: false
    mask_prompt_labels: true
    only_last_answer: false
    parent_id_column: None
    personalize: false
    prompt_column:
    - instruction
    prompt_column_separator: \n\n
    system_column: None
    text_answer_separator: <|answer|>
    text_prompt_start: <|prompt|>
    text_system_start: <|system|>
    train_dataframe: /home/pascal/h2o-llmstudio/data/user/oasst/train_full.pq
    validation_dataframe: None
    validation_size: 0.01
    validation_strategy: automatic
environment:
    compile_model: false
    deepspeed_allgather_bucket_size: 1000000
    deepspeed_method: ZeRO3
    deepspeed_reduce_bucket_size: 1000000
    deepspeed_stage3_param_persistence_threshold: 1000000
    deepspeed_stage3_prefetch_bucket_size: 1000000
    find_unused_parameters: false
    gpus:
    - '0'
    - '1'
    - '2'
    - '3'
    - '4'
    - '5'
    - '6'
    - '7'
    huggingface_branch: main
    mixed_precision: false
    mixed_precision_dtype: bfloat16
    number_of_workers: 8
    seed: -1
    trust_remote_code: true
    use_deepspeed: true
experiment_name: ruby-walrus
llm_backbone: meta-llama/Meta-Llama-3.1-70B
logging:
    logger: None
    neptune_project: ''
output_directory: /home/pascal/h2o-llmstudio/output/user/ruby-walrus/
prediction:
    batch_size_inference: 0
    do_sample: false
    max_length_inference: 256
    max_time: 0.0
    metric: Perplexity
    metric_gpt_model: gpt-3.5-turbo-0301
    metric_gpt_template: general
    min_length_inference: 2
    num_beams: 1
    num_history: 4
    repetition_penalty: 1.0
    stop_tokens: ''
    temperature: 0.0
    top_k: 0
    top_p: 1.0
problem_type: text_causal_language_modeling
tokenizer:
    add_prompt_answer_tokens: false
    max_length: 8096
    padding_quantile: 1.0
    tokenizer_kwargs: '{"use_fast": true, "add_prefix_space": false}'
training:
    attention_implementation: auto
    batch_size: 2
    differential_learning_rate: 1.0e-05
    differential_learning_rate_layers: []
    drop_last_batch: true
    epochs: 1
    evaluate_before_training: false
    evaluation_epochs: 1.0
    freeze_layers: []
    grad_accumulation: 1
    gradient_clip: 0.0
    learning_rate: 0.0001
    lora: true
    lora_alpha: 16
    lora_dropout: 0.05
    lora_r: 4
    lora_target_modules: ''
    lora_unfreeze_layers: []
    loss_function: TokenAveragedCrossEntropy
    optimizer: AdamW
    save_checkpoint: last
    schedule: Cosine
    train_validation_data: false
    use_dora: false
    warmup_epochs: 0.0
    weight_decay: 0.0

Training
[2024-07-26 08:48:22,582] [INFO] [logging.py:96:log_dist] [Rank 0] Creating torch.bfloat16 ZeRO stage 3 optimizer
[2024-07-26 08:48:22,981] [INFO] [utils.py:781:see_memory_usage] Stage 3 initialize beginning
[2024-07-26 08:48:22,982] [INFO] [utils.py:782:see_memory_usage] MA 16.52 GB         Max_MA 24.61 GB         CA 25.61 GB         Max_CA 26 GB
[2024-07-26 08:48:22,983] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 44.94 GB, percent = 2.2%
[2024-07-26 08:48:23,012] [INFO] [stage3.py:130:__init__] Reduce bucket size 1000000
[2024-07-26 08:48:23,012] [INFO] [stage3.py:131:__init__] Prefetch bucket size 1000000
[2024-07-26 08:48:23,289] [INFO] [utils.py:781:see_memory_usage] DeepSpeedZeRoOffload initialize [begin]
[2024-07-26 08:48:23,290] [INFO] [utils.py:782:see_memory_usage] MA 16.52 GB         Max_MA 16.52 GB         CA 25.61 GB         Max_CA 26 GB
[2024-07-26 08:48:23,290] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 44.94 GB, percent = 2.2%
Parameter Offload: Total persistent parameters: 53092352 in 1281 params
[2024-07-26 08:48:24,352] [INFO] [utils.py:781:see_memory_usage] DeepSpeedZeRoOffload initialize [end]
[2024-07-26 08:48:24,353] [INFO] [utils.py:782:see_memory_usage] MA 16.44 GB         Max_MA 16.52 GB         CA 25.61 GB         Max_CA 26 GB
[2024-07-26 08:48:24,353] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 45.17 GB, percent = 2.2%
[2024-07-26 08:48:24,530] [INFO] [utils.py:781:see_memory_usage] Before creating fp16 partitions
[2024-07-26 08:48:24,531] [INFO] [utils.py:782:see_memory_usage] MA 16.44 GB         Max_MA 16.44 GB         CA 25.61 GB         Max_CA 26 GB
[2024-07-26 08:48:24,531] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 45.18 GB, percent = 2.2%
[2024-07-26 08:48:25,043] [INFO] [utils.py:781:see_memory_usage] After creating fp16 partitions: 1
[2024-07-26 08:48:25,044] [INFO] [utils.py:782:see_memory_usage] MA 16.44 GB         Max_MA 16.44 GB         CA 16.64 GB         Max_CA 26 GB
[2024-07-26 08:48:25,045] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 45.28 GB, percent = 2.2%
[2024-07-26 08:48:25,217] [INFO] [utils.py:781:see_memory_usage] Before creating fp32 partitions
[2024-07-26 08:48:25,217] [INFO] [utils.py:782:see_memory_usage] MA 16.44 GB         Max_MA 16.44 GB         CA 16.64 GB         Max_CA 17 GB
[2024-07-26 08:48:25,218] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 45.28 GB, percent = 2.2%
[2024-07-26 08:48:25,420] [INFO] [utils.py:781:see_memory_usage] After creating fp32 partitions
[2024-07-26 08:48:25,420] [INFO] [utils.py:782:see_memory_usage] MA 16.46 GB         Max_MA 16.48 GB         CA 16.64 GB         Max_CA 17 GB
[2024-07-26 08:48:25,421] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 45.28 GB, percent = 2.2%
[2024-07-26 08:48:25,600] [INFO] [utils.py:781:see_memory_usage] Before initializing optimizer states
[2024-07-26 08:48:25,600] [INFO] [utils.py:782:see_memory_usage] MA 16.46 GB         Max_MA 16.46 GB         CA 16.64 GB         Max_CA 17 GB
[2024-07-26 08:48:25,601] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 45.28 GB, percent = 2.2%
[2024-07-26 08:48:25,770] [INFO] [utils.py:781:see_memory_usage] After initializing optimizer states
[2024-07-26 08:48:25,771] [INFO] [utils.py:782:see_memory_usage] MA 16.46 GB         Max_MA 16.49 GB         CA 16.64 GB         Max_CA 17 GB
[2024-07-26 08:48:25,771] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 45.28 GB, percent = 2.2%
[2024-07-26 08:48:25,771] [INFO] [stage3.py:486:_setup_for_real_optimizer] optimizer state initialized
[2024-07-26 08:48:26,387] [INFO] [utils.py:781:see_memory_usage] After initializing ZeRO optimizer
[2024-07-26 08:48:26,388] [INFO] [utils.py:782:see_memory_usage] MA 16.48 GB         Max_MA 16.48 GB         CA 16.64 GB         Max_CA 17 GB
[2024-07-26 08:48:26,388] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory:  used = 45.3 GB, percent = 2.2%
[...]
[2024-07-26 08:48:26,397] [INFO] [config.py:1001:print]   zero_enabled ................. True
[2024-07-26 08:48:26,397] [INFO] [config.py:1001:print]   zero_force_ds_cpu_optimizer .. False
[2024-07-26 08:48:26,397] [INFO] [config.py:1001:print]   zero_optimization_stage ...... 3
[2024-07-26 08:48:26,397] [INFO] [config.py:987:print_user_config]   json = {
    "fp16": {
        "enabled": false,
        "loss_scale_window": 100
    },
    "bf16": {
        "enabled": true,
        "loss_scale_window": 100
    },
    "zero_force_ds_cpu_optimizer": false,
    "zero_optimization": {
        "overlap_comm": true,
        "contiguous_gradients": true,
        "reduce_bucket_size": 1.000000e+06,
        "stage": 3,
        "stage3_prefetch_bucket_size": 1.000000e+06,
        "stage3_param_persistence_threshold": 1.000000e+06,
        "stage3_gather_16bit_weights_on_model_save": true
    },
    "steps_per_print": 2.000000e+03,
    "train_micro_batch_size_per_gpu": 2,
    "gradient_accumulation_steps": 1,
    "wall_clock_breakdown": false
}
2024-07-26 08:48:26,466 - INFO: Evaluation step: 161
2024-07-26 08:48:26,553 - INFO: Evaluation step: 161
2024-07-26 08:48:26,557 - INFO: Evaluation step: 161
2024-07-26 08:48:26,582 - INFO: Evaluation step: 161
2024-07-26 08:48:26,590 - INFO: Evaluation step: 161
2024-07-26 08:48:26,615 - INFO: Evaluation step: 161
2024-07-26 08:48:26,637 - INFO: Evaluation step: 161
2024-07-26 08:48:26,675 - INFO: Training Epoch: 1 / 1
2024-07-26 08:48:26,675 - INFO: train loss:   0%|          | 0/161 [00:00<?, ?it/s]
2024-07-26 08:48:26,807 - INFO: Evaluation step: 161
2024-07-26 08:48:28,215 - INFO: Stop token ids: [tensor([  27,   91, 9399,   91,   29]), tensor([  27,   91, 9125,   91,   29]), tensor([   27,    91,
 41681,    91,    29])]
2024-07-26 08:49:08,638 - INFO: train loss: 1.14:   5%|4         | 8/161 [00:41<13:22,  5.25s/it]
2024-07-26 08:49:23,998 - INFO: train loss: 1.14:   5%|4         | 8/161 [00:57<13:22,  5.25s/it]
2024-07-26 08:49:28,649 - INFO: train loss: 1.13:  10%|9         | 16/161 [01:01<08:46,  3.63s/it
[...]
2024-07-26 08:54:54,019 - INFO: train loss: 1.16:  84%|########4 | 136/161 [06:27<01:08,  2.75s/it]
2024-07-26 08:55:01,619 - INFO: train loss: 1.00:  89%|########9 | 144/161 [06:34<00:45,  2.69s/it]
2024-07-26 08:55:14,021 - INFO: train loss: 1.00:  89%|########9 | 144/161 [06:47<00:45,  2.69s/it]
2024-07-26 08:55:22,118 - INFO: train loss: 0.97:  94%|#########4| 152/161 [06:55<00:23,  2.65s/it]
2024-07-26 08:55:34,023 - INFO: train loss: 0.97:  94%|#########4| 152/161 [07:07<00:23,  2.65s/it]
2024-07-26 08:55:43,204 - INFO: train loss: 1.21:  99%|#########9| 160/161 [07:16<00:02,  2.65s/it]
2024-07-26 08:55:46,551 - INFO: Saving last model checkpoint to /home/pascal/h2o-llmstudio/output/user/ruby-walrus/
2024-07-26 08:55:54,024 - INFO: train loss: 1.15: 100%|##########| 161/161 [07:27<00:00,  2.65s/it]
[2024-07-26 08:56:53,065] [INFO] [torch_checkpoint_engine.py:33:commit] [Torch] Checkpoint global_step0 is ready now!
[2024-07-26 08:56:53,065] [INFO] [torch_checkpoint_engine.py:33:commit] [Torch] Checkpoint global_step0 is ready now!
[2024-07-26 08:56:53,066] [INFO] [torch_checkpoint_engine.py:33:commit] [Torch] Checkpoint global_step0 is ready now!
[2024-07-26 08:56:53,065] [INFO] [torch_checkpoint_engine.py:33:commit] [Torch] Checkpoint global_step0 is ready now!
[2024-07-26 08:56:53,066] [INFO] [torch_checkpoint_engine.py:33:commit] [Torch] Checkpoint global_step0 is ready now!
[2024-07-26 08:56:53,066] [INFO] [torch_checkpoint_engine.py:33:commit] [Torch] Checkpoint global_step0 is ready now!
[2024-07-26 08:56:53,066] [INFO] [torch_checkpoint_engine.py:33:commit] [Torch] Checkpoint global_step0 is ready now!
[2024-07-26 08:56:53,069] [INFO] [logging.py:96:log_dist] [Rank 0] [Torch] Checkpoint global_step0 is about to be saved!
[2024-07-26 08:56:53,069] [INFO] [engine.py:3591:save_16bit_model] Saving model weights to /home/pascal/h2o-llmstudio/output/user/ruby-walrus/checkpoi
nt.pth, tag: global_step0
[2024-07-26 08:56:53,070] [INFO] [torch_checkpoint_engine.py:21:save] [Torch] Saving /home/pascal/h2o-llmstudio/output/user/ruby-walrus/checkpoint.pth
...
[2024-07-26 08:59:22,348] [INFO] [torch_checkpoint_engine.py:23:save] [Torch] Saved /home/pascal/h2o-llmstudio/output/user/ruby-walrus/checkpoint.pth.
[2024-07-26 08:59:22,349] [INFO] [torch_checkpoint_engine.py:33:commit] [Torch] Checkpoint global_step0 is ready now!
[...]
2024-07-26 09:01:21,682 - INFO: Starting validation inference
2024-07-26 09:01:21,683 - INFO: validation progress:   0%|          | 0/9 [00:00<?, ?it/s]
2024-07-26 09:01:24,152 - INFO: validation progress:  11%|#1        | 1/9 [00:02<00:19,  2.47s/it]
2024-07-26 09:01:26,059 - INFO: validation progress:  22%|##2       | 2/9 [00:04<00:14,  2.14s/it]
2024-07-26 09:01:26,915 - INFO: validation progress:  33%|###3      | 3/9 [00:05<00:09,  1.55s/it]
2024-07-26 09:01:27,717 - INFO: validation progress:  44%|####4     | 4/9 [00:06<00:06,  1.26s/it]
2024-07-26 09:01:28,391 - INFO: validation progress:  56%|#####5    | 5/9 [00:06<00:04,  1.05s/it]
2024-07-26 09:01:29,159 - INFO: validation progress:  67%|######6   | 6/9 [00:07<00:02,  1.05it/s]
2024-07-26 09:01:29,803 - INFO: validation progress:  78%|#######7  | 7/9 [00:08<00:01,  1.17it/s]
2024-07-26 09:01:30,438 - INFO: validation progress:  89%|########8 | 8/9 [00:08<00:00,  1.28it/s]
2024-07-26 09:01:31,069 - INFO: validation progress: 100%|##########| 9/9 [00:09<00:00,  1.36it/s]
2024-07-26 09:01:31,077 - INFO: validation progress: 100%|##########| 9/9 [00:09<00:00,  1.04s/it]
2024-07-26 09:01:31,103 - INFO: Validation Perplexity: 18.37264
2024-07-26 09:01:31,103 - INFO: Mean validation loss: 1.09179
2024-07-26 09:01:34,473 - INFO: train loss: 1.15: 100%|##########| 161/161 [13:07<00:00,  4.89s/it]
[2024-07-26 09:01:38,110] [INFO] [launch.py:351:main] Process 1912825 exits successfully.
[2024-07-26 09:01:38,111] [INFO] [launch.py:351:main] Process 1912823 exits successfully.
[2024-07-26 09:01:38,111] [INFO] [launch.py:351:main] Process 1912821 exits successfully.
[2024-07-26 09:01:39,113] [INFO] [launch.py:351:main] Process 1912822 exits successfully.
[2024-07-26 09:01:39,113] [INFO] [launch.py:351:main] Process 1912824 exits successfully.
[2024-07-26 09:01:39,113] [INFO] [launch.py:351:main] Process 1912826 exits successfully.
[2024-07-26 09:01:39,114] [INFO] [launch.py:351:main] Process 1912827 exits successfully.
[2024-07-26 09:01:41,116] [INFO] [launch.py:351:main] Process 1912820 exits successfully.
[...]

Upload with cpu_shard
2024-07-26 09:07:20,750 - WARNING: PAD token id not matching between config and tokenizer. Overwriting with tokenizer id 128001.
2024-07-26 09:07:20,750 - INFO: Setting pretraining_tp of model config to 1.
2024-07-26 09:07:20,778 - INFO: Using bfloat16 for backbone
2024-07-26 09:36:05,021 - INFO: Attention implementation: sdpa
2024-07-26 09:36:05,026 - INFO: Lora module names: ['q_proj', 'k_proj', 'v_proj', 'o_proj', 'gate_proj', 'up_proj', 'down_proj']
2024-07-26 09:36:05,954 - INFO: Trainable parameters count: 51773440
2024-07-26 09:36:05,954 - INFO: Total parameters count:<PHONE_NUMBER>6
2024-07-26 09:36:05,955 - INFO: Trainable %: 0.0733%
2024-07-26 09:37:46,950 - INFO: Weights loaded from: /home/pascal/h2o-llmstudio/output/user/ruby-walrus/checkpoint.pth
2024-07-26 09:38:23,721 - INFO: Merging LORA layers with base model.
2024-07-26 09:38:24,035 - INFO: Enough space available for saving model weights.Required space: 138607.63MB, Available space: 3927968.00MB.
The token has not been saved to the git credentials helper. Pass `add_to_git_credential=True` in this function directly or `--add-to-git-credential` if using via `huggingface-cli` if you want to set the git credential as well.
Token is valid (permission: write).
Your token has been saved to /home/pascal/.cache/huggingface/token
Login successful
README.md: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 7.75k/7.75k [00:00<00:00, 22.1MB/s]
model-00014-of-00030.safetensors: 4.98GB [00:09, 522MB/s]
model-00025-of-00030.safetensors: 4.67GB [00:12, 382MB/s]
model-00024-of-00030.safetensors: 4.98GB [00:11, 449MB/s]
model-00006-of-00030.safetensors: 4.67GB [00:17, 273MB/s]
model-00029-of-00030.safetensors: 4.98GB [00:10, 478MB/s]
model-00023-of-00030.safetensors: 5.01GB [00:09, 534MB/s]
model-00020-of-00030.safetensors: 4.67GB [00:09, 509MB/s]
model-00026-of-00030.safetensors: 4.67GB [00:09, 497MB/s]
model-00013-of-00030.safetensors: 5.01GB [00:09, 511MB/s]
model-00027-of-00030.safetensors: 4.67GB [00:11, 392MB/s]
model-00009-of-00030.safetensors: 4.98GB [00:10, 467MB/s]                                                             | 10/30 [01:55<03:43, 11.17s/it]
model-00003-of-00030.safetensors: 5.01GB [00:09, 533MB/s]
model-00019-of-00030.safetensors: 4.98GB [00:09, 534MB/s]
model-00021-of-00030.safetensors: 4.67GB [00:09, 498MB/s]
model-00022-of-00030.safetensors: 4.67GB [00:09, 507MB/s]
model-00030-of-00030.safetensors: 2.11GB [00:04, 510MB/s]
model-00001-of-00030.safetensors: 4.59GB [00:09, 496MB/s]
model-00011-of-00030.safetensors: 4.67GB [00:09, 505MB/s]
model-00016-of-00030.safetensors: 4.67GB [00:09, 516MB/s]
model-00015-of-00030.safetensors: 4.67GB [00:09, 510MB/s]
model-00007-of-00030.safetensors: 4.67GB [00:10, 458MB/s]
model-00028-of-00030.safetensors: 5.01GB [00:09, 532MB/s]
model-00012-of-00030.safetensors: 4.67GB [00:09, 514MB/s]
model-00005-of-00030.safetensors: 4.67GB [00:09, 501MB/s]
model-00010-of-00030.safetensors: 4.67GB [00:11, 402MB/s]
model-00004-of-00030.safetensors: 4.98GB [00:09, 499MB/s]
model-00018-of-00030.safetensors: 5.01GB [00:09, 521MB/s]
model-00002-of-00030.safetensors: 4.67GB [00:09, 492MB/s]
model-00017-of-00030.safetensors: 4.67GB [00:12, 385MB/s]
model-00008-of-00030.safetensors: 5.01GB [00:10, 499MB/s]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 30/30 [06:50<00:00, 13.68s/it]

Memory allocation on the GPUs (yes, this indeed isn't freed but that is another issue https://github.com/h2oai/h2o-llmstudio/issues/736)
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.54.15              Driver Version: 550.54.15      CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA H100 80GB HBM3          Off |   00000000:18:00.0 Off |                    0 |
| N/A   33C    P0            143W /  700W |   17520MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA H100 80GB HBM3          Off |   00000000:2A:00.0 Off |                    0 |
| N/A   34C    P0            133W /  700W |   18969MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA H100 80GB HBM3          Off |   00000000:3A:00.0 Off |                    0 |
| N/A   35C    P0            133W /  700W |   18393MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   3  NVIDIA H100 80GB HBM3          Off |   00000000:5D:00.0 Off |                    0 |
| N/A   31C    P0            126W /  700W |   18841MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   4  NVIDIA H100 80GB HBM3          Off |   00000000:84:00.0 Off |                    0 |
| N/A   32C    P0            126W /  700W |   18841MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   5  NVIDIA H100 80GB HBM3          Off |   00000000:8B:00.0 Off |                    0 |
| N/A   34C    P0            131W /  700W |   18969MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   6  NVIDIA H100 80GB HBM3          Off |   00000000:91:00.0 Off |                    0 |
| N/A   35C    P0            134W /  700W |   18393MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   7  NVIDIA H100 80GB HBM3          Off |   00000000:E4:00.0 Off |                    0 |
| N/A   32C    P0            131W /  700W |   21741MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+

could some of these issues be related to this?
https://github.com/huggingface/transformers/pull/32214
maybe try updating transformer

It worked for me on current main/v1.9.0, so there seems to be at least one issue that isn't easily reproducible.

Hmm... as a sanity check I started a new instance and redid the dep install, trained again and got the same issue. I should note that I did make one change to requirements.txt to set transformers to the latest 4.43.3 version
transformers==4.43.3; python_full_version >= '3.8.0'
Full config file
architecture:
    backbone_dtype: bfloat16
    gradient_checkpointing: true
    intermediate_dropout: 0.0
    pretrained: true
    pretrained_weights: ''
augmentation:
    neftune_noise_alpha: 0.0
    random_parent_probability: 0.0
    skip_parent_probability: 0.0
    token_mask_probability: 0.0
dataset:
    add_eos_token_to_answer: true
    add_eos_token_to_prompt: true
    add_eos_token_to_system: true
    answer_column: answer
    chatbot_author: H2O.ai
    chatbot_name: h2oGPT
    data_sample: 1.0
    data_sample_choice:
    - Train
    - Validation
    limit_chained_samples: false
    mask_prompt_labels: true
    only_last_answer: false
    parent_id_column: None
    personalize: false
    prompt_column:
    - prompt
    prompt_column_separator: \n\n
    system_column: None
    text_answer_separator: ''
    text_prompt_start: ''
    text_system_start: <|system|>
    train_dataframe: /home/ubuntu/h2o-llmstudio/data/user/heavyiq_combo_v61_5_no_cte_judgements_3584_tokens_gen1/heavyiq_combo_v61_5_no_cte_judgements_3584_tokens_gen1_train.csv
    validation_dataframe: /home/ubuntu/h2o-llmstudio/data/user/heavyiq_combo_v61_5_no_cte_judgements_3584_tokens_gen1/heavyiq_combo_v61_5_no_cte_judgements_3584_tokens_gen1_eval.csv
    validation_size: 0.01
    validation_strategy: custom
environment:
    compile_model: false
    deepspeed_allgather_bucket_size: 1000000
    deepspeed_method: ZeRO3
    deepspeed_reduce_bucket_size: 1000000
    deepspeed_stage3_param_persistence_threshold: 1000000
    deepspeed_stage3_prefetch_bucket_size: 1000000
    find_unused_parameters: false
    gpus:
    - '0'
    - '1'
    - '2'
    - '3'
    - '4'
    - '5'
    - '6'
    - '7'
    huggingface_branch: main
    mixed_precision: false
    mixed_precision_dtype: bfloat16
    number_of_workers: 8
    seed: 2
    trust_remote_code: true
    use_deepspeed: true
experiment_name: heavyai-heavyiq-llama-3.1-70b-combo-v61-5-judge-3584-tokens-lora-r-512-a-1024-lr-1-1e-5.2.1
llm_backbone: meta-llama/Meta-Llama-3.1-70B
logging:
    logger: None
    neptune_project: ''
output_directory: /home/ubuntu/h2o-llmstudio/output/user/heavyai-heavyiq-llama-3.1-70b-combo-v61-5-judge-3584-tokens-lora-r-512-a-1024-lr-1-1e-5.2.1/
prediction:
    batch_size_inference: 0
    do_sample: false
    max_length_inference: 256
    max_time: 0.0
    metric: Perplexity
    metric_gpt_model: gpt-3.5-turbo-0301
    metric_gpt_template: general
    min_length_inference: 768
    num_beams: 1
    num_history: 4
    repetition_penalty: 1.0
    stop_tokens: ''
    temperature: 0.0
    top_k: 0
    top_p: 1.0
problem_type: text_causal_language_modeling
tokenizer:
    add_prompt_answer_tokens: false
    max_length: 4416
    padding_quantile: 1.0
    tokenizer_kwargs: '{"use_fast": true, "add_prefix_space": false}'
training:
    attention_implementation: auto
    batch_size: 1
    differential_learning_rate: 1.0e-05
    differential_learning_rate_layers: []
    drop_last_batch: true
    epochs: 1
    evaluate_before_training: false
    evaluation_epochs: 0.05
    freeze_layers: []
    grad_accumulation: 1
    gradient_clip: 0.0
    learning_rate: 1.2e-05
    lora: true
    lora_alpha: 1024
    lora_dropout: 0.05
    lora_r: 512
    lora_target_modules: ''
    lora_unfreeze_layers: []
    loss_function: TokenAveragedCrossEntropy
    optimizer: AdamW
    save_checkpoint: last
    schedule: Cosine
    train_validation_data: false
    use_dora: false
    warmup_epochs: 0.0

I will try training with a default dataset but not sure how that would make a difference.

Ok I trained with the default dataset but set lora_r: 4 and lora_alpha: 16 per the config shared by @pascal-pfeiffer. And indeed it succesfully merged the lora and is uploading.
This makes me think there's been some regression (assuming in the underlying peft library?) that is causing issues for large LoRA layers?
Here's my cfg
(base) ubuntu@164-152-107-167:~/h2o-llmstudio/output/user$ cat llama_3.1_70b_test/cfg.yaml
architecture:
    backbone_dtype: bfloat16
    gradient_checkpointing: true
    intermediate_dropout: 0.0
    pretrained: true
    pretrained_weights: ''
augmentation:
    neftune_noise_alpha: 0.0
    random_parent_probability: 0.0
    skip_parent_probability: 0.0
    token_mask_probability: 0.0
dataset:
    add_eos_token_to_answer: true
    add_eos_token_to_prompt: true
    add_eos_token_to_system: true
    answer_column: output
    chatbot_author: H2O.ai
    chatbot_name: h2oGPT
    data_sample: 0.05
    data_sample_choice:
    - Train
    - Validation
    limit_chained_samples: false
    mask_prompt_labels: true
    only_last_answer: false
    parent_id_column: None
    personalize: false
    prompt_column:
    - instruction
    prompt_column_separator: \n\n
    system_column: None
    text_answer_separator: <|answer|>
    text_prompt_start: <|prompt|>
    text_system_start: <|system|>
    train_dataframe: /home/ubuntu/h2o-llmstudio/data/user/oasst/train_full.pq
    validation_dataframe: None
    validation_size: 0.02
    validation_strategy: automatic
environment:
    compile_model: false
    deepspeed_allgather_bucket_size: 1000000
    deepspeed_method: ZeRO3
    deepspeed_reduce_bucket_size: 1000000
    deepspeed_stage3_param_persistence_threshold: 1000000
    deepspeed_stage3_prefetch_bucket_size: 1000000
    find_unused_parameters: false
    gpus:
    - '0'
    - '1'
    - '2'
    - '3'
    - '4'
    - '5'
    - '6'
    - '7'
    huggingface_branch: main
    mixed_precision: false
    mixed_precision_dtype: bfloat16
    number_of_workers: 8
    seed: 2
    trust_remote_code: true
    use_deepspeed: true
experiment_name: llama_3.1_70b_test
llm_backbone: meta-llama/Meta-Llama-3.1-70B
logging:
    logger: None
    neptune_project: ''
output_directory: /home/ubuntu/h2o-llmstudio/output/user/llama_3.1_70b_test/
prediction:
    batch_size_inference: 0
    do_sample: false
    max_length_inference: 4096
    max_time: 0.0
    metric: Perplexity
    metric_gpt_model: gpt-3.5-turbo-0301
    metric_gpt_template: general
    min_length_inference: 2
    num_beams: 1
    num_history: 4
    repetition_penalty: 1.0
    stop_tokens: ''
    temperature: 0.0
    top_k: 0
    top_p: 1.0
problem_type: text_causal_language_modeling
tokenizer:
    add_prompt_answer_tokens: false
    max_length: 4864
    padding_quantile: 1.0
    tokenizer_kwargs: '{"use_fast": true, "add_prefix_space": false}'
training:
    attention_implementation: auto
    batch_size: 1
    differential_learning_rate: 1.0e-05
    differential_learning_rate_layers: []
    drop_last_batch: true
    epochs: 1
    evaluate_before_training: false
    evaluation_epochs: 1.0
    freeze_layers: []
    grad_accumulation: 1
    gradient_clip: 0.0
    learning_rate: 1.2e-05
    lora: true
    lora_alpha: 16
    lora_dropout: 0.05
    lora_r: 4
    lora_target_modules: ''
    lora_unfreeze_layers: []
    loss_function: TokenAveragedCrossEntropy
    optimizer: AdamW
    save_checkpoint: last
    schedule: Cosine
    train_validation_data: false
    use_dora: false
    warmup_epochs: 0.0
    weight_decay: 0.0

Would you guys be able to try a bigger lora (i.e. rank 512 alpha 1024) as I did to see if you can repro? I'll try some sizes between 4/16 and 512/1024 to see if I can find the breaking point.

Yes, I am starting up the 512/1024 test right now. That could indeed be an issue then. Also, why I was asking for lora settings earlier, as default settings seemed to work fine.
So, seems that very large LoRA layers are split across GPUs, while smaller ones are on a single GPU and the deepspeed wrapper isn't gathering them on a single (meta) device.
Will see how we can deal with that and if there are any workarounds such as CPU only merge.

Thanks @pascal-pfeiffer... just should note I've been training and uploading r512/a1024 models (llama 3 70b) for some months, so seems there was a recent change that has caused the issues.

Also I tried a CPU-only merge and gave up after nearly 24 hours of waiting.

Ok, to follow up on this, altering my training config from above (https://github.com/h2oai/h2o-llmstudio/issues/782#issuecomment-2258743302) to use LoRA Rank 256 and Alpha 512 worked, but when I changed it to Rank 512 and Alpha 1024 I got the failure seen befoer
  File "/home/ubuntu/h2o-llmstudio/./llm_studio/app_utils/handlers.py", line 358, in handle
    await experiment_push_to_huggingface_dialog(q)
  File "/home/ubuntu/h2o-llmstudio/./llm_studio/app_utils/sections/experiment.py", line 2015, in experiment_push_to_huggingface_dialog
    publish_model_to_hugging_face(
  File "/home/ubuntu/h2o-llmstudio/./llm_studio/app_utils/hugging_face_utils.py", line 216, in publish_model_to_hugging_face
    cfg, model, tokenizer = load_cfg_model_tokenizer(
  File "/home/ubuntu/h2o-llmstudio/./llm_studio/app_utils/sections/chat.py", line 241, in load_cfg_model_tokenizer
    model.backbone = model.backbone.merge_and_unload()
  File "/home/ubuntu/miniconda3/envs/h2o_llm_studio/lib/python3.10/site-packages/peft/tuners/lora/model.py", line 838, in merge_and_unload
    return self._unload_and_optionally_merge(
  File "/home/ubuntu/miniconda3/envs/h2o_llm_studio/lib/python3.10/site-packages/peft/tuners/lora/model.py", line 457, in _unload_and_optionally_merge
    target.merge(safe_merge=safe_merge, adapter_names=adapter_names)
  File "/home/ubuntu/miniconda3/envs/h2o_llm_studio/lib/python3.10/site-packages/peft/tuners/lora/layer.py", line 470, in merge
    delta_weight = self.get_delta_weight(active_adapter)
  File "/home/ubuntu/miniconda3/envs/h2o_llm_studio/lib/python3.10/site-packages/peft/tuners/lora/layer.py", line 533, in get_delta_weight
    output_tensor = transpose(weight_B @ weight_A, self.fan_in_fan_out) * self.scaling[adapter]
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0! (when checking argument for argument mat2 in method wrapper_CUDA_mm)

Interesting, I used your config with 512/1024 and was able to merge and upload. Though, a bit different GPUs, so maybe it was down to luck there.

@pascal-pfeiffer would you be able to list all the versions of packages in your environment?

I checked out this commit (87c2978698545c758b639fb83e0ceef7e43e91e5) when testing and installed a fresh environment.
So, https://github.com/h2oai/h2o-llmstudio/blob/87c2978698545c758b639fb83e0ceef7e43e91e5/requirements.txt
Given that this is dependent on the size of LoRA, I have the strong feeling this can be very hardware dependent.
By chance, what is the disk space left on your primary disk? I noticed that the export uses always the primary disk for an intermediate saving, which is ~170GB for this model. Could be that this also affects somehow the sharding, as you also saw unusual distribution across the 8 GPUs.
With slightly different config, I was again able to export and upload. So, hard to replicate for me now.
Latest, I did update
transformers = "==4.43.3"
accelerate = "==0.33.0"
hf-transfer = "==0.1.8"
peft = "==0.12.0"

and export was fine again.
(though without HF_Transfer, the upload often fails, as you reported earlier) And setting it as an env var is required (#801).
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.54.15              Driver Version: 550.54.15      CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA H100 80GB HBM3          Off |   00000000:18:00.0 Off |                    0 |
| N/A   33C    P0            144W /  700W |   20997MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA H100 80GB HBM3          Off |   00000000:2A:00.0 Off |                    0 |
| N/A   35C    P0            135W /  700W |   22509MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA H100 80GB HBM3          Off |   00000000:3A:00.0 Off |                    0 |
| N/A   36C    P0            133W /  700W |   22509MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   3  NVIDIA H100 80GB HBM3          Off |   00000000:5D:00.0 Off |                    0 |
| N/A   32C    P0            127W /  700W |   22509MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   4  NVIDIA H100 80GB HBM3          Off |   00000000:84:00.0 Off |                    0 |
| N/A   32C    P0            128W /  700W |   22889MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   5  NVIDIA H100 80GB HBM3          Off |   00000000:8B:00.0 Off |                    0 |
| N/A   35C    P0            132W /  700W |   22509MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   6  NVIDIA H100 80GB HBM3          Off |   00000000:91:00.0 Off |                    0 |
| N/A   36C    P0            138W /  700W |   22509MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   7  NVIDIA H100 80GB HBM3          Off |   00000000:E4:00.0 Off |                    0 |
| N/A   33C    P0            134W /  700W |   24513MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+

I'll do some more testing with even more extreme LoRA dimensions.

I'll do some more testing with even more extreme LoRA dimensions.

1024 LoRA Rank also worked fine. Now thinking it might be something else.
2024-08-01 23:27:36,341 - INFO: Trainable parameters count:<PHONE_NUMBER>0
2024-08-01 23:27:36,342 - INFO: Total parameters count:<PHONE_NUMBER>6
2024-08-01 23:27:36,342 - INFO: Trainable %: 15.8148%
[...]
100%|███████████████████████████████████████████████████████████████████████████| 30/30 [05:20<00:00, 10

Though, that was again with the updated dependencies
transformers = "==4.43.3"
accelerate = "==0.33.0"
hf-transfer = "==0.1.8"
peft = "==0.12.0"

For 100 % reproduceability, I am on 6755a5866b598ae1832bd6e75b18e207a889bc52 (current main) and updated the dependencies as above. Attached, I have the Pipfile.lock and my train config
experiment_llama31.zip

Just to follow up on this, as a workaround I was able to start LLM Studio with 4 GPUs via the CUDA_VISIBLE_DEVICES environment variable, and it worked fine. Still don't know why it was/is still failing with 8 GPUs, but at least I was able to export my model.


Add playsInline attribute to HTMLAttributes

It is used by the <video> element.
From https://developer.mozilla.org/en-US/docs/Web/HTML/Element/video

playsinline: A Boolean attribute which indicates that the video is to be played "inline", that is within the element's playback area. Note that the absence of this attribute does not imply that the video will always be played in fullscreen.

Coverage remained the same at 100.0% when pulling 665dc1478297a93a2bd84c7a6187c184a2be4ed4 on andybons:patch-1 into 8be9c5f70ba8453fb3c262f38bbc852ee59afb64 on developit:master.

You're very welcome. Thank you for the quick merge!


Simplify access to the ClaimsPrincipal

Is your feature request related to a problem? Please describe.
Currently the authenticated user has to be accessed through state, which is annoying to type out and doesn't feel as integrated:
// Annotation-based
void Method([GlobalState(nameof(ClaimsPrincipal))] ClaimsPrincipal claims);

// Code-first
var claims = context.GetGlobalValue<ClaimsPrincipal>(nameof(ClaimsPrincipal))

You could also go through the IHttpContextAccessor, but then it is even more unnecessary code.
Describe the solution you'd like
Annotation-based: The Authorization package could introduce a new attribute that abstracts the state call, something like:
void Method([Claims] ClaimsPrincipal claims);

Code-first: The Authorization package could introduce a new extension method that abstracts the state call, something like:
context.Claims();

We need to decide on a good name for these APIs. I also considered User, since that's how MVC makes the ClaimsPrincipal available on the HttpContext.
Describe alternatives you've considered
For the Annotation-based approach we could also just allow for a ClaimsPrinicipal to be injected without the use of an attribute.
void Method(ClaimsPrincipal claims);

I would prefer the last option here, personally for annotation based code.
I don't think you'll realistically ever see someone passing around a ClaimsPrincipal to an actual query/mutation from the outside world.
To add to the above, I also think CancellationToken should be treated the same way as well.
async Task SomeQuery([GlobalState(nameof(ClaimsPrincipal))] ClaimsPrincipal user, [GlobalState(nameof(CancellationToken))] CancellationToken cancellation)

vs (using custom attributes)
async Task SomeQuery([ClaimsPrincipal] ClaimsPrincipal user, [CancellationToken] CancellationToken cancellation)

vs
async Task SomeQuery(ClaimsPrincipal user, CancellationToken cancellation)

Considering how often you (should) use the CancellationToken, I feel like the API to use it from HC is overbearing, and quite honestly makes code hard to read.

@ApocDev Yes, I also like the last approach the most.
About the CancellationToken, it should already be injected as expected from you or atleast I remember doing it...

@ApocDev Yes, I also like the last approach the most.
About the CancellationToken, it should already be injected as expected from you or atleast I remember doing it...

I'm honestly not sure if it is. The last time I was fighting with it, I had to go the attribute route, so now it's ingrained in my muscle memory. :)

@ApocDev  CancellationToken should already work like this :)

CancellationToken has always worked :)
In the same vain ClaimsPrincipal is now working. There is now attribute needed.
public Task<string> MyResolver(ClaimsPrincipal foo)

When using nullable annotations the ClaimsPrincipal will be injected with null if not available; otherwise we will cause a resolver error.
This will allow to get null whenever there is no claims principal:
public Task<string> MyResolver(ClaimsPrincipal? foo)

For the fluent api we are now mimicking the HttpContext and have en extension method GetUser.
services
    .AddGraphQL()
    .AddQueryType(d =>
    {
        d.Name("Query");
        d.Field("foo").Resolve(ctx => ctx.GetUser()?.Identity?.Name);
    })


Ignore FunctionOnlyReturningConstant for Dagger provides methods

It's common when using Dagger to have methods that are annotated with dagger.Provides to only return some hardcoded data. Such as:
@Provides
@ApplicationScope
@Named("deviceId")
fun provideDeviceId(): String {
  return "fake-device-id"
}

Currently this gets flagged by FunctionOnlyReturningConstant. I think it's best to provide some custom annoations similiar to #377 and default to dagger.Provides.

In my opinion this is good go. Please provide a list of annotations which should be excluded.

So far I've only seen dagger.Provides


Adding info on sections property in card object

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs/actionable-messages/card-reference.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Making OrbitService compile without OpenGL

That's a somehow hacky commit and just intended as a temporary solution.
In parallel to OrbitGl, this PR introduces another cmake target, called OrbitNoGl.
OrbitNoGl derives from OrbitGl in that sense, that it compiles the same cpp-files
(except some pure UI related cpp files) but with an additional preprocessor
definition called NOGL.
Additionally I added ifdef-guards to all calls into opengl-related code. These
preprocessor checks enable the calls in the absence of NOGL.
I tested OrbitService on a Stadia instance with Unreal's Infiltrator Demo:

Connecting to the instance works
Sampling works
Hooking functions works

I haven't done any further tests.

Thanks, overall looks good to me and thanks for jumping on this quickly!
Could you add a comment what kind of testing you have done? And since this is temporary, could you add an issue to make sure cleaning this up is tracked?

I added a comment to the CMakeLists.txt file, where this new target is defined.
I had the feeling, this is where all the strings come together and is probably the
best place to put such a comment.


refactor: abstract away language independent ui into separate component

Also fixes minor issues -

remove unnecessary re-renders on code change
fix: Do not render <p> inside another <p>

This pull request is being automatically deployed with Vercel (learn more). To see the status of your deployment, click below or on the icon next to each commit.
🔍 Inspect: https://vercel.com/sliphq/run-wasm/97Gvf6fJbEdPcbiUW8BhhyJz336v ✅ Preview: Failed

Not sure about the specific build commands used here, but I have deployed this in my own vercel account here - https://run-wasm-five.vercel.app/


Cms browser add string button

After clicking the Add button, no window opens. Currently I do not see what is wrong. when I look at the code in the core I do not think I need to add anything else.

After clicking the Add button, no window opens. Currently I do not see what is wrong. when I look at the code in the core I do not think I need to add anything else.

sorry, since it's pure UI/SWT code not accessible anywhere else I couldn't write a test for it either. So quit possible that it doesn't really work.  ... we could debug it together, once you have a client calling that stuff :)


Change fee dropdown breaks transaction request button

I noticed this behaviour earlier. Some error is thrown re BigNumber, in turn breaking the transaction request form (doesn't happen with send form)
https://user-images.githubusercontent.com/1618764/177587958-e560edff-a3d6-45e7-95f5-f1c0d142074f.mp4

I can look at this today.


CASMINST-4963 CAN to CHN

Description

Checklist Before Merging

[x] If I added any command snippets, the steps they belong to follow the prompt conventions (see example).
[x] If I added a new directory, I also updated .github/CODEOWNERS with the corresponding team in Cray-HPE.
[x] My commits or Pull-Request Title contain my JIRA information, or I don't have a JIRA.

I rebased, squashed the commits, and did minor linting.

Probably should get @trad511 to re-review to make sure he's satisfied that his concerns were addressed.


Support NodeViewWrapper, NodeViewContent, nodeViewProps

Hello,
In this guide at the Tiptap documentation they show how to use content editable in a custom view node. But these files seems to be missing in this library.
These are the files that Vue uses:
import { NodeViewWrapper, NodeViewContent, nodeViewProps } from '@tiptap/vue-2'
This is the documentation page:
https://www.tiptap.dev/guide/node-views/vue
Is there already supported in Ngx-tiptap?
Thanks!

Were you able to find it?

Yes, your example worked! I am just trying to figure out how not to create a new node when hitting enter now :)
Thanks!

Cool. I will leave this issue open until I add the doc.

Added to README. You can now use the tiptapNodeViewContent directive instead of the attribute.

Hi @sibiraj-s I could'nt able to find NodeViewWrapper. Sorry I am not able to fully understand the tiptap's vue part of the code


DynamoDB put_item with binary data

I can't work out how to use ExAws.Dynamo.put_item to save binary data to a DynamoDB table.
If I provide a bit string to put_item I receive a Poison encode error. If I base 64 encode the data then the data is saved to DynamoDB as a string.
binary_data = <<some binary data>>
ExAws.Dynamo.put_item("MyTable", %{my_field: binary_data})

** (FunctionClauseError) no function clause matching in Poison.Encoder.BitString.chunk_size/3

ExAws.Dynamo.put_item("MyTable", %{my_field: Base.encode64(binary_data)})
{:ok, %{}} # But my_field contains a string, not a binary.

Is there a way I can explicitly define the data types of the map I pass to put_item?

Hey there!
I'll respond with greater length tomorrow, but this is the purpose of the Encodable protocol. As I'm sure you're aware, the actual data sent to dynamo uses a bunch of type tags that look like https://github.com/CargoSense/ex_aws/blob/master/test/lib/ex_aws/dynamo/encoder_test.exs#L11
If you want to provide custom encoding information for a given type, you should create a struct for that record and implement the encodable protocol for it. I will create a relevant example probably tomorrow, out and about at the moment sorry!

Thanks @benwilson512, I figured it out.
You even mention it in your (docs)[https://hexdocs.pm/ex_aws/ExAws.Dynamo.Client.html] so I should really have a spent a bit more time experimenting.
But a question: looking at (dynamo/encodable.ex) [https://github.com/CargoSense/ex_aws/blob/master/lib/ex_aws/dynamo/encodable.ex] I see that the default encoder implementation for bitstrings is to send them to DynamoDB as a string data type.
Would it not make more sense to send them as a binary data type?
As I see it (and please correct me if I'm wrong) there's currently no way using the default encodings to send a binary data to DynamoDB.

Glad you figured it out.
The reason that the default for bitstrings is to send as a string type is that "hello" in Elixir is a bitstring. Now, I could try to case off of String.valid? or something to choose "S" or "B" but this will incur a cost as String.valid? has to walk the whole string linearly.
You do have another option I believe. The encode encode function is idempotent, so if you provide an already tagged value it will not be re-tagged as something else. Thus if instead of ExAws.Dynamo.put_item("MyTable", %{my_field: binary_data}) you did ExAws.Dynamo.put_item("MyTable", %{my_field: %{"B" => binary_data}}) it will use the provided type tag. I should write a test to that effect.

Yeah, I noticed that you allow for tagged values. Nice.
I'm happy with using a struct and implementing the encodable protocol.
Thanks for your help and nice job on the library.

Thanks!


Added language to download json files and not import from url

Changed the language to download file and import file, rather than import from URL.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

wrong repo. :)

wrong repo. :)


microsoft.workloadiam: Refactor subcommand invocation

The old code was unreliable. The new code should work in a similar way but it should be able to handle errors in a more graceful way.
For example, this is the output in a successful run:

And this is the output in a non-successful run when the user has already used that local authority before and the workload-iam extension fails to create a join token:

It also adds a check to ensure that the instance name isn't too long. The instance name is used to form labels for Kubernetes resources, and their labels are limited to 63 characters. If the instance name is too long, the labels will go over that limit.
https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#syntax-and-character-set

This checklist is used to make sure that common guidelines for a pull request are followed.
Related command

General Guidelines

[x] Have you run azdev style <YOUR_EXT> locally? (pip install azdev required)
[x] Have you run python scripts/ci/test_index.py -q locally? (pip install wheel==0.30.0 required)

For new extensions:

[ ] My extension description/summary conforms to the Extension Summary Guidelines.

About Extension Publish
There is a pipeline to automatically build, upload and publish extension wheels.
Once your pull request is merged into main branch, a new pull request will be created to update src/index.json automatically.
You only need to update the version information in file setup.py and historical information in file HISTORY.rst in your PR but do not modify src/index.json.

Is the 20 characters limit on the name documented anywhere? Is there a TSG that could be appended to the error if so?
LGTM other than the above comment.

Is the 20 characters limit on the name documented anywhere? Is there a TSG that could be appended to the error if so? LGTM other than the above comment.

We have a TSG that is in the extension metadata, I will add a comment there.


refactor: drop Table trait

I hereby agree to the terms of the GreptimeDB CLA.
Refer to a related PR or issue link (optional)
What's changed and what's your intention?
This refers to #2065.
Checklist

[ ]  I have written the necessary rustdoc comments.
[ ]  I have added the necessary unit tests and integration tests.
[x]  This PR does not require documentation updates.

All handled.


Possible CSRF detected - state parameter was required but no state could be found

Summary
I have a typical Spring Boot OAuth2 SSO Application integrated with external IdP.  I am running this application as docker container in ECS cluster.  It works as expected whenever I run single instance of this application but fails whenever I run more than one instance.
Actual Behavior
It fails with message "Possible CSRF detected - state parameter was required but no state could be found" whenever tried to run with more than one instance.
Looks like problem as described in 822
Expected Behavior
Success.
Configuration
Spring SSO Configuration -
security:     oauth2:     client:       client-id: appid       client-secret: seceret       access-token-uri:  idpaccesstokenuri       user-authorization-uri: authuri       scope: openid     resource:       user-info-uri: userinfouri       filter-order: 3   basic:     enabled: false
Version
Spring-boot: 2.0.6.RELEASE
spring-security-oauth2: 2.2.3.RELEASE
Sample
@EnableOAuth2Sso @SpringBootApplication @EnableAutoConfiguration public class SSOServiceApplication {}

I've notice the same error using keycloak as authorization server. In my case keycloak and spring application works on the same host but on different ports. Simple scenario is like follows:

user logins in into application in the first tab
user open the same page on the second tab
on the first tab user clicks logout and is redirected back to keycloak. On the keycloak side session is destroyed
users clicks logout on the second tab and is redirected to login page on keyloack
users go back to first tab and provides username and password
application fails with message "Possible CSRF detected - state parameter was required but no state could be found"

Interesting thing is if user in set 5 uses second tab there is no error.
org.springframework.security.oauth.boot:spring-security-oauth2-autoconfigure:2.1.5.RELEASE
org.springframework.boot:spring-boot:2.1.5.RELEASE
keycloak:8.0.0

@singharunkumar83 or @TomaszBelina Please provide a minimal sample (via GitHub repo) that reproduces the issue and I'll take a look.

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

Closing due to lack of requested feedback. If you would like us to look at this issue, please provide the requested information and we will re-open the issue.


Please Add King Edward VI Grammar School (KEGS)

this is a secondary school in the UK that reaches sixth form. It is located in Chelmsford.

@rudrrayan Pull request merged. Thank you!


feat: add subFolders option for generate

Hi there,
First of all, thank you for this awesome project & all the effort you are putting together to make Nuxt.js awesome!
I faced a little issue using nuxt generate as it creates a directory for every route & write an index.html inside it.
index.html
/about/index.html
/contact/index.html

However, most CDN (& or static hosting) do URL normalization to enforce consistent URL patterns (e.g Netlify).
Quoted from Netlify documentation:
When “Pretty URLs” is enabled under processing settings for your site, Netlify will enforce consistent URL patterns.

A link to /about.html will be rewritten to /about in your HTML files and Netlify will enforce a redirect from /about/ to /about

A link to /about/index.html will be rewritten to /about/ and Netlify will redirect from /about to /about/

With that said, the current generate does not suit the following scenario:
Let's say I have the following routes:
- /
- /about
- /contact

My webapp loads on / (e.g: https://example.com)
I navigate to a route using CSR (e.g https://example.com/about)
I refresh & been redirected to https://example.com/about/

Which means, if Google crawls /about, it will face a 301 redirection to /about/ (using Netlify for example).
Adding the subFolders option to generate fixes this problem.
Any thoughts/restrictions?
Fixes

[x] #721

Todo

[x] Tests
[x] Add documentation under Configuration / Generate

Codecov Report

Merging #2032 into dev will decrease coverage by 0.09%.
The diff coverage is 75%.

@@            Coverage Diff            @@
##              dev    #2032     +/-   ##
=========================================
- Coverage   96.21%   96.12%   -0.1%
=========================================
  Files          14       14
  Lines         952      954      +2
=========================================
+ Hits          916      917      +1
- Misses         36       37      +1

Impacted Files
Coverage Δ

lib/common/options.js
89.47% <ø> (ø)
:arrow_up:

lib/builder/generator.js
98.79% <75%> (-1.21%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e97bb1d...59ad478. Read the comment docs.

Hi @Gomah
I talked with the team and we agree with this suggestion, could you add tests for it so we can merge it afterwards?

Hi @Atinux, cool!
Just wondering what's the best approach to test this, as basic.generate.test.js generates the files by default before running the tests suite.
Should I just add the tests after closing the server or is there a different method?
Also, what's the best way to test it?
a) Generate & check if the routes rendered in fixtures/basic/dist are a folder or a file by checking the type with fs.
b) Generate & check if the routes rendered path based on the subFolders option.
c) A better option? 😄

As you will need to generate the files twice, you either have to add another test or re-generate within the same basic test. The first speaks for it self, for the later you could take a look how I approached that here: https://github.com/pimlie/nuxt-generate-cluster/blob/next/test/single.regenerate.test.js
So after the normal tests are done I first remove the full dist/users/ folder and check if they really have been removed (eg the routes should give a 404). After that I run the generate command again but this time I dont build Nuxt but only initiate. After that you could check either check the fs but inline with the current test implementation I would test if you can retrieve eg /user/1.html from the server.
For your test you would also need to change the nuxt config of course before generating the second time.

Ok, just got time to look at it!
@pimlie, Yeah, I'm trying to re-generate within the same basic test specs with a different configuration ;)
However, adding subFolders: false throws an error while trying to render the component in lib/app/components/nuxt-child.js.
TypeError: Cannot read property 'components' of undefined

# nuxt-child.js
# parent.$route
{ name: null,
  meta: {},
  path: '/stateless.html',
  hash: '',
  query: {},
  params: {},
  fullPath: '/stateless.html',
  matched: [] }

The routerViewKey computed function in /lib/app/components/nuxt.js actually return an invalid route:
/routename.html

instead of
/routename

I'll try to get some time to deeply look at it this week-end.
@Atinux, @pimlie, any ideas where I should look at?

Hi @Gomah
For TypeError: Cannot read property 'components' of undefined, I made a fix to nuxt.js core (https://github.com/nuxt/nuxt.js/commit/4b79c38f41786566f86b993ef2ae3f5151c6a9c0).
Actually I remember why it's not possible. Most CDN cannot serve /routename to routename.html but instead has to be /routename -> /routename/index.html.
Also, if you go to /routename.html, Vue does not know this route (since it has an extension in its name) and cannot load the app.
Actually, the right solution is to use canonical urls in your header: https://support.google.com/webmasters/answer/139066?hl=en
Example of your nuxt.config.js:
module.exports = {
  head: function () {
    const path = this.$route.path.replace(/\/$/, '') // Remove trailing /
    return {
      link: [{ rel: 'canonical', href: `https://my-website.com${path}` }]
    }
  }
}

I also made a fix so you can use head() {...} in the next release.
I believe we could add it to the documentation.

Thanks for the fix!
Are we still considering this option? Even if the canonical url is set, I believe it's still crucial as Google is crawling 301s for those routes.
I just got the test working actually, I added the extensions option to the serveStatic function to set an extension fallback so the local server mounted during the tests behave like Netlify servers (or any other server using html fallbacks).
I've added test/basic.generate.nosubfolders.test.js, let me know if there's a better way to structure this test.

I believe this option can be used with Netlify or the extensions set for serve-static.
I can merge it, but could you make the changes for the documentation: https://github.com/nuxt/docs/blob/master/en/api/configuration-generate.md

Awesome :raised_hands:

I was looking at the tests but I dont think they are specific enough. E.g. there is currently no check if subfolders: true really generates users/1/index.html and subfolders: false generates users/1.html as serveStatic will always magically translate the users/1 url to the correct one. Currently if someone changes the following line https://github.com/nuxt/nuxt.js/blob/dev/lib/builder/generator.js#L185 to read if (true) { the nosubfolder tests will still pass because of this.
As I mentioned in my comment above, I think you should make sure that in basic.generate.js you test one of the users as users/1/index.html and in the nosubfolders test you test one of the users as users/1.html.
But due to the fact that serveStatic can be an issue in the test itself, I actually think it would be best to also add a fs check. For the nosubfolders test it would look like this:
import { existsSync } from 'fs'
...
test('/users/1.html', async t => {
  const html = await rp(url('/users/1.html'))
  t.true(html.includes('
User: 1
'))
  t.true(existsSync(resolve(__dirname, 'fixtures/basic/dist', 'users/1.html')))
  t.false(existsSync(resolve(__dirname, 'fixtures/basic/dist', 'users/1/index.html')))
})

You never cant have enough tests ;)

@pimlie True, I was focused on rendering the component properly with serveStatic that I forgot my initials thoughts!
Just added your suggestion #2132 👍

Hi @Gomah , I tried the your subfolders branch but when we try to generate, the mounted method is not called in my case but is being called correctly in rc11 version

@piyushchauhan2011 Weird, all my mounted hooks are working using nuxt@dev.
Do you have an example?

@piyushchauhan2011 @Gomah
Same problem occurred for me, but I founded the two cases.

When I use static routing like below, actually the generated hoge.html doesn't call mounted method.

.
├── pages
│     ├── index.vue
│     └── hoge.vue

but when I use dynamic routing, the generated hoge.html also call mounted method correctly.

.
├── pages
│     └── _id.vue

  generate: {
    routes: [
      'hoge'
    ]
  }

I'm on Nuxt 1.4, and experiencing the same issue as @gonshi with subFolders:false. App looks great in dev or when generated with subFolders:true.
Would be great to see this working as it'd resolve some SEO  and URL canonicalization issues for non-dynamic routes very nicely.


isbat university kampala (ISBAT)

This is the international School of Business and Technology found in kampala. the website is isbatuniversity.ac.ug

@owinoalfred Thank you for your patience while waiting for a review and processing your request. Pull request merged.


Add IEditorOption copyWithSyntaxHighlighting

Please add this editor setting
Monaco.d.ts
in this class

Thanks @Swiftpaws for the suggestion.  It may have been a new setting I've overlooked.  I need to find a solution to better generate some/all of the C# from the TypeScript definitions.  There's a general discussion about that in the #26 PR from @hansmbakker.  As you can imagine writing the wrapper by hand isn't ideal.
I'll make sure to do a pass before the next release to get this one in though (and any other straight-forward ones missed).  I'm not sure when the next release/update will be at this time yet.


Remove nan and infinites from the package

Closes: #105

@Graur am I right that updating eo-maven-plugin dependency to 0.29.0 that contains nan and +- infs is not enough? We should add these objects to objectionary/home before?

@maxonfjvipon Yes, you are right. I will try to create new pull request in home

@Graur please have a look

@yegor256 please have a look

@rultor merge

@rultor merge

@yegor256 OK, I'll try to merge now. You can check the progress of the merge here

@rultor merge

@maxonfjvipon @yegor256 Oops, I failed. You can see the full log here (spent 2min)
+ shopt -s expand_aliases
+ alias 'sudo=sudo -i'
+ export HOME=/home/r
+ HOME=/home/r
+ cd /home/r/repo
+ export head=git@github.com:objectionary/eo-math.git
+ head=git@github.com:objectionary/eo-math.git
+ export pull_id=108
+ pull_id=108
+ export fork=git@github.com:maxonfjvipon/eo-math.git
+ fork=git@github.com:maxonfjvipon/eo-math.git
+ export fork_branch=remove-nan-and-infinites
+ fork_branch=remove-nan-and-infinites
+ export head_branch=master
+ head_branch=master
+ export 'pull_title=Remove `nan` and infinites from the package'
+ pull_title='Remove `nan` and infinites from the package'
+ export author=yegor256
+ author=yegor256
+ pdd --file=/dev/null
Found 6 lines in /home/r/repo/.pdd
My version is 0.20.8
Ruby version is 2.7.0 at x86_64-linux
Reading from root dir .
\u001b[38;5;214mExcluding\u001b[0m target/**
Reading renovate.json ...
Reading LICENSE.txt ...
Reading .github/workflows/pdd.yml ...
Reading .github/workflows/xcop.yml ...
Reading .github/workflows/mvn.yml ...
Reading .github/workflows/codecov.yml ...
Reading .pdd ...
Reading .0pdd.yml ...
Reading src/test/java/EOorg/EOeolang/EOmath/EOangleEOsinTest.java ...
Reading src/test/java/EOorg/EOeolang/EOmath/package-info.java ...
Reading src/test/java/EOorg/EOeolang/EOmath/EOangleEOcosTest.java ...
Reading src/test/eo/org/eolang/math/angle-tests.eo ...
Reading src/test/eo/org/eolang/math/e-tests.eo ...
Reading src/test/eo/org/eolang/math/integral-tests.eo ...
Puzzle 99-0a3cbf11 30/DEV at src/test/eo/org/eolang/math/integral-tests.eo
Puzzle 93-b28a2be4 30/DEV at src/test/eo/org/eolang/math/integral-tests.eo
Reading src/test/eo/org/eolang/math/random-tests.eo ...
Reading src/test/eo/org/eolang/math/series-tests.eo ...
Reading src/test/eo/org/eolang/math/number-tests.eo ...
Puzzle 99-2033025f 30/DEV at src/test/eo/org/eolang/math/number-tests.eo
Reading src/test/eo/org/eolang/math/pi-tests.eo ...
Reading src/main/java/EOorg/EOeolang/EOmath/EOangle$EOcos.java ...
Reading src/main/java/EOorg/EOeolang/EOmath/EOnumber$EOas_int.java ...
Reading src/main/java/EOorg/EOeolang/EOmath/EOnumber$EOas_string.java ...
Reading src/main/java/EOorg/EOeolang/EOmath/EOnumber$EOas_float.java ...
Reading src/main/java/EOorg/EOeolang/EOmath/package-info.java ...
Reading src/main/java/EOorg/EOeolang/EOmath/EOnumber$EOpow.java ...
Reading src/main/java/EOorg/EOeolang/EOmath/EOangle$EOsin.java ...
Reading src/main/eo/org/eolang/math/e.eo ...
Reading src/main/eo/org/eolang/math/integral.eo ...
\u001b[31mERROR\u001b[0m: src/main/eo/org/eolang/math/integral.eo; src/main/eo/org/eolang/math/integral.eo:29 Space expected at 32:2; make sure all lines in the puzzle body have a single leading space.
If you can't understand the cause of this issue or you don't know how to fix it, please submit a GitHub issue, we will try to help you: https://github.com/cqfn/pdd/issues. This tool is still in its beta version and we will appreciate your feedback. Here is where you can find more documentation: https://github.com/cqfn/pdd/blob/master/README.md.
Exit code is 1
container 0bcee4001220b81d936d3be6b5bc8f9f130b7875bece395edf673ce5aa1f7c3e is dead
Sun 19 Feb 2023 01:53:24 PM CET

@yegor256 please try to merge one more time

@rultor merge

@rultor merge

@yegor256 OK, I'll try to merge now. You can check the progress of the merge here

@rultor merge

@yegor256 Done! FYI, the full log is here (took me 11min)


Improve #include structure of commands ECFLOW-1932

Codecov Report
Attention: 239 lines in your changes are missing coverage. Please review.

Comparison is base (6689ac7) 80.16% compared to head (b81e4d7) 83.56%.

Files
Patch %
Lines

Base/src/ecflow/base/cts/task/CtsWaitCmd.cpp
39.16%
87 Missing :warning:

Base/src/ecflow/base/cts/task/QueueCmd.cpp
70.00%
33 Missing :warning:

Base/src/ecflow/base/cts/user/MoveCmd.cpp
64.86%
26 Missing :warning:

Base/src/ecflow/base/cts/task/InitCmd.cpp
60.65%
24 Missing :warning:

Base/src/ecflow/base/cts/task/LabelCmd.cpp
77.08%
11 Missing :warning:

Base/src/ecflow/base/cts/task/MeterCmd.cpp
81.35%
11 Missing :warning:

Base/src/ecflow/base/cts/task/EventCmd.cpp
83.01%
9 Missing :warning:

Base/src/ecflow/base/cts/task/CompleteCmd.cpp
84.09%
7 Missing :warning:

Base/src/ecflow/base/cts/task/AbortCmd.cpp
87.23%
6 Missing :warning:

Base/src/ecflow/base/cts/user/AlterCmd.hpp
94.28%
4 Missing :warning:

... and 13 more

Additional details and impacted files
@@             Coverage Diff             @@
##           develop      #68      +/-   ##
===========================================
+ Coverage    80.16%   83.56%   +3.39%
===========================================
  Files          580      589       +9
  Lines        57925    52567    -5358
===========================================
- Hits         46434    43925    -2509
+ Misses       11491     8642    -2849

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Remove bower.json

Is this file still up to date? Is it needed? Do we upload a bower package somewhere? Does it do something for bower to have it in repo? Is bower still used by anyone?

Can this issue be closed ?


How to map multiple rightsHolder

@pieterprovoost @rubenpp7 some campaigns in ESAS have multiple dataRightsHolders, e.g.:
https://esas.ices.dk/api/getCampaignRecords?campaignID=775
[
  {
    "tblUploadID": 231,
    "tblCampaignID": 208521,
    "dataRightsHolder": "3269~2299",
    "country": "DE",
    "campaignID": "775",
    "dataAccess": "Public",
    "startDate": "2011-07-27",
    "endDate": "2011-08-22",
    "notes": null
  }
]

3269~2299 indicates Federal Agency for Nature Conservation (BfN) | Research and Technology Centre (Buesum) (FTZ)
Can I map both in rightsHolder (separated with |) or do you prefer that I only keep the first one?

Sorry, submitted before issue was finished. Updated now in issue body.

@nicolasvanermen prefers to list all rightsHolders

Best is to map all data rights holders.

Yes, pipe separated rightsHolders is fine.

Fixed in 0a6c8fa


feat:add whats inside screenshot for nerdlet and launcher

This adds artifact screenshots to be used in the What's New section of the nerdpack detail view.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


issue-109 Rank API

translate Rank API
fix the problems reviewed in Task API

Thanks, seems u need rebase the website branch & solve conflicts first?

Hi, could u submit a new PR to master branch
even if u don't have time to address comments?

due to the website branch has deleted, I'll address the pr into the master soon


Discarding file changes crashes VS Code

VSCode Version: Code 1.19.0 (816be6780ca8bd0ab80314e11478c48c70d09383, 2017-12-14T12:06:43.492Z)
OS Version: Windows_NT x64 10.0.16299
Extensions: none

Steps to Reproduce:
1.Open any file or create a new one
2.Type in anything without saving
3.Close the fiile and select Don't save
4.VS Code should crash

I think something is not right with my VS Code installation.
How would I have a clean uninstallation?
I deleted .vscode and %APPDATA%/Roaming/Code but didnt help, I did it on another Computer and couldnt reproduce

Can you try to reproduce with our nightly insider builds? You can give our preview releases a try from: https://code.visualstudio.com/insiders/

Well I restarted and it appears no more. I think the hibernation screwed something up. I get this kind of problems quite frequently though.


Error compiling in MSys2

Hi guys,
Compiling the code in Windows 10, using the updated MSys2, mingw-64 (build as of feb15 2021). After updating all packages to the current editions, I'm getting an error in the last step "Linking Fortran executable." This seems to be due to an issue with the -static libraries for use with windows.
Error:
[100%] Linking Fortran executable ../bin/cactus
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0\libgfortran.a(read.o):(.text$_gfortrani_convert_real+0xb7): relocation truncated to fit: IMAGE_REL_AMD64_REL32 against undefined symbol strtoflt128' C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0\libgfortran.a(read.o):(.text$_gfortrani_convert_infnan+0xb3): relocation truncated to fit: IMAGE_REL_AMD64_REL32 against undefined symbol strtoflt128'
collect2.exe: error: ld returned 1 exit status
make[2]: *** [CMakeFiles/cactus.dir/build.make:1050: ../bin/cactus] Error 1
make[1]: *** [CMakeFiles/Makefile2:95: CMakeFiles/cactus.dir/all] Error 2
make: *** [Makefile:103: all] Error 2
Anyone else experiencing this issue? I'm not sure if this is innate to the MSys2 system or some easier work around.
I can successfully compile when the -static link is removed, but I can't run the program as it is missing essential .dlls.
Thanks for your time!
Bruce

Hey @bleblanc18 long time!
It does appear to be an msys issue, see https://github.com/msys2/MINGW-packages/issues/7023
There is a workaround listed, which is to add some flags during compilation.

That did the trick!
Thanks a ton!
It has been quite the while. What are you up to these days?


ipptool: Provide a way to run a subset of tests

From @KurtPfeifle in ippsample issue 60:
After meanwhile quite a few ipp-1.1.test runs and being entangled in related debugging activities, I think the following would be beneficial for future testers:

Split all the individual tests making up the IPP-1.1 test suite into separate files.
(Re-)Create a new ipp-1.1.test suite by using INCLUDE statements for the individual files.

This way it would be more easy to run specific tests individually, especially when it comes to debugging the respective parts of the now biiiig ipp-1.1.test file.
It would also allow to make a meaningful use of ipptool's capability to test with multiple test files within one run as its usage hint suggests:
 Usage: ipptool [options] URI filename [ ... filenameN ]

I'm glad you came around. 😊 The biggest issue is handling the dependency graph between DEFINEs used in a test and making sure earlier tests are also run. If they are spread across files that may make things awkward but might be fine if just done carefully.


Right to left support

We should make sure that DevToys goes Right To Left when in a RTl language like Arabic or Japanese.

I would like to work on this. Can you assign it to me?

Actually, I added the support to it a few weeks ago :) It's not released yet though.