error when saving ckpt

System Info
- `Accelerate` version: 0.33.0
- Platform: Linux-3.10.0-957.el7.x86_64-x86_64-with-glibc2.17
- `accelerate` bash location: /mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/bin/accelerate
- Python version: 3.8.19
- Numpy version: 1.24.3
- PyTorch version (GPU?): 2.4.0 (False)
- PyTorch XPU available: False
- PyTorch NPU available: False
- PyTorch MLU available: False
- PyTorch MUSA available: False
- System RAM: 754.36 GB
- `Accelerate` default config:
	- compute_environment: LOCAL_MACHINE
	- distributed_type: MULTI_GPU
	- mixed_precision: fp16
	- use_cpu: False
	- debug: False
	- num_processes: 2
	- machine_rank: 0
	- num_machines: 1
	- gpu_ids: 1,2
	- rdzv_backend: static
	- same_network: True
	- main_training_function: main
	- enable_cpu_affinity: False
	- downcast_bf16: no
	- tpu_use_cluster: False
	- tpu_use_sudo: False
	- tpu_env: []

Information

[ ] The official example scripts
[ ] My own modified scripts

Tasks

[ ] One of the scripts in the examples/ folder of Accelerate or an officially supported no_trainer script in the examples folder of the transformers repo (such as run_no_trainer_glue.py)
[ ] My own task or dataset (give details below)

Reproduction
1: script:
#!/bin/bash
#SBATCH -J train_job
#SBATCH -p partition

#SBATCH --nodes=2

#SBATCH --gres=gpu:8

#SBATCH --ntasks-per-node=1

# some parameters are ignored

srun accelerate launch --config_file ~/.cache/huggingface/accelerate/fsdp_multinodes.yaml \
  --num_processes $((SLURM_NNODES * GPUS_PER_NODE)) --machine_rank $NODE_RANK --num_machines $SLURM_JOB_NUM_NODES --rdzv_backend c10d --main_process_ip $MASTER_ADDR --main_process_port $MASTER_PORT \
  train.py

code:

def save_checkpoint(self):
    assert self.accelerator.sync_gradients
    if self.global_iter == self.last_saved_iter:
        self.logger.warning(
            f"checkpoint for global_iter: {self.global_iter} has been saved, pass"
        )
        return
    self.accelerator.project_configuration.iteration = self.global_iter + 1
    self.accelerator.save_state()
    self.last_saved_iter = self.global_iter
    self.logger.info(f"Save checkpoint for global_iter: {self.global_iter + 1}")

config:

compute_environment: LOCAL_MACHINE
debug: true
distributed_type: FSDP
downcast_bf16: 'no'
enable_cpu_affinity: false
fsdp_config:
  fsdp_activation_checkpointing: true
  fsdp_auto_wrap_policy: SIZE_BASED_WRAP
  fsdp_backward_prefetch: BACKWARD_PRE
  fsdp_cpu_ram_efficient_loading: false
  fsdp_forward_prefetch: false
  fsdp_min_num_params: 20000000
  fsdp_offload_params: false
  fsdp_sharding_strategy: FULL_SHARD
  fsdp_state_dict_type: SHARDED_STATE_DICT
  fsdp_sync_module_states: true
  fsdp_use_orig_params: true
machine_rank: 0
main_process_ip: <IP_ADDRESS>
main_process_port: 10078
main_training_function: main
mixed_precision: fp16
num_machines: 2
num_processes: 4
rdzv_backend: static
same_network: true
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: false

Expected behavior
It works well with one machine node, but an error occurs when using two machines.

07/26/2024 10:59:38   INFO  Saving FSDP model
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:689: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
  warnings.warn(
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:689: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
  warnings.warn(
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:689: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
  warnings.warn(
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:689: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
  warnings.warn(
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:689: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
  warnings.warn(
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:689: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
  warnings.warn(
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:689: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
  warnings.warn(
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:689: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
  warnings.warn(
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:689: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
  warnings.warn(
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:689: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
  warnings.warn(
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:689: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
  warnings.warn(
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:689: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
  warnings.warn(
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:689: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
  warnings.warn(
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:689: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
  warnings.warn(
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:689: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
  warnings.warn(
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:689: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
  warnings.warn(
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/_state_dict_utils.py:737: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  local_shape = tensor.shape
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/_state_dict_utils.py:749: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  tensor.shape,
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/_state_dict_utils.py:737: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  local_shape = tensor.shape
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/_state_dict_utils.py:737: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  local_shape = tensor.shape
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/_state_dict_utils.py:751: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  tensor.dtype,
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/_state_dict_utils.py:737: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  local_shape = tensor.shape
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/_state_dict_utils.py:752: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  tensor.device,
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/_state_dict_utils.py:749: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  tensor.shape,
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/_state_dict_utils.py:737: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  local_shape = tensor.shape
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/_state_dict_utils.py:749: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  tensor.shape,
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/_state_dict_utils.py:749: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  tensor.shape,
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/_state_dict_utils.py:751: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  tensor.dtype,
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/_state_dict_utils.py:751: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  tensor.dtype,
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/_state_dict_utils.py:749: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  tensor.shape,
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/_state_dict_utils.py:752: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  tensor.device,
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/_state_dict_utils.py:751: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  tensor.dtype,
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/_state_dict_utils.py:737: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  local_shape = tensor.shape
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/_state_dict_utils.py:752: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  tensor.device,
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/_state_dict_utils.py:749: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  tensor.shape,
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/_state_dict_utils.py:751: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  tensor.dtype,
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/_state_dict_utils.py:752: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  tensor.device,
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/_state_dict_utils.py:751: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  tensor.dtype,
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/_state_dict_utils.py:752: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  tensor.device,
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/_state_dict_utils.py:752: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  tensor.device,
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/_state_dict_utils.py:737: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  local_shape = tensor.shape
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/_state_dict_utils.py:737: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  local_shape = tensor.shape
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/_state_dict_utils.py:749: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  tensor.shape,
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/_state_dict_utils.py:749: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  tensor.shape,
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/_state_dict_utils.py:751: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  tensor.dtype,
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/_state_dict_utils.py:751: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  tensor.dtype,
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/_state_dict_utils.py:752: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  tensor.device,
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/_state_dict_utils.py:752: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  tensor.device,
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/_state_dict_utils.py:737: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  local_shape = tensor.shape
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/_state_dict_utils.py:737: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  local_shape = tensor.shape
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/_state_dict_utils.py:749: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  tensor.shape,
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/_state_dict_utils.py:749: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  tensor.shape,
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/_state_dict_utils.py:751: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  tensor.dtype,
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/_state_dict_utils.py:751: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  tensor.dtype,
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/_state_dict_utils.py:752: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  tensor.device,
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/_state_dict_utils.py:752: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  tensor.device,
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/_state_dict_utils.py:737: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  local_shape = tensor.shape
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/_state_dict_utils.py:737: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  local_shape = tensor.shape
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/_state_dict_utils.py:749: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  tensor.shape,
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/_state_dict_utils.py:737: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  local_shape = tensor.shape
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/_state_dict_utils.py:751: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  tensor.dtype,
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/_state_dict_utils.py:749: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  tensor.shape,
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/_state_dict_utils.py:752: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  tensor.device,
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/_state_dict_utils.py:749: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  tensor.shape,
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/_state_dict_utils.py:737: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  local_shape = tensor.shape
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/_state_dict_utils.py:751: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  tensor.dtype,
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/_state_dict_utils.py:749: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  tensor.shape,
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/_state_dict_utils.py:751: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  tensor.dtype,
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/_state_dict_utils.py:751: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  tensor.dtype,
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/_state_dict_utils.py:752: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  tensor.device,
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/_state_dict_utils.py:737: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  local_shape = tensor.shape
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/_state_dict_utils.py:752: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  tensor.device,
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/_state_dict_utils.py:752: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  tensor.device,
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/_state_dict_utils.py:737: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  local_shape = tensor.shape
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/_state_dict_utils.py:749: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  tensor.shape,
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/_state_dict_utils.py:749: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  tensor.shape,
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/_state_dict_utils.py:751: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  tensor.dtype,
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/_state_dict_utils.py:751: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  tensor.dtype,
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/_state_dict_utils.py:752: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  tensor.device,
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/fsdp/_state_dict_utils.py:752: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  tensor.device,
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/accelerate/utils/fsdp_utils.py:90: FutureWarning: `save_state_dict` is deprecated and will be removed in future versions.Please use `save` instead.
  dist_cp.save_state_dict(
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/accelerate/utils/fsdp_utils.py:90: FutureWarning: `save_state_dict` is deprecated and will be removed in future versions.Please use `save` instead.
  dist_cp.save_state_dict(
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/accelerate/utils/fsdp_utils.py:90: FutureWarning: `save_state_dict` is deprecated and will be removed in future versions.Please use `save` instead.
  dist_cp.save_state_dict(
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/accelerate/utils/fsdp_utils.py:90: FutureWarning: `save_state_dict` is deprecated and will be removed in future versions.Please use `save` instead.
  dist_cp.save_state_dict(
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/accelerate/utils/fsdp_utils.py:90: FutureWarning: `save_state_dict` is deprecated and will be removed in future versions.Please use `save` instead.
  dist_cp.save_state_dict(
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/accelerate/utils/fsdp_utils.py:90: FutureWarning: `save_state_dict` is deprecated and will be removed in future versions.Please use `save` instead.
  dist_cp.save_state_dict(
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/accelerate/utils/fsdp_utils.py:90: FutureWarning: `save_state_dict` is deprecated and will be removed in future versions.Please use `save` instead.
  dist_cp.save_state_dict(
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/accelerate/utils/fsdp_utils.py:90: FutureWarning: `save_state_dict` is deprecated and will be removed in future versions.Please use `save` instead.
  dist_cp.save_state_dict(
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/accelerate/utils/fsdp_utils.py:90: FutureWarning: `save_state_dict` is deprecated and will be removed in future versions.Please use `save` instead.
  dist_cp.save_state_dict(
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/accelerate/utils/fsdp_utils.py:90: FutureWarning: `save_state_dict` is deprecated and will be removed in future versions.Please use `save` instead.
  dist_cp.save_state_dict(
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/accelerate/utils/fsdp_utils.py:90: FutureWarning: `save_state_dict` is deprecated and will be removed in future versions.Please use `save` instead.
  dist_cp.save_state_dict(
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/accelerate/utils/fsdp_utils.py:90: FutureWarning: `save_state_dict` is deprecated and will be removed in future versions.Please use `save` instead.
  dist_cp.save_state_dict(
07/26/2024 10:59:41   INFO  Saving model to /mnt/petrelfs/yanghonghui/projects/OpenPCDet_torch2.4.0/output/game_models/sd3_rope/debug/checkpoints/checkpoint_1/pytorch_model_fsdp_0
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/accelerate/utils/fsdp_utils.py:90: FutureWarning: `save_state_dict` is deprecated and will be removed in future versions.Please use `save` instead.
  dist_cp.save_state_dict(
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/accelerate/utils/fsdp_utils.py:90: FutureWarning: `save_state_dict` is deprecated and will be removed in future versions.Please use `save` instead.
  dist_cp.save_state_dict(
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/accelerate/utils/fsdp_utils.py:90: FutureWarning: `save_state_dict` is deprecated and will be removed in future versions.Please use `save` instead.
  dist_cp.save_state_dict(
/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/accelerate/utils/fsdp_utils.py:90: FutureWarning: `save_state_dict` is deprecated and will be removed in future versions.Please use `save` instead.
  dist_cp.save_state_dict(
SH-IDC1-10-140-24-106:147874:192057 [6] NCCL INFO Channel 00/1 : 6[6] -> 0[0] via P2P/CUMEM/read
SH-IDC1-10-140-24-106:147866:192060 [1] NCCL INFO Channel 00/1 : 1[1] -> 0[0] via P2P/CUMEM/read
SH-IDC1-10-140-24-95:214042:258021 [7] NCCL INFO Channel 01/1 : 15[7] -> 0[0] [send] via NET/IB/0(8)/GDRDMA/Shared
SH-IDC1-10-140-24-95:214040:258019 [6] NCCL INFO Channel 01/1 : 14[6] -> 0[0] [send] via NET/IB/0(8)/GDRDMA/Shared
SH-IDC1-10-140-24-106:147867:192061 [2] NCCL INFO Channel 00/1 : 2[2] -> 0[0] via P2P/CUMEM/read
SH-IDC1-10-140-24-95:214035:258020 [4] NCCL INFO Channel 01/1 : 12[4] -> 0[0] [send] via NET/IB/0(8)/GDRDMA/Shared
SH-IDC1-10-140-24-106:147869:192059 [4] NCCL INFO Channel 00/1 : 4[4] -> 0[0] via P2P/CUMEM/read
SH-IDC1-10-140-24-95:214038:258026 [5] NCCL INFO Channel 01/1 : 13[5] -> 0[0] [send] via NET/IB/0(8)/GDRDMA/Shared
SH-IDC1-10-140-24-106:147872:192064 [5] NCCL INFO Channel 00/1 : 5[5] -> 0[0] via P2P/CUMEM/read
SH-IDC1-10-140-24-95:214033:258022 [2] NCCL INFO Channel 01/1 : 10[2] -> 0[0] [send] via NET/IB/0(8)/GDRDMA/Shared
SH-IDC1-10-140-24-106:147876:192058 [7] NCCL INFO Channel 00/1 : 7[7] -> 0[0] via P2P/CUMEM/read
SH-IDC1-10-140-24-95:214034:258023 [3] NCCL INFO Channel 01/1 : 11[3] -> 0[0] [send] via NET/IB/0(8)/GDRDMA/Shared
SH-IDC1-10-140-24-95:214032:258024 [1] NCCL INFO Channel 01/1 : 9[1] -> 0[0] [send] via NET/IB/0(8)/GDRDMA/Shared
SH-IDC1-10-140-24-106:147868:192062 [3] NCCL INFO Channel 00/1 : 3[3] -> 0[0] via P2P/CUMEM/read
SH-IDC1-10-140-24-95:214031:258025 [0] NCCL INFO Channel 01/1 : 8[0] -> 0[0] [send] via NET/IB/0/GDRDMA/Shared
SH-IDC1-10-140-24-106:147865:192063 [0] NCCL INFO Channel 01/1 : 15[7] -> 0[0] [receive] via NET/IB/0/GDRDMA/Shared
SH-IDC1-10-140-24-106:147865:192063 [0] NCCL INFO Channel 01/1 : 14[6] -> 0[0] [receive] via NET/IB/0/GDRDMA/Shared
SH-IDC1-10-140-24-106:147865:192063 [0] NCCL INFO Channel 01/1 : 13[5] -> 0[0] [receive] via NET/IB/0/GDRDMA/Shared
SH-IDC1-10-140-24-106:147865:192063 [0] NCCL INFO Channel 01/1 : 12[4] -> 0[0] [receive] via NET/IB/0/GDRDMA/Shared
SH-IDC1-10-140-24-106:147865:192063 [0] NCCL INFO Channel 01/1 : 11[3] -> 0[0] [receive] via NET/IB/0/GDRDMA/Shared
SH-IDC1-10-140-24-106:147865:192063 [0] NCCL INFO Channel 01/1 : 10[2] -> 0[0] [receive] via NET/IB/0/GDRDMA/Shared
SH-IDC1-10-140-24-106:147865:192063 [0] NCCL INFO Channel 01/1 : 9[1] -> 0[0] [receive] via NET/IB/0/GDRDMA/Shared
SH-IDC1-10-140-24-106:147865:192063 [0] NCCL INFO Channel 01/1 : 8[0] -> 0[0] [receive] via NET/IB/0/GDRDMA/Shared
SH-IDC1-10-140-24-95:214040:258033 [6] NCCL INFO Channel 01/1 : 0[0] -> 14[6] [receive] via NET/IB/0/GDRDMA/Shared
SH-IDC1-10-140-24-106:147865:192085 [0] NCCL INFO Channel 00/1 : 0[0] -> 1[1] via P2P/CUMEM/read
SH-IDC1-10-140-24-95:214035:258034 [4] NCCL INFO Channel 01/1 : 0[0] -> 12[4] [receive] via NET/IB/0/GDRDMA/Shared
SH-IDC1-10-140-24-95:214042:258032 [7] NCCL INFO Channel 01/1 : 0[0] -> 15[7] [receive] via NET/IB/0/GDRDMA/Shared
SH-IDC1-10-140-24-95:214034:258035 [3] NCCL INFO Channel 01/1 : 0[0] -> 11[3] [receive] via NET/IB/0/GDRDMA/Shared
SH-IDC1-10-140-24-95:214033:258030 [2] NCCL INFO Channel 01/1 : 0[0] -> 10[2] [receive] via NET/IB/0/GDRDMA/Shared
SH-IDC1-10-140-24-95:214031:258037 [0] NCCL INFO Channel 01/1 : 0[0] -> 8[0] [receive] via NET/IB/0/GDRDMA/Shared
SH-IDC1-10-140-24-106:147865:192085 [0] NCCL INFO Channel 00/1 : 0[0] -> 2[2] via P2P/CUMEM/read
SH-IDC1-10-140-24-95:214032:258036 [1] NCCL INFO Channel 01/1 : 0[0] -> 9[1] [receive] via NET/IB/0/GDRDMA/Shared
SH-IDC1-10-140-24-106:147865:192085 [0] NCCL INFO Channel 00/1 : 0[0] -> 3[3] via P2P/CUMEM/read
SH-IDC1-10-140-24-95:214038:258038 [5] NCCL INFO Channel 01/1 : 0[0] -> 13[5] [receive] via NET/IB/0/GDRDMA/Shared
SH-IDC1-10-140-24-106:147865:192085 [0] NCCL INFO Channel 00/1 : 0[0] -> 4[4] via P2P/CUMEM/read
SH-IDC1-10-140-24-106:147865:192085 [0] NCCL INFO Channel 00/1 : 0[0] -> 5[5] via P2P/CUMEM/read
SH-IDC1-10-140-24-106:147865:192085 [0] NCCL INFO Channel 00/1 : 0[0] -> 6[6] via P2P/CUMEM/read
SH-IDC1-10-140-24-106:147865:192085 [0] NCCL INFO Channel 00/1 : 0[0] -> 7[7] via P2P/CUMEM/read
SH-IDC1-10-140-24-106:147865:192085 [0] NCCL INFO Channel 01/1 : 0[0] -> 8[0] [send] via NET/IB/0/GDRDMA/Shared
SH-IDC1-10-140-24-106:147865:192085 [0] NCCL INFO Channel 01/1 : 0[0] -> 9[1] [send] via NET/IB/0/GDRDMA/Shared
SH-IDC1-10-140-24-106:147865:192085 [0] NCCL INFO Channel 01/1 : 0[0] -> 10[2] [send] via NET/IB/0/GDRDMA/Shared
SH-IDC1-10-140-24-106:147865:192085 [0] NCCL INFO Channel 01/1 : 0[0] -> 11[3] [send] via NET/IB/0/GDRDMA/Shared
SH-IDC1-10-140-24-106:147865:192085 [0] NCCL INFO Channel 01/1 : 0[0] -> 12[4] [send] via NET/IB/0/GDRDMA/Shared
SH-IDC1-10-140-24-106:147865:192085 [0] NCCL INFO Channel 01/1 : 0[0] -> 13[5] [send] via NET/IB/0/GDRDMA/Shared
SH-IDC1-10-140-24-106:147865:192085 [0] NCCL INFO Channel 01/1 : 0[0] -> 14[6] [send] via NET/IB/0/GDRDMA/Shared
SH-IDC1-10-140-24-106:147865:192085 [0] NCCL INFO Channel 01/1 : 0[0] -> 15[7] [send] via NET/IB/0/GDRDMA/Shared
mlx5: SH-IDC1-10-140-24-95: got completion with error:
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000001 00000000 00000000 00000000
00000000 00008914 100009e8 000045d2
mlx5: SH-IDC1-10-140-24-95: got completion with error:
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000001 00000000 00000000 00000000
00000000 00008914 100009e6 00004bd2

SH-IDC1-10-140-24-95:214034:214782 [3] transport/net_ib.cc:1696 NCCL WARN NET/IB : Got completion from peer <IP_ADDRESS><37760> with status=11 opcode=129 len=0 vendor err 137 (Flush)
SH-IDC1-10-140-24-95:214034:214782 [3] NCCL INFO transport/net.cc:1374 -> 6

SH-IDC1-10-140-24-95:214033:214783 [2] transport/net_ib.cc:1696 NCCL WARN NET/IB : Got completion from peer <IP_ADDRESS><27076> with status=11 opcode=129 len=0 vendor err 137 (Flush)
SH-IDC1-10-140-24-95:214034:214782 [3] NCCL INFO proxy.cc:694 -> 6
SH-IDC1-10-140-24-95:214033:214783 [2] NCCL INFO transport/net.cc:1374 -> 6
SH-IDC1-10-140-24-95:214034:214782 [3] NCCL INFO proxy.cc:874 -> 6 [Progress Thread]
SH-IDC1-10-140-24-95:214033:214783 [2] NCCL INFO proxy.cc:694 -> 6
SH-IDC1-10-140-24-95:214033:214783 [2] NCCL INFO proxy.cc:874 -> 6 [Progress Thread]
mlx5: SH-IDC1-10-140-24-95: got completion with error:
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000001 00000000 00000000 00000000
00000000 00008914 100009ec 000041d2

SH-IDC1-10-140-24-95:214034:214782 [3] transport/net_ib.cc:1696 NCCL WARN NET/IB : Got completion from peer <IP_ADDRESS><37760> with status=5 opcode=129 len=3 vendor err 249 (Flush)

SH-IDC1-10-140-24-95:214033:214783 [2] transport/net_ib.cc:1696 NCCL WARN NET/IB : Got completion from peer <IP_ADDRESS><27076> with status=5 opcode=129 len=2 vendor err 249 (Flush)
SH-IDC1-10-140-24-95:214034:214782 [3] NCCL INFO transport/net.cc:1374 -> 6
SH-IDC1-10-140-24-95:214033:214783 [2] NCCL INFO transport/net.cc:1374 -> 6
SH-IDC1-10-140-24-95:214034:214782 [3] NCCL INFO proxy.cc:694 -> 6
SH-IDC1-10-140-24-95:214033:214783 [2] NCCL INFO proxy.cc:694 -> 6
SH-IDC1-10-140-24-95:214034:214782 [3] NCCL INFO proxy.cc:874 -> 6 [Progress Thread]
SH-IDC1-10-140-24-95:214033:214783 [2] NCCL INFO proxy.cc:874 -> 6 [Progress Thread]

SH-IDC1-10-140-24-95:214033:214783 [2] transport/net_ib.cc:1696 NCCL WARN NET/IB : Got completion from peer <IP_ADDRESS><27076> with status=5 opcode=129 len=2 vendor err 249 (Flush)
SH-IDC1-10-140-24-95:214033:214783 [2] NCCL INFO transport/net.cc:1374 -> 6
SH-IDC1-10-140-24-95:214033:214783 [2] NCCL INFO proxy.cc:694 -> 6
SH-IDC1-10-140-24-95:214033:214783 [2] NCCL INFO proxy.cc:874 -> 6 [Progress Thread]
mlx5: SH-IDC1-10-140-24-95: got completion with error:
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000001 00000000 00000000 00000000
00000000 00008914 100009ea 000047d2

SH-IDC1-10-140-24-95:214038:214779 [5] transport/net_ib.cc:1696 NCCL WARN NET/IB : Got completion from peer <IP_ADDRESS><47174> with status=11 opcode=129 len=0 vendor err 137 (Flush)
SH-IDC1-10-140-24-95:214038:214779 [5] NCCL INFO transport/net.cc:1374 -> 6
SH-IDC1-10-140-24-95:214038:214779 [5] NCCL INFO proxy.cc:694 -> 6
SH-IDC1-10-140-24-95:214038:214779 [5] NCCL INFO proxy.cc:874 -> 6 [Progress Thread]

SH-IDC1-10-140-24-95:214038:214779 [5] transport/net_ib.cc:1696 NCCL WARN NET/IB : Got completion from peer <IP_ADDRESS><47174> with status=5 opcode=129 len=5 vendor err 249 (Flush)
SH-IDC1-10-140-24-95:214038:214779 [5] NCCL INFO transport/net.cc:1374 -> 6
SH-IDC1-10-140-24-95:214038:214779 [5] NCCL INFO proxy.cc:694 -> 6
SH-IDC1-10-140-24-95:214038:214779 [5] NCCL INFO proxy.cc:874 -> 6 [Progress Thread]

SH-IDC1-10-140-24-95:214038:214779 [5] transport/net_ib.cc:1696 NCCL WARN NET/IB : Got completion from peer <IP_ADDRESS><47174> with status=5 opcode=129 len=5 vendor err 249 (Flush)
SH-IDC1-10-140-24-95:214038:214779 [5] NCCL INFO transport/net.cc:1374 -> 6
SH-IDC1-10-140-24-95:214038:214779 [5] NCCL INFO proxy.cc:694 -> 6
SH-IDC1-10-140-24-95:214038:214779 [5] NCCL INFO proxy.cc:874 -> 6 [Progress Thread]

SH-IDC1-10-140-24-95:214035:214781 [4] transport/net_ib.cc:1696 NCCL WARN NET/IB : Got completion from peer <IP_ADDRESS><5776> with status=11 opcode=129 len=0 vendor err 137 (Flush)
SH-IDC1-10-140-24-95:214035:214781 [4] NCCL INFO transport/net.cc:1374 -> 6
SH-IDC1-10-140-24-95:214035:214781 [4] NCCL INFO proxy.cc:694 -> 6
SH-IDC1-10-140-24-95:214035:214781 [4] NCCL INFO proxy.cc:874 -> 6 [Progress Thread]

SH-IDC1-10-140-24-95:214035:214781 [4] transport/net_ib.cc:1696 NCCL WARN NET/IB : Got completion from peer <IP_ADDRESS><5776> with status=5 opcode=129 len=4 vendor err 249 (Flush)
SH-IDC1-10-140-24-95:214035:214781 [4] NCCL INFO transport/net.cc:1374 -> 6
SH-IDC1-10-140-24-95:214035:214781 [4] NCCL INFO proxy.cc:694 -> 6
SH-IDC1-10-140-24-95:214035:214781 [4] NCCL INFO proxy.cc:874 -> 6 [Progress Thread]

SH-IDC1-10-140-24-95:214035:214781 [4] transport/net_ib.cc:1696 NCCL WARN NET/IB : Got completion from peer <IP_ADDRESS><5776> with status=5 opcode=129 len=4 vendor err 249 (Flush)
SH-IDC1-10-140-24-95:214035:214781 [4] NCCL INFO transport/net.cc:1374 -> 6
SH-IDC1-10-140-24-95:214035:214781 [4] NCCL INFO proxy.cc:694 -> 6
SH-IDC1-10-140-24-95:214035:214781 [4] NCCL INFO proxy.cc:874 -> 6 [Progress Thread]
mlx5: SH-IDC1-10-140-24-95: got completion with error:
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000001 00000000 00000000 00000000
00000000 00008914 100009f0 00005dd2

SH-IDC1-10-140-24-95:214042:214778 [7] transport/net_ib.cc:1696 NCCL WARN NET/IB : Got completion from peer <IP_ADDRESS><61176> with status=11 opcode=129 len=0 vendor err 137 (Flush)
SH-IDC1-10-140-24-95:214042:214778 [7] NCCL INFO transport/net.cc:1374 -> 6
SH-IDC1-10-140-24-95:214042:214778 [7] NCCL INFO proxy.cc:694 -> 6
SH-IDC1-10-140-24-95:214042:214778 [7] NCCL INFO proxy.cc:874 -> 6 [Progress Thread]

SH-IDC1-10-140-24-95:214042:214778 [7] transport/net_ib.cc:1696 NCCL WARN NET/IB : Got completion from peer <IP_ADDRESS><61176> with status=5 opcode=129 len=7 vendor err 249 (Flush)
SH-IDC1-10-140-24-95:214042:214778 [7] NCCL INFO transport/net.cc:1374 -> 6
SH-IDC1-10-140-24-95:214042:214778 [7] NCCL INFO proxy.cc:694 -> 6
SH-IDC1-10-140-24-95:214042:214778 [7] NCCL INFO proxy.cc:874 -> 6 [Progress Thread]

SH-IDC1-10-140-24-95:214042:214778 [7] transport/net_ib.cc:1696 NCCL WARN NET/IB : Got completion from peer <IP_ADDRESS><61176> with status=5 opcode=129 len=7 vendor err 249 (Flush)
SH-IDC1-10-140-24-95:214042:214778 [7] NCCL INFO transport/net.cc:1374 -> 6
SH-IDC1-10-140-24-95:214042:214778 [7] NCCL INFO proxy.cc:694 -> 6
SH-IDC1-10-140-24-95:214042:214778 [7] NCCL INFO proxy.cc:874 -> 6 [Progress Thread]

SH-IDC1-10-140-24-95:214042:214778 [7] transport/net_ib.cc:1696 NCCL WARN NET/IB : Got completion from peer <IP_ADDRESS><61176> with status=5 opcode=129 len=7 vendor err 249 (Flush)
SH-IDC1-10-140-24-95:214042:214778 [7] NCCL INFO transport/net.cc:1374 -> 6
SH-IDC1-10-140-24-95:214042:214778 [7] NCCL INFO proxy.cc:694 -> 6
SH-IDC1-10-140-24-95:214042:214778 [7] NCCL INFO proxy.cc:874 -> 6 [Progress Thread]
mlx5: SH-IDC1-10-140-24-95: got completion with error:
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000001 00000000 00000000 00000000
00000000 00008914 100009ee 000043d2

SH-IDC1-10-140-24-95:214040:214780 [6] transport/net_ib.cc:1696 NCCL WARN NET/IB : Got completion from peer <IP_ADDRESS><37762> with status=11 opcode=129 len=0 vendor err 137 (Flush)
SH-IDC1-10-140-24-95:214040:214780 [6] NCCL INFO transport/net.cc:1374 -> 6
SH-IDC1-10-140-24-95:214040:214780 [6] NCCL INFO proxy.cc:694 -> 6
SH-IDC1-10-140-24-95:214040:214780 [6] NCCL INFO proxy.cc:874 -> 6 [Progress Thread]

SH-IDC1-10-140-24-95:214040:214780 [6] transport/net_ib.cc:1696 NCCL WARN NET/IB : Got completion from peer <IP_ADDRESS><37762> with status=5 opcode=129 len=6 vendor err 249 (Flush)
SH-IDC1-10-140-24-95:214040:214780 [6] NCCL INFO transport/net.cc:1374 -> 6
SH-IDC1-10-140-24-95:214040:214780 [6] NCCL INFO proxy.cc:694 -> 6
SH-IDC1-10-140-24-95:214040:214780 [6] NCCL INFO proxy.cc:874 -> 6 [Progress Thread]

SH-IDC1-10-140-24-95:214040:214780 [6] transport/net_ib.cc:1696 NCCL WARN NET/IB : Got completion from peer <IP_ADDRESS><37762> with status=5 opcode=129 len=6 vendor err 249 (Flush)
SH-IDC1-10-140-24-95:214040:214780 [6] NCCL INFO transport/net.cc:1374 -> 6
SH-IDC1-10-140-24-95:214040:214780 [6] NCCL INFO proxy.cc:694 -> 6
SH-IDC1-10-140-24-95:214040:214780 [6] NCCL INFO proxy.cc:874 -> 6 [Progress Thread]
[rank15]:[E726 10:59:43.692836266 ProcessGroupNCCL.cpp:1664] [PG 0 (default_pg) Rank 15] Exception (either an error or timeout) detected by watchdog at work: 184, last enqueued NCCL work: 184, last completed NCCL work: 183.
[rank10]:[E726 10:59:43.692913143 ProcessGroupNCCL.cpp:1664] [PG 0 (default_pg) Rank 10] Exception (either an error or timeout) detected by watchdog at work: 184, last enqueued NCCL work: 184, last completed NCCL work: 183.
[rank11]:[E726 10:59:43.692913153 ProcessGroupNCCL.cpp:1664] [PG 0 (default_pg) Rank 11] Exception (either an error or timeout) detected by watchdog at work: 184, last enqueued NCCL work: 184, last completed NCCL work: 183.
[rank13]:[E726 10:59:43.693201512 ProcessGroupNCCL.cpp:1664] [PG 0 (default_pg) Rank 13] Exception (either an error or timeout) detected by watchdog at work: 184, last enqueued NCCL work: 184, last completed NCCL work: 183.
[rank12]:[E726 10:59:43.695610972 ProcessGroupNCCL.cpp:1664] [PG 0 (default_pg) Rank 12] Exception (either an error or timeout) detected by watchdog at work: 184, last enqueued NCCL work: 184, last completed NCCL work: 183.
[rank14]:[E726 10:59:43.724105047 ProcessGroupNCCL.cpp:1664] [PG 0 (default_pg) Rank 14] Exception (either an error or timeout) detected by watchdog at work: 184, last enqueued NCCL work: 184, last completed NCCL work: 183.
SH-IDC1-10-140-24-95:214031:214765 [0] NCCL INFO [Service thread] Connection closed by localRank 2
SH-IDC1-10-140-24-95:214033:214763 [2] NCCL INFO [Service thread] Connection closed by localRank 2
SH-IDC1-10-140-24-95:214033:214190 [0] NCCL INFO comm 0x99e59c0 rank 10 nranks 16 cudaDev 2 busId 65000 - Abort COMPLETE
[rank10]:[E726 10:59:44.260357278 ProcessGroupNCCL.cpp:621] [Rank 10] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[rank10]:[E726 10:59:44.260366596 ProcessGroupNCCL.cpp:627] [Rank 10] To avoid data inconsistency, we are taking the entire process down.
[rank10]:[E726 10:59:44.260431229 ProcessGroupNCCL.cpp:1515] [PG 0 (default_pg) Rank 10] Process group watchdog thread terminated with exception: NCCL error: remote process exited or there was a network error, NCCL version 2.20.5
ncclRemoteError: A call failed possibly due to a network error or a remote process exiting prematurely.
Last error:
NET/IB : Got completion from peer <IP_ADDRESS><27076> with status=5 opcode=129 len=2 vendor err 249 (Flush)
Exception raised from checkForNCCLErrorsInternal at /opt/conda/conda-bld/pytorch_1720538622298/work/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1892 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fc0e41f2f86 in /mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::checkForNCCLErrorsInternal(std::shared_ptr<c10d::NCCLComm>&) + 0x220 (0x7fc0e54df7f0 in /mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::WorkNCCL::checkAndSetException() + 0x7c (0x7fc0e54dfa3c in /mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::watchdogHandler() + 0x213 (0x7fc0e54e6923 in /mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7fc0e54e8d2c in /mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so)
frame #5: <unknown function> + 0xdbbf4 (0x7fc147bb6bf4 in /mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/lib/../../../.././libstdc++.so.6)
frame #6: <unknown function> + 0x7dd5 (0x7fc156e4edd5 in /lib64/libpthread.so.0)
frame #7: clone + 0x6d (0x7fc15646eead in /lib64/libc.so.6)

SH-IDC1-10-140-24-95:214031:214765 [0] NCCL INFO [Service thread] Connection closed by localRank 4
SH-IDC1-10-140-24-95:214035:214760 [4] NCCL INFO [Service thread] Connection closed by localRank 4
SH-IDC1-10-140-24-95:214031:214765 [0] NCCL INFO [Service thread] Connection closed by localRank 5
SH-IDC1-10-140-24-95:214038:214761 [5] NCCL INFO [Service thread] Connection closed by localRank 5
SH-IDC1-10-140-24-95:214031:214765 [0] NCCL INFO [Service thread] Connection closed by localRank 6
SH-IDC1-10-140-24-95:214040:214762 [6] NCCL INFO [Service thread] Connection closed by localRank 6
SH-IDC1-10-140-24-95:214031:214765 [0] NCCL INFO [Service thread] Connection closed by localRank 7
SH-IDC1-10-140-24-95:214042:214774 [7] NCCL INFO [Service thread] Connection closed by localRank 7
SH-IDC1-10-140-24-95:214035:214179 [0] NCCL INFO comm 0x93a8c00 rank 12 nranks 16 cudaDev 4 busId a3000 - Abort COMPLETE
[rank12]:[E726 10:59:44.284494485 ProcessGroupNCCL.cpp:621] [Rank 12] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[rank12]:[E726 10:59:44.284505756 ProcessGroupNCCL.cpp:627] [Rank 12] To avoid data inconsistency, we are taking the entire process down.
[rank12]:[E726 10:59:44.284578655 ProcessGroupNCCL.cpp:1515] [PG 0 (default_pg) Rank 12] Process group watchdog thread terminated with exception: NCCL error: remote process exited or there was a network error, NCCL version 2.20.5
ncclRemoteError: A call failed possibly due to a network error or a remote process exiting prematurely.
Last error:
NET/IB : Got completion from peer <IP_ADDRESS><5776> with status=5 opcode=129 len=4 vendor err 249 (Flush)
Exception raised from checkForNCCLErrorsInternal at /opt/conda/conda-bld/pytorch_1720538622298/work/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1892 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f0b05001f86 in /mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::checkForNCCLErrorsInternal(std::shared_ptr<c10d::NCCLComm>&) + 0x220 (0x7f0b062ee7f0 in /mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::WorkNCCL::checkAndSetException() + 0x7c (0x7f0b062eea3c in /mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::watchdogHandler() + 0x213 (0x7f0b062f5923 in /mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7f0b062f7d2c in /mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so)
frame #5: <unknown function> + 0xdbbf4 (0x7f0b689c5bf4 in /mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/lib/../../../.././libstdc++.so.6)
frame #6: <unknown function> + 0x7dd5 (0x7f0b77c5ddd5 in /lib64/libpthread.so.0)
frame #7: clone + 0x6d (0x7f0b7727dead in /lib64/libc.so.6)

SH-IDC1-10-140-24-95:214038:214193 [0] NCCL INFO comm 0xaa13c80 rank 13 nranks 16 cudaDev 5 busId a8000 - Abort COMPLETE
[rank13]:[E726 10:59:44.290108228 ProcessGroupNCCL.cpp:621] [Rank 13] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[rank13]:[E726 10:59:44.290121924 ProcessGroupNCCL.cpp:627] [Rank 13] To avoid data inconsistency, we are taking the entire process down.
[rank13]:[E726 10:59:44.290391618 ProcessGroupNCCL.cpp:1515] [PG 0 (default_pg) Rank 13] Process group watchdog thread terminated with exception: NCCL error: remote process exited or there was a network error, NCCL version 2.20.5
ncclRemoteError: A call failed possibly due to a network error or a remote process exiting prematurely.
Last error:
NET/IB : Got completion from peer <IP_ADDRESS><47174> with status=5 opcode=129 len=5 vendor err 249 (Flush)
Exception raised from checkForNCCLErrorsInternal at /opt/conda/conda-bld/pytorch_1720538622298/work/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1892 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f7832bd1f86 in /mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::checkForNCCLErrorsInternal(std::shared_ptr<c10d::NCCLComm>&) + 0x220 (0x7f7833ebe7f0 in /mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::WorkNCCL::checkAndSetException() + 0x7c (0x7f7833ebea3c in /mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::watchdogHandler() + 0x213 (0x7f7833ec5923 in /mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7f7833ec7d2c in /mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so)
frame #5: <unknown function> + 0xdbbf4 (0x7f7896595bf4 in /mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/lib/../../../.././libstdc++.so.6)
frame #6: <unknown function> + 0x7dd5 (0x7f78a582ddd5 in /lib64/libpthread.so.0)
frame #7: clone + 0x6d (0x7f78a4e4dead in /lib64/libc.so.6)

SH-IDC1-10-140-24-95:214040:214204 [0] NCCL INFO comm 0x9a22940 rank 14 nranks 16 cudaDev 6 busId e1000 - Abort COMPLETE
[rank14]:[E726 10:59:44.293817263 ProcessGroupNCCL.cpp:621] [Rank 14] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[rank14]:[E726 10:59:44.293827222 ProcessGroupNCCL.cpp:627] [Rank 14] To avoid data inconsistency, we are taking the entire process down.
[rank14]:[E726 10:59:44.293894821 ProcessGroupNCCL.cpp:1515] [PG 0 (default_pg) Rank 14] Process group watchdog thread terminated with exception: NCCL error: remote process exited or there was a network error, NCCL version 2.20.5
ncclRemoteError: A call failed possibly due to a network error or a remote process exiting prematurely.
Last error:
NET/IB : Got completion from peer <IP_ADDRESS><37762> with status=5 opcode=129 len=6 vendor err 249 (Flush)
Exception raised from checkForNCCLErrorsInternal at /opt/conda/conda-bld/pytorch_1720538622298/work/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1892 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f2019fe6f86 in /mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::checkForNCCLErrorsInternal(std::shared_ptr<c10d::NCCLComm>&) + 0x220 (0x7f201b2d37f0 in /mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::WorkNCCL::checkAndSetException() + 0x7c (0x7f201b2d3a3c in /mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::watchdogHandler() + 0x213 (0x7f201b2da923 in /mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7f201b2dcd2c in /mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so)
frame #5: <unknown function> + 0xdbbf4 (0x7f207d9aabf4 in /mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/lib/../../../.././libstdc++.so.6)
frame #6: <unknown function> + 0x7dd5 (0x7f208cc42dd5 in /lib64/libpthread.so.0)
frame #7: clone + 0x6d (0x7f208c262ead in /lib64/libc.so.6)

SH-IDC1-10-140-24-95:214042:214187 [0] NCCL INFO comm 0xaf1b6c0 rank 15 nranks 16 cudaDev 7 busId e7000 - Abort COMPLETE
[rank15]:[E726 10:59:44.300514320 ProcessGroupNCCL.cpp:621] [Rank 15] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[rank15]:[E726 10:59:44.300529779 ProcessGroupNCCL.cpp:627] [Rank 15] To avoid data inconsistency, we are taking the entire process down.
[rank15]:[E726 10:59:44.300604481 ProcessGroupNCCL.cpp:1515] [PG 0 (default_pg) Rank 15] Process group watchdog thread terminated with exception: NCCL error: remote process exited or there was a network error, NCCL version 2.20.5
ncclRemoteError: A call failed possibly due to a network error or a remote process exiting prematurely.
Last error:
NET/IB : Got completion from peer <IP_ADDRESS><61176> with status=5 opcode=129 len=7 vendor err 249 (Flush)
Exception raised from checkForNCCLErrorsInternal at /opt/conda/conda-bld/pytorch_1720538622298/work/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1892 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f3a48ee4f86 in /mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::checkForNCCLErrorsInternal(std::shared_ptr<c10d::NCCLComm>&) + 0x220 (0x7f3a4a1d17f0 in /mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::WorkNCCL::checkAndSetException() + 0x7c (0x7f3a4a1d1a3c in /mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::watchdogHandler() + 0x213 (0x7f3a4a1d8923 in /mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7f3a4a1dad2c in /mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so)
frame #5: <unknown function> + 0xdbbf4 (0x7f3aac8a8bf4 in /mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/lib/../../../.././libstdc++.so.6)
frame #6: <unknown function> + 0x7dd5 (0x7f3abbb40dd5 in /lib64/libpthread.so.0)
frame #7: clone + 0x6d (0x7f3abb160ead in /lib64/libc.so.6)

SH-IDC1-10-140-24-95:214031:214765 [0] NCCL INFO [Service thread] Connection closed by localRank 3
SH-IDC1-10-140-24-95:214034:214764 [3] NCCL INFO [Service thread] Connection closed by localRank 3
SH-IDC1-10-140-24-95:214034:214181 [0] NCCL INFO comm 0xdd87e550 rank 11 nranks 16 cudaDev 3 busId 6a000 - Abort COMPLETE
[rank11]:[E726 10:59:44.365679787 ProcessGroupNCCL.cpp:621] [Rank 11] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[rank11]:[E726 10:59:44.365693163 ProcessGroupNCCL.cpp:627] [Rank 11] To avoid data inconsistency, we are taking the entire process down.
[rank11]:[E726 10:59:44.365753788 ProcessGroupNCCL.cpp:1515] [PG 0 (default_pg) Rank 11] Process group watchdog thread terminated with exception: NCCL error: remote process exited or there was a network error, NCCL version 2.20.5
ncclRemoteError: A call failed possibly due to a network error or a remote process exiting prematurely.
Last error:
NET/IB : Got completion from peer <IP_ADDRESS><37760> with status=5 opcode=129 len=3 vendor err 249 (Flush)
Exception raised from checkForNCCLErrorsInternal at /opt/conda/conda-bld/pytorch_1720538622298/work/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1892 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f84811b2f86 in /mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::checkForNCCLErrorsInternal(std::shared_ptr<c10d::NCCLComm>&) + 0x220 (0x7f848249f7f0 in /mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::WorkNCCL::checkAndSetException() + 0x7c (0x7f848249fa3c in /mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::watchdogHandler() + 0x213 (0x7f84824a6923 in /mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7f84824a8d2c in /mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so)
frame #5: <unknown function> + 0xdbbf4 (0x7f84e4b76bf4 in /mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/lib/../../../.././libstdc++.so.6)
frame #6: <unknown function> + 0x7dd5 (0x7f84f3e0edd5 in /lib64/libpthread.so.0)
frame #7: clone + 0x6d (0x7f84f342eead in /lib64/libc.so.6)

W0726 10:59:45.447573<PHONE_NUMBER>13184 torch/distributed/elastic/multiprocessing/api.py:858] Sending process 214031 closing signal SIGTERM
W0726 10:59:45.448162<PHONE_NUMBER>13184 torch/distributed/elastic/multiprocessing/api.py:858] Sending process 214032 closing signal SIGTERM
W0726 10:59:45.504283<PHONE_NUMBER>13184 torch/distributed/elastic/multiprocessing/api.py:858] Sending process 214034 closing signal SIGTERM
W0726 10:59:45.509025<PHONE_NUMBER>13184 torch/distributed/elastic/multiprocessing/api.py:858] Sending process 214035 closing signal SIGTERM
W0726 10:59:45.542702<PHONE_NUMBER>13184 torch/distributed/elastic/multiprocessing/api.py:858] Sending process 214038 closing signal SIGTERM
W0726 10:59:45.546265<PHONE_NUMBER>13184 torch/distributed/elastic/multiprocessing/api.py:858] Sending process 214040 closing signal SIGTERM
W0726 10:59:45.588805<PHONE_NUMBER>13184 torch/distributed/elastic/multiprocessing/api.py:858] Sending process 214042 closing signal SIGTERM
E0726 10:59:49.951559<PHONE_NUMBER>13184 torch/distributed/elastic/multiprocessing/api.py:833] failed (exitcode: -6) local_rank: 2 (pid: 214033) of binary: /mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/bin/python
Traceback (most recent call last):
  File "/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/bin/accelerate", line 8, in <module>
    sys.exit(main())
  File "/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/accelerate/commands/accelerate_cli.py", line 48, in main
    args.func(args)
  File "/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/accelerate/commands/launch.py", line 1093, in launch_command
    multi_gpu_launcher(args)
  File "/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/accelerate/commands/launch.py", line 734, in multi_gpu_launcher
    distrib_run.run(args)
  File "/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/run.py", line 892, in run
    elastic_launch(
  File "/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 133, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
=======================================================
train_wrapper.py FAILED
-------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
-------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2024-07-26_10:59:45
  host      : SH-IDC1-10-140-24-95
  rank      : 10 (local_rank: 2)
  exitcode  : -6 (pid: 214033)
  error_file: <N/A>
  traceback : Signal 6 (SIGABRT) received by PID 214033
=======================================================
srun: error: SH-IDC1-10-140-24-95: task 0: Exited with exit code 1
W0726 10:59:50.786825<PHONE_NUMBER>84672 torch/distributed/elastic/multiprocessing/api.py:858] Sending process 147865 closing signal SIGTERM
W0726 10:59:50.787296<PHONE_NUMBER>84672 torch/distributed/elastic/multiprocessing/api.py:858] Sending process 147866 closing signal SIGTERM
W0726 10:59:50.842426<PHONE_NUMBER>84672 torch/distributed/elastic/multiprocessing/api.py:858] Sending process 147867 closing signal SIGTERM
W0726 10:59:50.868046<PHONE_NUMBER>84672 torch/distributed/elastic/multiprocessing/api.py:858] Sending process 147868 closing signal SIGTERM
W0726 10:59:50.893290<PHONE_NUMBER>84672 torch/distributed/elastic/multiprocessing/api.py:858] Sending process 147869 closing signal SIGTERM
W0726 10:59:50.923330<PHONE_NUMBER>84672 torch/distributed/elastic/multiprocessing/api.py:858] Sending process 147872 closing signal SIGTERM
W0726 10:59:50.944047<PHONE_NUMBER>84672 torch/distributed/elastic/multiprocessing/api.py:858] Sending process 147874 closing signal SIGTERM
W0726 10:59:50.962826<PHONE_NUMBER>84672 torch/distributed/elastic/multiprocessing/api.py:858] Sending process 147876 closing signal SIGTERM
W0726 10:59:52.793814<PHONE_NUMBER>99488 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1267] The node 'SH-IDC1-10-140-24-106_147766_0' has failed to send a keep-alive heartbeat to the rendezvous 'none' due to an error of type RendezvousConnectionError.
W0726 10:59:56.810442<PHONE_NUMBER>84672 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1218] The node 'SH-IDC1-10-140-24-106_147766_0' has failed to shutdown the rendezvous 'none' due to an error of type RendezvousConnectionError.
W0726 10:59:56.811984<PHONE_NUMBER>84672 torch/distributed/elastic/rendezvous/dynamic_rendezvous.py:1218] The node 'SH-IDC1-10-140-24-106_147766_0' has failed to shutdown the rendezvous 'none' due to an error of type RendezvousConnectionError.
Traceback (most recent call last):
  File "/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py", line 114, in _call_store
    return getattr(self._store, store_op)(*args, **kwargs)
torch.distributed.DistNetworkError: Connection reset by peer

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/bin/accelerate", line 8, in <module>
    sys.exit(main())
  File "/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/accelerate/commands/accelerate_cli.py", line 48, in main
    args.func(args)
  File "/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/accelerate/commands/launch.py", line 1093, in launch_command
    multi_gpu_launcher(args)
  File "/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/accelerate/commands/launch.py", line 734, in multi_gpu_launcher
    distrib_run.run(args)
  File "/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/run.py", line 892, in run
    elastic_launch(
  File "/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 133, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 255, in launch_agent
    result = agent.run()
  File "/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/elastic/metrics/api.py", line 124, in wrapper
    result = f(*args, **kwargs)
  File "/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/elastic/agent/server/api.py", line 680, in run
    result = self._invoke_run(role)
  File "/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/elastic/agent/server/api.py", line 867, in _invoke_run
    num_nodes_waiting = rdzv_handler.num_nodes_waiting()
  File "/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/elastic/rendezvous/dynamic_rendezvous.py", line 1189, in num_nodes_waiting
    self._state_holder.sync()
  File "/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/elastic/rendezvous/dynamic_rendezvous.py", line 428, in sync
    get_response = self._backend.get_state()
  File "/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py", line 74, in get_state
    base64_state: bytes = self._call_store("get", self._key)
  File "/mnt/hwfile/3dv/yhh/miniconda3/envs/pcdet-torch2.4.0-cuda11.8-py3.8/lib/python3.8/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py", line 116, in _call_store
    raise RendezvousConnectionError(
torch.distributed.elastic.rendezvous.api.RendezvousConnectionError: The connection to the C10d store has failed. See inner exception for details.
srun: error: SH-IDC1-10-140-24-106: task 1: Exited with exit code 1

change fsdp_state_dict_type: SHARDED_STATE_DICT to FULL_STATE_DICT can avoid this error. But when resuming the ckpt, multiple processes will occur on the same GPU.

I have the save issue, any suggestions?


Update image on Docker Hub to use Piwik 2.15.0

Piwik 2.15.0 is now out.  It has improvements for performance and to the UI.

and I'm on it ;), hope nothing breaks ;)

Pulled down the new image and everything seems to work. Thanks!

nice thanks for testing, I'll close this then.
and i should look for a automation of docker container creation...


Simplification rule involving List sort does not apply

test.k:
module TEST
  imports INT
  imports BOOL
  imports LIST

  configuration <k> $PGM:Pgm </k>

  syntax Pgm ::= foo() | bar()

  rule foo() => bar() ensures size(ListItem(?A:Int) ListItem(?B:Int) ?L:List) >Int 0
  rule [simplify-size]: size(ListItem(_) WS) => size(WS) +Int 1 [simplification]

endmodule

kompile test.k --backend haskell
krun -cPGM="foo()"

I expect to see the simplification rule labeled "simplify-size" to apply, but it does not. This is blocking pi^2's ability to modify the KEVM configuration. This applies to the legacy haskell backend. We either need a fix or some kind of workaround.

Here is what I get with --debug-attempted-equations:
kore-exec: [173020] Debug (DebugAttemptEquation):
    applying equation (label:  TEST.simplify-size ) at /home/dwightguth/test/test.k:11:25-11:64 to term:
        /* T Fn D Spa */
        Lblsize'LParUndsRParUnds'LIST'Unds'Int'Unds'List{}(
            /* T Fn D Sfa */
            Lbl'Unds'List'Unds'{}(
                /* T Fn D Sfa */
                /* InternalList: */ Lbl'Unds'List'Unds'{}(
                    LblListItem{}(
                        /* T Fn D Sfa */
                        /* Inj: */ inj{SortInt{}, SortKItem{}}(
                            /* T Fn D Sfa */ RuleVar'Ques'A:SortInt{}
                        )
                    ),
                    LblListItem{}(
                        /* T Fn D Sfa */
                        /* Inj: */ inj{SortInt{}, SortKItem{}}(
                            /* T Fn D Sfa */ RuleVar'Ques'B:SortInt{}
                        )
                    )
                ),
                /* T Fn D Sfa */ RuleVar'Ques'L:SortList{}
            )
        )
Context:
    (DebugAttemptEquation) while applying equation (label:  TEST.simplify-size ) at /home/dwightguth/test/test.k:11:25-11:64
kore-exec: [178422] Debug (DebugAttemptEquation):
    equation is not applicable:
    equation did not match term:  list lengths are not equal
Context:
    (DebugAttemptEquation) while applying equation (label:  TEST.simplify-size ) at /home/dwightguth/test/test.k:11:25-11:64

Seems to confirm it's a pattern matching but, but it doesn't have a lot to go on.

This indeed seems to be a list matching bug. The error is produced in patternMatch' function, in the case for lists.
A simple workaround would be two have a specialized simplification for the double-headed list:
  rule [simplify-size-one]: size(ListItem(_) WS) => size(WS) +Int 1 [simplification]
  rule [simplify-size-two]: size(ListItem(_) ListItem(_) WS) => size(WS) +Int 2 [simplification]

Would such a workaround work for you @dwightguth?
In the meantime, we should fix the list matching code.

That depends: would having a two element rule handle the case where the subject has three elements and a variable? If not, it's not really a powerful enough workaround.

I've tries this variation with three exsitentials:
module TEST
  imports INT
  imports BOOL
  imports LIST

  configuration <k> $PGM:Pgm </k>

  syntax Pgm ::= foo() | bar()

  rule foo() => bar() ensures size(ListItem(?A:Int) ListItem(?B:Int) ListItem(?C:Int) ?L:List) >Int 0

  rule [simplify-size]: size(L:List WS:List) => size(L) +Int size(WS) [simplification]

endmodule

And it works, producing the desired result:
  <k>
    bar ( ) ~> .K
  </k>
#And
  {
    true
  #Equals
    size ( ?L:List ) +Int 3 >Int 0
  }

Hi, this second variation was able to make progress, but it eventually ended up stuck on the exact same bug of needing to be able to match a list with an arbitrary number of elements plus a variable at the end against the pattern ListItem(_) L:List. In this latter example that I eventually ran into, the rule cannot be rewritten in the manner you describe.

Incidentally, I discovered another related bug:
The term (ListItem ( A:Int ) ListItem ( B:Int ) WS:List) [ 0 ] ought to reduce to I:Int but it does not. This one might better be solved via a LIST-SYMBOLIC module in the K prelude, however. We might want to discuss this in a meeting. Either way, the matching bug still needs either a workaround to handle subjects with arbitrary numbers of elements, or a fix.

@dwightguth git blame shows that you have written the current pattern code. I have little experience implementing pattern matching algorithms, but I am very keen to learn. Could we meet next week to look at the code together and figure out a plan to modify it so that it supports your use-case?


Release version 4.0

We added support for compression and a bunch of other improvements since 3.4. I think this is a good time to release v4.0 (all the more since I've had less time for websockets since a few weeks and I'm not sure that'll improve going forwards).
@cjerdonek @homerunisgood @mayeut Is there anything you'd like to get into that release? If so, please make sure there's an issue or PR tagged with the 4.0 milestone. A major release is a good time to make backwards-incompatible changes (even though we're trying to avoid these in general).

I'm planning to merge #301, #302 and #303 before making the release. Anything else?

Not that I can think of. Thanks for all your work!! 👍

Same for me, I'm pretty much done. I will deploy this version and see how it runs over time!
Thanks a lot for making that happen!

Done!

Something wrong with Travis CI settings? Only Windows wheels are uploaded to pypi.

Yes I'm investigating this right now.

I've pushed the tags again, Travis CI is building.

We have Linux and Windows wheels but no OS X wheels, see #304.
This is a modest concern as OS X is a dev platform only (I hope).

Hmm, but source package isn't uploaded either, so OS X users will not be able to upgrade to 4.0 for now.

I'm making a 4.0.1 version and trying to upload it correctly.
PyPI isn't my friend today...

I uploaded a sdist for 4.0.1, Linux and Windows wheels are there, I haven't repaired macOS wheel generation yet but I'm optimistic.

macOS wheels are there.

Indeed now it installs with wheel on OSX!
Strange thing though, on ubuntu 16.04 and python 3.5.2 it is still stuck to version 3.4???
(wsproxy) user@test.vm:~$ pip install -U websockets
Collecting websockets
  Using cached websockets-3.4-cp35-cp35m-manylinux1_x86_64.whl
Installing collected packages: websockets
Successfully installed websockets-3.4

Try with --no-cache-dir?


Readme suggestions

Hey @dzucker-tab, let's review this next week. I added some small changes to your README.md but overall it is fantastic. Thank you!

@dzucker-tab  I think we should also add some text on:

the X-Tableau-Auth header used to send the API key to Tableau after authentication (can be added to the "Advanced Usage" section)
explain that the pre-request script is what runs the auto-auth feature (added to the auto-auth section)
explain how the testing script works (added to "Advanced Usage")
perhaps record a quick video showing people around (add it early in README perhaps in the intro)

For contributions, suggest:

About the repository: This GitHub repository is only used for issue tracking. Use the Postman environment to make requests to your Tableau service, and tomake contributions or enhancements to the collection.

Clone or fork the Tableau REST API collection in the Salesforce Developers Postman workspace.
The collection's files are hosted on Postman collection and synchronized to this repository for version control.


crash in 'cellForItem:item'

I get a crash with this error:
*** Terminating app due to uncaught exception 'NSRangeException', reason: '*** -[__NSArrayM objectAtIndex:]: index 10 beyond bounds [0 .. 9]'
fixed it:
 - (UITableViewCell *)cellForItem:(id)item
 {
   NSIndexPath *indexPath = [self indexPathForItem:item];
-  return [self.tableView cellForRowAtIndexPath:indexPath];
+  UITableViewCell *cell = nil;
+  @try {
+    cell = [self.tableView cellForRowAtIndexPath:indexPath];
+  }
+  @catch (NSException *exception) {
+    cell = nil;
+  }
+  return cell;
 }

@rp-snt  I added an example project which is entirely written in Swift. It can be found at https://github.com/Augustyniak/RATreeView/tree/master/Examples. Just run example project and select RATreeViewBasicExampleSwift target to run it.
Please take a look at #83 for more details.
I hope that it helps.

Closing. Feel free to reopen if necessary.


Remove gcstub.

@rainers this is in your area of expertise. Are you ok with removing gcstub from Druntime?

I'm fine with removing it, but it's more @WalterBright s baby. See also https://issues.dlang.org/show_bug.cgi?id=17326

Probably worth mentioning that the "manual" gc (src/impl/manual/gc.d) is the same implementation as gcstub, or it was when I added it.
gcstub probably should have been removed when it was added.

@ZombineDev
Well if rainers is alright with it and @WalterBright doesn't have anything to say against it, can this be merged in along with the druntime pull request?

I think it's fine, it's been long enough and it's not like this is removing a working feature ; it's been broken for a year or more.

Maybe a bit of information how to implement something similar to the gcstub (with a big red warning about its limitations) in the wiki would be nice, i.e. not linking in the regular GC, but allowing to proxy it elsewhere. I believe linking in something hacky like this should work (not tested but I used something similar recently):
import core.thread;
import core.demangle;
import gc.gcinterface;
import gc.impl.manual.gc;

// access to private symbol
pragma(mangle, mangle!GC("gc.proxy.instance")) extern __gshared GC instance;

extern (C)
{
   // replace COMDATs in druntime lib
    void gc_init()
    {
        config.gc = "manual";
        ManualGC.initialize(instance);
        thread_init();
    }
    void gc_term()
    {
        thread_term();
        ManualGC.finalize(instance);
    }
}

@andralex Any thoughts? Walter seems to be busy.

Okay as no one is responding and this has been broken for quite a while and existing as a copy in src/impl/manual/gc.d too, I will set it to auto-merge.
Thanks @skl131313 for pushing this!

Maybe a bit of information how to implement something similar to the gcstub (with a big red warning about its limitations) in the wiki would be nice, i.e. not linking in the regular GC, but allowing to proxy it elsewhere.

I didn't find a good place in the wiki for this (maybe someone else did?). However, I believe this would be more than interesting for a new article in the @nogc series...

Thanks for merging it!


Cant deploy anymore my app

Does this occur consistently?
Repro steps:

Action: appService.Deploy
Error type: 409
Error Message: There is a deployment currently in progress. Please try again when it completes.
Version: 0.16.2
OS: win32
OS Release: 10.0.17134
Product: Visual Studio Code
Product Version: 1.41.0
Language: en

Call Stack
extension.bundle.js:323:392866extension.bundle.js:323:392866
s extension.bundle.js:218:111699
s extension.bundle.js:157:26069
extension.bundle.js:157:27311extension.bundle.js:157:27311
o extension.bundle.js:164:7076
extension.bundle.js:157:24982extension.bundle.js:157:24982
$._callback extension.bundle.js:1:579996
$.t._callback.t.callback.t.callback extension.bundle.js:194:5449
$.emit events.js:200:13
$.<anonymous> extension.bundle.js:194:18309
$.emit events.js:205:15
IncomingMessage.<anonymous> extension.bundle.js:194:17141
Object.onceWrapper events.js:288:20
IncomingMessage.emit events.js:205:15
endReadableNT _stream_readable.js:1154:12
processTicksAndRejections task_queues.js:84:9

Sorry for the delayed response.  I'm investigating this, but unfortunately, this is most likely a platform issue, so I'm not sure how much control we have over it.

Unfortunately we don't have enough info to investigate on the VS Code side. I'd recommend getting support directly from the Azure team - there should be options in the portal to do so


Plugin E2E: Change FE feature toggles during test session

What this PR does / why we need it:
This is a bit experimental, and I'm not sure we want to go down this road so feedback is much appreciated.
I foresee that many plugin authors are going to want to test their plugins with certain feature toggles both enabled and disabled. Currently, the only way to change the value of a feature toggle in CI is by setting the GF_FEATURE_TOGGLES_ENABLE environment variable when starting the grafana container in the github workflow. Then they can use the isFeatureToggleEnabled fixture inside a test to check whether a certain feature is enabled. This works, but in order to try different values for a certain feature, they'll have to start multiple versions of Grafana in the workflow. This is complicated and it will also demand more resources.
As you know, Grafana feature toggles are being passed from Grafana backend to the frontend by setting window.grafanaBootData.settings.featureToggles. Playwright provide some guidelines for how to mock browser apis, such as the window object. In this PR, I'm adding a addInitScript function that is executed whenever the page is navigated. The function will poll for the existence of window.grafanaBootData, and when it exists it will override and/or add features that are defined in the playwright test session. The plugin-e2e consumer can then define their feature toggles in the playwright.config.ts like this:
  use: {
    baseURL: 'http://localhost:3000',
    featureToggles: {
      redshiftAsyncQueryDataSupport: false,
    },
    ...
  },

Of override the feature toggle for a certain file:
#queryEditor.spec.ts
test.use({
  featureToggles: {
    redshiftAsyncQueryDataSupport: true,
  },
});

I know this solution is kind of brittle and I wouldn't raise the PR if it wasn't for the fact the problem feels important to solve.
Which issue(s) this PR fixes:

Fixes #
Special notes for your reviewer:

@sunker We have a mechanism to enable and disable feature toggles in frontend already that uses local storage. you can see how it works here https://github.com/grafana/grafana/blob/4fa6bad7c03948cd648f784283bc60d7b2199ad3/packages/grafana-runtime/src/config.ts#L217

@sunker We have a mechanism to enable and disable feature toggles in frontend already that uses local storage. you can see how it works here so there's no need to write code that modifies the grafanaBootData https://github.com/grafana/grafana/blob/4fa6bad7c03948cd648f784283bc60d7b2199ad3/packages/grafana-runtime/src/config.ts#L217

Oh I did not know that. Thanks @academo, I'll try that instead.

Thanks for great input @mckn!

Use the following API to update the feature toggle prior to navigating to the page you want to test (https://github.com/grafana/grafana/blob/main/pkg/api/featuremgmt.go#L48)

Great, I did not know we had this API either.
One concern that I have with using this API is that it was introduced in Grafana 10.2.0. So far, all features in plugin-e2e are backwards compatible to Grafana 8.3.0. We could probably limit this feature in plugin-e2e to Grafana versions >=10.2.0, but it would not be ideal. Actually the same issue applies to using the overrideFeatureTogglesFromLocalStorage method @academo as it was introduced in Grafana 10.1.0.

On the other hand, if we update the value across the entire stack we might have problems running tests affected by the same toggle in parallel.

I think this is a very valid concern. Just to circumvent this problem, I wonder if it's best to limit this capability in plugin-e2e to only change the frontend part of feature toggle. Maybe it's fine as long as we are very clear about that in the docs? If a plugin authors needs to tests a feature toggle across the entire stack, it needs to be defined when configuring the Grafana instance.

Thanks for great input @mckn!

Use the following API to update the feature toggle prior to navigating to the page you want to test (https://github.com/grafana/grafana/blob/main/pkg/api/featuremgmt.go#L48)

Great, I did not know we had this API either.
One concern that I have with using this API is that it was introduced in Grafana 10.2.0. So far, all features in plugin-e2e are backwards compatible to Grafana 8.3.0. We could probably limit this feature in plugin-e2e to Grafana versions >=10.2.0, but it would not be ideal. Actually the same issue applies to using the overrideFeatureTogglesFromLocalStorage method @academo as it was introduced in Grafana 10.1.0.

On the other hand, if we update the value across the entire stack we might have problems running tests affected by the same toggle in parallel.

I think this is a very valid concern. Just to circumvent this problem, I wonder if it's best to limit this capability in plugin-e2e to only change the frontend part of feature toggle. Maybe it's fine as long as we are very clear about that in the docs? If a plugin authors needs to tests a feature toggle across the entire stack, it needs to be defined when configuring the Grafana instance.

That would work! An alternative would be to only do the frontend part of the toggle in versions earlier than 10.2.0? If that would be a way forward.

An alternative would be to only do the frontend part of the toggle in versions earlier than 10.2.0? If that would be a way forward.

Think it could become a bit confusing. Also, then they may have to write different test code for different versions of Grafana, which is not ideal.

@academo this PR should be ready for review now. Now I'm updating local storage in case Grafana version => 10.1.0. If not, I'm updating the bootData directly.


Add GenericByteArray (#2946)

Which issue does this PR close?

Closes #2946
Rationale for this change

See ticket
What changes are included in this PR?

Defines a GenericByteArray and migrates some methods across, subsequent PRs can then work on eliminating duplication
Are there any user-facing changes?
No :tada:

I intend to merge this this evening, i.e. in about 6 hours

Removed api-change label as this is not a breaking change


Update nightly report track

fix nightly report issue
add on-demand test track
1 model launch for on-demand

@chuanqi129 This is ready for review. https://github.com/intel/torch-xpu-ops/issues/426


fix: use requestPath for OpenAPI spec in Swagger Options

Fixes #161.
This will also work when you don't specify any prefix to Elysia but uses Traefik Middlewares to strip /api prefix too.

This doesn't work, swagger ui and scalar should append from the browser URL, not server.

Updated. Now this work with externally set prefix (from reverse proxy), but I'm not sure to get the root Elysia app prefix after your change in ebf0e542695ce92bcd940634622ba20f65921b9b where elysia-swagger creates it's own Elysia instance.

Just want to bump this, I use supabase functions and have a prefix on the root elysia instance, so my swagger ends up living at /functions/v1/api/swagger/json but the browser tries to query for just /swagger/json, dropping both the prefix and externally set routing components.

Is there any expected merge date for this? I just hit this hurdle when using prefix and using the documentation since our projects are separated by services and it fetches the wrong json url


Member leaves the stream event missing

https://github.com/zerodytrash/TikTok-Live-Connector#member
This event notifies when member joins the stream but there is no way to notify when member leaves
Can you please fix it?

https://github.com/zerodytrash/TikTok-Live-Connector#member
This event notifies when member joins the stream but there is no way to notify when member leaves
Can you please fix it?

This is a TikTok problem. There is no leave stream event coming from TikTok.

correct.


Complete br element on README

Hey just for add the missing part of the br in README.md
Hope you found it usefull ^^

Shame on me i didn't figure its a br (empty element) 🤣
But the br tag must be an open tag ( without /) unless the document is xhtml type...
But thx for your WC it was usefull to us 👌

Hm I think <br /> is compatible with both XHTML and HTML. Therefore I am inclined to keep it like it is. Is the non-closing element breaking something for you?
Glad this element is useful for you :smile:

Closing this per above comment. Thanks again for your interest in the project!


is it possible to create a special directory color ?

Hi and thank you for powerlevel9k, very nice prompt (i use it with zsh, it is perfect).
I would like to create a special directory color.
For example, i can define a POWERLEVEL9K_DIR_HOME_BACKGOUND color...
In the same way, i would like to define a kind of POWERLEVEL9K_DIR_HTTP_BACKGROUND color for directories where user is "http" (for http user), but not the home directory of http user. Is it possible and how to do that please ?

Would a custom prompt segment work for you instead of modifying a core segment?
zsh_dir_owner(){
  owner=$(stat -c "%U" .)
  if [[ "$owner" == 'www-data' ]]; then
    echo $owner
  fi
}
POWERLEVEL9K_CUSTOM_DIR_OWNER="zsh_dir_owner"

POWERLEVEL9K_LEFT_PROMPT_ELEMENTS=($POWERLEVEL9K_LEFT_PROMPT_ELEMENTS custom_dir_owner)
POWERLEVEL9K_CUSTOM_DIR_OWNER_BACKGROUND=yellow
POWERLEVEL9K_CUSTOM_DIR_OWNER_FORGROUND=black

You could even change the check to something like this to show the segment if you're in a directory owned by anyone other than the current user:
  if [[ "$owner" != "$USER" ]]; then
    echo $owner
  fi


It is considering Tailwind Classes as custom classes

First of all, thank you very much for this plugin. I'm dropping the prettier plugin in favor of this for my entire team.
Describe the bug
The projects I work on usually define or extend the color palette in the tailwind.config.js. You could have a color called primary that generates bg-primary for example.
Those classes are being treated as custom and being moved to the beginning next to the actual custom ones. In the example below I added a bg-red-100 and a text-secondary as well as a real custom class and you will notice that the text-secondary is now the first one, behind the custom class.
To Reproduce

Customize your color palette in the TailwindCSS config file
Try to use your color class
See error
Double-check your compiled CSS file for order
Confirm error

Expected behavior
I would expect it to know that it is a color class and move it next to the bg-red-100. I also checked my compiled CSS file and the bg-red-100 was on line 516 and text-secondary was on line 527 so in theory, they should go together.
Screenshots

Desktop:

OS: iOS
VSCode version: 1.85.2
TW Version: 3.4.1

@fclaussen Thank you for submitting this. I can't seem to replicate locally, I get the proper reordering when following your reproduction steps. Would it be possible for you to create a small workspace where you still see the issue that you can submit? Another thing that might be at play is other tailwind plugins you have installed.
I want to make sure this is no longer an issue for you or your team, I just need some additional detail to determine the cause of the discrepancy.

Hi @Trapfether
Absolutely, what would be the best way of sending you this? What files are you looking for?

You can make a small zip file and include it as an attachment to a comment, alternatively a small repo is always acceptable. I would be looking for the general structure of the workspace where the issue is occuring and the an example file where the issue is present. Once that is provided, I can narrow down where the issue is coming from.
I also pushed an update this morning that allows for specifying a custom location for the tailwind.config.js file. (I forgot to ask you where your config file was relative to the impacted file). So if your config file is not at the base workspace directory, you can override this setting to point to your desired location.