[Bug] torch.distributed.DistBackendError: NCCL error in  docker.io/vllm/vllm-openai:v0.6.4.post1

Your current environment

The output of `python collect_env.py`
Collecting environment information...
PyTorch version: 2.5.1+cu124
Is debug build: False
CUDA used to build PyTorch: 12.4
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.4 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.35

Python version: 3.12.7 (main, Oct  1 2024, 08:52:12) [GCC 11.4.0] (64-bit runtime)
Python platform: Linux-5.15.0-125-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
GPU 0: NVIDIA H100 80GB HBM3
GPU 1: NVIDIA H100 80GB HBM3
GPU 2: NVIDIA H100 80GB HBM3
GPU 3: NVIDIA H100 80GB HBM3

Nvidia driver version: 565.57.01
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                         x86_64
CPU op-mode(s):                       32-bit, 64-bit
Address sizes:                        52 bits physical, 57 bits virtual
Byte Order:                           Little Endian
CPU(s):                               64
On-line CPU(s) list:                  0-63
Vendor ID:                            AuthenticAMD
Model name:                           AMD EPYC 9454 48-Core Processor
CPU family:                           25
Model:                                17
Thread(s) per core:                   1
Core(s) per socket:                   32
Socket(s):                            2
Stepping:                             1
BogoMIPS:                             5500.00
Flags:                                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm rep_good nopl cpuid extd_apicid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy svm cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw perfctr_core invpcid_single ssbd ibrs ibpb stibp vmmcall fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves avx512_bf16 clzero xsaveerptr wbnoinvd arat npt nrip_save avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq la57 rdpid fsrm arch_capabilities
Virtualization:                       AMD-V
Hypervisor vendor:                    KVM
Virtualization type:                  full
L1d cache:                            4 MiB (64 instances)
L1i cache:                            4 MiB (64 instances)
L2 cache:                             32 MiB (64 instances)
L3 cache:                             1 GiB (64 instances)
NUMA node(s):                         2
NUMA node0 CPU(s):                    0-31
NUMA node1 CPU(s):                    32-63
Vulnerability Gather data sampling:   Not affected
Vulnerability Itlb multihit:          Not affected
Vulnerability L1tf:                   Not affected
Vulnerability Mds:                    Not affected
Vulnerability Meltdown:               Not affected
Vulnerability Mmio stale data:        Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed:               Not affected
Vulnerability Spec rstack overflow:   Mitigation; safe RET
Vulnerability Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:             Mitigation; Retpolines; IBPB conditional; IBRS_FW; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds:                  Not affected
Vulnerability Tsx async abort:        Not affected

Versions of relevant libraries:
[pip3] flashinfer==0.1.6+cu121torch2.4
[pip3] numpy==1.26.4
[pip3] nvidia-cublas-cu12==<IP_ADDRESS>
[pip3] nvidia-cuda-cupti-cu12==12.4.127
[pip3] nvidia-cuda-nvrtc-cu12==12.4.127
[pip3] nvidia-cuda-runtime-cu12==12.4.127
[pip3] nvidia-cudnn-cu12==<IP_ADDRESS>
[pip3] nvidia-cufft-cu12==<IP_ADDRESS>
[pip3] nvidia-curand-cu12==<IP_ADDRESS>
[pip3] nvidia-cusolver-cu12==<IP_ADDRESS>
[pip3] nvidia-cusparse-cu12==<IP_ADDRESS>
[pip3] nvidia-ml-py==12.560.30
[pip3] nvidia-nccl-cu12==2.21.5
[pip3] nvidia-nvjitlink-cu12==12.4.127
[pip3] nvidia-nvtx-cu12==12.4.127
[pip3] pyzmq==26.2.0
[pip3] torch==2.5.1
[pip3] torchvision==0.20.1
[pip3] transformers==4.46.2
[pip3] triton==3.1.0
[conda] Could not collect
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.6.4.post1
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0	GPU1	GPU2	GPU3	NIC0	NIC1	NIC2	NIC3	NIC4	NIC5	NIC6	NIC7	CPU Affinity	NUMA Affinity	GPU NUMA ID
GPU0	 X 	NV18	NV18	NV18	PHB	PHB	PHB	PHB	PHB	PHB	PHB	PHB	0-63	0-1		N/A
GPU1	NV18	 X 	NV18	NV18	PHB	PHB	PHB	PHB	PHB	PHB	PHB	PHB	0-63	0-1		N/A
GPU2	NV18	NV18	 X 	NV18	PHB	PHB	PHB	PHB	PHB	PHB	PHB	PHB	0-63	0-1		N/A
GPU3	NV18	NV18	NV18	 X 	PHB	PHB	PHB	PHB	PHB	PHB	PHB	PHB	0-63	0-1		N/A
NIC0	PHB	PHB	PHB	PHB	 X 	PHB	PHB	PHB	PHB	PHB	PHB	PHB
NIC1	PHB	PHB	PHB	PHB	PHB	 X 	PHB	PHB	PHB	PHB	PHB	PHB
NIC2	PHB	PHB	PHB	PHB	PHB	PHB	 X 	PHB	PHB	PHB	PHB	PHB
NIC3	PHB	PHB	PHB	PHB	PHB	PHB	PHB	 X 	PHB	PHB	PHB	PHB
NIC4	PHB	PHB	PHB	PHB	PHB	PHB	PHB	PHB	 X 	PHB	PHB	PHB
NIC5	PHB	PHB	PHB	PHB	PHB	PHB	PHB	PHB	PHB	 X 	PHB	PHB
NIC6	PHB	PHB	PHB	PHB	PHB	PHB	PHB	PHB	PHB	PHB	 X 	PHB
NIC7	PHB	PHB	PHB	PHB	PHB	PHB	PHB	PHB	PHB	PHB	PHB	 X

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: mlx5_0
  NIC1: mlx5_1
  NIC2: mlx5_2
  NIC3: mlx5_3
  NIC4: mlx5_4
  NIC5: mlx5_5
  NIC6: mlx5_6
  NIC7: mlx5_7

NVIDIA_VISIBLE_DEVICES=all
NVIDIA_REQUIRE_CUDA=cuda>=12.4 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 brand=tesla,driver>=535,driver<536 brand=unknown,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=geforce,driver>=535,driver<536 brand=geforcertx,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=titan,driver>=535,driver<536 brand=titanrtx,driver>=535,driver<536
NVIDIA_DRIVER_CAPABILITIES=compute,utility
VLLM_USAGE_SOURCE=production-docker-image
CUDA_VERSION=12.4.1
LD_LIBRARY_PATH=/usr/local/lib/python3.12/dist-packages/cv2/../../lib64:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
CUDA_MODULE_LOADING=LAZY

Model Input Dumps
No response
🐛 Describe the bug
I tried to run model in docker.io/vllm/vllm-openai:v0.6.4.post1
docker.io/vllm/vllm-openai:latest  \
 --model Qwen/Qwen2-VL-72B-Instruct  \
  --served-model-name "Qwen2-VL-72B"  \
  --port 8003 \
 --gpu-memory-utilization 0.95 \
 --tensor-parallel-size 4

and got
INFO 11-19 06:08:46 utils.py:961] Found nccl from library libnccl.so.2
INFO 11-19 06:08:46 pynccl.py:69] vLLM is using nccl==2.21.5
(VllmWorkerProcess pid=409) INFO 11-19 06:08:46 utils.py:961] Found nccl from library libnccl.so.2
(VllmWorkerProcess pid=409) INFO 11-19 06:08:46 pynccl.py:69] vLLM is using nccl==2.21.5
(VllmWorkerProcess pid=411) INFO 11-19 06:08:46 utils.py:961] Found nccl from library libnccl.so.2
(VllmWorkerProcess pid=411) INFO 11-19 06:08:46 pynccl.py:69] vLLM is using nccl==2.21.5
(VllmWorkerProcess pid=409) ERROR 11-19 06:08:46 multiproc_worker_utils.py:229] Exception in worker VllmWorkerProcess while processing method init_device.
(VllmWorkerProcess pid=409) ERROR 11-19 06:08:46 multiproc_worker_utils.py:229] Traceback (most recent call last):
(VllmWorkerProcess pid=409) ERROR 11-19 06:08:46 multiproc_worker_utils.py:229]   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/multiproc_worker_utils.py", line 223, in _run_worker_process
(VllmWorkerProcess pid=409) ERROR 11-19 06:08:46 multiproc_worker_utils.py:229]     output = executor(*args, **kwargs)
(VllmWorkerProcess pid=409) ERROR 11-19 06:08:46 multiproc_worker_utils.py:229]              ^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=409) ERROR 11-19 06:08:46 multiproc_worker_utils.py:229]   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker.py", line 145, in init_device
(VllmWorkerProcess pid=409) ERROR 11-19 06:08:46 multiproc_worker_utils.py:229]     init_worker_distributed_environment(self.parallel_config, self.rank,
(VllmWorkerProcess pid=409) ERROR 11-19 06:08:46 multiproc_worker_utils.py:229]   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker.py", line 462, in init_worker_distributed_environment
(VllmWorkerProcess pid=409) ERROR 11-19 06:08:46 multiproc_worker_utils.py:229]     ensure_model_parallel_initialized(parallel_config.tensor_parallel_size,
(VllmWorkerProcess pid=409) ERROR 11-19 06:08:46 multiproc_worker_utils.py:229]   File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 1105, in ensure_model_parallel_initialized
(VllmWorkerProcess pid=409) ERROR 11-19 06:08:46 multiproc_worker_utils.py:229]     initialize_model_parallel(tensor_model_parallel_size,
(VllmWorkerProcess pid=409) ERROR 11-19 06:08:46 multiproc_worker_utils.py:229]   File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 1069, in initialize_model_parallel
(VllmWorkerProcess pid=409) ERROR 11-19 06:08:46 multiproc_worker_utils.py:229]     _TP = init_model_parallel_group(group_ranks,
(VllmWorkerProcess pid=409) ERROR 11-19 06:08:46 multiproc_worker_utils.py:229]           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=409) ERROR 11-19 06:08:46 multiproc_worker_utils.py:229]   File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 908, in init_model_parallel_group
(VllmWorkerProcess pid=409) ERROR 11-19 06:08:46 multiproc_worker_utils.py:229]     return GroupCoordinator(
(VllmWorkerProcess pid=409) ERROR 11-19 06:08:46 multiproc_worker_utils.py:229]            ^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=409) ERROR 11-19 06:08:46 multiproc_worker_utils.py:229]   File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 228, in __init__
(VllmWorkerProcess pid=409) ERROR 11-19 06:08:46 multiproc_worker_utils.py:229]     self.pynccl_comm = PyNcclCommunicator(
(VllmWorkerProcess pid=409) ERROR 11-19 06:08:46 multiproc_worker_utils.py:229]                        ^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=409) ERROR 11-19 06:08:46 multiproc_worker_utils.py:229]   File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl.py", line 99, in __init__
(VllmWorkerProcess pid=409) ERROR 11-19 06:08:46 multiproc_worker_utils.py:229]     self.comm: ncclComm_t = self.nccl.ncclCommInitRank(
(VllmWorkerProcess pid=409) ERROR 11-19 06:08:46 multiproc_worker_utils.py:229]                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=411) ERROR 11-19 06:08:46 multiproc_worker_utils.py:229] Exception in worker VllmWorkerProcess while processing method init_device.
(VllmWorkerProcess pid=409) ERROR 11-19 06:08:46 multiproc_worker_utils.py:229]   File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl_wrapper.py", line 244, in ncclCommInitRank
(VllmWorkerProcess pid=411) ERROR 11-19 06:08:46 multiproc_worker_utils.py:229] Traceback (most recent call last):
(VllmWorkerProcess pid=409) ERROR 11-19 06:08:46 multiproc_worker_utils.py:229]     self.NCCL_CHECK(self._funcs["ncclCommInitRank"](ctypes.byref(comm),
(VllmWorkerProcess pid=411) ERROR 11-19 06:08:46 multiproc_worker_utils.py:229]   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/multiproc_worker_utils.py", line 223, in _run_worker_process
(VllmWorkerProcess pid=409) ERROR 11-19 06:08:46 multiproc_worker_utils.py:229]   File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl_wrapper.py", line 223, in NCCL_CHECK
(VllmWorkerProcess pid=411) ERROR 11-19 06:08:46 multiproc_worker_utils.py:229]     output = executor(*args, **kwargs)
(VllmWorkerProcess pid=409) ERROR 11-19 06:08:46 multiproc_worker_utils.py:229]     raise RuntimeError(f"NCCL error: {error_str}")
(VllmWorkerProcess pid=411) ERROR 11-19 06:08:46 multiproc_worker_utils.py:229]              ^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=409) ERROR 11-19 06:08:46 multiproc_worker_utils.py:229] RuntimeError: NCCL error: internal error - please report this issue to the NCCL developers
(VllmWorkerProcess pid=411) ERROR 11-19 06:08:46 multiproc_worker_utils.py:229]   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker.py", line 145, in init_device
(VllmWorkerProcess pid=411) ERROR 11-19 06:08:46 multiproc_worker_utils.py:229]     init_worker_distributed_environment(self.parallel_config, self.rank,
(VllmWorkerProcess pid=411) ERROR 11-19 06:08:46 multiproc_worker_utils.py:229]   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker.py", line 462, in init_worker_distributed_environment
(VllmWorkerProcess pid=411) ERROR 11-19 06:08:46 multiproc_worker_utils.py:229]     ensure_model_parallel_initialized(parallel_config.tensor_parallel_size,
(VllmWorkerProcess pid=411) ERROR 11-19 06:08:46 multiproc_worker_utils.py:229]   File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 1105, in ensure_model_parallel_initialized
(VllmWorkerProcess pid=411) ERROR 11-19 06:08:46 multiproc_worker_utils.py:229]     initialize_model_parallel(tensor_model_parallel_size,
(VllmWorkerProcess pid=411) ERROR 11-19 06:08:46 multiproc_worker_utils.py:229]   File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 1069, in initialize_model_parallel
(VllmWorkerProcess pid=411) ERROR 11-19 06:08:46 multiproc_worker_utils.py:229]     _TP = init_model_parallel_group(group_ranks,
(VllmWorkerProcess pid=411) ERROR 11-19 06:08:46 multiproc_worker_utils.py:229]           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=411) ERROR 11-19 06:08:46 multiproc_worker_utils.py:229]   File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 908, in init_model_parallel_group
(VllmWorkerProcess pid=411) ERROR 11-19 06:08:46 multiproc_worker_utils.py:229]     return GroupCoordinator(
(VllmWorkerProcess pid=411) ERROR 11-19 06:08:46 multiproc_worker_utils.py:229]            ^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=411) ERROR 11-19 06:08:46 multiproc_worker_utils.py:229]   File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 228, in __init__
(VllmWorkerProcess pid=411) ERROR 11-19 06:08:46 multiproc_worker_utils.py:229]     self.pynccl_comm = PyNcclCommunicator(
(VllmWorkerProcess pid=411) ERROR 11-19 06:08:46 multiproc_worker_utils.py:229]                        ^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=411) ERROR 11-19 06:08:46 multiproc_worker_utils.py:229]   File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl.py", line 99, in __init__
(VllmWorkerProcess pid=411) ERROR 11-19 06:08:46 multiproc_worker_utils.py:229]     self.comm: ncclComm_t = self.nccl.ncclCommInitRank(
(VllmWorkerProcess pid=411) ERROR 11-19 06:08:46 multiproc_worker_utils.py:229]                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=411) ERROR 11-19 06:08:46 multiproc_worker_utils.py:229]   File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl_wrapper.py", line 244, in ncclCommInitRank
(VllmWorkerProcess pid=411) ERROR 11-19 06:08:46 multiproc_worker_utils.py:229]     self.NCCL_CHECK(self._funcs["ncclCommInitRank"](ctypes.byref(comm),
(VllmWorkerProcess pid=411) ERROR 11-19 06:08:46 multiproc_worker_utils.py:229]   File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl_wrapper.py", line 223, in NCCL_CHECK
(VllmWorkerProcess pid=411) ERROR 11-19 06:08:46 multiproc_worker_utils.py:229]     raise RuntimeError(f"NCCL error: {error_str}")
(VllmWorkerProcess pid=411) ERROR 11-19 06:08:46 multiproc_worker_utils.py:229] RuntimeError: NCCL error: internal error - please report this issue to the NCCL developers
ERROR 11-19 06:08:46 engine.py:366] NCCL error: internal error - please report this issue to the NCCL developers

I have the latest nvidia drivers
root@h100-2:~/victor# nvidia-smi
Tue Nov 19 14:49:02 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 565.57.01              Driver Version: 565.57.01      CUDA Version: 12.7     |
|-----------------------------------------+------------------------+----------------------+

I run https://docs.vllm.ai/en/latest/getting_started/debugging.html#incorrect-hardware-driver and got
W1119 06:29:00.316000 347 torch/distributed/run.py:793]
W1119 06:29:00.316000 347 torch/distributed/run.py:793] *****************************************
W1119 06:29:00.316000 347 torch/distributed/run.py:793] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
W1119 06:29:00.316000 347 torch/distributed/run.py:793] *****************************************
h100-2:412:412 [0] NCCL INFO Bootstrap : Using enp3s0:<IP_ADDRESS><0>
h100-2:412:412 [0] NCCL INFO NET/Plugin: No plugin found (libnccl-net.so)
h100-2:412:412 [0] NCCL INFO NET/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-net.so
h100-2:412:412 [0] NCCL INFO NET/Plugin: Using internal network plugin.
h100-2:412:412 [0] NCCL INFO cudaDriverVersion 12070
NCCL version 2.21.5+cuda12.4
h100-2:413:413 [1] NCCL INFO cudaDriverVersion 12070
h100-2:413:413 [1] NCCL INFO Bootstrap : Using enp3s0:<IP_ADDRESS><0>
h100-2:413:413 [1] NCCL INFO NET/Plugin: No plugin found (libnccl-net.so)
h100-2:413:413 [1] NCCL INFO NET/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-net.so
h100-2:413:413 [1] NCCL INFO NET/Plugin: Using internal network plugin.
h100-2:415:415 [3] NCCL INFO cudaDriverVersion 12070
h100-2:414:414 [2] NCCL INFO cudaDriverVersion 12070
h100-2:415:415 [3] NCCL INFO Bootstrap : Using enp3s0:<IP_ADDRESS><0>
h100-2:415:415 [3] NCCL INFO NET/Plugin: No plugin found (libnccl-net.so)
h100-2:415:415 [3] NCCL INFO NET/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-net.so
h100-2:415:415 [3] NCCL INFO NET/Plugin: Using internal network plugin.
h100-2:414:414 [2] NCCL INFO Bootstrap : Using enp3s0:<IP_ADDRESS><0>
h100-2:414:414 [2] NCCL INFO NET/Plugin: No plugin found (libnccl-net.so)
h100-2:414:414 [2] NCCL INFO NET/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-net.so
h100-2:414:414 [2] NCCL INFO NET/Plugin: Using internal network plugin.
h100-2:415:435 [3] NCCL INFO NCCL_IB_HCA set to mlx5_0:1,mlx5_1:1,mlx5_2:1,mlx5_3:1,mlx5_4:1,mlx5_5:1,mlx5_6:1,mlx5_7:1
h100-2:412:433 [0] NCCL INFO NCCL_IB_HCA set to mlx5_0:1,mlx5_1:1,mlx5_2:1,mlx5_3:1,mlx5_4:1,mlx5_5:1,mlx5_6:1,mlx5_7:1
h100-2:414:436 [2] NCCL INFO NCCL_IB_HCA set to mlx5_0:1,mlx5_1:1,mlx5_2:1,mlx5_3:1,mlx5_4:1,mlx5_5:1,mlx5_6:1,mlx5_7:1
h100-2:413:434 [1] NCCL INFO NCCL_IB_HCA set to mlx5_0:1,mlx5_1:1,mlx5_2:1,mlx5_3:1,mlx5_4:1,mlx5_5:1,mlx5_6:1,mlx5_7:1
h100-2:415:435 [3] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [4]mlx5_4:1/IB [5]mlx5_5:1/IB [6]mlx5_6:1/IB [7]mlx5_7:1/IB [RO]; OOB enp3s0:<IP_ADDRESS><0>
h100-2:415:435 [3] NCCL INFO Using non-device net plugin version 0
h100-2:415:435 [3] NCCL INFO Using network IB
h100-2:414:436 [2] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [4]mlx5_4:1/IB [5]mlx5_5:1/IB [6]mlx5_6:1/IB [7]mlx5_7:1/IB [RO]; OOB enp3s0:<IP_ADDRESS><0>
h100-2:414:436 [2] NCCL INFO Using non-device net plugin version 0
h100-2:414:436 [2] NCCL INFO Using network IB
h100-2:412:433 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [4]mlx5_4:1/IB [5]mlx5_5:1/IB [6]mlx5_6:1/IB [7]mlx5_7:1/IB [RO]; OOB enp3s0:<IP_ADDRESS><0>
h100-2:412:433 [0] NCCL INFO Using non-device net plugin version 0
h100-2:412:433 [0] NCCL INFO Using network IB
h100-2:413:434 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [4]mlx5_4:1/IB [5]mlx5_5:1/IB [6]mlx5_6:1/IB [7]mlx5_7:1/IB [RO]; OOB enp3s0:<IP_ADDRESS><0>
h100-2:413:434 [1] NCCL INFO Using non-device net plugin version 0
h100-2:413:434 [1] NCCL INFO Using network IB
h100-2:415:435 [3] NCCL INFO DMA-BUF is available on GPU device 3
h100-2:414:436 [2] NCCL INFO DMA-BUF is available on GPU device 2
h100-2:412:433 [0] NCCL INFO DMA-BUF is available on GPU device 0
h100-2:413:434 [1] NCCL INFO DMA-BUF is available on GPU device 1
h100-2:415:435 [3] NCCL INFO ncclCommInitRank comm 0x41a22970 rank 3 nranks 4 cudaDev 3 nvmlDev 3 busId d000 commId 0x96ab0f7dc9ffe518 - Init START
h100-2:414:436 [2] NCCL INFO ncclCommInitRank comm 0x2dcd5c50 rank 2 nranks 4 cudaDev 2 nvmlDev 2 busId c000 commId 0x96ab0f7dc9ffe518 - Init START
h100-2:412:433 [0] NCCL INFO ncclCommInitRank comm 0x16a70470 rank 0 nranks 4 cudaDev 0 nvmlDev 0 busId a000 commId 0x96ab0f7dc9ffe518 - Init START
h100-2:413:434 [1] NCCL INFO ncclCommInitRank comm 0x358e6fb0 rank 1 nranks 4 cudaDev 1 nvmlDev 1 busId b000 commId 0x96ab0f7dc9ffe518 - Init START
h100-2:415:435 [3] NCCL INFO MNNVL busId 0xd000 fabric UUID 0.0 cliqueId 0x0 state 3 healthMask 0x0
h100-2:414:436 [2] NCCL INFO MNNVL busId 0xc000 fabric UUID 0.0 cliqueId 0x0 state 3 healthMask 0x0
h100-2:412:433 [0] NCCL INFO MNNVL busId 0xa000 fabric UUID 0.0 cliqueId 0x0 state 3 healthMask 0x0
h100-2:413:434 [1] NCCL INFO MNNVL busId 0xb000 fabric UUID 0.0 cliqueId 0x0 state 3 healthMask 0x0
h100-2:415:435 [3] NCCL INFO NCCL_TOPO_FILE set by environment to /etc/4psa/nccl_topo/h100-80gb-sxm-ib.xml
h100-2:414:436 [2] NCCL INFO NCCL_TOPO_FILE set by environment to /etc/4psa/nccl_topo/h100-80gb-sxm-ib.xml
h100-2:412:433 [0] NCCL INFO NCCL_TOPO_FILE set by environment to /etc/4psa/nccl_topo/h100-80gb-sxm-ib.xml
h100-2:413:434 [1] NCCL INFO NCCL_TOPO_FILE set by environment to /etc/4psa/nccl_topo/h100-80gb-sxm-ib.xml
h100-2:415:435 [3] NCCL INFO graph/paths.cc:260 -> 3
h100-2:415:435 [3] NCCL INFO transport/p2p.cc:123 -> 3
h100-2:415:435 [3] NCCL INFO graph/paths.cc:588 -> 3
h100-2:415:435 [3] NCCL INFO init.cc:1014 -> 3
h100-2:415:435 [3] NCCL INFO init.cc:1548 -> 3
h100-2:415:435 [3] NCCL INFO group.cc:64 -> 3 [Async thread]
h100-2:415:415 [3] NCCL INFO group.cc:418 -> 3
h100-2:415:415 [3] NCCL INFO init.cc:1929 -> 3
h100-2:413:434 [1] NCCL INFO graph/paths.cc:260 -> 3
h100-2:413:434 [1] NCCL INFO transport/p2p.cc:123 -> 3
h100-2:413:434 [1] NCCL INFO graph/paths.cc:588 -> 3
h100-2:413:434 [1] NCCL INFO init.cc:1014 -> 3
h100-2:413:434 [1] NCCL INFO init.cc:1548 -> 3
h100-2:413:434 [1] NCCL INFO group.cc:64 -> 3 [Async thread]
h100-2:413:413 [1] NCCL INFO group.cc:418 -> 3
h100-2:413:413 [1] NCCL INFO init.cc:1929 -> 3
h100-2:414:436 [2] NCCL INFO graph/paths.cc:260 -> 3
h100-2:414:436 [2] NCCL INFO transport/p2p.cc:123 -> 3
h100-2:414:436 [2] NCCL INFO graph/paths.cc:588 -> 3
h100-2:414:436 [2] NCCL INFO init.cc:1014 -> 3
h100-2:414:436 [2] NCCL INFO init.cc:1548 -> 3
h100-2:414:436 [2] NCCL INFO group.cc:64 -> 3 [Async thread]
h100-2:414:414 [2] NCCL INFO group.cc:418 -> 3
h100-2:414:414 [2] NCCL INFO init.cc:1929 -> 3
h100-2:412:433 [0] NCCL INFO graph/paths.cc:260 -> 3
h100-2:412:433 [0] NCCL INFO transport/p2p.cc:123 -> 3
h100-2:412:433 [0] NCCL INFO graph/paths.cc:588 -> 3
h100-2:412:433 [0] NCCL INFO init.cc:1014 -> 3
h100-2:412:433 [0] NCCL INFO init.cc:1548 -> 3
h100-2:412:433 [0] NCCL INFO group.cc:64 -> 3 [Async thread]
h100-2:412:412 [0] NCCL INFO group.cc:418 -> 3
h100-2:412:412 [0] NCCL INFO init.cc:1929 -> 3
[rank1]: Traceback (most recent call last):
[rank1]:   File "/vllm-workspace/test.py", line 8, in <module>
[rank1]:     dist.all_reduce(data, op=dist.ReduceOp.SUM)
[rank1]:   File "/usr/local/lib/python3.12/dist-packages/torch/distributed/c10d_logger.py", line 83, in wrapper
[rank1]:     return func(*args, **kwargs)
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/usr/local/lib/python3.12/dist-packages/torch/distributed/distributed_c10d.py", line 2501, in all_reduce
[rank1]:     work = group.allreduce([tensor], opts)
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: torch.distributed.DistBackendError: NCCL error in: ../torch/csrc/distributed/c10d/NCCLUtils.hpp:317, internal error - please report this issue to the NCCL developers, NCCL version 2.21.5
[rank1]: ncclInternalError: Internal check failed.
[rank1]: Last error:

[rank2]: Traceback (most recent call last):
[rank2]:   File "/vllm-workspace/test.py", line 8, in <module>
[rank2]:     dist.all_reduce(data, op=dist.ReduceOp.SUM)
[rank2]:   File "/usr/local/lib/python3.12/dist-packages/torch/distributed/c10d_logger.py", line 83, in wrapper
[rank2]:     return func(*args, **kwargs)
[rank2]:            ^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/usr/local/lib/python3.12/dist-packages/torch/distributed/distributed_c10d.py", line 2501, in all_reduce
[rank2]:     work = group.allreduce([tensor], opts)
[rank2]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: torch.distributed.DistBackendError: NCCL error in: ../torch/csrc/distributed/c10d/NCCLUtils.hpp:317, internal error - please report this issue to the NCCL developers, NCCL version 2.21.5
[rank2]: ncclInternalError: Internal check failed.
[rank2]: Last error:

[rank0]: Traceback (most recent call last):
[rank0]:   File "/vllm-workspace/test.py", line 8, in <module>
[rank0]:     dist.all_reduce(data, op=dist.ReduceOp.SUM)
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/torch/distributed/c10d_logger.py", line 83, in wrapper
[rank0]:     return func(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/torch/distributed/distributed_c10d.py", line 2501, in all_reduce
[rank0]:     work = group.allreduce([tensor], opts)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: torch.distributed.DistBackendError: NCCL error in: ../torch/csrc/distributed/c10d/NCCLUtils.hpp:317, internal error - please report this issue to the NCCL developers, NCCL version 2.21.5
[rank0]: ncclInternalError: Internal check failed.
[rank0]: Last error:

[rank3]: Traceback (most recent call last):
[rank3]:   File "/vllm-workspace/test.py", line 8, in <module>
[rank3]:     dist.all_reduce(data, op=dist.ReduceOp.SUM)
[rank3]:   File "/usr/local/lib/python3.12/dist-packages/torch/distributed/c10d_logger.py", line 83, in wrapper
[rank3]:     return func(*args, **kwargs)
[rank3]:            ^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/usr/local/lib/python3.12/dist-packages/torch/distributed/distributed_c10d.py", line 2501, in all_reduce
[rank3]:     work = group.allreduce([tensor], opts)
[rank3]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: torch.distributed.DistBackendError: NCCL error in: ../torch/csrc/distributed/c10d/NCCLUtils.hpp:317, internal error - please report this issue to the NCCL developers, NCCL version 2.21.5
[rank3]: ncclInternalError: Internal check failed.
[rank3]: Last error:

[rank0]:[W1119 06:29:04.501858728 ProcessGroupNCCL.cpp:1250] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present,  but this warning has only been added since PyTorch 2.4 (function operator())
W1119 06:29:04.946000 347 torch/distributed/elastic/multiprocessing/api.py:897] Sending process 412 closing signal SIGTERM
W1119 06:29:04.947000 347 torch/distributed/elastic/multiprocessing/api.py:897] Sending process 414 closing signal SIGTERM
W1119 06:29:04.947000 347 torch/distributed/elastic/multiprocessing/api.py:897] Sending process 415 closing signal SIGTERM
E1119 06:29:05.312000 347 torch/distributed/elastic/multiprocessing/api.py:869] failed (exitcode: 1) local_rank: 1 (pid: 413) of binary: /usr/bin/python3
Traceback (most recent call last):
  File "/usr/local/bin/torchrun", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 355, in wrapper
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/distributed/run.py", line 919, in main
    run(args)
  File "/usr/local/lib/python3.12/dist-packages/torch/distributed/run.py", line 910, in run
    elastic_launch(
  File "/usr/local/lib/python3.12/dist-packages/torch/distributed/launcher/api.py", line 138, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/distributed/launcher/api.py", line 269, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
test.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2024-11-19_06:29:04
  host      : host.containers.internal
  rank      : 1 (local_rank: 1)
  exitcode  : 1 (pid: 413)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================

Before submitting a new issue...

[X] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

With same nvidia drivers but in docker.io/vllm/vllm-openai:v0.6.3 i get
W1119 06:35:34.032000<PHONE_NUMBER>07392 torch/distributed/run.py:779]
W1119 06:35:34.032000<PHONE_NUMBER>07392 torch/distributed/run.py:779] *****************************************
W1119 06:35:34.032000<PHONE_NUMBER>07392 torch/distributed/run.py:779] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
W1119 06:35:34.032000<PHONE_NUMBER>07392 torch/distributed/run.py:779] *****************************************
h100-2:146:146 [0] NCCL INFO Bootstrap : Using enp3s0:<IP_ADDRESS><0>
h100-2:146:146 [0] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation
h100-2:146:146 [0] NCCL INFO cudaDriverVersion 12070
NCCL version 2.20.5+cuda12.4
h100-2:149:149 [3] NCCL INFO cudaDriverVersion 12070
h100-2:149:149 [3] NCCL INFO Bootstrap : Using enp3s0:<IP_ADDRESS><0>
h100-2:149:149 [3] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation
h100-2:147:147 [1] NCCL INFO cudaDriverVersion 12070
h100-2:148:148 [2] NCCL INFO cudaDriverVersion 12070
h100-2:147:147 [1] NCCL INFO Bootstrap : Using enp3s0:<IP_ADDRESS><0>
h100-2:147:147 [1] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation
h100-2:148:148 [2] NCCL INFO Bootstrap : Using enp3s0:<IP_ADDRESS><0>
h100-2:148:148 [2] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation
h100-2:146:167 [0] NCCL INFO NCCL_IB_HCA set to mlx5_0:1,mlx5_1:1,mlx5_2:1,mlx5_3:1,mlx5_4:1,mlx5_5:1,mlx5_6:1,mlx5_7:1
h100-2:146:167 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [4]mlx5_4:1/IB [5]mlx5_5:1/IB [6]mlx5_6:1/IB [7]mlx5_7:1/IB [RO]; OOB enp3s0:<IP_ADDRESS><0>
h100-2:146:167 [0] NCCL INFO Using non-device net plugin version 0
h100-2:146:167 [0] NCCL INFO Using network IB
h100-2:146:167 [0] NCCL INFO DMA-BUF is available on GPU device 0
h100-2:149:168 [3] NCCL INFO NCCL_IB_HCA set to mlx5_0:1,mlx5_1:1,mlx5_2:1,mlx5_3:1,mlx5_4:1,mlx5_5:1,mlx5_6:1,mlx5_7:1
h100-2:149:168 [3] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [4]mlx5_4:1/IB [5]mlx5_5:1/IB [6]mlx5_6:1/IB [7]mlx5_7:1/IB [RO]; OOB enp3s0:<IP_ADDRESS><0>
h100-2:149:168 [3] NCCL INFO Using non-device net plugin version 0
h100-2:149:168 [3] NCCL INFO Using network IB
h100-2:149:168 [3] NCCL INFO DMA-BUF is available on GPU device 3
h100-2:147:169 [1] NCCL INFO NCCL_IB_HCA set to mlx5_0:1,mlx5_1:1,mlx5_2:1,mlx5_3:1,mlx5_4:1,mlx5_5:1,mlx5_6:1,mlx5_7:1
h100-2:148:170 [2] NCCL INFO NCCL_IB_HCA set to mlx5_0:1,mlx5_1:1,mlx5_2:1,mlx5_3:1,mlx5_4:1,mlx5_5:1,mlx5_6:1,mlx5_7:1
h100-2:148:170 [2] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [4]mlx5_4:1/IB [5]mlx5_5:1/IB [6]mlx5_6:1/IB [7]mlx5_7:1/IB [RO]; OOB enp3s0:<IP_ADDRESS><0>
h100-2:147:169 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [4]mlx5_4:1/IB [5]mlx5_5:1/IB [6]mlx5_6:1/IB [7]mlx5_7:1/IB [RO]; OOB enp3s0:<IP_ADDRESS><0>
h100-2:148:170 [2] NCCL INFO Using non-device net plugin version 0
h100-2:147:169 [1] NCCL INFO Using non-device net plugin version 0
h100-2:148:170 [2] NCCL INFO Using network IB
h100-2:147:169 [1] NCCL INFO Using network IB
h100-2:147:169 [1] NCCL INFO DMA-BUF is available on GPU device 1
h100-2:148:170 [2] NCCL INFO DMA-BUF is available on GPU device 2
h100-2:146:167 [0] NCCL INFO comm 0x16452ca0 rank 0 nranks 4 cudaDev 0 nvmlDev 0 busId a000 commId 0xbfb92b42c4ae96e - Init START
h100-2:149:168 [3] NCCL INFO comm 0x324b3b30 rank 3 nranks 4 cudaDev 3 nvmlDev 3 busId d000 commId 0xbfb92b42c4ae96e - Init START
h100-2:148:170 [2] NCCL INFO comm 0x18971f80 rank 2 nranks 4 cudaDev 2 nvmlDev 2 busId c000 commId 0xbfb92b42c4ae96e - Init START
h100-2:147:169 [1] NCCL INFO comm 0xa7aa6e0 rank 1 nranks 4 cudaDev 1 nvmlDev 1 busId b000 commId 0xbfb92b42c4ae96e - Init START
h100-2:148:170 [2] NCCL INFO MNNVL busId 0xc000 fabric UUID 0.0 cliqueId 0x0 state 3 healthMask 0x0
h100-2:149:168 [3] NCCL INFO MNNVL busId 0xd000 fabric UUID 0.0 cliqueId 0x0 state 3 healthMask 0x0
h100-2:146:167 [0] NCCL INFO MNNVL busId 0xa000 fabric UUID 0.0 cliqueId 0x0 state 3 healthMask 0x0
h100-2:147:169 [1] NCCL INFO MNNVL busId 0xb000 fabric UUID 0.0 cliqueId 0x0 state 3 healthMask 0x0
h100-2:146:167 [0] NCCL INFO NCCL_TOPO_FILE set by environment to /etc/4psa/nccl_topo/h100-80gb-sxm-ib.xml
h100-2:149:168 [3] NCCL INFO NCCL_TOPO_FILE set by environment to /etc/4psa/nccl_topo/h100-80gb-sxm-ib.xml
h100-2:148:170 [2] NCCL INFO NCCL_TOPO_FILE set by environment to /etc/4psa/nccl_topo/h100-80gb-sxm-ib.xml
h100-2:147:169 [1] NCCL INFO NCCL_TOPO_FILE set by environment to /etc/4psa/nccl_topo/h100-80gb-sxm-ib.xml
h100-2:146:167 [0] NCCL INFO Setting affinity for GPU 1 to ffffffff
h100-2:146:167 [0] NCCL INFO NVLS multicast support is available on dev 0
h100-2:149:168 [3] NCCL INFO Setting affinity for GPU 3 to ffffffff
h100-2:148:170 [2] NCCL INFO NVLS multicast support is available on dev 2
h100-2:149:168 [3] NCCL INFO NVLS multicast support is available on dev 3
h100-2:147:169 [1] NCCL INFO Setting affinity for GPU 2 to ffffffff
h100-2:147:169 [1] NCCL INFO NVLS multicast support is available on dev 1
h100-2:149:168 [3] NCCL INFO comm 0x324b3b30 rank 3 nRanks 4 nNodes 1 localRanks 4 localRank 3 MNNVL 0
h100-2:148:170 [2] NCCL INFO comm 0x18971f80 rank 2 nRanks 4 nNodes 1 localRanks 4 localRank 2 MNNVL 0
h100-2:146:167 [0] NCCL INFO comm 0x16452ca0 rank 0 nRanks 4 nNodes 1 localRanks 4 localRank 0 MNNVL 0
h100-2:147:169 [1] NCCL INFO comm 0xa7aa6e0 rank 1 nRanks 4 nNodes 1 localRanks 4 localRank 1 MNNVL 0
h100-2:149:168 [3] NCCL INFO Trees [0] 2/-1/-1->3->1 [1] 2/-1/-1->3->1 [2] 2/-1/-1->3->1 [3] 2/-1/-1->3->1 [4] 2/-1/-1->3->1 [5] 2/-1/-1->3->1 [6] 2/-1/-1->3->1 [7] 2/-1/-1->3->1 [8] 2/-1/-1->3->1 [9] 2/-1/-1->3->1 [10] 2/-1/-1->3->1 [11] 2/-1/-1->3->1 [12] 2/-1/-1->3->1 [13] 2/-1/-1->3->1 [14] 2/-1/-1->3->1 [15] 2/-1/-1->3->1 [16] 2/-1/-1->3->1 [17] 2/-1/-1->3->1 [18] 2/-1/-1->3->1 [19] 2/-1/-1->3->1 [20] 2/-1/-1->3->1 [21] 2/-1/-1->3->1 [22] 2/-1/-1->3->1 [23] 2/-1/-1->3->1
h100-2:148:170 [2] NCCL INFO Trees [0] -1/-1/-1->2->3 [1] -1/-1/-1->2->3 [2] -1/-1/-1->2->3 [3] -1/-1/-1->2->3 [4] -1/-1/-1->2->3 [5] -1/-1/-1->2->3 [6] -1/-1/-1->2->3 [7] -1/-1/-1->2->3 [8] -1/-1/-1->2->3 [9] -1/-1/-1->2->3 [10] -1/-1/-1->2->3 [11] -1/-1/-1->2->3 [12] -1/-1/-1->2->3 [13] -1/-1/-1->2->3 [14] -1/-1/-1->2->3 [15] -1/-1/-1->2->3 [16] -1/-1/-1->2->3 [17] -1/-1/-1->2->3 [18] -1/-1/-1->2->3 [19] -1/-1/-1->2->3 [20] -1/-1/-1->2->3 [21] -1/-1/-1->2->3 [22] -1/-1/-1->2->3 [23] -1/-1/-1->2->3
h100-2:146:167 [0] NCCL INFO Channel 00/24 :    0   1   3   2
h100-2:149:168 [3] NCCL INFO P2P Chunksize set to 524288
h100-2:148:170 [2] NCCL INFO P2P Chunksize set to 524288
h100-2:146:167 [0] NCCL INFO Channel 01/24 :    0   1   3   2
h100-2:147:169 [1] NCCL INFO Trees [0] 3/-1/-1->1->0 [1] 3/-1/-1->1->0 [2] 3/-1/-1->1->0 [3] 3/-1/-1->1->0 [4] 3/-1/-1->1->0 [5] 3/-1/-1->1->0 [6] 3/-1/-1->1->0 [7] 3/-1/-1->1->0 [8] 3/-1/-1->1->0 [9] 3/-1/-1->1->0 [10] 3/-1/-1->1->0 [11] 3/-1/-1->1->0 [12] 3/-1/-1->1->0 [13] 3/-1/-1->1->0 [14] 3/-1/-1->1->0 [15] 3/-1/-1->1->0 [16] 3/-1/-1->1->0 [17] 3/-1/-1->1->0 [18] 3/-1/-1->1->0 [19] 3/-1/-1->1->0 [20] 3/-1/-1->1->0 [21] 3/-1/-1->1->0 [22] 3/-1/-1->1->0 [23] 3/-1/-1->1->0
h100-2:146:167 [0] NCCL INFO Channel 02/24 :    0   1   3   2
h100-2:147:169 [1] NCCL INFO P2P Chunksize set to 524288
h100-2:146:167 [0] NCCL INFO Channel 03/24 :    0   1   3   2
h100-2:146:167 [0] NCCL INFO Channel 04/24 :    0   1   3   2
h100-2:146:167 [0] NCCL INFO Channel 05/24 :    0   1   3   2
h100-2:146:167 [0] NCCL INFO Channel 06/24 :    0   1   3   2
h100-2:146:167 [0] NCCL INFO Channel 07/24 :    0   1   3   2
h100-2:146:167 [0] NCCL INFO Channel 08/24 :    0   1   3   2
h100-2:146:167 [0] NCCL INFO Channel 09/24 :    0   1   3   2
h100-2:146:167 [0] NCCL INFO Channel 10/24 :    0   1   3   2
h100-2:146:167 [0] NCCL INFO Channel 11/24 :    0   1   3   2
h100-2:146:167 [0] NCCL INFO Channel 12/24 :    0   1   3   2
h100-2:146:167 [0] NCCL INFO Channel 13/24 :    0   1   3   2
h100-2:146:167 [0] NCCL INFO Channel 14/24 :    0   1   3   2
h100-2:146:167 [0] NCCL INFO Channel 15/24 :    0   1   3   2
h100-2:146:167 [0] NCCL INFO Channel 16/24 :    0   1   3   2
h100-2:146:167 [0] NCCL INFO Channel 17/24 :    0   1   3   2
h100-2:146:167 [0] NCCL INFO Channel 18/24 :    0   1   3   2
h100-2:146:167 [0] NCCL INFO Channel 19/24 :    0   1   3   2
h100-2:146:167 [0] NCCL INFO Channel 20/24 :    0   1   3   2
h100-2:146:167 [0] NCCL INFO Channel 21/24 :    0   1   3   2
h100-2:146:167 [0] NCCL INFO Channel 22/24 :    0   1   3   2
h100-2:146:167 [0] NCCL INFO Channel 23/24 :    0   1   3   2
h100-2:146:167 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] 1/-1/-1->0->-1 [7] 1/-1/-1->0->-1 [8] 1/-1/-1->0->-1 [9] 1/-1/-1->0->-1 [10] 1/-1/-1->0->-1 [11] 1/-1/-1->0->-1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] 1/-1/-1->0->-1 [19] 1/-1/-1->0->-1 [20] 1/-1/-1->0->-1 [21] 1/-1/-1->0->-1 [22] 1/-1/-1->0->-1 [23] 1/-1/-1->0->-1
h100-2:146:167 [0] NCCL INFO P2P Chunksize set to 524288
h100-2:146:167 [0] NCCL INFO Channel 00/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:146:167 [0] NCCL INFO Channel 01/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:146:167 [0] NCCL INFO Channel 02/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:146:167 [0] NCCL INFO Channel 03/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:146:167 [0] NCCL INFO Channel 04/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:146:167 [0] NCCL INFO Channel 05/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:146:167 [0] NCCL INFO Channel 06/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:146:167 [0] NCCL INFO Channel 07/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 00/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:146:167 [0] NCCL INFO Channel 08/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:146:167 [0] NCCL INFO Channel 09/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 01/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 02/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:146:167 [0] NCCL INFO Channel 10/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 03/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:146:167 [0] NCCL INFO Channel 11/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 04/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:146:167 [0] NCCL INFO Channel 12/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 05/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:146:167 [0] NCCL INFO Channel 13/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 06/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:146:167 [0] NCCL INFO Channel 14/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 07/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:146:167 [0] NCCL INFO Channel 15/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 08/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:146:167 [0] NCCL INFO Channel 16/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 09/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:146:167 [0] NCCL INFO Channel 17/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 10/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:146:167 [0] NCCL INFO Channel 18/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 11/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:146:167 [0] NCCL INFO Channel 19/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 12/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:146:167 [0] NCCL INFO Channel 20/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 13/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:146:167 [0] NCCL INFO Channel 21/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 14/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:146:167 [0] NCCL INFO Channel 22/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 15/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:146:167 [0] NCCL INFO Channel 23/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 16/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 17/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 18/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 19/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 00/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 20/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 01/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 21/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 02/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 03/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 22/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 04/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 23/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 05/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 06/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 07/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 08/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 09/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 10/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 11/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 12/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 13/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 14/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 15/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 16/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 17/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 18/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 19/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 20/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 21/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 22/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 23/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 00/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 01/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 02/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 03/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 04/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 05/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 06/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 07/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 08/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 09/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 10/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 11/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 12/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 13/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 14/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 15/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 16/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 17/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 18/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 19/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 20/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 21/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 22/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 23/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:146:167 [0] NCCL INFO Connected all rings
h100-2:147:169 [1] NCCL INFO Connected all rings
h100-2:148:170 [2] NCCL INFO Connected all rings
h100-2:149:168 [3] NCCL INFO Connected all rings
h100-2:148:170 [2] NCCL INFO Channel 00/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 01/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 02/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 03/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 04/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 05/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 06/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 07/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 08/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 09/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 10/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 11/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 12/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 13/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 14/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 15/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 16/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 17/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 18/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 19/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 20/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 21/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 22/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 23/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 00/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 01/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 02/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 03/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 04/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 05/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 06/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 07/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 08/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 09/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 10/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 11/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 12/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 13/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 14/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 15/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 16/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 17/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 18/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 19/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 20/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 21/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 22/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 23/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 00/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 01/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 02/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 03/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 04/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 05/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 06/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 07/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 08/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 09/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 10/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 11/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 12/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 13/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 14/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 15/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 16/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 17/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 18/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 19/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 20/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 21/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 22/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 23/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Connected all trees
h100-2:148:170 [2] NCCL INFO NVLS comm 0x18971f80 headRank 3 nHeads 4 buffSize 4194304 memSize 2097152 nvlsPerRankSize 301989888 nvlsTotalSize<PHONE_NUMBER>
h100-2:149:168 [3] NCCL INFO Connected all trees
h100-2:147:169 [1] NCCL INFO Connected all trees
h100-2:146:167 [0] NCCL INFO Connected all trees
h100-2:147:169 [1] NCCL INFO NVLS comm 0xa7aa6e0 headRank 1 nHeads 4 buffSize 4194304 memSize 2097152 nvlsPerRankSize 301989888 nvlsTotalSize<PHONE_NUMBER>
h100-2:146:167 [0] NCCL INFO NVLS comm 0x16452ca0 headRank 0 nHeads 4 buffSize 4194304 memSize 2097152 nvlsPerRankSize 301989888 nvlsTotalSize<PHONE_NUMBER>
h100-2:149:168 [3] NCCL INFO NVLS comm 0x324b3b30 headRank 2 nHeads 4 buffSize 4194304 memSize 2097152 nvlsPerRankSize 301989888 nvlsTotalSize<PHONE_NUMBER>
h100-2:147:169 [1] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 512 | 512
h100-2:147:169 [1] NCCL INFO 24 coll channels, 0 collnet channels, 16 nvls channels, 32 p2p channels, 32 p2p channels per peer
h100-2:148:170 [2] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 512 | 512
h100-2:148:170 [2] NCCL INFO 24 coll channels, 0 collnet channels, 16 nvls channels, 32 p2p channels, 32 p2p channels per peer
h100-2:146:167 [0] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 512 | 512
h100-2:146:167 [0] NCCL INFO 24 coll channels, 0 collnet channels, 16 nvls channels, 32 p2p channels, 32 p2p channels per peer
h100-2:149:168 [3] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 512 | 512
h100-2:149:168 [3] NCCL INFO 24 coll channels, 0 collnet channels, 16 nvls channels, 32 p2p channels, 32 p2p channels per peer
h100-2:149:168 [3] NCCL INFO comm 0x324b3b30 rank 3 nranks 4 cudaDev 3 nvmlDev 3 busId d000 commId 0xbfb92b42c4ae96e - Init COMPLETE
h100-2:147:169 [1] NCCL INFO comm 0xa7aa6e0 rank 1 nranks 4 cudaDev 1 nvmlDev 1 busId b000 commId 0xbfb92b42c4ae96e - Init COMPLETE
h100-2:148:170 [2] NCCL INFO comm 0x18971f80 rank 2 nranks 4 cudaDev 2 nvmlDev 2 busId c000 commId 0xbfb92b42c4ae96e - Init COMPLETE
h100-2:146:167 [0] NCCL INFO comm 0x16452ca0 rank 0 nranks 4 cudaDev 0 nvmlDev 0 busId a000 commId 0xbfb92b42c4ae96e - Init COMPLETE
PyTorch NCCL is successful!
PyTorch NCCL is successful!
PyTorch NCCL is successful!
PyTorch NCCL is successful!
PyTorch GLOO is successful!PyTorch GLOO is successful!

PyTorch GLOO is successful!PyTorch GLOO is successful!

/usr/local/lib/python3.12/dist-packages/vllm/connections.py:8: RuntimeWarning: Failed to read commit hash:
No module named 'vllm._version'
  from vllm.version import __version__ as VLLM_VERSION
/usr/local/lib/python3.12/dist-packages/vllm/connections.py:8: RuntimeWarning: Failed to read commit hash:
No module named 'vllm._version'
  from vllm.version import __version__ as VLLM_VERSION
/usr/local/lib/python3.12/dist-packages/vllm/connections.py:8: RuntimeWarning: Failed to read commit hash:
No module named 'vllm._version'
  from vllm.version import __version__ as VLLM_VERSION
/usr/local/lib/python3.12/dist-packages/vllm/connections.py:8: RuntimeWarning: Failed to read commit hash:
No module named 'vllm._version'
  from vllm.version import __version__ as VLLM_VERSION
INFO 11-19 06:35:41 utils.py:1008] Found nccl from library libnccl.so.2
INFO 11-19 06:35:41 utils.py:1008] Found nccl from library libnccl.so.2
INFO 11-19 06:35:41 utils.py:1008] Found nccl from library libnccl.so.2
INFO 11-19 06:35:41 pynccl.py:63] vLLM is using nccl==2.20.5
INFO 11-19 06:35:41 pynccl.py:63] vLLM is using nccl==2.20.5
INFO 11-19 06:35:41 pynccl.py:63] vLLM is using nccl==2.20.5
h100-2:148:148 [2] NCCL INFO Using non-device net plugin version 0
h100-2:148:148 [2] NCCL INFO Using network IB
h100-2:148:148 [2] NCCL INFO DMA-BUF is available on GPU device 2
INFO 11-19 06:35:41 utils.py:1008] Found nccl from library libnccl.so.2
INFO 11-19 06:35:41 pynccl.py:63] vLLM is using nccl==2.20.5
h100-2:146:146 [0] NCCL INFO Using non-device net plugin version 0
h100-2:146:146 [0] NCCL INFO Using network IB
h100-2:146:146 [0] NCCL INFO DMA-BUF is available on GPU device 0
h100-2:147:147 [1] NCCL INFO Using non-device net plugin version 0
h100-2:147:147 [1] NCCL INFO Using network IB
h100-2:147:147 [1] NCCL INFO DMA-BUF is available on GPU device 1
h100-2:149:149 [3] NCCL INFO Using non-device net plugin version 0
h100-2:149:149 [3] NCCL INFO Using network IB
h100-2:149:149 [3] NCCL INFO DMA-BUF is available on GPU device 3
h100-2:146:146 [0] NCCL INFO comm 0x1a374220 rank 0 nranks 4 cudaDev 0 nvmlDev 0 busId a000 commId 0x6fbfd5d1beb402af - Init START
h100-2:148:148 [2] NCCL INFO comm 0x1c8ac350 rank 2 nranks 4 cudaDev 2 nvmlDev 2 busId c000 commId 0x6fbfd5d1beb402af - Init START
h100-2:149:149 [3] NCCL INFO comm 0x363d3850 rank 3 nranks 4 cudaDev 3 nvmlDev 3 busId d000 commId 0x6fbfd5d1beb402af - Init START
h100-2:147:147 [1] NCCL INFO comm 0xe6eac50 rank 1 nranks 4 cudaDev 1 nvmlDev 1 busId b000 commId 0x6fbfd5d1beb402af - Init START
h100-2:148:148 [2] NCCL INFO MNNVL busId 0xc000 fabric UUID 0.0 cliqueId 0x0 state 3 healthMask 0x0
h100-2:146:146 [0] NCCL INFO MNNVL busId 0xa000 fabric UUID 0.0 cliqueId 0x0 state 3 healthMask 0x0
h100-2:147:147 [1] NCCL INFO MNNVL busId 0xb000 fabric UUID 0.0 cliqueId 0x0 state 3 healthMask 0x0
h100-2:149:149 [3] NCCL INFO MNNVL busId 0xd000 fabric UUID 0.0 cliqueId 0x0 state 3 healthMask 0x0
h100-2:149:149 [3] NCCL INFO NCCL_TOPO_FILE set by environment to /etc/4psa/nccl_topo/h100-80gb-sxm-ib.xml
h100-2:148:148 [2] NCCL INFO NCCL_TOPO_FILE set by environment to /etc/4psa/nccl_topo/h100-80gb-sxm-ib.xml
h100-2:147:147 [1] NCCL INFO NCCL_TOPO_FILE set by environment to /etc/4psa/nccl_topo/h100-80gb-sxm-ib.xml
h100-2:146:146 [0] NCCL INFO NCCL_TOPO_FILE set by environment to /etc/4psa/nccl_topo/h100-80gb-sxm-ib.xml
h100-2:146:146 [0] NCCL INFO Setting affinity for GPU 1 to ffffffff
h100-2:146:146 [0] NCCL INFO NVLS multicast support is available on dev 0
h100-2:149:149 [3] NCCL INFO Setting affinity for GPU 3 to ffffffff
h100-2:148:148 [2] NCCL INFO NVLS multicast support is available on dev 2
h100-2:149:149 [3] NCCL INFO NVLS multicast support is available on dev 3
h100-2:147:147 [1] NCCL INFO Setting affinity for GPU 2 to ffffffff
h100-2:147:147 [1] NCCL INFO NVLS multicast support is available on dev 1
h100-2:149:149 [3] NCCL INFO comm 0x363d3850 rank 3 nRanks 4 nNodes 1 localRanks 4 localRank 3 MNNVL 0
h100-2:148:148 [2] NCCL INFO comm 0x1c8ac350 rank 2 nRanks 4 nNodes 1 localRanks 4 localRank 2 MNNVL 0
h100-2:146:146 [0] NCCL INFO comm 0x1a374220 rank 0 nRanks 4 nNodes 1 localRanks 4 localRank 0 MNNVL 0
h100-2:147:147 [1] NCCL INFO comm 0xe6eac50 rank 1 nRanks 4 nNodes 1 localRanks 4 localRank 1 MNNVL 0
h100-2:149:149 [3] NCCL INFO Trees [0] 2/-1/-1->3->1 [1] 2/-1/-1->3->1 [2] 2/-1/-1->3->1 [3] 2/-1/-1->3->1 [4] 2/-1/-1->3->1 [5] 2/-1/-1->3->1 [6] 2/-1/-1->3->1 [7] 2/-1/-1->3->1 [8] 2/-1/-1->3->1 [9] 2/-1/-1->3->1 [10] 2/-1/-1->3->1 [11] 2/-1/-1->3->1 [12] 2/-1/-1->3->1 [13] 2/-1/-1->3->1 [14] 2/-1/-1->3->1 [15] 2/-1/-1->3->1 [16] 2/-1/-1->3->1 [17] 2/-1/-1->3->1 [18] 2/-1/-1->3->1 [19] 2/-1/-1->3->1 [20] 2/-1/-1->3->1 [21] 2/-1/-1->3->1 [22] 2/-1/-1->3->1 [23] 2/-1/-1->3->1
h100-2:146:146 [0] NCCL INFO Channel 00/24 :    0   1   3   2
h100-2:148:148 [2] NCCL INFO Trees [0] -1/-1/-1->2->3 [1] -1/-1/-1->2->3 [2] -1/-1/-1->2->3 [3] -1/-1/-1->2->3 [4] -1/-1/-1->2->3 [5] -1/-1/-1->2->3 [6] -1/-1/-1->2->3 [7] -1/-1/-1->2->3 [8] -1/-1/-1->2->3 [9] -1/-1/-1->2->3 [10] -1/-1/-1->2->3 [11] -1/-1/-1->2->3 [12] -1/-1/-1->2->3 [13] -1/-1/-1->2->3 [14] -1/-1/-1->2->3 [15] -1/-1/-1->2->3 [16] -1/-1/-1->2->3 [17] -1/-1/-1->2->3 [18] -1/-1/-1->2->3 [19] -1/-1/-1->2->3 [20] -1/-1/-1->2->3 [21] -1/-1/-1->2->3 [22] -1/-1/-1->2->3 [23] -1/-1/-1->2->3
h100-2:149:149 [3] NCCL INFO P2P Chunksize set to 524288
h100-2:146:146 [0] NCCL INFO Channel 01/24 :    0   1   3   2
h100-2:148:148 [2] NCCL INFO P2P Chunksize set to 524288
h100-2:147:147 [1] NCCL INFO Trees [0] 3/-1/-1->1->0 [1] 3/-1/-1->1->0 [2] 3/-1/-1->1->0 [3] 3/-1/-1->1->0 [4] 3/-1/-1->1->0 [5] 3/-1/-1->1->0 [6] 3/-1/-1->1->0 [7] 3/-1/-1->1->0 [8] 3/-1/-1->1->0 [9] 3/-1/-1->1->0 [10] 3/-1/-1->1->0 [11] 3/-1/-1->1->0 [12] 3/-1/-1->1->0 [13] 3/-1/-1->1->0 [14] 3/-1/-1->1->0 [15] 3/-1/-1->1->0 [16] 3/-1/-1->1->0 [17] 3/-1/-1->1->0 [18] 3/-1/-1->1->0 [19] 3/-1/-1->1->0 [20] 3/-1/-1->1->0 [21] 3/-1/-1->1->0 [22] 3/-1/-1->1->0 [23] 3/-1/-1->1->0
h100-2:146:146 [0] NCCL INFO Channel 02/24 :    0   1   3   2
h100-2:147:147 [1] NCCL INFO P2P Chunksize set to 524288
h100-2:146:146 [0] NCCL INFO Channel 03/24 :    0   1   3   2
h100-2:146:146 [0] NCCL INFO Channel 04/24 :    0   1   3   2
h100-2:146:146 [0] NCCL INFO Channel 05/24 :    0   1   3   2
h100-2:146:146 [0] NCCL INFO Channel 06/24 :    0   1   3   2
h100-2:146:146 [0] NCCL INFO Channel 07/24 :    0   1   3   2
h100-2:146:146 [0] NCCL INFO Channel 08/24 :    0   1   3   2
h100-2:146:146 [0] NCCL INFO Channel 09/24 :    0   1   3   2
h100-2:146:146 [0] NCCL INFO Channel 10/24 :    0   1   3   2
h100-2:146:146 [0] NCCL INFO Channel 11/24 :    0   1   3   2
h100-2:146:146 [0] NCCL INFO Channel 12/24 :    0   1   3   2
h100-2:146:146 [0] NCCL INFO Channel 13/24 :    0   1   3   2
h100-2:146:146 [0] NCCL INFO Channel 14/24 :    0   1   3   2
h100-2:146:146 [0] NCCL INFO Channel 15/24 :    0   1   3   2
h100-2:146:146 [0] NCCL INFO Channel 16/24 :    0   1   3   2
h100-2:146:146 [0] NCCL INFO Channel 17/24 :    0   1   3   2
h100-2:146:146 [0] NCCL INFO Channel 18/24 :    0   1   3   2
h100-2:146:146 [0] NCCL INFO Channel 19/24 :    0   1   3   2
h100-2:146:146 [0] NCCL INFO Channel 20/24 :    0   1   3   2
h100-2:146:146 [0] NCCL INFO Channel 21/24 :    0   1   3   2
h100-2:146:146 [0] NCCL INFO Channel 22/24 :    0   1   3   2
h100-2:146:146 [0] NCCL INFO Channel 23/24 :    0   1   3   2
h100-2:146:146 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] 1/-1/-1->0->-1 [7] 1/-1/-1->0->-1 [8] 1/-1/-1->0->-1 [9] 1/-1/-1->0->-1 [10] 1/-1/-1->0->-1 [11] 1/-1/-1->0->-1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] 1/-1/-1->0->-1 [19] 1/-1/-1->0->-1 [20] 1/-1/-1->0->-1 [21] 1/-1/-1->0->-1 [22] 1/-1/-1->0->-1 [23] 1/-1/-1->0->-1
h100-2:146:146 [0] NCCL INFO P2P Chunksize set to 524288
h100-2:146:146 [0] NCCL INFO Channel 00/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:146:146 [0] NCCL INFO Channel 01/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:146:146 [0] NCCL INFO Channel 02/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:146:146 [0] NCCL INFO Channel 03/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:146:146 [0] NCCL INFO Channel 04/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:146:146 [0] NCCL INFO Channel 05/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:146:146 [0] NCCL INFO Channel 06/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:146:146 [0] NCCL INFO Channel 07/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:146:146 [0] NCCL INFO Channel 08/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 00/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:146:146 [0] NCCL INFO Channel 09/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 01/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 02/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:146:146 [0] NCCL INFO Channel 10/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 03/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:146:146 [0] NCCL INFO Channel 11/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:146:146 [0] NCCL INFO Channel 12/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 04/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 05/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:146:146 [0] NCCL INFO Channel 13/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 06/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 07/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:146:146 [0] NCCL INFO Channel 14/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 08/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 09/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 10/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:146:146 [0] NCCL INFO Channel 15/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 11/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 12/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:146:146 [0] NCCL INFO Channel 16/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 13/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 14/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:146:146 [0] NCCL INFO Channel 17/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 15/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:146:146 [0] NCCL INFO Channel 18/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 16/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 17/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:146:146 [0] NCCL INFO Channel 19/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 18/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 19/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:146:146 [0] NCCL INFO Channel 20/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 20/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 21/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:146:146 [0] NCCL INFO Channel 21/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 22/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 23/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:146:146 [0] NCCL INFO Channel 22/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:146:146 [0] NCCL INFO Channel 23/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 00/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 01/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 02/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 03/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 04/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 05/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 06/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 07/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 08/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 09/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 10/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 11/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 12/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 13/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 14/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 15/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 16/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 17/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 18/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 19/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 20/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 21/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 22/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 23/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 00/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 01/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 02/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 03/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 04/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 05/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 06/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 07/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 08/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 09/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 10/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 11/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 12/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 13/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 14/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 15/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 16/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 17/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 18/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 19/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 20/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 21/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 22/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 23/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Connected all rings
h100-2:146:146 [0] NCCL INFO Connected all rings
h100-2:149:149 [3] NCCL INFO Connected all rings
h100-2:148:148 [2] NCCL INFO Connected all rings
h100-2:148:148 [2] NCCL INFO Channel 00/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 01/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 02/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 03/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 04/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 05/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 06/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 07/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 08/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 09/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 10/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 11/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 12/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 13/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 14/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 15/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 16/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 17/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 18/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 19/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 20/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 21/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 22/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 23/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 00/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 01/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 02/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 03/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 04/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 05/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 06/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 07/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 08/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 09/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 10/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 11/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 12/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 13/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 14/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 15/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 16/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 17/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 18/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 19/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 20/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 21/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 22/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 23/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 00/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 01/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 02/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 03/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 04/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 05/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 06/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 07/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 08/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 09/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 10/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 11/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 12/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 13/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 14/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 15/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 16/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 17/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 18/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 19/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 20/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 21/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 22/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 23/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Connected all trees
h100-2:149:149 [3] NCCL INFO Connected all trees
h100-2:147:147 [1] NCCL INFO Connected all trees
h100-2:146:146 [0] NCCL INFO Connected all trees
h100-2:148:148 [2] NCCL INFO NVLS comm 0x1c8ac350 headRank 3 nHeads 4 buffSize 4194304 memSize 2097152 nvlsPerRankSize 301989888 nvlsTotalSize<PHONE_NUMBER>
h100-2:149:149 [3] NCCL INFO NVLS comm 0x363d3850 headRank 2 nHeads 4 buffSize 4194304 memSize 2097152 nvlsPerRankSize 301989888 nvlsTotalSize<PHONE_NUMBER>
h100-2:146:146 [0] NCCL INFO NVLS comm 0x1a374220 headRank 0 nHeads 4 buffSize 4194304 memSize 2097152 nvlsPerRankSize 301989888 nvlsTotalSize<PHONE_NUMBER>
h100-2:147:147 [1] NCCL INFO NVLS comm 0xe6eac50 headRank 1 nHeads 4 buffSize 4194304 memSize 2097152 nvlsPerRankSize 301989888 nvlsTotalSize<PHONE_NUMBER>
h100-2:147:147 [1] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 512 | 512
h100-2:147:147 [1] NCCL INFO 24 coll channels, 0 collnet channels, 16 nvls channels, 32 p2p channels, 32 p2p channels per peer
h100-2:148:148 [2] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 512 | 512
h100-2:148:148 [2] NCCL INFO 24 coll channels, 0 collnet channels, 16 nvls channels, 32 p2p channels, 32 p2p channels per peer
h100-2:146:146 [0] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 512 | 512
h100-2:146:146 [0] NCCL INFO 24 coll channels, 0 collnet channels, 16 nvls channels, 32 p2p channels, 32 p2p channels per peer
h100-2:149:149 [3] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 512 | 512
h100-2:149:149 [3] NCCL INFO 24 coll channels, 0 collnet channels, 16 nvls channels, 32 p2p channels, 32 p2p channels per peer
h100-2:148:148 [2] NCCL INFO comm 0x1c8ac350 rank 2 nranks 4 cudaDev 2 nvmlDev 2 busId c000 commId 0x6fbfd5d1beb402af - Init COMPLETE
h100-2:146:146 [0] NCCL INFO comm 0x1a374220 rank 0 nranks 4 cudaDev 0 nvmlDev 0 busId a000 commId 0x6fbfd5d1beb402af - Init COMPLETE
h100-2:149:149 [3] NCCL INFO comm 0x363d3850 rank 3 nranks 4 cudaDev 3 nvmlDev 3 busId d000 commId 0x6fbfd5d1beb402af - Init COMPLETE
h100-2:147:147 [1] NCCL INFO comm 0xe6eac50 rank 1 nranks 4 cudaDev 1 nvmlDev 1 busId b000 commId 0x6fbfd5d1beb402af - Init COMPLETE
vLLM NCCL is successful!
vLLM NCCL is successful!
vLLM NCCL is successful!
vLLM NCCL is successful!
vLLM NCCL with cuda graph is successful!
vLLM NCCL with cuda graph is successful!
vLLM NCCL with cuda graph is successful!vLLM NCCL with cuda graph is successful!

h100-2:146:205 [0] NCCL INFO [Service thread] Connection closed by localRank 0
h100-2:149:203 [3] NCCL INFO [Service thread] Connection closed by localRank 3
h100-2:147:207 [1] NCCL INFO [Service thread] Connection closed by localRank 1
h100-2:148:209 [2] NCCL INFO [Service thread] Connection closed by localRank 2
h100-2:148:261 [2] NCCL INFO comm 0x18971f80 rank 2 nranks 4 cudaDev 2 busId c000 - Abort COMPLETE
h100-2:146:262 [0] NCCL INFO comm 0x16452ca0 rank 0 nranks 4 cudaDev 0 busId a000 - Abort COMPLETE
h100-2:147:260 [1] NCCL INFO comm 0xa7aa6e0 rank 1 nranks 4 cudaDev 1 busId b000 - Abort COMPLETE
h100-2:149:263 [3] NCCL INFO comm 0x324b3b30 rank 3 nranks 4 cudaDev 3 busId d000 - Abort COMPLETE

With same nvidia drivers but in docker.io/vllm/vllm-openai:v0.6.3 i get
W1119 06:35:34.032000<PHONE_NUMBER>07392 torch/distributed/run.py:779]
W1119 06:35:34.032000<PHONE_NUMBER>07392 torch/distributed/run.py:779] *****************************************
W1119 06:35:34.032000<PHONE_NUMBER>07392 torch/distributed/run.py:779] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
W1119 06:35:34.032000<PHONE_NUMBER>07392 torch/distributed/run.py:779] *****************************************
h100-2:146:146 [0] NCCL INFO Bootstrap : Using enp3s0:<IP_ADDRESS><0>
h100-2:146:146 [0] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation
h100-2:146:146 [0] NCCL INFO cudaDriverVersion 12070
NCCL version 2.20.5+cuda12.4
h100-2:149:149 [3] NCCL INFO cudaDriverVersion 12070
h100-2:149:149 [3] NCCL INFO Bootstrap : Using enp3s0:<IP_ADDRESS><0>
h100-2:149:149 [3] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation
h100-2:147:147 [1] NCCL INFO cudaDriverVersion 12070
h100-2:148:148 [2] NCCL INFO cudaDriverVersion 12070
h100-2:147:147 [1] NCCL INFO Bootstrap : Using enp3s0:<IP_ADDRESS><0>
h100-2:147:147 [1] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation
h100-2:148:148 [2] NCCL INFO Bootstrap : Using enp3s0:<IP_ADDRESS><0>
h100-2:148:148 [2] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation
h100-2:146:167 [0] NCCL INFO NCCL_IB_HCA set to mlx5_0:1,mlx5_1:1,mlx5_2:1,mlx5_3:1,mlx5_4:1,mlx5_5:1,mlx5_6:1,mlx5_7:1
h100-2:146:167 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [4]mlx5_4:1/IB [5]mlx5_5:1/IB [6]mlx5_6:1/IB [7]mlx5_7:1/IB [RO]; OOB enp3s0:<IP_ADDRESS><0>
h100-2:146:167 [0] NCCL INFO Using non-device net plugin version 0
h100-2:146:167 [0] NCCL INFO Using network IB
h100-2:146:167 [0] NCCL INFO DMA-BUF is available on GPU device 0
h100-2:149:168 [3] NCCL INFO NCCL_IB_HCA set to mlx5_0:1,mlx5_1:1,mlx5_2:1,mlx5_3:1,mlx5_4:1,mlx5_5:1,mlx5_6:1,mlx5_7:1
h100-2:149:168 [3] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [4]mlx5_4:1/IB [5]mlx5_5:1/IB [6]mlx5_6:1/IB [7]mlx5_7:1/IB [RO]; OOB enp3s0:<IP_ADDRESS><0>
h100-2:149:168 [3] NCCL INFO Using non-device net plugin version 0
h100-2:149:168 [3] NCCL INFO Using network IB
h100-2:149:168 [3] NCCL INFO DMA-BUF is available on GPU device 3
h100-2:147:169 [1] NCCL INFO NCCL_IB_HCA set to mlx5_0:1,mlx5_1:1,mlx5_2:1,mlx5_3:1,mlx5_4:1,mlx5_5:1,mlx5_6:1,mlx5_7:1
h100-2:148:170 [2] NCCL INFO NCCL_IB_HCA set to mlx5_0:1,mlx5_1:1,mlx5_2:1,mlx5_3:1,mlx5_4:1,mlx5_5:1,mlx5_6:1,mlx5_7:1
h100-2:148:170 [2] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [4]mlx5_4:1/IB [5]mlx5_5:1/IB [6]mlx5_6:1/IB [7]mlx5_7:1/IB [RO]; OOB enp3s0:<IP_ADDRESS><0>
h100-2:147:169 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [4]mlx5_4:1/IB [5]mlx5_5:1/IB [6]mlx5_6:1/IB [7]mlx5_7:1/IB [RO]; OOB enp3s0:<IP_ADDRESS><0>
h100-2:148:170 [2] NCCL INFO Using non-device net plugin version 0
h100-2:147:169 [1] NCCL INFO Using non-device net plugin version 0
h100-2:148:170 [2] NCCL INFO Using network IB
h100-2:147:169 [1] NCCL INFO Using network IB
h100-2:147:169 [1] NCCL INFO DMA-BUF is available on GPU device 1
h100-2:148:170 [2] NCCL INFO DMA-BUF is available on GPU device 2
h100-2:146:167 [0] NCCL INFO comm 0x16452ca0 rank 0 nranks 4 cudaDev 0 nvmlDev 0 busId a000 commId 0xbfb92b42c4ae96e - Init START
h100-2:149:168 [3] NCCL INFO comm 0x324b3b30 rank 3 nranks 4 cudaDev 3 nvmlDev 3 busId d000 commId 0xbfb92b42c4ae96e - Init START
h100-2:148:170 [2] NCCL INFO comm 0x18971f80 rank 2 nranks 4 cudaDev 2 nvmlDev 2 busId c000 commId 0xbfb92b42c4ae96e - Init START
h100-2:147:169 [1] NCCL INFO comm 0xa7aa6e0 rank 1 nranks 4 cudaDev 1 nvmlDev 1 busId b000 commId 0xbfb92b42c4ae96e - Init START
h100-2:148:170 [2] NCCL INFO MNNVL busId 0xc000 fabric UUID 0.0 cliqueId 0x0 state 3 healthMask 0x0
h100-2:149:168 [3] NCCL INFO MNNVL busId 0xd000 fabric UUID 0.0 cliqueId 0x0 state 3 healthMask 0x0
h100-2:146:167 [0] NCCL INFO MNNVL busId 0xa000 fabric UUID 0.0 cliqueId 0x0 state 3 healthMask 0x0
h100-2:147:169 [1] NCCL INFO MNNVL busId 0xb000 fabric UUID 0.0 cliqueId 0x0 state 3 healthMask 0x0
h100-2:146:167 [0] NCCL INFO NCCL_TOPO_FILE set by environment to /etc/4psa/nccl_topo/h100-80gb-sxm-ib.xml
h100-2:149:168 [3] NCCL INFO NCCL_TOPO_FILE set by environment to /etc/4psa/nccl_topo/h100-80gb-sxm-ib.xml
h100-2:148:170 [2] NCCL INFO NCCL_TOPO_FILE set by environment to /etc/4psa/nccl_topo/h100-80gb-sxm-ib.xml
h100-2:147:169 [1] NCCL INFO NCCL_TOPO_FILE set by environment to /etc/4psa/nccl_topo/h100-80gb-sxm-ib.xml
h100-2:146:167 [0] NCCL INFO Setting affinity for GPU 1 to ffffffff
h100-2:146:167 [0] NCCL INFO NVLS multicast support is available on dev 0
h100-2:149:168 [3] NCCL INFO Setting affinity for GPU 3 to ffffffff
h100-2:148:170 [2] NCCL INFO NVLS multicast support is available on dev 2
h100-2:149:168 [3] NCCL INFO NVLS multicast support is available on dev 3
h100-2:147:169 [1] NCCL INFO Setting affinity for GPU 2 to ffffffff
h100-2:147:169 [1] NCCL INFO NVLS multicast support is available on dev 1
h100-2:149:168 [3] NCCL INFO comm 0x324b3b30 rank 3 nRanks 4 nNodes 1 localRanks 4 localRank 3 MNNVL 0
h100-2:148:170 [2] NCCL INFO comm 0x18971f80 rank 2 nRanks 4 nNodes 1 localRanks 4 localRank 2 MNNVL 0
h100-2:146:167 [0] NCCL INFO comm 0x16452ca0 rank 0 nRanks 4 nNodes 1 localRanks 4 localRank 0 MNNVL 0
h100-2:147:169 [1] NCCL INFO comm 0xa7aa6e0 rank 1 nRanks 4 nNodes 1 localRanks 4 localRank 1 MNNVL 0
h100-2:149:168 [3] NCCL INFO Trees [0] 2/-1/-1->3->1 [1] 2/-1/-1->3->1 [2] 2/-1/-1->3->1 [3] 2/-1/-1->3->1 [4] 2/-1/-1->3->1 [5] 2/-1/-1->3->1 [6] 2/-1/-1->3->1 [7] 2/-1/-1->3->1 [8] 2/-1/-1->3->1 [9] 2/-1/-1->3->1 [10] 2/-1/-1->3->1 [11] 2/-1/-1->3->1 [12] 2/-1/-1->3->1 [13] 2/-1/-1->3->1 [14] 2/-1/-1->3->1 [15] 2/-1/-1->3->1 [16] 2/-1/-1->3->1 [17] 2/-1/-1->3->1 [18] 2/-1/-1->3->1 [19] 2/-1/-1->3->1 [20] 2/-1/-1->3->1 [21] 2/-1/-1->3->1 [22] 2/-1/-1->3->1 [23] 2/-1/-1->3->1
h100-2:148:170 [2] NCCL INFO Trees [0] -1/-1/-1->2->3 [1] -1/-1/-1->2->3 [2] -1/-1/-1->2->3 [3] -1/-1/-1->2->3 [4] -1/-1/-1->2->3 [5] -1/-1/-1->2->3 [6] -1/-1/-1->2->3 [7] -1/-1/-1->2->3 [8] -1/-1/-1->2->3 [9] -1/-1/-1->2->3 [10] -1/-1/-1->2->3 [11] -1/-1/-1->2->3 [12] -1/-1/-1->2->3 [13] -1/-1/-1->2->3 [14] -1/-1/-1->2->3 [15] -1/-1/-1->2->3 [16] -1/-1/-1->2->3 [17] -1/-1/-1->2->3 [18] -1/-1/-1->2->3 [19] -1/-1/-1->2->3 [20] -1/-1/-1->2->3 [21] -1/-1/-1->2->3 [22] -1/-1/-1->2->3 [23] -1/-1/-1->2->3
h100-2:146:167 [0] NCCL INFO Channel 00/24 :    0   1   3   2
h100-2:149:168 [3] NCCL INFO P2P Chunksize set to 524288
h100-2:148:170 [2] NCCL INFO P2P Chunksize set to 524288
h100-2:146:167 [0] NCCL INFO Channel 01/24 :    0   1   3   2
h100-2:147:169 [1] NCCL INFO Trees [0] 3/-1/-1->1->0 [1] 3/-1/-1->1->0 [2] 3/-1/-1->1->0 [3] 3/-1/-1->1->0 [4] 3/-1/-1->1->0 [5] 3/-1/-1->1->0 [6] 3/-1/-1->1->0 [7] 3/-1/-1->1->0 [8] 3/-1/-1->1->0 [9] 3/-1/-1->1->0 [10] 3/-1/-1->1->0 [11] 3/-1/-1->1->0 [12] 3/-1/-1->1->0 [13] 3/-1/-1->1->0 [14] 3/-1/-1->1->0 [15] 3/-1/-1->1->0 [16] 3/-1/-1->1->0 [17] 3/-1/-1->1->0 [18] 3/-1/-1->1->0 [19] 3/-1/-1->1->0 [20] 3/-1/-1->1->0 [21] 3/-1/-1->1->0 [22] 3/-1/-1->1->0 [23] 3/-1/-1->1->0
h100-2:146:167 [0] NCCL INFO Channel 02/24 :    0   1   3   2
h100-2:147:169 [1] NCCL INFO P2P Chunksize set to 524288
h100-2:146:167 [0] NCCL INFO Channel 03/24 :    0   1   3   2
h100-2:146:167 [0] NCCL INFO Channel 04/24 :    0   1   3   2
h100-2:146:167 [0] NCCL INFO Channel 05/24 :    0   1   3   2
h100-2:146:167 [0] NCCL INFO Channel 06/24 :    0   1   3   2
h100-2:146:167 [0] NCCL INFO Channel 07/24 :    0   1   3   2
h100-2:146:167 [0] NCCL INFO Channel 08/24 :    0   1   3   2
h100-2:146:167 [0] NCCL INFO Channel 09/24 :    0   1   3   2
h100-2:146:167 [0] NCCL INFO Channel 10/24 :    0   1   3   2
h100-2:146:167 [0] NCCL INFO Channel 11/24 :    0   1   3   2
h100-2:146:167 [0] NCCL INFO Channel 12/24 :    0   1   3   2
h100-2:146:167 [0] NCCL INFO Channel 13/24 :    0   1   3   2
h100-2:146:167 [0] NCCL INFO Channel 14/24 :    0   1   3   2
h100-2:146:167 [0] NCCL INFO Channel 15/24 :    0   1   3   2
h100-2:146:167 [0] NCCL INFO Channel 16/24 :    0   1   3   2
h100-2:146:167 [0] NCCL INFO Channel 17/24 :    0   1   3   2
h100-2:146:167 [0] NCCL INFO Channel 18/24 :    0   1   3   2
h100-2:146:167 [0] NCCL INFO Channel 19/24 :    0   1   3   2
h100-2:146:167 [0] NCCL INFO Channel 20/24 :    0   1   3   2
h100-2:146:167 [0] NCCL INFO Channel 21/24 :    0   1   3   2
h100-2:146:167 [0] NCCL INFO Channel 22/24 :    0   1   3   2
h100-2:146:167 [0] NCCL INFO Channel 23/24 :    0   1   3   2
h100-2:146:167 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] 1/-1/-1->0->-1 [7] 1/-1/-1->0->-1 [8] 1/-1/-1->0->-1 [9] 1/-1/-1->0->-1 [10] 1/-1/-1->0->-1 [11] 1/-1/-1->0->-1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] 1/-1/-1->0->-1 [19] 1/-1/-1->0->-1 [20] 1/-1/-1->0->-1 [21] 1/-1/-1->0->-1 [22] 1/-1/-1->0->-1 [23] 1/-1/-1->0->-1
h100-2:146:167 [0] NCCL INFO P2P Chunksize set to 524288
h100-2:146:167 [0] NCCL INFO Channel 00/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:146:167 [0] NCCL INFO Channel 01/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:146:167 [0] NCCL INFO Channel 02/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:146:167 [0] NCCL INFO Channel 03/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:146:167 [0] NCCL INFO Channel 04/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:146:167 [0] NCCL INFO Channel 05/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:146:167 [0] NCCL INFO Channel 06/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:146:167 [0] NCCL INFO Channel 07/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 00/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:146:167 [0] NCCL INFO Channel 08/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:146:167 [0] NCCL INFO Channel 09/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 01/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 02/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:146:167 [0] NCCL INFO Channel 10/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 03/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:146:167 [0] NCCL INFO Channel 11/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 04/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:146:167 [0] NCCL INFO Channel 12/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 05/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:146:167 [0] NCCL INFO Channel 13/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 06/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:146:167 [0] NCCL INFO Channel 14/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 07/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:146:167 [0] NCCL INFO Channel 15/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 08/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:146:167 [0] NCCL INFO Channel 16/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 09/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:146:167 [0] NCCL INFO Channel 17/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 10/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:146:167 [0] NCCL INFO Channel 18/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 11/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:146:167 [0] NCCL INFO Channel 19/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 12/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:146:167 [0] NCCL INFO Channel 20/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 13/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:146:167 [0] NCCL INFO Channel 21/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 14/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:146:167 [0] NCCL INFO Channel 22/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 15/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:146:167 [0] NCCL INFO Channel 23/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 16/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 17/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 18/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 19/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 00/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 20/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 01/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 21/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 02/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 03/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 22/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 04/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 23/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 05/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 06/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 07/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 08/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 09/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 10/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 11/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 12/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 13/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 14/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 15/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 16/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 17/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 18/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 19/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 20/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 21/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 22/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 23/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 00/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 01/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 02/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 03/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 04/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 05/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 06/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 07/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 08/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 09/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 10/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 11/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 12/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 13/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 14/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 15/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 16/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 17/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 18/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 19/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 20/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 21/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 22/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 23/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:146:167 [0] NCCL INFO Connected all rings
h100-2:147:169 [1] NCCL INFO Connected all rings
h100-2:148:170 [2] NCCL INFO Connected all rings
h100-2:149:168 [3] NCCL INFO Connected all rings
h100-2:148:170 [2] NCCL INFO Channel 00/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 01/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 02/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 03/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 04/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 05/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 06/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 07/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 08/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 09/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 10/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 11/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 12/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 13/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 14/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 15/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 16/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 17/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 18/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 19/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 20/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 21/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 22/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Channel 23/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 00/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 01/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 02/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 03/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 04/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 05/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 06/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 07/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 08/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 09/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 10/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 11/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 12/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 13/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 14/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 15/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 16/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 17/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 18/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 19/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 20/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 21/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 22/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:168 [3] NCCL INFO Channel 23/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 00/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 01/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 02/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 03/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 04/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 05/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 06/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 07/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 08/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 09/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 10/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 11/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 12/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 13/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 14/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 15/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 16/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 17/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 18/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 19/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 20/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 21/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 22/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:169 [1] NCCL INFO Channel 23/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:148:170 [2] NCCL INFO Connected all trees
h100-2:148:170 [2] NCCL INFO NVLS comm 0x18971f80 headRank 3 nHeads 4 buffSize 4194304 memSize 2097152 nvlsPerRankSize 301989888 nvlsTotalSize<PHONE_NUMBER>
h100-2:149:168 [3] NCCL INFO Connected all trees
h100-2:147:169 [1] NCCL INFO Connected all trees
h100-2:146:167 [0] NCCL INFO Connected all trees
h100-2:147:169 [1] NCCL INFO NVLS comm 0xa7aa6e0 headRank 1 nHeads 4 buffSize 4194304 memSize 2097152 nvlsPerRankSize 301989888 nvlsTotalSize<PHONE_NUMBER>
h100-2:146:167 [0] NCCL INFO NVLS comm 0x16452ca0 headRank 0 nHeads 4 buffSize 4194304 memSize 2097152 nvlsPerRankSize 301989888 nvlsTotalSize<PHONE_NUMBER>
h100-2:149:168 [3] NCCL INFO NVLS comm 0x324b3b30 headRank 2 nHeads 4 buffSize 4194304 memSize 2097152 nvlsPerRankSize 301989888 nvlsTotalSize<PHONE_NUMBER>
h100-2:147:169 [1] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 512 | 512
h100-2:147:169 [1] NCCL INFO 24 coll channels, 0 collnet channels, 16 nvls channels, 32 p2p channels, 32 p2p channels per peer
h100-2:148:170 [2] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 512 | 512
h100-2:148:170 [2] NCCL INFO 24 coll channels, 0 collnet channels, 16 nvls channels, 32 p2p channels, 32 p2p channels per peer
h100-2:146:167 [0] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 512 | 512
h100-2:146:167 [0] NCCL INFO 24 coll channels, 0 collnet channels, 16 nvls channels, 32 p2p channels, 32 p2p channels per peer
h100-2:149:168 [3] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 512 | 512
h100-2:149:168 [3] NCCL INFO 24 coll channels, 0 collnet channels, 16 nvls channels, 32 p2p channels, 32 p2p channels per peer
h100-2:149:168 [3] NCCL INFO comm 0x324b3b30 rank 3 nranks 4 cudaDev 3 nvmlDev 3 busId d000 commId 0xbfb92b42c4ae96e - Init COMPLETE
h100-2:147:169 [1] NCCL INFO comm 0xa7aa6e0 rank 1 nranks 4 cudaDev 1 nvmlDev 1 busId b000 commId 0xbfb92b42c4ae96e - Init COMPLETE
h100-2:148:170 [2] NCCL INFO comm 0x18971f80 rank 2 nranks 4 cudaDev 2 nvmlDev 2 busId c000 commId 0xbfb92b42c4ae96e - Init COMPLETE
h100-2:146:167 [0] NCCL INFO comm 0x16452ca0 rank 0 nranks 4 cudaDev 0 nvmlDev 0 busId a000 commId 0xbfb92b42c4ae96e - Init COMPLETE
PyTorch NCCL is successful!
PyTorch NCCL is successful!
PyTorch NCCL is successful!
PyTorch NCCL is successful!
PyTorch GLOO is successful!PyTorch GLOO is successful!

PyTorch GLOO is successful!PyTorch GLOO is successful!

/usr/local/lib/python3.12/dist-packages/vllm/connections.py:8: RuntimeWarning: Failed to read commit hash:
No module named 'vllm._version'
  from vllm.version import __version__ as VLLM_VERSION
/usr/local/lib/python3.12/dist-packages/vllm/connections.py:8: RuntimeWarning: Failed to read commit hash:
No module named 'vllm._version'
  from vllm.version import __version__ as VLLM_VERSION
/usr/local/lib/python3.12/dist-packages/vllm/connections.py:8: RuntimeWarning: Failed to read commit hash:
No module named 'vllm._version'
  from vllm.version import __version__ as VLLM_VERSION
/usr/local/lib/python3.12/dist-packages/vllm/connections.py:8: RuntimeWarning: Failed to read commit hash:
No module named 'vllm._version'
  from vllm.version import __version__ as VLLM_VERSION
INFO 11-19 06:35:41 utils.py:1008] Found nccl from library libnccl.so.2
INFO 11-19 06:35:41 utils.py:1008] Found nccl from library libnccl.so.2
INFO 11-19 06:35:41 utils.py:1008] Found nccl from library libnccl.so.2
INFO 11-19 06:35:41 pynccl.py:63] vLLM is using nccl==2.20.5
INFO 11-19 06:35:41 pynccl.py:63] vLLM is using nccl==2.20.5
INFO 11-19 06:35:41 pynccl.py:63] vLLM is using nccl==2.20.5
h100-2:148:148 [2] NCCL INFO Using non-device net plugin version 0
h100-2:148:148 [2] NCCL INFO Using network IB
h100-2:148:148 [2] NCCL INFO DMA-BUF is available on GPU device 2
INFO 11-19 06:35:41 utils.py:1008] Found nccl from library libnccl.so.2
INFO 11-19 06:35:41 pynccl.py:63] vLLM is using nccl==2.20.5
h100-2:146:146 [0] NCCL INFO Using non-device net plugin version 0
h100-2:146:146 [0] NCCL INFO Using network IB
h100-2:146:146 [0] NCCL INFO DMA-BUF is available on GPU device 0
h100-2:147:147 [1] NCCL INFO Using non-device net plugin version 0
h100-2:147:147 [1] NCCL INFO Using network IB
h100-2:147:147 [1] NCCL INFO DMA-BUF is available on GPU device 1
h100-2:149:149 [3] NCCL INFO Using non-device net plugin version 0
h100-2:149:149 [3] NCCL INFO Using network IB
h100-2:149:149 [3] NCCL INFO DMA-BUF is available on GPU device 3
h100-2:146:146 [0] NCCL INFO comm 0x1a374220 rank 0 nranks 4 cudaDev 0 nvmlDev 0 busId a000 commId 0x6fbfd5d1beb402af - Init START
h100-2:148:148 [2] NCCL INFO comm 0x1c8ac350 rank 2 nranks 4 cudaDev 2 nvmlDev 2 busId c000 commId 0x6fbfd5d1beb402af - Init START
h100-2:149:149 [3] NCCL INFO comm 0x363d3850 rank 3 nranks 4 cudaDev 3 nvmlDev 3 busId d000 commId 0x6fbfd5d1beb402af - Init START
h100-2:147:147 [1] NCCL INFO comm 0xe6eac50 rank 1 nranks 4 cudaDev 1 nvmlDev 1 busId b000 commId 0x6fbfd5d1beb402af - Init START
h100-2:148:148 [2] NCCL INFO MNNVL busId 0xc000 fabric UUID 0.0 cliqueId 0x0 state 3 healthMask 0x0
h100-2:146:146 [0] NCCL INFO MNNVL busId 0xa000 fabric UUID 0.0 cliqueId 0x0 state 3 healthMask 0x0
h100-2:147:147 [1] NCCL INFO MNNVL busId 0xb000 fabric UUID 0.0 cliqueId 0x0 state 3 healthMask 0x0
h100-2:149:149 [3] NCCL INFO MNNVL busId 0xd000 fabric UUID 0.0 cliqueId 0x0 state 3 healthMask 0x0
h100-2:149:149 [3] NCCL INFO NCCL_TOPO_FILE set by environment to /etc/4psa/nccl_topo/h100-80gb-sxm-ib.xml
h100-2:148:148 [2] NCCL INFO NCCL_TOPO_FILE set by environment to /etc/4psa/nccl_topo/h100-80gb-sxm-ib.xml
h100-2:147:147 [1] NCCL INFO NCCL_TOPO_FILE set by environment to /etc/4psa/nccl_topo/h100-80gb-sxm-ib.xml
h100-2:146:146 [0] NCCL INFO NCCL_TOPO_FILE set by environment to /etc/4psa/nccl_topo/h100-80gb-sxm-ib.xml
h100-2:146:146 [0] NCCL INFO Setting affinity for GPU 1 to ffffffff
h100-2:146:146 [0] NCCL INFO NVLS multicast support is available on dev 0
h100-2:149:149 [3] NCCL INFO Setting affinity for GPU 3 to ffffffff
h100-2:148:148 [2] NCCL INFO NVLS multicast support is available on dev 2
h100-2:149:149 [3] NCCL INFO NVLS multicast support is available on dev 3
h100-2:147:147 [1] NCCL INFO Setting affinity for GPU 2 to ffffffff
h100-2:147:147 [1] NCCL INFO NVLS multicast support is available on dev 1
h100-2:149:149 [3] NCCL INFO comm 0x363d3850 rank 3 nRanks 4 nNodes 1 localRanks 4 localRank 3 MNNVL 0
h100-2:148:148 [2] NCCL INFO comm 0x1c8ac350 rank 2 nRanks 4 nNodes 1 localRanks 4 localRank 2 MNNVL 0
h100-2:146:146 [0] NCCL INFO comm 0x1a374220 rank 0 nRanks 4 nNodes 1 localRanks 4 localRank 0 MNNVL 0
h100-2:147:147 [1] NCCL INFO comm 0xe6eac50 rank 1 nRanks 4 nNodes 1 localRanks 4 localRank 1 MNNVL 0
h100-2:149:149 [3] NCCL INFO Trees [0] 2/-1/-1->3->1 [1] 2/-1/-1->3->1 [2] 2/-1/-1->3->1 [3] 2/-1/-1->3->1 [4] 2/-1/-1->3->1 [5] 2/-1/-1->3->1 [6] 2/-1/-1->3->1 [7] 2/-1/-1->3->1 [8] 2/-1/-1->3->1 [9] 2/-1/-1->3->1 [10] 2/-1/-1->3->1 [11] 2/-1/-1->3->1 [12] 2/-1/-1->3->1 [13] 2/-1/-1->3->1 [14] 2/-1/-1->3->1 [15] 2/-1/-1->3->1 [16] 2/-1/-1->3->1 [17] 2/-1/-1->3->1 [18] 2/-1/-1->3->1 [19] 2/-1/-1->3->1 [20] 2/-1/-1->3->1 [21] 2/-1/-1->3->1 [22] 2/-1/-1->3->1 [23] 2/-1/-1->3->1
h100-2:146:146 [0] NCCL INFO Channel 00/24 :    0   1   3   2
h100-2:148:148 [2] NCCL INFO Trees [0] -1/-1/-1->2->3 [1] -1/-1/-1->2->3 [2] -1/-1/-1->2->3 [3] -1/-1/-1->2->3 [4] -1/-1/-1->2->3 [5] -1/-1/-1->2->3 [6] -1/-1/-1->2->3 [7] -1/-1/-1->2->3 [8] -1/-1/-1->2->3 [9] -1/-1/-1->2->3 [10] -1/-1/-1->2->3 [11] -1/-1/-1->2->3 [12] -1/-1/-1->2->3 [13] -1/-1/-1->2->3 [14] -1/-1/-1->2->3 [15] -1/-1/-1->2->3 [16] -1/-1/-1->2->3 [17] -1/-1/-1->2->3 [18] -1/-1/-1->2->3 [19] -1/-1/-1->2->3 [20] -1/-1/-1->2->3 [21] -1/-1/-1->2->3 [22] -1/-1/-1->2->3 [23] -1/-1/-1->2->3
h100-2:149:149 [3] NCCL INFO P2P Chunksize set to 524288
h100-2:146:146 [0] NCCL INFO Channel 01/24 :    0   1   3   2
h100-2:148:148 [2] NCCL INFO P2P Chunksize set to 524288
h100-2:147:147 [1] NCCL INFO Trees [0] 3/-1/-1->1->0 [1] 3/-1/-1->1->0 [2] 3/-1/-1->1->0 [3] 3/-1/-1->1->0 [4] 3/-1/-1->1->0 [5] 3/-1/-1->1->0 [6] 3/-1/-1->1->0 [7] 3/-1/-1->1->0 [8] 3/-1/-1->1->0 [9] 3/-1/-1->1->0 [10] 3/-1/-1->1->0 [11] 3/-1/-1->1->0 [12] 3/-1/-1->1->0 [13] 3/-1/-1->1->0 [14] 3/-1/-1->1->0 [15] 3/-1/-1->1->0 [16] 3/-1/-1->1->0 [17] 3/-1/-1->1->0 [18] 3/-1/-1->1->0 [19] 3/-1/-1->1->0 [20] 3/-1/-1->1->0 [21] 3/-1/-1->1->0 [22] 3/-1/-1->1->0 [23] 3/-1/-1->1->0
h100-2:146:146 [0] NCCL INFO Channel 02/24 :    0   1   3   2
h100-2:147:147 [1] NCCL INFO P2P Chunksize set to 524288
h100-2:146:146 [0] NCCL INFO Channel 03/24 :    0   1   3   2
h100-2:146:146 [0] NCCL INFO Channel 04/24 :    0   1   3   2
h100-2:146:146 [0] NCCL INFO Channel 05/24 :    0   1   3   2
h100-2:146:146 [0] NCCL INFO Channel 06/24 :    0   1   3   2
h100-2:146:146 [0] NCCL INFO Channel 07/24 :    0   1   3   2
h100-2:146:146 [0] NCCL INFO Channel 08/24 :    0   1   3   2
h100-2:146:146 [0] NCCL INFO Channel 09/24 :    0   1   3   2
h100-2:146:146 [0] NCCL INFO Channel 10/24 :    0   1   3   2
h100-2:146:146 [0] NCCL INFO Channel 11/24 :    0   1   3   2
h100-2:146:146 [0] NCCL INFO Channel 12/24 :    0   1   3   2
h100-2:146:146 [0] NCCL INFO Channel 13/24 :    0   1   3   2
h100-2:146:146 [0] NCCL INFO Channel 14/24 :    0   1   3   2
h100-2:146:146 [0] NCCL INFO Channel 15/24 :    0   1   3   2
h100-2:146:146 [0] NCCL INFO Channel 16/24 :    0   1   3   2
h100-2:146:146 [0] NCCL INFO Channel 17/24 :    0   1   3   2
h100-2:146:146 [0] NCCL INFO Channel 18/24 :    0   1   3   2
h100-2:146:146 [0] NCCL INFO Channel 19/24 :    0   1   3   2
h100-2:146:146 [0] NCCL INFO Channel 20/24 :    0   1   3   2
h100-2:146:146 [0] NCCL INFO Channel 21/24 :    0   1   3   2
h100-2:146:146 [0] NCCL INFO Channel 22/24 :    0   1   3   2
h100-2:146:146 [0] NCCL INFO Channel 23/24 :    0   1   3   2
h100-2:146:146 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] 1/-1/-1->0->-1 [7] 1/-1/-1->0->-1 [8] 1/-1/-1->0->-1 [9] 1/-1/-1->0->-1 [10] 1/-1/-1->0->-1 [11] 1/-1/-1->0->-1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] 1/-1/-1->0->-1 [19] 1/-1/-1->0->-1 [20] 1/-1/-1->0->-1 [21] 1/-1/-1->0->-1 [22] 1/-1/-1->0->-1 [23] 1/-1/-1->0->-1
h100-2:146:146 [0] NCCL INFO P2P Chunksize set to 524288
h100-2:146:146 [0] NCCL INFO Channel 00/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:146:146 [0] NCCL INFO Channel 01/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:146:146 [0] NCCL INFO Channel 02/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:146:146 [0] NCCL INFO Channel 03/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:146:146 [0] NCCL INFO Channel 04/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:146:146 [0] NCCL INFO Channel 05/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:146:146 [0] NCCL INFO Channel 06/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:146:146 [0] NCCL INFO Channel 07/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:146:146 [0] NCCL INFO Channel 08/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 00/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:146:146 [0] NCCL INFO Channel 09/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 01/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 02/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:146:146 [0] NCCL INFO Channel 10/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 03/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:146:146 [0] NCCL INFO Channel 11/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:146:146 [0] NCCL INFO Channel 12/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 04/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 05/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:146:146 [0] NCCL INFO Channel 13/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 06/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 07/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:146:146 [0] NCCL INFO Channel 14/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 08/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 09/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 10/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:146:146 [0] NCCL INFO Channel 15/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 11/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 12/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:146:146 [0] NCCL INFO Channel 16/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 13/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 14/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:146:146 [0] NCCL INFO Channel 17/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 15/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:146:146 [0] NCCL INFO Channel 18/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 16/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 17/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:146:146 [0] NCCL INFO Channel 19/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 18/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 19/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:146:146 [0] NCCL INFO Channel 20/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 20/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 21/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:146:146 [0] NCCL INFO Channel 21/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 22/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 23/0 : 2[2] -> 0[0] via P2P/CUMEM
h100-2:146:146 [0] NCCL INFO Channel 22/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:146:146 [0] NCCL INFO Channel 23/0 : 0[0] -> 1[1] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 00/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 01/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 02/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 03/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 04/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 05/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 06/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 07/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 08/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 09/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 10/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 11/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 12/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 13/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 14/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 15/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 16/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 17/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 18/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 19/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 20/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 21/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 22/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 23/0 : 1[1] -> 3[3] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 00/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 01/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 02/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 03/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 04/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 05/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 06/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 07/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 08/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 09/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 10/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 11/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 12/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 13/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 14/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 15/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 16/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 17/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 18/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 19/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 20/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 21/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 22/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 23/0 : 3[3] -> 2[2] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Connected all rings
h100-2:146:146 [0] NCCL INFO Connected all rings
h100-2:149:149 [3] NCCL INFO Connected all rings
h100-2:148:148 [2] NCCL INFO Connected all rings
h100-2:148:148 [2] NCCL INFO Channel 00/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 01/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 02/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 03/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 04/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 05/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 06/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 07/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 08/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 09/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 10/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 11/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 12/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 13/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 14/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 15/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 16/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 17/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 18/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 19/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 20/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 21/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 22/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Channel 23/0 : 2[2] -> 3[3] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 00/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 01/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 02/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 03/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 04/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 05/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 06/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 07/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 08/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 09/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 10/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 11/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 12/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 13/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 14/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 15/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 16/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 17/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 18/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 19/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 20/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 21/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 22/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:149:149 [3] NCCL INFO Channel 23/0 : 3[3] -> 1[1] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 00/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 01/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 02/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 03/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 04/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 05/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 06/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 07/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 08/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 09/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 10/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 11/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 12/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 13/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 14/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 15/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 16/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 17/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 18/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 19/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 20/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 21/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 22/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:147:147 [1] NCCL INFO Channel 23/0 : 1[1] -> 0[0] via P2P/CUMEM
h100-2:148:148 [2] NCCL INFO Connected all trees
h100-2:149:149 [3] NCCL INFO Connected all trees
h100-2:147:147 [1] NCCL INFO Connected all trees
h100-2:146:146 [0] NCCL INFO Connected all trees
h100-2:148:148 [2] NCCL INFO NVLS comm 0x1c8ac350 headRank 3 nHeads 4 buffSize 4194304 memSize 2097152 nvlsPerRankSize 301989888 nvlsTotalSize<PHONE_NUMBER>
h100-2:149:149 [3] NCCL INFO NVLS comm 0x363d3850 headRank 2 nHeads 4 buffSize 4194304 memSize 2097152 nvlsPerRankSize 301989888 nvlsTotalSize<PHONE_NUMBER>
h100-2:146:146 [0] NCCL INFO NVLS comm 0x1a374220 headRank 0 nHeads 4 buffSize 4194304 memSize 2097152 nvlsPerRankSize 301989888 nvlsTotalSize<PHONE_NUMBER>
h100-2:147:147 [1] NCCL INFO NVLS comm 0xe6eac50 headRank 1 nHeads 4 buffSize 4194304 memSize 2097152 nvlsPerRankSize 301989888 nvlsTotalSize<PHONE_NUMBER>
h100-2:147:147 [1] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 512 | 512
h100-2:147:147 [1] NCCL INFO 24 coll channels, 0 collnet channels, 16 nvls channels, 32 p2p channels, 32 p2p channels per peer
h100-2:148:148 [2] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 512 | 512
h100-2:148:148 [2] NCCL INFO 24 coll channels, 0 collnet channels, 16 nvls channels, 32 p2p channels, 32 p2p channels per peer
h100-2:146:146 [0] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 512 | 512
h100-2:146:146 [0] NCCL INFO 24 coll channels, 0 collnet channels, 16 nvls channels, 32 p2p channels, 32 p2p channels per peer
h100-2:149:149 [3] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 512 | 512
h100-2:149:149 [3] NCCL INFO 24 coll channels, 0 collnet channels, 16 nvls channels, 32 p2p channels, 32 p2p channels per peer
h100-2:148:148 [2] NCCL INFO comm 0x1c8ac350 rank 2 nranks 4 cudaDev 2 nvmlDev 2 busId c000 commId 0x6fbfd5d1beb402af - Init COMPLETE
h100-2:146:146 [0] NCCL INFO comm 0x1a374220 rank 0 nranks 4 cudaDev 0 nvmlDev 0 busId a000 commId 0x6fbfd5d1beb402af - Init COMPLETE
h100-2:149:149 [3] NCCL INFO comm 0x363d3850 rank 3 nranks 4 cudaDev 3 nvmlDev 3 busId d000 commId 0x6fbfd5d1beb402af - Init COMPLETE
h100-2:147:147 [1] NCCL INFO comm 0xe6eac50 rank 1 nranks 4 cudaDev 1 nvmlDev 1 busId b000 commId 0x6fbfd5d1beb402af - Init COMPLETE
vLLM NCCL is successful!
vLLM NCCL is successful!
vLLM NCCL is successful!
vLLM NCCL is successful!
vLLM NCCL with cuda graph is successful!
vLLM NCCL with cuda graph is successful!
vLLM NCCL with cuda graph is successful!vLLM NCCL with cuda graph is successful!

h100-2:146:205 [0] NCCL INFO [Service thread] Connection closed by localRank 0
h100-2:149:203 [3] NCCL INFO [Service thread] Connection closed by localRank 3
h100-2:147:207 [1] NCCL INFO [Service thread] Connection closed by localRank 1
h100-2:148:209 [2] NCCL INFO [Service thread] Connection closed by localRank 2
h100-2:148:261 [2] NCCL INFO comm 0x18971f80 rank 2 nranks 4 cudaDev 2 busId c000 - Abort COMPLETE
h100-2:146:262 [0] NCCL INFO comm 0x16452ca0 rank 0 nranks 4 cudaDev 0 busId a000 - Abort COMPLETE
h100-2:147:260 [1] NCCL INFO comm 0xa7aa6e0 rank 1 nranks 4 cudaDev 1 busId b000 - Abort COMPLETE
h100-2:149:263 [3] NCCL INFO comm 0x324b3b30 rank 3 nranks 4 cudaDev 3 busId d000 - Abort COMPLETE

I think this NCCL error is from there not being enough shared memory in the docker container to run with TP=4. If running directly in docker, try running it with --shm-size=256m.
From the docs:

You can either use the ipc=host flag or --shm-size flag to allow the container to access the host’s shared memory. vLLM uses PyTorch, which uses shared memory to share data between processes under the hood, particularly for tensor parallel inference.

REF

Sorry for the incomplete command, I already have set these parameters
podman run ...
  --cap-add=IPC_LOCK \
  --pids-limit=-1 \
  --shm-size 16g  \
  --ulimit memlock=-1  \

It seems that with an empty /etc/nccl.conf it works so it seems it is a configuration issue on my server and de vllm v0.4.1.post1 image is good.
i thought that server is configured correctly since it worked with previous vllm image.