[Installation]: Build vllm with nightly pytorch

Your current environment
why is it important:
This is  a prerequisite to the work on enabling troch.compile on vllm, we  need to be able to build vllm with nightly so that we can iterate on changes and try features that are not released yet.
current error:
Failed to import from vllm._C with ImportError('/home/lsakka/vllm/vllm/_C.abi3.so: undefined symbol: cuTensorMapEncodeTiled')
any idea what this could be?
It was mentioned that vllm was struggling to upgrade one step version
Collecting environment information...
WARNING 07-11 16:20:31 _custom_ops.py:14] Failed to import from vllm._C with ImportError('/home/lsakka/vllm/vllm/_C.abi3.so: undefined symbol: cuTensorMapEncodeTiled')
PyTorch version: 2.5.0a0+git9c9744c
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: CentOS Stream 9 (x86_64)
GCC version: (GCC) 11.4.1 20231218 (Red Hat 11.4.1-3)
Clang version: 17.0.6 (CentOS 17.0.6-5.el9)
CMake version: version 3.30.0
Libc version: glibc-2.34

Python version: 3.11.9 (main, Apr 19 2024, 16:48:06) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.12.0-0_fbk16_zion_7661_geb00762ce6d2-x86_64-with-glibc2.34
Is CUDA available: True
CUDA runtime version: 12.1.105
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
GPU 0: NVIDIA PG509-210
GPU 1: NVIDIA PG509-210
GPU 2: NVIDIA PG509-210
GPU 3: NVIDIA PG509-210
GPU 4: NVIDIA PG509-210
GPU 5: NVIDIA PG509-210
GPU 6: NVIDIA PG509-210
GPU 7: NVIDIA PG509-210

Nvidia driver version: 525.105.17
cuDNN version: Probably one of the following:
/usr/lib64/libcudnn.so.8.8.0
/usr/lib64/libcudnn_adv_infer.so.8.8.0
/usr/lib64/libcudnn_adv_train.so.8.8.0
/usr/lib64/libcudnn_cnn_infer.so.8.8.0
/usr/lib64/libcudnn_cnn_train.so.8.8.0
/usr/lib64/libcudnn_ops_infer.so.8.8.0
/usr/lib64/libcudnn_ops_train.so.8.8.0
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Address sizes:                   46 bits physical, 48 bits virtual
Byte Order:                      Little Endian
CPU(s):                          192
On-line CPU(s) list:             0-191
Vendor ID:                       GenuineIntel
Model name:                      Intel(R) Xeon(R) Platinum 8339HC CPU @ 1.80GHz
CPU family:                      6
Model:                           85
Thread(s) per core:              2
Core(s) per socket:              24
Socket(s):                       4
Stepping:                        11
Frequency boost:                 enabled
CPU(s) scaling MHz:              99%
CPU max MHz:                     1801.0000
CPU min MHz:                     800.0000
BogoMIPS:                        3600.00
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single intel_ppin ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local avx512_bf16 dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke avx512_vnni md_clear flush_l1d arch_capabilities
Virtualization:                  VT-x
L1d cache:                       3 MiB (96 instances)
L1i cache:                       3 MiB (96 instances)
L2 cache:                        96 MiB (96 instances)
L3 cache:                        132 MiB (4 instances)
NUMA node(s):                    4
NUMA node0 CPU(s):               0-23,96-119
NUMA node1 CPU(s):               24-47,120-143
NUMA node2 CPU(s):               48-71,144-167
NUMA node3 CPU(s):               72-95,168-191
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Enhanced IBRS, IBPB conditional, RSB filling
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected

Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] nvidia-nccl-cu12==2.20.5
[pip3] torch==2.5.0a0+git9c9744c
[pip3] torchvision==0.18.0
[pip3] torchvision==0.19.0a0+d23a6e1
[pip3] transformers==4.42.3
[pip3] triton==3.0.0
[conda] blas                      1.0                         mkl
[conda] mkl                       2023.1.0         h213fc3f_46344
[conda] mkl-service               2.4.0           py311h5eee18b_1
[conda] mkl_fft                   1.3.8           py311h5eee18b_0
[conda] mkl_random                1.2.4           py311hdb19cb5_0
[conda] numpy                     1.26.4          py311h08b1b3b_0
[conda] numpy-base                1.26.4          py311hf175353_0
[conda] nvidia-nccl-cu12          2.20.5                   pypi_0    pypi
[conda] torch                     2.5.0a0+git9c9744c           dev_0    <develop>
[conda] torchvision               0.18.0                   pypi_0    pypi
[conda] transformers              4.42.3                   pypi_0    pypi
[conda] triton                    3.0.0                    pypi_0    pypi
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.5.1
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    CPU Affinity    NUMA Affinity
GPU0     X      NV4     SYS     SYS     NV2     NV2     SYS     NV4     0-23,96-119     0
GPU1    NV4      X      SYS     SYS     NV4     NV2     NV2     SYS     0-23,96-119     0
GPU2    SYS     SYS      X      NV4     SYS     NV4     NV2     NV2     24-47,120-143   1
GPU3    SYS     SYS     NV4      X      NV2     SYS     NV4     NV2     24-47,120-143   1
GPU4    NV2     NV4     SYS     NV2      X      NV4     SYS     SYS     48-71,144-167   2
GPU5    NV2     NV2     NV4     SYS     NV4      X      SYS     SYS     48-71,144-167   2
GPU6    SYS     NV2     NV2     NV4     SYS     SYS      X      NV4     72-95,168-191   3
GPU7    NV4     SYS     NV2     NV2     SYS     SYS     NV4      X      72-95,168-191   3

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

diff file
diff --git a/requirements-build.txt b/requirements-build.txt
index 1a07a94e..76f44f0c 100644
--- a/requirements-build.txt
+++ b/requirements-build.txt
@@ -3,5 +3,4 @@ cmake>=3.21
 ninja
 packaging
 setuptools>=49.4.0
-torch==2.3.0
 wheel
diff --git a/requirements-cpu.txt b/requirements-cpu.txt
index 754070df..8145cf37 100644
--- a/requirements-cpu.txt
+++ b/requirements-cpu.txt
@@ -2,6 +2,4 @@
 -r requirements-common.txt

 # Dependencies for x86_64 CPUs
-torch == 2.3.1+cpu; platform_machine != "ppc64le"
-torchvision == 0.18.1+cpu; platform_machine != "ppc64le"   # required for the image processor of phi3v, this must be updated alongside torch
 triton >= 2.2.0  # FIXME(woosuk): This is a hack to avoid import error.
diff --git a/requirements-cuda.txt b/requirements-cuda.txt
index 10596ed8..776b8338 100644
--- a/requirements-cuda.txt
+++ b/requirements-cuda.txt
@@ -4,8 +4,4 @@
 # Dependencies for NVIDIA GPUs
 ray >= 2.9
 nvidia-ml-py # for pynvml package
-torch == 2.3.0
 # These must be updated alongside torch
-torchvision == 0.18.0   # Required for phi3v processor, also see https://github.com/pytorch/vision?tab=readme-ov-file#installation for corresponding version
-xformers == 0.0.26.post1  # Requires PyTorch 2.3.0
-vllm-flash-attn == 2.5.9  # Requires PyTorch 2.3.0
diff --git a/requirements-openvino.txt b/requirements-openvino.txt
index e555d525..5012b6d4 100644
--- a/requirements-openvino.txt
+++ b/requirements-openvino.txt
@@ -2,7 +2,6 @@
 -r requirements-common.txt

 # OpenVINO dependencies
-torch >= 2.1.2
 openvino ~= 2024.3.0.dev
 optimum-intel[openvino] >= 1.17.2

How you are installing vllm
what did I do:

I updated vllm requirments files as the following and remove pinned versions of xformers, flash_attn, torch, triton. (see diff file bellow)
build pytorch nightly, xformers, flash_attn (python setup.py develop)  and install latest triton.
build vllm
load flash_attn directly see the PR above.
when Import vllm i get the error bellow:

current error:
Failed to import from vllm._C with ImportError('/home/lsakka/vllm/vllm/_C.abi3.so: undefined symbol: cuTensorMapEncodeTiled')

full compilation log
running develop
/home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages/setuptools/command/develop.py:40: EasyInstallDeprecationWarning: easy_install command is deprecated.
!!

        ********************************************************************************
        Please avoid running ``setup.py`` and ``easy_install``.
        Instead, use pypa/build, pypa/installer or other
        standards-based tools.

        See https://github.com/pypa/setuptools/issues/917 for details.
        ********************************************************************************

!!
  easy_install.initialize_options(self)
/home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages/setuptools/_distutils/cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated.
!!

        ********************************************************************************
        Please avoid running ``setup.py`` directly.
        Instead, use pypa/build, pypa/installer or other
        standards-based tools.

        See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.
        ********************************************************************************

!!
  self.initialize_options()
running egg_info
writing vllm.egg-info/PKG-INFO
writing dependency_links to vllm.egg-info/dependency_links.txt
writing requirements to vllm.egg-info/requires.txt
writing top-level names to vllm.egg-info/top_level.txt
reading manifest file 'vllm.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
adding license file 'LICENSE'
writing manifest file 'vllm.egg-info/SOURCES.txt'
running build_ext
Using MAX_JOBS=20 as the number of jobs.
-- The CXX compiler identification is GNU 11.4.1
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Build type: RelWithDebInfo
-- Target device: cuda
-- Found Python: /home/lsakka/.conda/envs/laith1991/bin/python (found version "3.11.9") found components: Interpreter Development.Module Development.SABIModule
-- Found python matching: /home/lsakka/.conda/envs/laith1991/bin/python.
-- Found CUDA: /usr/local/cuda-12.1 (found version "12.1")
-- The CUDA compiler identification is NVIDIA 12.1.105
-- Detecting CUDA compiler ABI info
-- Detecting CUDA compiler ABI info - done
-- Check for working CUDA compiler: /usr/local/cuda-12.1/bin/nvcc - skipped
-- Detecting CUDA compile features
-- Detecting CUDA compile features - done
-- Found CUDAToolkit: /usr/local/cuda-12.1/include (found version "12.1.105")
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- Caffe2: CUDA detected: 12.1
-- Caffe2: CUDA nvcc is: /usr/local/cuda-12.1/bin/nvcc
-- Caffe2: CUDA toolkit directory: /usr/local/cuda-12.1
-- Caffe2: Header version is: 12.1
CMake Warning at /home/lsakka/pytorch/torch/share/cmake/Caffe2/public/cuda.cmake:143 (message):
  Failed to compute shorthash for libnvrtc.so
Call Stack (most recent call first):
  /home/lsakka/pytorch/torch/share/cmake/Caffe2/Caffe2Config.cmake:86 (include)
  /home/lsakka/pytorch/torch/share/cmake/Torch/TorchConfig.cmake:68 (find_package)
  CMakeLists.txt:67 (find_package)

-- USE_CUDNN is set to 0. Compiling without cuDNN support
-- USE_CUSPARSELT is set to 0. Compiling without cuSPARSELt support
-- Autodetected CUDA architecture(s):  8.0 8.0 8.0 8.0 8.0 8.0 8.0 8.0
-- Added CUDA NVCC flags for: -gencode;arch=compute_80,code=sm_80
-- Found Torch: /home/lsakka/pytorch/torch/lib/libtorch.so
CMake Warning at CMakeLists.txt:90 (message):
  Pytorch version 2.3.0 expected for CUDA build, saw 2.5.0 instead.

-- CUDA supported arches: 7.0;7.5;8.0;8.6;8.9;9.0
-- CUDA target arches: 80-real
-- CMake Version: 3.30.0
-- CUTLASS 3.5.0
-- CUDART: /usr/local/cuda-12.1/lib64/libcudart.so
-- CUDA Driver: /usr/local/cuda-12.1/lib64/stubs/libcuda.so
-- NVRTC: /usr/local/cuda-12.1/lib64/libnvrtc.so
-- Default Install Location: install
-- Found Python3: /home/lsakka/.conda/envs/laith1991/bin/python3.11 (found suitable version "3.11.9", minimum required is "3.5") found components: Interpreter
-- CUDA Compilation Architectures: 70;72;75;80;86;87;89;90;90a
-- Enable caching of reference results in conv unit tests
-- Enable rigorous conv problem sizes in conv unit tests
-- Using NVCC flags: --expt-relaxed-constexpr;-DCUTLASS_TEST_LEVEL=0;-DCUTLASS_TEST_ENABLE_CACHED_RESULTS=1;-DCUTLASS_CONV_UNIT_TEST_RIGOROUS_SIZE_ENABLED=1;-DCUTLASS_DEBUG_TRACE_LEVEL=0;-Xcompiler=-Wconversion;-Xcompiler=-fno-strict-aliasing;-lineinfo
-- CUTLASS Revision: 966fe721
-- Configuring cublas ...
-- cuBLAS Disabled.
-- Configuring cuBLAS ... done.
-- Completed generation of library instances. See /home/lsakka/vllm/build/temp.linux-x86_64-cpython-311/_deps/cutlass-build/tools/library/library_instance_generation.log for more information.
-- Punica target arches: 80-real
-- Enabling C extension.
-- Enabling moe extension.
-- Configuring done (19.3s)
-- Generating done (2.2s)
-- Build files have been written to: /home/lsakka/vllm/build/temp.linux-x86_64-cpython-311
Using MAX_JOBS=20 as the number of jobs.
[0/2] Re-checking globbed directories...
[1/27] Building CUDA object CMakeFiles/_C.dir/csrc/cuda_utils_kernels.cu.o
[2/27] Building CXX object CMakeFiles/_moe_C.dir/csrc/moe/torch_bindings.cpp.o
[3/27] Building CXX object CMakeFiles/_C.dir/csrc/torch_bindings.cpp.o
[4/27] Building CUDA object CMakeFiles/_C.dir/csrc/moe_align_block_size_kernels.cu.o
[5/27] Building CUDA object CMakeFiles/_C.dir/csrc/quantization/compressed_tensors/int8_quant_kernels.cu.o
[6/27] Building CUDA object CMakeFiles/_C.dir/csrc/quantization/fp8/common.cu.o
[7/27] Building CUDA object CMakeFiles/_C.dir/csrc/quantization/squeezellm/quant_cuda_kernel.cu.o
/home/lsakka/vllm/csrc/quantization/squeezellm/quant_cuda_kernel.cu: In function 'void squeezellm_gemm(at::Tensor, at::Tensor, at::Tensor, at::Tensor)':
/home/lsakka/vllm/csrc/quantization/squeezellm/quant_cuda_kernel.cu:198:136: warning: 'T* at::Tensor::data() const [with T = c10::Half]' is deprecated: Tensor.data<T>() is deprecated. Please use Tensor.data_ptr<T>() instead. [-Wdeprecated-declarations]
  198 |   vllm::squeezellm::NUQ4MatMulKernel<<<blocks, threads, 0, stream>>>(
      |                                                                                ^
/home/lsakka/pytorch/torch/include/ATen/core/TensorBody.h:247:1: note: declared here
  247 |   T * data() const {
      | ^ ~~
/home/lsakka/vllm/csrc/quantization/squeezellm/quant_cuda_kernel.cu:198:193: warning: 'T* at::Tensor::data() const [with T = c10::Half]' is deprecated: Tensor.data<T>() is deprecated. Please use Tensor.data_ptr<T>() instead. [-Wdeprecated-declarations]
  198 |   vllm::squeezellm::NUQ4MatMulKernel<<<blocks, threads, 0, stream>>>(
      |                                                                                ^
/home/lsakka/pytorch/torch/include/ATen/core/TensorBody.h:247:1: note: declared here
  247 |   T * data() const {
      | ^ ~~
/home/lsakka/vllm/csrc/quantization/squeezellm/quant_cuda_kernel.cu:198:237: warning: 'T* at::Tensor::data() const [with T = c10::Half]' is deprecated: Tensor.data<T>() is deprecated. Please use Tensor.data_ptr<T>() instead. [-Wdeprecated-declarations]
  198 |   vllm::squeezellm::NUQ4MatMulKernel<<<blocks, threads, 0, stream>>>(
      |                                                                                ^
/home/lsakka/pytorch/torch/include/ATen/core/TensorBody.h:247:1: note: declared here
  247 |   T * data() const {
      | ^ ~~
[8/27] Building CUDA object CMakeFiles/_C.dir/csrc/layernorm_kernels.cu.o
[9/27] Building CUDA object CMakeFiles/_C.dir/csrc/activation_kernels.cu.o
[10/27] Building CUDA object CMakeFiles/_C.dir/csrc/pos_encoding_kernels.cu.o
[11/27] Building CUDA object CMakeFiles/_C.dir/csrc/quantization/awq/gemm_kernels.cu.o
/home/lsakka/vllm/csrc/quantization/awq/gemm_kernels.cu(370): warning #177-D: variable "j_factors1" was declared but never referenced
    int j_factors1 = 4;
        ^

Remark: The warnings can be suppressed with "-diag-suppress <warning-number>"

/home/lsakka/vllm/csrc/quantization/awq/gemm_kernels.cu(371): warning #177-D: variable "row_stride2" was declared but never referenced
    int row_stride2 = 4;
        ^

/home/lsakka/vllm/csrc/quantization/awq/gemm_kernels.cu(372): warning #177-D: variable "split_k_iters" was declared but never referenced
    int split_k_iters = 1;
        ^

/home/lsakka/vllm/csrc/quantization/awq/gemm_kernels.cu(378): warning #177-D: variable "B_shared_warp" was declared but never referenced
    half B_shared_warp[32];
         ^

/home/lsakka/vllm/csrc/quantization/awq/gemm_kernels.cu(379): warning #177-D: variable "OC" was declared but never referenced
    int OC = 512;
        ^

/home/lsakka/vllm/csrc/quantization/awq/gemm_kernels.cu(45): warning #177-D: variable "scaling_factors_shared" was declared but never referenced
              half scaling_factors_shared[N];
                   ^

/home/lsakka/vllm/csrc/quantization/awq/gemm_kernels.cu(46): warning #177-D: variable "zeros_shared" was declared but never referenced
              half zeros_shared[N];
                   ^

/home/lsakka/vllm/csrc/quantization/awq/gemm_kernels.cu(49): warning #177-D: variable "blockIdx_x" was declared but never referenced
    int blockIdx_x = 0;
        ^

/home/lsakka/vllm/csrc/quantization/awq/gemm_kernels.cu(63): warning #177-D: variable "ld_zero_flag" was declared but never referenced
    bool ld_zero_flag = (threadIdx.y * 32 + threadIdx.x) * 8 < N;
         ^

/home/lsakka/vllm/csrc/quantization/awq/gemm_kernels.cu(21): warning #177-D: function "vllm::awq::__pack_half2" was declared but never referenced
                                   unsigned __pack_half2(const half x,
                                            ^

[12/27] Building CUDA object CMakeFiles/_C.dir/csrc/quantization/gptq_marlin/gptq_marlin_repack.cu.o
[13/27] Building CUDA object CMakeFiles/_C.dir/csrc/quantization/aqlm/gemm_kernels.cu.o
/home/lsakka/vllm/csrc/quantization/aqlm/gemm_kernels.cu(276): warning #177-D: variable "res" was declared but never referenced
    float res = 0;
          ^

Remark: The warnings can be suppressed with "-diag-suppress <warning-number>"

[14/27] Building CUDA object CMakeFiles/_C.dir/csrc/cache_kernels.cu.o
[15/27] Building CUDA object CMakeFiles/_moe_C.dir/csrc/moe/topk_softmax_kernels.cu.o
[16/27] Linking CXX shared module /home/lsakka/vllm/build/lib.linux-x86_64-cpython-311/vllm/_moe_C.abi3.so
[17/27] Building CUDA object CMakeFiles/_C.dir/csrc/quantization/gptq/q_gemm.cu.o
[18/27] Building CUDA object CMakeFiles/_C.dir/csrc/quantization/cutlass_w8a8/scaled_mm_entry.cu.o
[19/27] Building CUDA object CMakeFiles/_C.dir/csrc/custom_all_reduce.cu.o
[20/27] Building CUDA object CMakeFiles/_C.dir/csrc/quantization/marlin/sparse/marlin_24_cuda_kernel.cu.o
/home/lsakka/vllm/csrc/quantization/marlin/sparse/marlin_24_cuda_kernel.cu(407): warning #179-D: right operand of "%" is zero
        if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
                                         ^
          detected during instantiation of "void marlin_24::Marlin_24<num_bits,threads,thread_m_blocks,thread_n_blocks,thread_k_blocks,stages,group_blocks>(const int4 *, const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with num_bits=4, threads=256, thread_m_blocks=1, thread_n_blocks=8, thread_k_blocks=4, stages=4, group_blocks=-1]" at line 959

Remark: The warnings can be suppressed with "-diag-suppress <warning-number>"

/home/lsakka/vllm/csrc/quantization/marlin/sparse/marlin_24_cuda_kernel.cu(438): warning #39-D: division by zero
                                 (pipe / (group_blocks / thread_k_blocks)));
                                         ^
          detected during instantiation of "void marlin_24::Marlin_24<num_bits,threads,thread_m_blocks,thread_n_blocks,thread_k_blocks,stages,group_blocks>(const int4 *, const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with num_bits=4, threads=256, thread_m_blocks=1, thread_n_blocks=8, thread_k_blocks=4, stages=4, group_blocks=-1]" at line 959

/home/lsakka/vllm/csrc/quantization/marlin/sparse/marlin_24_cuda_kernel.cu(407): warning #179-D: right operand of "%" is zero
        if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
                                         ^
          detected during instantiation of "void marlin_24::Marlin_24<num_bits,threads,thread_m_blocks,thread_n_blocks,thread_k_blocks,stages,group_blocks>(const int4 *, const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with num_bits=4, threads=256, thread_m_blocks=1, thread_n_blocks=16, thread_k_blocks=2, stages=4, group_blocks=-1]" at line 962

/home/lsakka/vllm/csrc/quantization/marlin/sparse/marlin_24_cuda_kernel.cu(438): warning #39-D: division by zero
                                 (pipe / (group_blocks / thread_k_blocks)));
                                         ^
          detected during instantiation of "void marlin_24::Marlin_24<num_bits,threads,thread_m_blocks,thread_n_blocks,thread_k_blocks,stages,group_blocks>(const int4 *, const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with num_bits=4, threads=256, thread_m_blocks=1, thread_n_blocks=16, thread_k_blocks=2, stages=4, group_blocks=-1]" at line 962

/home/lsakka/vllm/csrc/quantization/marlin/sparse/marlin_24_cuda_kernel.cu(407): warning #179-D: right operand of "%" is zero
        if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
                                         ^
          detected during instantiation of "void marlin_24::Marlin_24<num_bits,threads,thread_m_blocks,thread_n_blocks,thread_k_blocks,stages,group_blocks>(const int4 *, const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with num_bits=4, threads=256, thread_m_blocks=2, thread_n_blocks=16, thread_k_blocks=2, stages=4, group_blocks=-1]" at line 964

/home/lsakka/vllm/csrc/quantization/marlin/sparse/marlin_24_cuda_kernel.cu(438): warning #39-D: division by zero
                                 (pipe / (group_blocks / thread_k_blocks)));
                                         ^
          detected during instantiation of "void marlin_24::Marlin_24<num_bits,threads,thread_m_blocks,thread_n_blocks,thread_k_blocks,stages,group_blocks>(const int4 *, const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with num_bits=4, threads=256, thread_m_blocks=2, thread_n_blocks=16, thread_k_blocks=2, stages=4, group_blocks=-1]" at line 964

/home/lsakka/vllm/csrc/quantization/marlin/sparse/marlin_24_cuda_kernel.cu(407): warning #179-D: right operand of "%" is zero
        if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
                                         ^
          detected during instantiation of "void marlin_24::Marlin_24<num_bits,threads,thread_m_blocks,thread_n_blocks,thread_k_blocks,stages,group_blocks>(const int4 *, const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with num_bits=4, threads=256, thread_m_blocks=3, thread_n_blocks=16, thread_k_blocks=2, stages=4, group_blocks=-1]" at line 966

/home/lsakka/vllm/csrc/quantization/marlin/sparse/marlin_24_cuda_kernel.cu(438): warning #39-D: division by zero
                                 (pipe / (group_blocks / thread_k_blocks)));
                                         ^
          detected during instantiation of "void marlin_24::Marlin_24<num_bits,threads,thread_m_blocks,thread_n_blocks,thread_k_blocks,stages,group_blocks>(const int4 *, const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with num_bits=4, threads=256, thread_m_blocks=3, thread_n_blocks=16, thread_k_blocks=2, stages=4, group_blocks=-1]" at line 966

/home/lsakka/vllm/csrc/quantization/marlin/sparse/marlin_24_cuda_kernel.cu(407): warning #179-D: right operand of "%" is zero
        if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
                                         ^
          detected during instantiation of "void marlin_24::Marlin_24<num_bits,threads,thread_m_blocks,thread_n_blocks,thread_k_blocks,stages,group_blocks>(const int4 *, const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with num_bits=4, threads=256, thread_m_blocks=4, thread_n_blocks=16, thread_k_blocks=2, stages=4, group_blocks=-1]" at line 968

/home/lsakka/vllm/csrc/quantization/marlin/sparse/marlin_24_cuda_kernel.cu(438): warning #39-D: division by zero
                                 (pipe / (group_blocks / thread_k_blocks)));
                                         ^
          detected during instantiation of "void marlin_24::Marlin_24<num_bits,threads,thread_m_blocks,thread_n_blocks,thread_k_blocks,stages,group_blocks>(const int4 *, const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with num_bits=4, threads=256, thread_m_blocks=4, thread_n_blocks=16, thread_k_blocks=2, stages=4, group_blocks=-1]" at line 968

/home/lsakka/vllm/csrc/quantization/marlin/sparse/marlin_24_cuda_kernel.cu(407): warning #179-D: right operand of "%" is zero
        if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
                                         ^
          detected during instantiation of "void marlin_24::Marlin_24<num_bits,threads,thread_m_blocks,thread_n_blocks,thread_k_blocks,stages,group_blocks>(const int4 *, const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with num_bits=8, threads=256, thread_m_blocks=1, thread_n_blocks=8, thread_k_blocks=4, stages=4, group_blocks=-1]" at line 981

/home/lsakka/vllm/csrc/quantization/marlin/sparse/marlin_24_cuda_kernel.cu(438): warning #39-D: division by zero
                                 (pipe / (group_blocks / thread_k_blocks)));
                                         ^
          detected during instantiation of "void marlin_24::Marlin_24<num_bits,threads,thread_m_blocks,thread_n_blocks,thread_k_blocks,stages,group_blocks>(const int4 *, const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with num_bits=8, threads=256, thread_m_blocks=1, thread_n_blocks=8, thread_k_blocks=4, stages=4, group_blocks=-1]" at line 981

/home/lsakka/vllm/csrc/quantization/marlin/sparse/marlin_24_cuda_kernel.cu(407): warning #179-D: right operand of "%" is zero
        if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
                                         ^
          detected during instantiation of "void marlin_24::Marlin_24<num_bits,threads,thread_m_blocks,thread_n_blocks,thread_k_blocks,stages,group_blocks>(const int4 *, const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with num_bits=8, threads=256, thread_m_blocks=1, thread_n_blocks=16, thread_k_blocks=2, stages=4, group_blocks=-1]" at line 984

/home/lsakka/vllm/csrc/quantization/marlin/sparse/marlin_24_cuda_kernel.cu(438): warning #39-D: division by zero
                                 (pipe / (group_blocks / thread_k_blocks)));
                                         ^
          detected during instantiation of "void marlin_24::Marlin_24<num_bits,threads,thread_m_blocks,thread_n_blocks,thread_k_blocks,stages,group_blocks>(const int4 *, const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with num_bits=8, threads=256, thread_m_blocks=1, thread_n_blocks=16, thread_k_blocks=2, stages=4, group_blocks=-1]" at line 984

/home/lsakka/vllm/csrc/quantization/marlin/sparse/marlin_24_cuda_kernel.cu(407): warning #179-D: right operand of "%" is zero
        if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
                                         ^
          detected during instantiation of "void marlin_24::Marlin_24<num_bits,threads,thread_m_blocks,thread_n_blocks,thread_k_blocks,stages,group_blocks>(const int4 *, const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with num_bits=8, threads=256, thread_m_blocks=2, thread_n_blocks=16, thread_k_blocks=2, stages=4, group_blocks=-1]" at line 986

/home/lsakka/vllm/csrc/quantization/marlin/sparse/marlin_24_cuda_kernel.cu(438): warning #39-D: division by zero
                                 (pipe / (group_blocks / thread_k_blocks)));
                                         ^
          detected during instantiation of "void marlin_24::Marlin_24<num_bits,threads,thread_m_blocks,thread_n_blocks,thread_k_blocks,stages,group_blocks>(const int4 *, const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with num_bits=8, threads=256, thread_m_blocks=2, thread_n_blocks=16, thread_k_blocks=2, stages=4, group_blocks=-1]" at line 986

/home/lsakka/vllm/csrc/quantization/marlin/sparse/marlin_24_cuda_kernel.cu(407): warning #179-D: right operand of "%" is zero
        if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
                                         ^
          detected during instantiation of "void marlin_24::Marlin_24<num_bits,threads,thread_m_blocks,thread_n_blocks,thread_k_blocks,stages,group_blocks>(const int4 *, const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with num_bits=8, threads=256, thread_m_blocks=3, thread_n_blocks=16, thread_k_blocks=2, stages=4, group_blocks=-1]" at line 988

/home/lsakka/vllm/csrc/quantization/marlin/sparse/marlin_24_cuda_kernel.cu(438): warning #39-D: division by zero
                                 (pipe / (group_blocks / thread_k_blocks)));
                                         ^
          detected during instantiation of "void marlin_24::Marlin_24<num_bits,threads,thread_m_blocks,thread_n_blocks,thread_k_blocks,stages,group_blocks>(const int4 *, const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with num_bits=8, threads=256, thread_m_blocks=3, thread_n_blocks=16, thread_k_blocks=2, stages=4, group_blocks=-1]" at line 988

/home/lsakka/vllm/csrc/quantization/marlin/sparse/marlin_24_cuda_kernel.cu(407): warning #179-D: right operand of "%" is zero
        if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
                                         ^
          detected during instantiation of "void marlin_24::Marlin_24<num_bits,threads,thread_m_blocks,thread_n_blocks,thread_k_blocks,stages,group_blocks>(const int4 *, const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with num_bits=8, threads=256, thread_m_blocks=4, thread_n_blocks=16, thread_k_blocks=2, stages=4, group_blocks=-1]" at line 990

/home/lsakka/vllm/csrc/quantization/marlin/sparse/marlin_24_cuda_kernel.cu(438): warning #39-D: division by zero
                                 (pipe / (group_blocks / thread_k_blocks)));
                                         ^
          detected during instantiation of "void marlin_24::Marlin_24<num_bits,threads,thread_m_blocks,thread_n_blocks,thread_k_blocks,stages,group_blocks>(const int4 *, const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with num_bits=8, threads=256, thread_m_blocks=4, thread_n_blocks=16, thread_k_blocks=2, stages=4, group_blocks=-1]" at line 990

[21/27] Building CUDA object CMakeFiles/_C.dir/csrc/quantization/marlin/dense/marlin_cuda_kernel.cu.o
/home/lsakka/vllm/csrc/quantization/marlin/dense/marlin_cuda_kernel.cu(455): warning #179-D: right operand of "%" is zero
        if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
                                         ^
          detected during instantiation of "void marlin::Marlin<threads,thread_m_blocks,thread_n_blocks,thread_k_blocks,stages,group_blocks>(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=1, thread_n_blocks=8, thread_k_blocks=8, stages=4, group_blocks=-1]" at line 1025

Remark: The warnings can be suppressed with "-diag-suppress <warning-number>"

/home/lsakka/vllm/csrc/quantization/marlin/dense/marlin_cuda_kernel.cu(486): warning #39-D: division by zero
                                 (pipe / (group_blocks / thread_k_blocks)));
                                         ^
          detected during instantiation of "void marlin::Marlin<threads,thread_m_blocks,thread_n_blocks,thread_k_blocks,stages,group_blocks>(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=1, thread_n_blocks=8, thread_k_blocks=8, stages=4, group_blocks=-1]" at line 1025

/home/lsakka/vllm/csrc/quantization/marlin/dense/marlin_cuda_kernel.cu(455): warning #179-D: right operand of "%" is zero
        if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
                                         ^
          detected during instantiation of "void marlin::Marlin<threads,thread_m_blocks,thread_n_blocks,thread_k_blocks,stages,group_blocks>(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=2, thread_n_blocks=8, thread_k_blocks=8, stages=4, group_blocks=-1]" at line 1025

/home/lsakka/vllm/csrc/quantization/marlin/dense/marlin_cuda_kernel.cu(486): warning #39-D: division by zero
                                 (pipe / (group_blocks / thread_k_blocks)));
                                         ^
          detected during instantiation of "void marlin::Marlin<threads,thread_m_blocks,thread_n_blocks,thread_k_blocks,stages,group_blocks>(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=2, thread_n_blocks=8, thread_k_blocks=8, stages=4, group_blocks=-1]" at line 1025

/home/lsakka/vllm/csrc/quantization/marlin/dense/marlin_cuda_kernel.cu(455): warning #179-D: right operand of "%" is zero
        if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
                                         ^
          detected during instantiation of "void marlin::Marlin<threads,thread_m_blocks,thread_n_blocks,thread_k_blocks,stages,group_blocks>(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=3, thread_n_blocks=8, thread_k_blocks=8, stages=4, group_blocks=-1]" at line 1025

/home/lsakka/vllm/csrc/quantization/marlin/dense/marlin_cuda_kernel.cu(486): warning #39-D: division by zero
                                 (pipe / (group_blocks / thread_k_blocks)));
                                         ^
          detected during instantiation of "void marlin::Marlin<threads,thread_m_blocks,thread_n_blocks,thread_k_blocks,stages,group_blocks>(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=3, thread_n_blocks=8, thread_k_blocks=8, stages=4, group_blocks=-1]" at line 1025

/home/lsakka/vllm/csrc/quantization/marlin/dense/marlin_cuda_kernel.cu(455): warning #179-D: right operand of "%" is zero
        if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
                                         ^
          detected during instantiation of "void marlin::Marlin<threads,thread_m_blocks,thread_n_blocks,thread_k_blocks,stages,group_blocks>(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=4, thread_n_blocks=8, thread_k_blocks=8, stages=4, group_blocks=-1]" at line 1025

/home/lsakka/vllm/csrc/quantization/marlin/dense/marlin_cuda_kernel.cu(486): warning #39-D: division by zero
                                 (pipe / (group_blocks / thread_k_blocks)));
                                         ^
          detected during instantiation of "void marlin::Marlin<threads,thread_m_blocks,thread_n_blocks,thread_k_blocks,stages,group_blocks>(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=4, thread_n_blocks=8, thread_k_blocks=8, stages=4, group_blocks=-1]" at line 1025

/home/lsakka/vllm/csrc/quantization/marlin/dense/marlin_cuda_kernel.cu(455): warning #179-D: right operand of "%" is zero
        if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
                                         ^
          detected during instantiation of "void marlin::Marlin<threads,thread_m_blocks,thread_n_blocks,thread_k_blocks,stages,group_blocks>(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=1, thread_n_blocks=16, thread_k_blocks=4, stages=4, group_blocks=-1]" at line 1026

/home/lsakka/vllm/csrc/quantization/marlin/dense/marlin_cuda_kernel.cu(486): warning #39-D: division by zero
                                 (pipe / (group_blocks / thread_k_blocks)));
                                         ^
          detected during instantiation of "void marlin::Marlin<threads,thread_m_blocks,thread_n_blocks,thread_k_blocks,stages,group_blocks>(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=1, thread_n_blocks=16, thread_k_blocks=4, stages=4, group_blocks=-1]" at line 1026

/home/lsakka/vllm/csrc/quantization/marlin/dense/marlin_cuda_kernel.cu(455): warning #179-D: right operand of "%" is zero
        if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
                                         ^
          detected during instantiation of "void marlin::Marlin<threads,thread_m_blocks,thread_n_blocks,thread_k_blocks,stages,group_blocks>(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=2, thread_n_blocks=16, thread_k_blocks=4, stages=4, group_blocks=-1]" at line 1026

/home/lsakka/vllm/csrc/quantization/marlin/dense/marlin_cuda_kernel.cu(486): warning #39-D: division by zero
                                 (pipe / (group_blocks / thread_k_blocks)));
                                         ^
          detected during instantiation of "void marlin::Marlin<threads,thread_m_blocks,thread_n_blocks,thread_k_blocks,stages,group_blocks>(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=2, thread_n_blocks=16, thread_k_blocks=4, stages=4, group_blocks=-1]" at line 1026

/home/lsakka/vllm/csrc/quantization/marlin/dense/marlin_cuda_kernel.cu(455): warning #179-D: right operand of "%" is zero
        if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
                                         ^
          detected during instantiation of "void marlin::Marlin<threads,thread_m_blocks,thread_n_blocks,thread_k_blocks,stages,group_blocks>(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=3, thread_n_blocks=16, thread_k_blocks=4, stages=4, group_blocks=-1]" at line 1026

/home/lsakka/vllm/csrc/quantization/marlin/dense/marlin_cuda_kernel.cu(486): warning #39-D: division by zero
                                 (pipe / (group_blocks / thread_k_blocks)));
                                         ^
          detected during instantiation of "void marlin::Marlin<threads,thread_m_blocks,thread_n_blocks,thread_k_blocks,stages,group_blocks>(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=3, thread_n_blocks=16, thread_k_blocks=4, stages=4, group_blocks=-1]" at line 1026

/home/lsakka/vllm/csrc/quantization/marlin/dense/marlin_cuda_kernel.cu(455): warning #179-D: right operand of "%" is zero
        if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
                                         ^
          detected during instantiation of "void marlin::Marlin<threads,thread_m_blocks,thread_n_blocks,thread_k_blocks,stages,group_blocks>(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=4, thread_n_blocks=16, thread_k_blocks=4, stages=4, group_blocks=-1]" at line 1026

/home/lsakka/vllm/csrc/quantization/marlin/dense/marlin_cuda_kernel.cu(486): warning #39-D: division by zero
                                 (pipe / (group_blocks / thread_k_blocks)));
                                         ^
          detected during instantiation of "void marlin::Marlin<threads,thread_m_blocks,thread_n_blocks,thread_k_blocks,stages,group_blocks>(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=4, thread_n_blocks=16, thread_k_blocks=4, stages=4, group_blocks=-1]" at line 1026

/home/lsakka/vllm/csrc/quantization/marlin/dense/marlin_cuda_kernel.cu(455): warning #179-D: right operand of "%" is zero
        if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
                                         ^
          detected during instantiation of "void marlin::Marlin<threads,thread_m_blocks,thread_n_blocks,thread_k_blocks,stages,group_blocks>(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=1, thread_n_blocks=8, thread_k_blocks=4, stages=4, group_blocks=-1]" at line 1027

/home/lsakka/vllm/csrc/quantization/marlin/dense/marlin_cuda_kernel.cu(486): warning #39-D: division by zero
                                 (pipe / (group_blocks / thread_k_blocks)));
                                         ^
          detected during instantiation of "void marlin::Marlin<threads,thread_m_blocks,thread_n_blocks,thread_k_blocks,stages,group_blocks>(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=1, thread_n_blocks=8, thread_k_blocks=4, stages=4, group_blocks=-1]" at line 1027

/home/lsakka/vllm/csrc/quantization/marlin/dense/marlin_cuda_kernel.cu(455): warning #179-D: right operand of "%" is zero
        if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
                                         ^
          detected during instantiation of "void marlin::Marlin<threads,thread_m_blocks,thread_n_blocks,thread_k_blocks,stages,group_blocks>(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=2, thread_n_blocks=8, thread_k_blocks=4, stages=4, group_blocks=-1]" at line 1027

/home/lsakka/vllm/csrc/quantization/marlin/dense/marlin_cuda_kernel.cu(486): warning #39-D: division by zero
                                 (pipe / (group_blocks / thread_k_blocks)));
                                         ^
          detected during instantiation of "void marlin::Marlin<threads,thread_m_blocks,thread_n_blocks,thread_k_blocks,stages,group_blocks>(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=2, thread_n_blocks=8, thread_k_blocks=4, stages=4, group_blocks=-1]" at line 1027

/home/lsakka/vllm/csrc/quantization/marlin/dense/marlin_cuda_kernel.cu(455): warning #179-D: right operand of "%" is zero
        if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
                                         ^
          detected during instantiation of "void marlin::Marlin<threads,thread_m_blocks,thread_n_blocks,thread_k_blocks,stages,group_blocks>(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=3, thread_n_blocks=8, thread_k_blocks=4, stages=4, group_blocks=-1]" at line 1027

/home/lsakka/vllm/csrc/quantization/marlin/dense/marlin_cuda_kernel.cu(486): warning #39-D: division by zero
                                 (pipe / (group_blocks / thread_k_blocks)));
                                         ^
          detected during instantiation of "void marlin::Marlin<threads,thread_m_blocks,thread_n_blocks,thread_k_blocks,stages,group_blocks>(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=3, thread_n_blocks=8, thread_k_blocks=4, stages=4, group_blocks=-1]" at line 1027

/home/lsakka/vllm/csrc/quantization/marlin/dense/marlin_cuda_kernel.cu(455): warning #179-D: right operand of "%" is zero
        if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
                                         ^
          detected during instantiation of "void marlin::Marlin<threads,thread_m_blocks,thread_n_blocks,thread_k_blocks,stages,group_blocks>(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=4, thread_n_blocks=8, thread_k_blocks=4, stages=4, group_blocks=-1]" at line 1027

/home/lsakka/vllm/csrc/quantization/marlin/dense/marlin_cuda_kernel.cu(486): warning #39-D: division by zero
                                 (pipe / (group_blocks / thread_k_blocks)));
                                         ^
          detected during instantiation of "void marlin::Marlin<threads,thread_m_blocks,thread_n_blocks,thread_k_blocks,stages,group_blocks>(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=4, thread_n_blocks=8, thread_k_blocks=4, stages=4, group_blocks=-1]" at line 1027

/home/lsakka/vllm/csrc/quantization/marlin/dense/marlin_cuda_kernel.cu(455): warning #179-D: right operand of "%" is zero
        if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
                                         ^
          detected during instantiation of "void marlin::Marlin<threads,thread_m_blocks,thread_n_blocks,thread_k_blocks,stages,group_blocks>(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=1, thread_n_blocks=4, thread_k_blocks=8, stages=4, group_blocks=-1]" at line 1028

/home/lsakka/vllm/csrc/quantization/marlin/dense/marlin_cuda_kernel.cu(486): warning #39-D: division by zero
                                 (pipe / (group_blocks / thread_k_blocks)));
                                         ^
          detected during instantiation of "void marlin::Marlin<threads,thread_m_blocks,thread_n_blocks,thread_k_blocks,stages,group_blocks>(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=1, thread_n_blocks=4, thread_k_blocks=8, stages=4, group_blocks=-1]" at line 1028

/home/lsakka/vllm/csrc/quantization/marlin/dense/marlin_cuda_kernel.cu(455): warning #179-D: right operand of "%" is zero
        if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
                                         ^
          detected during instantiation of "void marlin::Marlin<threads,thread_m_blocks,thread_n_blocks,thread_k_blocks,stages,group_blocks>(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=2, thread_n_blocks=4, thread_k_blocks=8, stages=4, group_blocks=-1]" at line 1028

/home/lsakka/vllm/csrc/quantization/marlin/dense/marlin_cuda_kernel.cu(486): warning #39-D: division by zero
                                 (pipe / (group_blocks / thread_k_blocks)));
                                         ^
          detected during instantiation of "void marlin::Marlin<threads,thread_m_blocks,thread_n_blocks,thread_k_blocks,stages,group_blocks>(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=2, thread_n_blocks=4, thread_k_blocks=8, stages=4, group_blocks=-1]" at line 1028

/home/lsakka/vllm/csrc/quantization/marlin/dense/marlin_cuda_kernel.cu(455): warning #179-D: right operand of "%" is zero
        if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
                                         ^
          detected during instantiation of "void marlin::Marlin<threads,thread_m_blocks,thread_n_blocks,thread_k_blocks,stages,group_blocks>(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=3, thread_n_blocks=4, thread_k_blocks=8, stages=4, group_blocks=-1]" at line 1028

/home/lsakka/vllm/csrc/quantization/marlin/dense/marlin_cuda_kernel.cu(486): warning #39-D: division by zero
                                 (pipe / (group_blocks / thread_k_blocks)));
                                         ^
          detected during instantiation of "void marlin::Marlin<threads,thread_m_blocks,thread_n_blocks,thread_k_blocks,stages,group_blocks>(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=3, thread_n_blocks=4, thread_k_blocks=8, stages=4, group_blocks=-1]" at line 1028

/home/lsakka/vllm/csrc/quantization/marlin/dense/marlin_cuda_kernel.cu(455): warning #179-D: right operand of "%" is zero
        if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
                                         ^
          detected during instantiation of "void marlin::Marlin<threads,thread_m_blocks,thread_n_blocks,thread_k_blocks,stages,group_blocks>(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=4, thread_n_blocks=4, thread_k_blocks=8, stages=4, group_blocks=-1]" at line 1028

/home/lsakka/vllm/csrc/quantization/marlin/dense/marlin_cuda_kernel.cu(486): warning #39-D: division by zero
                                 (pipe / (group_blocks / thread_k_blocks)));
                                         ^
          detected during instantiation of "void marlin::Marlin<threads,thread_m_blocks,thread_n_blocks,thread_k_blocks,stages,group_blocks>(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=4, thread_n_blocks=4, thread_k_blocks=8, stages=4, group_blocks=-1]" at line 1028

[22/27] Building CUDA object CMakeFiles/_C.dir/csrc/quantization/fp8/fp8_marlin.cu.o
[23/27] Building CUDA object CMakeFiles/_C.dir/csrc/quantization/cutlass_w8a8/scaled_mm_c2x.cu.o
[24/27] Building CUDA object CMakeFiles/_C.dir/csrc/quantization/cutlass_w8a8/scaled_mm_c3x.cu.o
/home/lsakka/vllm/build/temp.linux-x86_64-cpython-311/_deps/cutlass-src/include/cutlass/device_kernel.h: In function 'void cutlass::device_kernel(typename Operator::Params) [with Operator = _GLOBAL__N__d5a9f71e_16_scaled_mm_c3x_cu_22d95651::cutlass_3x_gemm<signed char, cutlass::bfloat16_t, _GLOBAL__N__d5a9f71e_16_scaled_mm_c3x_cu_22d95651::ScaledEpilogueBias, cute::tuple<cute::C<64>, cute::C<64>, cute::C<256> >, cute::tuple<cute::C<1>, cute::C<8>, cute::C<1> >, cutlass::gemm::KernelTmaWarpSpecialized, cutlass::epilogue::TmaWarpSpecialized>::GemmKernel]':
/home/lsakka/vllm/build/temp.linux-x86_64-cpython-311/_deps/cutlass-src/include/cutlass/device_kernel.h:104:1: note: the ABI for passing parameters with 64-byte alignment has changed in GCC 4.6
  104 | void device_kernel(CUTLASS_GRID_CONSTANT typename Operator::Params const params)
      | ^~~~~~~~~~~~~
[25/27] Building CUDA object CMakeFiles/_C.dir/csrc/quantization/gptq_marlin/gptq_marlin.cu.o
[26/27] Building CUDA object CMakeFiles/_C.dir/csrc/attention/attention_kernels.cu.o
/home/lsakka/vllm/csrc/attention/attention_kernels.cu(709): warning #177-D: variable "thread_group_size" was declared but never referenced
    int thread_group_size = ((32 / BLOCK_SIZE) > (1) ? (32 / BLOCK_SIZE) : (1));
        ^

Remark: The warnings can be suppressed with "-diag-suppress <warning-number>"

/home/lsakka/vllm/csrc/attention/attention_kernels.cu(865): warning #177-D: variable "thread_group_size" was declared but never referenced
    int thread_group_size = ((32 / BLOCK_SIZE) > (1) ? (32 / BLOCK_SIZE) : (1));
        ^

[27/27] Linking CXX shared module /home/lsakka/vllm/build/lib.linux-x86_64-cpython-311/vllm/_C.abi3.so
copying build/lib.linux-x86_64-cpython-311/vllm/_moe_C.abi3.so -> vllm
copying build/lib.linux-x86_64-cpython-311/vllm/_C.abi3.so -> vllm
Creating /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages/vllm.egg-link (link to .)
Adding vllm 0.5.0.post1 to easy-install.pth file

Installed /home/lsakka/vllm
Processing dependencies for vllm==0.5.0.post1
Searching for nvidia-ml-py==12.555.43
Best match: nvidia-ml-py 12.555.43
Adding nvidia-ml-py 12.555.43 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for ray==2.31.0
Best match: ray 2.31.0
Adding ray 2.31.0 to easy-install.pth file
Installing ray script to /home/lsakka/.conda/envs/laith1991/bin
Installing rllib script to /home/lsakka/.conda/envs/laith1991/bin
Installing serve script to /home/lsakka/.conda/envs/laith1991/bin
Installing tune script to /home/lsakka/.conda/envs/laith1991/bin

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for filelock==3.13.1
Best match: filelock 3.13.1
Adding filelock 3.13.1 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for typing-extensions==4.11.0
Best match: typing-extensions 4.11.0
Adding typing-extensions 4.11.0 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for outlines==0.0.46
Best match: outlines 0.0.46
Adding outlines 0.0.46 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for lm-format-enforcer==0.10.1
Best match: lm-format-enforcer 0.10.1
Adding lm-format-enforcer 0.10.1 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for tiktoken==0.7.0
Best match: tiktoken 0.7.0
Adding tiktoken 0.7.0 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for prometheus-fastapi-instrumentator==7.0.0
Best match: prometheus-fastapi-instrumentator 7.0.0
Adding prometheus-fastapi-instrumentator 7.0.0 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for prometheus-client==0.20.0
Best match: prometheus-client 0.20.0
Adding prometheus-client 0.20.0 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for pillow==10.4.0
Best match: pillow 10.4.0
Adding pillow 10.4.0 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for pydantic==2.8.0
Best match: pydantic 2.8.0
Adding pydantic 2.8.0 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for uvicorn==0.30.1
Best match: uvicorn 0.30.1
Adding uvicorn 0.30.1 to easy-install.pth file
Installing uvicorn script to /home/lsakka/.conda/envs/laith1991/bin

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for openai==1.35.9
Best match: openai 1.35.9
Adding openai 1.35.9 to easy-install.pth file
Installing openai script to /home/lsakka/.conda/envs/laith1991/bin

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for aiohttp==3.9.5
Best match: aiohttp 3.9.5
Adding aiohttp 3.9.5 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for fastapi==0.111.0
Best match: fastapi 0.111.0
Adding fastapi 0.111.0 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for tokenizers==0.19.1
Best match: tokenizers 0.19.1
Adding tokenizers 0.19.1 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for transformers==4.42.3
Best match: transformers 4.42.3
Adding transformers 4.42.3 to easy-install.pth file
Installing transformers-cli script to /home/lsakka/.conda/envs/laith1991/bin

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for py-cpuinfo==9.0.0
Best match: py-cpuinfo 9.0.0
Adding py-cpuinfo 9.0.0 to easy-install.pth file
Installing cpuinfo script to /home/lsakka/.conda/envs/laith1991/bin

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for tqdm==4.66.4
Best match: tqdm 4.66.4
Adding tqdm 4.66.4 to easy-install.pth file
Installing tqdm script to /home/lsakka/.conda/envs/laith1991/bin

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for requests==2.32.3
Best match: requests 2.32.3
Adding requests 2.32.3 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for numpy==1.26.4
Best match: numpy 1.26.4
Adding numpy 1.26.4 to easy-install.pth file
Installing f2py script to /home/lsakka/.conda/envs/laith1991/bin

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for sentencepiece==0.2.0
Best match: sentencepiece 0.2.0
Adding sentencepiece 0.2.0 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for psutil==6.0.0
Best match: psutil 6.0.0
Adding psutil 6.0.0 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for ninja==<IP_ADDRESS>
Best match: ninja <IP_ADDRESS>
Adding ninja <IP_ADDRESS> to easy-install.pth file
Installing ninja script to /home/lsakka/.conda/envs/laith1991/bin

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for cmake==3.30.0
Best match: cmake 3.30.0
Adding cmake 3.30.0 to easy-install.pth file
Installing cmake script to /home/lsakka/.conda/envs/laith1991/bin
Installing cpack script to /home/lsakka/.conda/envs/laith1991/bin
Installing ctest script to /home/lsakka/.conda/envs/laith1991/bin

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for frozenlist==1.4.1
Best match: frozenlist 1.4.1
Adding frozenlist 1.4.1 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for aiosignal==1.3.1
Best match: aiosignal 1.3.1
Adding aiosignal 1.3.1 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for PyYAML==6.0.1
Best match: PyYAML 6.0.1
Adding PyYAML 6.0.1 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for protobuf==5.27.2
Best match: protobuf 5.27.2
Adding protobuf 5.27.2 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for packaging==23.2
Best match: packaging 23.2
Adding packaging 23.2 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for msgpack==1.0.8
Best match: msgpack 1.0.8
Adding msgpack 1.0.8 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for jsonschema==4.22.0
Best match: jsonschema 4.22.0
Adding jsonschema 4.22.0 to easy-install.pth file
Installing jsonschema script to /home/lsakka/.conda/envs/laith1991/bin

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for click==8.1.7
Best match: click 8.1.7
Adding click 8.1.7 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for pyairports==2.1.1
Best match: pyairports 2.1.1
Adding pyairports 2.1.1 to easy-install.pth file
Installing pyairports script to /home/lsakka/.conda/envs/laith1991/bin

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for pycountry==24.6.1
Best match: pycountry 24.6.1
Adding pycountry 24.6.1 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for datasets==2.20.0
Best match: datasets 2.20.0
Adding datasets 2.20.0 to easy-install.pth file
Installing datasets-cli script to /home/lsakka/.conda/envs/laith1991/bin

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for referencing==0.35.1
Best match: referencing 0.35.1
Adding referencing 0.35.1 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for numba==0.60.0
Best match: numba 0.60.0
Adding numba 0.60.0 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for diskcache==5.6.3
Best match: diskcache 5.6.3
Adding diskcache 5.6.3 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for cloudpickle==3.0.0
Best match: cloudpickle 3.0.0
Adding cloudpickle 3.0.0 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for nest-asyncio==1.6.0
Best match: nest-asyncio 1.6.0
Adding nest-asyncio 1.6.0 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for lark==1.1.9
Best match: lark 1.1.9
Adding lark 1.1.9 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for jinja2==3.1.4
Best match: jinja2 3.1.4
Adding jinja2 3.1.4 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for interegular==0.3.3
Best match: interegular 0.3.3
Adding interegular 0.3.3 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for regex==2024.5.15
Best match: regex 2024.5.15
Adding regex 2024.5.15 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for starlette==0.37.2
Best match: starlette 0.37.2
Adding starlette 0.37.2 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for pydantic-core==2.20.0
Best match: pydantic-core 2.20.0
Adding pydantic-core 2.20.0 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for annotated-types==0.7.0
Best match: annotated-types 0.7.0
Adding annotated-types 0.7.0 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for websockets==12.0
Best match: websockets 12.0
Adding websockets 12.0 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for watchfiles==0.22.0
Best match: watchfiles 0.22.0
Adding watchfiles 0.22.0 to easy-install.pth file
Installing watchfiles script to /home/lsakka/.conda/envs/laith1991/bin

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for uvloop==0.19.0
Best match: uvloop 0.19.0
Adding uvloop 0.19.0 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for python-dotenv==1.0.1
Best match: python-dotenv 1.0.1
Adding python-dotenv 1.0.1 to easy-install.pth file
Installing dotenv script to /home/lsakka/.conda/envs/laith1991/bin

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for httptools==0.6.1
Best match: httptools 0.6.1
Adding httptools 0.6.1 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for h11==0.14.0
Best match: h11 0.14.0
Adding h11 0.14.0 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for sniffio==1.3.1
Best match: sniffio 1.3.1
Adding sniffio 1.3.1 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for httpx==0.27.0
Best match: httpx 0.27.0
Adding httpx 0.27.0 to easy-install.pth file
Installing httpx script to /home/lsakka/.conda/envs/laith1991/bin

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for distro==1.9.0
Best match: distro 1.9.0
Adding distro 1.9.0 to easy-install.pth file
Installing distro script to /home/lsakka/.conda/envs/laith1991/bin

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for anyio==4.4.0
Best match: anyio 4.4.0
Adding anyio 4.4.0 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for yarl==1.9.4
Best match: yarl 1.9.4
Adding yarl 1.9.4 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for multidict==6.0.5
Best match: multidict 6.0.5
Adding multidict 6.0.5 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for attrs==23.2.0
Best match: attrs 23.2.0
Adding attrs 23.2.0 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for email-validator==2.2.0
Best match: email-validator 2.2.0
Adding email-validator 2.2.0 to easy-install.pth file
Installing email_validator script to /home/lsakka/.conda/envs/laith1991/bin

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for orjson==3.10.6
Best match: orjson 3.10.6
Adding orjson 3.10.6 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for ujson==5.10.0
Best match: ujson 5.10.0
Adding ujson 5.10.0 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for python-multipart==0.0.9
Best match: python-multipart 0.0.9
Adding python-multipart 0.0.9 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for fastapi-cli==0.0.4
Best match: fastapi-cli 0.0.4
Adding fastapi-cli 0.0.4 to easy-install.pth file
Installing fastapi script to /home/lsakka/.conda/envs/laith1991/bin

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for huggingface-hub==0.23.4
Best match: huggingface-hub 0.23.4
Adding huggingface-hub 0.23.4 to easy-install.pth file
Installing huggingface-cli script to /home/lsakka/.conda/envs/laith1991/bin

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for safetensors==0.4.3
Best match: safetensors 0.4.3
Adding safetensors 0.4.3 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for certifi==2024.6.2
Best match: certifi 2024.6.2
Adding certifi 2024.6.2 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for urllib3==2.2.2
Best match: urllib3 2.2.2
Adding urllib3 2.2.2 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for idna==3.7
Best match: idna 3.7
Adding idna 3.7 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for charset-normalizer==3.3.2
Best match: charset-normalizer 3.3.2
Adding charset-normalizer 3.3.2 to easy-install.pth file
Installing normalizer script to /home/lsakka/.conda/envs/laith1991/bin

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for rpds-py==0.18.1
Best match: rpds-py 0.18.1
Adding rpds-py 0.18.1 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for jsonschema-specifications==2023.12.1
Best match: jsonschema-specifications 2023.12.1
Adding jsonschema-specifications 2023.12.1 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for fsspec==2024.3.1
Best match: fsspec 2024.3.1
Adding fsspec 2024.3.1 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for multiprocess==0.70.16
Best match: multiprocess 0.70.16
Adding multiprocess 0.70.16 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for xxhash==3.4.1
Best match: xxhash 3.4.1
Adding xxhash 3.4.1 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for pandas==2.2.2
Best match: pandas 2.2.2
Adding pandas 2.2.2 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for dill==0.3.8
Best match: dill 0.3.8
Adding dill 0.3.8 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for pyarrow-hotfix==0.6
Best match: pyarrow-hotfix 0.6
Adding pyarrow-hotfix 0.6 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for pyarrow==16.1.0
Best match: pyarrow 16.1.0
Adding pyarrow 16.1.0 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for llvmlite==0.43.0
Best match: llvmlite 0.43.0
Adding llvmlite 0.43.0 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for MarkupSafe==2.1.3
Best match: MarkupSafe 2.1.3
Adding MarkupSafe 2.1.3 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for httpcore==1.0.5
Best match: httpcore 1.0.5
Adding httpcore 1.0.5 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for dnspython==2.6.1
Best match: dnspython 2.6.1
Adding dnspython 2.6.1 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for typer==0.12.3
Best match: typer 0.12.3
Adding typer 0.12.3 to easy-install.pth file
Installing typer script to /home/lsakka/.conda/envs/laith1991/bin

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for tzdata==2024.1
Best match: tzdata 2024.1
Adding tzdata 2024.1 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for pytz==2024.1
Best match: pytz 2024.1
Adding pytz 2024.1 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for python-dateutil==2.9.0.post0
Best match: python-dateutil 2.9.0.post0
Adding python-dateutil 2.9.0.post0 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for rich==13.7.1
Best match: rich 13.7.1
Adding rich 13.7.1 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for shellingham==1.5.4
Best match: shellingham 1.5.4
Adding shellingham 1.5.4 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for six==1.16.0
Best match: six 1.16.0
Adding six 1.16.0 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for pygments==2.18.0
Best match: pygments 2.18.0
Adding pygments 2.18.0 to easy-install.pth file
Installing pygmentize script to /home/lsakka/.conda/envs/laith1991/bin

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for markdown-it-py==3.0.0
Best match: markdown-it-py 3.0.0
Adding markdown-it-py 3.0.0 to easy-install.pth file
Installing markdown-it script to /home/lsakka/.conda/envs/laith1991/bin

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Searching for mdurl==0.1.2
Best match: mdurl 0.1.2
Adding mdurl 0.1.2 to easy-install.pth file

Using /home/lsakka/.conda/envs/laith1991/lib/python3.11/site-packages
Finished processing dependencies for vllm==0.5.0.post1

the undefined symbol cuTensorMapEncodeTiled is solved by workaround
export LD_PRELOAD=/usr/lib64/libcuda.so
``
nm -C  _C.abi3.so   | grep "cuTensorMap"
U cuTensorMapEncodeTiled
the workaround that you suggested to
export LD_PRELOAD=/usr/lib64/libcuda.so
now I am into new issue

import vllm
Traceback (most recent call last):
File "", line 1, in
File "/home/lsakka/vllm/vllm/init.py", line 3, in
from vllm.engine.arg_utils import AsyncEngineArgs, EngineArgs
File "/home/lsakka/vllm/vllm/engine/arg_utils.py", line 7, in
from vllm.config import (CacheConfig, DecodingConfig, DeviceConfig,
File "/home/lsakka/vllm/vllm/config.py", line 6, in
import torch
File "/home/lsakka/pytorch/torch/init.py", line 47, in
from torch._utils import (
File "/home/lsakka/pytorch/torch/_utils.py", line 4, in
import logging
File "/home/lsakka/vllm/vllm/logging/init.py", line 1, in
from vllm.logging.formatter import NewLineFormatter
File "/home/lsakka/vllm/vllm/logging/init.py", line 1, in
from vllm.logging.formatter import NewLineFormatter
File "/home/lsakka/vllm/vllm/logging/formatter.py", line 4, in
class NewLineFormatter(logging.Formatter):

```

import logging
File "/home/lsakka/vllm/vllm/logging/init.py", line 1, in

import logging goes to vllm/logging, which is caused by incorret python path i think. it should go to python builtin logging module.

I resolved the issue , I was running python from vllm/vllm folder and python was confused
for now I can import vllm with not issue
running
python benchmark_latency.py --model CohereForAI/c4ai-command-r-v01 -tp=4 --batch-size=1
hands crossed !

i get the following messages and running vllm stucks on the benchmark
 Found nccl from library libnccl.so.2
(VllmWorkerProcess pid=3656580) ERROR 07-14 19:17:50 pynccl_wrapper.py:196] Failed to load NCCL library from libnccl.so.2 .It is expected if you are not running on NVIDIA/AMD GPUs.Otherwise, the nccl library might not exist, be corrupted or it does not support the current platform Linux-5.12.0-0_fbk16_zion_7661_geb00762ce6d2-x86_64-with-glibc2.34.If you already have the library, please set the environment variable VLLM_NCCL_SO_PATH to point to the correct nccl library path.

th1991<EMAIL_ADDRESS>~/vllm/benchmarks (main)]$ python benchmark_latency.py --model CohereForAI/c4ai-command-r-v01 -tp=4 --batch-size=1
WARNING 07-14 19:17:44 _custom_ops.py:14] Failed to import from vllm._C with ImportError('/home/lsakka/vllm/vllm/_C.abi3.so: undefined symbol: cuTensorMapEncodeTiled')
Namespace(model='CohereForAI/c4ai-command-r-v01', speculative_model=None, num_speculative_tokens=None, speculative_draft_tensor_parallel_size=None, tokenizer=None, quantization=None, tensor_parallel_size=4, input_len=32, output_len=128, batch_size=1, n=1, use_beam_search=False, num_iters_warmup=10, num_iters=30, trust_remote_code=False, max_model_len=None, dtype='auto', enforce_eager=False, kv_cache_dtype='auto', quantization_param_path=None, profile=False, profile_result_dir=None, device='auto', block_size=16, enable_chunked_prefill=False, enable_prefix_caching=False, use_v2_block_manager=False, ray_workers_use_nsight=False, download_dir=None, output_json=None, gpu_memory_utilization=0.9, load_format='auto', distributed_executor_backend=None, otlp_traces_endpoint=None)
INFO 07-14 19:17:45 config.py:696] Defaulting to use mp for distributed inference
INFO 07-14 19:17:45 llm_engine.py:174] Initializing an LLM engine (v0.5.1) with config: model='CohereForAI/c4ai-command-r-v01', speculative_config=None, tokenizer='CohereForAI/c4ai-command-r-v01', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=8192, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=4, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None), seed=0, served_model_name=CohereForAI/c4ai-command-r-v01, use_v2_block_manager=False, enable_prefix_caching=False)
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
INFO 07-14 19:17:46 selector.py:191] Cannot use FlashAttention-2 backend because the vllm_flash_attn package is not found. `pip install vllm-flash-attn` for better performance.
INFO 07-14 19:17:46 selector.py:53] Using XFormers backend.
(VllmWorkerProcess pid=3656571) INFO 07-14 19:17:46 selector.py:191] Cannot use FlashAttention-2 backend because the vllm_flash_attn package is not found. `pip install vllm-flash-attn` for better performance.
(VllmWorkerProcess pid=3656580) INFO 07-14 19:17:46 selector.py:191] Cannot use FlashAttention-2 backend because the vllm_flash_attn package is not found. `pip install vllm-flash-attn` for better performance.
(VllmWorkerProcess pid=3656571) INFO 07-14 19:17:46 selector.py:53] Using XFormers backend.
(VllmWorkerProcess pid=3656580) INFO 07-14 19:17:46 selector.py:53] Using XFormers backend.
(VllmWorkerProcess pid=3656573) INFO 07-14 19:17:46 selector.py:191] Cannot use FlashAttention-2 backend because the vllm_flash_attn package is not found. `pip install vllm-flash-attn` for better performance.
(VllmWorkerProcess pid=3656573) INFO 07-14 19:17:46 selector.py:53] Using XFormers backend.
/home/lsakka/xformers/xformers/ops/fmha/flash.py:211: FutureWarning: `torch.library.impl_abstract` was renamed to `torch.library.register_fake`. Please use that instead; we will remove `torch.library.impl_abstract` in a future version of PyTorch.
  @torch.library.impl_abstract("xformers_flash::flash_fwd")
/home/lsakka/xformers/xformers/ops/fmha/flash.py:344: FutureWarning: `torch.library.impl_abstract` was renamed to `torch.library.register_fake`. Please use that instead; we will remove `torch.library.impl_abstract` in a future version of PyTorch.
  @torch.library.impl_abstract("xformers_flash::flash_bwd")
/home/lsakka/xformers/xformers/ops/swiglu_op.py:127: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd
/home/lsakka/xformers/xformers/ops/swiglu_op.py:148: FutureWarning: `torch.cuda.amp.custom_bwd(args...)` is deprecated. Please use `torch.amp.custom_bwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_bwd
(VllmWorkerProcess pid=3656580) /home/lsakka/xformers/xformers/ops/fmha/flash.py:211: FutureWarning: `torch.library.impl_abstract` was renamed to `torch.library.register_fake`. Please use that instead; we will remove `torch.library.impl_abstract` in a future version of PyTorch.
(VllmWorkerProcess pid=3656580)   @torch.library.impl_abstract("xformers_flash::flash_fwd")
(VllmWorkerProcess pid=3656580) /home/lsakka/xformers/xformers/ops/fmha/flash.py:344: FutureWarning: `torch.library.impl_abstract` was renamed to `torch.library.register_fake`. Please use that instead; we will remove `torch.library.impl_abstract` in a future version of PyTorch.
(VllmWorkerProcess pid=3656580)   @torch.library.impl_abstract("xformers_flash::flash_bwd")
(VllmWorkerProcess pid=3656580) /home/lsakka/xformers/xformers/ops/swiglu_op.py:127: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
(VllmWorkerProcess pid=3656580)   @torch.cuda.amp.custom_fwd
(VllmWorkerProcess pid=3656580) /home/lsakka/xformers/xformers/ops/swiglu_op.py:148: FutureWarning: `torch.cuda.amp.custom_bwd(args...)` is deprecated. Please use `torch.amp.custom_bwd(args..., device_type='cuda')` instead.
(VllmWorkerProcess pid=3656580)   @torch.cuda.amp.custom_bwd
(VllmWorkerProcess pid=3656580) INFO 07-14 19:17:49 multiproc_worker_utils.py:215] Worker ready; awaiting tasks
(VllmWorkerProcess pid=3656571) /home/lsakka/xformers/xformers/ops/fmha/flash.py:211: FutureWarning: `torch.library.impl_abstract` was renamed to `torch.library.register_fake`. Please use that instead; we will remove `torch.library.impl_abstract` in a future version of PyTorch.
(VllmWorkerProcess pid=3656571)   @torch.library.impl_abstract("xformers_flash::flash_fwd")
(VllmWorkerProcess pid=3656571) /home/lsakka/xformers/xformers/ops/fmha/flash.py:344: FutureWarning: `torch.library.impl_abstract` was renamed to `torch.library.register_fake`. Please use that instead; we will remove `torch.library.impl_abstract` in a future version of PyTorch.
(VllmWorkerProcess pid=3656571)   @torch.library.impl_abstract("xformers_flash::flash_bwd")
(VllmWorkerProcess pid=3656573) /home/lsakka/xformers/xformers/ops/fmha/flash.py:211: FutureWarning: `torch.library.impl_abstract` was renamed to `torch.library.register_fake`. Please use that instead; we will remove `torch.library.impl_abstract` in a future version of PyTorch.
(VllmWorkerProcess pid=3656573)   @torch.library.impl_abstract("xformers_flash::flash_fwd")
(VllmWorkerProcess pid=3656573) /home/lsakka/xformers/xformers/ops/fmha/flash.py:344: FutureWarning: `torch.library.impl_abstract` was renamed to `torch.library.register_fake`. Please use that instead; we will remove `torch.library.impl_abstract` in a future version of PyTorch.
(VllmWorkerProcess pid=3656573)   @torch.library.impl_abstract("xformers_flash::flash_bwd")
(VllmWorkerProcess pid=3656571) /home/lsakka/xformers/xformers/ops/swiglu_op.py:127: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
(VllmWorkerProcess pid=3656571)   @torch.cuda.amp.custom_fwd
(VllmWorkerProcess pid=3656571) /home/lsakka/xformers/xformers/ops/swiglu_op.py:148: FutureWarning: `torch.cuda.amp.custom_bwd(args...)` is deprecated. Please use `torch.amp.custom_bwd(args..., device_type='cuda')` instead.
(VllmWorkerProcess pid=3656571)   @torch.cuda.amp.custom_bwd
(VllmWorkerProcess pid=3656571) INFO 07-14 19:17:49 multiproc_worker_utils.py:215] Worker ready; awaiting tasks
(VllmWorkerProcess pid=3656573) /home/lsakka/xformers/xformers/ops/swiglu_op.py:127: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
(VllmWorkerProcess pid=3656573)   @torch.cuda.amp.custom_fwd
(VllmWorkerProcess pid=3656573) /home/lsakka/xformers/xformers/ops/swiglu_op.py:148: FutureWarning: `torch.cuda.amp.custom_bwd(args...)` is deprecated. Please use `torch.amp.custom_bwd(args..., device_type='cuda')` instead.
(VllmWorkerProcess pid=3656573)   @torch.cuda.amp.custom_bwd
(VllmWorkerProcess pid=3656573) INFO 07-14 19:17:49 multiproc_worker_utils.py:215] Worker ready; awaiting tasks
(VllmWorkerProcess pid=3656571) INFO 07-14 19:17:50 utils.py:741] Found nccl from library libnccl.so.2
INFO 07-14 19:17:50 utils.py:741] Found nccl from library libnccl.so.2
(VllmWorkerProcess pid=3656571) ERROR 07-14 19:17:50 pynccl_wrapper.py:196] Failed to load NCCL library from libnccl.so.2 .It is expected if you are not running on NVIDIA/AMD GPUs.Otherwise, the nccl library might not exist, be corrupted or it does not support the current platform Linux-5.12.0-0_fbk16_zion_7661_geb00762ce6d2-x86_64-with-glibc2.34.If you already have the library, please set the environment variable VLLM_NCCL_SO_PATH to point to the correct nccl library path.
(VllmWorkerProcess pid=3656573) INFO 07-14 19:17:50 utils.py:741] Found nccl from library libnccl.so.2
ERROR 07-14 19:17:50 pynccl_wrapper.py:196] Failed to load NCCL library from libnccl.so.2 .It is expected if you are not running on NVIDIA/AMD GPUs.Otherwise, the nccl library might not exist, be corrupted or it does not support the current platform Linux-5.12.0-0_fbk16_zion_7661_geb00762ce6d2-x86_64-with-glibc2.34.If you already have the library, please set the environment variable VLLM_NCCL_SO_PATH to point to the correct nccl library path.
(VllmWorkerProcess pid=3656573) ERROR 07-14 19:17:50 pynccl_wrapper.py:196] Failed to load NCCL library from libnccl.so.2 .It is expected if you are not running on NVIDIA/AMD GPUs.Otherwise, the nccl library might not exist, be corrupted or it does not support the current platform Linux-5.12.0-0_fbk16_zion_7661_geb00762ce6d2-x86_64-with-glibc2.34.If you already have the library, please set the environment variable VLLM_NCCL_SO_PATH to point to the correct nccl library path.
(VllmWorkerProcess pid=3656580) INFO 07-14 19:17:50 utils.py:741] Found nccl from library libnccl.so.2
(VllmWorkerProcess pid=3656580) ERROR 07-14 19:17:50 pynccl_wrapper.py:196] Failed to load NCCL library from libnccl.so.2 .It is expected if you are not running on NVIDIA/AMD GPUs.Otherwise, the nccl library might not exist, be corrupted or it does not support the current platform Linux-5.12.0-0_fbk16_zion_7661_geb00762ce6d2-x86_64-with-glibc2.34.If you already have the library, please set the environment variable VLLM_NCCL_SO_PATH to point to the correct nccl library path.

how does your nightly pytorch use NCCL? static link or dynamic link? which NCCL does it use?

I fixed that by setting the version of the nccl libarary
VLLM_NCCL_SO_PATH=/home/lsakka/pytorch/build/nccl/lib/libnccl.so.2  python benchmark_latency.py --model CohereForAI/c4ai-command-r-v01 -tp=4 --batch-size=1

the benchmark still gets stuck though, i have seen local posts about nccl getting stuck and having tor revert not sure if related.

can you try to follow the debugging guide https://docs.vllm.ai/en/latest/getting_started/debugging.html ? there is a sanity check script to help you locate the problem.

I used the VLLM_TRACE and it get stuck at
2024-07-16 13:56:37.432732 Call to wait_until_ready in /home/lsakka/vllm/vllm/distributed/device_communicators/shm_broadcast.py:284 from create_from_process_group in /home/lsakka/vllm/vllm/distributed/device_communicators/shm_broadcast.py:489

the warning is irrelevant.
can you stably reproduce this hang? if so, you can remove VLLM_TRACE_FUNCTION, and set a breakpoint in /home/lsakka/vllm/vllm/distributed/device_communicators/shm_broadcast.py:284 to see which line hangs.
that part of code sets up a publish-subscribe message queue for communication.

I used the VLLM_TRACE and it get stuck at 2024-07-16 13:56:37.432732 Call to wait_until_ready in /home/lsakka/vllm/vllm/distributed/device_communicators/shm_broadcast.py:284 from create_from_process_group in /home/lsakka/vllm/vllm/distributed/device_communicators/shm_broadcast.py:489
note the following warning which i am unusre if it related
llmWorkerProcess pid=1194815) WARNING 07-16 14:31:56 custom_all_reduce.py:118] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.
(VllmWorkerProcess pid=1194818) WARNING 07-16 14:31:56 custom_all_reduce.py:118] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.
(VllmWorkerProcess pid=1194814) WARNING 07-16 14:31:56 custom_all_reduce.py:118] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.
WARNING 07-16 14:31:56 custom_all_reduce.py:118] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.

+1 hanged there at wait_until_ready()

I used the VLLM_TRACE and it get stuck at 2024-07-16 13:56:37.432732 Call to wait_until_ready in /home/lsakka/vllm/vllm/distributed/device_communicators/shm_broadcast.py:284 from create_from_process_group in /home/lsakka/vllm/vllm/distributed/device_communicators/shm_broadcast.py:489
note the following warning which i am unusre if it related
llmWorkerProcess pid=1194815) WARNING 07-16 14:31:56 custom_all_reduce.py:118] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.
(VllmWorkerProcess pid=1194818) WARNING 07-16 14:31:56 custom_all_reduce.py:118] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.
(VllmWorkerProcess pid=1194814) WARNING 07-16 14:31:56 custom_all_reduce.py:118] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.
WARNING 07-16 14:31:56 custom_all_reduce.py:118] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.

+1 hanged there at wait_until_ready() vllm 0.5.2 run in Docker on host network

It turns out that using host network on a IPv6 only machine is problematic. Bridged network works just fine on my side

In v0.5.2, v0.5.3, and v0.5.3.post1, there is a bug caused by zmq , which can cause hangs at a low probability (once in about 20 times, depending on the machine configuration). The solution is to upgrade to the latest version of vllm to include the fix .


Maxime/update gitlab trigger

What does this PR do?

Move release jobs to run only when the pipeline it triggered. All release jobs (i.e. jobs run when producing a nightly or building a tag) are now only run when the pipeline is triggered. This avoids building a version named after a tag but with the nightly versions from release.json inside.

Avoid updating the latest build when needed. This is useful when building a custom beta for example.

@JulienBalestra could you please confirm this meets what we want for e2e?

LGTM @xvello @hush-hush