View the dashboard at <IP_ADDRESS>:8265
INFO 03-29 13:57:09 llm_engine.py:70] Initializing an LLM engine (v0.3.3) with config: model=/models/openchat-3.5-0106/, tokenizer=/models/openchat-3.5-0106/, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=8192, download_dir=None, load_format=auto, tensor_parallel_size=4, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, kv_cache_dtype=auto, device_config=cuda, seed=0)
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
INFO 03-29 13:57:22 pynccl.py:49] Loading nccl from library libnccl.so
INFO 03-29 13:57:22 pynccl_utils.py:13] vLLM is using nccl==2.17.1
INFO 03-29 13:57:23 selector.py:33] Cannot use FlashAttention backend for Volta and Turing GPUs.
INFO 03-29 13:57:23 selector.py:20] Using XFormers backend.
(RayWorkerVllm pid=392, ip=<IP_ADDRESS>) INFO 03-29 13:57:16 pynccl.py:49] Loading nccl from library libnccl.so
(RayWorkerVllm pid=392, ip=<IP_ADDRESS>) INFO 03-29 13:57:16 pynccl_utils.py:13] vLLM is using nccl==2.17.1
(RayWorkerVllm pid=11442) INFO 03-29 13:57:25 selector.py:33] Cannot use FlashAttention backend for Volta and Turing GPUs.
(RayWorkerVllm pid=11442) INFO 03-29 13:57:25 selector.py:20] Using XFormers backend.
(RayWorkerVllm pid=309, ip=<IP_ADDRESS>) ERROR 03-29 13:57:18 ray_utils.py:44] Error executing method init_device. This might cause deadlock in distributed execution.
(RayWorkerVllm pid=309, ip=<IP_ADDRESS>) ERROR 03-29 13:57:18 ray_utils.py:44] Traceback (most recent call last):
(RayWorkerVllm pid=309, ip=<IP_ADDRESS>) ERROR 03-29 13:57:18 ray_utils.py:44]   File /usr/local/lib/python3.10/dist-packages/vllm/engine/ray_utils.py, line 37, in execute_method
(RayWorkerVllm pid=309, ip=<IP_ADDRESS>) ERROR 03-29 13:57:18 ray_utils.py:44]     return executor(*args, **kwargs)
(RayWorkerVllm pid=309, ip=<IP_ADDRESS>) ERROR 03-29 13:57:18 ray_utils.py:44]   File /usr/local/lib/python3.10/dist-packages/vllm/worker/worker.py, line 100, in init_device
(RayWorkerVllm pid=309, ip=<IP_ADDRESS>) ERROR 03-29 13:57:18 ray_utils.py:44]     init_distributed_environment(self.parallel_config, self.rank,
(RayWorkerVllm pid=309, ip=<IP_ADDRESS>) ERROR 03-29 13:57:18 ray_utils.py:44]   File /usr/local/lib/python3.10/dist-packages/vllm/worker/worker.py, line 286, in init_distributed_environment
(RayWorkerVllm pid=309, ip=<IP_ADDRESS>) ERROR 03-29 13:57:18 ray_utils.py:44]     pynccl_utils.init_process_group(
(RayWorkerVllm pid=309, ip=<IP_ADDRESS>) ERROR 03-29 13:57:18 ray_utils.py:44]   File /usr/local/lib/python3.10/dist-packages/vllm/model_executor/parallel_utils/pynccl_utils.py, line 42, in init_process_group
(RayWorkerVllm pid=309, ip=<IP_ADDRESS>) ERROR 03-29 13:57:18 ray_utils.py:44]     comm = NCCLCommunicator(init_method=init_method,
(RayWorkerVllm pid=309, ip=<IP_ADDRESS>) ERROR 03-29 13:57:18 ray_utils.py:44]   File /usr/local/lib/python3.10/dist-packages/vllm/model_executor/parallel_utils/pynccl.py, line 226, in __init__
(RayWorkerVllm pid=309, ip=<IP_ADDRESS>) ERROR 03-29 13:57:18 ray_utils.py:44]     torch.cuda.set_device(self.rank)
(RayWorkerVllm pid=309, ip=<IP_ADDRESS>) ERROR 03-29 13:57:18 ray_utils.py:44]   File /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py, line 404, in set_device
(RayWorkerVllm pid=309, ip=<IP_ADDRESS>) ERROR 03-29 13:57:18 ray_utils.py:44]     torch._C._cuda_setDevice(device)
(RayWorkerVllm pid=309, ip=<IP_ADDRESS>) ERROR 03-29 13:57:18 ray_utils.py:44] RuntimeError: CUDA error: invalid device ordinal
(RayWorkerVllm pid=309, ip=<IP_ADDRESS>) ERROR 03-29 13:57:18 ray_utils.py:44] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
(RayWorkerVllm pid=309, ip=<IP_ADDRESS>) ERROR 03-29 13:57:18 ray_utils.py:44] For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
(RayWorkerVllm pid=309, ip=<IP_ADDRESS>) ERROR 03-29 13:57:18 ray_utils.py:44] Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
(RayWorkerVllm pid=309, ip=<IP_ADDRESS>) ERROR 03-29 13:57:18 ray_utils.py:44]
(RayWorkerVllm pid=309, ip=<IP_ADDRESS>) Exception ignored in:
(RayWorkerVllm pid=309, ip=<IP_ADDRESS>) Traceback (most recent call last):
(RayWorkerVllm pid=309, ip=<IP_ADDRESS>)   File /usr/local/lib/python3.10/dist-packages/vllm/model_executor/parallel_utils/pynccl.py, line 260, in __del__
(RayWorkerVllm pid=309, ip=<IP_ADDRESS>)     _c_ncclCommDestroy(self.comm)
(RayWorkerVllm pid=309, ip=<IP_ADDRESS>) AttributeError: NCCLCommunicator object has no attribute comm

Hi, can you try to build from source with the latest main? https://github.com/vllm-project/vllm/pull/3686 should resolve your problem I think.
BTW, when you use two nodes, do you use ray to set up the two nodes as a cluster?

Hi, can you try to build from source with the latest main? #3686 should resolve your problem I think.
BTW, when you use two nodes, do you use ray to set up the two nodes as a cluster?

I'll try build vllm from latest main
the cluster is set up with two nodes

new vllm-build(0.4.0) comes new ERROR.

    ncclRemoteError: A call failed possibly due to a network error or a remote process exiting prematurely.

root@ai151:/# python3 -m vllm.entrypoints.api_server --model /models/openchat-3.5-0106/ --tensor-parallel-size 4 --dtype float16 --enforce-eager
WARNING 04-01 10:49:31 config.py:748] Casting torch.bfloat16 to torch.float16.
2024-04-01 10:49:31,995	INFO worker.py:1567 -- Connecting to existing Ray cluster at address: <IP_ADDRESS>:6379...
2024-04-01 10:49:32,005	INFO worker.py:1743 -- Connected to Ray cluster. View the dashboard at <IP_ADDRESS>:8265
INFO 04-01 10:49:32 llm_engine.py:75] Initializing an LLM engine (v0.4.0) with config: model=/models/openchat-3.5-0106/, tokenizer=/models/openchat-3.5-0106/, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=8192, download_dir=None, load_format=auto, tensor_parallel_size=4, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, kv_cache_dtype=auto, device_config=cuda, seed=0)
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
(RayWorkerVllm pid=306, ip=<IP_ADDRESS>) INFO 04-01 10:49:42 selector.py:34] Cannot use FlashAttention backend for Volta and Turing GPUs.
INFO 04-01 10:49:52 selector.py:34] Cannot use FlashAttention backend for Volta and Turing GPUs.
INFO 04-01 10:49:52 selector.py:21] Using XFormers backend.
(RayWorkerVllm pid=306, ip=<IP_ADDRESS>) INFO 04-01 10:49:42 selector.py:21] Using XFormers backend.
(RayWorkerVllm pid=11716) INFO 04-01 10:49:53 pynccl_utils.py:45] vLLM is using nccl==2.18.1
INFO 04-01 10:49:54 pynccl_utils.py:45] vLLM is using nccl==2.18.1
(RayWorkerVllm pid=394, ip=<IP_ADDRESS>) Exception ignored in:
(RayWorkerVllm pid=394, ip=<IP_ADDRESS>) Traceback (most recent call last):
(RayWorkerVllm pid=394, ip=<IP_ADDRESS>)   File /usr/local/lib/python3.10/dist-packages/vllm-0.4.0-py3.10-linux-x86_64.egg/vllm/model_executor/parallel_utils/pynccl.py, line 264, in __del__
(RayWorkerVllm pid=394, ip=<IP_ADDRESS>)     _c_ncclCommDestroy(self.comm)
(RayWorkerVllm pid=394, ip=<IP_ADDRESS>) AttributeError: NCCLCommunicator object has no attribute comm
(RayWorkerVllm pid=394, ip=<IP_ADDRESS>) ERROR 04-01 10:49:46 ray_utils.py:44] Error executing method init_device. This might cause deadlock in distributed execution.
(RayWorkerVllm pid=394, ip=<IP_ADDRESS>) ERROR 04-01 10:49:46 ray_utils.py:44] Traceback (most recent call last):
(RayWorkerVllm pid=394, ip=<IP_ADDRESS>) ERROR 04-01 10:49:46 ray_utils.py:44]   File /usr/local/lib/python3.10/dist-packages/vllm-0.4.0-py3.10-linux-x86_64.egg/vllm/engine/ray_utils.py, line 37, in execute_method
(RayWorkerVllm pid=394, ip=<IP_ADDRESS>) ERROR 04-01 10:49:46 ray_utils.py:44]     return executor(*args, **kwargs)
(RayWorkerVllm pid=394, ip=<IP_ADDRESS>) ERROR 04-01 10:49:46 ray_utils.py:44]   File /usr/local/lib/python3.10/dist-packages/vllm-0.4.0-py3.10-linux-x86_64.egg/vllm/worker/worker.py, line 100, in init_device
(RayWorkerVllm pid=394, ip=<IP_ADDRESS>) ERROR 04-01 10:49:46 ray_utils.py:44]     init_distributed_environment(self.parallel_config, self.rank,
(RayWorkerVllm pid=394, ip=<IP_ADDRESS>) ERROR 04-01 10:49:46 ray_utils.py:44]   File /usr/local/lib/python3.10/dist-packages/vllm-0.4.0-py3.10-linux-x86_64.egg/vllm/worker/worker.py, line 287, in init_distributed_environment
(RayWorkerVllm pid=394, ip=<IP_ADDRESS>) ERROR 04-01 10:49:46 ray_utils.py:44]     pynccl_utils.init_process_group(
(RayWorkerVllm pid=394, ip=<IP_ADDRESS>) ERROR 04-01 10:49:46 ray_utils.py:44]   File /usr/local/lib/python3.10/dist-packages/vllm-0.4.0-py3.10-linux-x86_64.egg/vllm/model_executor/parallel_utils/pynccl_utils.py, line 46, in init_process_group
(RayWorkerVllm pid=394, ip=<IP_ADDRESS>) ERROR 04-01 10:49:46 ray_utils.py:44]     comm = NCCLCommunicator(init_method=init_method,
(RayWorkerVllm pid=394, ip=<IP_ADDRESS>) ERROR 04-01 10:49:46 ray_utils.py:44]   File /usr/local/lib/python3.10/dist-packages/vllm-0.4.0-py3.10-linux-x86_64.egg/vllm/model_executor/parallel_utils/pynccl.py, line 236, in __init__
(RayWorkerVllm pid=394, ip=<IP_ADDRESS>) ERROR 04-01 10:49:46 ray_utils.py:44]     dist.broadcast(tensor, src=0)
(RayWorkerVllm pid=394, ip=<IP_ADDRESS>) ERROR 04-01 10:49:46 ray_utils.py:44]   File /usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py, line 47, in wrapper
(RayWorkerVllm pid=394, ip=<IP_ADDRESS>) ERROR 04-01 10:49:46 ray_utils.py:44]     return func(*args, **kwargs)
(RayWorkerVllm pid=394, ip=<IP_ADDRESS>) ERROR 04-01 10:49:46 ray_utils.py:44]   File /usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py, line 1906, in broadcast
(RayWorkerVllm pid=394, ip=<IP_ADDRESS>) ERROR 04-01 10:49:46 ray_utils.py:44]     work = default_pg.broadcast([tensor], opts)
(RayWorkerVllm pid=394, ip=<IP_ADDRESS>) ERROR 04-01 10:49:46 ray_utils.py:44] torch.distributed.DistBackendError: NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1333, remote process exited or there was a network error, NCCL version 2.18.1
(RayWorkerVllm pid=394, ip=<IP_ADDRESS>) ERROR 04-01 10:49:46 ray_utils.py:44] ncclRemoteError: A call failed possibly due to a network error or a remote process exiting prematurely.
(RayWorkerVllm pid=394, ip=<IP_ADDRESS>) ERROR 04-01 10:49:46 ray_utils.py:44] Last error:
(RayWorkerVllm pid=394, ip=<IP_ADDRESS>) ERROR 04-01 10:49:46 ray_utils.py:44] socketProgressOpt: Call to recv from <IP_ADDRESS><57269> failed : Broken pipe

here's build log, looks like good.

vllm master(@563c1d7ec56aa0f9fdc28720f3517bf9297f5476) build log

root@7fc7fec8839f:/tmp/vllm# python3 setup.py  install
No CUDA runtime is found, using CUDA_HOME=/usr/local/cuda
running install
/usr/local/lib/python3.10/dist-packages/setuptools/_distutils/cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated.
!!
********************************************************************************
Please avoid running ``setup.py`` directly.
Instead, use pypa/build, pypa/installer or other
standards-based tools.
See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.
********************************************************************************
!!
self.initialize_options()
/usr/local/lib/python3.10/dist-packages/setuptools/_distutils/cmd.py:66: EasyInstallDeprecationWarning: easy_install command is deprecated.
!!
********************************************************************************
Please avoid running ``setup.py`` and ``easy_install``.
Instead, use pypa/build, pypa/installer or other
standards-based tools.
See https://github.com/pypa/setuptools/issues/917 for details.
********************************************************************************
!!
self.initialize_options()
running bdist_egg
running egg_info
writing vllm.egg-info/PKG-INFO
writing dependency_links to vllm.egg-info/dependency_links.txt
writing requirements to vllm.egg-info/requires.txt
writing top-level names to vllm.egg-info/top_level.txt
reading manifest file vllm.egg-info/SOURCES.txt
reading manifest template MANIFEST.in
adding license file LICENSE
writing manifest file vllm.egg-info/SOURCES.txt
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
running build_ext
-- Build type: RelWithDebInfo
-- Found python matching: /usr/bin/python3.
-- Caffe2: CUDA detected: 12.1
-- Caffe2: CUDA nvcc is: /usr/local/cuda/bin/nvcc
-- Caffe2: CUDA toolkit directory: /usr/local/cuda
-- Caffe2: Header version is: 12.1
-- /usr/local/cuda/lib64/libnvrtc.so shorthash is b51b459d
-- USE_CUDNN is set to 0. Compiling without cuDNN support
-- USE_CUSPARSELT is set to 0. Compiling without cuSPARSELt support
-- Automatic GPU detection failed. Building for common architectures.
-- Autodetected CUDA architecture(s): 3.5;5.0;8.0;8.6;8.9;9.0
-- Added CUDA NVCC flags for: -gencode;arch=compute_35,code=sm_35;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_89,code=sm_89;-gencode;arch=compute_90,code=sm_90
CMake Warning at /usr/local/lib/python3.10/dist-packages/torch/share/cmake/Torch/TorchConfig.cmake:22 (message):
static library kineto_LIBRARY-NOTFOUND not found.
Call Stack (most recent call first):
/usr/local/lib/python3.10/dist-packages/torch/share/cmake/Torch/TorchConfig.cmake:127 (append_torchlib_if_found)
CMakeLists.txt:64 (find_package)
-- CUDA supported arches: 7.0;7.5;8.0;8.6;8.9;9.0
-- discarding unsupported CUDA arch 3.10.
-- discarding unsupported CUDA arch 3.10.
-- CUDA target arches: 80-real;86-real;89-real;90-real
-- Punica target arches: 80-real;86-real;89-real;90-real
-- Enabling C extension.
-- Enabling moe extension.
-- Configuring done (8.2s)
-- Generating done (0.0s)
-- Build files have been written to: /tmp/vllm/build/temp.linux-x86_64-cpython-310
[3/3] Linking CXX shared module /tmp/vllm/build/lib.linux-x86_64-cpython-310/vllm/_moe_C.cpython-310-x86_64-linux-gnu.so
[6/14] Building CUDA object CMakeFiles/_C.dir/csrc/quantization/awq/gemm_kernels.cu.o
/tmp/vllm/csrc/quantization/awq/gemm_kernels.cu(282): warning #177-D: variable j_factors1 was declared but never referenced
int j_factors1 = 4;
^
Remark: The warnings can be suppressed with -diag-suppress
/tmp/vllm/csrc/quantization/awq/gemm_kernels.cu(283): warning #177-D: variable row_stride2 was declared but never referenced
int row_stride2 = 4;
^
/tmp/vllm/csrc/quantization/awq/gemm_kernels.cu(284): warning #177-D: variable split_k_iters was declared but never referenced
int split_k_iters = 1;
^
/tmp/vllm/csrc/quantization/awq/gemm_kernels.cu(290): warning #177-D: variable B_shared_warp was declared but never referenced
half B_shared_warp[32];
^
/tmp/vllm/csrc/quantization/awq/gemm_kernels.cu(291): warning #177-D: variable OC was declared but never referenced
int OC = 512;
^
/tmp/vllm/csrc/quantization/awq/gemm_kernels.cu(53): warning #177-D: variable scaling_factors_shared was declared but never referenced
half scaling_factors_shared[N];
^
/tmp/vllm/csrc/quantization/awq/gemm_kernels.cu(54): warning #177-D: variable zeros_shared was declared but never referenced
half zeros_shared[N];
^
/tmp/vllm/csrc/quantization/awq/gemm_kernels.cu(57): warning #177-D: variable blockIdx_x was declared but never referenced
int blockIdx_x = 0;
^
/tmp/vllm/csrc/quantization/awq/gemm_kernels.cu(71): warning #177-D: variable ld_zero_flag was declared but never referenced
bool ld_zero_flag = (threadIdx.y * 32 + threadIdx.x) * 8 < N;
^
/tmp/vllm/csrc/quantization/awq/gemm_kernels.cu(24): warning #177-D: function vllm::awq::__pack_half2 was declared but never referenced
__pack_half2(const half x, const half y) {
^
/tmp/vllm/csrc/quantization/awq/gemm_kernels.cu(282): warning #177-D: variable j_factors1 was declared but never referenced
int j_factors1 = 4;
^
Remark: The warnings can be suppressed with -diag-suppress
/tmp/vllm/csrc/quantization/awq/gemm_kernels.cu(283): warning #177-D: variable row_stride2 was declared but never referenced
int row_stride2 = 4;
^
/tmp/vllm/csrc/quantization/awq/gemm_kernels.cu(284): warning #177-D: variable split_k_iters was declared but never referenced
int split_k_iters = 1;
^
/tmp/vllm/csrc/quantization/awq/gemm_kernels.cu(290): warning #177-D: variable B_shared_warp was declared but never referenced
half B_shared_warp[32];
^
/tmp/vllm/csrc/quantization/awq/gemm_kernels.cu(291): warning #177-D: variable OC was declared but never referenced
int OC = 512;
^
/tmp/vllm/csrc/quantization/awq/gemm_kernels.cu(53): warning #177-D: variable scaling_factors_shared was declared but never referenced
half scaling_factors_shared[N];
^
/tmp/vllm/csrc/quantization/awq/gemm_kernels.cu(54): warning #177-D: variable zeros_shared was declared but never referenced
half zeros_shared[N];
^
/tmp/vllm/csrc/quantization/awq/gemm_kernels.cu(57): warning #177-D: variable blockIdx_x was declared but never referenced
int blockIdx_x = 0;
^
/tmp/vllm/csrc/quantization/awq/gemm_kernels.cu(71): warning #177-D: variable ld_zero_flag was declared but never referenced
bool ld_zero_flag = (threadIdx.y * 32 + threadIdx.x) * 8 < N;
^
/tmp/vllm/csrc/quantization/awq/gemm_kernels.cu(24): warning #177-D: function vllm::awq::__pack_half2 was declared but never referenced
__pack_half2(const half x, const half y) {
^
/tmp/vllm/csrc/quantization/awq/gemm_kernels.cu(282): warning #177-D: variable j_factors1 was declared but never referenced
int j_factors1 = 4;
^
Remark: The warnings can be suppressed with -diag-suppress
/tmp/vllm/csrc/quantization/awq/gemm_kernels.cu(283): warning #177-D: variable row_stride2 was declared but never referenced
int row_stride2 = 4;
^
/tmp/vllm/csrc/quantization/awq/gemm_kernels.cu(284): warning #177-D: variable split_k_iters was declared but never referenced
int split_k_iters = 1;
^
/tmp/vllm/csrc/quantization/awq/gemm_kernels.cu(290): warning #177-D: variable B_shared_warp was declared but never referenced
half B_shared_warp[32];
^
/tmp/vllm/csrc/quantization/awq/gemm_kernels.cu(291): warning #177-D: variable OC was declared but never referenced
int OC = 512;
^
/tmp/vllm/csrc/quantization/awq/gemm_kernels.cu(53): warning #177-D: variable scaling_factors_shared was declared but never referenced
half scaling_factors_shared[N];
^
/tmp/vllm/csrc/quantization/awq/gemm_kernels.cu(54): warning #177-D: variable zeros_shared was declared but never referenced
half zeros_shared[N];
^
/tmp/vllm/csrc/quantization/awq/gemm_kernels.cu(57): warning #177-D: variable blockIdx_x was declared but never referenced
int blockIdx_x = 0;
^
/tmp/vllm/csrc/quantization/awq/gemm_kernels.cu(71): warning #177-D: variable ld_zero_flag was declared but never referenced
bool ld_zero_flag = (threadIdx.y * 32 + threadIdx.x) * 8 < N;
^
/tmp/vllm/csrc/quantization/awq/gemm_kernels.cu(24): warning #177-D: function vllm::awq::__pack_half2 was declared but never referenced
__pack_half2(const half x, const half y) {
^
/tmp/vllm/csrc/quantization/awq/gemm_kernels.cu(282): warning #177-D: variable j_factors1 was declared but never referenced
int j_factors1 = 4;
^
Remark: The warnings can be suppressed with -diag-suppress
/tmp/vllm/csrc/quantization/awq/gemm_kernels.cu(283): warning #177-D: variable row_stride2 was declared but never referenced
int row_stride2 = 4;
^
/tmp/vllm/csrc/quantization/awq/gemm_kernels.cu(284): warning #177-D: variable split_k_iters was declared but never referenced
int split_k_iters = 1;
^
/tmp/vllm/csrc/quantization/awq/gemm_kernels.cu(290): warning #177-D: variable B_shared_warp was declared but never referenced
half B_shared_warp[32];
^
/tmp/vllm/csrc/quantization/awq/gemm_kernels.cu(291): warning #177-D: variable OC was declared but never referenced
int OC = 512;
^
/tmp/vllm/csrc/quantization/awq/gemm_kernels.cu(53): warning #177-D: variable scaling_factors_shared was declared but never referenced
half scaling_factors_shared[N];
^
/tmp/vllm/csrc/quantization/awq/gemm_kernels.cu(54): warning #177-D: variable zeros_shared was declared but never referenced
half zeros_shared[N];
^
/tmp/vllm/csrc/quantization/awq/gemm_kernels.cu(57): warning #177-D: variable blockIdx_x was declared but never referenced
int blockIdx_x = 0;
^
/tmp/vllm/csrc/quantization/awq/gemm_kernels.cu(71): warning #177-D: variable ld_zero_flag was declared but never referenced
bool ld_zero_flag = (threadIdx.y * 32 + threadIdx.x) * 8 < N;
^
/tmp/vllm/csrc/quantization/awq/gemm_kernels.cu(24): warning #177-D: function vllm::awq::__pack_half2 was declared but never referenced
__pack_half2(const half x, const half y) {
^
[7/14] Building CUDA object CMakeFiles/_C.dir/csrc/quantization/squeezellm/quant_cuda_kernel.cu.o
/tmp/vllm/csrc/quantization/squeezellm/quant_cuda_kernel.cu: In function ‘void squeezellm_gemm(at::Tensor, at::Tensor, at::Tensor, at::Tensor)’:
/tmp/vllm/csrc/quantization/squeezellm/quant_cuda_kernel.cu:206:136: warning: ‘T* at::Tensor::data() const [with T = c10::Half]’ is deprecated: Tensor.data() is deprecated. Please use Tensor.data_ptr() instead. [-Wdeprecated-declarations]
206 |   vllm::squeezellm::NUQ4MatMulKernel<<>>(
|                                                                                ^
/usr/local/lib/python3.10/dist-packages/torch/include/ATen/core/TensorBody.h:247:1: note: declared here
247 |   T * data() const {
| ^ ~~
/tmp/vllm/csrc/quantization/squeezellm/quant_cuda_kernel.cu:206:193: warning: ‘T* at::Tensor::data() const [with T = c10::Half]’ is deprecated: Tensor.data() is deprecated. Please use Tensor.data_ptr() instead. [-Wdeprecated-declarations]
206 |   vllm::squeezellm::NUQ4MatMulKernel<<>>(
|                                                                                ^
/usr/local/lib/python3.10/dist-packages/torch/include/ATen/core/TensorBody.h:247:1: note: declared here
247 |   T * data() const {
| ^ ~~
/tmp/vllm/csrc/quantization/squeezellm/quant_cuda_kernel.cu:206:237: warning: ‘T* at::Tensor::data() const [with T = c10::Half]’ is deprecated: Tensor.data() is deprecated. Please use Tensor.data_ptr() instead. [-Wdeprecated-declarations]
206 |   vllm::squeezellm::NUQ4MatMulKernel<<>>(
|                                                                                ^
/usr/local/lib/python3.10/dist-packages/torch/include/ATen/core/TensorBody.h:247:1: note: declared here
247 |   T * data() const {
| ^ ~~
[12/14] Building CUDA object CMakeFiles/_C.dir/csrc/quantization/marlin/marlin_cuda_kernel.cu.o
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(455): warning #179-D: right operand of % is zero
if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=1, thread_n_blocks=8, thread_k_blocks=8, stages=4, group_blocks=-1] at line 1033
Remark: The warnings can be suppressed with -diag-suppress
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(487): warning #39-D: division by zero
(pipe / (group_blocks / thread_k_blocks)));
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=1, thread_n_blocks=8, thread_k_blocks=8, stages=4, group_blocks=-1] at line 1033
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(455): warning #179-D: right operand of % is zero
if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=2, thread_n_blocks=8, thread_k_blocks=8, stages=4, group_blocks=-1] at line 1033
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(487): warning #39-D: division by zero
(pipe / (group_blocks / thread_k_blocks)));
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=2, thread_n_blocks=8, thread_k_blocks=8, stages=4, group_blocks=-1] at line 1033
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(455): warning #179-D: right operand of % is zero
if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=3, thread_n_blocks=8, thread_k_blocks=8, stages=4, group_blocks=-1] at line 1033
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(487): warning #39-D: division by zero
(pipe / (group_blocks / thread_k_blocks)));
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=3, thread_n_blocks=8, thread_k_blocks=8, stages=4, group_blocks=-1] at line 1033
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(455): warning #179-D: right operand of % is zero
if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=4, thread_n_blocks=8, thread_k_blocks=8, stages=4, group_blocks=-1] at line 1033
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(487): warning #39-D: division by zero
(pipe / (group_blocks / thread_k_blocks)));
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=4, thread_n_blocks=8, thread_k_blocks=8, stages=4, group_blocks=-1] at line 1033
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(455): warning #179-D: right operand of % is zero
if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=1, thread_n_blocks=16, thread_k_blocks=4, stages=4, group_blocks=-1] at line 1034
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(487): warning #39-D: division by zero
(pipe / (group_blocks / thread_k_blocks)));
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=1, thread_n_blocks=16, thread_k_blocks=4, stages=4, group_blocks=-1] at line 1034
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(455): warning #179-D: right operand of % is zero
if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=2, thread_n_blocks=16, thread_k_blocks=4, stages=4, group_blocks=-1] at line 1034
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(487): warning #39-D: division by zero
(pipe / (group_blocks / thread_k_blocks)));
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=2, thread_n_blocks=16, thread_k_blocks=4, stages=4, group_blocks=-1] at line 1034
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(455): warning #179-D: right operand of % is zero
if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=3, thread_n_blocks=16, thread_k_blocks=4, stages=4, group_blocks=-1] at line 1034
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(487): warning #39-D: division by zero
(pipe / (group_blocks / thread_k_blocks)));
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=3, thread_n_blocks=16, thread_k_blocks=4, stages=4, group_blocks=-1] at line 1034
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(455): warning #179-D: right operand of % is zero
if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=4, thread_n_blocks=16, thread_k_blocks=4, stages=4, group_blocks=-1] at line 1034
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(487): warning #39-D: division by zero
(pipe / (group_blocks / thread_k_blocks)));
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=4, thread_n_blocks=16, thread_k_blocks=4, stages=4, group_blocks=-1] at line 1034
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(455): warning #179-D: right operand of % is zero
if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=1, thread_n_blocks=8, thread_k_blocks=4, stages=4, group_blocks=-1] at line 1035
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(487): warning #39-D: division by zero
(pipe / (group_blocks / thread_k_blocks)));
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=1, thread_n_blocks=8, thread_k_blocks=4, stages=4, group_blocks=-1] at line 1035
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(455): warning #179-D: right operand of % is zero
if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=2, thread_n_blocks=8, thread_k_blocks=4, stages=4, group_blocks=-1] at line 1035
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(487): warning #39-D: division by zero
(pipe / (group_blocks / thread_k_blocks)));
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=2, thread_n_blocks=8, thread_k_blocks=4, stages=4, group_blocks=-1] at line 1035
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(455): warning #179-D: right operand of % is zero
if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=3, thread_n_blocks=8, thread_k_blocks=4, stages=4, group_blocks=-1] at line 1035
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(487): warning #39-D: division by zero
(pipe / (group_blocks / thread_k_blocks)));
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=3, thread_n_blocks=8, thread_k_blocks=4, stages=4, group_blocks=-1] at line 1035
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(455): warning #179-D: right operand of % is zero
if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=4, thread_n_blocks=8, thread_k_blocks=4, stages=4, group_blocks=-1] at line 1035
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(487): warning #39-D: division by zero
(pipe / (group_blocks / thread_k_blocks)));
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=4, thread_n_blocks=8, thread_k_blocks=4, stages=4, group_blocks=-1] at line 1035
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(455): warning #179-D: right operand of % is zero
if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=1, thread_n_blocks=4, thread_k_blocks=8, stages=4, group_blocks=-1] at line 1036
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(487): warning #39-D: division by zero
(pipe / (group_blocks / thread_k_blocks)));
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=1, thread_n_blocks=4, thread_k_blocks=8, stages=4, group_blocks=-1] at line 1036
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(455): warning #179-D: right operand of % is zero
if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=2, thread_n_blocks=4, thread_k_blocks=8, stages=4, group_blocks=-1] at line 1036
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(487): warning #39-D: division by zero
(pipe / (group_blocks / thread_k_blocks)));
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=2, thread_n_blocks=4, thread_k_blocks=8, stages=4, group_blocks=-1] at line 1036
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(455): warning #179-D: right operand of % is zero
if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=3, thread_n_blocks=4, thread_k_blocks=8, stages=4, group_blocks=-1] at line 1036
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(487): warning #39-D: division by zero
(pipe / (group_blocks / thread_k_blocks)));
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=3, thread_n_blocks=4, thread_k_blocks=8, stages=4, group_blocks=-1] at line 1036
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(455): warning #179-D: right operand of % is zero
if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=4, thread_n_blocks=4, thread_k_blocks=8, stages=4, group_blocks=-1] at line 1036
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(487): warning #39-D: division by zero
(pipe / (group_blocks / thread_k_blocks)));
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=4, thread_n_blocks=4, thread_k_blocks=8, stages=4, group_blocks=-1] at line 1036
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(455): warning #179-D: right operand of % is zero
if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=1, thread_n_blocks=8, thread_k_blocks=8, stages=4, group_blocks=-1] at line 1033
Remark: The warnings can be suppressed with -diag-suppress
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(487): warning #39-D: division by zero
(pipe / (group_blocks / thread_k_blocks)));
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=1, thread_n_blocks=8, thread_k_blocks=8, stages=4, group_blocks=-1] at line 1033
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(455): warning #179-D: right operand of % is zero
if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=2, thread_n_blocks=8, thread_k_blocks=8, stages=4, group_blocks=-1] at line 1033
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(487): warning #39-D: division by zero
(pipe / (group_blocks / thread_k_blocks)));
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=2, thread_n_blocks=8, thread_k_blocks=8, stages=4, group_blocks=-1] at line 1033
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(455): warning #179-D: right operand of % is zero
if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=3, thread_n_blocks=8, thread_k_blocks=8, stages=4, group_blocks=-1] at line 1033
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(487): warning #39-D: division by zero
(pipe / (group_blocks / thread_k_blocks)));
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=3, thread_n_blocks=8, thread_k_blocks=8, stages=4, group_blocks=-1] at line 1033
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(455): warning #179-D: right operand of % is zero
if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=4, thread_n_blocks=8, thread_k_blocks=8, stages=4, group_blocks=-1] at line 1033
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(487): warning #39-D: division by zero
(pipe / (group_blocks / thread_k_blocks)));
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=4, thread_n_blocks=8, thread_k_blocks=8, stages=4, group_blocks=-1] at line 1033
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(455): warning #179-D: right operand of % is zero
if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=1, thread_n_blocks=16, thread_k_blocks=4, stages=4, group_blocks=-1] at line 1034
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(487): warning #39-D: division by zero
(pipe / (group_blocks / thread_k_blocks)));
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=1, thread_n_blocks=16, thread_k_blocks=4, stages=4, group_blocks=-1] at line 1034
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(455): warning #179-D: right operand of % is zero
if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=2, thread_n_blocks=16, thread_k_blocks=4, stages=4, group_blocks=-1] at line 1034
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(487): warning #39-D: division by zero
(pipe / (group_blocks / thread_k_blocks)));
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=2, thread_n_blocks=16, thread_k_blocks=4, stages=4, group_blocks=-1] at line 1034
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(455): warning #179-D: right operand of % is zero
if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=3, thread_n_blocks=16, thread_k_blocks=4, stages=4, group_blocks=-1] at line 1034
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(487): warning #39-D: division by zero
(pipe / (group_blocks / thread_k_blocks)));
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=3, thread_n_blocks=16, thread_k_blocks=4, stages=4, group_blocks=-1] at line 1034
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(455): warning #179-D: right operand of % is zero
if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=4, thread_n_blocks=16, thread_k_blocks=4, stages=4, group_blocks=-1] at line 1034
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(487): warning #39-D: division by zero
(pipe / (group_blocks / thread_k_blocks)));
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=4, thread_n_blocks=16, thread_k_blocks=4, stages=4, group_blocks=-1] at line 1034
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(455): warning #179-D: right operand of % is zero
if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=1, thread_n_blocks=8, thread_k_blocks=4, stages=4, group_blocks=-1] at line 1035
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(487): warning #39-D: division by zero
(pipe / (group_blocks / thread_k_blocks)));
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=1, thread_n_blocks=8, thread_k_blocks=4, stages=4, group_blocks=-1] at line 1035
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(455): warning #179-D: right operand of % is zero
if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=2, thread_n_blocks=8, thread_k_blocks=4, stages=4, group_blocks=-1] at line 1035
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(487): warning #39-D: division by zero
(pipe / (group_blocks / thread_k_blocks)));
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=2, thread_n_blocks=8, thread_k_blocks=4, stages=4, group_blocks=-1] at line 1035
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(455): warning #179-D: right operand of % is zero
if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=3, thread_n_blocks=8, thread_k_blocks=4, stages=4, group_blocks=-1] at line 1035
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(487): warning #39-D: division by zero
(pipe / (group_blocks / thread_k_blocks)));
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=3, thread_n_blocks=8, thread_k_blocks=4, stages=4, group_blocks=-1] at line 1035
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(455): warning #179-D: right operand of % is zero
if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=4, thread_n_blocks=8, thread_k_blocks=4, stages=4, group_blocks=-1] at line 1035
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(487): warning #39-D: division by zero
(pipe / (group_blocks / thread_k_blocks)));
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=4, thread_n_blocks=8, thread_k_blocks=4, stages=4, group_blocks=-1] at line 1035
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(455): warning #179-D: right operand of % is zero
if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=1, thread_n_blocks=4, thread_k_blocks=8, stages=4, group_blocks=-1] at line 1036
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(487): warning #39-D: division by zero
(pipe / (group_blocks / thread_k_blocks)));
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=1, thread_n_blocks=4, thread_k_blocks=8, stages=4, group_blocks=-1] at line 1036
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(455): warning #179-D: right operand of % is zero
if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=2, thread_n_blocks=4, thread_k_blocks=8, stages=4, group_blocks=-1] at line 1036
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(487): warning #39-D: division by zero
(pipe / (group_blocks / thread_k_blocks)));
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=2, thread_n_blocks=4, thread_k_blocks=8, stages=4, group_blocks=-1] at line 1036
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(455): warning #179-D: right operand of % is zero
if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=3, thread_n_blocks=4, thread_k_blocks=8, stages=4, group_blocks=-1] at line 1036
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(487): warning #39-D: division by zero
(pipe / (group_blocks / thread_k_blocks)));
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=3, thread_n_blocks=4, thread_k_blocks=8, stages=4, group_blocks=-1] at line 1036
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(455): warning #179-D: right operand of % is zero
if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=4, thread_n_blocks=4, thread_k_blocks=8, stages=4, group_blocks=-1] at line 1036
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(487): warning #39-D: division by zero
(pipe / (group_blocks / thread_k_blocks)));
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=4, thread_n_blocks=4, thread_k_blocks=8, stages=4, group_blocks=-1] at line 1036
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(455): warning #179-D: right operand of % is zero
if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=1, thread_n_blocks=8, thread_k_blocks=8, stages=4, group_blocks=-1] at line 1033
Remark: The warnings can be suppressed with -diag-suppress
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(487): warning #39-D: division by zero
(pipe / (group_blocks / thread_k_blocks)));
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=1, thread_n_blocks=8, thread_k_blocks=8, stages=4, group_blocks=-1] at line 1033
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(455): warning #179-D: right operand of % is zero
if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=2, thread_n_blocks=8, thread_k_blocks=8, stages=4, group_blocks=-1] at line 1033
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(487): warning #39-D: division by zero
(pipe / (group_blocks / thread_k_blocks)));
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=2, thread_n_blocks=8, thread_k_blocks=8, stages=4, group_blocks=-1] at line 1033
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(455): warning #179-D: right operand of % is zero
if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=3, thread_n_blocks=8, thread_k_blocks=8, stages=4, group_blocks=-1] at line 1033
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(487): warning #39-D: division by zero
(pipe / (group_blocks / thread_k_blocks)));
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=3, thread_n_blocks=8, thread_k_blocks=8, stages=4, group_blocks=-1] at line 1033
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(455): warning #179-D: right operand of % is zero
if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=4, thread_n_blocks=8, thread_k_blocks=8, stages=4, group_blocks=-1] at line 1033
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(487): warning #39-D: division by zero
(pipe / (group_blocks / thread_k_blocks)));
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=4, thread_n_blocks=8, thread_k_blocks=8, stages=4, group_blocks=-1] at line 1033
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(455): warning #179-D: right operand of % is zero
if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=1, thread_n_blocks=16, thread_k_blocks=4, stages=4, group_blocks=-1] at line 1034
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(487): warning #39-D: division by zero
(pipe / (group_blocks / thread_k_blocks)));
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=1, thread_n_blocks=16, thread_k_blocks=4, stages=4, group_blocks=-1] at line 1034
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(455): warning #179-D: right operand of % is zero
if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=2, thread_n_blocks=16, thread_k_blocks=4, stages=4, group_blocks=-1] at line 1034
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(487): warning #39-D: division by zero
(pipe / (group_blocks / thread_k_blocks)));
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=2, thread_n_blocks=16, thread_k_blocks=4, stages=4, group_blocks=-1] at line 1034
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(455): warning #179-D: right operand of % is zero
if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=3, thread_n_blocks=16, thread_k_blocks=4, stages=4, group_blocks=-1] at line 1034
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(487): warning #39-D: division by zero
(pipe / (group_blocks / thread_k_blocks)));
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=3, thread_n_blocks=16, thread_k_blocks=4, stages=4, group_blocks=-1] at line 1034
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(455): warning #179-D: right operand of % is zero
if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=4, thread_n_blocks=16, thread_k_blocks=4, stages=4, group_blocks=-1] at line 1034
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(487): warning #39-D: division by zero
(pipe / (group_blocks / thread_k_blocks)));
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=4, thread_n_blocks=16, thread_k_blocks=4, stages=4, group_blocks=-1] at line 1034
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(455): warning #179-D: right operand of % is zero
if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=1, thread_n_blocks=8, thread_k_blocks=4, stages=4, group_blocks=-1] at line 1035
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(487): warning #39-D: division by zero
(pipe / (group_blocks / thread_k_blocks)));
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=1, thread_n_blocks=8, thread_k_blocks=4, stages=4, group_blocks=-1] at line 1035
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(455): warning #179-D: right operand of % is zero
if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=2, thread_n_blocks=8, thread_k_blocks=4, stages=4, group_blocks=-1] at line 1035
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(487): warning #39-D: division by zero
(pipe / (group_blocks / thread_k_blocks)));
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=2, thread_n_blocks=8, thread_k_blocks=4, stages=4, group_blocks=-1] at line 1035
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(455): warning #179-D: right operand of % is zero
if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=3, thread_n_blocks=8, thread_k_blocks=4, stages=4, group_blocks=-1] at line 1035
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(487): warning #39-D: division by zero
(pipe / (group_blocks / thread_k_blocks)));
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=3, thread_n_blocks=8, thread_k_blocks=4, stages=4, group_blocks=-1] at line 1035
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(455): warning #179-D: right operand of % is zero
if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=4, thread_n_blocks=8, thread_k_blocks=4, stages=4, group_blocks=-1] at line 1035
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(487): warning #39-D: division by zero
(pipe / (group_blocks / thread_k_blocks)));
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=4, thread_n_blocks=8, thread_k_blocks=4, stages=4, group_blocks=-1] at line 1035
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(455): warning #179-D: right operand of % is zero
if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=1, thread_n_blocks=4, thread_k_blocks=8, stages=4, group_blocks=-1] at line 1036
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(487): warning #39-D: division by zero
(pipe / (group_blocks / thread_k_blocks)));
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=1, thread_n_blocks=4, thread_k_blocks=8, stages=4, group_blocks=-1] at line 1036
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(455): warning #179-D: right operand of % is zero
if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=2, thread_n_blocks=4, thread_k_blocks=8, stages=4, group_blocks=-1] at line 1036
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(487): warning #39-D: division by zero
(pipe / (group_blocks / thread_k_blocks)));
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=2, thread_n_blocks=4, thread_k_blocks=8, stages=4, group_blocks=-1] at line 1036
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(455): warning #179-D: right operand of % is zero
if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=3, thread_n_blocks=4, thread_k_blocks=8, stages=4, group_blocks=-1] at line 1036
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(487): warning #39-D: division by zero
(pipe / (group_blocks / thread_k_blocks)));
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=3, thread_n_blocks=4, thread_k_blocks=8, stages=4, group_blocks=-1] at line 1036
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(455): warning #179-D: right operand of % is zero
if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=4, thread_n_blocks=4, thread_k_blocks=8, stages=4, group_blocks=-1] at line 1036
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(487): warning #39-D: division by zero
(pipe / (group_blocks / thread_k_blocks)));
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=4, thread_n_blocks=4, thread_k_blocks=8, stages=4, group_blocks=-1] at line 1036
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(455): warning #179-D: right operand of % is zero
if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=1, thread_n_blocks=8, thread_k_blocks=8, stages=4, group_blocks=-1] at line 1033
Remark: The warnings can be suppressed with -diag-suppress
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(487): warning #39-D: division by zero
(pipe / (group_blocks / thread_k_blocks)));
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=1, thread_n_blocks=8, thread_k_blocks=8, stages=4, group_blocks=-1] at line 1033
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(455): warning #179-D: right operand of % is zero
if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=2, thread_n_blocks=8, thread_k_blocks=8, stages=4, group_blocks=-1] at line 1033
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(487): warning #39-D: division by zero
(pipe / (group_blocks / thread_k_blocks)));
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=2, thread_n_blocks=8, thread_k_blocks=8, stages=4, group_blocks=-1] at line 1033
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(455): warning #179-D: right operand of % is zero
if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=3, thread_n_blocks=8, thread_k_blocks=8, stages=4, group_blocks=-1] at line 1033
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(487): warning #39-D: division by zero
(pipe / (group_blocks / thread_k_blocks)));
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=3, thread_n_blocks=8, thread_k_blocks=8, stages=4, group_blocks=-1] at line 1033
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(455): warning #179-D: right operand of % is zero
if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=4, thread_n_blocks=8, thread_k_blocks=8, stages=4, group_blocks=-1] at line 1033
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(487): warning #39-D: division by zero
(pipe / (group_blocks / thread_k_blocks)));
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=4, thread_n_blocks=8, thread_k_blocks=8, stages=4, group_blocks=-1] at line 1033
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(455): warning #179-D: right operand of % is zero
if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=1, thread_n_blocks=16, thread_k_blocks=4, stages=4, group_blocks=-1] at line 1034
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(487): warning #39-D: division by zero
(pipe / (group_blocks / thread_k_blocks)));
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=1, thread_n_blocks=16, thread_k_blocks=4, stages=4, group_blocks=-1] at line 1034
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(455): warning #179-D: right operand of % is zero
if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=2, thread_n_blocks=16, thread_k_blocks=4, stages=4, group_blocks=-1] at line 1034
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(487): warning #39-D: division by zero
(pipe / (group_blocks / thread_k_blocks)));
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=2, thread_n_blocks=16, thread_k_blocks=4, stages=4, group_blocks=-1] at line 1034
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(455): warning #179-D: right operand of % is zero
if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=3, thread_n_blocks=16, thread_k_blocks=4, stages=4, group_blocks=-1] at line 1034
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(487): warning #39-D: division by zero
(pipe / (group_blocks / thread_k_blocks)));
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=3, thread_n_blocks=16, thread_k_blocks=4, stages=4, group_blocks=-1] at line 1034
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(455): warning #179-D: right operand of % is zero
if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=4, thread_n_blocks=16, thread_k_blocks=4, stages=4, group_blocks=-1] at line 1034
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(487): warning #39-D: division by zero
(pipe / (group_blocks / thread_k_blocks)));
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=256, thread_m_blocks=4, thread_n_blocks=16, thread_k_blocks=4, stages=4, group_blocks=-1] at line 1034
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(455): warning #179-D: right operand of % is zero
if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=1, thread_n_blocks=8, thread_k_blocks=4, stages=4, group_blocks=-1] at line 1035
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(487): warning #39-D: division by zero
(pipe / (group_blocks / thread_k_blocks)));
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=1, thread_n_blocks=8, thread_k_blocks=4, stages=4, group_blocks=-1] at line 1035
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(455): warning #179-D: right operand of % is zero
if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=2, thread_n_blocks=8, thread_k_blocks=4, stages=4, group_blocks=-1] at line 1035
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(487): warning #39-D: division by zero
(pipe / (group_blocks / thread_k_blocks)));
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=2, thread_n_blocks=8, thread_k_blocks=4, stages=4, group_blocks=-1] at line 1035
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(455): warning #179-D: right operand of % is zero
if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=3, thread_n_blocks=8, thread_k_blocks=4, stages=4, group_blocks=-1] at line 1035
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(487): warning #39-D: division by zero
(pipe / (group_blocks / thread_k_blocks)));
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=3, thread_n_blocks=8, thread_k_blocks=4, stages=4, group_blocks=-1] at line 1035
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(455): warning #179-D: right operand of % is zero
if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=4, thread_n_blocks=8, thread_k_blocks=4, stages=4, group_blocks=-1] at line 1035
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(487): warning #39-D: division by zero
(pipe / (group_blocks / thread_k_blocks)));
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=4, thread_n_blocks=8, thread_k_blocks=4, stages=4, group_blocks=-1] at line 1035
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(455): warning #179-D: right operand of % is zero
if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=1, thread_n_blocks=4, thread_k_blocks=8, stages=4, group_blocks=-1] at line 1036
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(487): warning #39-D: division by zero
(pipe / (group_blocks / thread_k_blocks)));
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=1, thread_n_blocks=4, thread_k_blocks=8, stages=4, group_blocks=-1] at line 1036
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(455): warning #179-D: right operand of % is zero
if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=2, thread_n_blocks=4, thread_k_blocks=8, stages=4, group_blocks=-1] at line 1036
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(487): warning #39-D: division by zero
(pipe / (group_blocks / thread_k_blocks)));
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=2, thread_n_blocks=4, thread_k_blocks=8, stages=4, group_blocks=-1] at line 1036
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(455): warning #179-D: right operand of % is zero
if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=3, thread_n_blocks=4, thread_k_blocks=8, stages=4, group_blocks=-1] at line 1036
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(487): warning #39-D: division by zero
(pipe / (group_blocks / thread_k_blocks)));
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=3, thread_n_blocks=4, thread_k_blocks=8, stages=4, group_blocks=-1] at line 1036
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(455): warning #179-D: right operand of % is zero
if (group_blocks != -1 && pipe % (group_blocks / thread_k_blocks) == 0) {
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=4, thread_n_blocks=4, thread_k_blocks=8, stages=4, group_blocks=-1] at line 1036
/tmp/vllm/csrc/quantization/marlin/marlin_cuda_kernel.cu(487): warning #39-D: division by zero
(pipe / (group_blocks / thread_k_blocks)));
^
detected during instantiation of void marlin::Marlin(const int4 *, const int4 *, int4 *, const int4 *, int, int, int, int *) [with threads=128, thread_m_blocks=4, thread_n_blocks=4, thread_k_blocks=8, stages=4, group_blocks=-1] at line 1036
[13/14] Building CUDA object CMakeFiles/_C.dir/csrc/attention/attention_kernels.cu.o
/tmp/vllm/csrc/attention/attention_kernels.cu(625): warning #177-D: variable thread_group_size was declared but never referenced
int thread_group_size = ((32 / BLOCK_SIZE) > (1) ? (32 / BLOCK_SIZE) : (1));
^
Remark: The warnings can be suppressed with -diag-suppress
/tmp/vllm/csrc/attention/attention_kernels.cu(806): warning #177-D: variable thread_group_size was declared but never referenced
int thread_group_size = ((32 / BLOCK_SIZE) > (1) ? (32 / BLOCK_SIZE) : (1));
^
/tmp/vllm/csrc/attention/attention_kernels.cu(625): warning #177-D: variable thread_group_size was declared but never referenced
int thread_group_size = ((32 / BLOCK_SIZE) > (1) ? (32 / BLOCK_SIZE) : (1));
^
Remark: The warnings can be suppressed with -diag-suppress
/tmp/vllm/csrc/attention/attention_kernels.cu(806): warning #177-D: variable thread_group_size was declared but never referenced
int thread_group_size = ((32 / BLOCK_SIZE) > (1) ? (32 / BLOCK_SIZE) : (1));
^
/tmp/vllm/csrc/attention/attention_kernels.cu(625): warning #177-D: variable thread_group_size was declared but never referenced
int thread_group_size = ((32 / BLOCK_SIZE) > (1) ? (32 / BLOCK_SIZE) : (1));
^
Remark: The warnings can be suppressed with -diag-suppress
/tmp/vllm/csrc/attention/attention_kernels.cu(806): warning #177-D: variable thread_group_size was declared but never referenced
int thread_group_size = ((32 / BLOCK_SIZE) > (1) ? (32 / BLOCK_SIZE) : (1));
^
/tmp/vllm/csrc/attention/attention_kernels.cu(625): warning #177-D: variable thread_group_size was declared but never referenced
int thread_group_size = ((32 / BLOCK_SIZE) > (1) ? (32 / BLOCK_SIZE) : (1));
^
Remark: The warnings can be suppressed with -diag-suppress
/tmp/vllm/csrc/attention/attention_kernels.cu(806): warning #177-D: variable thread_group_size was declared but never referenced
int thread_group_size = ((32 / BLOCK_SIZE) > (1) ? (32 / BLOCK_SIZE) : (1));
^
[14/14] Linking CXX shared module /tmp/vllm/build/lib.linux-x86_64-cpython-310/vllm/_C.cpython-310-x86_64-linux-gnu.so
creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/vllm
copying build/lib.linux-x86_64-cpython-310/vllm/test_utils.py -> build/bdist.linux-x86_64/egg/vllm
copying build/lib.linux-x86_64-cpython-310/vllm/utils.py -> build/bdist.linux-x86_64/egg/vllm
creating build/bdist.linux-x86_64/egg/vllm/attention
copying build/lib.linux-x86_64-cpython-310/vllm/attention/layer.py -> build/bdist.linux-x86_64/egg/vllm/attention
copying build/lib.linux-x86_64-cpython-310/vllm/attention/__init__.py -> build/bdist.linux-x86_64/egg/vllm/attention
creating build/bdist.linux-x86_64/egg/vllm/attention/ops
copying build/lib.linux-x86_64-cpython-310/vllm/attention/ops/paged_attn.py -> build/bdist.linux-x86_64/egg/vllm/attention/ops
copying build/lib.linux-x86_64-cpython-310/vllm/attention/ops/prefix_prefill.py -> build/bdist.linux-x86_64/egg/vllm/attention/ops
copying build/lib.linux-x86_64-cpython-310/vllm/attention/ops/__init__.py -> build/bdist.linux-x86_64/egg/vllm/attention/ops
copying build/lib.linux-x86_64-cpython-310/vllm/attention/selector.py -> build/bdist.linux-x86_64/egg/vllm/attention
creating build/bdist.linux-x86_64/egg/vllm/attention/backends
copying build/lib.linux-x86_64-cpython-310/vllm/attention/backends/flash_attn.py -> build/bdist.linux-x86_64/egg/vllm/attention/backends
copying build/lib.linux-x86_64-cpython-310/vllm/attention/backends/xformers.py -> build/bdist.linux-x86_64/egg/vllm/attention/backends
copying build/lib.linux-x86_64-cpython-310/vllm/attention/backends/__init__.py -> build/bdist.linux-x86_64/egg/vllm/attention/backends
copying build/lib.linux-x86_64-cpython-310/vllm/attention/backends/abstract.py -> build/bdist.linux-x86_64/egg/vllm/attention/backends
creating build/bdist.linux-x86_64/egg/vllm/worker
copying build/lib.linux-x86_64-cpython-310/vllm/worker/neuron_model_runner.py -> build/bdist.linux-x86_64/egg/vllm/worker
copying build/lib.linux-x86_64-cpython-310/vllm/worker/worker.py -> build/bdist.linux-x86_64/egg/vllm/worker
copying build/lib.linux-x86_64-cpython-310/vllm/worker/cache_engine.py -> build/bdist.linux-x86_64/egg/vllm/worker
copying build/lib.linux-x86_64-cpython-310/vllm/worker/model_runner.py -> build/bdist.linux-x86_64/egg/vllm/worker
copying build/lib.linux-x86_64-cpython-310/vllm/worker/__init__.py -> build/bdist.linux-x86_64/egg/vllm/worker
copying build/lib.linux-x86_64-cpython-310/vllm/worker/neuron_worker.py -> build/bdist.linux-x86_64/egg/vllm/worker
copying build/lib.linux-x86_64-cpython-310/vllm/py.typed -> build/bdist.linux-x86_64/egg/vllm
creating build/bdist.linux-x86_64/egg/vllm/transformers_utils
copying build/lib.linux-x86_64-cpython-310/vllm/transformers_utils/tokenizer.py -> build/bdist.linux-x86_64/egg/vllm/transformers_utils
creating build/bdist.linux-x86_64/egg/vllm/transformers_utils/configs
copying build/lib.linux-x86_64-cpython-310/vllm/transformers_utils/configs/__init__.py -> build/bdist.linux-x86_64/egg/vllm/transformers_utils/configs
copying build/lib.linux-x86_64-cpython-310/vllm/transformers_utils/configs/chatglm.py -> build/bdist.linux-x86_64/egg/vllm/transformers_utils/configs
copying build/lib.linux-x86_64-cpython-310/vllm/transformers_utils/configs/jais.py -> build/bdist.linux-x86_64/egg/vllm/transformers_utils/configs
copying build/lib.linux-x86_64-cpython-310/vllm/transformers_utils/configs/mpt.py -> build/bdist.linux-x86_64/egg/vllm/transformers_utils/configs
copying build/lib.linux-x86_64-cpython-310/vllm/transformers_utils/configs/dbrx.py -> build/bdist.linux-x86_64/egg/vllm/transformers_utils/configs
copying build/lib.linux-x86_64-cpython-310/vllm/transformers_utils/configs/falcon.py -> build/bdist.linux-x86_64/egg/vllm/transformers_utils/configs
copying build/lib.linux-x86_64-cpython-310/vllm/transformers_utils/detokenizer.py -> build/bdist.linux-x86_64/egg/vllm/transformers_utils
copying build/lib.linux-x86_64-cpython-310/vllm/transformers_utils/__init__.py -> build/bdist.linux-x86_64/egg/vllm/transformers_utils
copying build/lib.linux-x86_64-cpython-310/vllm/transformers_utils/config.py -> build/bdist.linux-x86_64/egg/vllm/transformers_utils
creating build/bdist.linux-x86_64/egg/vllm/transformers_utils/tokenizers
copying build/lib.linux-x86_64-cpython-310/vllm/transformers_utils/tokenizers/__init__.py -> build/bdist.linux-x86_64/egg/vllm/transformers_utils/tokenizers
copying build/lib.linux-x86_64-cpython-310/vllm/transformers_utils/tokenizers/baichuan.py -> build/bdist.linux-x86_64/egg/vllm/transformers_utils/tokenizers
creating build/bdist.linux-x86_64/egg/vllm/transformers_utils/tokenizer_group
copying build/lib.linux-x86_64-cpython-310/vllm/transformers_utils/tokenizer_group/ray_tokenizer_group.py -> build/bdist.linux-x86_64/egg/vllm/transformers_utils/tokenizer_group
copying build/lib.linux-x86_64-cpython-310/vllm/transformers_utils/tokenizer_group/tokenizer_group.py -> build/bdist.linux-x86_64/egg/vllm/transformers_utils/tokenizer_group
copying build/lib.linux-x86_64-cpython-310/vllm/transformers_utils/tokenizer_group/__init__.py -> build/bdist.linux-x86_64/egg/vllm/transformers_utils/tokenizer_group
copying build/lib.linux-x86_64-cpython-310/vllm/transformers_utils/tokenizer_group/base_tokenizer_group.py -> build/bdist.linux-x86_64/egg/vllm/transformers_utils/tokenizer_group
creating build/bdist.linux-x86_64/egg/vllm/lora
copying build/lib.linux-x86_64-cpython-310/vllm/lora/models.py -> build/bdist.linux-x86_64/egg/vllm/lora
copying build/lib.linux-x86_64-cpython-310/vllm/lora/utils.py -> build/bdist.linux-x86_64/egg/vllm/lora
copying build/lib.linux-x86_64-cpython-310/vllm/lora/worker_manager.py -> build/bdist.linux-x86_64/egg/vllm/lora
copying build/lib.linux-x86_64-cpython-310/vllm/lora/request.py -> build/bdist.linux-x86_64/egg/vllm/lora
copying build/lib.linux-x86_64-cpython-310/vllm/lora/__init__.py -> build/bdist.linux-x86_64/egg/vllm/lora
copying build/lib.linux-x86_64-cpython-310/vllm/lora/punica.py -> build/bdist.linux-x86_64/egg/vllm/lora
copying build/lib.linux-x86_64-cpython-310/vllm/lora/lora.py -> build/bdist.linux-x86_64/egg/vllm/lora
copying build/lib.linux-x86_64-cpython-310/vllm/lora/layers.py -> build/bdist.linux-x86_64/egg/vllm/lora
creating build/bdist.linux-x86_64/egg/vllm/core
copying build/lib.linux-x86_64-cpython-310/vllm/core/block_manager_v1.py -> build/bdist.linux-x86_64/egg/vllm/core
copying build/lib.linux-x86_64-cpython-310/vllm/core/evictor.py -> build/bdist.linux-x86_64/egg/vllm/core
copying build/lib.linux-x86_64-cpython-310/vllm/core/block_manager_v2.py -> build/bdist.linux-x86_64/egg/vllm/core
copying build/lib.linux-x86_64-cpython-310/vllm/core/scheduler.py -> build/bdist.linux-x86_64/egg/vllm/core
copying build/lib.linux-x86_64-cpython-310/vllm/core/__init__.py -> build/bdist.linux-x86_64/egg/vllm/core
copying build/lib.linux-x86_64-cpython-310/vllm/core/policy.py -> build/bdist.linux-x86_64/egg/vllm/core
copying build/lib.linux-x86_64-cpython-310/vllm/core/interfaces.py -> build/bdist.linux-x86_64/egg/vllm/core
creating build/bdist.linux-x86_64/egg/vllm/model_executor
creating build/bdist.linux-x86_64/egg/vllm/model_executor/layers
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/layers/sampler.py -> build/bdist.linux-x86_64/egg/vllm/model_executor/layers
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/layers/layernorm.py -> build/bdist.linux-x86_64/egg/vllm/model_executor/layers
creating build/bdist.linux-x86_64/egg/vllm/model_executor/layers/fused_moe
creating build/bdist.linux-x86_64/egg/vllm/model_executor/layers/fused_moe/configs
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/layers/fused_moe/configs/E=16,N=1344,device_name=NVIDIA_A100-SXM4-80GB.json -> build/bdist.linux-x86_64/egg/vllm/model_executor/layers/fused_moe/configs
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/layers/fused_moe/configs/E=16,N=2688,device_name=NVIDIA_A100-SXM4-80GB.json -> build/bdist.linux-x86_64/egg/vllm/model_executor/layers/fused_moe/configs
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/layers/fused_moe/configs/E=8,N=3584,device_name=NVIDIA_A100-SXM4-80GB.json -> build/bdist.linux-x86_64/egg/vllm/model_executor/layers/fused_moe/configs
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/layers/fused_moe/configs/E=8,N=1792,device_name=NVIDIA_H100_80GB_HBM3.json -> build/bdist.linux-x86_64/egg/vllm/model_executor/layers/fused_moe/configs
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/layers/fused_moe/configs/E=8,N=7168,device_name=NVIDIA_A100-SXM4-80GB.json -> build/bdist.linux-x86_64/egg/vllm/model_executor/layers/fused_moe/configs
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/layers/fused_moe/configs/E=8,N=1792,device_name=NVIDIA_A100-SXM4-80GB.json -> build/bdist.linux-x86_64/egg/vllm/model_executor/layers/fused_moe/configs
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/layers/fused_moe/configs/E=8,N=3584,device_name=NVIDIA_H100_80GB_HBM3.json -> build/bdist.linux-x86_64/egg/vllm/model_executor/layers/fused_moe/configs
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/layers/fused_moe/configs/E=16,N=1344,device_name=NVIDIA_A100-SXM4-40GB.json -> build/bdist.linux-x86_64/egg/vllm/model_executor/layers/fused_moe/configs
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/layers/fused_moe/configs/E=8,N=7168,device_name=NVIDIA_H100_80GB_HBM3.json -> build/bdist.linux-x86_64/egg/vllm/model_executor/layers/fused_moe/configs
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/layers/fused_moe/configs/E=8,N=1792,device_name=NVIDIA_A100-SXM4-40GB.json -> build/bdist.linux-x86_64/egg/vllm/model_executor/layers/fused_moe/configs
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/layers/fused_moe/configs/E=16,N=1344,device_name=NVIDIA_H100_80GB_HBM3.json -> build/bdist.linux-x86_64/egg/vllm/model_executor/layers/fused_moe/configs
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/layers/fused_moe/configs/E=16,N=2688,device_name=NVIDIA_H100_80GB_HBM3.json -> build/bdist.linux-x86_64/egg/vllm/model_executor/layers/fused_moe/configs
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/layers/fused_moe/configs/E=8,N=3584,device_name=NVIDIA_A100-SXM4-40GB.json -> build/bdist.linux-x86_64/egg/vllm/model_executor/layers/fused_moe/configs
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/layers/fused_moe/__init__.py -> build/bdist.linux-x86_64/egg/vllm/model_executor/layers/fused_moe
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/layers/fused_moe/fused_moe.py -> build/bdist.linux-x86_64/egg/vllm/model_executor/layers/fused_moe
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/layers/vocab_parallel_embedding.py -> build/bdist.linux-x86_64/egg/vllm/model_executor/layers
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/layers/rotary_embedding.py -> build/bdist.linux-x86_64/egg/vllm/model_executor/layers
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/layers/linear.py -> build/bdist.linux-x86_64/egg/vllm/model_executor/layers
creating build/bdist.linux-x86_64/egg/vllm/model_executor/layers/quantization
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/layers/quantization/squeezellm.py -> build/bdist.linux-x86_64/egg/vllm/model_executor/layers/quantization
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/layers/quantization/gptq.py -> build/bdist.linux-x86_64/egg/vllm/model_executor/layers/quantization
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/layers/quantization/__init__.py -> build/bdist.linux-x86_64/egg/vllm/model_executor/layers/quantization
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/layers/quantization/marlin.py -> build/bdist.linux-x86_64/egg/vllm/model_executor/layers/quantization
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/layers/quantization/base_config.py -> build/bdist.linux-x86_64/egg/vllm/model_executor/layers/quantization
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/layers/quantization/awq.py -> build/bdist.linux-x86_64/egg/vllm/model_executor/layers/quantization
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/layers/__init__.py -> build/bdist.linux-x86_64/egg/vllm/model_executor/layers
creating build/bdist.linux-x86_64/egg/vllm/model_executor/layers/ops
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/layers/ops/sample.py -> build/bdist.linux-x86_64/egg/vllm/model_executor/layers/ops
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/layers/ops/__init__.py -> build/bdist.linux-x86_64/egg/vllm/model_executor/layers/ops
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/layers/ops/rand.py -> build/bdist.linux-x86_64/egg/vllm/model_executor/layers/ops
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/layers/rejection_sampler.py -> build/bdist.linux-x86_64/egg/vllm/model_executor/layers
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/layers/logits_processor.py -> build/bdist.linux-x86_64/egg/vllm/model_executor/layers
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/layers/activation.py -> build/bdist.linux-x86_64/egg/vllm/model_executor/layers
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/utils.py -> build/bdist.linux-x86_64/egg/vllm/model_executor
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/sampling_metadata.py -> build/bdist.linux-x86_64/egg/vllm/model_executor
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/model_loader.py -> build/bdist.linux-x86_64/egg/vllm/model_executor
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/guided_decoding.py -> build/bdist.linux-x86_64/egg/vllm/model_executor
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/__init__.py -> build/bdist.linux-x86_64/egg/vllm/model_executor
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/guided_logits_processors.py -> build/bdist.linux-x86_64/egg/vllm/model_executor
creating build/bdist.linux-x86_64/egg/vllm/model_executor/parallel_utils
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/parallel_utils/parallel_state.py -> build/bdist.linux-x86_64/egg/vllm/model_executor/parallel_utils
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/parallel_utils/utils.py -> build/bdist.linux-x86_64/egg/vllm/model_executor/parallel_utils
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/parallel_utils/custom_all_reduce.py -> build/bdist.linux-x86_64/egg/vllm/model_executor/parallel_utils
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/parallel_utils/communication_op.py -> build/bdist.linux-x86_64/egg/vllm/model_executor/parallel_utils
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/parallel_utils/__init__.py -> build/bdist.linux-x86_64/egg/vllm/model_executor/parallel_utils
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/parallel_utils/pynccl.py -> build/bdist.linux-x86_64/egg/vllm/model_executor/parallel_utils
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/parallel_utils/pynccl_utils.py -> build/bdist.linux-x86_64/egg/vllm/model_executor/parallel_utils
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/weight_utils.py -> build/bdist.linux-x86_64/egg/vllm/model_executor
creating build/bdist.linux-x86_64/egg/vllm/model_executor/models
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/models/bloom.py -> build/bdist.linux-x86_64/egg/vllm/model_executor/models
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/models/orion.py -> build/bdist.linux-x86_64/egg/vllm/model_executor/models
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/models/qwen2_moe.py -> build/bdist.linux-x86_64/egg/vllm/model_executor/models
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/models/decilm.py -> build/bdist.linux-x86_64/egg/vllm/model_executor/models
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/models/gpt2.py -> build/bdist.linux-x86_64/egg/vllm/model_executor/models
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/models/gpt_bigcode.py -> build/bdist.linux-x86_64/egg/vllm/model_executor/models
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/models/qwen2.py -> build/bdist.linux-x86_64/egg/vllm/model_executor/models
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/models/starcoder2.py -> build/bdist.linux-x86_64/egg/vllm/model_executor/models
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/models/opt.py -> build/bdist.linux-x86_64/egg/vllm/model_executor/models
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/models/mixtral_quant.py -> build/bdist.linux-x86_64/egg/vllm/model_executor/models
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/models/olmo.py -> build/bdist.linux-x86_64/egg/vllm/model_executor/models
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/models/__init__.py -> build/bdist.linux-x86_64/egg/vllm/model_executor/models
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/models/gemma.py -> build/bdist.linux-x86_64/egg/vllm/model_executor/models
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/models/phi.py -> build/bdist.linux-x86_64/egg/vllm/model_executor/models
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/models/llama.py -> build/bdist.linux-x86_64/egg/vllm/model_executor/models
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/models/gpt_j.py -> build/bdist.linux-x86_64/egg/vllm/model_executor/models
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/models/chatglm.py -> build/bdist.linux-x86_64/egg/vllm/model_executor/models
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/models/commandr.py -> build/bdist.linux-x86_64/egg/vllm/model_executor/models
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/models/xverse.py -> build/bdist.linux-x86_64/egg/vllm/model_executor/models
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/models/stablelm.py -> build/bdist.linux-x86_64/egg/vllm/model_executor/models
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/models/jais.py -> build/bdist.linux-x86_64/egg/vllm/model_executor/models
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/models/internlm2.py -> build/bdist.linux-x86_64/egg/vllm/model_executor/models
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/models/mixtral.py -> build/bdist.linux-x86_64/egg/vllm/model_executor/models
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/models/qwen.py -> build/bdist.linux-x86_64/egg/vllm/model_executor/models
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/models/deepseek.py -> build/bdist.linux-x86_64/egg/vllm/model_executor/models
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/models/mpt.py -> build/bdist.linux-x86_64/egg/vllm/model_executor/models
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/models/gpt_neox.py -> build/bdist.linux-x86_64/egg/vllm/model_executor/models
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/models/llava.py -> build/bdist.linux-x86_64/egg/vllm/model_executor/models
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/models/dbrx.py -> build/bdist.linux-x86_64/egg/vllm/model_executor/models
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/models/baichuan.py -> build/bdist.linux-x86_64/egg/vllm/model_executor/models
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/models/falcon.py -> build/bdist.linux-x86_64/egg/vllm/model_executor/models
copying build/lib.linux-x86_64-cpython-310/vllm/model_executor/neuron_model_loader.py -> build/bdist.linux-x86_64/egg/vllm/model_executor
creating build/bdist.linux-x86_64/egg/vllm/executor
copying build/lib.linux-x86_64-cpython-310/vllm/executor/utils.py -> build/bdist.linux-x86_64/egg/vllm/executor
copying build/lib.linux-x86_64-cpython-310/vllm/executor/ray_gpu_executor.py -> build/bdist.linux-x86_64/egg/vllm/executor
copying build/lib.linux-x86_64-cpython-310/vllm/executor/executor_base.py -> build/bdist.linux-x86_64/egg/vllm/executor
copying build/lib.linux-x86_64-cpython-310/vllm/executor/gpu_executor.py -> build/bdist.linux-x86_64/egg/vllm/executor
copying build/lib.linux-x86_64-cpython-310/vllm/executor/neuron_executor.py -> build/bdist.linux-x86_64/egg/vllm/executor
copying build/lib.linux-x86_64-cpython-310/vllm/executor/__init__.py -> build/bdist.linux-x86_64/egg/vllm/executor
copying build/lib.linux-x86_64-cpython-310/vllm/sampling_params.py -> build/bdist.linux-x86_64/egg/vllm
copying build/lib.linux-x86_64-cpython-310/vllm/__init__.py -> build/bdist.linux-x86_64/egg/vllm
copying build/lib.linux-x86_64-cpython-310/vllm/config.py -> build/bdist.linux-x86_64/egg/vllm
creating build/bdist.linux-x86_64/egg/vllm/entrypoints
copying build/lib.linux-x86_64-cpython-310/vllm/entrypoints/llm.py -> build/bdist.linux-x86_64/egg/vllm/entrypoints
copying build/lib.linux-x86_64-cpython-310/vllm/entrypoints/__init__.py -> build/bdist.linux-x86_64/egg/vllm/entrypoints
copying build/lib.linux-x86_64-cpython-310/vllm/entrypoints/api_server.py -> build/bdist.linux-x86_64/egg/vllm/entrypoints
creating build/bdist.linux-x86_64/egg/vllm/entrypoints/openai
copying build/lib.linux-x86_64-cpython-310/vllm/entrypoints/openai/serving_completion.py -> build/bdist.linux-x86_64/egg/vllm/entrypoints/openai
copying build/lib.linux-x86_64-cpython-310/vllm/entrypoints/openai/serving_chat.py -> build/bdist.linux-x86_64/egg/vllm/entrypoints/openai
copying build/lib.linux-x86_64-cpython-310/vllm/entrypoints/openai/__init__.py -> build/bdist.linux-x86_64/egg/vllm/entrypoints/openai
copying build/lib.linux-x86_64-cpython-310/vllm/entrypoints/openai/api_server.py -> build/bdist.linux-x86_64/egg/vllm/entrypoints/openai
copying build/lib.linux-x86_64-cpython-310/vllm/entrypoints/openai/protocol.py -> build/bdist.linux-x86_64/egg/vllm/entrypoints/openai
copying build/lib.linux-x86_64-cpython-310/vllm/entrypoints/openai/cli_args.py -> build/bdist.linux-x86_64/egg/vllm/entrypoints/openai
copying build/lib.linux-x86_64-cpython-310/vllm/entrypoints/openai/serving_engine.py -> build/bdist.linux-x86_64/egg/vllm/entrypoints/openai
copying build/lib.linux-x86_64-cpython-310/vllm/outputs.py -> build/bdist.linux-x86_64/egg/vllm
creating build/bdist.linux-x86_64/egg/vllm/usage
copying build/lib.linux-x86_64-cpython-310/vllm/usage/usage_lib.py -> build/bdist.linux-x86_64/egg/vllm/usage
copying build/lib.linux-x86_64-cpython-310/vllm/usage/__init__.py -> build/bdist.linux-x86_64/egg/vllm/usage
copying build/lib.linux-x86_64-cpython-310/vllm/block.py -> build/bdist.linux-x86_64/egg/vllm
copying build/lib.linux-x86_64-cpython-310/vllm/logger.py -> build/bdist.linux-x86_64/egg/vllm
creating build/bdist.linux-x86_64/egg/vllm/engine
copying build/lib.linux-x86_64-cpython-310/vllm/engine/async_llm_engine.py -> build/bdist.linux-x86_64/egg/vllm/engine
copying build/lib.linux-x86_64-cpython-310/vllm/engine/arg_utils.py -> build/bdist.linux-x86_64/egg/vllm/engine
copying build/lib.linux-x86_64-cpython-310/vllm/engine/metrics.py -> build/bdist.linux-x86_64/egg/vllm/engine
copying build/lib.linux-x86_64-cpython-310/vllm/engine/__init__.py -> build/bdist.linux-x86_64/egg/vllm/engine
copying build/lib.linux-x86_64-cpython-310/vllm/engine/ray_utils.py -> build/bdist.linux-x86_64/egg/vllm/engine
copying build/lib.linux-x86_64-cpython-310/vllm/engine/llm_engine.py -> build/bdist.linux-x86_64/egg/vllm/engine
copying build/lib.linux-x86_64-cpython-310/vllm/sequence.py -> build/bdist.linux-x86_64/egg/vllm
copying build/lib.linux-x86_64-cpython-310/vllm/_moe_C.cpython-310-x86_64-linux-gnu.so -> build/bdist.linux-x86_64/egg/vllm
copying build/lib.linux-x86_64-cpython-310/vllm/_C.cpython-310-x86_64-linux-gnu.so -> build/bdist.linux-x86_64/egg/vllm
creating build/bdist.linux-x86_64/egg/tests
creating build/bdist.linux-x86_64/egg/tests/spec_decode
copying build/lib.linux-x86_64-cpython-310/tests/spec_decode/test_utils.py -> build/bdist.linux-x86_64/egg/tests/spec_decode
copying build/lib.linux-x86_64-cpython-310/tests/spec_decode/utils.py -> build/bdist.linux-x86_64/egg/tests/spec_decode
copying build/lib.linux-x86_64-cpython-310/tests/spec_decode/test_batch_expansion.py -> build/bdist.linux-x86_64/egg/tests/spec_decode
copying build/lib.linux-x86_64-cpython-310/tests/spec_decode/test_metrics.py -> build/bdist.linux-x86_64/egg/tests/spec_decode
copying build/lib.linux-x86_64-cpython-310/tests/spec_decode/__init__.py -> build/bdist.linux-x86_64/egg/tests/spec_decode
copying build/lib.linux-x86_64-cpython-310/tests/spec_decode/test_spec_decode_worker.py -> build/bdist.linux-x86_64/egg/tests/spec_decode
copying build/lib.linux-x86_64-cpython-310/tests/spec_decode/test_multi_step_worker.py -> build/bdist.linux-x86_64/egg/tests/spec_decode
creating build/bdist.linux-x86_64/egg/tests/worker
copying build/lib.linux-x86_64-cpython-310/tests/worker/test_swap.py -> build/bdist.linux-x86_64/egg/tests/worker
copying build/lib.linux-x86_64-cpython-310/tests/worker/__init__.py -> build/bdist.linux-x86_64/egg/tests/worker
copying build/lib.linux-x86_64-cpython-310/tests/worker/test_model_runner.py -> build/bdist.linux-x86_64/egg/tests/worker
creating build/bdist.linux-x86_64/egg/tests/tokenization
copying build/lib.linux-x86_64-cpython-310/tests/tokenization/test_tokenizer_group.py -> build/bdist.linux-x86_64/egg/tests/tokenization
copying build/lib.linux-x86_64-cpython-310/tests/tokenization/test_detokenize.py -> build/bdist.linux-x86_64/egg/tests/tokenization
copying build/lib.linux-x86_64-cpython-310/tests/tokenization/__init__.py -> build/bdist.linux-x86_64/egg/tests/tokenization
copying build/lib.linux-x86_64-cpython-310/tests/tokenization/test_cached_tokenizer.py -> build/bdist.linux-x86_64/egg/tests/tokenization
creating build/bdist.linux-x86_64/egg/tests/lora
copying build/lib.linux-x86_64-cpython-310/tests/lora/test_tokenizer_group.py -> build/bdist.linux-x86_64/egg/tests/lora
copying build/lib.linux-x86_64-cpython-310/tests/lora/test_utils.py -> build/bdist.linux-x86_64/egg/tests/lora
copying build/lib.linux-x86_64-cpython-310/tests/lora/test_lora.py -> build/bdist.linux-x86_64/egg/tests/lora
copying build/lib.linux-x86_64-cpython-310/tests/lora/utils.py -> build/bdist.linux-x86_64/egg/tests/lora
copying build/lib.linux-x86_64-cpython-310/tests/lora/test_layers.py -> build/bdist.linux-x86_64/egg/tests/lora
copying build/lib.linux-x86_64-cpython-310/tests/lora/test_mixtral.py -> build/bdist.linux-x86_64/egg/tests/lora
copying build/lib.linux-x86_64-cpython-310/tests/lora/test_chatglm3.py -> build/bdist.linux-x86_64/egg/tests/lora
copying build/lib.linux-x86_64-cpython-310/tests/lora/test_lora_manager.py -> build/bdist.linux-x86_64/egg/tests/lora
copying build/lib.linux-x86_64-cpython-310/tests/lora/conftest.py -> build/bdist.linux-x86_64/egg/tests/lora
copying build/lib.linux-x86_64-cpython-310/tests/lora/test_llama.py -> build/bdist.linux-x86_64/egg/tests/lora
copying build/lib.linux-x86_64-cpython-310/tests/lora/test_worker.py -> build/bdist.linux-x86_64/egg/tests/lora
copying build/lib.linux-x86_64-cpython-310/tests/lora/test_punica.py -> build/bdist.linux-x86_64/egg/tests/lora
copying build/lib.linux-x86_64-cpython-310/tests/lora/__init__.py -> build/bdist.linux-x86_64/egg/tests/lora
copying build/lib.linux-x86_64-cpython-310/tests/lora/test_baichuan.py -> build/bdist.linux-x86_64/egg/tests/lora
copying build/lib.linux-x86_64-cpython-310/tests/lora/test_layer_variation.py -> build/bdist.linux-x86_64/egg/tests/lora
copying build/lib.linux-x86_64-cpython-310/tests/lora/test_gemma.py -> build/bdist.linux-x86_64/egg/tests/lora
creating build/bdist.linux-x86_64/egg/tests/core
copying build/lib.linux-x86_64-cpython-310/tests/core/utils.py -> build/bdist.linux-x86_64/egg/tests/core
copying build/lib.linux-x86_64-cpython-310/tests/core/test_block_manager.py -> build/bdist.linux-x86_64/egg/tests/core
copying build/lib.linux-x86_64-cpython-310/tests/core/test_scheduler.py -> build/bdist.linux-x86_64/egg/tests/core
creating build/bdist.linux-x86_64/egg/tests/core/block
copying build/lib.linux-x86_64-cpython-310/tests/core/block/test_naive_block.py -> build/bdist.linux-x86_64/egg/tests/core/block
copying build/lib.linux-x86_64-cpython-310/tests/core/block/test_cpu_gpu_block_allocator.py -> build/bdist.linux-x86_64/egg/tests/core/block
copying build/lib.linux-x86_64-cpython-310/tests/core/block/test_common.py -> build/bdist.linux-x86_64/egg/tests/core/block
copying build/lib.linux-x86_64-cpython-310/tests/core/block/test_prefix_caching_block.py -> build/bdist.linux-x86_64/egg/tests/core/block
copying build/lib.linux-x86_64-cpython-310/tests/core/block/__init__.py -> build/bdist.linux-x86_64/egg/tests/core/block
copying build/lib.linux-x86_64-cpython-310/tests/core/block/test_block_table.py -> build/bdist.linux-x86_64/egg/tests/core/block
copying build/lib.linux-x86_64-cpython-310/tests/core/block/test_block_space_manager.py -> build/bdist.linux-x86_64/egg/tests/core/block
copying build/lib.linux-x86_64-cpython-310/tests/core/__init__.py -> build/bdist.linux-x86_64/egg/tests/core
byte-compiling build/bdist.linux-x86_64/egg/vllm/test_utils.py to test_utils.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/utils.py to utils.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/attention/layer.py to layer.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/attention/__init__.py to __init__.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/attention/ops/paged_attn.py to paged_attn.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/attention/ops/prefix_prefill.py to prefix_prefill.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/attention/ops/__init__.py to __init__.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/attention/selector.py to selector.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/attention/backends/flash_attn.py to flash_attn.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/attention/backends/xformers.py to xformers.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/attention/backends/__init__.py to __init__.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/attention/backends/abstract.py to abstract.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/worker/neuron_model_runner.py to neuron_model_runner.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/worker/worker.py to worker.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/worker/cache_engine.py to cache_engine.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/worker/model_runner.py to model_runner.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/worker/__init__.py to __init__.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/worker/neuron_worker.py to neuron_worker.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/transformers_utils/tokenizer.py to tokenizer.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/transformers_utils/configs/__init__.py to __init__.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/transformers_utils/configs/chatglm.py to chatglm.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/transformers_utils/configs/jais.py to jais.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/transformers_utils/configs/mpt.py to mpt.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/transformers_utils/configs/dbrx.py to dbrx.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/transformers_utils/configs/falcon.py to falcon.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/transformers_utils/detokenizer.py to detokenizer.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/transformers_utils/__init__.py to __init__.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/transformers_utils/config.py to config.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/transformers_utils/tokenizers/__init__.py to __init__.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/transformers_utils/tokenizers/baichuan.py to baichuan.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/transformers_utils/tokenizer_group/ray_tokenizer_group.py to ray_tokenizer_group.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/transformers_utils/tokenizer_group/tokenizer_group.py to tokenizer_group.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/transformers_utils/tokenizer_group/__init__.py to __init__.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/transformers_utils/tokenizer_group/base_tokenizer_group.py to base_tokenizer_group.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/lora/models.py to models.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/lora/utils.py to utils.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/lora/worker_manager.py to worker_manager.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/lora/request.py to request.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/lora/__init__.py to __init__.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/lora/punica.py to punica.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/lora/lora.py to lora.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/lora/layers.py to layers.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/core/block_manager_v1.py to block_manager_v1.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/core/evictor.py to evictor.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/core/block_manager_v2.py to block_manager_v2.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/core/scheduler.py to scheduler.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/core/__init__.py to __init__.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/core/policy.py to policy.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/core/interfaces.py to interfaces.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/model_executor/layers/sampler.py to sampler.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/model_executor/layers/layernorm.py to layernorm.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/model_executor/layers/fused_moe/__init__.py to __init__.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/model_executor/layers/fused_moe/fused_moe.py to fused_moe.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/model_executor/layers/vocab_parallel_embedding.py to vocab_parallel_embedding.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/model_executor/layers/rotary_embedding.py to rotary_embedding.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/model_executor/layers/linear.py to linear.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/model_executor/layers/quantization/squeezellm.py to squeezellm.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/model_executor/layers/quantization/gptq.py to gptq.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/model_executor/layers/quantization/__init__.py to __init__.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/model_executor/layers/quantization/marlin.py to marlin.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/model_executor/layers/quantization/base_config.py to base_config.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/model_executor/layers/quantization/awq.py to awq.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/model_executor/layers/__init__.py to __init__.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/model_executor/layers/ops/sample.py to sample.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/model_executor/layers/ops/__init__.py to __init__.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/model_executor/layers/ops/rand.py to rand.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/model_executor/layers/rejection_sampler.py to rejection_sampler.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/model_executor/layers/logits_processor.py to logits_processor.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/model_executor/layers/activation.py to activation.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/model_executor/utils.py to utils.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/model_executor/sampling_metadata.py to sampling_metadata.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/model_executor/model_loader.py to model_loader.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/model_executor/guided_decoding.py to guided_decoding.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/model_executor/__init__.py to __init__.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/model_executor/guided_logits_processors.py to guided_logits_processors.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/model_executor/parallel_utils/parallel_state.py to parallel_state.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/model_executor/parallel_utils/utils.py to utils.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/model_executor/parallel_utils/custom_all_reduce.py to custom_all_reduce.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/model_executor/parallel_utils/communication_op.py to communication_op.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/model_executor/parallel_utils/__init__.py to __init__.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/model_executor/parallel_utils/pynccl.py to pynccl.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/model_executor/parallel_utils/pynccl_utils.py to pynccl_utils.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/model_executor/weight_utils.py to weight_utils.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/model_executor/models/bloom.py to bloom.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/model_executor/models/orion.py to orion.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/model_executor/models/qwen2_moe.py to qwen2_moe.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/model_executor/models/decilm.py to decilm.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/model_executor/models/gpt2.py to gpt2.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/model_executor/models/gpt_bigcode.py to gpt_bigcode.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/model_executor/models/qwen2.py to qwen2.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/model_executor/models/starcoder2.py to starcoder2.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/model_executor/models/opt.py to opt.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/model_executor/models/mixtral_quant.py to mixtral_quant.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/model_executor/models/olmo.py to olmo.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/model_executor/models/__init__.py to __init__.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/model_executor/models/gemma.py to gemma.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/model_executor/models/phi.py to phi.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/model_executor/models/llama.py to llama.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/model_executor/models/gpt_j.py to gpt_j.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/model_executor/models/chatglm.py to chatglm.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/model_executor/models/commandr.py to commandr.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/model_executor/models/xverse.py to xverse.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/model_executor/models/stablelm.py to stablelm.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/model_executor/models/jais.py to jais.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/model_executor/models/internlm2.py to internlm2.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/model_executor/models/mixtral.py to mixtral.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/model_executor/models/qwen.py to qwen.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/model_executor/models/deepseek.py to deepseek.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/model_executor/models/mpt.py to mpt.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/model_executor/models/gpt_neox.py to gpt_neox.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/model_executor/models/llava.py to llava.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/model_executor/models/dbrx.py to dbrx.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/model_executor/models/baichuan.py to baichuan.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/model_executor/models/falcon.py to falcon.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/model_executor/neuron_model_loader.py to neuron_model_loader.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/executor/utils.py to utils.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/executor/ray_gpu_executor.py to ray_gpu_executor.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/executor/executor_base.py to executor_base.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/executor/gpu_executor.py to gpu_executor.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/executor/neuron_executor.py to neuron_executor.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/executor/__init__.py to __init__.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/sampling_params.py to sampling_params.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/__init__.py to __init__.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/config.py to config.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/entrypoints/llm.py to llm.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/entrypoints/__init__.py to __init__.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/entrypoints/api_server.py to api_server.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/entrypoints/openai/serving_completion.py to serving_completion.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/entrypoints/openai/serving_chat.py to serving_chat.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/entrypoints/openai/__init__.py to __init__.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/entrypoints/openai/api_server.py to api_server.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/entrypoints/openai/protocol.py to protocol.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/entrypoints/openai/cli_args.py to cli_args.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/entrypoints/openai/serving_engine.py to serving_engine.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/outputs.py to outputs.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/usage/usage_lib.py to usage_lib.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/usage/__init__.py to __init__.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/block.py to block.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/logger.py to logger.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/engine/async_llm_engine.py to async_llm_engine.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/engine/arg_utils.py to arg_utils.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/engine/metrics.py to metrics.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/engine/__init__.py to __init__.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/engine/ray_utils.py to ray_utils.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/engine/llm_engine.py to llm_engine.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/sequence.py to sequence.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/tests/spec_decode/test_utils.py to test_utils.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/tests/spec_decode/utils.py to utils.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/tests/spec_decode/test_batch_expansion.py to test_batch_expansion.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/tests/spec_decode/test_metrics.py to test_metrics.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/tests/spec_decode/__init__.py to __init__.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/tests/spec_decode/test_spec_decode_worker.py to test_spec_decode_worker.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/tests/spec_decode/test_multi_step_worker.py to test_multi_step_worker.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/tests/worker/test_swap.py to test_swap.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/tests/worker/__init__.py to __init__.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/tests/worker/test_model_runner.py to test_model_runner.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/tests/tokenization/test_tokenizer_group.py to test_tokenizer_group.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/tests/tokenization/test_detokenize.py to test_detokenize.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/tests/tokenization/__init__.py to __init__.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/tests/tokenization/test_cached_tokenizer.py to test_cached_tokenizer.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/tests/lora/test_tokenizer_group.py to test_tokenizer_group.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/tests/lora/test_utils.py to test_utils.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/tests/lora/test_lora.py to test_lora.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/tests/lora/utils.py to utils.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/tests/lora/test_layers.py to test_layers.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/tests/lora/test_mixtral.py to test_mixtral.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/tests/lora/test_chatglm3.py to test_chatglm3.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/tests/lora/test_lora_manager.py to test_lora_manager.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/tests/lora/conftest.py to conftest.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/tests/lora/test_llama.py to test_llama.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/tests/lora/test_worker.py to test_worker.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/tests/lora/test_punica.py to test_punica.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/tests/lora/__init__.py to __init__.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/tests/lora/test_baichuan.py to test_baichuan.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/tests/lora/test_layer_variation.py to test_layer_variation.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/tests/lora/test_gemma.py to test_gemma.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/tests/core/utils.py to utils.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/tests/core/test_block_manager.py to test_block_manager.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/tests/core/test_scheduler.py to test_scheduler.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/tests/core/block/test_naive_block.py to test_naive_block.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/tests/core/block/test_cpu_gpu_block_allocator.py to test_cpu_gpu_block_allocator.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/tests/core/block/test_common.py to test_common.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/tests/core/block/test_prefix_caching_block.py to test_prefix_caching_block.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/tests/core/block/__init__.py to __init__.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/tests/core/block/test_block_table.py to test_block_table.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/tests/core/block/test_block_space_manager.py to test_block_space_manager.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/tests/core/__init__.py to __init__.cpython-310.pyc
creating stub loader for vllm/_moe_C.cpython-310-x86_64-linux-gnu.so
creating stub loader for vllm/_C.cpython-310-x86_64-linux-gnu.so
byte-compiling build/bdist.linux-x86_64/egg/vllm/_moe_C.py to _moe_C.cpython-310.pyc
byte-compiling build/bdist.linux-x86_64/egg/vllm/_C.py to _C.cpython-310.pyc
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying vllm.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying vllm.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying vllm.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying vllm.egg-info/requires.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying vllm.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
writing build/bdist.linux-x86_64/egg/EGG-INFO/native_libs.txt
zip_safe flag not set; analyzing archive contents...
vllm.__pycache__._C.cpython-310: module references __file__
vllm.__pycache__._moe_C.cpython-310: module references __file__
vllm.model_executor.layers.fused_moe.__pycache__.fused_moe.cpython-310: module references __file__
creating dist
creating dist/vllm-0.4.0-py3.10-linux-x86_64.egg and adding build/bdist.linux-x86_64/egg to it
removing build/bdist.linux-x86_64/egg (and everything under it)
Processing vllm-0.4.0-py3.10-linux-x86_64.egg
creating /usr/local/lib/python3.10/dist-packages/vllm-0.4.0-py3.10-linux-x86_64.egg