[Model] Meta Llama 3.1 Know Issues & FAQ

Please checkout Announcing Llama 3.1 Support in vLLM

Chunked prefill is turned on for all Llama 3.1 models. However, it is currently incompatible with prefix caching, sliding window, and multi-lora. In order to use those features, you can set --enable-chunked-prefill=false then optionally combine it with --max-model-len=4096 if turning it out cause OOM. You can change the length for the context window you desired.
Rope scaling if rope_scaling is not None and rope_scaling["type"] not in, KeyError: 'type'.

Please update to v0.5.3.post1 which included a fix.

Rope scaling ValueError: 'rope_scaling' must be a dictionary with two fields, 'type' and 'factor', got {'factor': 8.0, 'low_freq_factor': 1.0, 'high_freq_factor': 4.0, 'original_max_position_embeddings': 8192, 'rope_type': 'llama3'}

Please upgrade transformers to 4.43.1 (pip install transformers --upgrade)

Using a per-request random seed currently does not work with pipeline parallel deployments (https://github.com/vllm-project/vllm/issues/6449). This will be fixed soon.

Just me or are other people also having issues running llama 3.1 models? My error:
if rope_scaling is not None and rope_scaling["type"] not in, KeyError: 'type'.
Config:
llm = LLM(model=MODEL,
              tensor_parallel_size=NUM_GPUS,
              enable_prefix_caching=False,
              gpu_memory_utilization=0.80,
              max_model_len=4096,
              trust_remote_code=True,
              max_num_seqs=16,
              )

I think there's some issue with parsing the config fetched from huggingface:
$ vllm serve meta-llama/Meta-Llama-3.1-405B-Instruct-FP8 --tensor-parallel-size 8
INFO 07-23 16:04:42 api_server.py:219] vLLM API server version 0.5.3
INFO 07-23 16:04:42 api_server.py:220] args: Namespace(model_tag='meta-llama/Meta-Llama-3.1-405B-Instruct-FP8', host=None, port=8000, uvicorn_log_level='info', allow_credential
s=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], model='meta-llama/Meta-Llama-3.1-405B-Instruct-FP8', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, download_dir=None, load_format='auto', dtype='auto', kv_cache_dtype='auto', quantization_param_path=None, max_model_len=None, guided_decoding_backend='outlines', distributed_executor_backend=None, worker_use_ray=False, pipeline_parallel_size=1, tensor_parallel_size=8, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=16, enable_prefix_caching=False, disable_sliding_window=False, use_v2_block_manager=False, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.9, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_seqs=256, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, enforce_eager=False, max_context_len_to_capture=None, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, enable_lora=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_model=None, num_speculative_tokens=None, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=None, qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, engine_use_ray=False, disable_log_requests=False, max_log_len=None, dispatch_function=<function serve at 0x7fee7856c5e0>)
config.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 22.5k/22.5k [00:00<00:00, 85.5MB/s]
Traceback (most recent call last):
  File "/opt/conda/bin/vllm", line 8, in <module>
    sys.exit(main())
  File "/opt/conda/lib/python3.10/site-packages/vllm/scripts.py", line 148, in main
    args.dispatch_function(args)
  File "/opt/conda/lib/python3.10/site-packages/vllm/scripts.py", line 28, in serve
    run_server(args)
  File "/opt/conda/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 231, in run_server
    if llm_engine is not None else AsyncLLMEngine.from_engine_args(
  File "/opt/conda/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 457, in from_engine_args
    engine_config = engine_args.create_engine_config()
  File "/opt/conda/lib/python3.10/site-packages/vllm/engine/arg_utils.py", line 699, in create_engine_config
    model_config = ModelConfig(
  File "/opt/conda/lib/python3.10/site-packages/vllm/config.py", line 176, in __init__
    self.max_model_len = _get_and_verify_max_len(
  File "/opt/conda/lib/python3.10/site-packages/vllm/config.py", line 1497, in _get_and_verify_max_len
    if rope_scaling is not None and rope_scaling["type"] not in {
KeyError: 'type'

Version:
pip freeze | grep vllm
vllm==0.5.3
vllm-flash-attn==2.5.9.post1

I'm having the same issue as @Vaibhav-Sahai
Command:
python -m vllm.entrypoints.openai.api_server --model meta-llama/Meta-Llama-3.1-70B-Instruct --tensor-parallel-size 2

Error trace:
File "envs/synthenv/lib/python3.10/site-packages/vllm/config.py", line 176, in __init__
  self.max_model_len = _get_and_verify_max_len(
File "envs/synthenv/lib/python3.10/site-packages/vllm/config.py", line 1497, in _get_and_verify_max_len
  if rope_scaling is not None and rope_scaling["type"] not in {
KeyError: 'type'

Version:
vllm==0.5.3
vllm-flash-attn==2.5.9.post1

Hello there! Thanks for reporting and issue and we are aware of it. This is due to the fact that HuggingFace decides to rename this key ("rope_type" instead of "type") in the repo of all Llama 3.1 models.

In CLI or Python, pass in --rope-scaling='{"type": "dummy"} or rope_scaling={"type": "dummy"} should get around this for now

@simon-mo maybe i missed something, but getting the following log:
assert "factor" in rope_scaling
AssertionError

LLM config:
llm = LLM(model=MODEL,
              tensor_parallel_size=NUM_GPUS,
              enable_prefix_caching=False,
              gpu_memory_utilization=0.90,
              max_model_len=4096,
              trust_remote_code=True,
              max_num_seqs=16,
              rope_scaling={"type": "dummy"},
 )

Version:
vllm==0.5.3

@Vaibhav-Sahai updated my hack, PTAL.

@Vaibhav-Sahai @romilbhardwaj @casper-hansen We fixed the RoPE issue in #6693. The model should work without any extra args now in the main branch.

Thanks @WoosukKwon. Are you guys planning any post release today or should we build from source until 0.5.4?

We plan to release ASAP after confirmation with the HuggingFace team.

Llama 3.1 405B base in fp8 is just generating !!!! over and over for me, is anyone else having this issue? I verified that 70B base works for me (not in fp8 though).

Hey is this the same issue?
ValueError: `rope_scaling` must be a dictionary with two fields, `type` and `factor`, got {'factor': 8.0, 'low_freq_factor': 1.0, 'high_freq_factor': 4.0, 'original_max_position_embeddings': 8192, 'rope_type': 'llama3'}

I am using the following configuration
Namespace(host=None, port=8000, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], model='meta-llama/Meta-Llama-3.1-405B-Instruct-FP8', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, download_dir=None, load_format='auto', dtype='auto', kv_cache_dtype='auto', quantization_param_path=None, max_model_len=8192, guided_decoding_backend='outlines', distributed_executor_backend=None, worker_use_ray=False, pipeline_parallel_size=1, tensor_parallel_size=8, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=16, enable_prefix_caching=False, disable_sliding_window=False, use_v2_block_manager=False, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.9, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_seqs=256, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling={'type': 'extended', 'factor': 8.0}, rope_theta=None, enforce_eager=False, max_context_len_to_capture=None, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, enable_lora=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_model=None, num_speculative_tokens=None, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=None, qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, engine_use_ray=False, disable_log_requests=False, max_log_len=None)

As suggested used rope_scaling={"type": "extended", "factor": 8.0} this with vllm openai server

Please try update the vLLM version and you don't need the rope scaling anymore

@simon-mo, same issue as @akhil-netomi for me. Just updated to post1, I unfortunately get the same issue?

https://github.com/vllm-project/vllm/blob/38c4b7e863570a045308af814c72f4504297222e/vllm/transformers_utils/configs/chameleon.py#L81
This might be where the issue lies - although I'm not familiar with the codebase

Hey!
trying to get Llama3.1-405B-FP8 working with vLLM, getting this error
RuntimeError: "fill_empty_deterministic_" not implemented for 'Float8_e4m3fn'
I updated to the latest vLLM version and Llama3.1-70B works

@sumukshashidhar are you trying to use chameleon? That class won't be touched unless you're trying to serve ChameleonForConditionalGeneration.

Llama 3.1 405B base in fp8 is just generating !!!! over and over for me, is anyone else having this issue? I verified that 70B base works for me (but it's not in fp8).

405B instruct FP8 works for me. It's just the base model that is not working. Also, base and instruct seem to use different amounts of GPU memory, which should not happen (base uses less).

@simon-mo @akhil-netomi Please try upgrading transformers with pip install --upgrade transformers

For those that want to test vLLM with gptq_marlin compatible 4bit quants we have just pushed both 8B models to HF:
https://huggingface.co/ModelCloud/Meta-Llama-3.1-8B-Instruct-gptq-4bit
https://huggingface.co/ModelCloud/Meta-Llama-3.1-8B-gptq-4bit

@ywang96  my bad, I'm not trying to use chameleon, my error is in the normal transformers module.  @WoosukKwon yes, upgrading transformers fixes it. Thanks!

Just sharing - vllm serve meta-llama/Meta-Llama-3.1-405B-Instruct-FP8 --tensor-parallel-size 8 --max-model-len 4096 works well on A100-80GB:8 with v0.5.3.post1.
I'm also able to run the 8B model on L4:1 with vllm serve meta-llama/Meta-Llama-3.1-8B-Instruct --tensor-parallel-size 1 --max-model-len 1024.
Thanks for the great work vllm team!

Also posting on AWQ: the latest patch fixed it and benchmark now runs.
python benchmarks/benchmark_throughput.py \
    --input-len 512 \
    --output-len 256 \
    --model hugging-quants/Meta-Llama-3.1-405B-Instruct-AWQ-INT4 \
    --quantization marlin \
    -tp 4 \
    --num-prompts 100 \
    --max-model-len 1024 \
    --dtype half

@romilbhardwaj @casper-hansen Thanks for sharing! Could you please share why you limit max-mode-len? Did you experience OOMs without it?

I no longer have the error on hand, but it was along the lines of [rank0]: ValueError: The model's max seq len (...) is larger than the maximum number of tokens that can be stored in KV cache (...) ...

QQ, what is good choice for max-model-len for 4*H100?
Should I just find out?  If anyone has suggestions welcome.  That is, 128k is default, I presume.  I'm using
--max-num-batched-tokens=16384 without any --max-model-len, so I presume that's limiting it indirectly.
But should I go for 16k, 32k, 64k, 128k?  What can 4*H100 support at full context length?

Llama 3.1 405B base in fp8 is just generating !!!! over and over for me, is anyone else having this issue? I verified that 70B base works for me (but it's not in fp8).

405B instruct FP8 works for me. It's just the base model that is not working. Also, base and instruct seem to use different amounts of GPU memory, which should not happen (base uses less).

I figured this out. The 405B FP8 base model config.json has a list of "modules_to_not_convert" that is different from the instruct model. vllm apparently does not run submodules of modules in this list as 16-bit, so I pasted the list from the instruct model config.json, which includes all of the submodules, into the base model config.json and this fixed the issue where I was getting NaNs in inference.

@romilbhardwaj @casper-hansen Thanks for sharing! Could you please share why you limit max-mode-len? Did you experience OOMs without it?

I limited the sequence length just for testing/benchmarking purposes. If my memory serves me correctly, I don't believe it can load 128k context length on the setup of 4x H100 that I was running earlier.

I am trying to deploy fP8 model with 8 A100 and getting below error
WARNING 07-23 22:25:48 utils.py:569] Your GPU does not have native support for FP8 computation but FP8 quantization is being used. Weight-only FP8 compression will be used leveraging the Marlin kernel. This may degrade performance for compute-heavy workloads. INFO 07-23 22:25:49 model_runner.py:692] Loading model weights took 57.7485 GB [rank0]: Traceback (most recent call last): [rank0]:   File "/usr/local/lib/python3.10/runpy.py", line 196, in _run_module_as_main [rank0]:     return _run_code(code, main_globals, None, [rank0]:   File "/usr/local/lib/python3.10/runpy.py", line 86, in _run_code [rank0]:     exec(code, run_globals) [rank0]:   File "/usr/local/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 317, in <module> [rank0]:     run_server(args) [rank0]:   File "/usr/local/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 231, in run_server [rank0]:     if llm_engine is not None else AsyncLLMEngine.from_engine_args( [rank0]:   File "/usr/local/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 466, in from_engine_args [rank0]:     engine = cls( [rank0]:   File "/usr/local/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 380, in __init__ [rank0]:     self.engine = self._init_engine(*args, **kwargs) [rank0]:   File "/usr/local/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 547, in _init_engine [rank0]:     return engine_class(*args, **kwargs) [rank0]:   File "/usr/local/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 265, in __init__ [rank0]:     self._initialize_kv_caches() [rank0]:   File "/usr/local/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 364, in _initialize_kv_caches [rank0]:     self.model_executor.determine_num_available_blocks()) [rank0]:   File "/usr/local/lib/python3.10/site-packages/vllm/executor/distributed_gpu_executor.py", line 38, in determine_num_available_blocks [rank0]:     num_blocks = self._run_workers("determine_num_available_blocks", ) [rank0]:   File "/usr/local/lib/python3.10/site-packages/vllm/executor/multiproc_gpu_executor.py", line 178, in _run_workers [rank0]:     driver_worker_output = driver_worker_method(*args, **kwargs) [rank0]:   File "/usr/local/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context [rank0]:     return func(*args, **kwargs) [rank0]:   File "/usr/local/lib/python3.10/site-packages/vllm/worker/worker.py", line 179, in determine_num_available_blocks [rank0]:     self.model_runner.profile_run() [rank0]:   File "/usr/local/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context [rank0]:     return func(*args, **kwargs) [rank0]:   File "/usr/local/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 896, in profile_run [rank0]:     self.execute_model(model_input, kv_caches, intermediate_tensors) [rank0]:   File "/usr/local/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context [rank0]:     return func(*args, **kwargs) [rank0]:   File "/usr/local/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 1314, in execute_model [rank0]:     hidden_or_intermediate_states = model_executable( [rank0]:   File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl [rank0]:     return self._call_impl(*args, **kwargs) [rank0]:   File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl [rank0]:     return forward_call(*args, **kwargs) [rank0]:   File "/usr/local/lib/python3.10/site-packages/vllm/model_executor/models/llama.py", line 422, in forward [rank0]:     model_output = self.model(input_ids, positions, kv_caches, [rank0]:   File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl [rank0]:     return self._call_impl(*args, **kwargs) [rank0]:   File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl [rank0]:     return forward_call(*args, **kwargs) [rank0]:   File "/usr/local/lib/python3.10/site-packages/vllm/model_executor/models/llama.py", line 313, in forward [rank0]:     hidden_states = self.get_input_embeddings(input_ids) [rank0]:   File "/usr/local/lib/python3.10/site-packages/vllm/model_executor/models/llama.py", line 298, in get_input_embeddings [rank0]:     return self.embed_tokens(input_ids) [rank0]:   File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl [rank0]:     return self._call_impl(*args, **kwargs) [rank0]:   File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl [rank0]:     return forward_call(*args, **kwargs) [rank0]:   File "/usr/local/lib/python3.10/site-packages/vllm/model_executor/layers/vocab_parallel_embedding.py", line 352, in forward [rank0]:     output = tensor_model_parallel_all_reduce(output_parallel) [rank0]:   File "/usr/local/lib/python3.10/site-packages/vllm/distributed/communication_op.py", line 11, in tensor_model_parallel_all_reduce [rank0]:     return get_tp_group().all_reduce(input_) [rank0]:   File "/usr/local/lib/python3.10/site-packages/vllm/distributed/parallel_state.py", line 293, in all_reduce [rank0]:     torch.distributed.all_reduce(input_, group=self.device_group) [rank0]:   File "/usr/local/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper [rank0]:     return func(*args, **kwargs) [rank0]:   File "/usr/local/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 2219, in all_reduce [rank0]:     work = group.allreduce([tensor], opts) [rank0]: torch.distributed.DistBackendError: NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1970, unhandled system error (run with NCCL_DEBUG=INFO for details), NCCL version 2.20.5 [rank0]: ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error. [rank0]: Last error: [rank0]: Error while creating shared memory segment /dev/shm/nccl-YYwZmV (size 5767520)

Hi team,I am trying deploy fp8 llama3.1 405b model and running into below issue ..any help is appreciated
torch.distributed.DistBackendError: NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1970, unhandled system error (run with NCCL_DEBUG=INFO for details), NCCL version 2.20.5 [rank0]: ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error. [rank0]: Last error: [rank0]: Error while creating shared memory segment /dev/shm/nccl-eQz2sT (size 5767520) ERROR 07-23 22:46:18 multiproc_worker_utils.py:120] Worker VllmWorkerProcess pid 82 died, exit code: -15 INFO 07-23 22:46:18 multiproc_worker_utils.py:123] Killing local vLLM worker processes [rank0]:[W CudaIPCTypes.cpp:16] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors] /usr/local/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown   warnings.warn('resource_tracker: There appear to be %d '

@nani1149

Error while creating shared memory segment /dev/shm/nccl-eQz2sT (size 5767520)

you don't give enough size for the shared memory. if you are using docker, please see https://docs.vllm.ai/en/latest/serving/deploying_with_docker.html for how to adjust shm size.

python -m vllm.entrypoints.openai.api_server
--model meta-llama/Meta-Llama-3.1-70B
--tensor-parallel-size 4
--gpu-memory-utilization 0.95
--rope-scaling='{"type": "extended", "factor": 8.0}'
Above command work for me, but it result a very strange model name, "facebook/opt-125m", and the answer from this model is very strange and crazy. Anyone has the same problem?

@alexchenyu you need \ in the end of the first line

@alexchenyu you need \ in the end of the first line

Oh, my bad, thanks for your reply.

I'm currently using 0.4.2, is this version support llama3.1? We only need a deployable version for demo.

405B-FP8 base worked well for me on 8xH100 after changing the config (thanks @crowsonkb!), but at some point it started throwing CUDA errors:
2024-07-24T04:34:21.448550002Z Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:43 (most recent call first):
2024-07-24T04:34:21.448553024Z frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7065336bf897 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
2024-07-24T04:34:21.448555571Z frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x70653366fb25 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
2024-07-24T04:34:21.448558757Z frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x706533797718 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10_cuda.so)
2024-07-24T04:34:21.448561932Z frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7065349948e6 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
2024-07-24T04:34:21.448564812Z frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0x58 (0x7065349989e8 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
2024-07-24T04:34:21.448567287Z frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x77c (0x70653499e05c in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
2024-07-24T04:34:21.448584270Z frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x70653499edcc in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
2024-07-24T04:34:21.448587724Z frame #7: <unknown function> + 0xd6df4 (0x706580455df4 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
2024-07-24T04:34:21.448590886Z frame #8: <unknown function> + 0x8609 (0x706581517609 in /usr/lib/x86_64-linux-gnu/libpthread.so.0)
2024-07-24T04:34:21.448593376Z frame #9: clone + 0x43 (0x706581651353 in /usr/lib/x86_64-linux-gnu/libc.so.6)
2024-07-24T04:34:21.448596260Z
2024-07-24T04:34:21.449893760Z terminate called after throwing an instance of 'c10::DistBackendError'
2024-07-24T04:34:21.449947144Z   what():  [PG 2 Rank 3] Process group watchdog thread terminated with exception: CUDA error: an illegal memory access was encountered
2024-07-24T04:34:21.449954579Z CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
2024-07-24T04:34:21.449960388Z For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
2024-07-24T04:34:21.449966439Z Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
2024-07-24T04:34:21.449971923Z
2024-07-24T04:34:21.449977084Z Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:43 (most recent call first):
2024-07-24T04:34:21.449982622Z frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7065336bf897 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
2024-07-24T04:34:21.449988402Z frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x70653366fb25 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
2024-07-24T04:34:21.449994893Z frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x706533797718 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10_cuda.so)
2024-07-24T04:34:21.450000848Z frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7065349948e6 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
2024-07-24T04:34:21.450006518Z frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0x58 (0x7065349989e8 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
2024-07-24T04:34:21.450012246Z frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x77c (0x70653499e05c in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
2024-07-24T04:34:21.450017501Z frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x70653499edcc in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
2024-07-24T04:34:21.450022907Z frame #7: <unknown function> + 0xd6df4 (0x706580455df4 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
2024-07-24T04:34:21.450028932Z frame #8: <unknown function> + 0x8609 (0x706581517609 in /usr/lib/x86_64-linux-gnu/libpthread.so.0)
2024-07-24T04:34:21.450034511Z frame #9: clone + 0x43 (0x706581651353 in /usr/lib/x86_64-linux-gnu/libc.so.6)
2024-07-24T04:34:21.450039916Z
2024-07-24T04:34:21.450047055Z Exception raised from ncclCommWatchdog at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1418 (most recent call first):
2024-07-24T04:34:21.450052480Z frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7065336bf897 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
2024-07-24T04:34:21.450058156Z frame #1: <unknown function> + 0xe32119 (0x706534622119 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
2024-07-24T04:34:21.450062761Z frame #2: <unknown function> + 0xd6df4 (0x706580455df4 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
2024-07-24T04:34:21.450068309Z frame #3: <unknown function> + 0x8609 (0x706581517609 in /usr/lib/x86_64-linux-gnu/libpthread.so.0)
2024-07-24T04:34:21.450073485Z frame #4: clone + 0x43 (0x706581651353 in /usr/lib/x86_64-linux-gnu/libc.so.6)
2024-07-24T04:34:21.450078404Z
2024-07-24T04:34:26.082870754Z INFO 07-24 04:34:26 metrics.py:396] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 5.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2%, CPU KV cache usage: 0.0%.
2024-07-24T04:34:27.165587884Z ERROR 07-24 04:34:27 multiproc_worker_utils.py:120] Worker VllmWorkerProcess pid 83 died, exit code: -6
2024-07-24T04:34:27.165637073Z INFO 07-24 04:34:27 multiproc_worker_utils.py:123] Killing local vLLM worker processes
2024-07-24T04:34:36.085684429Z INFO 07-24 04:34:36 metrics.py:396] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2%, CPU KV cache usage: 0.0%.
2024-07-24T04:34:46.087863055Z INFO 07-24 04:34:46 metrics.py:396] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2%, CPU KV cache usage: 0.0%.
2024-07-24T04:34:56.090458203Z INFO 07-24 04:34:56 metrics.py:396] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2%, CPU KV cache usage: 0.0%.

how to deploy fp16 405B model by using vllm?

Got this:
UNAVAILABLE: Internal: RuntimeError: Expected a.dtype() == torch::kInt8 to be true, but got false.  (Could this error message be improved?  If so, please report an enhancement request .
I'm using vllm 0.5.3-post1

@crowsonkb "We’ve worked with key community projects like vLLM, TensorRT, and PyTorch to build in support from day one to ensure the community is ready for production deployment." so much about that, didn't even test all models apparently lol

@kharvd How many TPS did you get? ^^

405B-FP8 base worked well for me on 8xH100 after changing the config (thanks @crowsonkb!), but at some point it started throwing CUDA errors:
2024-07-24T04:34:21.448550002Z Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:43 (most recent call first):
2024-07-24T04:34:21.448553024Z frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7065336bf897 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
2024-07-24T04:34:21.448555571Z frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x70653366fb25 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
2024-07-24T04:34:21.448558757Z frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x706533797718 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10_cuda.so)
2024-07-24T04:34:21.448561932Z frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7065349948e6 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
2024-07-24T04:34:21.448564812Z frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0x58 (0x7065349989e8 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
2024-07-24T04:34:21.448567287Z frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x77c (0x70653499e05c in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
2024-07-24T04:34:21.448584270Z frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x70653499edcc in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
2024-07-24T04:34:21.448587724Z frame #7: <unknown function> + 0xd6df4 (0x706580455df4 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
2024-07-24T04:34:21.448590886Z frame #8: <unknown function> + 0x8609 (0x706581517609 in /usr/lib/x86_64-linux-gnu/libpthread.so.0)
2024-07-24T04:34:21.448593376Z frame #9: clone + 0x43 (0x706581651353 in /usr/lib/x86_64-linux-gnu/libc.so.6)
2024-07-24T04:34:21.448596260Z
2024-07-24T04:34:21.449893760Z terminate called after throwing an instance of 'c10::DistBackendError'
2024-07-24T04:34:21.449947144Z   what():  [PG 2 Rank 3] Process group watchdog thread terminated with exception: CUDA error: an illegal memory access was encountered
2024-07-24T04:34:21.449954579Z CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
2024-07-24T04:34:21.449960388Z For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
2024-07-24T04:34:21.449966439Z Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
2024-07-24T04:34:21.449971923Z
2024-07-24T04:34:21.449977084Z Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:43 (most recent call first):
2024-07-24T04:34:21.449982622Z frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7065336bf897 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
2024-07-24T04:34:21.449988402Z frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x70653366fb25 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
2024-07-24T04:34:21.449994893Z frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x706533797718 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10_cuda.so)
2024-07-24T04:34:21.450000848Z frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7065349948e6 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
2024-07-24T04:34:21.450006518Z frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0x58 (0x7065349989e8 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
2024-07-24T04:34:21.450012246Z frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x77c (0x70653499e05c in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
2024-07-24T04:34:21.450017501Z frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x70653499edcc in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
2024-07-24T04:34:21.450022907Z frame #7: <unknown function> + 0xd6df4 (0x706580455df4 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
2024-07-24T04:34:21.450028932Z frame #8: <unknown function> + 0x8609 (0x706581517609 in /usr/lib/x86_64-linux-gnu/libpthread.so.0)
2024-07-24T04:34:21.450034511Z frame #9: clone + 0x43 (0x706581651353 in /usr/lib/x86_64-linux-gnu/libc.so.6)
2024-07-24T04:34:21.450039916Z
2024-07-24T04:34:21.450047055Z Exception raised from ncclCommWatchdog at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1418 (most recent call first):
2024-07-24T04:34:21.450052480Z frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7065336bf897 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
2024-07-24T04:34:21.450058156Z frame #1: <unknown function> + 0xe32119 (0x706534622119 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
2024-07-24T04:34:21.450062761Z frame #2: <unknown function> + 0xd6df4 (0x706580455df4 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
2024-07-24T04:34:21.450068309Z frame #3: <unknown function> + 0x8609 (0x706581517609 in /usr/lib/x86_64-linux-gnu/libpthread.so.0)
2024-07-24T04:34:21.450073485Z frame #4: clone + 0x43 (0x706581651353 in /usr/lib/x86_64-linux-gnu/libc.so.6)
2024-07-24T04:34:21.450078404Z
2024-07-24T04:34:26.082870754Z INFO 07-24 04:34:26 metrics.py:396] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 5.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2%, CPU KV cache usage: 0.0%.
2024-07-24T04:34:27.165587884Z ERROR 07-24 04:34:27 multiproc_worker_utils.py:120] Worker VllmWorkerProcess pid 83 died, exit code: -6
2024-07-24T04:34:27.165637073Z INFO 07-24 04:34:27 multiproc_worker_utils.py:123] Killing local vLLM worker processes
2024-07-24T04:34:36.085684429Z INFO 07-24 04:34:36 metrics.py:396] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2%, CPU KV cache usage: 0.0%.
2024-07-24T04:34:46.087863055Z INFO 07-24 04:34:46 metrics.py:396] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2%, CPU KV cache usage: 0.0%.
2024-07-24T04:34:56.090458203Z INFO 07-24 04:34:56 metrics.py:396] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2%, CPU KV cache usage: 0.0%.

hello @kharvd , I have the same issue with the new vllm version 0.5.3post1 and I saw 2 other issues has been created due to that : #6734 and #6732

405B-FP8 base worked well for me on 8xH100 after changing the config (thanks @crowsonkb!), but at some point it started throwing CUDA errors:
hello @kharvd , I have the same issue with the new vllm version 0.5.3post1 and I saw 2 other issues has been created due to that : #6734 and #6732

I'm having a similar issue...
Error: Failed to initialize the TMA descriptor 700
[rank2]:[E ProcessGroupNCCL.cpp:1414] [PG 2 Rank 2] Process group watchdog thread terminated with exception: CUD
A error: an illegal memory access was encountered
``` etc.

Is hf_transfer still working? Weight download seems to be quite slow...

405B-FP8 base worked well for me on 8xH100 after changing the config (thanks @crowsonkb!), but at some point it started throwing CUDA errors:
hello @kharvd , I have the same issue with the new vllm version 0.5.3post1 and I saw 2 other issues has been created due to that : #6734 and #6732

I'm having a similar issue...
Error: Failed to initialize the TMA descriptor 700
[rank2]:[E ProcessGroupNCCL.cpp:1414] [PG 2 Rank 2] Process group watchdog thread terminated with exception: CUD
A error: an illegal memory access was encountered

etc.
Edited: If I don't disable vLLM's custom all reduce it works on prompts that previously failed. I haven't seen this error again yet.

Hello @crowsonkb , what do you mean by disanling vLLM’s custom all reduce ?

@RonanKMcGovern might be an issue with HF's ability to scale since everyone is rushing to get the weights for the first time

405B-FP8 base worked well for me on 8xH100 after changing the config (thanks @crowsonkb!), but at some point it started throwing CUDA errors:
hello @kharvd , I have the same issue with the new vllm version 0.5.3post1 and I saw 2 other issues has been created due to that : #6734 and #6732

I'm having a similar issue...
Error: Failed to initialize the TMA descriptor 700
[rank2]:[E ProcessGroupNCCL.cpp:1414] [PG 2 Rank 2] Process group watchdog thread terminated with exception: CUD
A error: an illegal memory access was encountered

etc.
Edited: If I don't disable vLLM's custom all reduce it works on prompts that previously failed. I haven't seen this error again yet.

I am trying to repro this issue, so any information on it would be helpful. Are you using chunked prefill? And could you share your collect-env.py output?

We saw the same TMA descriptor 700 error as well. Our configuration is:

TP=8 on H100.
Chunked prefill with 2048 chunk size.

The error can be avoided if I manually disabled CUTLASS FP8 kernel for fbgemm, but most importantly, we cannot easily reproduce this error as it happens randomly.

I also got this

[rank3]:[E ProcessGroupNCCL.cpp:1414] [PG 2 Rank 3] Process group watchdog thread terminated with exception: CUDA error: an illegal memory access was encountered

We saw the same TMA descriptor 700 error as well. Our configuration is:

TP=8 on H100.
Chunked prefill with 2048 chunk size.

The error can be avoided if I manually disabled CUTLASS FP8 kernel for fbgemm, but most importantly, we cannot easily reproduce this error as it happens randomly.

@comaniac How did you manually disable the CUTLASS FP8 kernel?

@comaniac How did you manually disable the CUTLASS FP8 kernel?

I just changed this to False
https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/layers/quantization/fbgemm_fp8.py#L148

Llama 3.1 405B base in fp8 is just generating !!!! over and over for me, is anyone else having this issue? I verified that 70B base works for me (but it's not in fp8).

I am similarly getting this issue with 70B GPTQ on ROCm with 4 x MI100

Having upgraded both transformers and vllm, I'm still getting ValueError: 'rope_scaling' must be a dictionary with two fields, 'type' and 'factor', got {'factor': 8.0, 'low_freq_factor': 1.0, 'high_freq_factor': 4.0, 'original_max_position_embeddings': 8192, 'rope_type': 'llama3'} . Does anyone have any more suggestions on how to fix?

I made #6767 to track my progress in debugging the issue @comaniac, @crowsonkb, and others are running into:

Having upgraded both transformers and vllm, I'm still getting ValueError: 'rope_scaling' must be a dictionary with two fields, 'type' and 'factor', got {'factor': 8.0, 'low_freq_factor': 1.0, 'high_freq_factor': 4.0, 'original_max_position_embeddings': 8192, 'rope_type': 'llama3'} . Does anyone have any more suggestions on how to fix?

this is probably a dependency issue with your local environment. can you try installing the latest version of vLLM into a clean venv / conda environment?

Having upgraded both transformers and vllm, I'm still getting ValueError: 'rope_scaling' must be a dictionary with two fields, 'type' and 'factor', got {'factor': 8.0, 'low_freq_factor': 1.0, 'high_freq_factor': 4.0, 'original_max_position_embeddings': 8192, 'rope_type': 'llama3'} . Does anyone have any more suggestions on how to fix?

this is probably a dependency issue with your local environment. can you try installing the latest version of vLLM into a clean venv / conda environment?

I was already working in a clean venv :/

Llama 3.1 uses a triton operation that fails on V100s, see https://github.com/vllm-project/vllm/issues/6723

Getting error
[rank0]:     raise ValueError(f"Unknown RoPE scaling type {scaling_type}")
[rank0]: ValueError: Unknown RoPE scaling type extended

Running config

      - args:
        - --model
        - meta-llama/Meta-Llama-3.1-8B
        - --rope-scaling
        - '{"type": "extended", "factor": 8.0}'
        - --gpu-memory-utilization
        - "0.95"
        - --enforce-eager
        - --dtype=half

Full logs

INFO 07-25 18:00:10 api_server.py:219] vLLM API server version 0.5.3.post1
INFO 07-25 18:00:10 api_server.py:220] args: Namespace(host=None, port=8000, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], model='meta-llama/Meta-Llama-3.1-8B', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, download_dir=None, load_format='auto', dtype='half', kv_cache_dtype='auto', quantization_param_path=None, max_model_len=None, guided_decoding_backend='outlines', distributed_executor_backend=None, worker_use_ray=False, pipeline_parallel_size=1, tensor_parallel_size=1, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=16, enable_prefix_caching=False, disable_sliding_window=False, use_v2_block_manager=False, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.95, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_seqs=256, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling={'type': 'extended', 'factor': 8.0}, rope_theta=None, enforce_eager=True, max_context_len_to_capture=None, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, enable_lora=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_model=None, num_speculative_tokens=None, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=None, qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, engine_use_ray=False, disable_log_requests=False, max_log_len=None)
INFO 07-25 18:00:11 config.py:68] Updating rope_scaling from {'factor': 8.0, 'low_freq_factor': 1.0, 'high_freq_factor': 4.0, 'original_max_position_embeddings': 8192, 'rope_type': 'llama3'} to {'type': 'extended', 'factor': 8.0}
WARNING 07-25 18:00:11 config.py:1425] Casting torch.bfloat16 to torch.float16.
WARNING 07-25 18:00:11 arg_utils.py:762] Chunked prefill is enabled by default for models with max_model_len > 32K. Currently, chunked prefill might not work with some features or models. If you encounter any issues, please disable chunked prefill by setting --enable-chunked-prefill=False.
INFO 07-25 18:00:11 config.py:806] Chunked prefill is enabled with max_num_batched_tokens=512.
INFO 07-25 18:00:11 llm_engine.py:176] Initializing an LLM engine (v0.5.3.post1) with config: model='meta-llama/Meta-Llama-3.1-8B', speculative_config=None, tokenizer='meta-llama/Meta-Llama-3.1-8B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, rope_scaling={'type': 'extended', 'factor': 8.0}, rope_theta=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=1048576, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None), seed=0, served_model_name=meta-llama/Meta-Llama-3.1-8B, use_v2_block_manager=False, enable_prefix_caching=False)
INFO 07-25 18:00:15 selector.py:151] Cannot use FlashAttention-2 backend for Volta and Turing GPUs.
INFO 07-25 18:00:15 selector.py:54] Using XFormers backend.
INFO 07-25 18:00:16 model_runner.py:680] Starting to load model meta-llama/Meta-Llama-3.1-8B...
[rank0]: Traceback (most recent call last):
[rank0]:   File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
[rank0]:     return _run_code(code, main_globals, None,
[rank0]:   File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
[rank0]:     exec(code, run_globals)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/api_server.py", line 317, in <module>
[rank0]:     run_server(args)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/api_server.py", line 231, in run_server
[rank0]:     if llm_engine is not None else AsyncLLMEngine.from_engine_args(
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 466, in from_engine_args
[rank0]:     engine = cls(
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 380, in __init__
[rank0]:     self.engine = self._init_engine(*args, **kwargs)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 547, in _init_engine
[rank0]:     return engine_class(*args, **kwargs)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/llm_engine.py", line 251, in __init__
[rank0]:     self.model_executor = executor_class(
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/executor/executor_base.py", line 47, in __init__
[rank0]:     self._init_executor()
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/executor/gpu_executor.py", line 36, in _init_executor
[rank0]:     self.driver_worker.load_model()
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker.py", line 139, in load_model
[rank0]:     self.model_runner.load_model()
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/worker/model_runner.py", line 682, in load_model
[rank0]:     self.model = get_model(model_config=self.model_config,
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/model_loader/__init__.py", line 21, in get_model
[rank0]:     return loader.load_model(model_config=model_config,
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/model_loader/loader.py", line 280, in load_model
[rank0]:     model = _initialize_model(model_config, self.load_config,
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/model_loader/loader.py", line 111, in _initialize_model
[rank0]:     return model_class(config=model_config.hf_config,
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/llama.py", line 384, in __init__
[rank0]:     self.model = LlamaModel(config,
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/llama.py", line 285, in __init__
[rank0]:     self.start_layer, self.end_layer, self.layers = make_layers(
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/utils.py", line 144, in make_layers
[rank0]:     [PPMissingLayer() for _ in range(start_layer)] + [
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/utils.py", line 145, in <listcomp>
[rank0]:     maybe_offload_to_cpu(layer_fn(prefix=f"{prefix}.{idx}"))
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/llama.py", line 287, in <lambda>
[rank0]:     lambda prefix: LlamaDecoderLayer(config=config,
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/llama.py", line 203, in __init__
[rank0]:     self.self_attn = LlamaAttention(
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/llama.py", line 151, in __init__
[rank0]:     self.rotary_emb = get_rope(
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/layers/rotary_embedding.py", line 860, in get_rope
[rank0]:     raise ValueError(f"Unknown RoPE scaling type {scaling_type}")
[rank0]: ValueError: Unknown RoPE scaling type extended

Anyone any suggestions to get pass this ?

--rope-scaling
- '{"type": "extended", "factor": 8.0}'
- --gpu-memory-utilization
- "0.95"
- --enforce-eager
- --dtype=half

You shouldn't need to pass any of these rope scaling. If you are OOM just limit the max context (although if you really have to, you can save a few GB by using enforce-eager). If you want you can copy some of the templates from github.com/trelisresearch/one-click-llms - I have tested those on A6000s, A100, H100.

In case of using transformer hf library  - updating helped.
Example:
# Load base model
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    quantization_config=bnb_config,
    device_map={"": 0},
    low_cpu_mem_usage=True,
    # rope_scaling={"type": "extended", "factor": 8.0}     <---  didn't had to use this
)

I upgraded my vllm version to use Llama 3.1 but it seems to break everything. I can't run the newer models and the older models which used to run also break now -
(ragenv) afkhan@sws-2h100-01:/NS/llm-1/work/afkhan$ python -m vllm.entrypoints.openai.api_server --model /NS/llm-1/nobackup/afkhan/Model_Saves/Nous-Capybara-34B --tensor-parallel-size 2 --trust-remote-code --max-seq-len-to-capture 32000 --max-model-len 32000
INFO 07-29 07:08:51 api_server.py:219] vLLM API server version 0.5.3.post1
INFO 07-29 07:08:51 api_server.py:220] args: Namespace(host=None, port=8000, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], model='/NS/llm-1/nobackup/afkhan/Model_Saves/Nous-Capybara-34B', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=True, download_dir=None, load_format='auto', dtype='auto', kv_cache_dtype='auto', quantization_param_path=None, max_model_len=32000, guided_decoding_backend='outlines', distributed_executor_backend=None, worker_use_ray=False, pipeline_parallel_size=1, tensor_parallel_size=2, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=16, enable_prefix_caching=False, disable_sliding_window=False, use_v2_block_manager=False, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.9, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_seqs=256, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, enforce_eager=False, max_context_len_to_capture=None, max_seq_len_to_capture=32000, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, enable_lora=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_model=None, num_speculative_tokens=None, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=None, qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, engine_use_ray=False, disable_log_requests=False, max_log_len=None)
INFO 07-29 07:08:51 config.py:715] Defaulting to use mp for distributed inference
INFO 07-29 07:08:51 llm_engine.py:176] Initializing an LLM engine (v0.5.3.post1) with config: model='/NS/llm-1/nobackup/afkhan/Model_Saves/Nous-Capybara-34B', speculative_config=None, tokenizer='/NS/llm-1/nobackup/afkhan/Model_Saves/Nous-Capybara-34B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32000, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=2, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None), seed=0, served_model_name=/NS/llm-1/nobackup/afkhan/Model_Saves/Nous-Capybara-34B, use_v2_block_manager=False, enable_prefix_caching=False)
WARNING 07-29 07:08:52 tokenizer.py:129] Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead.
INFO 07-29 07:08:52 custom_cache_manager.py:17] Setting Triton cache manager to: vllm.triton_utils.custom_cache_manager:CustomCacheManager
Traceback (most recent call last):
  File "/NS/llm-1/nobackup/afkhan/anaconda3/envs/ragenv/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/NS/llm-1/nobackup/afkhan/anaconda3/envs/ragenv/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/NS/llm-1/nobackup/afkhan/anaconda3/envs/ragenv/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 317, in <module>
    run_server(args)
  File "/NS/llm-1/nobackup/afkhan/anaconda3/envs/ragenv/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 231, in run_server
    if llm_engine is not None else AsyncLLMEngine.from_engine_args(
  File "/NS/llm-1/nobackup/afkhan/anaconda3/envs/ragenv/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 466, in from_engine_args
    engine = cls(
  File "/NS/llm-1/nobackup/afkhan/anaconda3/envs/ragenv/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 380, in __init__
    self.engine = self._init_engine(*args, **kwargs)
  File "/NS/llm-1/nobackup/afkhan/anaconda3/envs/ragenv/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 547, in _init_engine
    return engine_class(*args, **kwargs)
  File "/NS/llm-1/nobackup/afkhan/anaconda3/envs/ragenv/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 251, in __init__
    self.model_executor = executor_class(
  File "/NS/llm-1/nobackup/afkhan/anaconda3/envs/ragenv/lib/python3.10/site-packages/vllm/executor/multiproc_gpu_executor.py", line 201, in __init__
    super().__init__(*args, **kwargs)
  File "/NS/llm-1/nobackup/afkhan/anaconda3/envs/ragenv/lib/python3.10/site-packages/vllm/executor/distributed_gpu_executor.py", line 25, in __init__
    super().__init__(*args, **kwargs)
  File "/NS/llm-1/nobackup/afkhan/anaconda3/envs/ragenv/lib/python3.10/site-packages/vllm/executor/executor_base.py", line 47, in __init__
    self._init_executor()
  File "/NS/llm-1/nobackup/afkhan/anaconda3/envs/ragenv/lib/python3.10/site-packages/vllm/executor/multiproc_gpu_executor.py", line 121, in _init_executor
    self.driver_worker = self._create_worker(
  File "/NS/llm-1/nobackup/afkhan/anaconda3/envs/ragenv/lib/python3.10/site-packages/vllm/executor/gpu_executor.py", line 85, in _create_worker
    return create_worker(**self._get_create_worker_kwargs(
  File "/NS/llm-1/nobackup/afkhan/anaconda3/envs/ragenv/lib/python3.10/site-packages/vllm/executor/gpu_executor.py", line 20, in create_worker
    wrapper.init_worker(**kwargs)
  File "/NS/llm-1/nobackup/afkhan/anaconda3/envs/ragenv/lib/python3.10/site-packages/vllm/worker/worker_base.py", line 367, in init_worker
    self.worker = worker_class(*args, **kwargs)
  File "/NS/llm-1/nobackup/afkhan/anaconda3/envs/ragenv/lib/python3.10/site-packages/vllm/worker/worker.py", line 90, in __init__
    self.model_runner: GPUModelRunnerBase = ModelRunnerClass(
  File "/NS/llm-1/nobackup/afkhan/anaconda3/envs/ragenv/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 651, in __init__
    self.attn_backend = get_attn_backend(
  File "/NS/llm-1/nobackup/afkhan/anaconda3/envs/ragenv/lib/python3.10/site-packages/vllm/attention/selector.py", line 46, in get_attn_backend
    backend = which_attn_to_use(num_heads, head_size, num_kv_heads,
  File "/NS/llm-1/nobackup/afkhan/anaconda3/envs/ragenv/lib/python3.10/site-packages/vllm/attention/selector.py", line 149, in which_attn_to_use
    if current_platform.get_device_capability()[0] < 8:
  File "/NS/llm-1/nobackup/afkhan/anaconda3/envs/ragenv/lib/python3.10/site-packages/vllm/platforms/cuda.py", line 49, in get_device_capability
    return get_physical_device_capability(physical_device_id)
  File "/NS/llm-1/nobackup/afkhan/anaconda3/envs/ragenv/lib/python3.10/site-packages/vllm/platforms/cuda.py", line 20, in wrapper
    return fn(*args, **kwargs)
  File "/NS/llm-1/nobackup/afkhan/anaconda3/envs/ragenv/lib/python3.10/site-packages/vllm/platforms/cuda.py", line 31, in get_physical_device_capability
    return pynvml.nvmlDeviceGetCudaComputeCapability(handle)
AttributeError: module 'pynvml' has no attribute 'nvmlDeviceGetCudaComputeCapability'
INFO 07-29 07:08:53 multiproc_worker_utils.py:136] Terminating local vLLM worker processes

@aflah02  can you open a separate issue to report your environment? I believe your environment setup might be incorrect. This should never happen in a normal environment:

AttributeError: module 'pynvml' has no attribute 'nvmlDeviceGetCudaComputeCapability'

Deploy llama3.1 int4 awq got error below when have some concurrency requests:
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 399, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "/usr/local/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
    return await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/applications.py", line 123, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/cors.py", line 85, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 756, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 776, in app
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 297, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 77, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 72, in app
    response = await func(request)
  File "/usr/local/lib/python3.10/site-packages/fastapi/routing.py", line 278, in app
    raw_response = await run_endpoint_function(
  File "/usr/local/lib/python3.10/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
    return await dependant.call(**values)
  File "/usr/local/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 129, in create_chat_completion
    generator = await openai_serving_chat.create_chat_completion(
  File "/usr/local/lib/python3.10/site-packages/vllm/entrypoints/openai/serving_chat.py", line 195, in create_chat_completion
    return await self.chat_completion_full_generator(
  File "/usr/local/lib/python3.10/site-packages/vllm/entrypoints/openai/serving_chat.py", line 404, in chat_completion_full_generator
    async for res in result_generator:
  File "/usr/local/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 772, in generate
    async for output in self._process_request(
  File "/usr/local/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 888, in _process_request
    raise e
  File "/usr/local/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 884, in _process_request
    async for request_output in stream:
  File "/usr/local/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 93, in __anext__
    raise result
  File "/usr/local/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 399, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "/usr/local/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
    return await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/applications.py", line 123, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/cors.py", line 85, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 756, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 776, in app
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 297, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 77, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 72, in app
    response = await func(request)
  File "/usr/local/lib/python3.10/site-packages/fastapi/routing.py", line 278, in app
    raw_response = await run_endpoint_function(
  File "/usr/local/lib/python3.10/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
    return await dependant.call(**values)
  File "/usr/local/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 129, in create_chat_completion
    generator = await openai_serving_chat.create_chat_completion(
  File "/usr/local/lib/python3.10/site-packages/vllm/entrypoints/openai/serving_chat.py", line 195, in create_chat_completion
    return await self.chat_completion_full_generator(
  File "/usr/local/lib/python3.10/site-packages/vllm/entrypoints/openai/serving_chat.py", line 404, in chat_completion_full_generator
    async for res in result_generator:
  File "/usr/local/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 772, in generate
    async for output in self._process_request(
  File "/usr/local/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 888, in _process_request
    raise e
  File "/usr/local/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 884, in _process_request
    async for request_output in stream:
  File "/usr/local/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 93, in __anext__
    raise result
  File "/usr/local/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 46, in _log_task_completion
    return_value = task.result()
  File "/usr/local/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 637, in run_engine_loop
    result = task.result()
  File "/usr/local/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 580, in engine_step
    request_outputs = await self.engine.step_async(virtual_engine)
  File "/usr/local/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 253, in step_async
    output = await self.model_executor.execute_model_async(
  File "/usr/local/lib/python3.10/site-packages/vllm/executor/distributed_gpu_executor.py", line 175, in execute_model_async
    return await self._driver_execute_model_async(execute_model_req)
  File "/usr/local/lib/python3.10/site-packages/vllm/executor/multiproc_gpu_executor.py", line 210, in _driver_execute_model_async
    return await self.driver_exec_model(execute_model_req)
  File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.10/site-packages/vllm/worker/worker_base.py", line 272, in execute_model
    output = self.model_runner.execute_model(
  File "/output/.pylibs/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 1314, in execute_model
    hidden_or_intermediate_states = model_executable(
  File "/output/.pylibs/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/output/.pylibs/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/vllm/model_executor/models/llama.py", line 422, in forward
    model_output = self.model(input_ids, positions, kv_caches,
  File "/output/.pylibs/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/output/.pylibs/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/vllm/model_executor/models/llama.py", line 322, in forward
    hidden_states, residual = layer(
  File "/output/.pylibs/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/output/.pylibs/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/vllm/model_executor/models/llama.py", line 255, in forward
    hidden_states = self.mlp(hidden_states)
  File "/output/.pylibs/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/output/.pylibs/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/vllm/model_executor/models/llama.py", line 87, in forward
    gate_up, _ = self.gate_up_proj(x)
  File "/output/.pylibs/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/output/.pylibs/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/vllm/model_executor/layers/linear.py", line 330, in forward
    output_parallel = self.quant_method.apply(self, input_, bias)
  File "/usr/local/lib/python3.10/site-packages/vllm/model_executor/layers/quantization/awq.py", line 169, in apply
    out = ops.awq_dequantize(qweight, scales, qzeros, 0, 0, 0)
  File "/usr/local/lib/python3.10/site-packages/vllm/_custom_ops.py", line 34, in wrapper
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/vllm/_custom_ops.py", line 186, in awq_dequantize
    return torch.ops._C.awq_dequantize(qweight, scales, zeros, split_k_iters,
  File "/output/.pylibs/lib/python3.10/site-packages/torch/_ops.py", line 854, in __call__
    return self_._op(*args, **(kwargs or {}))
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 416.00 MiB. GPU
INFO:     <IP_ADDRESS>:41576 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 399, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "/usr/local/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
    return await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/applications.py", line 123, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/cors.py", line 85, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 756, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 776, in app
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 297, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 77, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 72, in app
    response = await func(request)
  File "/usr/local/lib/python3.10/site-packages/fastapi/routing.py", line 278, in app
    raw_response = await run_endpoint_function(
  File "/usr/local/lib/python3.10/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
    return await dependant.call(**values)
  File "/usr/local/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 129, in create_chat_completion
    generator = await openai_serving_chat.create_chat_completion(
  File "/usr/local/lib/python3.10/site-packages/vllm/entrypoints/openai/serving_chat.py", line 195, in create_chat_completion
    return await self.chat_completion_full_generator(
  File "/usr/local/lib/python3.10/site-packages/vllm/entrypoints/openai/serving_chat.py", line 404, in chat_completion_full_generator
    async for res in result_generator:
  File "/usr/local/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 772, in generate
    async for output in self._process_request(
  File "/usr/local/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 888, in _process_request
    raise e
  File "/usr/local/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 884, in _process_request
    async for request_output in stream:
  File "/usr/local/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 93, in __anext__
    raise result
  File "/usr/local/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 399, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "/usr/local/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
    return await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/applications.py", line 123, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/cors.py", line 85, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 756, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 776, in app
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 297, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 77, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 72, in app
    response = await func(request)
  File "/usr/local/lib/python3.10/site-packages/fastapi/routing.py", line 278, in app
    raw_response = await run_endpoint_function(
  File "/usr/local/lib/python3.10/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
    return await dependant.call(**values)
  File "/usr/local/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 129, in create_chat_completion
    generator = await openai_serving_chat.create_chat_completion(
  File "/usr/local/lib/python3.10/site-packages/vllm/entrypoints/openai/serving_chat.py", line 195, in create_chat_completion
    return await self.chat_completion_full_generator(
  File "/usr/local/lib/python3.10/site-packages/vllm/entrypoints/openai/serving_chat.py", line 404, in chat_completion_full_generator
    async for res in result_generator:
  File "/usr/local/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 772, in generate
    async for output in self._process_request(
  File "/usr/local/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 888, in _process_request
    raise e
  File "/usr/local/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 884, in _process_request
    async for request_output in stream:
  File "/usr/local/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 93, in __anext__
    raise result
  File "/usr/local/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 46, in _log_task_completion
    return_value = task.result()
  File "/usr/local/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 637, in run_engine_loop
    result = task.result()
  File "/usr/local/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 580, in engine_step
    request_outputs = await self.engine.step_async(virtual_engine)
  File "/usr/local/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 253, in step_async
    output = await self.model_executor.execute_model_async(
  File "/usr/local/lib/python3.10/site-packages/vllm/executor/distributed_gpu_executor.py", line 175, in execute_model_async
    return await self._driver_execute_model_async(execute_model_req)
  File "/usr/local/lib/python3.10/site-packages/vllm/executor/multiproc_gpu_executor.py", line 210, in _driver_execute_model_async
    return await self.driver_exec_model(execute_model_req)
  File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.10/site-packages/vllm/worker/worker_base.py", line 272, in execute_model
    output = self.model_runner.execute_model(
  File "/output/.pylibs/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 1314, in execute_model
    hidden_or_intermediate_states = model_executable(
  File "/output/.pylibs/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/output/.pylibs/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/vllm/model_executor/models/llama.py", line 422, in forward
    model_output = self.model(input_ids, positions, kv_caches,
  File "/output/.pylibs/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/output/.pylibs/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/vllm/model_executor/models/llama.py", line 322, in forward
    hidden_states, residual = layer(
  File "/output/.pylibs/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/output/.pylibs/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/vllm/model_executor/models/llama.py", line 255, in forward
    hidden_states = self.mlp(hidden_states)
  File "/output/.pylibs/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/output/.pylibs/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/vllm/model_executor/models/llama.py", line 87, in forward
    gate_up, _ = self.gate_up_proj(x)
  File "/output/.pylibs/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/output/.pylibs/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/vllm/model_executor/layers/linear.py", line 330, in forward
    output_parallel = self.quant_method.apply(self, input_, bias)
  File "/usr/local/lib/python3.10/site-packages/vllm/model_executor/layers/quantization/awq.py", line 169, in apply
    out = ops.awq_dequantize(qweight, scales, qzeros, 0, 0, 0)
  File "/usr/local/lib/python3.10/site-packages/vllm/_custom_ops.py", line 34, in wrapper
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/vllm/_custom_ops.py", line 186, in awq_dequantize
    return torch.ops._C.awq_dequantize(qweight, scales, zeros, split_k_iters,
  File "/output/.pylibs/lib/python3.10/site-packages/torch/_ops.py", line 854, in __call__
    return self_._op(*args, **(kwargs or {}))
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 416.00 MiB. GPU

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 399, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "/usr/local/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
    return await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/applications.py", line 123, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/cors.py", line 85, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 756, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 776, in app
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 297, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 77, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 72, in app
    response = await func(request)
  File "/usr/local/lib/python3.10/site-packages/fastapi/routing.py", line 278, in app
    raw_response = await run_endpoint_function(
  File "/usr/local/lib/python3.10/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
    return await dependant.call(**values)
  File "/usr/local/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 129, in create_chat_completion
    generator = await openai_serving_chat.create_chat_completion(
  File "/usr/local/lib/python3.10/site-packages/vllm/entrypoints/openai/serving_chat.py", line 195, in create_chat_completion
    return await self.chat_completion_full_generator(
  File "/usr/local/lib/python3.10/site-packages/vllm/entrypoints/openai/serving_chat.py", line 404, in chat_completion_full_generator
    async for res in result_generator:
  File "/usr/local/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 772, in generate
    async for output in self._process_request(
  File "/usr/local/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 873, in _process_request
    stream = await self.add_request(
  File "/usr/local/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 676, in add_request
    self.start_background_loop()
  File "/usr/local/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 516, in start_background_loop
    raise AsyncEngineDeadError(
vllm.engine.async_llm_engine.AsyncEngineDeadError: Background loop has errored already.

The deploy script looks like this:
vllm serve Meta-Llama-3___1-405B-Instruct-AWQ-INT4 \
    --port 8080 \
    --host <IP_ADDRESS> \
    --served-model-name llama-3.1-405b \
    --trust-remote-code \
    --max-model-len 8192 -q awq \
    --tensor-parallel-size 8 \
    --enable-chunked-prefill=false \
    --disable-log-requests \
    --disable-log-stats

The gpu is A6000.

@aflah02 can you open a separate issue to report your environment? I believe your environment setup might be incorrect. This should never happen in a normal environment:

AttributeError: module 'pynvml' has no attribute 'nvmlDeviceGetCudaComputeCapability'

Hi
I just created a new env with Python 3.11
I only installed vllm with pip install vllm
Getting this error now -
(vllm_env) afkhan@sws-2h100-01:/NS/llm-1/work/afkhan$ python -m vllm.entrypoints.openai.api_server --model /NS/llm-1/nobackup/shared/huggingface_cache/hub/models--meta-llama--Meta-Llama-3.1-70B-Instruct --tensor-parallel-size 2  --max-seq-len-to-capture 32000 --max-model-len 32000

Bus error

(vllm_env) afkhan@sws-2h100-01:/NS/llm-1/work/afkhan$ python -m vllm.entrypoints.openai.api_server --model /NS/llm-1/nobackup/afkhan/Model_Saves/dolphin-2.6-mixtral-8x7b --tensor-parallel-size 2  --max-seq-len-to-capture 32000 --max-model-len 32000

Bus error

I have updated vllm and transformers to the latest version and now when I load an instance of VLLM class. I get the following error:
[rank0]: ValueError: No available memory for the cache blocks. Try increasing `gpu_memory_utilization` when initializing the engine.```

Got this:
UNAVAILABLE: Internal: RuntimeError: Expected a.dtype() == torch::kInt8 to be true, but got false. (Could this error message be improved? If so, please report an enhancement request .
I'm using vllm 0.5.3-post1 and Meta-Llama-3.1-405B-Instruct-FP8

Do you run 405B with cuda 11.8? The cuda version should be 12.X to support FP8

gptq_marlin not found.

QQ, what is good choice for max-model-len for 8*H100 for 405B?
Should I just find out? If anyone has suggestions welcome. That is, 128k is default, I presume.
But should I go for 16k, 32k, 64k, 128k? What can 8*H100 support at full context length?

@pseudotensor
did you got answer for this?
did you tested with 64K or more context length on 8*H100 ?

I keep getting the following error when launching Llama 3.1 70B/70B-Instruct:
(VllmWorkerProcess pid=1118) ERROR 08-06 14:58:39 multiproc_worker_utils.py:226] Exception in worker VllmWorkerProcess while processing method initialize_cache: The model's max seq len (131072) is larger than the maximum number of tokens that can be stored in KV cache (106384). Try increasing `gpu_memory_utilization` or decreasing `max_model_len` when initializing the engine., Traceback (most recent call last):
(VllmWorkerProcess pid=1118) ERROR 08-06 14:58:39 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.10/site-packages/vllm/executor/multiproc_worker_utils.py", line 223, in _run_worker_process
(VllmWorkerProcess pid=1118) ERROR 08-06 14:58:39 multiproc_worker_utils.py:226]     output = executor(*args, **kwargs)
(VllmWorkerProcess pid=1118) ERROR 08-06 14:58:39 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.10/site-packages/vllm/worker/worker.py", line 214, in initialize_cache
(VllmWorkerProcess pid=1118) ERROR 08-06 14:58:39 multiproc_worker_utils.py:226]     raise_if_cache_size_invalid(num_gpu_blocks,
(VllmWorkerProcess pid=1118) ERROR 08-06 14:58:39 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.10/site-packages/vllm/worker/worker.py", line 374, in raise_if_cache_size_invalid
(VllmWorkerProcess pid=1118) ERROR 08-06 14:58:39 multiproc_worker_utils.py:226]     raise ValueError(
(VllmWorkerProcess pid=1118) ERROR 08-06 14:58:39 multiproc_worker_utils.py:226] ValueError: The model's max seq len (131072) is larger than the maximum number of tokens that can be stored in KV cache (106384). Try increasing `gpu_memory_utilization` or decreasing `max_model_len` when initializing the engine.
(VllmWorkerProcess pid=1118) ERROR 08-06 14:58:39 multiproc_worker_utils.py:226]

I'm launching the model using 4 a40 GPUs, and the same hardwares have no issues hosting Llama 3 70B. I first noticed this error with vllm v0.5.3.post1, and it persists with v0.5.4. Every time I try to launch one of these models the KV cache size is different, so fixing max_model_len to a smaller number wouldn't work. I've already increased gpu_memory_utilization to 0.95, and setting this parameter to 1 seems to cause CUDA out of memory errors.

@XkunW llama 3.1 has longer context length (128k by default). so this is expected.

@XkunW llama 3.1 has longer context length (128k by default). so this is expected.

So I should set context length to a number small enough that it should always be smaller than KV cache size?

yes, as the error suggests, you can add --max_model_len 106000

ERROR:    Exception in ASGI application

Exception Group Traceback (most recent call last):
|   File "/home/datagrand/miniconda3/envs/vllm_new_hw/lib/python3.12/site-packages/starlette/_utils.py", line 87, in collapse_excgroups
|     yield
|   File "/home/datagrand/miniconda3/envs/vllm_new_hw/lib/python3.12/site-packages/starlette/middleware/base.py", line 190, in call
|     async with anyio.create_task_group() as task_group:
|   File "/home/datagrand/miniconda3/envs/vllm_new_hw/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 680, in aexit
|     raise BaseExceptionGroup(
| ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
+-+---------------- 1 ----------------
| Traceback (most recent call last):
|   File "/home/datagrand/miniconda3/envs/vllm_new_hw/lib/python3.12/site-packages/uvicorn/protocols/http/httptools_impl.py", line 399, in run_asgi
|     result = await app(  # type: ignore[func-returns-value]
|              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|   File "/home/datagrand/miniconda3/envs/vllm_new_hw/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in call
|     return await self.app(scope, receive, send)
|            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|   File "/home/datagrand/miniconda3/envs/vllm_new_hw/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in call
|     await super().call(scope, receive, send)
|   File "/home/datagrand/miniconda3/envs/vllm_new_hw/lib/python3.12/site-packages/starlette/applications.py", line 123, in call
|     await self.middleware_stack(scope, receive, send)
|   File "/home/datagrand/miniconda3/envs/vllm_new_hw/lib/python3.12/site-packages/starlette/middleware/errors.py", line 186, in call
|     raise exc
|   File "/home/datagrand/miniconda3/envs/vllm_new_hw/lib/python3.12/site-packages/starlette/middleware/errors.py", line 164, in call
|     await self.app(scope, receive, _send)
|   File "/home/datagrand/miniconda3/envs/vllm_new_hw/lib/python3.12/site-packages/starlette/middleware/base.py", line 189, in call
|     with collapse_excgroups():
|   File "/home/datagrand/miniconda3/envs/vllm_new_hw/lib/python3.12/contextlib.py", line 158, in exit
|     self.gen.throw(value)
|   File "/home/datagrand/miniconda3/envs/vllm_new_hw/lib/python3.12/site-packages/starlette/_utils.py", line 93, in collapse_excgroups
|     raise exc
|   File "/home/datagrand/miniconda3/envs/vllm_new_hw/lib/python3.12/site-packages/starlette/middleware/base.py", line 191, in call
|     response = await self.dispatch_func(request, call_next)
|                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|   File "/home/datagrand/miniconda3/envs/vllm_new_hw/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 260, in authentication
|     return await call_next(request)
|            ^^^^^^^^^^^^^^^^^^^^^^^^
|   File "/home/datagrand/miniconda3/envs/vllm_new_hw/lib/python3.12/site-packages/starlette/middleware/base.py", line 165, in call_next
|     raise app_exc
|   File "/home/datagrand/miniconda3/envs/vllm_new_hw/lib/python3.12/site-packages/starlette/middleware/base.py", line 151, in coro
|     await self.app(scope, receive_or_disconnect, send_no_error)
|   File "/home/datagrand/miniconda3/envs/vllm_new_hw/lib/python3.12/site-packages/starlette/middleware/cors.py", line 85, in call
|     await self.app(scope, receive, send)
|   File "/home/datagrand/miniconda3/envs/vllm_new_hw/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 65, in call
|     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
|   File "/home/datagrand/miniconda3/envs/vllm_new_hw/lib/python3.12/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
|     raise exc
|   File "/home/datagrand/miniconda3/envs/vllm_new_hw/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
|     await app(scope, receive, sender)
|   File "/home/datagrand/miniconda3/envs/vllm_new_hw/lib/python3.12/site-packages/starlette/routing.py", line 756, in call
|     await self.middleware_stack(scope, receive, send)
|   File "/home/datagrand/miniconda3/envs/vllm_new_hw/lib/python3.12/site-packages/starlette/routing.py", line 776, in app
|     await route.handle(scope, receive, send)
|   File "/home/datagrand/miniconda3/envs/vllm_new_hw/lib/python3.12/site-packages/starlette/routing.py", line 297, in handle
|     await self.app(scope, receive, send)
|   File "/home/datagrand/miniconda3/envs/vllm_new_hw/lib/python3.12/site-packages/starlette/routing.py", line 77, in app
|     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
|   File "/home/datagrand/miniconda3/envs/vllm_new_hw/lib/python3.12/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
|     raise exc
|   File "/home/datagrand/miniconda3/envs/vllm_new_hw/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
|     await app(scope, receive, sender)
|   File "/home/datagrand/miniconda3/envs/vllm_new_hw/lib/python3.12/site-packages/starlette/routing.py", line 72, in app
|     response = await func(request)
|                ^^^^^^^^^^^^^^^^^^^
|   File "/home/datagrand/miniconda3/envs/vllm_new_hw/lib/python3.12/site-packages/fastapi/routing.py", line 278, in app
|     raw_response = await run_endpoint_function(
|                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|   File "/home/datagrand/miniconda3/envs/vllm_new_hw/lib/python3.12/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
|     return await dependant.call(**values)
|            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|   File "/home/datagrand/miniconda3/envs/vllm_new_hw/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 189, in create_chat_completion
|     generator = await openai_serving_chat.create_chat_completion(
|                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|   File "/home/datagrand/miniconda3/envs/vllm_new_hw/lib/python3.12/site-packages/vllm/entrypoints/openai/serving_chat.py", line 185, in create_chat_completion
|     return await self.chat_completion_full_generator(
|            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|   File "/home/datagrand/miniconda3/envs/vllm_new_hw/lib/python3.12/site-packages/vllm/entrypoints/openai/serving_chat.py", line 436, in chat_completion_full_generator
|     async for res in result_generator:
|   File "/home/datagrand/miniconda3/envs/vllm_new_hw/lib/python3.12/site-packages/vllm/entrypoints/openai/rpc/client.py", line 216, in generate
|     raise request_output
| vllm.engine.async_llm_engine.AsyncEngineDeadError: Background loop has errored already.
+------------------------------------

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/datagrand/miniconda3/envs/vllm_new_hw/lib/python3.12/site-packages/uvicorn/protocols/http/httptools_impl.py", line 399, in run_asgi
result = await app(  # type: ignore[func-returns-value]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/datagrand/miniconda3/envs/vllm_new_hw/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in call
return await self.app(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/datagrand/miniconda3/envs/vllm_new_hw/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in call
await super().call(scope, receive, send)
File "/home/datagrand/miniconda3/envs/vllm_new_hw/lib/python3.12/site-packages/starlette/applications.py", line 123, in call
await self.middleware_stack(scope, receive, send)
File "/home/datagrand/miniconda3/envs/vllm_new_hw/lib/python3.12/site-packages/starlette/middleware/errors.py", line 186, in call
raise exc
File "/home/datagrand/miniconda3/envs/vllm_new_hw/lib/python3.12/site-packages/starlette/middleware/errors.py", line 164, in call
await self.app(scope, receive, _send)
File "/home/datagrand/miniconda3/envs/vllm_new_hw/lib/python3.12/site-packages/starlette/middleware/base.py", line 189, in call
with collapse_excgroups():
File "/home/datagrand/miniconda3/envs/vllm_new_hw/lib/python3.12/contextlib.py", line 158, in exit
self.gen.throw(value)
File "/home/datagrand/miniconda3/envs/vllm_new_hw/lib/python3.12/site-packages/starlette/_utils.py", line 93, in collapse_excgroups
raise exc
File "/home/datagrand/miniconda3/envs/vllm_new_hw/lib/python3.12/site-packages/starlette/middleware/base.py", line 191, in call
response = await self.dispatch_func(request, call_next)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/datagrand/miniconda3/envs/vllm_new_hw/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 260, in authentication
return await call_next(request)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/datagrand/miniconda3/envs/vllm_new_hw/lib/python3.12/site-packages/starlette/middleware/base.py", line 165, in call_next
raise app_exc
File "/home/datagrand/miniconda3/envs/vllm_new_hw/lib/python3.12/site-packages/starlette/middleware/base.py", line 151, in coro
await self.app(scope, receive_or_disconnect, send_no_error)
File "/home/datagrand/miniconda3/envs/vllm_new_hw/lib/python3.12/site-packages/starlette/middleware/cors.py", line 85, in call
await self.app(scope, receive, send)
File "/home/datagrand/miniconda3/envs/vllm_new_hw/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 65, in call
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "/home/datagrand/miniconda3/envs/vllm_new_hw/lib/python3.12/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
raise exc
File "/home/datagrand/miniconda3/envs/vllm_new_hw/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
await app(scope, receive, sender)
File "/home/datagrand/miniconda3/envs/vllm_new_hw/lib/python3.12/site-packages/starlette/routing.py", line 756, in call
await self.middleware_stack(scope, receive, send)
File "/home/datagrand/miniconda3/envs/vllm_new_hw/lib/python3.12/site-packages/starlette/routing.py", line 776, in app
await route.handle(scope, receive, send)
File "/home/datagrand/miniconda3/envs/vllm_new_hw/lib/python3.12/site-packages/starlette/routing.py", line 297, in handle
await self.app(scope, receive, send)
File "/home/datagrand/miniconda3/envs/vllm_new_hw/lib/python3.12/site-packages/starlette/routing.py", line 77, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "/home/datagrand/miniconda3/envs/vllm_new_hw/lib/python3.12/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
raise exc
File "/home/datagrand/miniconda3/envs/vllm_new_hw/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
await app(scope, receive, sender)
File "/home/datagrand/miniconda3/envs/vllm_new_hw/lib/python3.12/site-packages/starlette/routing.py", line 72, in app
response = await func(request)
^^^^^^^^^^^^^^^^^^^
File "/home/datagrand/miniconda3/envs/vllm_new_hw/lib/python3.12/site-packages/fastapi/routing.py", line 278, in app
raw_response = await run_endpoint_function(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/datagrand/miniconda3/envs/vllm_new_hw/lib/python3.12/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
return await dependant.call(**values)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/datagrand/miniconda3/envs/vllm_new_hw/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 189, in create_chat_completion
generator = await openai_serving_chat.create_chat_completion(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/datagrand/miniconda3/envs/vllm_new_hw/lib/python3.12/site-packages/vllm/entrypoints/openai/serving_chat.py", line 185, in create_chat_completion
return await self.chat_completion_full_generator(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/datagrand/miniconda3/envs/vllm_new_hw/lib/python3.12/site-packages/vllm/entrypoints/openai/serving_chat.py", line 436, in chat_completion_full_generator
async for res in result_generator:
File "/home/datagrand/miniconda3/envs/vllm_new_hw/lib/python3.12/site-packages/vllm/entrypoints/openai/rpc/client.py", line 216, in generate
raise request_output
vllm.engine.async_llm_engine.AsyncEngineDeadError: Background loop has errored already.
I got this error,
env:
vllm 0.5.4+cu118
command:
vllm serve /mnt/models/Meta-Llama-3.1-70B-Instruct --tensor-parallel-size 8 --api-key token-abc123 --host <IP_ADDRESS> --port 88

@youkaichao I am having an issue when trying to run llama 3.1 405b on 8 nodes each with 4 64Gb A100 gpus, you can see the specific here https://leonardo-supercomputer.cineca.eu/hpc-system/
This is the full stack trace, it appears to hang when creating cuda graphs, though this might not be the case, any chance you could help?
/var/spool/slurmd/job6730439/slurm_script: line 14: /leonardo_scratch/large/userexternal/gpuccett/Repos/llm_newest/conda_venv/bin/conda: No such file or directory
IP Head: lrdn1699:6379
STARTING HEAD at lrdn1699
STARTING WORKER 1 at lrdn1739
IP Head: lrdn1699:6379
Node: lrdn1739 IP Local: <IP_ADDRESS>
STARTING WORKER 2 at lrdn1760
IP Head: lrdn1699:6379
Node: lrdn1760 IP Local: <IP_ADDRESS>
STARTING WORKER 3 at lrdn1766
IP Head: lrdn1699:6379
Node: lrdn1766 IP Local: <IP_ADDRESS>
STARTING WORKER 4 at lrdn1778
IP Head: lrdn1699:6379
Node: lrdn1778 IP Local: <IP_ADDRESS>
STARTING WORKER 5 at lrdn1788
IP Head: lrdn1699:6379
Node: lrdn1788 IP Local: <IP_ADDRESS>
STARTING WORKER 6 at lrdn1813
IP Head: lrdn1699:6379
Node: lrdn1813 IP Local: <IP_ADDRESS>
STARTING WORKER 7 at lrdn1855
IP Head: lrdn1699:6379
Node: lrdn1855 IP Local: <IP_ADDRESS>
STARTING python command
2024-08-07 10:14:26,200 - INFO - Note: NumExpr detected 32 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2024-08-07 10:14:26,200 - INFO - Note: NumExpr detected 32 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2024-08-07 10:14:26,200 - INFO - Note: NumExpr detected 32 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2024-08-07 10:14:26,200 - INFO - Note: NumExpr detected 32 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2024-08-07 10:14:26,201 - INFO - NumExpr defaulting to 8 threads.
2024-08-07 10:14:26,200 - INFO - Note: NumExpr detected 32 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2024-08-07 10:14:26,200 - INFO - Note: NumExpr detected 32 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2024-08-07 10:14:26,200 - INFO - Note: NumExpr detected 32 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2024-08-07 10:14:26,200 - INFO - Note: NumExpr detected 32 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2024-08-07 10:14:26,200 - INFO - NumExpr defaulting to 8 threads.
2024-08-07 10:14:26,200 - INFO - NumExpr defaulting to 8 threads.
2024-08-07 10:14:26,200 - INFO - NumExpr defaulting to 8 threads.
2024-08-07 10:14:26,200 - INFO - NumExpr defaulting to 8 threads.
2024-08-07 10:14:26,200 - INFO - NumExpr defaulting to 8 threads.
2024-08-07 10:14:26,201 - INFO - NumExpr defaulting to 8 threads.
2024-08-07 10:14:26,200 - INFO - NumExpr defaulting to 8 threads.
2024-08-07 10:14:30,579	INFO usage_lib.py:443 -- Usage stats collection is disabled.
2024-08-07 10:14:30,579	INFO scripts.py:763 -- Local node IP: lrdn1699
2024-08-07 10:14:30,582	INFO scripts.py:945 -- Local node IP: <IP_ADDRESS>
2024-08-07 10:14:30,585	INFO scripts.py:945 -- Local node IP: <IP_ADDRESS>
2024-08-07 10:14:30,585	INFO scripts.py:945 -- Local node IP: <IP_ADDRESS>
2024-08-07 10:14:30,586	INFO scripts.py:945 -- Local node IP: <IP_ADDRESS>
2024-08-07 10:14:30,585	INFO scripts.py:945 -- Local node IP: <IP_ADDRESS>
2024-08-07 10:14:30,586	INFO scripts.py:945 -- Local node IP: <IP_ADDRESS>
2024-08-07 10:14:30,586	INFO scripts.py:945 -- Local node IP: <IP_ADDRESS>
2024-08-07 10:14:35,102	SUCC scripts.py:800 -- --------------------
2024-08-07 10:14:35,102	SUCC scripts.py:801 -- Ray runtime started.
2024-08-07 10:14:35,102	SUCC scripts.py:802 -- --------------------
2024-08-07 10:14:35,102	INFO scripts.py:804 -- Next steps
2024-08-07 10:14:35,102	INFO scripts.py:807 -- To add another node to this Ray cluster, run
2024-08-07 10:14:35,102	INFO scripts.py:810 --   ray start --address='lrdn1699:6379'
2024-08-07 10:14:35,102	INFO scripts.py:819 -- To connect to this Ray cluster:
2024-08-07 10:14:35,102	INFO scripts.py:821 -- import ray
2024-08-07 10:14:35,102	INFO scripts.py:822 -- ray.init(_node_ip_address='lrdn1699')
2024-08-07 10:14:35,102	INFO scripts.py:853 -- To terminate the Ray runtime, run
2024-08-07 10:14:35,102	INFO scripts.py:854 --   ray stop
2024-08-07 10:14:35,102	INFO scripts.py:857 -- To view the status of the cluster, use
2024-08-07 10:14:35,102	INFO scripts.py:858 --   ray status
2024-08-07 10:14:35,103	INFO scripts.py:971 -- --block
2024-08-07 10:14:35,103	INFO scripts.py:972 -- This command will now block forever until terminated by a signal.
2024-08-07 10:14:35,103	INFO scripts.py:975 -- Running subprocesses are monitored and a message will be printed if any of them terminate unexpectedly. Subprocesses exit with SIGTERM will be treated as graceful, thus NOT reported.
[2024-08-07 10:14:35,261 I<PHONE_NUMBER>34] global_state_accessor.cc:432: This node has an IP address of <IP_ADDRESS>, but we cannot find a local Raylet with the same address. This can happen when you connect to the Ray cluster with a different IP address or when connecting to a container.
[2024-08-07 10:14:35,261 I<PHONE_NUMBER>03] global_state_accessor.cc:432: This node has an IP address of <IP_ADDRESS>, but we cannot find a local Raylet with the same address. This can happen when you connect to the Ray cluster with a different IP address or when connecting to a container.
[2024-08-07 10:14:35,261 I<PHONE_NUMBER>95] global_state_accessor.cc:432: This node has an IP address of <IP_ADDRESS>, but we cannot find a local Raylet with the same address. This can happen when you connect to the Ray cluster with a different IP address or when connecting to a container.
[2024-08-07 10:14:35,261 I<PHONE_NUMBER>64] global_state_accessor.cc:432: This node has an IP address of <IP_ADDRESS>, but we cannot find a local Raylet with the same address. This can happen when you connect to the Ray cluster with a different IP address or when connecting to a container.
[2024-08-07 10:14:35,261 I<PHONE_NUMBER>56] global_state_accessor.cc:432: This node has an IP address of <IP_ADDRESS>, but we cannot find a local Raylet with the same address. This can happen when you connect to the Ray cluster with a different IP address or when connecting to a container.
[2024-08-07 10:14:35,261 I<PHONE_NUMBER>30] global_state_accessor.cc:432: This node has an IP address of <IP_ADDRESS>, but we cannot find a local Raylet with the same address. This can happen when you connect to the Ray cluster with a different IP address or when connecting to a container.
[2024-08-07 10:14:35,261 I<PHONE_NUMBER>49] global_state_accessor.cc:432: This node has an IP address of <IP_ADDRESS>, but we cannot find a local Raylet with the same address. This can happen when you connect to the Ray cluster with a different IP address or when connecting to a container.
2024-08-07 10:14:35,296	SUCC scripts.py:958 -- --------------------
2024-08-07 10:14:35,296	SUCC scripts.py:958 -- --------------------
2024-08-07 10:14:35,296	SUCC scripts.py:959 -- Ray runtime started.
2024-08-07 10:14:35,296	SUCC scripts.py:959 -- Ray runtime started.
2024-08-07 10:14:35,296	SUCC scripts.py:960 -- --------------------
2024-08-07 10:14:35,296	SUCC scripts.py:960 -- --------------------
2024-08-07 10:14:35,297	INFO scripts.py:962 -- To terminate the Ray runtime, run
2024-08-07 10:14:35,297	INFO scripts.py:962 -- To terminate the Ray runtime, run
2024-08-07 10:14:35,297	INFO scripts.py:963 --   ray stop
2024-08-07 10:14:35,297	INFO scripts.py:963 --   ray stop
2024-08-07 10:14:35,297	INFO scripts.py:971 -- --block
2024-08-07 10:14:35,297	INFO scripts.py:971 -- --block
2024-08-07 10:14:35,297	INFO scripts.py:972 -- This command will now block forever until terminated by a signal.
2024-08-07 10:14:35,297	INFO scripts.py:972 -- This command will now block forever until terminated by a signal.
2024-08-07 10:14:35,297	INFO scripts.py:975 -- Running subprocesses are monitored and a message will be printed if any of them terminate unexpectedly. Subprocesses exit with SIGTERM will be treated as graceful, thus NOT reported.
2024-08-07 10:14:35,297	INFO scripts.py:975 -- Running subprocesses are monitored and a message will be printed if any of them terminate unexpectedly. Subprocesses exit with SIGTERM will be treated as graceful, thus NOT reported.
2024-08-07 10:14:35,297	SUCC scripts.py:958 -- --------------------
2024-08-07 10:14:35,297	SUCC scripts.py:959 -- Ray runtime started.
2024-08-07 10:14:35,297	SUCC scripts.py:960 -- --------------------
2024-08-07 10:14:35,297	INFO scripts.py:962 -- To terminate the Ray runtime, run
2024-08-07 10:14:35,297	SUCC scripts.py:958 -- --------------------
2024-08-07 10:14:35,297	INFO scripts.py:963 --   ray stop
2024-08-07 10:14:35,297	SUCC scripts.py:959 -- Ray runtime started.
2024-08-07 10:14:35,297	SUCC scripts.py:960 -- --------------------
2024-08-07 10:14:35,297	INFO scripts.py:962 -- To terminate the Ray runtime, run
2024-08-07 10:14:35,297	INFO scripts.py:963 --   ray stop
2024-08-07 10:14:35,297	SUCC scripts.py:958 -- --------------------
2024-08-07 10:14:35,297	INFO scripts.py:971 -- --block
2024-08-07 10:14:35,297	SUCC scripts.py:959 -- Ray runtime started.
2024-08-07 10:14:35,297	SUCC scripts.py:960 -- --------------------
2024-08-07 10:14:35,297	INFO scripts.py:962 -- To terminate the Ray runtime, run
2024-08-07 10:14:35,297	INFO scripts.py:963 --   ray stop
2024-08-07 10:14:35,297	INFO scripts.py:972 -- This command will now block forever until terminated by a signal.
2024-08-07 10:14:35,297	INFO scripts.py:975 -- Running subprocesses are monitored and a message will be printed if any of them terminate unexpectedly. Subprocesses exit with SIGTERM will be treated as graceful, thus NOT reported.
2024-08-07 10:14:35,297	INFO scripts.py:971 -- --block
2024-08-07 10:14:35,297	INFO scripts.py:972 -- This command will now block forever until terminated by a signal.
2024-08-07 10:14:35,297	INFO scripts.py:975 -- Running subprocesses are monitored and a message will be printed if any of them terminate unexpectedly. Subprocesses exit with SIGTERM will be treated as graceful, thus NOT reported.
2024-08-07 10:14:35,297	INFO scripts.py:971 -- --block
2024-08-07 10:14:35,297	INFO scripts.py:972 -- This command will now block forever until terminated by a signal.
2024-08-07 10:14:35,297	INFO scripts.py:975 -- Running subprocesses are monitored and a message will be printed if any of them terminate unexpectedly. Subprocesses exit with SIGTERM will be treated as graceful, thus NOT reported.
2024-08-07 10:14:35,298	SUCC scripts.py:958 -- --------------------
2024-08-07 10:14:35,298	SUCC scripts.py:959 -- Ray runtime started.
2024-08-07 10:14:35,298	SUCC scripts.py:958 -- --------------------
2024-08-07 10:14:35,298	SUCC scripts.py:960 -- --------------------
2024-08-07 10:14:35,298	SUCC scripts.py:959 -- Ray runtime started.
2024-08-07 10:14:35,298	INFO scripts.py:962 -- To terminate the Ray runtime, run
2024-08-07 10:14:35,298	SUCC scripts.py:960 -- --------------------
2024-08-07 10:14:35,298	INFO scripts.py:963 --   ray stop
2024-08-07 10:14:35,298	INFO scripts.py:962 -- To terminate the Ray runtime, run
2024-08-07 10:14:35,298	INFO scripts.py:963 --   ray stop
2024-08-07 10:14:35,298	INFO scripts.py:971 -- --block
2024-08-07 10:14:35,298	INFO scripts.py:971 -- --block
2024-08-07 10:14:35,298	INFO scripts.py:972 -- This command will now block forever until terminated by a signal.
2024-08-07 10:14:35,298	INFO scripts.py:975 -- Running subprocesses are monitored and a message will be printed if any of them terminate unexpectedly. Subprocesses exit with SIGTERM will be treated as graceful, thus NOT reported.
2024-08-07 10:14:35,298	INFO scripts.py:972 -- This command will now block forever until terminated by a signal.
2024-08-07 10:14:35,298	INFO scripts.py:975 -- Running subprocesses are monitored and a message will be printed if any of them terminate unexpectedly. Subprocesses exit with SIGTERM will be treated as graceful, thus NOT reported.
INFO 08-07 10:15:57 config.py:715] Defaulting to use ray for distributed inference
2024-08-07 10:15:57,398	INFO worker.py:1567 -- Connecting to existing Ray cluster at address: lrdn1699:6379...
2024-08-07 10:15:57,406	INFO worker.py:1752 -- Connected to Ray cluster.
INFO 08-07 10:16:00 llm_engine.py:176] Initializing an LLM engine (v0.5.3.post1) with config: model='/leonardo_scratch/large/userexternal/gpuccett/models/hf_llama/llama-3.1-405b-instruct-hf', speculative_config=None, tokenizer='/leonardo_scratch/large/userexternal/gpuccett/models/hf_llama/llama-3.1-405b-instruct-hf', skip_tokenizer_init=False, tokenizer_mode=slow, revision=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=8192, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=32, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None), seed=0, served_model_name=/leonardo_scratch/large/userexternal/gpuccett/models/hf_llama/llama-3.1-405b-instruct-hf, use_v2_block_manager=False, enable_prefix_caching=False)
INFO 08-07 10:27:36 utils.py:784] Found nccl from library libnccl.so.2
INFO 08-07 10:27:36 pynccl.py:63] vLLM is using nccl==2.20.5
(RayWorkerWrapper pid=700343, ip=<IP_ADDRESS>) INFO 08-07 10:27:36 utils.py:784] Found nccl from library libnccl.so.2
(RayWorkerWrapper pid=700343, ip=<IP_ADDRESS>) INFO 08-07 10:27:36 pynccl.py:63] vLLM is using nccl==2.20.5
WARNING 08-07 10:27:40 custom_all_reduce.py:69] Custom allreduce is disabled because this process group spans across nodes.
(RayWorkerWrapper pid=669372, ip=<IP_ADDRESS>) WARNING 08-07 10:27:40 custom_all_reduce.py:69] Custom allreduce is disabled because this process group spans across nodes.
INFO 08-07 10:27:40 shm_broadcast.py:241] vLLM message queue communication handle: Handle(connect_ip='<IP_ADDRESS>', local_reader_ranks=[1, 2, 3], buffer=<vllm.distributed.device_communicators.shm_broadcast.ShmRingBuffer object at 0x14e7518b0f50>, local_subscribe_port=40815, local_sync_port=38345, remote_subscribe_port=57791, remote_sync_port=35819)
INFO 08-07 10:27:40 model_runner.py:680] Starting to load model /leonardo_scratch/large/userexternal/gpuccett/models/hf_llama/llama-3.1-405b-instruct-hf...
(RayWorkerWrapper pid=700343, ip=<IP_ADDRESS>) INFO 08-07 10:27:40 model_runner.py:680] Starting to load model /leonardo_scratch/large/userexternal/gpuccett/models/hf_llama/llama-3.1-405b-instruct-hf...

Loading safetensors checkpoint shards:   0% Completed | 0/191 [00:00<?, ?it/s]

Loading safetensors checkpoint shards:   1% Completed | 1/191 [00:25<1:19:52, 25.22s/it]

Loading safetensors checkpoint shards:   1% Completed | 2/191 [00:25<33:40, 10.69s/it]

Loading safetensors checkpoint shards:   2% Completed | 3/191 [00:26<19:01,  6.07s/it]

Loading safetensors checkpoint shards:   2% Completed | 4/191 [00:48<38:15, 12.28s/it]

Loading safetensors checkpoint shards:   3% Completed | 5/191 [01:04<42:30, 13.71s/it]

Loading safetensors checkpoint shards:   3% Completed | 6/191 [01:25<49:51, 16.17s/it]

Loading safetensors checkpoint shards:   4% Completed | 7/191 [01:42<50:16, 16.39s/it]

Loading safetensors checkpoint shards:   4% Completed | 8/191 [01:44<36:47, 12.06s/it]

Loading safetensors checkpoint shards:   5% Completed | 9/191 [01:52<32:12, 10.62s/it]

Loading safetensors checkpoint shards:   5% Completed | 10/191 [01:52<22:42,  7.53s/it]

Loading safetensors checkpoint shards:   6% Completed | 11/191 [01:57<20:15,  6.75s/it]

Loading safetensors checkpoint shards:   6% Completed | 12/191 [01:58<14:27,  4.85s/it]

Loading safetensors checkpoint shards:   7% Completed | 13/191 [02:07<18:15,  6.15s/it]

Loading safetensors checkpoint shards:   7% Completed | 14/191 [02:11<16:10,  5.48s/it]

Loading safetensors checkpoint shards:   8% Completed | 15/191 [02:12<11:43,  4.00s/it]

Loading safetensors checkpoint shards:   8% Completed | 16/191 [02:18<13:43,  4.70s/it]

Loading safetensors checkpoint shards:   9% Completed | 17/191 [02:23<14:17,  4.93s/it]

Loading safetensors checkpoint shards:   9% Completed | 18/191 [02:27<13:06,  4.54s/it]

Loading safetensors checkpoint shards:  10% Completed | 19/191 [02:32<13:11,  4.60s/it]

Loading safetensors checkpoint shards:  10% Completed | 20/191 [02:32<09:36,  3.37s/it]

Loading safetensors checkpoint shards:  11% Completed | 21/191 [02:35<09:08,  3.23s/it]

Loading safetensors checkpoint shards:  12% Completed | 22/191 [02:40<10:42,  3.80s/it]

Loading safetensors checkpoint shards:  12% Completed | 23/191 [03:04<27:22,  9.78s/it]

Loading safetensors checkpoint shards:  13% Completed | 24/191 [03:10<23:53,  8.59s/it]

Loading safetensors checkpoint shards:  13% Completed | 25/191 [03:10<16:59,  6.14s/it]

Loading safetensors checkpoint shards:  14% Completed | 26/191 [03:18<18:04,  6.57s/it]

Loading safetensors checkpoint shards:  14% Completed | 27/191 [03:18<13:03,  4.78s/it]

Loading safetensors checkpoint shards:  15% Completed | 28/191 [03:20<10:09,  3.74s/it]

Loading safetensors checkpoint shards:  15% Completed | 29/191 [03:23<09:53,  3.66s/it]

Loading safetensors checkpoint shards:  16% Completed | 30/191 [03:27<09:50,  3.67s/it]

Loading safetensors checkpoint shards:  16% Completed | 31/191 [03:36<13:49,  5.19s/it]

Loading safetensors checkpoint shards:  17% Completed | 32/191 [03:36<09:59,  3.77s/it]

Loading safetensors checkpoint shards:  17% Completed | 33/191 [03:37<07:37,  2.90s/it]

Loading safetensors checkpoint shards:  18% Completed | 34/191 [03:41<08:13,  3.14s/it]

Loading safetensors checkpoint shards:  18% Completed | 35/191 [03:46<09:57,  3.83s/it]

Loading safetensors checkpoint shards:  19% Completed | 36/191 [03:55<13:24,  5.19s/it]

Loading safetensors checkpoint shards:  19% Completed | 37/191 [03:55<09:28,  3.69s/it]

Loading safetensors checkpoint shards:  20% Completed | 38/191 [03:55<07:00,  2.75s/it]

Loading safetensors checkpoint shards:  20% Completed | 39/191 [03:56<05:15,  2.08s/it]

Loading safetensors checkpoint shards:  21% Completed | 40/191 [04:02<08:00,  3.18s/it]

Loading safetensors checkpoint shards:  21% Completed | 41/191 [04:07<09:50,  3.94s/it]

Loading safetensors checkpoint shards:  22% Completed | 42/191 [04:12<10:19,  4.16s/it]

Loading safetensors checkpoint shards:  23% Completed | 43/191 [04:15<09:33,  3.88s/it]

Loading safetensors checkpoint shards:  23% Completed | 44/191 [04:15<06:54,  2.82s/it]

Loading safetensors checkpoint shards:  24% Completed | 45/191 [04:16<05:09,  2.12s/it]

Loading safetensors checkpoint shards:  24% Completed | 46/191 [04:23<08:32,  3.53s/it]

Loading safetensors checkpoint shards:  25% Completed | 47/191 [04:28<09:32,  3.98s/it]

Loading safetensors checkpoint shards:  25% Completed | 48/191 [04:32<09:40,  4.06s/it]

Loading safetensors checkpoint shards:  26% Completed | 49/191 [04:33<07:04,  2.99s/it]

Loading safetensors checkpoint shards:  26% Completed | 50/191 [04:40<10:17,  4.38s/it]

Loading safetensors checkpoint shards:  27% Completed | 51/191 [04:43<09:16,  3.97s/it]

Loading safetensors checkpoint shards:  27% Completed | 52/191 [04:44<06:50,  2.96s/it]

Loading safetensors checkpoint shards:  28% Completed | 53/191 [04:44<05:00,  2.18s/it]

Loading safetensors checkpoint shards:  28% Completed | 54/191 [04:44<03:39,  1.60s/it]

Loading safetensors checkpoint shards:  29% Completed | 55/191 [04:49<05:57,  2.63s/it]

Loading safetensors checkpoint shards:  29% Completed | 56/191 [04:55<07:37,  3.39s/it]

Loading safetensors checkpoint shards:  30% Completed | 57/191 [05:00<08:35,  3.84s/it]

Loading safetensors checkpoint shards:  30% Completed | 58/191 [05:04<09:05,  4.10s/it]

Loading safetensors checkpoint shards:  31% Completed | 59/191 [05:05<06:44,  3.07s/it]

Loading safetensors checkpoint shards:  31% Completed | 60/191 [05:06<05:07,  2.34s/it]

Loading safetensors checkpoint shards:  32% Completed | 61/191 [05:16<10:12,  4.71s/it]

Loading safetensors checkpoint shards:  32% Completed | 62/191 [05:16<07:29,  3.49s/it]

Loading safetensors checkpoint shards:  33% Completed | 63/191 [05:28<12:50,  6.02s/it]

Loading safetensors checkpoint shards:  34% Completed | 64/191 [05:29<09:18,  4.40s/it]

Loading safetensors checkpoint shards:  34% Completed | 65/191 [05:36<11:00,  5.24s/it]