[Usage]: The 8xH100 device failed to run meta-llama/Meta-Llama-3.1-405B-Instruct-FP8.

Your current environment
The output of `python collect_env.py`
Collecting environment information...
 However, it does not run correctly and returned the following error message:
root@dbe9716daf8891-sz5t2:/workspace# vllm serve /models/Meta-Llama-3.1-405B-Instruct-FP8 --tensor-parallel-size 8 --served-model-name "Meta-Llama-3.1-405B-Instruct-FP8" --gpu-memory-utilization 0.98
INFO 07-24 14:03:05 api_server.py:219] vLLM API server version 0.5.3.post1
INFO 07-24 14:03:05 api_server.py:220] args: Namespace(model_tag='/models/Meta-Llama-3.1-405B-Instruct-FP8', host=None, port=8000, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], model='/models/Meta-Llama-3.1-405B-Instruct-FP8', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, download_dir=None, load_format='auto', dtype='auto', kv_cache_dtype='auto', quantization_param_path=None, max_model_len=None, guided_decoding_backend='outlines', distributed_executor_backend=None, worker_use_ray=False, pipeline_parallel_size=1, tensor_parallel_size=8, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=16, enable_prefix_caching=False, disable_sliding_window=False, use_v2_block_manager=False, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.98, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_seqs=256, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, enforce_eager=False, max_context_len_to_capture=None, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, enable_lora=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_model=None, num_speculative_tokens=None, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=['Meta-Llama-3.1-405B-Instruct-FP8'], qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, engine_use_ray=False, disable_log_requests=False, max_log_len=None, dispatch_function=<function serve at 0x7fe0c5fee710>)
INFO 07-24 14:03:06 config.py:715] Defaulting to use mp for distributed inference
WARNING 07-24 14:03:06 arg_utils.py:762] Chunked prefill is enabled by default for models with max_model_len > 32K. Currently, chunked prefill might not work with some features or models. If you encounter any issues, please disable chunked prefill by setting --enable-chunked-prefill=False.
INFO 07-24 14:03:06 config.py:806] Chunked prefill is enabled with max_num_batched_tokens=512.
INFO 07-24 14:03:06 llm_engine.py:176] Initializing an LLM engine (v0.5.3.post1) with config: model='/models/Meta-Llama-3.1-405B-Instruct-FP8', speculative_config=None, tokenizer='/models/Meta-Llama-3.1-405B-Instruct-FP8', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=131072, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=8, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=fbgemm_fp8, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None), seed=0, served_model_name=Meta-Llama-3.1-405B-Instruct-FP8, use_v2_block_manager=False, enable_prefix_caching=False)
INFO 07-24 14:03:06 custom_cache_manager.py:17] Setting Triton cache manager to: vllm.triton_utils.custom_cache_manager:CustomCacheManager
(VllmWorkerProcess pid=6495) INFO 07-24 14:03:07 multiproc_worker_utils.py:215] Worker ready; awaiting tasks
(VllmWorkerProcess pid=6499) INFO 07-24 14:03:07 multiproc_worker_utils.py:215] Worker ready; awaiting tasks
(VllmWorkerProcess pid=6494) INFO 07-24 14:03:07 multiproc_worker_utils.py:215] Worker ready; awaiting tasks
(VllmWorkerProcess pid=6500) INFO 07-24 14:03:07 multiproc_worker_utils.py:215] Worker ready; awaiting tasks
(VllmWorkerProcess pid=6498) INFO 07-24 14:03:07 multiproc_worker_utils.py:215] Worker ready; awaiting tasks
(VllmWorkerProcess pid=6496) INFO 07-24 14:03:07 multiproc_worker_utils.py:215] Worker ready; awaiting tasks
(VllmWorkerProcess pid=6497) INFO 07-24 14:03:07 multiproc_worker_utils.py:215] Worker ready; awaiting tasks
INFO 07-24 14:03:23 utils.py:784] Found nccl from library libnccl.so.2
(VllmWorkerProcess pid=6498) INFO 07-24 14:03:23 utils.py:784] Found nccl from library libnccl.so.2
(VllmWorkerProcess pid=6496) INFO 07-24 14:03:23 utils.py:784] Found nccl from library libnccl.so.2
INFO 07-24 14:03:23 pynccl.py:63] vLLM is using nccl==2.20.5
(VllmWorkerProcess pid=6498) INFO 07-24 14:03:23 pynccl.py:63] vLLM is using nccl==2.20.5
(VllmWorkerProcess pid=6496) INFO 07-24 14:03:23 pynccl.py:63] vLLM is using nccl==2.20.5
(VllmWorkerProcess pid=6494) INFO 07-24 14:03:23 utils.py:784] Found nccl from library libnccl.so.2
(VllmWorkerProcess pid=6497) INFO 07-24 14:03:23 utils.py:784] Found nccl from library libnccl.so.2
(VllmWorkerProcess pid=6499) INFO 07-24 14:03:23 utils.py:784] Found nccl from library libnccl.so.2
(VllmWorkerProcess pid=6497) INFO 07-24 14:03:23 pynccl.py:63] vLLM is using nccl==2.20.5
(VllmWorkerProcess pid=6495) INFO 07-24 14:03:23 utils.py:784] Found nccl from library libnccl.so.2
(VllmWorkerProcess pid=6494) INFO 07-24 14:03:23 pynccl.py:63] vLLM is using nccl==2.20.5
(VllmWorkerProcess pid=6499) INFO 07-24 14:03:23 pynccl.py:63] vLLM is using nccl==2.20.5
(VllmWorkerProcess pid=6495) INFO 07-24 14:03:23 pynccl.py:63] vLLM is using nccl==2.20.5
(VllmWorkerProcess pid=6500) INFO 07-24 14:03:23 utils.py:784] Found nccl from library libnccl.so.2
(VllmWorkerProcess pid=6500) INFO 07-24 14:03:23 pynccl.py:63] vLLM is using nccl==2.20.5
(VllmWorkerProcess pid=6498) INFO 07-24 14:03:36 custom_all_reduce_utils.py:232] reading GPU P2P access cache from /root/.cache/vllm/gpu_p2p_access_cache_for_0,1,2,3,4,5,6,7.json
(VllmWorkerProcess pid=6499) INFO 07-24 14:03:36 custom_all_reduce_utils.py:232] reading GPU P2P access cache from /root/.cache/vllm/gpu_p2p_access_cache_for_0,1,2,3,4,5,6,7.json
(VllmWorkerProcess pid=6495) INFO 07-24 14:03:36 custom_all_reduce_utils.py:232] reading GPU P2P access cache from /root/.cache/vllm/gpu_p2p_access_cache_for_0,1,2,3,4,5,6,7.json
(VllmWorkerProcess pid=6500) INFO 07-24 14:03:36 custom_all_reduce_utils.py:232] reading GPU P2P access cache from /root/.cache/vllm/gpu_p2p_access_cache_for_0,1,2,3,4,5,6,7.json
(VllmWorkerProcess pid=6497) INFO 07-24 14:03:36 custom_all_reduce_utils.py:232] reading GPU P2P access cache from /root/.cache/vllm/gpu_p2p_access_cache_for_0,1,2,3,4,5,6,7.json
(VllmWorkerProcess pid=6496) INFO 07-24 14:03:36 custom_all_reduce_utils.py:232] reading GPU P2P access cache from /root/.cache/vllm/gpu_p2p_access_cache_for_0,1,2,3,4,5,6,7.json
INFO 07-24 14:03:36 custom_all_reduce_utils.py:232] reading GPU P2P access cache from /root/.cache/vllm/gpu_p2p_access_cache_for_0,1,2,3,4,5,6,7.json
(VllmWorkerProcess pid=6494) INFO 07-24 14:03:36 custom_all_reduce_utils.py:232] reading GPU P2P access cache from /root/.cache/vllm/gpu_p2p_access_cache_for_0,1,2,3,4,5,6,7.json
INFO 07-24 14:03:36 shm_broadcast.py:241] vLLM message queue communication handle: Handle(connect_ip='<IP_ADDRESS>', local_reader_ranks=[1, 2, 3, 4, 5, 6, 7], buffer=<vllm.distributed.device_communicators.shm_broadcast.ShmRingBuffer object at 0x7fe0c4bd3040>, local_subscribe_port=32775, local_sync_port=49501, remote_subscribe_port=None, remote_sync_port=None)
(VllmWorkerProcess pid=6495) INFO 07-24 14:03:36 model_runner.py:680] Starting to load model /models/Meta-Llama-3.1-405B-Instruct-FP8...
INFO 07-24 14:03:36 model_runner.py:680] Starting to load model /models/Meta-Llama-3.1-405B-Instruct-FP8...
(VllmWorkerProcess pid=6499) INFO 07-24 14:03:36 model_runner.py:680] Starting to load model /models/Meta-Llama-3.1-405B-Instruct-FP8...
(VllmWorkerProcess pid=6500) INFO 07-24 14:03:36 model_runner.py:680] Starting to load model /models/Meta-Llama-3.1-405B-Instruct-FP8...
(VllmWorkerProcess pid=6498) INFO 07-24 14:03:36 model_runner.py:680] Starting to load model /models/Meta-Llama-3.1-405B-Instruct-FP8...
(VllmWorkerProcess pid=6496) INFO 07-24 14:03:36 model_runner.py:680] Starting to load model /models/Meta-Llama-3.1-405B-Instruct-FP8...
(VllmWorkerProcess pid=6497) INFO 07-24 14:03:36 model_runner.py:680] Starting to load model /models/Meta-Llama-3.1-405B-Instruct-FP8...
(VllmWorkerProcess pid=6494) INFO 07-24 14:03:36 model_runner.py:680] Starting to load model /models/Meta-Llama-3.1-405B-Instruct-FP8...
(VllmWorkerProcess pid=6494) INFO 07-24 14:07:31 model_runner.py:692] Loading model weights took 57.7520 GB
(VllmWorkerProcess pid=6498) INFO 07-24 14:07:31 model_runner.py:692] Loading model weights took 57.7520 GB
INFO 07-24 14:07:31 model_runner.py:692] Loading model weights took 57.7520 GB
(VllmWorkerProcess pid=6500) INFO 07-24 14:07:31 model_runner.py:692] Loading model weights took 57.7520 GB
(VllmWorkerProcess pid=6497) INFO 07-24 14:07:31 model_runner.py:692] Loading model weights took 57.7520 GB
(VllmWorkerProcess pid=6496) INFO 07-24 14:07:31 model_runner.py:692] Loading model weights took 57.7520 GB
(VllmWorkerProcess pid=6499) INFO 07-24 14:07:32 model_runner.py:692] Loading model weights took 57.7520 GB
(VllmWorkerProcess pid=6495) INFO 07-24 14:07:32 model_runner.py:692] Loading model weights took 57.7520 GB
INFO 07-24 14:07:40 distributed_gpu_executor.py:56] # GPU blocks: 4962, # CPU blocks: 2080
(VllmWorkerProcess pid=6494) ERROR 07-24 14:07:40 multiproc_worker_utils.py:226] Exception in worker VllmWorkerProcess while processing method initialize_cache: The model's max seq len (131072) is larger than the maximum number of tokens that can be stored in KV cache (79392). Try increasing `gpu_memory_utilization` or decreasing `max_model_len` when initializing the engine., Traceback (most recent call last):
(VllmWorkerProcess pid=6494) ERROR 07-24 14:07:40 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.10/dist-packages/vllm/executor/multiproc_worker_utils.py", line 223, in _run_worker_process
(VllmWorkerProcess pid=6494) ERROR 07-24 14:07:40 multiproc_worker_utils.py:226]     output = executor(*args, **kwargs)
(VllmWorkerProcess pid=6494) ERROR 07-24 14:07:40 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker.py", line 212, in initialize_cache
(VllmWorkerProcess pid=6494) ERROR 07-24 14:07:40 multiproc_worker_utils.py:226]     raise_if_cache_size_invalid(num_gpu_blocks,
(VllmWorkerProcess pid=6494) ERROR 07-24 14:07:40 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker.py", line 372, in raise_if_cache_size_invalid
(VllmWorkerProcess pid=6494) ERROR 07-24 14:07:40 multiproc_worker_utils.py:226]     raise ValueError(
(VllmWorkerProcess pid=6497) ERROR 07-24 14:07:40 multiproc_worker_utils.py:226] Exception in worker VllmWorkerProcess while processing method initialize_cache: The model's max seq len (131072) is larger than the maximum number of tokens that can be stored in KV cache (79392). Try increasing `gpu_memory_utilization` or decreasing `max_model_len` when initializing the engine., Traceback (most recent call last):
(VllmWorkerProcess pid=6494) ERROR 07-24 14:07:40 multiproc_worker_utils.py:226] ValueError: The model's max seq len (131072) is larger than the maximum number of tokens that can be stored in KV cache (79392). Try increasing `gpu_memory_utilization` or decreasing `max_model_len` when initializing the engine.
(VllmWorkerProcess pid=6494) ERROR 07-24 14:07:40 multiproc_worker_utils.py:226]
(VllmWorkerProcess pid=6497) ERROR 07-24 14:07:40 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.10/dist-packages/vllm/executor/multiproc_worker_utils.py", line 223, in _run_worker_process
(VllmWorkerProcess pid=6497) ERROR 07-24 14:07:40 multiproc_worker_utils.py:226]     output = executor(*args, **kwargs)
(VllmWorkerProcess pid=6497) ERROR 07-24 14:07:40 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker.py", line 212, in initialize_cache
(VllmWorkerProcess pid=6497) ERROR 07-24 14:07:40 multiproc_worker_utils.py:226]     raise_if_cache_size_invalid(num_gpu_blocks,
(VllmWorkerProcess pid=6497) ERROR 07-24 14:07:40 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker.py", line 372, in raise_if_cache_size_invalid
(VllmWorkerProcess pid=6497) ERROR 07-24 14:07:40 multiproc_worker_utils.py:226]     raise ValueError(
(VllmWorkerProcess pid=6497) ERROR 07-24 14:07:40 multiproc_worker_utils.py:226] ValueError: The model's max seq len (131072) is larger than the maximum number of tokens that can be stored in KV cache (79392). Try increasing `gpu_memory_utilization` or decreasing `max_model_len` when initializing the engine.
(VllmWorkerProcess pid=6497) ERROR 07-24 14:07:40 multiproc_worker_utils.py:226]
(VllmWorkerProcess pid=6499) ERROR 07-24 14:07:40 multiproc_worker_utils.py:226] Exception in worker VllmWorkerProcess while processing method initialize_cache: The model's max seq len (131072) is larger than the maximum number of tokens that can be stored in KV cache (79392). Try increasing `gpu_memory_utilization` or decreasing `max_model_len` when initializing the engine., Traceback (most recent call last):
(VllmWorkerProcess pid=6499) ERROR 07-24 14:07:40 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.10/dist-packages/vllm/executor/multiproc_worker_utils.py", line 223, in _run_worker_process
(VllmWorkerProcess pid=6499) ERROR 07-24 14:07:40 multiproc_worker_utils.py:226]     output = executor(*args, **kwargs)
(VllmWorkerProcess pid=6499) ERROR 07-24 14:07:40 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker.py", line 212, in initialize_cache
(VllmWorkerProcess pid=6499) ERROR 07-24 14:07:40 multiproc_worker_utils.py:226]     raise_if_cache_size_invalid(num_gpu_blocks,
(VllmWorkerProcess pid=6499) ERROR 07-24 14:07:40 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker.py", line 372, in raise_if_cache_size_invalid
(VllmWorkerProcess pid=6499) ERROR 07-24 14:07:40 multiproc_worker_utils.py:226]     raise ValueError(
(VllmWorkerProcess pid=6499) ERROR 07-24 14:07:40 multiproc_worker_utils.py:226] ValueError: The model's max seq len (131072) is larger than the maximum number of tokens that can be stored in KV cache (79392). Try increasing `gpu_memory_utilization` or decreasing `max_model_len` when initializing the engine.
(VllmWorkerProcess pid=6499) ERROR 07-24 14:07:40 multiproc_worker_utils.py:226]
(VllmWorkerProcess pid=6500) ERROR 07-24 14:07:40 multiproc_worker_utils.py:226] Exception in worker VllmWorkerProcess while processing method initialize_cache: The model's max seq len (131072) is larger than the maximum number of tokens that can be stored in KV cache (79392). Try increasing `gpu_memory_utilization` or decreasing `max_model_len` when initializing the engine., Traceback (most recent call last):
(VllmWorkerProcess pid=6500) ERROR 07-24 14:07:40 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.10/dist-packages/vllm/executor/multiproc_worker_utils.py", line 223, in _run_worker_process
(VllmWorkerProcess pid=6500) ERROR 07-24 14:07:40 multiproc_worker_utils.py:226]     output = executor(*args, **kwargs)
(VllmWorkerProcess pid=6500) ERROR 07-24 14:07:40 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker.py", line 212, in initialize_cache
(VllmWorkerProcess pid=6500) ERROR 07-24 14:07:40 multiproc_worker_utils.py:226]     raise_if_cache_size_invalid(num_gpu_blocks,
(VllmWorkerProcess pid=6500) ERROR 07-24 14:07:40 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker.py", line 372, in raise_if_cache_size_invalid
(VllmWorkerProcess pid=6500) ERROR 07-24 14:07:40 multiproc_worker_utils.py:226]     raise ValueError(
(VllmWorkerProcess pid=6500) ERROR 07-24 14:07:40 multiproc_worker_utils.py:226] ValueError: The model's max seq len (131072) is larger than the maximum number of tokens that can be stored in KV cache (79392). Try increasing `gpu_memory_utilization` or decreasing `max_model_len` when initializing the engine.
(VllmWorkerProcess pid=6500) ERROR 07-24 14:07:40 multiproc_worker_utils.py:226]
(VllmWorkerProcess pid=6498) ERROR 07-24 14:07:40 multiproc_worker_utils.py:226] Exception in worker VllmWorkerProcess while processing method initialize_cache: The model's max seq len (131072) is larger than the maximum number of tokens that can be stored in KV cache (79392). Try increasing `gpu_memory_utilization` or decreasing `max_model_len` when initializing the engine., Traceback (most recent call last):
(VllmWorkerProcess pid=6498) ERROR 07-24 14:07:40 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.10/dist-packages/vllm/executor/multiproc_worker_utils.py", line 223, in _run_worker_process
(VllmWorkerProcess pid=6498) ERROR 07-24 14:07:40 multiproc_worker_utils.py:226]     output = executor(*args, **kwargs)
(VllmWorkerProcess pid=6498) ERROR 07-24 14:07:40 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker.py", line 212, in initialize_cache
(VllmWorkerProcess pid=6498) ERROR 07-24 14:07:40 multiproc_worker_utils.py:226]     raise_if_cache_size_invalid(num_gpu_blocks,
(VllmWorkerProcess pid=6498) ERROR 07-24 14:07:40 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker.py", line 372, in raise_if_cache_size_invalid
(VllmWorkerProcess pid=6498) ERROR 07-24 14:07:40 multiproc_worker_utils.py:226]     raise ValueError(
(VllmWorkerProcess pid=6498) ERROR 07-24 14:07:40 multiproc_worker_utils.py:226] ValueError: The model's max seq len (131072) is larger than the maximum number of tokens that can be stored in KV cache (79392). Try increasing `gpu_memory_utilization` or decreasing `max_model_len` when initializing the engine.
(VllmWorkerProcess pid=6498) ERROR 07-24 14:07:40 multiproc_worker_utils.py:226]
(VllmWorkerProcess pid=6496) ERROR 07-24 14:07:40 multiproc_worker_utils.py:226] Exception in worker VllmWorkerProcess while processing method initialize_cache: The model's max seq len (131072) is larger than the maximum number of tokens that can be stored in KV cache (79392). Try increasing `gpu_memory_utilization` or decreasing `max_model_len` when initializing the engine., Traceback (most recent call last):
(VllmWorkerProcess pid=6496) ERROR 07-24 14:07:40 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.10/dist-packages/vllm/executor/multiproc_worker_utils.py", line 223, in _run_worker_process
(VllmWorkerProcess pid=6496) ERROR 07-24 14:07:40 multiproc_worker_utils.py:226]     output = executor(*args, **kwargs)
(VllmWorkerProcess pid=6496) ERROR 07-24 14:07:40 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker.py", line 212, in initialize_cache
(VllmWorkerProcess pid=6496) ERROR 07-24 14:07:40 multiproc_worker_utils.py:226]     raise_if_cache_size_invalid(num_gpu_blocks,
(VllmWorkerProcess pid=6496) ERROR 07-24 14:07:40 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker.py", line 372, in raise_if_cache_size_invalid
(VllmWorkerProcess pid=6496) ERROR 07-24 14:07:40 multiproc_worker_utils.py:226]     raise ValueError(
(VllmWorkerProcess pid=6496) ERROR 07-24 14:07:40 multiproc_worker_utils.py:226] ValueError: The model's max seq len (131072) is larger than the maximum number of tokens that can be stored in KV cache (79392). Try increasing `gpu_memory_utilization` or decreasing `max_model_len` when initializing the engine.
(VllmWorkerProcess pid=6496) ERROR 07-24 14:07:40 multiproc_worker_utils.py:226]
[rank0]: Traceback (most recent call last):
[rank0]:   File "/usr/local/bin/vllm", line 8, in <module>
[rank0]:     sys.exit(main())
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/scripts.py", line 148, in main
[rank0]:     args.dispatch_function(args)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/scripts.py", line 28, in serve
[rank0]:     run_server(args)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/api_server.py", line 231, in run_server
[rank0]:     if llm_engine is not None else AsyncLLMEngine.from_engine_args(
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 466, in from_engine_args
[rank0]:     engine = cls(
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 380, in __init__
[rank0]:     self.engine = self._init_engine(*args, **kwargs)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 547, in _init_engine
[rank0]:     return engine_class(*args, **kwargs)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/llm_engine.py", line 265, in __init__
[rank0]:     self._initialize_kv_caches()
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/llm_engine.py", line 377, in _initialize_kv_caches
[rank0]:     self.model_executor.initialize_cache(num_gpu_blocks, num_cpu_blocks)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/executor/distributed_gpu_executor.py", line 62, in initialize_cache
[rank0]:     self._run_workers("initialize_cache",
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/executor/multiproc_gpu_executor.py", line 178, in _run_workers
[rank0]:     driver_worker_output = driver_worker_method(*args, **kwargs)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker.py", line 212, in initialize_cache
[rank0]:     raise_if_cache_size_invalid(num_gpu_blocks,
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker.py", line 372, in raise_if_cache_size_invalid
[rank0]:     raise ValueError(
[rank0]: ValueError: The model's max seq len (131072) is larger than the maximum number of tokens that can be stored in KV cache (79392). Try increasing `gpu_memory_utilization` or decreasing `max_model_len` when initializing the engine.
ERROR 07-24 14:07:41 multiproc_worker_utils.py:120] Worker VllmWorkerProcess pid 6500 died, exit code: -15
INFO 07-24 14:07:41 multiproc_worker_utils.py:123] Killing local vLLM worker processes
[rank0]:[W CudaIPCTypes.cpp:16] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]
/usr/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

Set --max-model-len 79392 or lower as needed, you do not have enough vram for the full context length

Set --max-model-len 79392 or lower as needed, you do not have enough vram for the full context length

Thank you, after adjusting to 8192, it works fine. How should I evaluate the maximum context length that my device can support?

Just to confirm, I'm only able to run 8k context for 405B FP8 with 8*H100

I'd suggest enabling chunked prefill!

@ywang96 By default 70b and 8b enable that already for > 32K:
WARNING 07-23 19:40:35 arg_utils.py:762] Chunked prefill is enabled by default for models with max_model_len > 32K. Currently, chunked prefill might not work with some features or models. If you encounter any issues, please disable chunked prefill by setting --enable-chunked-prefill=False.

However, 32K does not run without GPU OOM during startup for 405B.
So I'm not sure what you mean to do.

Sorry that I didn't read the stacktrace carefully.
I don't have an H100 box right now but this works on an A100-80G box for me. @pseudotensor @jueming0312
vllm serve meta-llama/Meta-Llama-3.1-405B-FP8 -tp 8 --gpu-memory-utilization 0.98 —max-num-seqs 32

I see, so reduce the max num sequences.  I used that for phi-3 in past, not sure why I didn't try it.  I'm unclear on how it affects memory.  I would have thought only something like max-num-batched-tokens matters since that is total tokens per iteration, and I was only trying to launch vLLM and didn't even have 1 sequence going yet.  But if it pre-allocates all those sequences in static way, maybe that's why.  But I thought it would pre-allocate based upon tokens, not sequences.

Ok, I was already using --max_num_seqs=17
i.e.
I didn't yet check if         --enable_chunked_prefill=True explicitly added will help, since I assumed it would be automatic for >32k.
But I'll review again.

Enabling chunked prefill will set max-num-batched-tokens to 512 by default.

I see, so reduce the max num sequences. I used that for phi-3 in past, not sure why I didn't try it. I'm unclear on how it affects memory. I would have thought only something like max-num-batched-tokens matters since that is total tokens per iteration, and I was only trying to launch vLLM and didn't even have 1 sequence going yet. But if it pre-allocates all those sequences in static way, maybe that's why. But I thought it would pre-allocate based upon tokens, not sequences.
Will review: #2492

indeed max_num_batched_tokens is fixed, but during the profiling run, we create max_num_seqs which each sequence length roughly equals to max_num_batched_tokens // max_num_seqs.
However, you can see from the code here that we're using a top_k = vocab_size - 1 (this value will be ~130k for Llama 3) sampling for profiling:
https://github.com/vllm-project/vllm/blob/316a41ac1de4e6e46933cadb39b9b7af65b01abd/vllm/worker/model_runner.py#L843-L848
thus more sequences do lead to more memory usage, even though the total batched tokens are the same.
I would suggest try building vLLM from source and changing this top_k value and see how it affects profiling!

For the first part of what you said, I think what you are saying is I shouldn't set both these:
        --max-model-len=8192 \
        --max-num-batched-tokens=8192 \

because when prefill is enabled, then --max-num-batched-tokens=512 is optimal and default and doesn't affect the actual maximum number of batches of tokens vLLM can process.

 Try increasing `gpu_memory_utilization` or decreasing `max_model_len` when initializing the engine.
ERROR 07-25 07:03:55 multiproc_worker_utils.py:120] Worker VllmWorkerProcess pid 76 died, exit code: -15
INFO 07-25 07:03:55 multiproc_worker_utils.py:123] Killing local vLLM worker processes
[rank0]:[W CudaIPCTypes.cpp:16] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]
/usr/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

I also tried 25, but that failed with saying 86k, so closer.
I tried 16, then says 89k, so not moving quickly to 128k
So close I guess.   See Note [Sharing CUDA tensors]
Failed: Cuda error /workspace/csrc/custom_all_reduce.cuh:307 'invalid argument'
Failed: Cuda error /workspace/csrc/custom_all_reduce.cuh:307 'invalid argument'
[rank0]: Traceback (most recent call last):
[rank0]:   File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
[rank0]:     return _run_code(code, main_globals, None,
[rank0]:   File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
[rank0]:     exec(code, run_globals)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/api_server.py", line 317, in <module>
[rank0]:     run_server(args)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/api_server.py", line 231, in run_server
[rank0]:     if llm_engine is not None else AsyncLLMEngine.from_engine_args(
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 466, in from_engine_args
[rank0]:     engine = cls(
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 380, in __init__
[rank0]:     self.engine = self._init_engine(*args, **kwargs)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 547, in _init_engine
[rank0]:     return engine_class(*args, **kwargs)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/llm_engine.py", line 265, in __init__
[rank0]:     self._initialize_kv_caches()
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/llm_engine.py", line 377, in _initialize_kv_caches
[rank0]:     self.model_executor.initialize_cache(num_gpu_blocks, num_cpu_blocks)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/executor/distributed_gpu_executor.py", line 62, in initialize_cache
[rank0]:     self._run_workers("initialize_cache",
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/executor/multiproc_gpu_executor.py", line 178, in _run_workers
[rank0]:     driver_worker_output = driver_worker_method(*args, **kwargs)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker.py", line 220, in initialize_cache
[rank0]:     self._warm_up_model()
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker.py", line 236, in _warm_up_model
[rank0]:     self.model_runner.capture_model(self.gpu_cache)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
[rank0]:     return func(*args, **kwargs)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/worker/model_runner.py", line 1173, in capture_model
[rank0]:     graph_runner.capture(**capture_inputs)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/worker/model_runner.py", line 1410, in capture
[rank0]:     with torch.cuda.graph(self._graph, pool=memory_pool, stream=stream):
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/torch/cuda/graphs.py", line 184, in __exit__
[rank0]:     self.cuda_graph.capture_end()
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/torch/cuda/graphs.py", line 82, in capture_end
[rank0]:     super().capture_end()
[rank0]: RuntimeError: CUDA error: out of memory
[rank0]: CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
[rank0]: For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
[rank0]: Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

ERROR 07-25 07:14:56 multiproc_worker_utils.py:120] Worker VllmWorkerProcess pid 82 died, exit code: -15
INFO 07-25 07:14:56 multiproc_worker_utils.py:123] Killing local vLLM worker processes
[rank0]:[W CudaIPCTypes.cpp:16] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]

That's really weird... I thought on A100-80G it would only require more GPU memory during profiling, since activation is done with FP16.
Have you confirmed before each launch, the GPU memory is cleared up properly?

Yes, I ensure the prior docker run I had is gone, nvidia-smi is clear.
 See Note [Sharing CUDA tensors]
Failed: Cuda error /workspace/csrc/custom_all_reduce.cuh:307 'invalid argument'
[rank0]: Traceback (most recent call last):
[rank0]:   File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
[rank0]:     return _run_code(code, main_globals, None,
[rank0]:   File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
[rank0]:     exec(code, run_globals)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/api_server.py", line 317, in <module>
[rank0]:     run_server(args)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/api_server.py", line 231, in run_server
[rank0]:     if llm_engine is not None else AsyncLLMEngine.from_engine_args(
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 466, in from_engine_args
[rank0]:     engine = cls(
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 380, in __init__
[rank0]:     self.engine = self._init_engine(*args, **kwargs)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 547, in _init_engine
[rank0]:     return engine_class(*args, **kwargs)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/llm_engine.py", line 265, in __init__
[rank0]:     self._initialize_kv_caches()
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/llm_engine.py", line 377, in _initialize_kv_caches
[rank0]:     self.model_executor.initialize_cache(num_gpu_blocks, num_cpu_blocks)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/executor/distributed_gpu_executor.py", line 62, in initialize_cache
[rank0]:     self._run_workers("initialize_cache",
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/executor/multiproc_gpu_executor.py", line 178, in _run_workers
[rank0]:     driver_worker_output = driver_worker_method(*args, **kwargs)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker.py", line 220, in initialize_cache
[rank0]:     self._warm_up_model()
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker.py", line 236, in _warm_up_model
[rank0]:     self.model_runner.capture_model(self.gpu_cache)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
[rank0]:     return func(*args, **kwargs)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/worker/model_runner.py", line 1173, in capture_model
[rank0]:     graph_runner.capture(**capture_inputs)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/worker/model_runner.py", line 1410, in capture
[rank0]:     with torch.cuda.graph(self._graph, pool=memory_pool, stream=stream):
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/torch/cuda/graphs.py", line 184, in __exit__
[rank0]:     self.cuda_graph.capture_end()
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/torch/cuda/graphs.py", line 82, in capture_end
[rank0]:     super().capture_end()
[rank0]: RuntimeError: CUDA error: out of memory
[rank0]: CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
[rank0]: For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
[rank0]: Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

So seems the chunked_prefill is cause, since 8k worked without it.
Here is full logs for this 8k case in case useful:
base) ubuntu@compute-permanent-node-406:~$ docker logs llama31-405b
INFO 07-25 07:22:50 api_server.py:219] vLLM API server version 0.5.3.post1
INFO 07-25 07:22:50 api_server.py:220] args: Namespace(host='<IP_ADDRESS>', port=5020, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key='EMPTY', lora_modules=None, prompt_adapters=None, chat_template=None, response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], model='meta-llama/Meta-Llama-3.1-405B-Instruct-FP8', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, download_dir='/home/ubuntu/.cache/huggingface/hub', load_format='auto', dtype='auto', kv_cache_dtype='auto', quantization_param_path=None, max_model_len=8192, guided_decoding_backend='outlines', distributed_executor_backend=None, worker_use_ray=False, pipeline_parallel_size=1, tensor_parallel_size=8, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=16, enable_prefix_caching=False, disable_sliding_window=False, use_v2_block_manager=False, num_lookahead_slots=0, seed=1234, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.99, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_seqs=16, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, enforce_eager=False, max_context_len_to_capture=None, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, enable_lora=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', scheduler_delay_factor=0.0, enable_chunked_prefill=True, speculative_model=None, num_speculative_tokens=None, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=None, qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, engine_use_ray=False, disable_log_requests=False, max_log_len=100)
INFO 07-25 07:22:50 config.py:715] Defaulting to use mp for distributed inference
INFO 07-25 07:22:50 config.py:806] Chunked prefill is enabled with max_num_batched_tokens=512.
INFO 07-25 07:22:50 llm_engine.py:176] Initializing an LLM engine (v0.5.3.post1) with config: model='meta-llama/Meta-Llama-3.1-405B-Instruct-FP8', speculative_config=None, tokenizer='meta-llama/Meta-Llama-3.1-405B-Instruct-FP8', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=8192, download_dir='/home/ubuntu/.cache/huggingface/hub', load_format=LoadFormat.AUTO, tensor_parallel_size=8, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=fbgemm_fp8, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None), seed=1234, served_model_name=meta-llama/Meta-Llama-3.1-405B-Instruct-FP8, use_v2_block_manager=False, enable_prefix_caching=False)
INFO 07-25 07:22:50 custom_cache_manager.py:17] Setting Triton cache manager to: vllm.triton_utils.custom_cache_manager:CustomCacheManager
(VllmWorkerProcess pid=76) INFO 07-25 07:22:50 multiproc_worker_utils.py:215] Worker ready; awaiting tasks
(VllmWorkerProcess pid=77) INFO 07-25 07:22:51 multiproc_worker_utils.py:215] Worker ready; awaiting tasks
(VllmWorkerProcess pid=78) INFO 07-25 07:22:51 multiproc_worker_utils.py:215] Worker ready; awaiting tasks
(VllmWorkerProcess pid=79) INFO 07-25 07:22:51 multiproc_worker_utils.py:215] Worker ready; awaiting tasks
(VllmWorkerProcess pid=80) INFO 07-25 07:22:51 multiproc_worker_utils.py:215] Worker ready; awaiting tasks
(VllmWorkerProcess pid=81) INFO 07-25 07:22:51 multiproc_worker_utils.py:215] Worker ready; awaiting tasks
(VllmWorkerProcess pid=82) INFO 07-25 07:22:51 multiproc_worker_utils.py:215] Worker ready; awaiting tasks
INFO 07-25 07:22:53 utils.py:774] Found nccl from environment variable VLLM_NCCL_SO_PATH=/usr/local/lib/python3.10/dist-packages/nvidia/nccl/lib/libnccl.so.2
(VllmWorkerProcess pid=76) INFO 07-25 07:22:53 utils.py:774] Found nccl from environment variable VLLM_NCCL_SO_PATH=/usr/local/lib/python3.10/dist-packages/nvidia/nccl/lib/libnccl.so.2
INFO 07-25 07:22:53 pynccl.py:63] vLLM is using nccl==2.20.5
(VllmWorkerProcess pid=76) INFO 07-25 07:22:53 pynccl.py:63] vLLM is using nccl==2.20.5
(VllmWorkerProcess pid=79) INFO 07-25 07:22:53 utils.py:774] Found nccl from environment variable VLLM_NCCL_SO_PATH=/usr/local/lib/python3.10/dist-packages/nvidia/nccl/lib/libnccl.so.2
(VllmWorkerProcess pid=79) INFO 07-25 07:22:53 pynccl.py:63] vLLM is using nccl==2.20.5
(VllmWorkerProcess pid=77) INFO 07-25 07:22:53 utils.py:774] Found nccl from environment variable VLLM_NCCL_SO_PATH=/usr/local/lib/python3.10/dist-packages/nvidia/nccl/lib/libnccl.so.2
(VllmWorkerProcess pid=78) INFO 07-25 07:22:53 utils.py:774] Found nccl from environment variable VLLM_NCCL_SO_PATH=/usr/local/lib/python3.10/dist-packages/nvidia/nccl/lib/libnccl.so.2
(VllmWorkerProcess pid=77) INFO 07-25 07:22:53 pynccl.py:63] vLLM is using nccl==2.20.5
(VllmWorkerProcess pid=78) INFO 07-25 07:22:53 pynccl.py:63] vLLM is using nccl==2.20.5
(VllmWorkerProcess pid=80) INFO 07-25 07:22:53 utils.py:774] Found nccl from environment variable VLLM_NCCL_SO_PATH=/usr/local/lib/python3.10/dist-packages/nvidia/nccl/lib/libnccl.so.2
(VllmWorkerProcess pid=81) INFO 07-25 07:22:53 utils.py:774] Found nccl from environment variable VLLM_NCCL_SO_PATH=/usr/local/lib/python3.10/dist-packages/nvidia/nccl/lib/libnccl.so.2
(VllmWorkerProcess pid=82) INFO 07-25 07:22:53 utils.py:774] Found nccl from environment variable VLLM_NCCL_SO_PATH=/usr/local/lib/python3.10/dist-packages/nvidia/nccl/lib/libnccl.so.2
(VllmWorkerProcess pid=80) INFO 07-25 07:22:53 pynccl.py:63] vLLM is using nccl==2.20.5
(VllmWorkerProcess pid=81) INFO 07-25 07:22:53 pynccl.py:63] vLLM is using nccl==2.20.5
(VllmWorkerProcess pid=82) INFO 07-25 07:22:53 pynccl.py:63] vLLM is using nccl==2.20.5
INFO 07-25 07:23:00 custom_all_reduce_utils.py:232] reading GPU P2P access cache from /home/ubuntu/.cache/vllm/gpu_p2p_access_cache_for_0,1,2,3,4,5,6,7.json
(VllmWorkerProcess pid=76) INFO 07-25 07:23:00 custom_all_reduce_utils.py:232] reading GPU P2P access cache from /home/ubuntu/.cache/vllm/gpu_p2p_access_cache_for_0,1,2,3,4,5,6,7.json
(VllmWorkerProcess pid=77) INFO 07-25 07:23:00 custom_all_reduce_utils.py:232] reading GPU P2P access cache from /home/ubuntu/.cache/vllm/gpu_p2p_access_cache_for_0,1,2,3,4,5,6,7.json
(VllmWorkerProcess pid=78) INFO 07-25 07:23:00 custom_all_reduce_utils.py:232] reading GPU P2P access cache from /home/ubuntu/.cache/vllm/gpu_p2p_access_cache_for_0,1,2,3,4,5,6,7.json
(VllmWorkerProcess pid=80) INFO 07-25 07:23:00 custom_all_reduce_utils.py:232] reading GPU P2P access cache from /home/ubuntu/.cache/vllm/gpu_p2p_access_cache_for_0,1,2,3,4,5,6,7.json
(VllmWorkerProcess pid=79) INFO 07-25 07:23:00 custom_all_reduce_utils.py:232] reading GPU P2P access cache from /home/ubuntu/.cache/vllm/gpu_p2p_access_cache_for_0,1,2,3,4,5,6,7.json
(VllmWorkerProcess pid=82) INFO 07-25 07:23:00 custom_all_reduce_utils.py:232] reading GPU P2P access cache from /home/ubuntu/.cache/vllm/gpu_p2p_access_cache_for_0,1,2,3,4,5,6,7.json
(VllmWorkerProcess pid=81) INFO 07-25 07:23:00 custom_all_reduce_utils.py:232] reading GPU P2P access cache from /home/ubuntu/.cache/vllm/gpu_p2p_access_cache_for_0,1,2,3,4,5,6,7.json
INFO 07-25 07:23:00 shm_broadcast.py:241] vLLM message queue communication handle: Handle(connect_ip='<IP_ADDRESS>', local_reader_ranks=[1, 2, 3, 4, 5, 6, 7], buffer=<vllm.distributed.device_communicators.shm_broadcast.ShmRingBuffer object at 0x79a3a49171f0>, local_subscribe_port=45519, local_sync_port=44165, remote_subscribe_port=None, remote_sync_port=None)
INFO 07-25 07:23:00 model_runner.py:680] Starting to load model meta-llama/Meta-Llama-3.1-405B-Instruct-FP8...
(VllmWorkerProcess pid=76) INFO 07-25 07:23:00 model_runner.py:680] Starting to load model meta-llama/Meta-Llama-3.1-405B-Instruct-FP8...
(VllmWorkerProcess pid=78) INFO 07-25 07:23:00 model_runner.py:680] Starting to load model meta-llama/Meta-Llama-3.1-405B-Instruct-FP8...
(VllmWorkerProcess pid=77) INFO 07-25 07:23:00 model_runner.py:680] Starting to load model meta-llama/Meta-Llama-3.1-405B-Instruct-FP8...
(VllmWorkerProcess pid=81) INFO 07-25 07:23:00 model_runner.py:680] Starting to load model meta-llama/Meta-Llama-3.1-405B-Instruct-FP8...
(VllmWorkerProcess pid=79) INFO 07-25 07:23:00 model_runner.py:680] Starting to load model meta-llama/Meta-Llama-3.1-405B-Instruct-FP8...
(VllmWorkerProcess pid=80) INFO 07-25 07:23:00 model_runner.py:680] Starting to load model meta-llama/Meta-Llama-3.1-405B-Instruct-FP8...
(VllmWorkerProcess pid=82) INFO 07-25 07:23:00 model_runner.py:680] Starting to load model meta-llama/Meta-Llama-3.1-405B-Instruct-FP8...
(VllmWorkerProcess pid=77) INFO 07-25 07:23:00 weight_utils.py:223] Using model weights format ['*.safetensors']
(VllmWorkerProcess pid=81) INFO 07-25 07:23:00 weight_utils.py:223] Using model weights format ['*.safetensors']
INFO 07-25 07:23:00 weight_utils.py:223] Using model weights format ['*.safetensors']
(VllmWorkerProcess pid=82) INFO 07-25 07:23:00 weight_utils.py:223] Using model weights format ['*.safetensors']
(VllmWorkerProcess pid=79) INFO 07-25 07:23:01 weight_utils.py:223] Using model weights format ['*.safetensors']
(VllmWorkerProcess pid=80) INFO 07-25 07:23:01 weight_utils.py:223] Using model weights format ['*.safetensors']
(VllmWorkerProcess pid=78) INFO 07-25 07:23:01 weight_utils.py:223] Using model weights format ['*.safetensors']
(VllmWorkerProcess pid=76) INFO 07-25 07:23:01 weight_utils.py:223] Using model weights format ['*.safetensors']
Loading safetensors checkpoint shards:   0% Completed | 0/109 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:   1% Completed | 1/109 [00:00<00:20,  5.35it/s]
Loading safetensors checkpoint shards:   2% Completed | 2/109 [00:00<00:21,  4.91it/s]
Loading safetensors checkpoint shards:   3% Completed | 3/109 [00:00<00:21,  5.02it/s]
Loading safetensors checkpoint shards:   4% Completed | 4/109 [00:00<00:22,  4.65it/s]
Loading safetensors checkpoint shards:   5% Completed | 5/109 [00:01<00:25,  4.15it/s]
Loading safetensors checkpoint shards:   6% Completed | 6/109 [00:01<00:23,  4.37it/s]
Loading safetensors checkpoint shards:   6% Completed | 7/109 [00:01<00:21,  4.67it/s]
Loading safetensors checkpoint shards:   7% Completed | 8/109 [00:01<00:21,  4.67it/s]
Loading safetensors checkpoint shards:   8% Completed | 9/109 [00:01<00:22,  4.35it/s]
Loading safetensors checkpoint shards:   9% Completed | 10/109 [00:02<00:23,  4.15it/s]
Loading safetensors checkpoint shards:  10% Completed | 11/109 [00:02<00:24,  3.97it/s]
Loading safetensors checkpoint shards:  11% Completed | 12/109 [00:02<00:24,  3.97it/s]
Loading safetensors checkpoint shards:  12% Completed | 13/109 [00:03<00:24,  3.92it/s]
Loading safetensors checkpoint shards:  13% Completed | 14/109 [00:03<00:24,  3.82it/s]
Loading safetensors checkpoint shards:  14% Completed | 15/109 [00:03<00:24,  3.90it/s]
Loading safetensors checkpoint shards:  15% Completed | 16/109 [00:03<00:24,  3.87it/s]
Loading safetensors checkpoint shards:  16% Completed | 17/109 [00:04<00:21,  4.20it/s]
Loading safetensors checkpoint shards:  17% Completed | 18/109 [00:04<00:20,  4.48it/s]
Loading safetensors checkpoint shards:  17% Completed | 19/109 [00:04<00:20,  4.49it/s]
Loading safetensors checkpoint shards:  18% Completed | 20/109 [00:04<00:20,  4.35it/s]
Loading safetensors checkpoint shards:  19% Completed | 21/109 [00:04<00:19,  4.49it/s]
Loading safetensors checkpoint shards:  20% Completed | 22/109 [00:05<00:19,  4.58it/s]
Loading safetensors checkpoint shards:  21% Completed | 23/109 [00:05<00:21,  4.01it/s]
Loading safetensors checkpoint shards:  22% Completed | 24/109 [00:05<00:21,  4.04it/s]
Loading safetensors checkpoint shards:  23% Completed | 25/109 [00:05<00:19,  4.24it/s]
Loading safetensors checkpoint shards:  24% Completed | 26/109 [00:06<00:18,  4.50it/s]
Loading safetensors checkpoint shards:  25% Completed | 27/109 [00:06<00:18,  4.40it/s]
Loading safetensors checkpoint shards:  26% Completed | 28/109 [00:06<00:18,  4.31it/s]
Loading safetensors checkpoint shards:  27% Completed | 29/109 [00:06<00:18,  4.43it/s]
Loading safetensors checkpoint shards:  28% Completed | 30/109 [00:06<00:18,  4.31it/s]
Loading safetensors checkpoint shards:  28% Completed | 31/109 [00:07<00:19,  4.09it/s]
Loading safetensors checkpoint shards:  29% Completed | 32/109 [00:07<00:18,  4.18it/s]
Loading safetensors checkpoint shards:  30% Completed | 33/109 [00:07<00:18,  4.04it/s]
Loading safetensors checkpoint shards:  31% Completed | 34/109 [00:08<00:18,  4.01it/s]
Loading safetensors checkpoint shards:  32% Completed | 35/109 [00:08<00:18,  4.00it/s]
Loading safetensors checkpoint shards:  33% Completed | 36/109 [00:08<00:18,  3.99it/s]
Loading safetensors checkpoint shards:  34% Completed | 37/109 [00:08<00:18,  3.96it/s]
Loading safetensors checkpoint shards:  35% Completed | 38/109 [00:09<00:18,  3.89it/s]
Loading safetensors checkpoint shards:  36% Completed | 39/109 [00:09<00:18,  3.78it/s]
Loading safetensors checkpoint shards:  37% Completed | 40/109 [00:09<00:18,  3.78it/s]
Loading safetensors checkpoint shards:  38% Completed | 41/109 [00:09<00:18,  3.69it/s]
Loading safetensors checkpoint shards:  39% Completed | 42/109 [00:10<00:18,  3.56it/s]
Loading safetensors checkpoint shards:  39% Completed | 43/109 [00:10<00:18,  3.55it/s]
Loading safetensors checkpoint shards:  40% Completed | 44/109 [00:10<00:17,  3.64it/s]
Loading safetensors checkpoint shards:  41% Completed | 45/109 [00:10<00:16,  3.92it/s]
Loading safetensors checkpoint shards:  42% Completed | 46/109 [00:11<00:16,  3.91it/s]
Loading safetensors checkpoint shards:  43% Completed | 47/109 [00:11<00:15,  3.98it/s]
Loading safetensors checkpoint shards:  44% Completed | 48/109 [00:11<00:14,  4.12it/s]
Loading safetensors checkpoint shards:  45% Completed | 49/109 [00:11<00:16,  3.73it/s]
Loading safetensors checkpoint shards:  46% Completed | 50/109 [00:12<00:15,  3.80it/s]
Loading safetensors checkpoint shards:  47% Completed | 51/109 [00:12<00:14,  4.05it/s]
Loading safetensors checkpoint shards:  48% Completed | 52/109 [00:12<00:13,  4.15it/s]
Loading safetensors checkpoint shards:  49% Completed | 53/109 [00:12<00:14,  3.97it/s]
Loading safetensors checkpoint shards:  50% Completed | 54/109 [00:13<00:14,  3.84it/s]
Loading safetensors checkpoint shards:  50% Completed | 55/109 [00:13<00:14,  3.70it/s]
Loading safetensors checkpoint shards:  51% Completed | 56/109 [00:13<00:14,  3.74it/s]
Loading safetensors checkpoint shards:  52% Completed | 57/109 [00:14<00:14,  3.62it/s]
Loading safetensors checkpoint shards:  53% Completed | 58/109 [00:14<00:13,  3.69it/s]
Loading safetensors checkpoint shards:  54% Completed | 59/109 [00:14<00:13,  3.76it/s]
Loading safetensors checkpoint shards:  55% Completed | 60/109 [00:14<00:13,  3.54it/s]
Loading safetensors checkpoint shards:  56% Completed | 61/109 [00:15<00:13,  3.54it/s]
Loading safetensors checkpoint shards:  57% Completed | 62/109 [00:15<00:13,  3.60it/s]
Loading safetensors checkpoint shards:  58% Completed | 63/109 [00:15<00:11,  3.93it/s]
Loading safetensors checkpoint shards:  59% Completed | 64/109 [00:15<00:11,  4.02it/s]
Loading safetensors checkpoint shards:  60% Completed | 65/109 [00:16<00:10,  4.01it/s]
Loading safetensors checkpoint shards:  61% Completed | 66/109 [00:16<00:10,  4.04it/s]
Loading safetensors checkpoint shards:  61% Completed | 67/109 [00:16<00:10,  4.00it/s]
Loading safetensors checkpoint shards:  62% Completed | 68/109 [00:16<00:10,  3.98it/s]
Loading safetensors checkpoint shards:  63% Completed | 69/109 [00:17<00:09,  4.00it/s]
Loading safetensors checkpoint shards:  64% Completed | 70/109 [00:17<00:10,  3.86it/s]
Loading safetensors checkpoint shards:  65% Completed | 71/109 [00:17<00:09,  3.84it/s]
Loading safetensors checkpoint shards:  66% Completed | 72/109 [00:17<00:10,  3.67it/s]
Loading safetensors checkpoint shards:  67% Completed | 73/109 [00:18<00:09,  3.73it/s]
Loading safetensors checkpoint shards:  68% Completed | 74/109 [00:18<00:09,  3.85it/s]
Loading safetensors checkpoint shards:  69% Completed | 75/109 [00:18<00:08,  3.85it/s]
Loading safetensors checkpoint shards:  70% Completed | 76/109 [00:19<00:08,  3.90it/s]
Loading safetensors checkpoint shards:  71% Completed | 77/109 [00:19<00:07,  4.02it/s]
Loading safetensors checkpoint shards:  72% Completed | 78/109 [00:19<00:07,  4.18it/s]
Loading safetensors checkpoint shards:  72% Completed | 79/109 [00:19<00:07,  4.26it/s]
Loading safetensors checkpoint shards:  73% Completed | 80/109 [00:19<00:07,  4.03it/s]
Loading safetensors checkpoint shards:  74% Completed | 81/109 [00:20<00:07,  3.62it/s]
Loading safetensors checkpoint shards:  75% Completed | 82/109 [00:20<00:07,  3.68it/s]
Loading safetensors checkpoint shards:  76% Completed | 83/109 [00:20<00:06,  3.72it/s]
Loading safetensors checkpoint shards:  77% Completed | 84/109 [00:21<00:06,  3.86it/s]
Loading safetensors checkpoint shards:  78% Completed | 85/109 [00:21<00:06,  3.73it/s]
Loading safetensors checkpoint shards:  79% Completed | 86/109 [00:21<00:05,  3.92it/s]
Loading safetensors checkpoint shards:  80% Completed | 87/109 [00:21<00:05,  3.79it/s]
Loading safetensors checkpoint shards:  81% Completed | 88/109 [00:22<00:05,  3.73it/s]
Loading safetensors checkpoint shards:  82% Completed | 89/109 [00:22<00:04,  4.01it/s]
Loading safetensors checkpoint shards:  83% Completed | 90/109 [00:22<00:04,  4.01it/s]
Loading safetensors checkpoint shards:  83% Completed | 91/109 [00:22<00:04,  4.06it/s]
Loading safetensors checkpoint shards:  84% Completed | 92/109 [00:23<00:04,  4.14it/s]
Loading safetensors checkpoint shards:  85% Completed | 93/109 [00:23<00:04,  3.97it/s]
Loading safetensors checkpoint shards:  86% Completed | 94/109 [00:23<00:03,  4.04it/s]
Loading safetensors checkpoint shards:  87% Completed | 95/109 [00:23<00:03,  3.97it/s]
Loading safetensors checkpoint shards:  88% Completed | 96/109 [00:24<00:03,  4.04it/s]
Loading safetensors checkpoint shards:  89% Completed | 97/109 [00:24<00:02,  4.16it/s]
Loading safetensors checkpoint shards:  90% Completed | 98/109 [00:24<00:02,  4.19it/s]
Loading safetensors checkpoint shards:  91% Completed | 99/109 [00:24<00:02,  4.13it/s]
Loading safetensors checkpoint shards:  92% Completed | 100/109 [00:25<00:02,  4.18it/s]
Loading safetensors checkpoint shards:  93% Completed | 101/109 [00:25<00:01,  4.14it/s]
Loading safetensors checkpoint shards:  94% Completed | 102/109 [00:25<00:01,  4.00it/s]
Loading safetensors checkpoint shards:  94% Completed | 103/109 [00:25<00:01,  4.03it/s]
Loading safetensors checkpoint shards:  95% Completed | 104/109 [00:25<00:01,  4.23it/s]
Loading safetensors checkpoint shards:  96% Completed | 105/109 [00:26<00:00,  4.85it/s]
Loading safetensors checkpoint shards:  98% Completed | 107/109 [00:26<00:00,  5.92it/s]
(VllmWorkerProcess pid=77) INFO 07-25 07:23:28 model_runner.py:692] Loading model weights took 57.7520 GB
Loading safetensors checkpoint shards:  99% Completed | 108/109 [00:26<00:00,  5.86it/s]
Loading safetensors checkpoint shards: 100% Completed | 109/109 [00:26<00:00,  5.86it/s]
Loading safetensors checkpoint shards: 100% Completed | 109/109 [00:26<00:00,  4.08it/s]

(VllmWorkerProcess pid=80) INFO 07-25 07:23:28 model_runner.py:692] Loading model weights took 57.7520 GB
(VllmWorkerProcess pid=81) INFO 07-25 07:23:28 model_runner.py:692] Loading model weights took 57.7520 GB
INFO 07-25 07:23:28 model_runner.py:692] Loading model weights took 57.7520 GB
(VllmWorkerProcess pid=79) INFO 07-25 07:23:28 model_runner.py:692] Loading model weights took 57.7520 GB
(VllmWorkerProcess pid=82) INFO 07-25 07:23:28 model_runner.py:692] Loading model weights took 57.7520 GB
(VllmWorkerProcess pid=78) INFO 07-25 07:23:28 model_runner.py:692] Loading model weights took 57.7520 GB
(VllmWorkerProcess pid=76) INFO 07-25 07:23:29 model_runner.py:692] Loading model weights took 57.7520 GB
INFO 07-25 07:23:35 distributed_gpu_executor.py:56] # GPU blocks: 5969, # CPU blocks: 2080
(VllmWorkerProcess pid=81) INFO 07-25 07:23:44 model_runner.py:980] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI.
(VllmWorkerProcess pid=81) INFO 07-25 07:23:44 model_runner.py:984] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing `gpu_memory_utilization` or enforcing eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
(VllmWorkerProcess pid=77) INFO 07-25 07:23:44 model_runner.py:980] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI.
(VllmWorkerProcess pid=77) INFO 07-25 07:23:44 model_runner.py:984] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing `gpu_memory_utilization` or enforcing eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
(VllmWorkerProcess pid=79) INFO 07-25 07:23:45 model_runner.py:980] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI.
(VllmWorkerProcess pid=79) INFO 07-25 07:23:45 model_runner.py:984] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing `gpu_memory_utilization` or enforcing eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
(VllmWorkerProcess pid=82) INFO 07-25 07:23:45 model_runner.py:980] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI.
(VllmWorkerProcess pid=82) INFO 07-25 07:23:45 model_runner.py:984] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing `gpu_memory_utilization` or enforcing eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
INFO 07-25 07:23:45 model_runner.py:980] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI.
INFO 07-25 07:23:45 model_runner.py:984] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing `gpu_memory_utilization` or enforcing eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
(VllmWorkerProcess pid=80) INFO 07-25 07:23:45 model_runner.py:980] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI.
(VllmWorkerProcess pid=80) INFO 07-25 07:23:45 model_runner.py:984] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing `gpu_memory_utilization` or enforcing eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
(VllmWorkerProcess pid=76) INFO 07-25 07:23:45 model_runner.py:980] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI.
(VllmWorkerProcess pid=76) INFO 07-25 07:23:45 model_runner.py:984] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing `gpu_memory_utilization` or enforcing eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
(VllmWorkerProcess pid=78) INFO 07-25 07:23:45 model_runner.py:980] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI.
(VllmWorkerProcess pid=78) INFO 07-25 07:23:45 model_runner.py:984] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing `gpu_memory_utilization` or enforcing eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
(VllmWorkerProcess pid=82) INFO 07-25 07:23:48 custom_all_reduce.py:219] Registering 1265 cuda graph addresses
(VllmWorkerProcess pid=78) INFO 07-25 07:23:48 custom_all_reduce.py:219] Registering 1265 cuda graph addresses
(VllmWorkerProcess pid=76) INFO 07-25 07:23:48 custom_all_reduce.py:219] Registering 1265 cuda graph addresses
(VllmWorkerProcess pid=80) INFO 07-25 07:23:48 custom_all_reduce.py:219] Registering 1265 cuda graph addresses
(VllmWorkerProcess pid=79) INFO 07-25 07:23:48 custom_all_reduce.py:219] Registering 1265 cuda graph addresses
INFO 07-25 07:23:48 custom_all_reduce.py:219] Registering 506 cuda graph addresses
(VllmWorkerProcess pid=77) INFO 07-25 07:23:48 custom_all_reduce.py:219] Registering 1265 cuda graph addresses
(VllmWorkerProcess pid=81) INFO 07-25 07:23:48 custom_all_reduce.py:219] Registering 1265 cuda graph addresses
Failed: Cuda error /workspace/csrc/custom_all_reduce.cuh:307 'invalid argument'
[rank4]:[W CudaIPCTypes.cpp:16] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]
Failed: Cuda error /workspace/csrc/custom_all_reduce.cuh:307 'invalid argument'
[rank0]: Traceback (most recent call last):
[rank0]:   File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
[rank0]:     return _run_code(code, main_globals, None,
[rank0]:   File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
[rank0]:     exec(code, run_globals)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/api_server.py", line 317, in <module>
[rank0]:     run_server(args)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/api_server.py", line 231, in run_server
[rank0]:     if llm_engine is not None else AsyncLLMEngine.from_engine_args(
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 466, in from_engine_args
[rank0]:     engine = cls(
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 380, in __init__
[rank0]:     self.engine = self._init_engine(*args, **kwargs)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 547, in _init_engine
[rank0]:     return engine_class(*args, **kwargs)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/llm_engine.py", line 265, in __init__
[rank0]:     self._initialize_kv_caches()
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/llm_engine.py", line 377, in _initialize_kv_caches
[rank0]:     self.model_executor.initialize_cache(num_gpu_blocks, num_cpu_blocks)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/executor/distributed_gpu_executor.py", line 62, in initialize_cache
[rank0]:     self._run_workers("initialize_cache",
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/executor/multiproc_gpu_executor.py", line 178, in _run_workers
[rank0]:     driver_worker_output = driver_worker_method(*args, **kwargs)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker.py", line 220, in initialize_cache
[rank0]:     self._warm_up_model()
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker.py", line 236, in _warm_up_model
[rank0]:     self.model_runner.capture_model(self.gpu_cache)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
[rank0]:     return func(*args, **kwargs)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/worker/model_runner.py", line 1173, in capture_model
[rank0]:     graph_runner.capture(**capture_inputs)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/worker/model_runner.py", line 1410, in capture
[rank0]:     with torch.cuda.graph(self._graph, pool=memory_pool, stream=stream):
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/torch/cuda/graphs.py", line 184, in __exit__
[rank0]:     self.cuda_graph.capture_end()
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/torch/cuda/graphs.py", line 82, in capture_end
[rank0]:     super().capture_end()
[rank0]: RuntimeError: CUDA error: out of memory
[rank0]: CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
[rank0]: For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
[rank0]: Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Failed: Cuda error /workspace/csrc/custom_all_reduce.cuh:307 'invalid argument'
ERROR 07-25 07:23:49 multiproc_worker_utils.py:120] Worker VllmWorkerProcess pid 77 died, exit code: -15
INFO 07-25 07:23:49 multiproc_worker_utils.py:123] Killing local vLLM worker processes
[rank0]:[W CudaIPCTypes.cpp:16] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]
INFO 07-25 07:24:00 api_server.py:219] vLLM API server version 0.5.3.post1
INFO 07-25 07:24:00 api_server.py:220] args: Namespace(host='<IP_ADDRESS>', port=5020, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key='EMPTY', lora_modules=None, prompt_adapters=None, chat_template=None, response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], model='meta-llama/Meta-Llama-3.1-405B-Instruct-FP8', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, download_dir='/home/ubuntu/.cache/huggingface/hub', load_format='auto', dtype='auto', kv_cache_dtype='auto', quantization_param_path=None, max_model_len=8192, guided_decoding_backend='outlines', distributed_executor_backend=None, worker_use_ray=False, pipeline_parallel_size=1, tensor_parallel_size=8, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=16, enable_prefix_caching=False, disable_sliding_window=False, use_v2_block_manager=False, num_lookahead_slots=0, seed=1234, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.99, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_seqs=16, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, enforce_eager=False, max_context_len_to_capture=None, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, enable_lora=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', scheduler_delay_factor=0.0, enable_chunked_prefill=True, speculative_model=None, num_speculative_tokens=None, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=None, qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, engine_use_ray=False, disable_log_requests=False, max_log_len=100)
(base) ubuntu@compute-permanent-node-406:~$

Your last error is caused by cuda graph not having enough memory actually, not kv cache.

So it's like the message is not true.

I guess this confirms that on H100 you can't really realistically get full 128k context length (and the highest you can hit here is 89k) unless we change something in the profiling run. I will bring it to the core dev team to discuss - feel free to share other observations you have!

So in summary, seems something is off with prefill.  It doesn't actually set --max-num-batched-tokens=512 despite saying so.  But if I manually set it, then I can load with 16 sequences (eager or not doesn't help) up to about 64K (which is better than 8k).
But I'd really like to use the full context.  Is there an issue with cuda driver 12.2 vs. 12.4+ or something?  What is missing here?
At minimum there seems bug with the auto-setting of things.

Worth cross-referencing with #6689

Update: With 16->8 reduction, I'm able to use full 128k context with FP8 405B on 8*H100 without hitting the k-v cache limit.  So that "20%" lower memory usage definitely helped with this.


Allow user to delete note

User should be able to delete notes by pressing Shift + Delete. If it's a new note, it should simply delete the HTML, else it should delete the note in the database as well.

Done, over! Also asking the user to confirm before deletion.


oml task_type inconsistent

The results of tasktype get by id and list task types are inconsistent.
Both return oml:task_type.
The list query does not return some fields, however it return creators. The query by id returns contributors, not creators and it returns also fields like output and input.
Furthemore, the creators are also not consistent, sometimes it is Name1, Name2,
other time it is "Name1","Name2".
Make this more consistent.
Reference:
http://www.openml.org/api_new/v1/tasktype/1
http://www.openml.org/api_new/v1/tasktype/list

This is no website issue.
However, I think this is even solved.

It is not solved as you can easily check by visiting the links.
How is this not a website issue? Isnt the API part of the website?
Thanks,
Jakub

Technically yes, but API issues are typically reported in the general issue
tracker. I'll fix this soon.
On Mon, 19 Sep 2016 at 19:40, Jakub Šmíd<EMAIL_ADDRESS>wrote:

It is not solved as you can easily check by visiting the links.
How is this not a website issue? Isnt the API part of the website?
Thanks,
Jakub
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/openml/website/issues/30#issuecomment-248064744, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABpQV9lgZJdx7A_qwID7ls5rixg6jM7Kks5qrskSgaJpZM4GL5qW
.

I see, I will report API related bugs in the general issue tracker further on.
Let me know if I can be of any help.
Regards,
Jakub


Modify pagination in page of commits

Re. issue #3716
I wanted to check if the changes that I have made are viewed as acceptable, because I observed that the code that I indirectly refer to (namely, paginate.tmpl )was last modified in the year 2017 A.D. (and the format of the hyperlinks may have changed since then, if you know what I am referring to.)
Here's a screenshot: (I draw your attention to the navigation at the bottom of the image)

Codecov Report

Merging #3766 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #3766   +/-   ##
=======================================
  Coverage   22.91%   22.91%
=======================================
  Files         124      124
  Lines       24750    24750
=======================================
  Hits         5672     5672
  Misses      18216    18216
  Partials      862      862

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7000b41...fc6531b. Read the comment docs.

Thank you @bkcsoft , @lafriks , @appleboy , @lunny , @JonasFranzDEV , @Bwko for your effort on the PR. (I listed your user-names in no particular order, to clarify).


Should we retry to open cache span on signal interrupt or out of memory

We currently retry on these error cases in Span::init, because SocketManager::open retries them, and as far as I can tell, nowhere else.
https://github.com/apache/trafficserver/blob/89264ff7da0ae5a2e6db9197c75d248b6bd9c3b9/src/iocore/cache/Store.cc#L370-L375
These are the cases in which the open() call will be retried:
https://github.com/apache/trafficserver/blob/89264ff7da0ae5a2e6db9197c75d248b6bd9c3b9/src/iocore/eventsystem/UnixSocket.h#L297-L308

This is resolved by https://github.com/apache/trafficserver/pull/11526 .


Ubuntu 16.04.201809120

What this PR does / why we need it: Freshen Ubuntu 16.04 LTS image for "ubuntu" distro option.
Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #
Special notes for your reviewer:
If applicable:

[ ] documentation
[ ] unit tests
[ ] tested backward compatibility (ie. deploy with previous version, upgrade with this branch)

Release note:

Ubuntu 16.04.201809120

/lgtm

/lgtm


Prevent NullPointerException

This patch prevents a NullPointerException whenever someone would access
the Annotation Tool with a media package identifier unknown to Opencast.

This is a change specific to the SWITCH version of the code. It is merged there and will land here if/when we get to backporting their customizations.


[OPENAI] Json parse exception while request OpenAI API in java

Hi Microsoft Dev Team,
I had an exception when i requested to your openai azure api.
I used the sample code
final List messages = new ArrayList<>();
messages.add(new.ChatMessage(ChatRole.USER).setContent("tôi cần gì đâu"));
com.azure.ai.openai.models.ChatCompletionsOptions options = new com.azure.ai.openai.models.ChatCompletionsOptions(messages);
options.setModel("gpt-35-turbo");
options.setStream(false);
options.setN(1);
options.setMaxTokens(256);
options.setLogitBias(new HashMap<>());

ChatCompletions chatCompletions = client.getChatCompletions("***", options);
Exception or Stack Trace
09:14:44.921 [main] ERROR com.azure.core.implementation.serializer.DefaultJsonSerializer - com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'data': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false') at [Source: (byte[])"data: {"id":"chatcmpl-7YPqqfztcROs341l4dOLDvFXvFOlv","object":"chat.completion.chunk","created":1688436880,"model":"gpt-35-turbo","choices":[{"index":0,"finish_reason":null,"delta":{"role":"assistant"}}],"usage":null}data: {"id":"chatcmpl-7YPqqfztcROs341l4dOLDvFXvFOlv","object":"chat.completion.chunk","created":1688436880,"model":"gpt-35-turbo","choices":[{"index":0,"finish_reason":null,"delta":{"content":"L"}}],"usage":null}data: {"id":"chatcmpl-7YPqqfztcROs341l4dOLDvFXvFOlv","object":"chat"[truncated 26688 bytes]; line: 1, column: 6]
java.io.UncheckedIOException: com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'data': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
at [Source: (byte[])"data: {"id":"chatcmpl-7YPqqfztcROs341l4dOLDvFXvFOlv","object":"chat.completion.chunk","created":1688436880,"model":"gpt-35-turbo","choices":[{"index":0,"finish_reason":null,"delta":{"role":"assistant"}}],"usage":null}
data: {"id":"chatcmpl-7YPqqfztcROs341l4dOLDvFXvFOlv","object":"chat.completion.chunk","created":1688436880,"model":"gpt-35-turbo","choices":[{"index":0,"finish_reason":null,"delta":{"content":"L"}}],"usage":null}
data: {"id":"chatcmpl-7YPqqfztcROs341l4dOLDvFXvFOlv","object":"chat"[truncated 26688 bytes]; line: 1, column: 6]
at com.azure.core.implementation.serializer.DefaultJsonSerializer.deserializeFromBytes(DefaultJsonSerializer.java:34)
at com.azure.core.implementation.util.FluxByteBufferContent.toObject(FluxByteBufferContent.java:94)
at com.azure.core.util.BinaryData.toObject(BinaryData.java:1241)
at com.azure.core.util.BinaryData.toObject(BinaryData.java:1003)
at com.azure.ai.openai.OpenAIClient.getChatCompletions(OpenAIClient.java:384) @

Thanks @alzimmermsft
@cuongnguyengit Thank you for reaching out to us. I won't be able to reproduce the error locally. Can you give more information, such as what version of SDK you are running and what is the complete sample you are using?
Given my local sample below and output for reference.
     OpenAIClient client = new OpenAIClientBuilder()
                .endpoint(endpoint)
                .credential(new AzureKeyCredential(azureOpenaiKey))
                .buildClient();

        final List messages = new ArrayList<>();
        messages.add(new ChatMessage(ChatRole.USER).setContent("tôi cần gì đâu"));
        ChatCompletionsOptions options = new ChatCompletionsOptions(messages)
                .setModel("gpt-35-turbo") // which can be removed.
                .setStream(false)
                .setN(1)
                .setMaxTokens(256)
                .setLogitBias(new HashMap<>());

        ChatCompletions chatCompletions = client.getChatCompletions("gpt-35-turbo", options);

        System.out.printf("Model ID=%s is created at %d.%n", chatCompletions.getId(), chatCompletions.getCreated());
        for (ChatChoice choice : chatCompletions.getChoices()) {
            ChatMessage message = choice.getMessage();
            System.out.printf("Index: %d, Chat Role: %s.%n", choice.getIndex(), message.getRole());
            System.out.println("Message:");
            System.out.println(message.getContent());
        }

        System.out.println();
        CompletionsUsage usage = chatCompletions.getUsage();
        System.out.printf("Usage: number of prompt token is %d, "
                        + "number of completion token is %d, and number of total tokens in request and response is %d.%n",
                usage.getPromptTokens(), usage.getCompletionTokens(), usage.getTotalTokens());

Output:
Model ID=chatcmpl-7Z3fwFiC3WNw0GE5QPi0WVQgsZiBI is created at<PHONE_NUMBER>.
Index: 0, Chat Role: assistant.
Message:
Là trợ lý ảo, tôi rất mong muốn được hỗ trợ bạn. Vui lòng cho tôi biết cần hỗ trợ gì, tôi sẽ cố gắng hết sức để giúp đỡ bạn.

Usage: number of prompt token is 18, number of completion token is 68, and number of total tokens in request and response is 86.

Vào Th 5, 6 thg 7, 2023 lúc 03:46 Shawn Fang @.***> đã
viết:

Thanks @alzimmermsft https://github.com/alzimmermsft
@cuongnguyengit https://github.com/cuongnguyengit Thank you for
reaching out to us. I won't be able to reproduce the error locally. Can you
give more information, such as what version of SDK you are running and what
is the complete sample you are using?
Given my local sample below and output for reference.
 OpenAIClient client = new OpenAIClientBuilder()
            .endpoint(endpoint)
            .credential(new AzureKeyCredential(azureOpenaiKey))
            .buildClient();

    final List messages = new ArrayList<>();
    messages.add(new ChatMessage(ChatRole.USER).setContent("tôi cần gì đâu"));
    ChatCompletionsOptions options = new ChatCompletionsOptions(messages)
            .setModel("gpt-35-turbo") // which can be removed.
            .setStream(false)
            .setN(1)
            .setMaxTokens(256)
            .setLogitBias(new HashMap<>());

    ChatCompletions chatCompletions = client.getChatCompletions("gpt-35-turbo", options);

    System.out.printf("Model ID=%s is created at %d.%n", chatCompletions.getId(), chatCompletions.getCreated());
    for (ChatChoice choice : chatCompletions.getChoices()) {
        ChatMessage message = choice.getMessage();
        System.out.printf("Index: %d, Chat Role: %s.%n", choice.getIndex(), message.getRole());
        System.out.println("Message:");
        System.out.println(message.getContent());
    }

    System.out.println();
    CompletionsUsage usage = chatCompletions.getUsage();
    System.out.printf("Usage: number of prompt token is %d, "
                    + "number of completion token is %d, and number of total tokens in request and response is %d.%n",
            usage.getPromptTokens(), usage.getCompletionTokens(), usage.getTotalTokens());

Output:
Model ID=chatcmpl-7Z3fwFiC3WNw0GE5QPi0WVQgsZiBI is created at<PHONE_NUMBER>.
Index: 0, Chat Role: assistant.
Message:
Là trợ lý ảo, tôi rất mong muốn được hỗ trợ bạn. Vui lòng cho tôi biết cần hỗ trợ gì, tôi sẽ cố gắng hết sức để giúp đỡ bạn.
Usage: number of prompt token is 18, number of completion token is 68, and number of total tokens in request and response is 86.
—
Reply to this email directly, view it on GitHub
https://github.com/Azure/azure-sdk-for-java/issues/35734#issuecomment-1622478324,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AKZGI7OON63B7GTRCMYUR53XOXHDVANCNFSM6AAAAAAZ5PCQXQ
.
You are receiving this because you were mentioned.Message ID:
@.***>

Sorry, I found a bug in my code and had no choice send it to you. I set
True for stream.
Thanks for your support.

@cuongnguyengit Please don't use setStream() method, which will be deleted in the future.  Please use the streaming API instead.
See sample: https://github.com/Azure/azure-sdk-for-java/blob/main/sdk/openai/azure-ai-openai/src/samples/java/com/azure/ai/openai/usage/GetChatCompletionsStreamSample.java#L50

Vào Th 5, 6 thg 7, 2023 lúc 06:25 Shawn Fang @.***> đã
viết:

@cuongnguyengit https://github.com/cuongnguyengit Please don't use
setStream() method, which will be deleted in the future. Please use the
streaming API instead.
See sample:
https://github.com/Azure/azure-sdk-for-java/blob/main/sdk/openai/azure-ai-openai/src/samples/java/com/azure/ai/openai/usage/GetChatCompletionsStreamSample.java#L50
—
Reply to this email directly, view it on GitHub
https://github.com/Azure/azure-sdk-for-java/issues/35734#issuecomment-1622670417,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AKZGI7LHSHYHBD6JDIK4OVDXOXZU3ANCNFSM6AAAAAAZ5PCQXQ
.
You are receiving this because you were mentioned.Message ID:
@.***>

Maybe I used old docs.
Thanks.


Wheels are now os-dependent but python-independent

This builds OS-dependent but Python-independent wheels.
I tested this on OSX, Win32, and Win64 and the wheels work fine.
The main point is that this should also work for python setup.py bdist_wheel upload.

That's great!
I thought we would have to use some ugly file-renaming script ... this is so much better!
I added a few remarks to the code.

The main point is that this should also work for python setup.py bdist_wheel upload

Note, however, that according to https://pypi.python.org/pypi/twine this exposes your credentials and you should use twine ...
But perhaps this is resolved already?

Thank you for all of these comments. I pretty much agree with all of them, and will implement them tomorrow.

This should incorporate all the changes you suggested. Please merge if you think it's ready.

BTW, I love the name bdist_wheel_half_pure! Thank you for coming up with that!

Great, thank you!
Just a teensy little thing:
Don't you like the else block after try/except?
I think it's better style to keep the try block as short as possible and have it only contain stuff that actually can throw ...
But if you dislike else, that's also OK with me ...


1.9.3 Nolonger connects using keys

I can no longer connect using the stored keys. I've tried recreating and using different key lengths but to no avail

Sorry, bug with pre-API 19 versions. Rolling out a fix now; follow #654

The last version that connect with my key only servers is v1.9.2 (git-v1.9.2 19 Feb 2017) every update since made the same errors (same with latest git-v1.9.4 from minutes ago) :

On the servers side noticeable settings are :
PasswordAuthentication no
ChallengeResponseAuthentication no

Interesting. What type of key are you using?

RSA 1024 SHA256, I'll try to generate a new one and let you know.

Juste tried with a brand new 4096 RSA key (generated with ssh-keygen -b 4096), same error (works with 1.9.2, not with higher version).

This might be related to https://github.com/openssh/openssh-portable/commit/4ba0d54794814ec0de1ec87987d0c3b89379b436
See https://github.com/open-keychain/sshlib/commit/686e8d44d80ea65167972f88a0a9c440559e5fd4 for a fix

If it can helps here is a screenshot of a successfull connection with v1.9.2 (in french, sorry) :

Logs on servers are not very useful :

v1.9.2, Accepted publickey then session opened,
v1.9.4, nothing when I connect and Received disconnect from then Disconnected from authenticating user

Ah, looks like it only fails with OpenSSH 7.8 and newer. The integration tests were running OpenSSH 7.7.

Should be fixed by 18e8fe030eb45d7606adff22292e2b0b6cc5d1d4.

Should be fixed by 18e8fe0.

I do confirm 1.9.5 works (OpenSSH 7.9)


Add editor localization support (WIP)

Adding editor localization support.

Specific language in editor setting

Reading translation strings from [editor setting folder]/lang/[locale]/editor.po

pls blame Goolge for bad translation :)

po template generated from xgettext and translated in POEditor

Bump. How is work on editor localization going (if any)? Is this pull request even mergeable today?

Planned for 1.2
On Apr 5, 2015 5:14 PM, "Calinou"<EMAIL_ADDRESS>wrote:

Bump. How is work on editor localization going (if any)? Is this pull
request even mergeable today?
—
Reply to this email directly or view it on GitHub
https://github.com/okamstudio/godot/pull/287#issuecomment-89846427.

Just so you know, that Japanese is cavespeak.

What's the status on this @reduz? You said it was planned for 1.2, do you still intend to get it into 2.0 or will it have to wait for 2.1 or later (as we're now trying to go towards releasing 2.0 final in the coming months)?

Rebased on current master.

Looks like @reduz went for his own implementation in the master branch, so I guess this PR must be closed. Thanks for the work you did though, I do hope it was used as an inspiration for the implemented system :)


Issue with setting the seed for experiment replication

I believe  ' random.seed() '  should be removed in all of the following files. Its use sets the seed to some random value every time the library is used. Because of this, it is impossible to replicate any results exactly. Instead, user should be able to specify the desired seed before the library is used - and package should not override his choice.
List of files:
libpgm/CPDtypes/discrete.py
libpgm/CPDtypes/lg.py
libpgm/CPDtypes/lgandd.py
libpgm/hybayesiannetwork.py
libpgm/lgbayesiannetwork.py
libpgm/discretebayesiannetwork.py
libpgm/dyndiscbayesiannetwork.py
libpgm/tablecpdfactorization.py

I agree, assuming there is an option to get a random seed if desired.  If you make a pull request, I will marge your changes in.


Allow addressbook to be source of truth for a token's oracleId

Any oracleId from a token in the addressbook will now take precedents over any set in any config file e.g. vaults json.
(Excluding native tokens which are loaded from main config)

IDK why fleek hasn't picked this up


Languages: German

Translate bot to german language.

This will be done after adding the reminder command, #8


Changed build task to remove dangling &

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[x] I have read the CONTRIBUTING document.

@lazd so the notnested() wasn't cleaning out the rulesets in the standalone outputs. I added the plugin to those tasks, but is there a better place to do it since postcss has already been run by that point?
Anyway, this works, I'm just not sure it's the best way to fix the problem.

This would close https://github.com/adobe/spectrum-css/issues/61

Turns out it was a bigger problem than just the & { - details in #61


1.6.2 release

Target date: Thursday 10/26.
Any other bug fixes you'd like to make @snorfalorpagus ?

Done! Share and enjoy: https://pypi.python.org/pypi/Shapely/1.6.2

maint-1.6 has been merged into master.

Shapely wheels on PyPI were made using version 1.5.0 of frs-wheel-builds, just like 1.6.1. This means GEOS 3.6.2: https://github.com/sgillies/frs-wheel-builds/blob/master/CHANGES.txt#L4.
The GEOS shared libraries in the linux wheels are huge for reasons I don't understand.
$ tar tzvf dist/Shapely-1.6.2-cp36-cp36m-manylinux1_x86_64.whl | grep libgeos
-rwxrwxrwx  0 0      0     1558288 Oct 26 19:38 shapely/.libs/libgeos_c-8e89822e.so.1.10.2
-rwxrwxrwx  0 0      0    22463536 Oct 26 19:38 shapely/.libs/libgeos-3-247346df.6.2.so


"This Key doesn't have any Mail Addresses"

Hi,
something is wrong with encrypt.to. Every key i tried brings an error. Even your own.
(Android 7, different browsers)

3 days later. It works again.
Thanks for such great communication. 😟

I think it is the same issue I had:
https://github.com/encrypt-to/encrypt.to/issues/48

Ja ist es. Key Server waren aber online in den 3 Tagen.
Yes, it is. But the key server were online during these 3 days.

Yes the problem is: if the service is not reliable, the people will not use it.

And it is brocken again. A Joke!

We changed the sks-keyserver pool, seems http://ha.pool.sks-keyservers.net  was unstable in the last month.

We found the reason, a key server in the pool changed the API, hence parsing the API response failed.
https://pgp.key-server.io/pks/lookup?op=vindex&search=0x0caf1e5b11489a1f&fingerprint=on&options=mr
Again the changed the pool and contacted the admin.

Yes, it works. Thanks.
btw, great service. 👍


🐛Fix argstable sync with useArgs

Resolves #1825

Table seems to be a separate issue from this to me. is density event supposed to be a prop for table? Is it not supposed to use edsprovider like everywhere else?

Table seems to be a separate issue from this to me. is density event supposed to be a prop for table? Is it not supposed to use edsprovider like everywhere else?

Good point, its not. Maybe we can make a tailored storybook component for all our "compact" stories? Either way, a separate issue :)

Why did you use @storybooj/client-api instead of @storybook/api? The first seems to be deprecated and is not mentioned in the storybook documentation.

Something i spent much of friday trying to figure out. The /api one simply will not work (TypeError: Cannot read properties of undefined (reading 'getCurrentStoryData')) whereas the "deprecated" /client-api one works fine and is what everyone seems to use if you search for import { useArgs } from '@storybook/client-api' vs import { useArgs } from '@storybook/api' on github

Why did you use @storybooj/client-api instead of @storybook/api? The first seems to be deprecated and is not mentioned in the storybook documentation.

Something i spent much of friday trying to figure out. The /api one simply will not work (TypeError: Cannot read properties of undefined (reading 'getCurrentStoryData')) whereas the "deprecated" /client-api one works fine and is what everyone seems to use if you search for import { useArgs } from '@storybook/client-api' vs import { useArgs } from '@storybook/api' on github

Fair enough. Maybe its fixed in the future, lets keep an eye out 👀


python310Packages.pysaml2: 7.1.1 -> 7.1.2

Automatic update generated by nixpkgs-update tools. This update was made based on information from passthru.updateScript.
meta.description for python310Packages.pysaml2 is: "Python implementation of SAML Version 2 Standard"
meta.homepage for python310Packages.pysaml2 is: "https://github.com/IdentityPython/pysaml2"
meta.changelog for python310Packages.pysaml2 is: ""
Updates performed

Ran passthru.UpdateScript

To inspect upstream changes

Release on GitHub

Compare changes on GitHub

Impact

Checks done (click to expand)

built on NixOS
Warning: a test defined in passthru.tests did not pass
Warning: no invocation of /nix/store/ly0rf04mvrf0x7r2a4s2arv3mmdy31k6-python3.10-pysaml2-7.1.2/bin/.merge_metadata.py-wrapped had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/ly0rf04mvrf0x7r2a4s2arv3mmdy31k6-python3.10-pysaml2-7.1.2/bin/make_metadata.py had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/ly0rf04mvrf0x7r2a4s2arv3mmdy31k6-python3.10-pysaml2-7.1.2/bin/.mdexport.py-wrapped had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/ly0rf04mvrf0x7r2a4s2arv3mmdy31k6-python3.10-pysaml2-7.1.2/bin/merge_metadata.py had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/ly0rf04mvrf0x7r2a4s2arv3mmdy31k6-python3.10-pysaml2-7.1.2/bin/mdexport.py had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/ly0rf04mvrf0x7r2a4s2arv3mmdy31k6-python3.10-pysaml2-7.1.2/bin/.make_metadata.py-wrapped had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/ly0rf04mvrf0x7r2a4s2arv3mmdy31k6-python3.10-pysaml2-7.1.2/bin/.parse_xsd2.py-wrapped had a zero exit code or showed the expected version
Warning: no invocation of /nix/store/ly0rf04mvrf0x7r2a4s2arv3mmdy31k6-python3.10-pysaml2-7.1.2/bin/parse_xsd2.py had a zero exit code or showed the expected version
0 of 8 passed binary check by having a zero exit code.
0 of 8 passed binary check by having the new version present in output.
found 7.1.2 with grep in /nix/store/ly0rf04mvrf0x7r2a4s2arv3mmdy31k6-python3.10-pysaml2-7.1.2
found 7.1.2 in filename of file in /nix/store/ly0rf04mvrf0x7r2a4s2arv3mmdy31k6-python3.10-pysaml2-7.1.2

Rebuild report (if merged into master) (click to expand)

10 total rebuild path(s)

5 package rebuild(s)

5 x86_64-linux rebuild(s)
5 x86_64-darwin rebuild(s)

First fifty rebuilds by attrpath
matrix-synapse
matrix-synapse-plugins.matrix-synapse-mjolnir-antispam
matrix-synapse-plugins.matrix-synapse-shared-secret-auth
python310Packages.pysaml2
python39Packages.pysaml2

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/ly0rf04mvrf0x7r2a4s2arv3mmdy31k6-python3.10-pysaml2-7.1.2 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A python310Packages.pysaml2 https://github.com/r-ryantm/nixpkgs/archive/b7c8a02354155e5a0311bc6668fddf43e670d0f4.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/ly0rf04mvrf0x7r2a4s2arv3mmdy31k6-python3.10-pysaml2-7.1.2
ls -la /nix/store/ly0rf04mvrf0x7r2a4s2arv3mmdy31k6-python3.10-pysaml2-7.1.2/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

5 packages built:

matrix-synapse
matrix-synapse-plugins.matrix-synapse-mjolnir-antispam
matrix-synapse-plugins.matrix-synapse-shared-secret-auth
python310Packages.pysaml2
python39Packages.pysaml2

Maintainer pings

Result of nixpkgs-review pr 162877 run on x86_64-linux 1

5 packages built:

matrix-synapse
matrix-synapse-plugins.matrix-synapse-mjolnir-antispam
matrix-synapse-plugins.matrix-synapse-shared-secret-auth
python310Packages.pysaml2
python39Packages.pysaml2


Port handling is confusing when web is not first in Procfile

In my Procfile I had the following:
release: rake db:migrate
web: bundle exec rails s -p $PORT

I then ran
$ myapp config:get PORT
6000

I found that when my app was scaled to 1, every time I ran systemctl status myapp-web-1 I would get
* Listening on tcp://<IP_ADDRESS>:6100

I then changed my Procfile to
web: bundle exec rails s -p $PORT
release: rake db:migrate

And got
* Listening on tcp://<IP_ADDRESS>:6000

My question here is how do I go about figuring out what port my app is up and running on so I can point a reverse proxy at it? It seems like changing the order of the Procfile should not cause the port number to jump around like it does.

The best thing I can think of is at least having an interface to find out what port a service is running on. The only reason I was able to realize what was happening was due to the fact that I checked systemctl status myservice-web-1 and saw it was coming up on not the port I expected. If I could've said myservice config to view current port bindings for specific services, I believe that would've been pretty helpful.


vscan bugs

There are the list of vscan bugs we found.

[ ]  Determine the 'MaxHeight' when a low ceiling above our Prius is present.

Can you also email to Alex?
2016/05/21 17:14 "Hiroki Ohta"<EMAIL_ADDRESS>
There are the list of vscan bugs we found.

Determine the 'MaxHeight' when a low ceiling above our Prius is
present.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
https://github.com/CPFL/Autoware/issues/317

Is this fixed?

Hi, I received this issue in May. This bug requires a significant modification, but it only happens with a very low ceiling above Prius. So we put a to-do event here as a reminder for us to solve this bug later.
The height information is a "byproduct" in VScan algorithm; therefore, the determination of the "MaxHeight" is very simple and only takes ordinary situations into consideration.