[Feature] Running multi-node offline engine inference ( via SLURM)

Checklist

[X] 1. If the issue you raised is not a feature but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
[X] 2. Please use English, otherwise it will be closed.

Motivation
A lot of academic institutions only allow access to larger node clusters via SLURM and it is not immediately clear how would I reuse the code to run Llama 405B BF16 on 2 nodes (by starting a server) to perform offline inference
Related resources
No response

@aflah02 Thanks for pointing this out. We are looking for contributors since we do not use slurm for long. 😂

@zhaochenyang20 If you have any pointers on how you might approach the problem I can take a stab at this. The issue right now is that I have 0 clue on how to get started with using either the runtime api or the engine api for multinode. They don't seem to support pipeline parallel so the only method seems to be tensor parallel across all GPUs but if I say use 16 GPUs it can't do that directly as it only sees 8 GPUs per node

I was thinking of using the Engine API and just converting all server args from the CLI commands but then my question would be that in the CLI version you run 2 commands one per node, how would you do that here via the Engine API. Do you run 2 engine calls (one on each node)?

Btw this is one of my attempts to just load the model but nothing seems to run on the worker node as I only see logs for head node -
#!/bin/bash -l

#SBATCH -o SLURM_Logs/%x_%j_node%t.out
#SBATCH -e SLURM_Logs/%x_%j_node%t.err
#SBATCH -D ./
#SBATCH -J 405B-FP8-Online-TP16-Sglang

#SBATCH --nodes=2
#SBATCH --tasks-per-node=1
#SBATCH --cpus-per-task=18
#SBATCH --mem=224GB

#SBATCH --partition="h100"
#SBATCH --gres=gpu:h100:8

#SBATCH --time=12:00:00

# Load required modules or set environment variables if needed
echo "[INFO] Activating environment on node $SLURM_NODEID"
source /NS/venvs/work/afkhan/slurm_sglang_env/bin/activate || { echo "[ERROR] Failed to activate environment"; exit 1; }

# Define parameters
model="/scratch/sws0/user/afkhan/Models/Llama-3.1-405B-Instruct-FP8"
tp_size=16
prompt_path="/NS/llm-pretraining/work/afkhan/vLLM-Serving-POC/Data/Prompt_OHB_Chat_Alpha.txt"
save_path="/NS/llm-pretraining/work/afkhan/vLLM-Serving-POC/Data/Outputs_Llama-3.1-405B-Instruct-FP8_OHB_Chat_Alpha_tp_16_sgl.json"
queries_path="/NS/llm-pretraining/work/afkhan/vLLM-Serving-POC/Data/FAQ_en.csv"

echo "[INFO] Running inference"
echo "[INFO] Model: $model"
echo "[INFO] TP Size: $tp_size"
echo "[INFO] Save Path: $save_path"
echo "[INFO] Queries Path: $queries_path"

# Define the NCCL init address using the hostname of the head node
HEAD_NODE=$(scontrol show hostname "$SLURM_NODELIST" | head -n 1)
NCCL_INIT_ADDR="${HEAD_NODE}:8000"
echo "[INFO] NCCL_INIT_ADDR for node $SLURM_NODEID: $NCCL_INIT_ADDR"

# Launch processes
if [ "$SLURM_NODEID" -eq 0 ]; then
    echo "[INFO] Launching head node process on $HOSTNAME"
    python3 -m sglang.launch_server \
        --model-path "$model" \
        --tp "$tp_size" \
        --nccl-init-addr "$NCCL_INIT_ADDR" \
        --nnodes 2 \
        --node-rank 0 &

    echo "[INFO] Head node process launched"
elif [ "$SLURM_NODEID" -eq 1 ]; then
    echo "[INFO] Launching worker node process on $HOSTNAME"
    python3 -m sglang.launch_server \
        --model-path "$model" \
        --tp "$tp_size" \
        --nccl-init-addr "$NCCL_INIT_ADDR" \
        --nnodes 2 \
        --node-rank 1 &

    echo "[INFO] Worker node process launched"
else
    echo "[ERROR] Unexpected SLURM_NODEID: $SLURM_NODEID"
fi

echo "[INFO] Waiting for all processes to complete on node $SLURM_NODEID"
wait
echo "[INFO] Processes completed on node $SLURM_NODEID"

Good points. We do not support pipeline parallelism but I do not think this would block the progress of running on slurm. Our team will discuss your issue this Friday. Before that, could you try out some quantization method for llama 405B? Or you can use llama 3.3 70B, which is pretty good.

BTW, would you like to join our bi-weekly meeting this Saturday?

Thanks for the invite
I'll be traveling over the weekend and it would be hard to join the meeting
I've already had success with running the FP8 version as well as the 70B one on a single node for offline inference. So the only thing left is to go multinode for the BF16 version + to get bigger context length for the FP8 version

Great! How do you run the FP8 version of the 70B model? I think the best way is to first quantize it and then load it, rather than quantizing it online. @aflah02

Great! How do you run the FP8 version of the 70B model? I think the best way is to first quantize it and then load it, rather than quantizing it online. @aflah02

Sorry for not being clear. I've run 2 models - 70B in BF16 and 405B in FP8. I'm not running 70B in FP8.
My goal now is to somehow run 405B in FP16

Yeah. I see. We will discuss this in our weekly meeting on this. BTW, how you quantize 405B model on fp8?

Yeah. I see. We will discuss this in our weekly meeting on this. BTW, how you quantize 405B model on fp8? @aflah02

Thanks that would be awesome!
For FP8 I'm directly using the official meta weights for FP8 and using that as model path when loading the model. I'm not doing any quantization myself but I do remember seeing some logs from SGLang about it doing some quantization related stuff

Cool. Thanks for pointing this out. @jamesand and I are working on quantization documentation. We will record that "use official repo first" 😂

For the slurm issue, let me update this week. If I haven't replied before next week, please reply to this issue and remind me. Thanks! @aflah02

Just for reference this is the current script which works well on 1 node -
Python file -
import sglang as sgl
import argparse
import pandas as pd
import json

def main(args):
    print("Arguments: ", args)
    print("Loading model...")

    runtime = sgl.Runtime(
        model_path=args.model,
        tokenizer_path=args.model,
        tp_size=args.tp_size,  # t_ensor p_arallel size, number of GPUs to split the model over
        log_level="error" ,
        random_seed = 20242
    )

    sgl.set_default_backend(runtime)

    print("Model loaded.")

    temperature = 1
    top_p = 1
    top_k = -1
    min_p = 0
    max_tokens = 20000
    message_role = 'assistant'

    @sgl.function
    def conv_generate(s, system_prompt, old_messages, new_message, max_tokens, message_role):
        s += sgl.system(system_prompt)
        for om in old_messages:
            content = om['content']
            role = om['role']
            if role == 'user':
                s += sgl.user(content)
            elif role == 'assistant':
                s += sgl.assistant(content)
        if message_role == 'user':
            s += sgl.user(new_message + sgl.gen("response", max_tokens=max_tokens))
        elif message_role == 'assistant':
            s += sgl.assistant(new_message + sgl.gen("response", max_tokens=max_tokens))

    # Read Queries
    df = pd.read_csv(args.queries_path)

    queries = df["Question"].tolist()

    # Read Prompt
    with open(args.prompt_path, "r") as f:
        system_prompt = f.read()

    completion_texts = []

    for query in queries:
        state = conv_generate.run(
            system_prompt = system_prompt,
            old_messages = [
                {'content': query, 'role': 'user'}
            ],
            new_message = '', # Start with empty message
            max_tokens=max_tokens,
            message_role=message_role,
            temperature=temperature,
            top_p=top_p,
            top_k=top_k,
            min_p=min_p,
        )
        model_generation  = state['response']
        completion_texts.append(model_generation)
        print("Ran query: ", query)
        print("Model generation: ", model_generation)

    # Save completions
    with open(args.save_path, "w") as f:
        json.dump(completion_texts, f)

if __name__ == "__main__":
    argparser = argparse.ArgumentParser()
    argparser.add_argument("--model", type=str)
    argparser.add_argument("--tp_size", type=int)
    argparser.add_argument("--save_path", type=str)
    argparser.add_argument("--queries_path", type=str)
    argparser.add_argument("--prompt_path", type=str)
    args = argparser.parse_args()
    main(args)

SLURM bash file -
#!/bin/bash -l

#SBATCH -o SLURM_Logs/%x_%j_%A-%T.out
#SBATCH -e SLURM_Logs/%x_%j_%A-%T.err
#SBATCH -D ./
#SBATCH -J 405B-FP8-Off-Sglang

#SBATCH --nodes=1
#SBATCH --tasks-per-node=1
#SBATCH --cpus-per-task=18
#SBATCH --mem=224GB

#SBATCH --partition="h100"
#SBATCH --gres=gpu:h100:8

# Wall clock limit (max. is 24 hours):
#SBATCH --time=12:00:00

# Load required modules or set environment variables if needed
source /NS/venvs/work/afkhan/slurm_sglang_env/bin/activate

model="/scratch/sws0/user/afkhan/Models/Llama-3.1-405B-Instruct-FP8"
tp_size=8
prompt_path="/NS/llm-pretraining/work/afkhan/vLLM-Serving-POC/Data/Prompt_OHB_Chat_Alpha.txt"
save_path="/NS/llm-pretraining/work/afkhan/vLLM-Serving-POC/Data/Outputs_Llama-3.1-405B-Instruct-FP8_OHB_Chat_Alpha_tp_8_sgl_fixed.json"
queries_path="/NS/llm-pretraining/work/afkhan/vLLM-Serving-POC/Data/FAQ_en.csv"

# Run Inference -

echo "Running Inference"
echo "Model: $model"
echo "TP Size: $tp_size"
echo "Save Path: $save_path"
echo "Queries Path: $queries_path"

python sglang_offline_runner.py --model $model --tp_size $tp_size --save_path $save_path --queries_path $queries_path --prompt_path $prompt_path

Just for reference this is the current script which works well on 1 node -
Python file -

import sglang as sgl

import argparse

import pandas as pd

import json

def main(args):

    print("Arguments: ", args)

    print("Loading model...")

    runtime = sgl.Runtime(

        model_path=args.model,

        tokenizer_path=args.model,

        tp_size=args.tp_size,  # t_ensor p_arallel size, number of GPUs to split the model over

        log_level="error" ,

        random_seed = 20242

    )

    sgl.set_default_backend(runtime)

    print("Model loaded.")

    temperature = 1

    top_p = 1

    top_k = -1

    min_p = 0

    max_tokens = 20000

    message_role = 'assistant'

    @sgl.function

    def conv_generate(s, system_prompt, old_messages, new_message, max_tokens, message_role):

        s += sgl.system(system_prompt)

        for om in old_messages:

            content = om['content']

            role = om['role']

            if role == 'user':

                s += sgl.user(content)

            elif role == 'assistant':

                s += sgl.assistant(content)

        if message_role == 'user':

            s += sgl.user(new_message + sgl.gen("response", max_tokens=max_tokens))

        elif message_role == 'assistant':

            s += sgl.assistant(new_message + sgl.gen("response", max_tokens=max_tokens))

    # Read Queries

    df = pd.read_csv(args.queries_path)

    queries = df["Question"].tolist()

    # Read Prompt

    with open(args.prompt_path, "r") as f:

        system_prompt = f.read()

    completion_texts = []

    for query in queries:

        state = conv_generate.run(

            system_prompt = system_prompt,

            old_messages = [

                {'content': query, 'role': 'user'}

            ],

            new_message = '', # Start with empty message

            max_tokens=max_tokens,

            message_role=message_role,

            temperature=temperature,

            top_p=top_p,

            top_k=top_k,

            min_p=min_p,

        )

        model_generation  = state['response']

        completion_texts.append(model_generation)

        print("Ran query: ", query)

        print("Model generation: ", model_generation)

    # Save completions

    with open(args.save_path, "w") as f:

        json.dump(completion_texts, f)

if __name__ == "__main__":

    argparser = argparse.ArgumentParser()

    argparser.add_argument("--model", type=str)

    argparser.add_argument("--tp_size", type=int)

    argparser.add_argument("--save_path", type=str)

    argparser.add_argument("--queries_path", type=str)

    argparser.add_argument("--prompt_path", type=str)

    args = argparser.parse_args()

    main(args)

SLURM bash file -

#!/bin/bash -l

#SBATCH -o SLURM_Logs/%x_%j_%A-%T.out

#SBATCH -e SLURM_Logs/%x_%j_%A-%T.err

#SBATCH -D ./

#SBATCH -J 405B-FP8-Off-Sglang

#SBATCH --nodes=1

#SBATCH --tasks-per-node=1

#SBATCH --cpus-per-task=18

#SBATCH --mem=224GB

#SBATCH --partition="h100"

#SBATCH --gres=gpu:h100:8

# Wall clock limit (max. is 24 hours):

#SBATCH --time=12:00:00

# Load required modules or set environment variables if needed

source /NS/venvs/work/afkhan/slurm_sglang_env/bin/activate

model="/scratch/sws0/user/afkhan/Models/Llama-3.1-405B-Instruct-FP8"

tp_size=8

prompt_path="/NS/llm-pretraining/work/afkhan/vLLM-Serving-POC/Data/Prompt_OHB_Chat_Alpha.txt"

save_path="/NS/llm-pretraining/work/afkhan/vLLM-Serving-POC/Data/Outputs_Llama-3.1-405B-Instruct-FP8_OHB_Chat_Alpha_tp_8_sgl_fixed.json"

queries_path="/NS/llm-pretraining/work/afkhan/vLLM-Serving-POC/Data/FAQ_en.csv"

# Run Inference -

echo "Running Inference"

echo "Model: $model"

echo "TP Size: $tp_size"

echo "Save Path: $save_path"

echo "Queries Path: $queries_path"

python sglang_offline_runner.py --model $model --tp_size $tp_size --save_path $save_path --queries_path $queries_path --prompt_path $prompt_path

Thanks

Some more updates. I tried to run the openai compatible version on SLURM on 2 nodes. For the 8B version it works across 2 nodes (tp=16) -
#!/bin/bash -l

#SBATCH -o SLURM_Logs/%x_%j_master.out
#SBATCH -e SLURM_Logs/%x_%j_master.err
#SBATCH -D ./
#SBATCH -J 8B-Online-TP16-Sglang

#SBATCH --nodes=2
#SBATCH --ntasks=2  # Total tasks across all nodes
#SBATCH --cpus-per-task=18
#SBATCH --mem=224GB

#SBATCH --partition="a100"
#SBATCH --gres=gpu:a100:8

#SBATCH --time=12:00:00

# Load required modules or set environment variables if needed
echo "[INFO] Activating environment on node $SLURM_PROCID"
if ! source /NS/venvs/work/afkhan/slurm_sglang_env/bin/activate; then
    echo "[ERROR] Failed to activate environment" >&2
    exit 1
fi

# Define parameters
model="/scratch/sws0/user/afkhan/Models/Llama-3.1-8B-Instruct"
tp_size=16

echo "[INFO] Running inference"
echo "[INFO] Model: $model"
echo "[INFO] TP Size: $tp_size"

# Define the NCCL init address using the hostname of the head node
HEAD_NODE=$(scontrol show hostname "$SLURM_NODELIST" | head -n 1)
NCCL_INIT_ADDR="${HEAD_NODE}:8000"
echo "[INFO] NCCL_INIT_ADDR: $NCCL_INIT_ADDR"

# Set OUTLINES_CACHE_DIR to /tmp/node_0_cache

export OUTLINES_CACHE_DIR="/tmp/node_0_cache"

# Launch processes with srun
srun --ntasks=1 --nodes=1 --exclusive --output="SLURM_Logs/8b_%x_%j_node0.out" \
    --error="SLURM_Logs/%x_%j_node0.err" \
    python3 -m sglang.launch_server \
    --model-path "$model" \
    --tp "$tp_size" \
    --nccl-init-addr "$NCCL_INIT_ADDR" \
    --nnodes 2 \
    --node-rank 0 &

# Set OUTLINES_CACHE_DIR to /tmp/node_1_cache

export OUTLINES_CACHE_DIR="/tmp/node_1_cache"

srun --ntasks=1 --nodes=1 --exclusive --output="SLURM_Logs/8b_%x_%j_node1.out" \
    --error="SLURM_Logs/%x_%j_node1.err" \
    python3 -m sglang.launch_server \
    --model-path "$model" \
    --tp "$tp_size" \
    --nccl-init-addr "$NCCL_INIT_ADDR" \
    --nnodes 2 \
    --node-rank 1 &

# Wait for localhost:30000 to accept connections

while ! nc -z localhost 30000; do
    sleep 1
    echo "[INFO] Waiting for localhost:30000 to accept connections"
done

echo "[INFO] localhost:30000 is ready to accept connections"

# curl -X POST http://<IP_ADDRESS>:30000/v1/chat/completions \
# -H "Authorization: Bearer None" \
# -H "Content-Type: application/json" \
# -d '{
#   "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
#   "messages": [
#     {
#       "role": "user",
#       "content": "List 3 countries and their capitals."
#     }
#   ],
#   "temperature": 0,
#   "max_tokens": 64
# }'

# Run the client and echo the output
response=$(curl -s -X POST http://<IP_ADDRESS>:30000/v1/chat/completions \
-H "Authorization: Bearer None" \
-H "Content-Type: application/json" \
-d '{
  "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
  "messages": [
    {
      "role": "user",
      "content": "List 3 countries and their capitals."
    }
  ],
  "temperature": 0,
  "max_tokens": 64
}')

echo "[INFO] Response from server:"
echo "$response"

However for 405B-FP8 I get a timeout (I use the same script with model path changed to 405B-FP8)
Logs for timeout -
[2024-12-26 15:33:50] server_args=ServerArgs(model_path='/scratch/sws0/user/afkhan/Models/Llama-3.1-405B-Instruct-FP8', tokenizer_path='/scratch/sws0/user/afkhan/Models/Llama-3.1-405B-Instruct-FP8', tokenizer_mode='auto', skip_tokenizer_init=False, load_format='auto', trust_remote_code=False, dtype='auto', kv_cache_dtype='auto', quantization=None, context_length=None, device='cuda', served_model_name='/scratch/sws0/user/afkhan/Models/Llama-3.1-405B-Instruct-FP8', chat_template=None, is_embedding=False, revision=None, host='<IP_ADDRESS>', port=30000, mem_fraction_static=0.79, max_running_requests=None, max_total_tokens=None, chunked_prefill_size=8192, max_prefill_tokens=16384, schedule_policy='lpm', schedule_conservativeness=1.0, cpu_offload_gb=0, tp_size=16, stream_interval=1, random_seed=341483684, constrained_json_whitespace_pattern=None, watchdog_timeout=300, download_dir=None, base_gpu_id=0, log_level='info', log_level_http=None, log_requests=False, show_time_cost=False, enable_metrics=False, decode_log_interval=40, api_key=None, file_storage_pth='SGLang_storage', enable_cache_report=False, dp_size=1, load_balance_method='round_robin', ep_size=1, dist_init_addr='sws-8h100grid-05:8000', nnodes=2, node_rank=1, json_model_override_args='{}', enable_double_sparsity=False, ds_channel_config_path=None, ds_heavy_channel_num=32, ds_heavy_token_num=256, ds_heavy_channel_type='qk', ds_sparse_decode_threshold=4096, lora_paths=None, max_loras_per_batch=8, attention_backend='flashinfer', sampling_backend='flashinfer', grammar_backend='outlines', disable_radix_cache=False, disable_jump_forward=False, disable_cuda_graph=False, disable_cuda_graph_padding=False, disable_outlines_disk_cache=False, disable_custom_all_reduce=False, disable_mla=False, disable_overlap_schedule=False, enable_mixed_chunk=False, enable_dp_attention=False, enable_ep_moe=False, enable_torch_compile=False, torch_compile_max_bs=32, cuda_graph_max_bs=160, torchao_config='', enable_nan_detection=False, enable_p2p_check=False, triton_attention_reduce_in_fp32=False, triton_attention_num_kv_splits=8, num_continuous_decode_steps=1, delete_ckpt_after_loading=False)
[2024-12-26 15:35:03 TP9] Init torch distributed begin.
[2024-12-26 15:35:03 TP15] Init torch distributed begin.
[2024-12-26 15:35:03 TP13] Init torch distributed begin.
[2024-12-26 15:35:04 TP14] Init torch distributed begin.
[2024-12-26 15:35:04 TP10] Init torch distributed begin.
[2024-12-26 15:35:04 TP12] Init torch distributed begin.
[2024-12-26 15:35:04 TP11] Init torch distributed begin.
[2024-12-26 15:35:04 TP8] Init torch distributed begin.
[W1226 15:35:11.626067741 socket.cpp:752] [c10d] The client socket cannot be initialized to connect to [sws-8h100grid-05.mpi-sws.org]:8000 (errno: 97 - Address family not supported by protocol).
[W1226 15:35:12.800892117 socket.cpp:752] [c10d] The client socket cannot be initialized to connect to [sws-8h100grid-05.mpi-sws.org]:8000 (errno: 97 - Address family not supported by protocol).
[W1226 15:35:12.813586747 socket.cpp:752] [c10d] The client socket cannot be initialized to connect to [sws-8h100grid-05.mpi-sws.org]:8000 (errno: 97 - Address family not supported by protocol).
[W1226 15:35:12.857323180 socket.cpp:752] [c10d] The client socket cannot be initialized to connect to [sws-8h100grid-05.mpi-sws.org]:8000 (errno: 97 - Address family not supported by protocol).
[W1226 15:35:12.876457914 socket.cpp:752] [c10d] The client socket cannot be initialized to connect to [sws-8h100grid-05.mpi-sws.org]:8000 (errno: 97 - Address family not supported by protocol).
[W1226 15:35:12.883440682 socket.cpp:752] [c10d] The client socket cannot be initialized to connect to [sws-8h100grid-05.mpi-sws.org]:8000 (errno: 97 - Address family not supported by protocol).
[W1226 15:35:12.902173353 socket.cpp:752] [c10d] The client socket cannot be initialized to connect to [sws-8h100grid-05.mpi-sws.org]:8000 (errno: 97 - Address family not supported by protocol).
[W1226 15:35:12.017366693 socket.cpp:752] [c10d] The client socket cannot be initialized to connect to [sws-8h100grid-05.mpi-sws.org]:8000 (errno: 97 - Address family not supported by protocol).
[E1226 15:44:07.920344489 socket.cpp:1011] [c10d] The client socket has timed out after 600000ms while trying to connect to (sws-8h100grid-05, 8000).
[W1226 15:44:07.032046284 TCPStore.cpp:358] [c10d] TCP client failed to connect/validate to host sws-8h100grid-05:8000 - retrying (try=0, timeout=600000ms, delay=71366ms): The client socket has timed out after 600000ms while trying to connect to (sws-8h100grid-05, 8000).
Exception raised from throwTimeoutError at ../torch/csrc/distributed/c10d/socket.cpp:1013 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f5b72d6c446 in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x15e04c6 (0x7f5b5df3c4c6 in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
frame #2: <unknown function> + 0x6029d95 (0x7f5b62985d95 in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
frame #3: <unknown function> + 0x602a22b (0x7f5b6298622b in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
frame #4: <unknown function> + 0x602a3a4 (0x7f5b629863a4 in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
frame #5: <unknown function> + 0x5fe8016 (0x7f5b62944016 in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
frame #6: c10d::TCPStore::TCPStore(std::string, c10d::TCPStoreOptions const&) + 0x20c (0x7f5b62946f7c in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
frame #7: <unknown function> + 0xda5bed (0x7f5b72334bed in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
frame #8: <unknown function> + 0x4ccad4 (0x7f5b71a5bad4 in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
frame #9: sglang::scheduler() [0x548aa6]
frame #10: _PyObject_MakeTpCall + 0x223 (0x5188b3 in sglang::scheduler)
frame #11: sglang::scheduler() [0x584ec4]
frame #12: _PyObject_Call + 0xa7 (0x56b2a7 in sglang::scheduler)
frame #13: sglang::scheduler() [0x5669b8]
frame #14: sglang::scheduler() [0x518c50]
frame #15: <unknown function> + 0x4cb31b (0x7f5b71a5a31b in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
frame #16: _PyObject_MakeTpCall + 0x223 (0x5188b3 in sglang::scheduler)
frame #17: _PyEval_EvalFrameDefault + 0x8f0 (0x52c2f0 in sglang::scheduler)
frame #18: sglang::scheduler() [0x5ee497]
frame #19: sglang::scheduler() [0x54edd8]
frame #20: PyObject_Vectorcall + 0x2c (0x53b6ac in sglang::scheduler)
frame #21: _PyEval_EvalFrameDefault + 0x8f0 (0x52c2f0 in sglang::scheduler)
frame #22: _PyFunction_Vectorcall + 0x191 (0x55d961 in sglang::scheduler)
frame #23: PyObject_Call + 0xb1 (0x56b081 in sglang::scheduler)
frame #24: _PyEval_EvalFrameDefault + 0x494e (0x53034e in sglang::scheduler)
frame #25: _PyFunction_Vectorcall + 0x191 (0x55d961 in sglang::scheduler)
frame #26: PyObject_Call + 0xb1 (0x56b081 in sglang::scheduler)
frame #27: _PyEval_EvalFrameDefault + 0x494e (0x53034e in sglang::scheduler)
frame #28: _PyFunction_Vectorcall + 0x191 (0x55d961 in sglang::scheduler)
frame #29: _PyObject_FastCallDictTstate + 0x2e6 (0x51fb46 in sglang::scheduler)
frame #30: sglang::scheduler() [0x5666d3]
frame #31: _PyObject_MakeTpCall + 0x1c5 (0x518855 in sglang::scheduler)
frame #32: _PyEval_EvalFrameDefault + 0x8f0 (0x52c2f0 in sglang::scheduler)
frame #33: _PyObject_FastCallDictTstate + 0x1c7 (0x51fa27 in sglang::scheduler)
frame #34: sglang::scheduler() [0x566772]
frame #35: _PyObject_MakeTpCall + 0x1c5 (0x518855 in sglang::scheduler)
frame #36: _PyEval_EvalFrameDefault + 0x8f0 (0x52c2f0 in sglang::scheduler)
frame #37: _PyFunction_Vectorcall + 0x191 (0x55d961 in sglang::scheduler)
frame #38: _PyObject_FastCallDictTstate + 0x2e6 (0x51fb46 in sglang::scheduler)
frame #39: sglang::scheduler() [0x5666d3]
frame #40: _PyObject_MakeTpCall + 0x1c5 (0x518855 in sglang::scheduler)
frame #41: _PyEval_EvalFrameDefault + 0x8f0 (0x52c2f0 in sglang::scheduler)
frame #42: _PyObject_FastCallDictTstate + 0x1c7 (0x51fa27 in sglang::scheduler)
frame #43: sglang::scheduler() [0x566772]
frame #44: _PyObject_MakeTpCall + 0x1c5 (0x518855 in sglang::scheduler)
frame #45: _PyEval_EvalFrameDefault + 0x8f0 (0x52c2f0 in sglang::scheduler)
frame #46: _PyFunction_Vectorcall + 0x191 (0x55d961 in sglang::scheduler)
frame #47: _PyEval_EvalFrameDefault + 0x494e (0x53034e in sglang::scheduler)
frame #48: PyEval_EvalCode + 0xbb (0x523fab in sglang::scheduler)
frame #49: sglang::scheduler() [0x649bd7]
frame #50: sglang::scheduler() [0x64752f]
frame #51: PyRun_StringFlags + 0x5d (0x56fd4d in sglang::scheduler)
frame #52: PyRun_SimpleStringFlags + 0x36 (0x640ae6 in sglang::scheduler)
frame #53: Py_RunMain + 0x454 (0x652224 in sglang::scheduler)
frame #54: Py_BytesMain + 0x27 (0x629917 in sglang::scheduler)
frame #55: <unknown function> + 0x2724a (0x7f5b7dc4124a in /lib/x86_64-linux-gnu/libc.so.6)
frame #56: __libc_start_main + 0x85 (0x7f5b7dc41305 in /lib/x86_64-linux-gnu/libc.so.6)
frame #57: _start + 0x21 (0x6297b1 in sglang::scheduler)

[E1226 15:44:09.437224181 socket.cpp:1011] [c10d] The client socket has timed out after 600000ms while trying to connect to (sws-8h100grid-05, 8000).
[W1226 15:44:09.437733885 TCPStore.cpp:358] [c10d] TCP client failed to connect/validate to host sws-8h100grid-05:8000 - retrying (try=0, timeout=600000ms, delay=44468ms): The client socket has timed out after 600000ms while trying to connect to (sws-8h100grid-05, 8000).
Exception raised from throwTimeoutError at ../torch/csrc/distributed/c10d/socket.cpp:1013 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f8ca7b6c446 in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x15e04c6 (0x7f8c92d3c4c6 in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
frame #2: <unknown function> + 0x6029d95 (0x7f8c97785d95 in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
frame #3: <unknown function> + 0x602a22b (0x7f8c9778622b in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
frame #4: <unknown function> + 0x602a3a4 (0x7f8c977863a4 in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
frame #5: <unknown function> + 0x5fe8016 (0x7f8c97744016 in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
frame #6: c10d::TCPStore::TCPStore(std::string, c10d::TCPStoreOptions const&) + 0x20c (0x7f8c97746f7c in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
frame #7: <unknown function> + 0xda5bed (0x7f8ca7134bed in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
frame #8: <unknown function> + 0x4ccad4 (0x7f8ca685bad4 in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
frame #9: sglang::scheduler() [0x548aa6]
frame #10: _PyObject_MakeTpCall + 0x223 (0x5188b3 in sglang::scheduler)
frame #11: sglang::scheduler() [0x584ec4]
frame #12: _PyObject_Call + 0xa7 (0x56b2a7 in sglang::scheduler)
frame #13: sglang::scheduler() [0x5669b8]
frame #14: sglang::scheduler() [0x518c50]
frame #15: <unknown function> + 0x4cb31b (0x7f8ca685a31b in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
frame #16: _PyObject_MakeTpCall + 0x223 (0x5188b3 in sglang::scheduler)
frame #17: _PyEval_EvalFrameDefault + 0x8f0 (0x52c2f0 in sglang::scheduler)
frame #18: sglang::scheduler() [0x5ee497]
frame #19: sglang::scheduler() [0x54edd8]
frame #20: PyObject_Vectorcall + 0x2c (0x53b6ac in sglang::scheduler)
frame #21: _PyEval_EvalFrameDefault + 0x8f0 (0x52c2f0 in sglang::scheduler)
frame #22: _PyFunction_Vectorcall + 0x191 (0x55d961 in sglang::scheduler)
frame #23: PyObject_Call + 0xb1 (0x56b081 in sglang::scheduler)
frame #24: _PyEval_EvalFrameDefault + 0x494e (0x53034e in sglang::scheduler)
frame #25: _PyFunction_Vectorcall + 0x191 (0x55d961 in sglang::scheduler)
frame #26: PyObject_Call + 0xb1 (0x56b081 in sglang::scheduler)
frame #27: _PyEval_EvalFrameDefault + 0x494e (0x53034e in sglang::scheduler)
frame #28: _PyFunction_Vectorcall + 0x191 (0x55d961 in sglang::scheduler)
frame #29: _PyObject_FastCallDictTstate + 0x2e6 (0x51fb46 in sglang::scheduler)
frame #30: sglang::scheduler() [0x5666d3]
frame #31: _PyObject_MakeTpCall + 0x1c5 (0x518855 in sglang::scheduler)
frame #32: _PyEval_EvalFrameDefault + 0x8f0 (0x52c2f0 in sglang::scheduler)
frame #33: _PyObject_FastCallDictTstate + 0x1c7 (0x51fa27 in sglang::scheduler)
frame #34: sglang::scheduler() [0x566772]
frame #35: _PyObject_MakeTpCall + 0x1c5 (0x518855 in sglang::scheduler)
frame #36: _PyEval_EvalFrameDefault + 0x8f0 (0x52c2f0 in sglang::scheduler)
frame #37: _PyFunction_Vectorcall + 0x191 (0x55d961 in sglang::scheduler)
frame #38: _PyObject_FastCallDictTstate + 0x2e6 (0x51fb46 in sglang::scheduler)
frame #39: sglang::scheduler() [0x5666d3]
frame #40: _PyObject_MakeTpCall + 0x1c5 (0x518855 in sglang::scheduler)
frame #41: _PyEval_EvalFrameDefault + 0x8f0 (0x52c2f0 in sglang::scheduler)
frame #42: _PyObject_FastCallDictTstate + 0x1c7 (0x51fa27 in sglang::scheduler)
frame #43: sglang::scheduler() [0x566772]
frame #44: _PyObject_MakeTpCall + 0x1c5 (0x518855 in sglang::scheduler)
frame #45: _PyEval_EvalFrameDefault + 0x8f0 (0x52c2f0 in sglang::scheduler)
frame #46: _PyFunction_Vectorcall + 0x191 (0x55d961 in sglang::scheduler)
frame #47: _PyEval_EvalFrameDefault + 0x494e (0x53034e in sglang::scheduler)
frame #48: PyEval_EvalCode + 0xbb (0x523fab in sglang::scheduler)
frame #49: sglang::scheduler() [0x649bd7]
frame #50: sglang::scheduler() [0x64752f]
frame #51: PyRun_StringFlags + 0x5d (0x56fd4d in sglang::scheduler)
frame #52: PyRun_SimpleStringFlags + 0x36 (0x640ae6 in sglang::scheduler)
frame #53: Py_RunMain + 0x454 (0x652224 in sglang::scheduler)
frame #54: Py_BytesMain + 0x27 (0x629917 in sglang::scheduler)
frame #55: <unknown function> + 0x2724a (0x7f8cb296a24a in /lib/x86_64-linux-gnu/libc.so.6)
frame #56: __libc_start_main + 0x85 (0x7f8cb296a305 in /lib/x86_64-linux-gnu/libc.so.6)
frame #57: _start + 0x21 (0x6297b1 in sglang::scheduler)

[E1226 15:44:24.795949033 socket.cpp:1011] [c10d] The client socket has timed out after 600000ms while trying to connect to (sws-8h100grid-05, 8000).
[W1226 15:44:24.796554711 TCPStore.cpp:358] [c10d] TCP client failed to connect/validate to host sws-8h100grid-05:8000 - retrying (try=0, timeout=600000ms, delay=58972ms): The client socket has timed out after 600000ms while trying to connect to (sws-8h100grid-05, 8000).
Exception raised from throwTimeoutError at ../torch/csrc/distributed/c10d/socket.cpp:1013 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f6cd876c446 in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x15e04c6 (0x7f6cc353c4c6 in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
frame #2: <unknown function> + 0x6029d95 (0x7f6cc7f85d95 in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
frame #3: <unknown function> + 0x602a22b (0x7f6cc7f8622b in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
frame #4: <unknown function> + 0x602a3a4 (0x7f6cc7f863a4 in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
frame #5: <unknown function> + 0x5fe8016 (0x7f6cc7f44016 in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
frame #6: c10d::TCPStore::TCPStore(std::string, c10d::TCPStoreOptions const&) + 0x20c (0x7f6cc7f46f7c in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
frame #7: <unknown function> + 0xda5bed (0x7f6cd7934bed in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
frame #8: <unknown function> + 0x4ccad4 (0x7f6cd705bad4 in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
frame #9: sglang::scheduler() [0x548aa6]
frame #10: _PyObject_MakeTpCall + 0x223 (0x5188b3 in sglang::scheduler)
frame #11: sglang::scheduler() [0x584ec4]
frame #12: _PyObject_Call + 0xa7 (0x56b2a7 in sglang::scheduler)
frame #13: sglang::scheduler() [0x5669b8]
frame #14: sglang::scheduler() [0x518c50]
frame #15: <unknown function> + 0x4cb31b (0x7f6cd705a31b in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
frame #16: _PyObject_MakeTpCall + 0x223 (0x5188b3 in sglang::scheduler)
frame #17: _PyEval_EvalFrameDefault + 0x8f0 (0x52c2f0 in sglang::scheduler)
frame #18: sglang::scheduler() [0x5ee497]
frame #19: sglang::scheduler() [0x54edd8]
frame #20: PyObject_Vectorcall + 0x2c (0x53b6ac in sglang::scheduler)
frame #21: _PyEval_EvalFrameDefault + 0x8f0 (0x52c2f0 in sglang::scheduler)
frame #22: _PyFunction_Vectorcall + 0x191 (0x55d961 in sglang::scheduler)
frame #23: PyObject_Call + 0xb1 (0x56b081 in sglang::scheduler)
frame #24: _PyEval_EvalFrameDefault + 0x494e (0x53034e in sglang::scheduler)
frame #25: _PyFunction_Vectorcall + 0x191 (0x55d961 in sglang::scheduler)
frame #26: PyObject_Call + 0xb1 (0x56b081 in sglang::scheduler)
frame #27: _PyEval_EvalFrameDefault + 0x494e (0x53034e in sglang::scheduler)
frame #28: _PyFunction_Vectorcall + 0x191 (0x55d961 in sglang::scheduler)
frame #29: _PyObject_FastCallDictTstate + 0x2e6 (0x51fb46 in sglang::scheduler)
frame #30: sglang::scheduler() [0x5666d3]
frame #31: _PyObject_MakeTpCall + 0x1c5 (0x518855 in sglang::scheduler)
frame #32: _PyEval_EvalFrameDefault + 0x8f0 (0x52c2f0 in sglang::scheduler)
frame #33: _PyObject_FastCallDictTstate + 0x1c7 (0x51fa27 in sglang::scheduler)
frame #34: sglang::scheduler() [0x566772]
frame #35: _PyObject_MakeTpCall + 0x1c5 (0x518855 in sglang::scheduler)
frame #36: _PyEval_EvalFrameDefault + 0x8f0 (0x52c2f0 in sglang::scheduler)
frame #37: _PyFunction_Vectorcall + 0x191 (0x55d961 in sglang::scheduler)
frame #38: _PyObject_FastCallDictTstate + 0x2e6 (0x51fb46 in sglang::scheduler)
frame #39: sglang::scheduler() [0x5666d3]
frame #40: _PyObject_MakeTpCall + 0x1c5 (0x518855 in sglang::scheduler)
frame #41: _PyEval_EvalFrameDefault + 0x8f0 (0x52c2f0 in sglang::scheduler)
frame #42: _PyObject_FastCallDictTstate + 0x1c7 (0x51fa27 in sglang::scheduler)
frame #43: sglang::scheduler() [0x566772]
frame #44: _PyObject_MakeTpCall + 0x1c5 (0x518855 in sglang::scheduler)
frame #45: _PyEval_EvalFrameDefault + 0x8f0 (0x52c2f0 in sglang::scheduler)
frame #46: _PyFunction_Vectorcall + 0x191 (0x55d961 in sglang::scheduler)
frame #47: _PyEval_EvalFrameDefault + 0x494e (0x53034e in sglang::scheduler)
frame #48: PyEval_EvalCode + 0xbb (0x523fab in sglang::scheduler)
frame #49: sglang::scheduler() [0x649bd7]
frame #50: sglang::scheduler() [0x64752f]
frame #51: PyRun_StringFlags + 0x5d (0x56fd4d in sglang::scheduler)
frame #52: PyRun_SimpleStringFlags + 0x36 (0x640ae6 in sglang::scheduler)
frame #53: Py_RunMain + 0x454 (0x652224 in sglang::scheduler)
frame #54: Py_BytesMain + 0x27 (0x629917 in sglang::scheduler)
frame #55: <unknown function> + 0x2724a (0x7f6ce331c24a in /lib/x86_64-linux-gnu/libc.so.6)
frame #56: __libc_start_main + 0x85 (0x7f6ce331c305 in /lib/x86_64-linux-gnu/libc.so.6)
frame #57: _start + 0x21 (0x6297b1 in sglang::scheduler)

[E1226 15:44:25.497183572 socket.cpp:1011] [c10d] The client socket has timed out after 600000ms while trying to connect to (sws-8h100grid-05, 8000).
[W1226 15:44:25.497497753 TCPStore.cpp:358] [c10d] TCP client failed to connect/validate to host sws-8h100grid-05:8000 - retrying (try=0, timeout=600000ms, delay=82879ms): The client socket has timed out after 600000ms while trying to connect to (sws-8h100grid-05, 8000).
Exception raised from throwTimeoutError at ../torch/csrc/distributed/c10d/socket.cpp:1013 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f68c796c446 in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x15e04c6 (0x7f68b273c4c6 in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
frame #2: <unknown function> + 0x6029d95 (0x7f68b7185d95 in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
frame #3: <unknown function> + 0x602a22b (0x7f68b718622b in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
frame #4: <unknown function> + 0x602a3a4 (0x7f68b71863a4 in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
frame #5: <unknown function> + 0x5fe8016 (0x7f68b7144016 in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
frame #6: c10d::TCPStore::TCPStore(std::string, c10d::TCPStoreOptions const&) + 0x20c (0x7f68b7146f7c in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
frame #7: <unknown function> + 0xda5bed (0x7f68c6b34bed in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
frame #8: <unknown function> + 0x4ccad4 (0x7f68c625bad4 in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
frame #9: sglang::scheduler() [0x548aa6]
frame #10: _PyObject_MakeTpCall + 0x223 (0x5188b3 in sglang::scheduler)
frame #11: sglang::scheduler() [0x584ec4]
frame #12: _PyObject_Call + 0xa7 (0x56b2a7 in sglang::scheduler)
frame #13: sglang::scheduler() [0x5669b8]
frame #14: sglang::scheduler() [0x518c50]
frame #15: <unknown function> + 0x4cb31b (0x7f68c625a31b in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
frame #16: _PyObject_MakeTpCall + 0x223 (0x5188b3 in sglang::scheduler)
frame #17: _PyEval_EvalFrameDefault + 0x8f0 (0x52c2f0 in sglang::scheduler)
frame #18: sglang::scheduler() [0x5ee497]
frame #19: sglang::scheduler() [0x54edd8]
frame #20: PyObject_Vectorcall + 0x2c (0x53b6ac in sglang::scheduler)
frame #21: _PyEval_EvalFrameDefault + 0x8f0 (0x52c2f0 in sglang::scheduler)
frame #22: _PyFunction_Vectorcall + 0x191 (0x55d961 in sglang::scheduler)
frame #23: PyObject_Call + 0xb1 (0x56b081 in sglang::scheduler)
frame #24: _PyEval_EvalFrameDefault + 0x494e (0x53034e in sglang::scheduler)
frame #25: _PyFunction_Vectorcall + 0x191 (0x55d961 in sglang::scheduler)
frame #26: PyObject_Call + 0xb1 (0x56b081 in sglang::scheduler)
frame #27: _PyEval_EvalFrameDefault + 0x494e (0x53034e in sglang::scheduler)
frame #28: _PyFunction_Vectorcall + 0x191 (0x55d961 in sglang::scheduler)
frame #29: _PyObject_FastCallDictTstate + 0x2e6 (0x51fb46 in sglang::scheduler)
frame #30: sglang::scheduler() [0x5666d3]
frame #31: _PyObject_MakeTpCall + 0x1c5 (0x518855 in sglang::scheduler)
frame #32: _PyEval_EvalFrameDefault + 0x8f0 (0x52c2f0 in sglang::scheduler)
frame #33: _PyObject_FastCallDictTstate + 0x1c7 (0x51fa27 in sglang::scheduler)
frame #34: sglang::scheduler() [0x566772]
frame #35: _PyObject_MakeTpCall + 0x1c5 (0x518855 in sglang::scheduler)
frame #36: _PyEval_EvalFrameDefault + 0x8f0 (0x52c2f0 in sglang::scheduler)
frame #37: _PyFunction_Vectorcall + 0x191 (0x55d961 in sglang::scheduler)
frame #38: _PyObject_FastCallDictTstate + 0x2e6 (0x51fb46 in sglang::scheduler)
frame #39: sglang::scheduler() [0x5666d3]
frame #40: _PyObject_MakeTpCall + 0x1c5 (0x518855 in sglang::scheduler)
frame #41: _PyEval_EvalFrameDefault + 0x8f0 (0x52c2f0 in sglang::scheduler)
frame #42: _PyObject_FastCallDictTstate + 0x1c7 (0x51fa27 in sglang::scheduler)
frame #43: sglang::scheduler() [0x566772]
frame #44: _PyObject_MakeTpCall + 0x1c5 (0x518855 in sglang::scheduler)
frame #45: _PyEval_EvalFrameDefault + 0x8f0 (0x52c2f0 in sglang::scheduler)
frame #46: _PyFunction_Vectorcall + 0x191 (0x55d961 in sglang::scheduler)
frame #47: _PyEval_EvalFrameDefault + 0x494e (0x53034e in sglang::scheduler)
frame #48: PyEval_EvalCode + 0xbb (0x523fab in sglang::scheduler)
frame #49: sglang::scheduler() [0x649bd7]
frame #50: sglang::scheduler() [0x64752f]
frame #51: PyRun_StringFlags + 0x5d (0x56fd4d in sglang::scheduler)
frame #52: PyRun_SimpleStringFlags + 0x36 (0x640ae6 in sglang::scheduler)
frame #53: Py_RunMain + 0x454 (0x652224 in sglang::scheduler)
frame #54: Py_BytesMain + 0x27 (0x629917 in sglang::scheduler)
frame #55: <unknown function> + 0x2724a (0x7f68d24d424a in /lib/x86_64-linux-gnu/libc.so.6)
frame #56: __libc_start_main + 0x85 (0x7f68d24d4305 in /lib/x86_64-linux-gnu/libc.so.6)
frame #57: _start + 0x21 (0x6297b1 in sglang::scheduler)

[E1226 15:44:31.548021308 socket.cpp:1011] [c10d] The client socket has timed out after 600000ms while trying to connect to (sws-8h100grid-05, 8000).
[W1226 15:44:31.548330712 TCPStore.cpp:358] [c10d] TCP client failed to connect/validate to host sws-8h100grid-05:8000 - retrying (try=0, timeout=600000ms, delay=77116ms): The client socket has timed out after 600000ms while trying to connect to (sws-8h100grid-05, 8000).
Exception raised from throwTimeoutError at ../torch/csrc/distributed/c10d/socket.cpp:1013 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fd08876c446 in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x15e04c6 (0x7fd07393c4c6 in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
frame #2: <unknown function> + 0x6029d95 (0x7fd078385d95 in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
frame #3: <unknown function> + 0x602a22b (0x7fd07838622b in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
frame #4: <unknown function> + 0x602a3a4 (0x7fd0783863a4 in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
frame #5: <unknown function> + 0x5fe8016 (0x7fd078344016 in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
frame #6: c10d::TCPStore::TCPStore(std::string, c10d::TCPStoreOptions const&) + 0x20c (0x7fd078346f7c in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
frame #7: <unknown function> + 0xda5bed (0x7fd087d34bed in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
frame #8: <unknown function> + 0x4ccad4 (0x7fd08745bad4 in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
frame #9: sglang::scheduler() [0x548aa6]
frame #10: _PyObject_MakeTpCall + 0x223 (0x5188b3 in sglang::scheduler)
frame #11: sglang::scheduler() [0x584ec4]
frame #12: _PyObject_Call + 0xa7 (0x56b2a7 in sglang::scheduler)
frame #13: sglang::scheduler() [0x5669b8]
frame #14: sglang::scheduler() [0x518c50]
frame #15: <unknown function> + 0x4cb31b (0x7fd08745a31b in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
frame #16: _PyObject_MakeTpCall + 0x223 (0x5188b3 in sglang::scheduler)
frame #17: _PyEval_EvalFrameDefault + 0x8f0 (0x52c2f0 in sglang::scheduler)
frame #18: sglang::scheduler() [0x5ee497]
frame #19: sglang::scheduler() [0x54edd8]
frame #20: PyObject_Vectorcall + 0x2c (0x53b6ac in sglang::scheduler)
frame #21: _PyEval_EvalFrameDefault + 0x8f0 (0x52c2f0 in sglang::scheduler)
frame #22: _PyFunction_Vectorcall + 0x191 (0x55d961 in sglang::scheduler)
frame #23: PyObject_Call + 0xb1 (0x56b081 in sglang::scheduler)
frame #24: _PyEval_EvalFrameDefault + 0x494e (0x53034e in sglang::scheduler)
frame #25: _PyFunction_Vectorcall + 0x191 (0x55d961 in sglang::scheduler)
frame #26: PyObject_Call + 0xb1 (0x56b081 in sglang::scheduler)
frame #27: _PyEval_EvalFrameDefault + 0x494e (0x53034e in sglang::scheduler)
frame #28: _PyFunction_Vectorcall + 0x191 (0x55d961 in sglang::scheduler)
frame #29: _PyObject_FastCallDictTstate + 0x2e6 (0x51fb46 in sglang::scheduler)
frame #30: sglang::scheduler() [0x5666d3]
frame #31: _PyObject_MakeTpCall + 0x1c5 (0x518855 in sglang::scheduler)
frame #32: _PyEval_EvalFrameDefault + 0x8f0 (0x52c2f0 in sglang::scheduler)
frame #33: _PyObject_FastCallDictTstate + 0x1c7 (0x51fa27 in sglang::scheduler)
frame #34: sglang::scheduler() [0x566772]
frame #35: _PyObject_MakeTpCall + 0x1c5 (0x518855 in sglang::scheduler)
frame #36: _PyEval_EvalFrameDefault + 0x8f0 (0x52c2f0 in sglang::scheduler)
frame #37: _PyFunction_Vectorcall + 0x191 (0x55d961 in sglang::scheduler)
frame #38: _PyObject_FastCallDictTstate + 0x2e6 (0x51fb46 in sglang::scheduler)
frame #39: sglang::scheduler() [0x5666d3]
frame #40: _PyObject_MakeTpCall + 0x1c5 (0x518855 in sglang::scheduler)
frame #41: _PyEval_EvalFrameDefault + 0x8f0 (0x52c2f0 in sglang::scheduler)
frame #42: _PyObject_FastCallDictTstate + 0x1c7 (0x51fa27 in sglang::scheduler)
frame #43: sglang::scheduler() [0x566772]
frame #44: _PyObject_MakeTpCall + 0x1c5 (0x518855 in sglang::scheduler)
frame #45: _PyEval_EvalFrameDefault + 0x8f0 (0x52c2f0 in sglang::scheduler)
frame #46: _PyFunction_Vectorcall + 0x191 (0x55d961 in sglang::scheduler)
frame #47: _PyEval_EvalFrameDefault + 0x494e (0x53034e in sglang::scheduler)
frame #48: PyEval_EvalCode + 0xbb (0x523fab in sglang::scheduler)
frame #49: sglang::scheduler() [0x649bd7]
frame #50: sglang::scheduler() [0x64752f]
frame #51: PyRun_StringFlags + 0x5d (0x56fd4d in sglang::scheduler)
frame #52: PyRun_SimpleStringFlags + 0x36 (0x640ae6 in sglang::scheduler)
frame #53: Py_RunMain + 0x454 (0x652224 in sglang::scheduler)
frame #54: Py_BytesMain + 0x27 (0x629917 in sglang::scheduler)
frame #55: <unknown function> + 0x2724a (0x7fd0935e624a in /lib/x86_64-linux-gnu/libc.so.6)
frame #56: __libc_start_main + 0x85 (0x7fd0935e6305 in /lib/x86_64-linux-gnu/libc.so.6)
frame #57: _start + 0x21 (0x6297b1 in sglang::scheduler)

[E1226 15:44:32.176606388 socket.cpp:1011] [c10d] The client socket has timed out after 600000ms while trying to connect to (sws-8h100grid-05, 8000).
[W1226 15:44:32.176918136 TCPStore.cpp:358] [c10d] TCP client failed to connect/validate to host sws-8h100grid-05:8000 - retrying (try=0, timeout=600000ms, delay=48822ms): The client socket has timed out after 600000ms while trying to connect to (sws-8h100grid-05, 8000).
Exception raised from throwTimeoutError at ../torch/csrc/distributed/c10d/socket.cpp:1013 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fb1c2d6c446 in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x15e04c6 (0x7fb1adb3c4c6 in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
frame #2: <unknown function> + 0x6029d95 (0x7fb1b2585d95 in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
frame #3: <unknown function> + 0x602a22b (0x7fb1b258622b in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
frame #4: <unknown function> + 0x602a3a4 (0x7fb1b25863a4 in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
frame #5: <unknown function> + 0x5fe8016 (0x7fb1b2544016 in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
frame #6: c10d::TCPStore::TCPStore(std::string, c10d::TCPStoreOptions const&) + 0x20c (0x7fb1b2546f7c in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
frame #7: <unknown function> + 0xda5bed (0x7fb1c1f34bed in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
frame #8: <unknown function> + 0x4ccad4 (0x7fb1c165bad4 in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
frame #9: sglang::scheduler() [0x548aa6]
frame #10: _PyObject_MakeTpCall + 0x223 (0x5188b3 in sglang::scheduler)
frame #11: sglang::scheduler() [0x584ec4]
frame #12: _PyObject_Call + 0xa7 (0x56b2a7 in sglang::scheduler)
frame #13: sglang::scheduler() [0x5669b8]
frame #14: sglang::scheduler() [0x518c50]
frame #15: <unknown function> + 0x4cb31b (0x7fb1c165a31b in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
frame #16: _PyObject_MakeTpCall + 0x223 (0x5188b3 in sglang::scheduler)
frame #17: _PyEval_EvalFrameDefault + 0x8f0 (0x52c2f0 in sglang::scheduler)
frame #18: sglang::scheduler() [0x5ee497]
frame #19: sglang::scheduler() [0x54edd8]
frame #20: PyObject_Vectorcall + 0x2c (0x53b6ac in sglang::scheduler)
frame #21: _PyEval_EvalFrameDefault + 0x8f0 (0x52c2f0 in sglang::scheduler)
frame #22: _PyFunction_Vectorcall + 0x191 (0x55d961 in sglang::scheduler)
frame #23: PyObject_Call + 0xb1 (0x56b081 in sglang::scheduler)
frame #24: _PyEval_EvalFrameDefault + 0x494e (0x53034e in sglang::scheduler)
frame #25: _PyFunction_Vectorcall + 0x191 (0x55d961 in sglang::scheduler)
frame #26: PyObject_Call + 0xb1 (0x56b081 in sglang::scheduler)
frame #27: _PyEval_EvalFrameDefault + 0x494e (0x53034e in sglang::scheduler)
frame #28: _PyFunction_Vectorcall + 0x191 (0x55d961 in sglang::scheduler)
frame #29: _PyObject_FastCallDictTstate + 0x2e6 (0x51fb46 in sglang::scheduler)
frame #30: sglang::scheduler() [0x5666d3]
frame #31: _PyObject_MakeTpCall + 0x1c5 (0x518855 in sglang::scheduler)
frame #32: _PyEval_EvalFrameDefault + 0x8f0 (0x52c2f0 in sglang::scheduler)
frame #33: _PyObject_FastCallDictTstate + 0x1c7 (0x51fa27 in sglang::scheduler)
frame #34: sglang::scheduler() [0x566772]
frame #35: _PyObject_MakeTpCall + 0x1c5 (0x518855 in sglang::scheduler)
frame #36: _PyEval_EvalFrameDefault + 0x8f0 (0x52c2f0 in sglang::scheduler)
frame #37: _PyFunction_Vectorcall + 0x191 (0x55d961 in sglang::scheduler)
frame #38: _PyObject_FastCallDictTstate + 0x2e6 (0x51fb46 in sglang::scheduler)
frame #39: sglang::scheduler() [0x5666d3]
frame #40: _PyObject_MakeTpCall + 0x1c5 (0x518855 in sglang::scheduler)
frame #41: _PyEval_EvalFrameDefault + 0x8f0 (0x52c2f0 in sglang::scheduler)
frame #42: _PyObject_FastCallDictTstate + 0x1c7 (0x51fa27 in sglang::scheduler)
frame #43: sglang::scheduler() [0x566772]
frame #44: _PyObject_MakeTpCall + 0x1c5 (0x518855 in sglang::scheduler)
frame #45: _PyEval_EvalFrameDefault + 0x8f0 (0x52c2f0 in sglang::scheduler)
frame #46: _PyFunction_Vectorcall + 0x191 (0x55d961 in sglang::scheduler)
frame #47: _PyEval_EvalFrameDefault + 0x494e (0x53034e in sglang::scheduler)
frame #48: PyEval_EvalCode + 0xbb (0x523fab in sglang::scheduler)
frame #49: sglang::scheduler() [0x649bd7]
frame #50: sglang::scheduler() [0x64752f]
frame #51: PyRun_StringFlags + 0x5d (0x56fd4d in sglang::scheduler)
frame #52: PyRun_SimpleStringFlags + 0x36 (0x640ae6 in sglang::scheduler)
frame #53: Py_RunMain + 0x454 (0x652224 in sglang::scheduler)
frame #54: Py_BytesMain + 0x27 (0x629917 in sglang::scheduler)
frame #55: <unknown function> + 0x2724a (0x7fb1cd92e24a in /lib/x86_64-linux-gnu/libc.so.6)
frame #56: __libc_start_main + 0x85 (0x7fb1cd92e305 in /lib/x86_64-linux-gnu/libc.so.6)
frame #57: _start + 0x21 (0x6297b1 in sglang::scheduler)

[E1226 15:44:45.235264822 socket.cpp:1011] [c10d] The client socket has timed out after 600000ms while trying to connect to (sws-8h100grid-05, 8000).
[W1226 15:44:45.235803349 TCPStore.cpp:358] [c10d] TCP client failed to connect/validate to host sws-8h100grid-05:8000 - retrying (try=0, timeout=600000ms, delay=38694ms): The client socket has timed out after 600000ms while trying to connect to (sws-8h100grid-05, 8000).
Exception raised from throwTimeoutError at ../torch/csrc/distributed/c10d/socket.cpp:1013 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f377856c446 in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x15e04c6 (0x7f376373c4c6 in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
frame #2: <unknown function> + 0x6029d95 (0x7f3768185d95 in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
frame #3: <unknown function> + 0x602a22b (0x7f376818622b in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
frame #4: <unknown function> + 0x602a3a4 (0x7f37681863a4 in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
frame #5: <unknown function> + 0x5fe8016 (0x7f3768144016 in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
frame #6: c10d::TCPStore::TCPStore(std::string, c10d::TCPStoreOptions const&) + 0x20c (0x7f3768146f7c in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
frame #7: <unknown function> + 0xda5bed (0x7f3777b34bed in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
frame #8: <unknown function> + 0x4ccad4 (0x7f377725bad4 in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
frame #9: sglang::scheduler() [0x548aa6]
frame #10: _PyObject_MakeTpCall + 0x223 (0x5188b3 in sglang::scheduler)
frame #11: sglang::scheduler() [0x584ec4]
frame #12: _PyObject_Call + 0xa7 (0x56b2a7 in sglang::scheduler)
frame #13: sglang::scheduler() [0x5669b8]
frame #14: sglang::scheduler() [0x518c50]
frame #15: <unknown function> + 0x4cb31b (0x7f377725a31b in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
frame #16: _PyObject_MakeTpCall + 0x223 (0x5188b3 in sglang::scheduler)
frame #17: _PyEval_EvalFrameDefault + 0x8f0 (0x52c2f0 in sglang::scheduler)
frame #18: sglang::scheduler() [0x5ee497]
frame #19: sglang::scheduler() [0x54edd8]
frame #20: PyObject_Vectorcall + 0x2c (0x53b6ac in sglang::scheduler)
frame #21: _PyEval_EvalFrameDefault + 0x8f0 (0x52c2f0 in sglang::scheduler)
frame #22: _PyFunction_Vectorcall + 0x191 (0x55d961 in sglang::scheduler)
frame #23: PyObject_Call + 0xb1 (0x56b081 in sglang::scheduler)
frame #24: _PyEval_EvalFrameDefault + 0x494e (0x53034e in sglang::scheduler)
frame #25: _PyFunction_Vectorcall + 0x191 (0x55d961 in sglang::scheduler)
frame #26: PyObject_Call + 0xb1 (0x56b081 in sglang::scheduler)
frame #27: _PyEval_EvalFrameDefault + 0x494e (0x53034e in sglang::scheduler)
frame #28: _PyFunction_Vectorcall + 0x191 (0x55d961 in sglang::scheduler)
frame #29: _PyObject_FastCallDictTstate + 0x2e6 (0x51fb46 in sglang::scheduler)
frame #30: sglang::scheduler() [0x5666d3]
frame #31: _PyObject_MakeTpCall + 0x1c5 (0x518855 in sglang::scheduler)
frame #32: _PyEval_EvalFrameDefault + 0x8f0 (0x52c2f0 in sglang::scheduler)
frame #33: _PyObject_FastCallDictTstate + 0x1c7 (0x51fa27 in sglang::scheduler)
frame #34: sglang::scheduler() [0x566772]
frame #35: _PyObject_MakeTpCall + 0x1c5 (0x518855 in sglang::scheduler)
frame #36: _PyEval_EvalFrameDefault + 0x8f0 (0x52c2f0 in sglang::scheduler)
frame #37: _PyFunction_Vectorcall + 0x191 (0x55d961 in sglang::scheduler)
frame #38: _PyObject_FastCallDictTstate + 0x2e6 (0x51fb46 in sglang::scheduler)
frame #39: sglang::scheduler() [0x5666d3]
frame #40: _PyObject_MakeTpCall + 0x1c5 (0x518855 in sglang::scheduler)
frame #41: _PyEval_EvalFrameDefault + 0x8f0 (0x52c2f0 in sglang::scheduler)
frame #42: _PyObject_FastCallDictTstate + 0x1c7 (0x51fa27 in sglang::scheduler)
frame #43: sglang::scheduler() [0x566772]
frame #44: _PyObject_MakeTpCall + 0x1c5 (0x518855 in sglang::scheduler)
frame #45: _PyEval_EvalFrameDefault + 0x8f0 (0x52c2f0 in sglang::scheduler)
frame #46: _PyFunction_Vectorcall + 0x191 (0x55d961 in sglang::scheduler)
frame #47: _PyEval_EvalFrameDefault + 0x494e (0x53034e in sglang::scheduler)
frame #48: PyEval_EvalCode + 0xbb (0x523fab in sglang::scheduler)
frame #49: sglang::scheduler() [0x649bd7]
frame #50: sglang::scheduler() [0x64752f]
frame #51: PyRun_StringFlags + 0x5d (0x56fd4d in sglang::scheduler)
frame #52: PyRun_SimpleStringFlags + 0x36 (0x640ae6 in sglang::scheduler)
frame #53: Py_RunMain + 0x454 (0x652224 in sglang::scheduler)
frame #54: Py_BytesMain + 0x27 (0x629917 in sglang::scheduler)
frame #55: <unknown function> + 0x2724a (0x7f37833d624a in /lib/x86_64-linux-gnu/libc.so.6)
frame #56: __libc_start_main + 0x85 (0x7f37833d6305 in /lib/x86_64-linux-gnu/libc.so.6)
frame #57: _start + 0x21 (0x6297b1 in sglang::scheduler)

[W1226 15:44:53.922117502 socket.cpp:752] [c10d] The client socket cannot be initialized to connect to [srv-24-0112.mpi-sws.org]:8000 (errno: 97 - Address family not supported by protocol).
[E1226 15:45:07.399109121 socket.cpp:1011] [c10d] The client socket has timed out after 600000ms while trying to connect to (sws-8h100grid-05, 8000).
[W1226 15:45:07.399623392 TCPStore.cpp:358] [c10d] TCP client failed to connect/validate to host sws-8h100grid-05:8000 - retrying (try=0, timeout=600000ms, delay=52324ms): The client socket has timed out after 600000ms while trying to connect to (sws-8h100grid-05, 8000).
Exception raised from throwTimeoutError at ../torch/csrc/distributed/c10d/socket.cpp:1013 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7ff8d7b6c446 in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x15e04c6 (0x7ff8c293c4c6 in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
frame #2: <unknown function> + 0x6029d95 (0x7ff8c7385d95 in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
frame #3: <unknown function> + 0x602a22b (0x7ff8c738622b in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
frame #4: <unknown function> + 0x602a3a4 (0x7ff8c73863a4 in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
frame #5: <unknown function> + 0x5fe8016 (0x7ff8c7344016 in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
frame #6: c10d::TCPStore::TCPStore(std::string, c10d::TCPStoreOptions const&) + 0x20c (0x7ff8c7346f7c in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
frame #7: <unknown function> + 0xda5bed (0x7ff8d6d34bed in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
frame #8: <unknown function> + 0x4ccad4 (0x7ff8d645bad4 in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
frame #9: sglang::scheduler() [0x548aa6]
frame #10: _PyObject_MakeTpCall + 0x223 (0x5188b3 in sglang::scheduler)
frame #11: sglang::scheduler() [0x584ec4]
frame #12: _PyObject_Call + 0xa7 (0x56b2a7 in sglang::scheduler)
frame #13: sglang::scheduler() [0x5669b8]
frame #14: sglang::scheduler() [0x518c50]
frame #15: <unknown function> + 0x4cb31b (0x7ff8d645a31b in /NS/venvs/work/afkhan/slurm_sglang_env/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
frame #16: _PyObject_MakeTpCall + 0x223 (0x5188b3 in sglang::scheduler)
frame #17: _PyEval_EvalFrameDefault + 0x8f0 (0x52c2f0 in sglang::scheduler)
frame #18: sglang::scheduler() [0x5ee497]
frame #19: sglang::scheduler() [0x54edd8]
frame #20: PyObject_Vectorcall + 0x2c (0x53b6ac in sglang::scheduler)
frame #21: _PyEval_EvalFrameDefault + 0x8f0 (0x52c2f0 in sglang::scheduler)
frame #22: _PyFunction_Vectorcall + 0x191 (0x55d961 in sglang::scheduler)
frame #23: PyObject_Call + 0xb1 (0x56b081 in sglang::scheduler)
frame #24: _PyEval_EvalFrameDefault + 0x494e (0x53034e in sglang::scheduler)
frame #25: _PyFunction_Vectorcall + 0x191 (0x55d961 in sglang::scheduler)
frame #26: PyObject_Call + 0xb1 (0x56b081 in sglang::scheduler)
frame #27: _PyEval_EvalFrameDefault + 0x494e (0x53034e in sglang::scheduler)
frame #28: _PyFunction_Vectorcall + 0x191 (0x55d961 in sglang::scheduler)
frame #29: _PyObject_FastCallDictTstate + 0x2e6 (0x51fb46 in sglang::scheduler)
frame #30: sglang::scheduler() [0x5666d3]
frame #31: _PyObject_MakeTpCall + 0x1c5 (0x518855 in sglang::scheduler)
frame #32: _PyEval_EvalFrameDefault + 0x8f0 (0x52c2f0 in sglang::scheduler)
frame #33: _PyObject_FastCallDictTstate + 0x1c7 (0x51fa27 in sglang::scheduler)
frame #34: sglang::scheduler() [0x566772]
frame #35: _PyObject_MakeTpCall + 0x1c5 (0x518855 in sglang::scheduler)
frame #36: _PyEval_EvalFrameDefault + 0x8f0 (0x52c2f0 in sglang::scheduler)
frame #37: _PyFunction_Vectorcall + 0x191 (0x55d961 in sglang::scheduler)
frame #38: _PyObject_FastCallDictTstate + 0x2e6 (0x51fb46 in sglang::scheduler)
frame #39: sglang::scheduler() [0x5666d3]
frame #40: _PyObject_MakeTpCall + 0x1c5 (0x518855 in sglang::scheduler)
frame #41: _PyEval_EvalFrameDefault + 0x8f0 (0x52c2f0 in sglang::scheduler)
frame #42: _PyObject_FastCallDictTstate + 0x1c7 (0x51fa27 in sglang::scheduler)
frame #43: sglang::scheduler() [0x566772]
frame #44: _PyObject_MakeTpCall + 0x1c5 (0x518855 in sglang::scheduler)
frame #45: _PyEval_EvalFrameDefault + 0x8f0 (0x52c2f0 in sglang::scheduler)
frame #46: _PyFunction_Vectorcall + 0x191 (0x55d961 in sglang::scheduler)
frame #47: _PyEval_EvalFrameDefault + 0x494e (0x53034e in sglang::scheduler)
frame #48: PyEval_EvalCode + 0xbb (0x523fab in sglang::scheduler)
frame #49: sglang::scheduler() [0x649bd7]
frame #50: sglang::scheduler() [0x64752f]
frame #51: PyRun_StringFlags + 0x5d (0x56fd4d in sglang::scheduler)
frame #52: PyRun_SimpleStringFlags + 0x36 (0x640ae6 in sglang::scheduler)
frame #53: Py_RunMain + 0x454 (0x652224 in sglang::scheduler)
frame #54: Py_BytesMain + 0x27 (0x629917 in sglang::scheduler)
frame #55: <unknown function> + 0x2724a (0x7ff8e26ae24a in /lib/x86_64-linux-gnu/libc.so.6)
frame #56: __libc_start_main + 0x85 (0x7ff8e26ae305 in /lib/x86_64-linux-gnu/libc.so.6)
frame #57: _start + 0x21 (0x6297b1 in sglang::scheduler)

[W1226 15:45:19.409442336 socket.cpp:752] [c10d] The client socket cannot be initialized to connect to [sws-8h100grid-05.mpi-sws.org]:8000 (errno: 97 - Address family not supported by protocol).
[W1226 15:45:20.008074382 socket.cpp:752] [c10d] The client socket cannot be initialized to connect to [sws-8h100grid-05.mpi-sws.org]:8000 (errno: 97 - Address family not supported by protocol).
[W1226 15:45:23.781919807 socket.cpp:752] [c10d] The client socket cannot be initialized to connect to [sws-8h100grid-05.mpi-sws.org]:8000 (errno: 97 - Address family not supported by protocol).
[W1226 15:45:23.941001448 socket.cpp:752] [c10d] The client socket cannot be initialized to connect to [sws-8h100grid-05.mpi-sws.org]:8000 (errno: 97 - Address family not supported by protocol).
[W1226 15:45:48.390112397 socket.cpp:752] [c10d] The client socket cannot be initialized to connect to [sws-8h100grid-05.mpi-sws.org]:8000 (errno: 97 - Address family not supported by protocol).
[W1226 15:45:48.674499580 socket.cpp:752] [c10d] The client socket cannot be initialized to connect to [sws-8h100grid-05.mpi-sws.org]:8000 (errno: 97 - Address family not supported by protocol).
[W1226 15:45:59.737848425 socket.cpp:752] [c10d] The client socket cannot be initialized to connect to [sws-8h100grid-05.mpi-sws.org]:8000 (errno: 97 - Address family not supported by protocol).

Update again: The same code worked on 2 A100 nodes with 8 GPUs each. I am now trying the BF16 version on 2 nodes (both H100 and A100)

Thanks! @aflah02. I don't know if things have worked out. If not, could you come to our meeting on this?
https://x.com/lmsysorg/status/1872797103107522932

Thanks! @aflah02. I don't know if things have worked out. If not, could you come to our meeting on this?
https://x.com/lmsysorg/status/1872797103107522932

Thanks @zhaochenyang20 however I'm travelling later today and unfortunately will not be able to make it to the meeting.
Just to update I was able to run the online inference (starting a server) across 2 nodes in slurm however I haven't been able to figure out how to run multinode via the SGLang engine or runtime for offline inference

Yeah. I remember even without slurm, we can't serve llama 405B across multiple nodes with offline engine. @aflah02
Also, would you like to provide a docs markdown or .py to us, demonstrating how to use SRT on slurm? This is a great contribution and we sincerely appreciate this. We will move on for the engine running llama 405B (perhaps also deepseek V3)

Yeah. I remember even without slurm, we can't serve llama 405B across multiple nodes with offline engine. @aflah02
Also, would you like to provide a docs markdown or .py to us, demonstrating how to use SRT on slurm? This is a great contribution and we sincerely appreciate this. We will move on for the engine running llama 405B (perhaps also deepseek V3)

I can share one on running the server via SLURM. Is that what you call SRT? I guess you can technically connect to the endpoint via setting the runtime backend so it makes sense. It's not running via python though and just carefully recreates how you would do it if you had complete access to both nodes but via slurm

SRT is the HTTP server @aflah02

SRT is the HTTP server @aflah02

Ah okay nice
Yeah I'll do that sometimes next week once I'm back from travels
It would also be great to have a way to do this via the python api though instead of via running commands on the terminal. Is it currently possible to do that? Running multinode by just setting backend model path and tp-size in the python api?

Sorry. I don't think that's feasible right now. We support running 405B llama in this way:
https://sgl-project.github.io/backend/backend.html#example-run-llama-3-1-405b
Also, my advisor told me that Llama 405B is rarely used since it's performance is worse than Qwen 2.5 72B instruct and Llama 3.3 70B Instruct. Maybe you can by pass this 😂
@aflah02

Sorry. I don't think that's feasible right now. We support running 405B llama in this way:
https://sgl-project.github.io/backend/backend.html#example-run-llama-3-1-405b
Also, my advisor told me that Llama 405B is rarely used since it's performance is worse than Qwen 2.5 72B instruct and Llama 3.3 70B Instruct. Maybe you can by pass this 😂
@aflah02

Yeah but it's not just about the performance. If I want to say benchmark the model I still need to run it and running it via the CLI is much less convenient as compared to having one python script. It would be really useful if this could be added in future releases.

Also sorry I couldn't join the meeting yesterday. Any updates on if this is in on the roadmap?

Hi @zhaochenyang20
I wrote a blog post on running LLMs across multiple SLURM nodes via SGLang - https://aflah02.substack.com/p/multi-node-llm-inference-with-sglang

@aflah02 Nice blog!

Thanks :)

Hi @zhaochenyang20 I wrote a blog post on running LLMs across multiple SLURM nodes via SGLang - https://aflah02.substack.com/p/multi-node-llm-inference-with-sglang

@aflah02 This blog looks good. Can you submit a PR to write some of the SLRUM commands into a script?

Hi @zhaochenyang20 I wrote a blog post on running LLMs across multiple SLURM nodes via SGLang - https://aflah02.substack.com/p/multi-node-llm-inference-with-sglang

@aflah02 This blog looks good. Can you submit a PR to write some of the SLRUM commands into a script?

Sure
Is there any reference which I should follow on where do I place this and what kind of file should this be (bash v/s markdown) etc?

https://github.com/sgl-project/sglang/tree/main/examples/runtime
I think here is the right place. Markdown is pretty okay, and you can refer to your blog at:
https://aflah02.substack.com/p/multi-node-llm-inference-with-sglang
Great job, thanks! @aflah02


Intune User interface config is not present

Reading the updated docs page I can see that you can allow/deny hardware classes however i cannot see an option to define the level of control that can be made by the OMA-URI settings. is this a limitation of Intune or the feature?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 7b7a5c17-1358-990b-968d-432a3aab2e17
Version Independent ID: 1b4d4eff-3e22-47d7-4054-6cc9ed0217d0
Content: Microsoft Defender for Endpoint Device Control Removable Storage Access Control, removable storage media
Content Source: microsoft-365/security/defender-endpoint/device-control-removable-storage-access-control.md
Product: m365-security
Technology: mde
GitHub Login: @Dansimp
Microsoft Alias: dansimp

@denisebmsft Please help us in resolving this issue. Thanks

I have opened work item 5731624 to be assigned to a writer.

Hello @yogkumgit! My work item has been assigned and is in the queue for April.

@MatthewWhiteMoJ  the PM would like to known if you are asking how to “Block specific removable storage class but allow specific media”? if that is the case, he suggests "you can use ‘IncludedIdList a group through PrimaryId and ExcludedIDList a group through DeviceId/HardwareId/etc.’ "
If this is not what you are inquiring about, please provide additional details, perhaps with a basic example.

@yogkumgit Is it possible to contact the customer for additional information (posted above)? The PM needs some clarification in order to proceed with a response or revisions to the documentation.

@MatthewWhiteMoJ Please respond to our query. Thanks

Fixed in pr 16338 Adds clarification in two topics, with cross-linking references, per PM.
To block a specific removable storage class but allow specific media, you can use ‘IncludedIdList a group through PrimaryId and ExcludedIDList a group through DeviceId/HardwareId/etc.` For additional guidance, see Deploy Removable Storage Access Control by using Intune OMA-URI. Will publish this morning 09/16.

Update to article went live about 5 minutes ago.

@MatthewWhiteMoJ Hope this update is helpful for you. Thanks for taking out some time to open the issue. Appreciate and encourage you to do the same in future also.


The following resource(s) failed to   create: [RestAPI].

Describe the bug
I'm trying to launch a Simple File Manager for Amazon EFS but it fails with
EFSFileManagerAPI
Embedded stack arn:aws:cloudformation:ca-central-1:IDIDIDIDIDID:stack/EFSAccessor-EFSFileManagerAPI-
  1W4QUA1DG6MRC/XXXXXXXX-6c41-11ed-9c09-0e965e613246 was not successfully created: The following resource(s) failed to
  create: [RestAPI].

To Reproduce

Navigate to Simple File Manager for Amazon EFS
Click Launch in the AWS Console
Enter a Stack Name
Enter an AdminEmail
Then click Next
Accept:

I acknowledge that AWS CloudFormation might create IAM resources with custom names.
I acknowledge that AWS CloudFormation might require the following capability: CAPABILITY_AUTO_EXPAND

Click Submit
Look at your CloudFormation / Stack Detail / Events for the error message under EFSFileManagerAPI

Expected behavior
I expect the stack to deploy completely without error.
Please complete the following information about the solution:

v1.5.0

[ca-central-1 ]

[No] Was the solution modified from the version published on this repository?

[N/A] If the answer to the previous question was yes, are the changes available on GitHub?

[No] Have you checked your service quotas for the sevices this solution uses?

[ ] Were there any errors in the CloudWatch Logs?

I accidentally managed to briefly catch a glance of an intermediate substack that had the following error.
RestAPI
User: arn:aws:iam::IDIDIDIDIDID:user/ME is not authorized to perform: apigateway:POST on resource: arn:aws:apigateway:ca-central-1::/restapis because no iden    tity-based policy allows the apigateway:POST action (Service: AmazonApiGateway; Status Code: 403; Error Code: AccessDeniedException; Request ID: ab2a8d19-8b5f-433    4-a291-b4802a75f519; Proxy: null)

Hi @SamuelLarkin, thanks for your patience here.
In order to deploy the RestAPI included as part of the Simple File Manager CloudFormation template, you are required to have the apigateway:POST permission, please check to ensure that your user has that permission.
Please either open a new issue or respond here if you continue to run into problems.
Thanks,
Simon


fix: reduce placeholder specificity #8366

Close https://github.com/zurb/foundation-sites/issues/8366

👍


Migrate ClientJavaE2E from Grakn Core

What is the goal of this PR?
https://github.com/graknlabs/grakn/issues/5043 lists that we want to remove client-java-e2e from grakn core and put it in an assembly test in this repository. This PR migrates the test and adapts it for the repository.
What are the changes implemented in this PR?

Migrate test into new package test/assembly
Adapt assetions and server bootup to match client-java style
Add to CI

@lolski you think this is a large integration test? The original issue indicates to place it into a assembly package but not sure that is the best place, it's really a huge integration test or sorta e2e. Assumiing assembly is the name of our new e2e in general!

good point!

Ok - the test executes a long series of Queries so I've renamed it to QueriesTest

Ok - the test executes a long series of Queries so I've renamed it to QueriesTest

Possible. Let's keep it singular. "QueryTest".


feat: Extend felt packing (128bits) to other operations (store_n, load_n, bytecode execution, ...)

Feature Request
Describe the Feature Request
In many parts of the app, we translate bytes array to felt packed bytes (16 bytes). We'd like to drop altogether translation between the two format and simply handle felt packed 16 bytes.

Right now we are translating between:
bytes array <-> 16bytes array
Which could be removed if the whole Evm works as 16bytes felts

Describe Preferred Solution

Describe Alternatives

Related Code

Additional Context

If the feature request is approved, would you be willing to submit a PR?
Yes / No (Help can be provided if you need assistance submitting a PR)

Please @danilowhk, describe here your implementation choices and later on propose a draft PR early so we can discuss together with @spapinistarkware / @thomas-quadratic and @ClementWalter about tech choices:)!

I think we can close this until Cairo 1.0 comes out


Do not click the logo in the top left!

Just don't; it's not worth it. 😔

Wait now it actually doesn't do anything anymore.
what

Ok found the code. sneaky

Until I have proper "lore"™ going on, it's basically a placeholder :P
Also, thanks for checking out my website :D

Ok found the code. sneaky

uh oh you can read my code
o_o


version of alias that checks that missing_blocks is empty

It should be pretty easy to add a version of alias that only succeeds in the case that the dag is complete.

why do you need it? Can you not just check if there are any missing links and then abort if that is not the case?

I guess you'd have to alias it first and then check, but sounds reasonable


Component model support

Context
https://github.com/bytecodealliance/wasmtime/tree/main/examples/component
Add support for making the same example from C#.
https://github.com/WebAssembly/component-model/tree/main
https://github.com/bytecodealliance/componentize-dotnet
Current state:
Only Wasm modules can be instantiated and functions ran from them. Loading .wasm file which is compiled as a component into a Module fails.
Steps
[ ] running components
[ ] imports and exports (with WIT data types)
[ ] converting modules to components
[ ] multiple components interacting
Remarks
This is blocked by https://github.com/bytecodealliance/wasmtime/issues/8036#issuecomment-2180272305

Now that we released wasmtime v22 support, we should have the right version but may need to expose some new apis

Since we use the c-api we need https://github.com/bytecodealliance/wasmtime/issues/8036.


Incorrect error messages when using or()

I'm working on a parser for a typescript-like language. I initially used nom, but I switched to chumsky because I really like the error generating and the synergy with ariadne. But I noticed just today that the parser gives incorrect error messages when using the or combinator. I should mention I'm using logos for the lexer and I'm using the latest commit of the chumsky crate.
The error is mainly here and here.
To give a quick explanation, this parser is trying to parse something like let x = 10;. The issue arises when the statement end token is missing (which is ;). When using the first option in the or combinator (which tries to parse an int), the error is correct, as it displays:

But when the value is something other than the first option (be it a bool or a float) it displays an incorrect message:

The message is incorrect because false is a valid value and it should not be replaced by ;, but instead be followed by it.
The parser should "know" that it correctly parsed a primitive_val (which is the bool or float val) and display the error message regarding the missing ;. I'd expect a message such as:

You can try it yourself by cloning the repo and running cargo run, only changing the input from z = 10.0 to z = 10 to see the incorrectly and correctly displayed message respectively.
Am I doing something wrong or is this intended behavior?

This is interesting. Do you get this with other error types such as Simple?

Do you think you could try to reduce the example parser / input to a minimum failing case?

Yes, I think this is the example with the least code that still produces the bug. It seems related to Streams, since I couldn't reproduce it with regular string parsers.
To give you a quick rundown, this parser just tries to parse either "f;" or "t;". We force the error by intentionally omitting the ';', I included both the correct and incorrect errors. I should note that if you change the order of the or combinator, only the one that's no the first variant gets the incorrect error.

After analyzing this for some time, or should I say a LOT (this library is extremely complicated, props to you for maintaining it). I realized that not only does this work with non-streaming parsers, but it's not related to the or() combinator at all.
I noticed the choice combinator (which is used by or under the hood) saves the errors on each alternative (which is totally intended), and replaces them or merges them depending on the position of the next found error.
This issue occurs because None (which is the end of input) shares the same priority as the last character of the input. Thus, when comparing the error, the parser decides that, since they share the same priority, they should be merged. This all happens here.
The solution I thought about is to add a conditional to the Ordering:Equal pattern to give priority to the incoming error if the found Token is None, which would look something like this.
This solution passes all tests, except for one on the pratt parsing module. This can be fixed by changing the 1..2 range into 2..2 like such.
Since I don't know if this is the best solution, I didn't open a PR, but please let me know if you would want me to. I hope this info is useful to you, and many thanks for creating and maintaining this library.

This seems like a reasonable approach! Error prioritisation is one of those things I've put on the back foot for 1.0 in favour of sorting out the API (exact semantics of errors can be changed later without breaking semver), so I'm sure your improvement does the job. I'd be happy to review a PR for this :)


GitHub Copilot fails to start

Does this issue occur when all extensions are disabled?: Yes/No

VS Code Version: 1.78.2
OS Version: Windows_NT x64 10.0.22621 (latest Windows 11 basically)

I have a problem with the GitHub Copilot extension not starting in our repository (private). It's stuck in the "Activating" phase and drives the usage of the CPU pretty high, particularly the rg.exe process. Possibly related, I have found this in the dev tools console:

I have tried manually editing product.json and adding terminalDataWriteEvent which helped get the error to no longer appear, but it unfortunately didn't help with the extension not running. Any other logs specific to the extension that I could look at? Extensions host output is not particularly useful in this case, just shows this:

This problem started happening about 2 weeks ago, before then everything worked as expected. Note that installing older versions of the extension (or the Nightly version) does not help. Copilot works as expected in all other repositories. Unfortunately the repository is private so I can't share it for reproduction.
Thanks in advance!

/extCopilot

It looks like this is caused by the Copilot extension. Please file the issue in the Copilot Discussion Forum. Make sure to check their issue reporting template and provide them relevant information such as the extension version you're using. See also our issue reporting guidelines for more information.
Happy Coding!


Add environment variable for --runtime-interface-emulator-address

At the moment it's not possible to configure the runtime interface emulator address without patching the docker image.
The default is <IP_ADDRESS>:8080 , and I would like to restrict it to the local network for security reasons.

Hi. There is already a parameter that you can pass called --runtime-interface-emulator-address
(Definition and Example usage in the tests)
Is this what you're looking for?

No, the lambda-entrypoint.sh looks like this (e.g. in public.ecr.aws/lambda/ruby:3.3-x86_64 ):
#!/bin/sh
# Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.

if [ $# -ne 1 ]; then
  echo "entrypoint requires the handler name to be the first argument" 1>&2
  exit 142
fi
export _HANDLER="$1"

RUNTIME_ENTRYPOINT=/var/runtime/bootstrap
if [ -z "${AWS_LAMBDA_RUNTIME_API}" ]; then
  exec /usr/local/bin/aws-lambda-rie $RUNTIME_ENTRYPOINT
else
  exec $RUNTIME_ENTRYPOINT
fi

The command line parameters are hard coded, so there's no easy way to pass those parameters. At the moment I need to create a docker image with a patched lambda-entrypoint.sh which is a lot more work than setting an environment variable.

Ok. I understand what's the use case now.
If you want to make a contribution, we would be happy to review and merge. It should be something similar to how the logLevel is handled that can be passed as a parameter or as an env var: https://github.com/aws/aws-lambda-runtime-interface-emulator/blob/develop/cmd/aws-lambda-rie/main.go#L44-L48 (plus tests)
In any case, we'll take this into consideration and see when we can take this.

@valerena , could you look into adding it?

Hi @shaicoleman. I don't have enough bandwidth to submit the changes myself at the moment, but I could help reviewing code if you want to send a PR.


Error with STM32F723IEKx

Here's what i get when i try using the provided "app-template" with the stm32f723 disco board via the onboard ST-LINK/V2-1
probe-run --chip STM32F723IEKx --defmt target/thumbv7em-none-eabihf/debug/hello`
flashing program ..
DONE
resetting device
Error: Error communicating with probe: A core architecture specific error occured

Caused by:
    0: A core architecture specific error occured
    1: Failed to read register DRW at address 0x0000000c because: An error specific to a probe type occured
    2: An error specific to a probe type occured
    3: Command failed with status SwdDpFault

I think this is really interesting to know! Because we see this error within cargo-embed/flash on some chips too. But atm I have no clue what causes it. So it seems like it's even different for two probe-rs users :/

See some progress after updating to latest:
probe-run --chip STM32F723IEKx target/thumbv7em-none-eabihf/debug/hello`
flashing program ..
DONE
resetting device
stack backtrace:
   0: 0x08000894 - __bkpt
   1: 0x08000864 - defmt_test::exit
   2: 0x0800045e - hello::__cortex_m_rt_main
   3: 0x08000408 - main
   4: 0x08000528 - ResetTrampoline
   5: 0x08000504 - Reset
Error: Error communicating with probe: A core architecture specific error occured

Caused by:
    0: A core architecture specific error occured
    1: Failed to read register DRW at address 0x0000000c because: An error specific to a probe type occured
    2: An error specific to a probe type occured
    3: Command failed with status SwdDpFault

It works with the Segger firmware on the ST-link for some reason...
https://www.segger.com/products/debug-probes/j-link/models/other-j-links/st-link-on-board/

cargo embed works fine with the board FYI...

this could be because cargo-embed was using probe-rs 0.9 and probe-run was using probe-rs 0.8
@kalkyl we have just released probe-run 0.1.6 which uses probe-rs 0.10. Could you give that a try and see if the problem persists?

Hmm, still the same problem using the original st-link firmware....
And still works when i switch to the J-Link firmware

No IT WORKS, sorry i accidently reverted back to an old version of the st-link firmware (which would not work...)
So all good using version  V2J37M26!
Thanks, great job!


[UI Dashboard Time Machine] Switch to script editor and add function starts with cursor in odd place

Steps to reproduce:
List the minimal actions needed to reproduce the behavior.
Generate some test data

Create a dashboard
Create a cell and define the initial query with query builder.  Save the cell.
Click configure on the cell again.  Time machine opens in Query builder view.
Click on the Script Editor.  The query appears as a flux script.

from(bucket: "qa")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r._measurement == "beat")
  |> filter(fn: (r) => r._field == "pulse")

Click on add a function.  e.g. aggregateWindow

Expected behavior:
Describe what you expected to happen.
Expected the function to be added at a logical place in the pipeline. For example at the end of the last call to a filter function generated from query builder.
Actual behavior:
Describe What actually happened.
Function is added at the start of the pipeline.
from(bucket: "qa")
  |> aggregateWindow(every: 1m, fn: mean)
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r._measurement == "beat")
  |> filter(fn: (r) => r._field == "pulse")

Submitting this results in the following error message.
error in building plan while starting program: cannot submit unbounded read to "qa"; try bounding 'from' with a call to 'range'

Generally
Would expect pipeline functions to be appended after the last function in the pipeline.
This can cause confusion for beginner or casual users.
Screencast

Environment info:

System info: Run uname -srm and copy the output here

Linux 4.15.0-76-generic x86_64

InfluxDB version: Run influxd version and copy the output here
Detected in nightly build.

ts=2020-02-13T11:50:50.800628Z lvl=info msg="Welcome to InfluxDB" log_id=0Kwb2Ch0000 version=nightly commit=a17471696 build_date=2020-02-13T11:50:50Z

Other relevant environment details: Container runtime, disk info, etc
Browser: Chrome 79

Config:
Copy any non-default config values here or attach the full config as a gist or file.
Started with e2e testing enabled.
--e2e-testing=true

possibly related to #16668

Will be fixed with #16668


docs: Fix minor issues in docs/usage/applications.rst

Description
Suggestions for better alignment between descriptions and code examples. Also attempted to fix stale link (but not used to rst, so not sure if that works like that).

Closes
Closes #3110
Closes #2688

@all-contributors add @aranvir for docs


No CSS for Table

No CSS rules for table tags
When creating tables in Markdown, the generated HTML files has no defined style to display the table fields/tags.
Using pelican-alchemy:

Using an in-line css:

Hi @alanorth  @raymondwanyoike, kindly check if this is a false positive or not.
I've opened the /theme/static/style/css and I see there are styles for table and two classes defined too, table-responsive and table-bordered.
I've added the following style to the CSS file and the table showed up well..
table{border-collapse:collapse;border-spacing:0;empty-cells:show;border:1px solid #cbcbcb;margin:inherit}
table caption{color:#000;font:inherit;padding:1em 0;text-align:center}
table td, th{border-left:1px solid #cbcbcb;border-width:0 0 0 1px;font-size:inherit;margin:0;overflow:visible;padding:.5em 1em}
table td:first-child, th:first-child{border-left-width:0}
table thead{background-color:#e0e0e0;color:#000;text-align:left;vertical-align:bottom}
table td{background-color:transparent;border-bottom:1px solid #cbcbcb}
table tbody>tr:last-child>td{border-bottom-width:0}

style.css is generated by, or at least comes from, Bootstrap. I think we have to include these outside of there...

Okay.. I know so little about the Bootstrap...

Hallo ,
sorry that i post my question here,but i don't know where to ask a question..
I have saw your wonderful work of “alchemy” pelican theme.
And I want to use your theme as a model to build up my own blog.
But I have some problems to set up the theme of yours.
Could you please give me some supports if you have time, thank you very much !
My blog is http://d-yu.github.io/
As you can see, the profile image cannot show up, as well as the menu items.
I thought it might be the problem of my pelicanconf.py(in attach) configurations.  I have created a images folder under the folder C:\Users\Yu\Dropbox\Blog\Pelican-themes\alchemy\static\images. And put the profile image.gif(200 pix *200 pix) in it.
I am a beginner of python and Html.. I only know a little about c language but I desire to have my own blog.
Thanks a lot for your help.

Regards,
dong

Seems like Pelican-alchemy is going viral... :heart_eyes_cat:

@JohnTroony  hallo, what are you mean? can you give me some tips? i like this theme

Hey @d-yu , You could have opened another issue with a question Label. Your problem is irrelevant to the current issue.

@JohnTroony ok，thanks！ at first i didnot finde where to post...

Fixed in #54

:+1:


Make schema validation more flexible?

Our schema validation is strict. When we add fields to the schema, files that contain them will not be usable in older versions of the publisher. Extension auto-updates mitigate the issue, but they're not always enabled. We need a strategy to deal with this.
Some ideas:

Do nothing - if you're collaborating with or deploying content created by  someone using a newer extension, you need to update. You can work around new fields by dealing with them manually, for example, by removing them.
Allow extra fields and preserve them when writing config files. This means that field name typos won't be errors (we could warn). Any new deployment features called out in the config file won't be applied as specified, because the older client doesn't understand those fields.
If the publisher sees a schema that is not built-in, load it from the schema URL and use that for validation. We would also need to preserve the extra values as above.

The strategy might be different for configurations (user opts in to using new fields, and expects them to be honored) vs. content records (Publisher creates them and decides what information to include).

In the short term, we are taking no action and relying on users who collaborate to use compatible Publisher versions. We have only one schema version at the moment.
When we introduce a new schema version:

We will always write files using the new schema version
We will support reading older files


Update our zipkin config to include the protocol

It complains without it.

This change is

Jenkins job mixer/presubmit passed

Jenkins job mixer/presubmit passed

Jenkins job mixer/pilot-regression passed

Jenkins job mixer/pilot-regression passed

Jenkins job mixer/e2e-smoketest passed


Added user bhavin

Added user bhavin

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Can one of the admins verify this patch?

this pr got polluted w/ other commits, closing this now.


Support anonymous completion counts

It's a common scenario that a quizlord member does quizzes with others that are not interested in joining Quizlord. In these cases it appears they are doing the quiz themselves but they may be doing it with many others. To better support this case an extra "unnamed" count option should be included when entering quiz results and a small (+1), (+3) etc shown next to the named people.

We could also consider optionally supporting adding email addresses of those you are doing the quiz with
This could:

Send a quizlord invite to that person
Once/if the person signed up they could have their quizzes backfilled


Search - check order and query fuzz

Is your feature request related to a problem? Please describe.
check if the order is working properly and add fuzz to the params accepted by the api

fixed in commit


benchmarks: surface zebrad mainnet sync time in CI

Motivation

Sometimes changes (like dependency updates) can affect total sync time, and we want to know this as early in the cycle as possible, so that we don't merge a performance regression.
We currently do a full sync if the other CI jobs pass and the changes get a passing review, and if the full sync is successful, Mergify will proceed with putting the PR into its respective queue. One thing we can do is track the timing results of these full syncs and fail a Mergify merge if the PR goes over the latest benchmark.
However, this still keeps the slowdown signal late in the process. But it's still better than what we have now.
We can spitball other designs.
This GitHub Action  supports Rust (cargo bench) and can store and render results on the gh-pages branch or a custom branch as a JSON blob in a .js file, or in a separate repo, or in an arbitrary file that gets populated however you like in the workflow, say by the cache Action. Supports failing a workflow + make a comment on a PR when benchmarks fall outside a configurable threshold: https://github.com/benchmark-action/github-action-benchmark/tree/master#alert-comment-on-commit-page

Security
Pushing the the GH pages (if used) on external PRs may be risky: https://github.com/benchmark-action/github-action-benchmark/tree/master#run-only-on-your-branches

We currently do a full sync if the other CI jobs pass and the changes get a passing review, and if the full sync is successful, Mergify will proceed with putting the PR into its respective queue. One thing we can do is track the timing results of these full syncs and fail a Mergify merge if the PR goes over the latest benchmark.

At the moment, we're only running a full sync after each PR merges to the main branch:
https://github.com/ZcashFoundation/zebra/blob/8d6957f39b036fc4be93092845fa4872378424bd/.github/workflows/continous-integration-docker.yml#L304
We decided not to run a full sync on each PR, because then each PR batch would take 6 hours to merge.
We could time the mandatory checkpoint sync instead, it only takes 3 hours.

I've marked this ticket as needing a design, and being blocked by sync performance improvements.
If we manage to decrease sync time by a few hours, that will change the design decisions we make.

Mainnet zebrad + lightwalletd full sync times are currently available at:
https://github.com/ZcashFoundation/zebra/actions/workflows/continous-integration-docker.yml?query=branch%3Amain+event%3Aschedule

I rewrote the issue description instead of closing it, because we still need some way to find out when the sync is slow.

This has been done already by automatically opening a ticket when the weekly scheduled full sync job fails in PR #7565.
We also have an open ticket for doing a sync with a few tens or hundreds of thousands of blocks, and timing it:

#7649