Instructions to use Qwen/Qwen3-235B-A22B-Thinking-2507-FP8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Qwen/Qwen3-235B-A22B-Thinking-2507-FP8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Qwen/Qwen3-235B-A22B-Thinking-2507-FP8")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-235B-A22B-Thinking-2507-FP8")
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-235B-A22B-Thinking-2507-FP8")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use Qwen/Qwen3-235B-A22B-Thinking-2507-FP8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Qwen/Qwen3-235B-A22B-Thinking-2507-FP8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Qwen/Qwen3-235B-A22B-Thinking-2507-FP8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Qwen/Qwen3-235B-A22B-Thinking-2507-FP8

SGLang

How to use Qwen/Qwen3-235B-A22B-Thinking-2507-FP8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Qwen/Qwen3-235B-A22B-Thinking-2507-FP8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Qwen/Qwen3-235B-A22B-Thinking-2507-FP8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Qwen/Qwen3-235B-A22B-Thinking-2507-FP8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Qwen/Qwen3-235B-A22B-Thinking-2507-FP8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Qwen/Qwen3-235B-A22B-Thinking-2507-FP8 with Docker Model Runner:
```
docker model run hf.co/Qwen/Qwen3-235B-A22B-Thinking-2507-FP8
```

The start of <think> is not been used on assistant response using vllm 0.13

by celsowm - opened Dec 19, 2025

Discussion

celsowm

Dec 19, 2025

•

edited Dec 19, 2025

Hi !
After uses:

docker run -d --name vllm-qwen-235b \
  --gpus '"device=1,2,3,4"' \
  --ipc=host \
  -p 8001:8001 \
  -v /srv/models:/models \
  vllm/vllm-openai:latest \
  --model /models/qwen3-235b-fp8 \
  --port 8001 \
  --tensor-parallel-size 4 \
  --max-model-len 65536 \
  --enable-auto-tool-choice \
  --tool-call-parser hermes \
  --enable-chunked-prefill

I tried:

root@srv-ia-020:/home/fontesc# curl -s http://10.120.191.20:8001/v1/chat/completions   -H "Content-Type: application/json"   -H "Authorization: Bearer dummy"   -d '{
    "model": "/models/qwen3-235b-fp8",
    "messages": [{"role":"user","content":"how many is 1+1?"}]
  }'

but the answer is:

{"id":"chatcmpl-a552f115b7aa5d68","object":"chat.completion","created":1766171761,"model":"/models/qwen3-235b-fp8","choices":[{"index":0,"message":{"role":"assistant","content":"Okay, the user asked "how many is 1+1?" Hmm, this seems like a very basic math question. Maybe they're a young child just learning addition, or perhaps someone testing if I'm paying attention. \n\nI should keep it simple and clear. The answer is obviously 2, but let me think if there's any trick here. In standard arithmetic? Definitely 2. In binary? Still 10 which is two. In weird modular systems? Nah, overcomplicating it. \n\nUser probably wants the straightforward answer. I'll say "2" but add a friendly explanation since they said "how many" - shows they might be thinking concretely. Should I mention it's the foundation of math? Yeah, a little educational touch won't hurt. \n\n...Wait, is this a meme reference? Like the "1+1=3" propaganda thing? No, the question seems genuine. Better assume good faith. \n\nSmiley face at the end. Kids like those. If it's an adult testing me, still harmless. Done.\n\n\nThe answer to 1 + 1 is 2. \n\nThis is the most fundamental operation in arithmetic: \n- 1 (one unit) + 1 (another unit) = 2 (two units). \n\nIt’s the foundation of counting, addition, and all higher mathematics! 😊 \nLet me know if you'd like to explore more math concepts!","refusal":null,"annotations":null,"audio":null,"function_call":null,"tool_calls":[],"reasoning":null,"reasoning_content":null},"logprobs":null,"finish_reason":"stop","stop_reason":null,"token_ids":null}],"service_tier":null,"system_fingerprint":null,"usage":{"prompt_tokens":18,"total_tokens":323,"completion_tokens":305,"prompt_tokens_details":null},"prompt_logprobs":null,"prompt_token_ids":null,"kv_transfer_params":null}root@srv-ia-020:/home/fontesc#

as you can see the first token is not <think> is directly Okay

What wrong am I doing?

yecp

Mar 6

I encountered the same issue, finally I set --reasoning-parser deepseek-r1 to resolve it.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment