Instructions to use backups/dots.llm1.base with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use backups/dots.llm1.base with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="backups/dots.llm1.base")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("backups/dots.llm1.base")
model = AutoModelForCausalLM.from_pretrained("backups/dots.llm1.base")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use backups/dots.llm1.base with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "backups/dots.llm1.base"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "backups/dots.llm1.base",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/backups/dots.llm1.base

SGLang

How to use backups/dots.llm1.base with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "backups/dots.llm1.base" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "backups/dots.llm1.base",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "backups/dots.llm1.base" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "backups/dots.llm1.base",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use backups/dots.llm1.base with Docker Model Runner:
```
docker model run hf.co/backups/dots.llm1.base
```

dots.llm1.base / README.md

mrfakename

Duplicate from rednote-hilab/dots.llm1.base

9d5c891 verified 12 months ago

preview code

raw

history blame contribute delete

7.9 kB

	---
	license: mit
	license_link: https://huggingface.co/rednote-hilab/dots.llm1.base/blob/main/LICENSE
	library_name: transformers
	language:
	- en
	- zh
	---

	# dots1

	<p align="center">
	<img src="figures/new_logo2.png" width="300"/>
	<p>

	<p align="center">
	&nbsp&nbsp🤗 <a href="https://huggingface.co/rednote-hilab">Hugging Face</a>&nbsp&nbsp \| &nbsp&nbsp 📑 <a href="https://github.com/rednote-hilab/dots.llm1/blob/main/dots1_tech_report.pdf">Paper</a> &nbsp&nbsp
	<br>
	🖥️ <a href="https://huggingface.co/spaces/rednote-hilab/dots-demo">Demo</a>&nbsp&nbsp \| &nbsp&nbsp💬 <a href="figures/wechat.png">WeChat (微信)</a>&nbsp&nbsp \| &nbsp&nbsp📕 <a href="https://www.xiaohongshu.com/user/profile/683ffe42000000001d021a4c">rednote</a>&nbsp&nbsp
	</p>


	Visit our Hugging Face (click links above), search checkpoints with names starting with `dots.llm1` or visit the [dots1 collection](https://huggingface.co/collections/rednote-hilab/dotsllm1-68246aaaaba3363374a8aa7c), and you will find all you need! Enjoy!


	## News

	- 2025.06.06: We released the `dots.llm1` series. Check our [report](https://github.com/rednote-hilab/dots.llm1/blob/main/dots1_tech_report.pdf) for more details!


	## 1. Introduction


	The `dots.llm1` model is a large-scale MoE model that activates 14B parameters out of a total of 142B parameters, delivering performance on par with state-of-the-art models.
	Leveraging our meticulously crafted and efficient data processing pipeline, `dots.llm1` achieves performance comparable to Qwen2.5-72B after pretrained on 11.2T high-quality tokens without synthetic data. To foster further research, we open-source intermediate training checkpoints at every one trillion tokens, providing valuable insights into the learning dynamics of large language models.


	<p align="center">
	<img width="90%" src="./figures/performance.png">
	</p>

	## 2. Model Summary

	This repo contains the base and instruction-tuned `dots.llm1` model. which has the following features:

	- Type: A MoE model with 14B activated and 142B total parameters trained on 11.2T tokens.
	- Training Stages: Pretraining and SFT.
	- Architecture: Multi-head Attention with QK-Norm in attention Layer, fine-grained MoE utilizing top-6 out of 128 routed experts, plus 2 shared experts.
	- Number of Layers: 62
	- Number of Attention Heads: 32
	- Supported Languages: English, Chinese
	- Context Length: 32,768 tokens
	- License: MIT

	The highlights from `dots.llm1` include:

	- Enhanced Data Processing: We propose a scalable and fine-grained three-stage data processing framework designed to generate large-scale, high-quality and diverse data for pretraining.
	- No Synthetic Data during Pretraining: 11.2 trillion high-quality non-synthetic tokens was used in base model pretraining.
	- Performance and Cost Efficiency: `dots.llm1` is an open-source model that activates only 14B parameters at inference, delivering both comprehensive capabilities and high computational efficiency.
	- Infrastructure: We introduce an innovative MoE all-to-all communication and computation overlapping recipe based on interleaved 1F1B pipeline scheduling and an efficient grouped GEMM implementation to boost computational efficiency.
	- Open Accessibility to Model Dynamics: Intermediate model checkpoints for every 1T tokens trained are released, facilitating future research into the learning dynamics of large language models.

	## 3. Example Usage

	### Model Downloads

	<div align="center">

	\| Model \| #Total Params \| #Activated Params \| Context Length \| Download Link \|
	\| :------------: \| :------------: \| :------------: \| :------------: \| :------------: \|
	\| dots.llm1.base \| 142B \| 14B \| 32K \| [🤗 Hugging Face](https://huggingface.co/rednote-hilab/dots.llm1.base) \|
	\| dots.llm1.inst \| 142B \| 14B \| 32K \| [🤗 Hugging Face](https://huggingface.co/rednote-hilab/dots.llm1.inst) \|

	</div>

	### Docker (recommended)


	The docker images are available on [Docker Hub](https://hub.docker.com/repository/docker/rednotehilab/dots1/tags), based on the official images.

	You can start a server via vllm.

	```shell
	docker run --gpus all \
	-v ~/.cache/huggingface:/root/.cache/huggingface \
	-p 8000:8000 \
	--ipc=host \
	rednotehilab/dots1:vllm-openai-v0.9.0.1 \
	--model rednote-hilab/dots.llm1.inst \
	--tensor-parallel-size 8 \
	--trust-remote-code \
	--served-model-name dots1
	```

	Then you can verify whether the model is running successfully in the following way.

	```shell
	curl http://localhost:8000/v1/chat/completions \
	-H "Content-Type: application/json" \
	-d '{
	"model": "dots1",
	"messages": [
	{"role": "system", "content": "You are a helpful assistant."},
	{"role": "user", "content": "Who won the world series in 2020?"}
	],
	"max_tokens": 32,
	"temperature": 0
	}'
	```


	### Inference with huggingface

	We are working to merge it into Transformers ([PR #38143](https://github.com/huggingface/transformers/pull/38143)).

	#### Text Completion

	```python
	import torch
	from transformers import AutoTokenizer, AutoModelForCausalLM, GenerationConfig

	model_name = "rednote-hilab/dots.llm1.base"
	tokenizer = AutoTokenizer.from_pretrained(model_name)

	model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto", torch_dtype=torch.bfloat16)

	text = "An attention function can be described as mapping a query and a set of key-value pairs to an output, where the query, keys, values, and output are all vectors. The output is"
	inputs = tokenizer(text, return_tensors="pt")
	outputs = model.generate(**inputs.to(model.device), max_new_tokens=100)
	result = tokenizer.decode(outputs[0], skip_special_tokens=True)
	print(result)
	```

	#### Chat Completion

	```python
	import torch
	from transformers import AutoTokenizer, AutoModelForCausalLM, GenerationConfig

	model_name = "rednote-hilab/dots.llm1.inst"
	tokenizer = AutoTokenizer.from_pretrained(model_name)

	model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto", torch_dtype=torch.bfloat16)

	messages = [
	{"role": "user", "content": "Write a piece of quicksort code in C++"}
	]
	input_tensor = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
	outputs = model.generate(input_tensor.to(model.device), max_new_tokens=200)

	result = tokenizer.decode(outputs[0][input_tensor.shape[1]:], skip_special_tokens=True)
	print(result)
	```

	### Inference with vllm

	[vLLM](https://github.com/vllm-project/vllm) is a high-throughput and memory-efficient inference and serving engine for LLMs. Official support for this feature is covered in [PR #18254](https://github.com/vllm-project/vllm/pull/18254).

	```shell
	vllm serve dots.llm1.inst --port 8000 --tensor-parallel-size 8
	```

	An OpenAI-compatible API will be available at `http://localhost:8000/v1`.

	### Inference with sglang

	[SGLang](https://github.com/sgl-project/sglang) is a fast serving framework for large language models and vision language models. SGLang could be used to launch a server with OpenAI-compatible API service. Official support for this feature is covered in [PR #6471](https://github.com/sgl-project/sglang/pull/6471).

	Getting started is as simple as running:

	```shell
	python -m sglang.launch_server --model-path dots.llm1.inst --tp 8 --host 0.0.0.0 --port 8000
	```

	An OpenAI-compatible API will be available at `http://localhost:8000/v1`.

	## 4. Evaluation Results

	Detailed evaluation results are reported in this [📑 report](https://github.com/rednote-hilab/dots.llm1/blob/main/dots1_tech_report.pdf).

	## Citation

	If you find `dots.llm1` is useful or want to use in your projects, please kindly cite our paper:

	```
	@article{dots1,
	title={dots.llm1 Technical Report},
	author={rednote-hilab},
	journal={arXiv preprint arXiv:TBD},
	year={2025}
	}
	```