| --- |
| library_name: transformers |
| license: apache-2.0 |
| --- |
| |
| [Model checkpoints will be released soon.] |
|
|
| ### Model Details |
| We introduce a new streaming paradigm that enables large language models to achieve strong performance and generalization in streaming settings, without requiring any architectural modifications. |
| * Streaming-processing: The LLMs process the input as it arrives, incrementally and in real time. |
| <img src="streaming.gif" alt="Streaming-processing" width="900"/> |
|
|
| * Batch-processing: The LLMs process inputs all at once after receiving the full sequence. |
| <img src="batch.gif" alt="Batch-processing" width="900"/> |
|
|
|
|
| ### Model Sources |
| - **Paper:** https://arxiv.org/abs/2505.16983 |
| - **Repository:** https://github.com/EIT-NLP/StreamingLLM |
|
|
|
|
|
|
|
|
| ### Citation |
| ```tex |
| @misc{tong2025llmeffectivestreamingprocessor, |
| title={LLM as Effective Streaming Processor: Bridging Streaming-Batch Mismatches with Group Position Encoding}, |
| author={Junlong Tong and Jinlan Fu and Zixuan Lin and Yingqi Fan and Anhao Zhao and Hui Su and Xiaoyu Shen}, |
| year={2025}, |
| eprint={2505.16983}, |
| archivePrefix={arXiv}, |
| primaryClass={cs.CL}, |
| url={https://arxiv.org/abs/2505.16983}, |
| } |
| ``` |
