Instructions to use LumiOpen/Poro-34B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use LumiOpen/Poro-34B with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="LumiOpen/Poro-34B")# Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("LumiOpen/Poro-34B") model = AutoModelForCausalLM.from_pretrained("LumiOpen/Poro-34B") - Notebooks
- Google Colab
- Kaggle
- Local Apps
- vLLM
How to use LumiOpen/Poro-34B with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "LumiOpen/Poro-34B" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "LumiOpen/Poro-34B", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/LumiOpen/Poro-34B
- SGLang
How to use LumiOpen/Poro-34B with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "LumiOpen/Poro-34B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "LumiOpen/Poro-34B", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "LumiOpen/Poro-34B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "LumiOpen/Poro-34B", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use LumiOpen/Poro-34B with Docker Model Runner:
docker model run hf.co/LumiOpen/Poro-34B
Commit History
Update README.md e9b22a3 verified
Aarne Talman commited on
Update README.md 42bcd40 verified
Update README.md e6580e9 verified
add citation information 7244b58
jonabur commited on
Update README.md 8cd74a7 verified
Aarne Talman commited on
Update README.md e24b393 verified
Aarne Talman commited on
update README for final release e9441f3
jonabur commited on
add 700B checkpoint 9f3d465
jonabur commited on
add 600B checkpoints dc0e31c
jonabur commited on
update for 500B release c251a8a
jonabur commited on
Update README.md f09efad
Update README.md 3949d66
Update README.md b270339
update note about GAS 4118a0f
jonabur commited on
add HPLT acknowledgment 564a58e
jonabur commited on
improve descriptions 8502bcf
jonabur commited on
add model card 7733eca
jonabur commited on
add logo 8f66e4d
jonabur commited on