Instructions to use Intel/gpt-j-6B-pytorch-int8-static-inc with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Intel/gpt-j-6B-pytorch-int8-static-inc with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Intel/gpt-j-6B-pytorch-int8-static-inc")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Intel/gpt-j-6B-pytorch-int8-static-inc")
model = AutoModelForCausalLM.from_pretrained("Intel/gpt-j-6B-pytorch-int8-static-inc")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Intel/gpt-j-6B-pytorch-int8-static-inc with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Intel/gpt-j-6B-pytorch-int8-static-inc"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Intel/gpt-j-6B-pytorch-int8-static-inc",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/Intel/gpt-j-6B-pytorch-int8-static-inc

SGLang

How to use Intel/gpt-j-6B-pytorch-int8-static-inc with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Intel/gpt-j-6B-pytorch-int8-static-inc" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Intel/gpt-j-6B-pytorch-int8-static-inc",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Intel/gpt-j-6B-pytorch-int8-static-inc" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Intel/gpt-j-6B-pytorch-int8-static-inc",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use Intel/gpt-j-6B-pytorch-int8-static-inc with Docker Model Runner:
```
docker model run hf.co/Intel/gpt-j-6B-pytorch-int8-static-inc
```

INT8 GPT-J 6B

Model Description

GPT-J 6B is a transformer model trained using Ben Wang's Mesh Transformer JAX. "GPT-J" refers to the class of model, while "6B" represents the number of trainable parameters.

This int8 PyTorch model is generated by intel-extension-for-transformers.

Package	Version
intel-extension-for-transformers	a4aba8ddb07c9b744b6ac106502ec059e0c47960
neural-compressor	2.4.1
torch	2.1.0+cpu
intel-extension-for-pytorch	2.1.0
transformers	4.32.0

Usage

Currently, we only support the method of downloading the model and then loading it. In this approach, the model files are downloaded from the server and stored locally on the user's machine.

Clone this model repository

# Make sure you have git-lfs installed (https://git-lfs.com)
git lfs install
git clone https://huggingface.co/Intel/gpt-j-6B-pytorch-int8-static

Load int8 model

from intel_extension_for_transformers.llm.evaluation.models import TSModelCausalLMForITREX

user_model = TSModelCausalLMForITREX.from_pretrained(
    args.output_dir, # Your saved path
    file_name="best_model.pt",
    trust_remote_code=args.trust_remote_code, # Default is False
)

Evaluation results

Evaluating the accuracy of the optimized model of gpt-j-6b using the lambada_openai dataset in lm_eval.

Dtype	Dataset	Precision
FP32	Lambada_openai	0.6831
INT8	Lambada_openai	0.6835

Downloads last month: 11

Dataset used to train Intel/gpt-j-6B-pytorch-int8-static-inc

Collection including Intel/gpt-j-6B-pytorch-int8-static-inc

GPT

Collection

Series of GPT fine-tuned models • 5 items • Updated Mar 2 • 1