[WIP] Support int8 KVCacheQuant and W8A8 inference in vllm

We have recently implemented and tested int8 KV-Cache quantization and W8A8 inference in vLLM. We find out that our quantization implementation can increase the throughput by about 20% and reduce the first token latency under heavy load. In contrast,  W4A16 quant methods(eg. AWQ-based method) provided in vllm cannot improve throughput according to pr1032 because it can not benefit from int8 tensor core.  So we propose this PR as an alternative quantization method.
What we have right now:

int8 KV-Cache quantization related works:
a. Quant\Dequant helper functions adapted from Faster Transformer
b. Quantized version CUDA kernels
c. Unit tests for the added kernels
W8A8 inference related works:
a. Int8 Gemm kernels adapted from torch-int
b. W8A8 linear layer modules
c. Support W8A8 inference on Llama model
Test result based on our own dataset

What we plan to do:

Further kernel fusion
Code refactoring and cleaning
Opimize int8 GEMM kernel
Release SmoothQuant for LLaMA
Add code for generating KV-Cache quantization parameters (scales and zero points)
Experiments on more datasets

PS.
current test result in our datasets:
Throughput of FP16 LLaMA-13B:
Throughput: 4.8510 requests/s, 527.4584 token/s

Throughput of Int8 LLaMA-13B with int8 KVCacheQuant:
Throughput: 5.8764 requests/s, 638.9578 token/s

Updates
We have released SmoothQuant for LLaMA in
https://github.com/AniZpZ/smoothquant/tree/llama-dev
https://github.com/AniZpZ/torch-int/tree/llama-dev
The code for generating KV-Cache quantization parameters is on the way

I fully support this, since the 4-bit AWQ model proved to have inferior quality for my use cases. Having 8 bit weights with 8 bit activation cache would be the best of both worlds, allowing for almost no loss of quality (perplexity) while being able to more efficiently. I would also keep the W8A16 mode as an option, should the precision of the cache count in same use cases.

Hi vLLM genius @WoosukKwon @zhuohan123
This is the latest development from our team regarding quantitative support for vllm, we have done something similar to https://github.com/vllm-project/vllm/pull/1032 before. At that time, we didn't mention pr after the benchmark results showed a drop in throughput, but later we found out that https://github.com/vllm-project/vllm/pull/1032 was merged, which is very encouraging. Therefore, we continue to do performance optimization on this basis, and send out the pr in WIP state in advance, hoping to get some comments and suggestions, and finally merge into the vllm codebase smoothly. Cheers!

@AniZpZ @zhyncs This is great work! My understanding is that this PR converts FP16 -> INT8 dynamically without computing a loss function to optimize perplexity. Have you evaluated perplexity on this approach?

@AniZpZ @zhyncs This is great work! My understanding is that this PR converts FP16 -> INT8 dynamically without computing a loss function to optimize perplexity. Have you evaluated perplexity on this approach?

We implement quantization with smoothquant method for W8A8 I will release the code later. The perplexity is identical to a standard smoothquant method if you do W8A8 inference without int8 KVCacheQuant

@AniZpZ @zhyncs This is great work! My understanding is that this PR converts FP16 -> INT8 dynamically without computing a loss function to optimize perplexity. Have you evaluated perplexity on this approach?

We implement quantization with smoothquant method for W8A8 I will release the code later. The perplexity is identical to a standard smoothquant method if you do W8A8 inference without int8 KVCacheQuant.
Quantization details are discussed in this paper(Xiao et. al)

SmoothQuant only supports OPT models. How can we test this PR when the SmoothQuant repository does not support LLaMa models? If you implement this PR without the quantization code, you will inevitably end up with a bad perplexity if you naively use W8A8 as you have no calibration dataset.
See this image, accuracy ends up being worse than INT4 if you naively convert weights to W8A8. You need the SmoothQuant or AWQ method to convert if you want to preserve accuracy. You need a framework for this, which is why I created AutoAWQ - I will look to implement INT8 quantization using the torch-int modules and would love your help with this so we can support all models in vLLM (LLaMa, MPT, Falcon, etc.) without accuracy degradation.

@AniZpZ @zhyncs This is great work! My understanding is that this PR converts FP16 -> INT8 dynamically without computing a loss function to optimize perplexity. Have you evaluated perplexity on this approach?

@AniZpZ @zhyncs This is great work! My understanding is that this PR converts FP16 -> INT8 dynamically without computing a loss function to optimize perplexity. Have you evaluated perplexity on this approach?

We implement quantization with smoothquant method for W8A8 I will release the code later. The perplexity is identical to a standard smoothquant method if you do W8A8 inference without int8 KVCacheQuant.
Quantization details are discussed in this paper(Xiao et. al)

SmoothQuant only supports OPT models. How can we test this PR when the SmoothQuant repository does not support LLaMa models? If you implement this PR without the quantization code, you will inevitably end up with a bad perplexity if you naively use W8A8 as you have no calibration dataset.
See this image, accuracy ends up being worse than INT4 if you naively convert weights to W8A8. You need the SmoothQuant or AWQ method to convert if you want to preserve accuracy. You need a framework for this, which is why I created AutoAWQ - I will look to implement INT8 quantization using the torch-int modules and would love your help with this so we can support all models in vLLM (LLaMa, MPT, Falcon, etc.) without accuracy degradation.

We implement smoothquant for llama ourself, you can find code here: https://github.com/AniZpZ/smoothquant/tree/llama-dev

Hi @AniZpZ @zhyncs, thank you for your great work with this PR.
I have now had more time to explore your fast implementation and found that Nvidia only has support for INT8 for high throughput, which makes this PR achieve higher throughput than INT4 due to software capabilities.
Is your proposal to run W8A16? Your code does not have A8 implemented in the llama.py model definition.
SmoothQuant implements W8A8, but it seems silly to run A8 as there should be little benefit speed-wise. Therefore, I see this as a natural choice. I want to confirm this with you for my implementation in AutoAWQ as I want to push INT8 models out using your initial LLaMa implementation, just using the AWQ method for minimum perplexity loss.

Hi @AniZpZ @zhyncs, thank you for your great work with this PR.
I have now had more time to explore your fast implementation and found that Nvidia only has support for INT8 for high throughput, which makes this PR achieve higher throughput than INT4 due to software capabilities.
Is your proposal to run W8A16? Your code does not have A8 implemented in the llama.py model definition.
SmoothQuant implements W8A8, but it seems silly to run A8 as there should be little benefit speed-wise. Therefore, I see this as a natural choice. I want to confirm this with you for my implementation in AutoAWQ as I want to push INT8 models out using your initial LLaMa implementation, just using the AWQ method for minimum perplexity loss.

Our proposal is to run in W8A8. If you enable smoothquant, we will replace rmsnorm and linear layer with our custom int8 layernorms and w8a8linears which quant activations and impelement int8 gemm. You can find the detail in w8a8linear.py

I agree that the RMSNorm and Linear layers run in INT8, making it W8. Running in A8 means running your activation functions with INT8, and this is not implemented (see below). Currently, this makes it W8A16.
Your MLP forward runs .half(), making the precision FP16 before running the Silu activation function.
Please see my annotated comments:
    def forward(self, x):
        gate_up, _ = self.gate_up_proj(x) # <-- INT8 inference (W8)
        gate_up = gate_up.half() # <-- INT8 to FP16 conversion
        x = self.act_fn(gate_up) # <-- FP16 Silu activation function (A16)
        x, _ = self.down_proj(x)
        x = x.half()
        return x

Is it possible to keep the W8A16 mode as well?

@viktor-ferenczi I am much in favor of W8A16 which is currently implemented. Quantized models will be easier to create in the W8A16 format without accuracy degradation.

@AniZpZ @zhyncs I believe throughput can be increased more if you implement PagedAttention as INT8. Do you have any idea if that is possible because it could increase throughput much more?
This loop dequantizes from INT8, but you could instead apply the INT8 BMM from torch-int if adapted to work with PagedAttention. @WoosukKwon perhaps you are better qualified to answer this.
https://github.com/AniZpZ/vllm/blob/kv-quant-merge/csrc/attention/attention_kernels.cu#L432

PagedAttention

We repalce the input and post layer norm of attention to I8rmsnorm and make the output of layernorm to in8 that makes qkv gemm and gateup gemm run in w8a8. And we use W8A8BFP32OFP32LinearWithSFactor in o_proj and down_proj. W8A8BFP32OFP32LinearWithSFactor quant input activations to int8 and do w8a8 gemm as well.

@viktor-ferenczi I am much in favor of W8A16 which is currently implemented. Quantized models will be easier to create in the W8A16 format without accuracy degradation.
@AniZpZ @zhyncs I believe throughput can be increased more if you implement PagedAttention as INT8. Do you have any idea if that is possible because it could increase throughput much more?
This loop dequantizes from INT8, but you could instead apply the INT8 BMM from torch-int if adapted to work with PagedAttention. @WoosukKwon perhaps you are better qualified to answer this. https://github.com/AniZpZ/vllm/blob/kv-quant-merge/csrc/attention/attention_kernels.cu#L432

You're correct that implementing PagedAttention as INT8 could potentially improve performance. However, our profiling of the inference process assumes that the gemm computation in PagedAttention only accounts for approximately 5% of the total， which suggests that performance gains from this particular optimization might be limited. We current focus is on kernel fusion and optimizing the INT8 GEMM kernel. After finishing works mentioned above, we will try to implemented PagedAttention as INT8.

it's possible to run in W8A16 but you can not benefit from int8 tensor core in that case

We have a misunderstanding of what W8 and A8 means here.

torch-int W8A8: Inputs = INT8/A8, Weights = INT8/W8 in W8A8BFP32OFP32LinearWithSFactor
In a normal sense, I would call this W8A16 because activation functions are running in FP16.

PagedAttention only accounts for approximately 5% of the total

Thank you for sharing those numbers with me. Your work in this PR is amazing, the performance is unmatched! I am looking forward to seeing how this develops and to support W8 quantization with AutoAWQ.

6.1147

We replaced cutlass gemm kernels adapted from torch-int with cublas gemm kernels and achieve 6% improvement on throughput.

I have implemented SmoothQuant quantization of input activations in AutoAWQ/smoothquant. However, I am getting an error around how the model is loaded.
This is the first layer when I print the state_dict.keys() in vLLM: model.layers.0.self_attn.qkv_proj.weight. However, the llama model definition looks a bit different in general. @AniZpZ can we sync up our versions to make this run smoothly? Ideally we can quantize models using AutoAWQ and we can have TheBloke help push some INT8 models.
LlamaForCausalLM(
  (model): LlamaModel(
    (embed_tokens): Embedding(32000, 4096, padding_idx=0)
    (layers): ModuleList(
      (0-31): 32 x LlamaDecoderLayer(
        (self_attn): LlamaAttention(
          (q_proj): WQLinear_INT8()
          (k_proj): WQLinear_INT8()
          (v_proj): WQLinear_INT8()
          (o_proj): WQLinear_INT8()
          (rotary_emb): LlamaRotaryEmbedding()
        )
        (mlp): LlamaMLP(
          (gate_proj): WQLinear_INT8()
          (up_proj): WQLinear_INT8()
          (down_proj): WQLinear_INT8()
          (act_fn): SiLUActivation()
        )
        (input_layernorm): LlamaRMSNorm()
        (post_attention_layernorm): LlamaRMSNorm()
      )
    )
    (norm): LlamaRMSNorm()
  )
  (lm_head): Linear(in_features=4096, out_features=32000, bias=False)
)

Error:
Traceback (most recent call last):
  File "/workspace/AutoAWQ/vllm/test.py", line 16, in <module>
    llm = LLM(model="../vicuna-7b-v1.5-awq", **{'quantization': 'smoothquant'})
  File "/workspace/AutoAWQ/vllm/vllm/entrypoints/llm.py", line 73, in __init__
    self.llm_engine = LLMEngine.from_engine_args(engine_args)
  File "/workspace/AutoAWQ/vllm/vllm/engine/llm_engine.py", line 226, in from_engine_args
    engine = cls(*engine_configs,
  File "/workspace/AutoAWQ/vllm/vllm/engine/llm_engine.py", line 105, in __init__
    self._init_workers(distributed_init_method)
  File "/workspace/AutoAWQ/vllm/vllm/engine/llm_engine.py", line 137, in _init_workers
    self._run_workers(
  File "/workspace/AutoAWQ/vllm/vllm/engine/llm_engine.py", line 685, in _run_workers
    output = executor(*args, **kwargs)
  File "/workspace/AutoAWQ/vllm/vllm/worker/worker.py", line 67, in init_model
    self.model = get_model(self.model_config, self.parallel_config, self.rank)
  File "/workspace/AutoAWQ/vllm/vllm/model_executor/model_loader.py", line 112, in get_model
    model.load_weights(model_config.model, model_config.download_dir,
  File "/workspace/AutoAWQ/vllm/vllm/model_executor/models/llama.py", line 346, in load_weights
    return self._load_int8_weights(
  File "/workspace/AutoAWQ/vllm/vllm/model_executor/models/llama.py", line 514, in _load_int8_weights
    param = state_dict[name]
KeyError: 'model.layers.0.self_attn.q_proj.weight'

state_dict

VLLM use qkv_proj to replace q_poj,k_proj and v_proj. So there has no model.layers.0.self_attn.q_proj.weight in state_dict.keys()

@AniZpZ
Loading works for now with this PR: https://github.com/AniZpZ/vllm/pull/1. Could you please accept the PR? Then I will focus on optimizing the accuracy/perplexity of models.
Also, what is the command to generate INT8 KV-Cache?

@AniZpZ Loading works for now with this PR: AniZpZ#1. Could you please accept the PR? Then I will focus on optimizing the accuracy/perplexity of models.
Also, what is the command to generate INT8 KV-Cache?

Thank you for you support!  We do not want to change origin model load method and trying to do minimum modify. Furthermore, we are trying to support tensor parallel in w8a8 linear. Thus I am going to do some code refactor later and will not accept the pr right now.
If you want to generate INT8 KV-Cache, there are two steps, first use vllm/kv_quant/calibrate.py and a calibration dataset to do the calibration. And then use vllm/kv_quant/export_kv_params.py to do the quanted param export.
You can run following command for usage details
python vllm/kv_quant/calibrate.py --help
python vllm/kv_quant/export_kv_params.py --help

Please look at the model loading code soon. Until then, I must use my fork to continue my development. The goal is to create W8A8 models with the KV8 cache smoothly without garbage output. Currently, I cannot get any models to generate normal output in vLLM, looking into this.

Please look at the model loading code soon. Until then, I must use my fork to continue my development. The goal is to create W8A8 models with the KV8 cache smoothly without garbage output. Currently, I cannot get any models to generate normal output in vLLM, looking into this.
EDIT: I have optimized the scaling of the llama model using SmoothQuant method, and the results are unfortunately not good for Llama models. It suggests we need new methods for quantizing into a W8A8 format.
This result is obtained by using TheBloke/Llama-2-7b-chat-fp16 and running my SmoothQuant repository which is optimized for better perplexity.
Quantization Type	Scale	Perplexity	% Higher than FP16
W16A16	N/A	6.365	N/A
W8A8	0.4	21.62	239.73%
W8A8	0.45	16.97	166.55%
W8A8	0.5	15.80	148.26%
W8A8	0.55	15.68	146.37%
W8A8	0.6	16.93	165.95%

Thank you for your experiment data, we will look into the problem. Our earlier experiments focused on accuracy in datasets like mmlu. We found there were accuracy drops, but they were within an acceptable range.

@AniZpZ smooth quant only get high accuracy with activation per-token dynamic quantization, weight per-channel quantization.

@AniZpZ smooth quant only get high accuracy with activation per-token dynamic quantization, weight per-channel quantization.

Yes, activation  per-token dynamic quantization, weight per-channel quantization can achieve  higher accuracy than activation per-tensor static, weight tensor-wise quantization. However the difference is minor according to smoothquant paper. We will look into the perplexity problem soon.

@AniZpZ the root cause is activation is still too large. you can dump the down proj's input tensor.
if you disable llama model down proj quantization, accuracy will get improved a lot.
if you disable output proj  futher, the final accuracy will only drop about 0.2 PPL.
layer: model.layers.2.self_attn.q_proj min: 0.00 max: 7.50
layer: model.layers.2.self_attn.q_proj_out min: 0.69 max: 11.41
layer: model.layers.2.self_attn.k_proj min: 0.00 max: 7.50
layer: model.layers.2.self_attn.k_proj_out min: 0.73 max: 16.66
layer: model.layers.2.self_attn.v_proj min: 0.00 max: 7.50
layer: model.layers.2.self_attn.v_proj_out min: 0.27 max: 4.79
layer: model.layers.2.self_attn.o_proj min: 0.12 max: 4.79
layer: model.layers.2.self_attn.o_proj_out min: 0.20 max: 9.45
layer: model.layers.2.mlp.gate_proj min: 0.26 max: 13.90
layer: model.layers.2.mlp.gate_proj_out min: 0.64 max: 37.94
layer: model.layers.2.mlp.up_proj min: 0.26 max: 13.90
layer: model.layers.2.mlp.up_proj_out min: 0.68 max: 33.28
layer: model.layers.2.mlp.down_proj min: 0.19 max: 1263.00
layer: model.layers.2.mlp.down_proj_out min: 0.28 max: 2426.00

In our work, performance impoved about 30-50% in this case compared to fp16.

2426.00

@AniZpZ the root cause is activation is still too large. you can dump the down proj's input tensor. if you disable llama model down proj quantization, accuracy will get improved a lot. if you disable output proj futher, the final accuracy will only drop about 0.2 PPL. And Opt model's activation is not such abnormal.
Here's llama activation and output MinMax.
layer: model.layers.2.self_attn.q_proj min: 0.00 max: 7.50
layer: model.layers.2.self_attn.q_proj_out min: 0.69 max: 11.41
layer: model.layers.2.self_attn.k_proj min: 0.00 max: 7.50
layer: model.layers.2.self_attn.k_proj_out min: 0.73 max: 16.66
layer: model.layers.2.self_attn.v_proj min: 0.00 max: 7.50
layer: model.layers.2.self_attn.v_proj_out min: 0.27 max: 4.79
layer: model.layers.2.self_attn.o_proj min: 0.12 max: 4.79
layer: model.layers.2.self_attn.o_proj_out min: 0.20 max: 9.45
layer: model.layers.2.mlp.gate_proj min: 0.26 max: 13.90
layer: model.layers.2.mlp.gate_proj_out min: 0.64 max: 37.94
layer: model.layers.2.mlp.up_proj min: 0.26 max: 13.90
layer: model.layers.2.mlp.up_proj_out min: 0.68 max: 33.28
layer: model.layers.2.mlp.down_proj min: 0.19 max: 1263.00
layer: model.layers.2.mlp.down_proj_out min: 0.28 max: 2426.00

In our work, performance impoved about 30-50% in this case compared to fp16.

We got it. The root cause is that the smooth operation do not work on down proj and output proj for now. The silu function    in mlp and the rotary embedding make us have to do dequantization after qkv proj and gate up proj,  which will lead to big activations. However, the dequantization is not neccessry in opt model, activations are kept within a limited range throughout the entire MLP and attention process.

I have conducted more experiments that achieve the same results as in the paper.
There is only one problem: per-channel weight quantization is not compatible with the CUDA kernels because you get multiple weight scales which is not compatible to be used for the alpha value.
Specifically, the alpha value needs to be just one number to be compatible with the current kernels that are available. I also suspect that if you use per-channel and make it work, it will decrease performance.

I have conducted more experiments that achieve the same results as in the paper.
There is only one problem: per-channel weight quantization is not compatible with the CUDA kernels because you get multiple weight scales which is not compatible to be used for the alpha value.
Specifically, the alpha value needs to be just one number to be compatible with the current kernels that are available. I also suspect that if you use per-channel and make it work, it will decrease performance.

Yes, but it could use tensor-core if you rewrite cuda kernel. and this will half the cost of time.
here aq and bq is quantized int8 value,  s and b is per-chanel scale.
A = [                = [
    a11, a12               aq11 * s1,  aq12 * s1
    a21, a22               aq21 * s2,  aq22 * s2
]                          ]
B = [                = [
    b11, b12             bq11 * b1,  bq12 * b2
    b21, b22             bq21 * b1,  bq22 * b2
]                          ]

A @ B = [
       aq11 * s1 * bq11 * b1 + aq12 * s1 * bq21 * b1,     aq11 * s1 * bq12 * b2 + aq12 * s1 * bq22 * b2
       aq21 * s2 * bq11 * b1 + aq22 * s2 * bq21 * b1,     aq21 * s2 * bq12 * b2 + aq22 * s2 * bq22 * b2
]
         = [                                                                                [
       aq11 * bq11 + aq12 * bq21,   aq11 * bq12 + aq12 * bq22      *            s1 * b1, s1 * b2
       aq21 * bq11 + aq22 * bq21 ,  aq21 * bq12 + aq22 * bq22                    s2 * b1, s2 * b2
           ]                                                                                ]

@AniZpZ What does partial means? Does it means partial quantize Linear Layer in Model?

@AniZpZ What does partial means? Does it means partial quantize Linear Layer in Model?

In our test case, it means not quantize down proj and o proj

@AniZpZ Could you tell me whether is the branch kv-quant-merge installed by pip install -e ., I try to install this branch with this command, but it keeps showing Installing build dependencies ... for along time

raise ValueError(f"Cannot find the config file for {quantization}")
ValueError: Cannot find the config file for smoothquant

raise ValueError(f"Cannot find the config file for {quantization}")
ValueError: Cannot find the config file for smoothquant

It's the requested way to use quantize in Vllm. You should add a 'quant_config.json' file in your model root folder and it looks like this:
{
"w_bit": 8,
"q_type": "tensor"
}

Thank you for your reply, I am trying to use this branch based on mmlu_eval. The problem currently encountered is
INFO:     <IP_ADDRESS>:55674 - "POST /generate HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/home/ft/vllm/vllm/vllm/engine/async_llm_engine.py", line 28, in _raise_exception_on_finish
    task.result()
  File "/home/ft/vllm/vllm/vllm/engine/async_llm_engine.py", line 351, in run_engine_loop
    has_requests_in_progress = await self.engine_step()
  File "/home/ft/vllm/vllm/vllm/engine/async_llm_engine.py", line 330, in engine_step
    request_outputs = await self.engine.step_async()
  File "/home/ft/vllm/vllm/vllm/engine/async_llm_engine.py", line 191, in step_async
    output = await self._run_workers_async(
  File "/home/ft/vllm/vllm/vllm/engine/async_llm_engine.py", line 216, in _run_workers_async
    output = executor(*args, **kwargs)
  File "/root/miniconda3/envs/vllm/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/ft/vllm/vllm/vllm/worker/worker.py", line 293, in execute_model
    output = self.model(
  File "/root/miniconda3/envs/vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/ft/vllm/vllm/vllm/model_executor/models/llama.py", line 366, in forward
    next_tokens = self.sampler(self.lm_head.weight, hidden_states,
  File "/root/miniconda3/envs/vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/ft/vllm/vllm/vllm/model_executor/layers/sampler.py", line 90, in forward
    return _sample(probs, logprobs, input_metadata)
  File "/home/ft/vllm/vllm/vllm/model_executor/layers/sampler.py", line 424, in _sample
    SequenceOutputs(parent_seq_id, next_token_id,
TypeError: SequenceOutputs.__init__() missing 1 required positional argument: 'probs'

The above exception was the direct cause of the following exception:

raise ValueError(f"Cannot find the config file for {quantization}")
ValueError: Cannot find the config file for smoothquant

It's the requested way to use quantize in Vllm. You should add a 'quant_config.json' file in your model root folder and it looks like this:
{
"w_bit": 8,
"q_type": "tensor"
}

Thank you for your reply, I am trying to use this branch based on mmlu_eval. The problem currently encountered is
INFO:     <IP_ADDRESS>:55674 - "POST /generate HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/home/ft/vllm/vllm/vllm/engine/async_llm_engine.py", line 28, in _raise_exception_on_finish
    task.result()
  File "/home/ft/vllm/vllm/vllm/engine/async_llm_engine.py", line 351, in run_engine_loop
    has_requests_in_progress = await self.engine_step()
  File "/home/ft/vllm/vllm/vllm/engine/async_llm_engine.py", line 330, in engine_step
    request_outputs = await self.engine.step_async()
  File "/home/ft/vllm/vllm/vllm/engine/async_llm_engine.py", line 191, in step_async
    output = await self._run_workers_async(
  File "/home/ft/vllm/vllm/vllm/engine/async_llm_engine.py", line 216, in _run_workers_async
    output = executor(*args, **kwargs)
  File "/root/miniconda3/envs/vllm/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/ft/vllm/vllm/vllm/worker/worker.py", line 293, in execute_model
    output = self.model(
  File "/root/miniconda3/envs/vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/ft/vllm/vllm/vllm/model_executor/models/llama.py", line 366, in forward
    next_tokens = self.sampler(self.lm_head.weight, hidden_states,
  File "/root/miniconda3/envs/vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/ft/vllm/vllm/vllm/model_executor/layers/sampler.py", line 90, in forward
    return _sample(probs, logprobs, input_metadata)
  File "/home/ft/vllm/vllm/vllm/model_executor/layers/sampler.py", line 424, in _sample
    SequenceOutputs(parent_seq_id, next_token_id,
TypeError: SequenceOutputs.__init__() missing 1 required positional argument: 'probs'

The above exception was the direct cause of the following exception:

mmlu_eval branch is modified only for  mmlu like dataset eval, it can not used for chat or other tasks because we change the sampler. You can use kv-quant-merge branch for normal tasks.

raise ValueError(f"Cannot find the config file for {quantization}")
ValueError: Cannot find the config file for smoothquant

It's the requested way to use quantize in Vllm. You should add a 'quant_config.json' file in your model root folder and it looks like this:
{
"w_bit": 8,
"q_type": "tensor"
}

Thank you for your reply, I am trying to use this branch based on mmlu_eval. The problem currently encountered is
INFO:     <IP_ADDRESS>:55674 - "POST /generate HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/home/ft/vllm/vllm/vllm/engine/async_llm_engine.py", line 28, in _raise_exception_on_finish
    task.result()
  File "/home/ft/vllm/vllm/vllm/engine/async_llm_engine.py", line 351, in run_engine_loop
    has_requests_in_progress = await self.engine_step()
  File "/home/ft/vllm/vllm/vllm/engine/async_llm_engine.py", line 330, in engine_step
    request_outputs = await self.engine.step_async()
  File "/home/ft/vllm/vllm/vllm/engine/async_llm_engine.py", line 191, in step_async
    output = await self._run_workers_async(
  File "/home/ft/vllm/vllm/vllm/engine/async_llm_engine.py", line 216, in _run_workers_async
    output = executor(*args, **kwargs)
  File "/root/miniconda3/envs/vllm/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/ft/vllm/vllm/vllm/worker/worker.py", line 293, in execute_model
    output = self.model(
  File "/root/miniconda3/envs/vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/ft/vllm/vllm/vllm/model_executor/models/llama.py", line 366, in forward
    next_tokens = self.sampler(self.lm_head.weight, hidden_states,
  File "/root/miniconda3/envs/vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/ft/vllm/vllm/vllm/model_executor/layers/sampler.py", line 90, in forward
    return _sample(probs, logprobs, input_metadata)
  File "/home/ft/vllm/vllm/vllm/model_executor/layers/sampler.py", line 424, in _sample
    SequenceOutputs(parent_seq_id, next_token_id,
TypeError: SequenceOutputs.__init__() missing 1 required positional argument: 'probs'

The above exception was the direct cause of the following exception:

mmlu_eval branch is modified only for mmlu like dataset eval, it can not used for chat or other tasks because we change the sampler. You can use kv-quant-merge branch for normal tasks. However， if you want to eval mmlu dataset, you can use following command:
python benchmarks/benchmark_evaluation.py --model=/path/to/model --tokenizer=/path/to/tokenizer --dev-data-path=/path/to/mmlu/dev --test-data-path=/path/to/mmlu/test --output-len=1 --origin --quantization=smoothquant

Thank you, After I reinstalled based on the kv-quant-merge branch, the following problem occurred.
Traceback (most recent call last):
  File "/root/miniconda3/envs/vllm/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/root/miniconda3/envs/vllm/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/ft/vllm/api_server.py", line 156, in <module>
    engine = AsyncLLMEngine.from_engine_args(engine_args)
  File "/home/ft/vllm/vllm/vllm/engine/async_llm_engine.py", line 486, in from_engine_args
    engine = cls(engine_args.worker_use_ray,
  File "/home/ft/vllm/vllm/vllm/engine/async_llm_engine.py", line 270, in __init__
    self.engine = self._init_engine(*args, **kwargs)
  File "/home/ft/vllm/vllm/vllm/engine/async_llm_engine.py", line 306, in _init_engine
    return engine_class(*args, **kwargs)
  File "/home/ft/vllm/vllm/vllm/engine/llm_engine.py", line 107, in __init__
    self._init_workers(distributed_init_method)
  File "/home/ft/vllm/vllm/vllm/engine/llm_engine.py", line 125, in _init_workers
    from vllm.worker.worker import Worker  # pylint: disable=import-outside-toplevel
  File "/home/ft/vllm/vllm/vllm/worker/worker.py", line 10, in <module>
    from vllm.model_executor import get_model, InputMetadata, set_random_seed
  File "/home/ft/vllm/vllm/vllm/model_executor/__init__.py", line 2, in <module>
    from vllm.model_executor.model_loader import get_model
  File "/home/ft/vllm/vllm/vllm/model_executor/model_loader.py", line 11, in <module>
    from vllm.model_executor.models import *  # pylint: disable=wildcard-import
  File "/home/ft/vllm/vllm/vllm/model_executor/models/__init__.py", line 11, in <module>
    from vllm.model_executor.models.llama import LlamaForCausalLM
  File "/home/ft/vllm/vllm/vllm/model_executor/models/llama.py", line 40, in <module>
    from vllm.model_executor.layers.fusion import DequantAddResidual
ModuleNotFoundError: No module named 'vllm.model_executor.layers.fusion'

raise ValueError(f"Cannot find the config file for {quantization}")
ValueError: Cannot find the config file for smoothquant

It's the requested way to use quantize in Vllm. You should add a 'quant_config.json' file in your model root folder and it looks like this:
{
"w_bit": 8,
"q_type": "tensor"
}

Thank you for your reply, I am trying to use this branch based on mmlu_eval. The problem currently encountered is
INFO:     <IP_ADDRESS>:55674 - "POST /generate HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/home/ft/vllm/vllm/vllm/engine/async_llm_engine.py", line 28, in _raise_exception_on_finish
    task.result()
  File "/home/ft/vllm/vllm/vllm/engine/async_llm_engine.py", line 351, in run_engine_loop
    has_requests_in_progress = await self.engine_step()
  File "/home/ft/vllm/vllm/vllm/engine/async_llm_engine.py", line 330, in engine_step
    request_outputs = await self.engine.step_async()
  File "/home/ft/vllm/vllm/vllm/engine/async_llm_engine.py", line 191, in step_async
    output = await self._run_workers_async(
  File "/home/ft/vllm/vllm/vllm/engine/async_llm_engine.py", line 216, in _run_workers_async
    output = executor(*args, **kwargs)
  File "/root/miniconda3/envs/vllm/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/ft/vllm/vllm/vllm/worker/worker.py", line 293, in execute_model
    output = self.model(
  File "/root/miniconda3/envs/vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/ft/vllm/vllm/vllm/model_executor/models/llama.py", line 366, in forward
    next_tokens = self.sampler(self.lm_head.weight, hidden_states,
  File "/root/miniconda3/envs/vllm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/ft/vllm/vllm/vllm/model_executor/layers/sampler.py", line 90, in forward
    return _sample(probs, logprobs, input_metadata)
  File "/home/ft/vllm/vllm/vllm/model_executor/layers/sampler.py", line 424, in _sample
    SequenceOutputs(parent_seq_id, next_token_id,
TypeError: SequenceOutputs.__init__() missing 1 required positional argument: 'probs'

The above exception was the direct cause of the following exception:

mmlu_eval branch is modified only for mmlu like dataset eval, it can not used for chat or other tasks because we change the sampler. You can use kv-quant-merge branch for normal tasks. However， if you want to eval mmlu dataset, you can use following command:
python benchmarks/benchmark_evaluation.py --model=/path/to/model --tokenizer=/path/to/tokenizer --dev-data-path=/path/to/mmlu/dev --test-data-path=/path/to/mmlu/test --output-len=1 --origin --quantization=smoothquant

Thank you, After I reinstalled based on the kv-quant-merge branch, the following problem occurred.
Traceback (most recent call last):
  File "/root/miniconda3/envs/vllm/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/root/miniconda3/envs/vllm/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/ft/vllm/api_server.py", line 156, in <module>
    engine = AsyncLLMEngine.from_engine_args(engine_args)
  File "/home/ft/vllm/vllm/vllm/engine/async_llm_engine.py", line 486, in from_engine_args
    engine = cls(engine_args.worker_use_ray,
  File "/home/ft/vllm/vllm/vllm/engine/async_llm_engine.py", line 270, in __init__
    self.engine = self._init_engine(*args, **kwargs)
  File "/home/ft/vllm/vllm/vllm/engine/async_llm_engine.py", line 306, in _init_engine
    return engine_class(*args, **kwargs)
  File "/home/ft/vllm/vllm/vllm/engine/llm_engine.py", line 107, in __init__
    self._init_workers(distributed_init_method)
  File "/home/ft/vllm/vllm/vllm/engine/llm_engine.py", line 125, in _init_workers
    from vllm.worker.worker import Worker  # pylint: disable=import-outside-toplevel
  File "/home/ft/vllm/vllm/vllm/worker/worker.py", line 10, in <module>
    from vllm.model_executor import get_model, InputMetadata, set_random_seed
  File "/home/ft/vllm/vllm/vllm/model_executor/__init__.py", line 2, in <module>
    from vllm.model_executor.model_loader import get_model
  File "/home/ft/vllm/vllm/vllm/model_executor/model_loader.py", line 11, in <module>
    from vllm.model_executor.models import *  # pylint: disable=wildcard-import
  File "/home/ft/vllm/vllm/vllm/model_executor/models/__init__.py", line 11, in <module>
    from vllm.model_executor.models.llama import LlamaForCausalLM
  File "/home/ft/vllm/vllm/vllm/model_executor/models/llama.py", line 40, in <module>
    from vllm.model_executor.layers.fusion import DequantAddResidual
ModuleNotFoundError: No module named 'vllm.model_executor.layers.fusion'

Sorry, there was a file missing. I have updated the code and restored the missing file.

I feel there's something wrong with sampling (may not be related to this PR). I'm getting output like
time curl http://localhost:8000/generate -d '{"prompt": "San Francisco is a", "temperature": 0, "max_tokens": 100}'

{"text":["San Francisco is a great place to live for singles, couples, and families alike. There are many different types of neighborhoods here too, ranging from those that are more expensive and those that are more towards the median priced. No matter where you decide...Learn to »\n"]}

output is the same if I increase max_tokens (e.g. to 1000) however generation time increased (e.g. 10x)
works fine if I use some other prompts

using both smoothquant and int8 kv cache on llama 2 70b model, i'm seeing some gibberish output like {"text":["San Francisco is a great place to be.\n\n\n\n\n\n\n\n\n\n\n"]}

\

This might occur when unsuitable scales are used for kv cache quantification.

\

This might occur when unsuitable scales are used for kv cache quantification.

Thanks. I'm getting pretty reasonable output with only kv cache quantization though if I run the model with
python -m vllm.entrypoints.api_server --model=/home/ubuntu/model_weights/llama-2-70b --tokenizer=/home/ubuntu/model_weights/llama-2-7b/ --tensor-parallel-size 2 --kv-cache-dtype=int8 --kv-quant-params-path=kv_params_tp2

@AniZpZ What does partial means? Does it means partial quantize Linear Layer in Model?

In our test case, it means not quantize down proj and o proj

does it mean not quantizing like in these lines? https://github.com/AniZpZ/vllm/blob/kv-quant-merge/vllm/model_executor/models/llama.py#L70-L81

@AniZpZ What does partial means? Does it means partial quantize Linear Layer in Model?

In our test case, it means not quantize down proj and o proj

does it mean not quantizing like in these lines? https://github.com/AniZpZ/vllm/blob/kv-quant-merge/vllm/model_executor/models/llama.py#L70-L81

Not implement quantizing in self.down_proj

install smoothquant and torch-int for llama. Use "examples/generate_act_scales.py" to generate act scale, and then use "examples/export_int8_llama.py" to export int8 model. Please note to check and change the 'architectures' field in the model's config.json from 'Int8LlamaForCausalLM' to 'LlamaForCausalLM'.
When I export int 8 model, it report errors.
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Traceback (most recent call last):
File "examples/export_int8_llama.py", line 31, in
smooth_lm(model, act_scales, 0.5)
File "/opt/conda/envs/int/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/opt/conda/envs/int/lib/python3.8/site-packages/smoothquant-0.0.0-py3.8.egg/smoothquant/smooth.py", line 64, in smooth_lm
KeyError: 'model.decoder.layers.0.self_attn.q_proj'
Would you help to check this?

smoothquant

Did you want to export int8 llama model? If yes, there is a bug in examples/export_int8_llama.py. Please put in this from transformers.models.llama.modeling_llama import LlamaForCausalLM and change line 28、29 from model = OPTForCausalLM.from_pretrained(         args.model_name, device_map="auto", torch_dtype=torch.float16) to model = LlamaForCausalLM.from_pretrained(         args.model_name, device_map="auto", torch_dtype=torch.float16) . We will fix it soon.

install smoothquant and torch-int for llama. Use "examples/generate_act_scales.py" to generate act scale, and then use "examples/export_int8_llama.py" to export int8 model. Please note to check and change the 'architectures' field in the model's config.json from 'Int8LlamaForCausalLM' to 'LlamaForCausalLM'. When I export int 8 model, it report errors. You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference. Traceback (most recent call last): File "examples/export_int8_llama.py", line 31, in smooth_lm(model, act_scales, 0.5) File "/opt/conda/envs/int/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context return func(*args, **kwargs) File "/opt/conda/envs/int/lib/python3.8/site-packages/smoothquant-0.0.0-py3.8.egg/smoothquant/smooth.py", line 64, in smooth_lm KeyError: 'model.decoder.layers.0.self_attn.q_proj'
Would you help to check this?

Did you want to export int8 llama model? If yes, there is a bug in examples/export_int8_llama.py. Please put in this from transformers.models.llama.modeling_llama import LlamaForCausalLM and change line 28、29 from model = OPTForCausalLM.from_pretrained(         args.model_name, device_map="auto", torch_dtype=torch.float16) to model = LlamaForCausalLM.from_pretrained(         args.model_name, device_map="auto", torch_dtype=torch.float16) . We will fix it soon.

So when I run vllm server with my exported int 8 model, it report errors like this? So do you know how to solve this?
My command is like this: /bin/bash vllm_service.sh --port 10038 --model /model/quantized/xxx_ck6800-smoothquant.pt --dtype float16 --tensor-parallel-size 2 --trust-remote-code --quantization awq
ray.exceptions.RayTaskError(ValueError): ray::RayWorker.execute_method() (pid=40824, ip=<IP_ADDRESS>, actor_id=f9190db5d8656170094d0bc401000000, repr=<vllm.engine.ray_utils.RayWorker object at 0x7fa6ca5e5dc0>)
File "/usr/local/lib/python3.8/dist-packages/vllm/engine/ray_utils.py", line 32, in execute_method
return executor(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/vllm/worker/worker.py", line 70, in init_model
self.model = get_model(self.model_config)
File "/usr/local/lib/python3.8/dist-packages/vllm/model_executor/model_loader.py", line 71, in get_model
quant_config = get_quant_config(model_config.quantization,
File "/usr/local/lib/python3.8/dist-packages/vllm/model_executor/weight_utils.py", line 107, in get_quant_config
raise ValueError(f"Cannot find the config file for {quantization}")
ValueError: Cannot find the config file for awq

So when I run vllm server with my exported int 8 model, it report errors like this? So do you know how to solve this? My command is like this: /bin/bash vllm_service.sh --port 10038 --model /model/quantized/xxx_ck6800-smoothquant.pt --dtype float16 --tensor-parallel-size 2 --trust-remote-code --quantization awq @HandH1998
ray.exceptions.RayTaskError(ValueError): ray::RayWorker.execute_method() (pid=40824, ip=<IP_ADDRESS>, actor_id=f9190db5d8656170094d0bc401000000, repr=<vllm.engine.ray_utils.RayWorker object at 0x7fa6ca5e5dc0>) File "/usr/local/lib/python3.8/dist-packages/vllm/engine/ray_utils.py", line 32, in execute_method return executor(*args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/vllm/worker/worker.py", line 70, in init_model self.model = get_model(self.model_config) File "/usr/local/lib/python3.8/dist-packages/vllm/model_executor/model_loader.py", line 71, in get_model quant_config = get_quant_config(model_config.quantization, File "/usr/local/lib/python3.8/dist-packages/vllm/model_executor/weight_utils.py", line 107, in get_quant_config raise ValueError(f"Cannot find the config file for {quantization}") ValueError: Cannot find the config file for awq
Thanks in advance!

Hello! You are using awq quant. So you should prepare a config file in your model path.

For details, you should refer to vllm/model_executor/quantization_utils/awq.py

This time with smoothquant, and the command is like this. Errors report like this,
python ./entrypoints/api_server.py --model=/model/smoothquant/quantized/xxx --tokenizer=/model/smoothquant/quantizedxxx   --max-num-batched-tokens=7000 --block-size=16 --swap-space=20 --quantization smoothquant
INFO 11-21 08:37:44 llm_engine.py:72] Initializing an LLM engine with config: model='/model/smoothquant/quantized/xxx', tokenizer='/model/smoothquant/quantized/xxx', tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=2600, download_dir=None, load_format=auto, tensor_parallel_size=1, quantization=smoothquant, seed=0)
Traceback (most recent call last):
File "./entrypoints/api_server.py", line 80, in
engine = AsyncLLMEngine.from_engine_args(engine_args)
File "/usr/local/lib/python3.8/dist-packages/vllm-0.2.1-py3.8-linux-x86_64.egg/vllm/engine/async_llm_engine.py", line 487, in from_engine_args
engine = cls(parallel_config.worker_use_ray,
File "/usr/local/lib/python3.8/dist-packages/vllm-0.2.1-py3.8-linux-x86_64.egg/vllm/engine/async_llm_engine.py", line 270, in init
self.engine = self._init_engine(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/vllm-0.2.1-py3.8-linux-x86_64.egg/vllm/engine/async_llm_engine.py", line 306, in _init_engine
return engine_class(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/vllm-0.2.1-py3.8-linux-x86_64.egg/vllm/engine/llm_engine.py", line 110, in init
self._init_workers(distributed_init_method)
File "/usr/local/lib/python3.8/dist-packages/vllm-0.2.1-py3.8-linux-x86_64.egg/vllm/engine/llm_engine.py", line 128, in _init_workers
from vllm.worker.worker import Worker  # pylint: disable=import-outside-toplevel
File "/usr/local/lib/python3.8/dist-packages/vllm-0.2.1-py3.8-linux-x86_64.egg/vllm/worker/worker.py", line 10, in
from vllm.model_executor import get_model, InputMetadata, set_random_seed
File "/usr/local/lib/python3.8/dist-packages/vllm-0.2.1-py3.8-linux-x86_64.egg/vllm/model_executor/init.py", line 2, in
from vllm.model_executor.model_loader import get_model
File "/usr/local/lib/python3.8/dist-packages/vllm-0.2.1-py3.8-linux-x86_64.egg/vllm/model_executor/model_loader.py", line 10, in
from vllm.model_executor.models import *  # pylint: disable=wildcard-import
File "/usr/local/lib/python3.8/dist-packages/vllm-0.2.1-py3.8-linux-x86_64.egg/vllm/model_executor/models/init.py", line 11, in
from vllm.model_executor.models.llama import LlamaForCausalLM
File "/usr/local/lib/python3.8/dist-packages/vllm-0.2.1-py3.8-linux-x86_64.egg/vllm/model_executor/models/llama.py", line 39, in
from vllm.model_executor.layers.quantized_linear import ParallelLinear
File "/usr/local/lib/python3.8/dist-packages/vllm-0.2.1-py3.8-linux-x86_64.egg/vllm/model_executor/layers/quantized_linear/init.py", line 5, in
from vllm.model_executor.layers.quantized_linear.smoothquant import (
File "/usr/local/lib/python3.8/dist-packages/vllm-0.2.1-py3.8-linux-x86_64.egg/vllm/model_executor/layers/quantized_linear/smoothquant.py", line 4, in
from vllm.model_executor.parallel_utils.tensor_parallel.layers import (
ModuleNotFoundError: No module named 'vllm.model_executor.parallel_utils.tensor_parallel'

This time with smoothquant, and the command is like this. Errors report like this, python ./entrypoints/api_server.py --model=/model/smoothquant/quantized/xxx --tokenizer=/model/smoothquant/quantizedxxx --max-num-batched-tokens=7000 --block-size=16 --swap-space=20 --quantization smoothquant INFO 11-21 08:37:44 llm_engine.py:72] Initializing an LLM engine with config: model='/model/smoothquant/quantized/xxx', tokenizer='/model/smoothquant/quantized/xxx', tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=2600, download_dir=None, load_format=auto, tensor_parallel_size=1, quantization=smoothquant, seed=0) Traceback (most recent call last): File "./entrypoints/api_server.py", line 80, in engine = AsyncLLMEngine.from_engine_args(engine_args) File "/usr/local/lib/python3.8/dist-packages/vllm-0.2.1-py3.8-linux-x86_64.egg/vllm/engine/async_llm_engine.py", line 487, in from_engine_args engine = cls(parallel_config.worker_use_ray, File "/usr/local/lib/python3.8/dist-packages/vllm-0.2.1-py3.8-linux-x86_64.egg/vllm/engine/async_llm_engine.py", line 270, in init self.engine = self._init_engine(*args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/vllm-0.2.1-py3.8-linux-x86_64.egg/vllm/engine/async_llm_engine.py", line 306, in _init_engine return engine_class(*args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/vllm-0.2.1-py3.8-linux-x86_64.egg/vllm/engine/llm_engine.py", line 110, in init self._init_workers(distributed_init_method) File "/usr/local/lib/python3.8/dist-packages/vllm-0.2.1-py3.8-linux-x86_64.egg/vllm/engine/llm_engine.py", line 128, in _init_workers from vllm.worker.worker import Worker # pylint: disable=import-outside-toplevel File "/usr/local/lib/python3.8/dist-packages/vllm-0.2.1-py3.8-linux-x86_64.egg/vllm/worker/worker.py", line 10, in from vllm.model_executor import get_model, InputMetadata, set_random_seed File "/usr/local/lib/python3.8/dist-packages/vllm-0.2.1-py3.8-linux-x86_64.egg/vllm/model_executor/init.py", line 2, in from vllm.model_executor.model_loader import get_model File "/usr/local/lib/python3.8/dist-packages/vllm-0.2.1-py3.8-linux-x86_64.egg/vllm/model_executor/model_loader.py", line 10, in from vllm.model_executor.models import * # pylint: disable=wildcard-import File "/usr/local/lib/python3.8/dist-packages/vllm-0.2.1-py3.8-linux-x86_64.egg/vllm/model_executor/models/init.py", line 11, in from vllm.model_executor.models.llama import LlamaForCausalLM File "/usr/local/lib/python3.8/dist-packages/vllm-0.2.1-py3.8-linux-x86_64.egg/vllm/model_executor/models/llama.py", line 39, in from vllm.model_executor.layers.quantized_linear import ParallelLinear File "/usr/local/lib/python3.8/dist-packages/vllm-0.2.1-py3.8-linux-x86_64.egg/vllm/model_executor/layers/quantized_linear/init.py", line 5, in from vllm.model_executor.layers.quantized_linear.smoothquant import ( File "/usr/local/lib/python3.8/dist-packages/vllm-0.2.1-py3.8-linux-x86_64.egg/vllm/model_executor/layers/quantized_linear/smoothquant.py", line 4, in from vllm.model_executor.parallel_utils.tensor_parallel.layers import ( ModuleNotFoundError: No module named 'vllm.model_executor.parallel_utils.tensor_parallel'
Would you help to check this?

Hello! This  is a merge problem. We have fixed it in our new per-token quant code which will be released tonight. Please try the new code then. Thanks!

This time with smoothquant, and the command is like this. Errors report like this, python ./entrypoints/api_server.py --model=/model/smoothquant/quantized/xxx --tokenizer=/model/smoothquant/quantizedxxx --max-num-batched-tokens=7000 --block-size=16 --swap-space=20 --quantization smoothquant INFO 11-21 08:37:44 llm_engine.py:72] Initializing an LLM engine with config: model='/model/smoothquant/quantized/xxx', tokenizer='/model/smoothquant/quantized/xxx', tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=2600, download_dir=None, load_format=auto, tensor_parallel_size=1, quantization=smoothquant, seed=0) Traceback (most recent call last): File "./entrypoints/api_server.py", line 80, in engine = AsyncLLMEngine.from_engine_args(engine_args) File "/usr/local/lib/python3.8/dist-packages/vllm-0.2.1-py3.8-linux-x86_64.egg/vllm/engine/async_llm_engine.py", line 487, in from_engine_args engine = cls(parallel_config.worker_use_ray, File "/usr/local/lib/python3.8/dist-packages/vllm-0.2.1-py3.8-linux-x86_64.egg/vllm/engine/async_llm_engine.py", line 270, in init self.engine = self._init_engine(*args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/vllm-0.2.1-py3.8-linux-x86_64.egg/vllm/engine/async_llm_engine.py", line 306, in _init_engine return engine_class(*args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/vllm-0.2.1-py3.8-linux-x86_64.egg/vllm/engine/llm_engine.py", line 110, in init self._init_workers(distributed_init_method) File "/usr/local/lib/python3.8/dist-packages/vllm-0.2.1-py3.8-linux-x86_64.egg/vllm/engine/llm_engine.py", line 128, in _init_workers from vllm.worker.worker import Worker # pylint: disable=import-outside-toplevel File "/usr/local/lib/python3.8/dist-packages/vllm-0.2.1-py3.8-linux-x86_64.egg/vllm/worker/worker.py", line 10, in from vllm.model_executor import get_model, InputMetadata, set_random_seed File "/usr/local/lib/python3.8/dist-packages/vllm-0.2.1-py3.8-linux-x86_64.egg/vllm/model_executor/init.py", line 2, in from vllm.model_executor.model_loader import get_model File "/usr/local/lib/python3.8/dist-packages/vllm-0.2.1-py3.8-linux-x86_64.egg/vllm/model_executor/model_loader.py", line 10, in from vllm.model_executor.models import * # pylint: disable=wildcard-import File "/usr/local/lib/python3.8/dist-packages/vllm-0.2.1-py3.8-linux-x86_64.egg/vllm/model_executor/models/init.py", line 11, in from vllm.model_executor.models.llama import LlamaForCausalLM File "/usr/local/lib/python3.8/dist-packages/vllm-0.2.1-py3.8-linux-x86_64.egg/vllm/model_executor/models/llama.py", line 39, in from vllm.model_executor.layers.quantized_linear import ParallelLinear File "/usr/local/lib/python3.8/dist-packages/vllm-0.2.1-py3.8-linux-x86_64.egg/vllm/model_executor/layers/quantized_linear/init.py", line 5, in from vllm.model_executor.layers.quantized_linear.smoothquant import ( File "/usr/local/lib/python3.8/dist-packages/vllm-0.2.1-py3.8-linux-x86_64.egg/vllm/model_executor/layers/quantized_linear/smoothquant.py", line 4, in from vllm.model_executor.parallel_utils.tensor_parallel.layers import ( ModuleNotFoundError: No module named 'vllm.model_executor.parallel_utils.tensor_parallel'
Would you help to check this?

Hello! This is a merge problem. We have fixed it in our new per-token quant code which will be released tonight. Please try the new code then. Thanks!

We have released w8a8 per-token quant (without kv quant) code. Please refer to https://github.com/vllm-project/vllm/pull/1507

@HandH1998
when compiling the vllm from this branch kv_quant, another issue:
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/2] c++ -MMD -MF /app/vllm/build/temp.linux-x86_64-3.8/csrc/cache.o.d -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/local/lib/python3.8/dist-packages/torch/include -I/usr/local/lib/python3.8/dist-packages/torch/include/torch/csrc/api/include -I/usr/local/lib/python3.8/dist-packages/torch/include/TH -I/usr/local/lib/python3.8/dist-packages/torch/include/THC -I/usr/local/cuda/include -I/usr/include/python3.8 -c -c /app/vllm/csrc/cache.cpp -o /app/vllm/build/temp.linux-x86_64-3.8/csrc/cache.o -g -O2 -std=c++17 -D_GLIBCXX_USE_CXX11_ABI=0 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="gcc"' '-DPYBIND11_STDLIB="libstdcpp"' '-DPYBIND11_BUILD_ABI="cxxabi1011"' -DTORCH_EXTENSION_NAME=cache_ops -D_GLIBCXX_USE_CXX11_ABI=0
[2/2] /usr/local/cuda/bin/nvcc  -I/usr/local/lib/python3.8/dist-packages/torch/include -I/usr/local/lib/python3.8/dist-packages/torch/include/torch/csrc/api/include -I/usr/local/lib/python3.8/dist-packages/torch/include/TH -I/usr/local/lib/python3.8/dist-packages/torch/include/THC -I/usr/local/cuda/include -I/usr/include/python3.8 -c -c /app/vllm/csrc/cache_kernels.cu -o /app/vllm/build/temp.linux-x86_64-3.8/csrc/cache_kernels.o -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -O2 -std=c++17 -D_GLIBCXX_USE_CXX11_ABI=0 -gencode arch=compute_86,code=sm_86 -gencode arch=compute_70,code=sm_70 -gencode arch=compute_90,code=sm_90 -gencode arch=compute_90,code=compute_90 -gencode arch=compute_75,code=sm_75 -gencode arch=compute_80,code=sm_80 --threads 8 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="gcc"' '-DPYBIND11_STDLIB="libstdcpp"' '-DPYBIND11_BUILD_ABI="cxxabi1011"' -DTORCH_EXTENSION_NAME=cache_ops -D_GLIBCXX_USE_CXX11_ABI=0
FAILED: /app/vllm/build/temp.linux-x86_64-3.8/csrc/cache_kernels.o
/usr/local/cuda/bin/nvcc  -I/usr/local/lib/python3.8/dist-packages/torch/include -I/usr/local/lib/python3.8/dist-packages/torch/include/torch/csrc/api/include -I/usr/local/lib/python3.8/dist-packages/torch/include/TH -I/usr/local/lib/python3.8/dist-packages/torch/include/THC -I/usr/local/cuda/include -I/usr/include/python3.8 -c -c /app/vllm/csrc/cache_kernels.cu -o /app/vllm/build/temp.linux-x86_64-3.8/csrc/cache_kernels.o -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -O2 -std=c++17 -D_GLIBCXX_USE_CXX11_ABI=0 -gencode arch=compute_86,code=sm_86 -gencode arch=compute_70,code=sm_70 -gencode arch=compute_90,code=sm_90 -gencode arch=compute_90,code=compute_90 -gencode arch=compute_75,code=sm_75 -gencode arch=compute_80,code=sm_80 --threads 8 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=cache_ops -D_GLIBCXX_USE_CXX11_ABI=0
/app/vllm/csrc/cache_kernels.cu(44): warning #1143-D: arithmetic on pointer to void or function type
/app/vllm/csrc/cache_kernels.cu(45): warning #1143-D: arithmetic on pointer to void or function type
/usr/local/lib/python3.8/dist-packages/torch/include/c10/util/irange.h(54): warning #186-D: pointless comparison of unsigned integer with zero
detected during:
instantiation of "__nv_bool c10::detail::integer_iterator<I, one_sided, >::operator==(const c10::detail::integer_iterator<I, one_sided, > &) const [with I=size_t, one_sided=false, =0]"
(61): here
instantiation of "__nv_bool c10::detail::integer_iterator<I, one_sided, >::operator!=(const c10::detail::integer_iterator<I, one_sided, > &) const [with I=size_t, one_sided=false, =0]"
/usr/local/lib/python3.8/dist-packages/torch/include/c10/core/TensorImpl.h(77): here
/usr/local/lib/python3.8/dist-packages/torch/include/c10/util/irange.h(54): warning #186-D: pointless comparison of unsigned integer with zero
detected during:
instantiation of "__nv_bool c10::detail::integer_iterator<I, one_sided, >::operator==(const c10::detail::integer_iterator<I, one_sided, > &) const [with I=std::size_t, one_sided=true, =0]"
(61): here
instantiation of "__nv_bool c10::detail::integer_iterator<I, one_sided, >::operator!=(const c10::detail::integer_iterator<I, one_sided, > &) const [with I=std::size_t, one_sided=true, =0]"
/usr/local/lib/python3.8/dist-packages/torch/include/ATen/core/qualified_name.h(73): here
/app/vllm/csrc/cache_kernels.cu(399): warning #550-D: variable "src_key_indices" was set but never used
/app/vllm/csrc/cache_kernels.cu(400): warning #550-D: variable "src_value_indices" was set but never used
/app/vllm/csrc/quant_utils.cuh(217): error: identifier "__float22bfloat162_rn" is undefined
/app/vllm/csrc/cache_kernels.cu(44): warning #1143-D: arithmetic on pointer to void or function type
/app/vllm/csrc/cache_kernels.cu(45): warning #1143-D: arithmetic on pointer to void or function type
/usr/local/lib/python3.8/dist-packages/torch/include/c10/util/irange.h(54): warning #186-D: pointless comparison of unsigned integer with zero
detected during:
instantiation of "__nv_bool c10::detail::integer_iterator<I, one_sided, >::operator==(const c10::detail::integer_iterator<I, one_sided, > &) const [with I=size_t, one_sided=false, =0]"
(61): here
instantiation of "__nv_bool c10::detail::integer_iterator<I, one_sided, >::operator!=(const c10::detail::integer_iterator<I, one_sided, > &) const [with I=size_t, one_sided=false, =0]"
/usr/local/lib/python3.8/dist-packages/torch/include/c10/core/TensorImpl.h(77): here
/usr/local/lib/python3.8/dist-packages/torch/include/c10/util/irange.h(54): warning #186-D: pointless comparison of unsigned integer with zero
detected during:
instantiation of "__nv_bool c10::detail::integer_iterator<I, one_sided, >::operator==(const c10::detail::integer_iterator<I, one_sided, > &) const [with I=std::size_t, one_sided=true, =0]"
(61): here
instantiation of "__nv_bool c10::detail::integer_iterator<I, one_sided, >::operator!=(const c10::detail::integer_iterator<I, one_sided, > &) const [with I=std::size_t, one_sided=true, =0]"
/usr/local/lib/python3.8/dist-packages/torch/include/ATen/core/qualified_name.h(73): here
/app/vllm/csrc/cache_kernels.cu(399): warning #550-D: variable "src_key_indices" was set but never used
/app/vllm/csrc/cache_kernels.cu(400): warning #550-D: variable "src_value_indices" was set but never used
1 error detected in the compilation of "/app/vllm/csrc/cache_kernels.cu".
/app/vllm/csrc/quant_utils.cuh(217): error: identifier "__float22bfloat162_rn" is undefined
/app/vllm/csrc/cache_kernels.cu(44): warning #1143-D: arithmetic on pointer to void or function type
/app/vllm/csrc/cache_kernels.cu(45): warning #1143-D: arithmetic on pointer to void or function type
/usr/local/lib/python3.8/dist-packages/torch/include/c10/util/irange.h(54): warning #186-D: pointless comparison of unsigned integer with zero
detected during:
instantiation of "__nv_bool c10::detail::integer_iterator<I, one_sided, >::operator==(const c10::detail::integer_iterator<I, one_sided, > &) const [with I=size_t, one_sided=false, =0]"
(61): here
instantiation of "__nv_bool c10::detail::integer_iterator<I, one_sided, >::operator!=(const c10::detail::integer_iterator<I, one_sided, > &) const [with I=size_t, one_sided=false, =0]"
/usr/local/lib/python3.8/dist-packages/torch/include/c10/core/TensorImpl.h(77): here
/usr/local/lib/python3.8/dist-packages/torch/include/c10/util/irange.h(54): warning #186-D: pointless comparison of unsigned integer with zero
detected during:
instantiation of "__nv_bool c10::detail::integer_iterator<I, one_sided, >::operator==(const c10::detail::integer_iterator<I, one_sided, > &) const [with I=std::size_t, one_sided=true, =0]"
(61): here
instantiation of "__nv_bool c10::detail::integer_iterator<I, one_sided, >::operator!=(const c10::detail::integer_iterator<I, one_sided, > &) const [with I=std::size_t, one_sided=true, =0]"
/usr/local/lib/python3.8/dist-packages/torch/include/ATen/core/qualified_name.h(73): here
/app/vllm/csrc/cache_kernels.cu(399): warning #550-D: variable "src_key_indices" was set but never used
/app/vllm/csrc/cache_kernels.cu(400): warning #550-D: variable "src_value_indices" was set but never used
1 error detected in the compilation of "/app/vllm/csrc/cache_kernels.cu".
/app/vllm/csrc/cache_kernels.cu(44): warning #1143-D: arithmetic on pointer to void or function type
/app/vllm/csrc/cache_kernels.cu(45): warning #1143-D: arithmetic on pointer to void or function type
/usr/local/lib/python3.8/dist-packages/torch/include/c10/util/irange.h(54): warning #186-D: pointless comparison of unsigned integer with zero
detected during:
instantiation of "__nv_bool c10::detail::integer_iterator<I, one_sided, >::operator==(const c10::detail::integer_iterator<I, one_sided, > &) const [with I=size_t, one_sided=false, =0]"
(61): here
instantiation of "__nv_bool c10::detail::integer_iterator<I, one_sided, >::operator!=(const c10::detail::integer_iterator<I, one_sided, > &) const [with I=size_t, one_sided=false, =0]"
/usr/local/lib/python3.8/dist-packages/torch/include/c10/core/TensorImpl.h(77): here
/usr/local/lib/python3.8/dist-packages/torch/include/c10/util/irange.h(54): warning #186-D: pointless comparison of unsigned integer with zero
detected during:
instantiation of "__nv_bool c10::detail::integer_iterator<I, one_sided, >::operator==(const c10::detail::integer_iterator<I, one_sided, > &) const [with I=std::size_t, one_sided=true, =0]"
(61): here
instantiation of "__nv_bool c10::detail::integer_iterator<I, one_sided, >::operator!=(const c10::detail::integer_iterator<I, one_sided, > &) const [with I=std::size_t, one_sided=true, =0]"
/usr/local/lib/python3.8/dist-packages/torch/include/ATen/core/qualified_name.h(73): here
/app/vllm/csrc/cache_kernels.cu(399): warning #550-D: variable "src_key_indices" was set but never used
/app/vllm/csrc/cache_kernels.cu(400): warning #550-D: variable "src_value_indices" was set but never used
/app/vllm/csrc/cache_kernels.cu(44): warning #1143-D: arithmetic on pointer to void or function type
/app/vllm/csrc/cache_kernels.cu(45): warning #1143-D: arithmetic on pointer to void or function type
/usr/local/lib/python3.8/dist-packages/torch/include/c10/util/irange.h(54): warning #186-D: pointless comparison of unsigned integer with zero
detected during:
instantiation of "__nv_bool c10::detail::integer_iterator<I, one_sided, >::operator==(const c10::detail::integer_iterator<I, one_sided, > &) const [with I=size_t, one_sided=false, =0]"
(61): here
instantiation of "__nv_bool c10::detail::integer_iterator<I, one_sided, >::operator!=(const c10::detail::integer_iterator<I, one_sided, > &) const [with I=size_t, one_sided=false, =0]"
/usr/local/lib/python3.8/dist-packages/torch/include/c10/core/TensorImpl.h(77): here
/usr/local/lib/python3.8/dist-packages/torch/include/c10/util/irange.h(54): warning #186-D: pointless comparison of unsigned integer with zero
detected during:
instantiation of "__nv_bool c10::detail::integer_iterator<I, one_sided, >::operator==(const c10::detail::integer_iterator<I, one_sided, > &) const [with I=std::size_t, one_sided=true, =0]"
(61): here
instantiation of "__nv_bool c10::detail::integer_iterator<I, one_sided, >::operator!=(const c10::detail::integer_iterator<I, one_sided, > &) const [with I=std::size_t, one_sided=true, =0]"
/usr/local/lib/python3.8/dist-packages/torch/include/ATen/core/qualified_name.h(73): here
/app/vllm/csrc/cache_kernels.cu(399): warning #550-D: variable "src_key_indices" was set but never used
/app/vllm/csrc/cache_kernels.cu(400): warning #550-D: variable "src_value_indices" was set but never used
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/torch/utils/cpp_extension.py", line 1893, in _run_ninja_build
subprocess.run(
File "/usr/lib/python3.8/subprocess.py", line 516, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "setup.py", line 257, in
setuptools.setup(
File "/usr/local/lib/python3.8/dist-packages/setuptools/init.py", line 153, in setup
return distutils.core.setup(**attrs)
File "/usr/lib/python3.8/distutils/core.py", line 148, in setup
dist.run_commands()
File "/usr/lib/python3.8/distutils/dist.py", line 966, in run_commands
self.run_command(cmd)
File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/usr/local/lib/python3.8/dist-packages/setuptools/command/install.py", line 74, in run
self.do_egg_install()
File "/usr/local/lib/python3.8/dist-packages/setuptools/command/install.py", line 116, in do_egg_install
self.run_command('bdist_egg')
File "/usr/lib/python3.8/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/usr/local/lib/python3.8/dist-packages/setuptools/command/bdist_egg.py", line 164, in run
cmd = self.call_command('install_lib', warn_dir=0)
File "/usr/local/lib/python3.8/dist-packages/setuptools/command/bdist_egg.py", line 150, in call_command
self.run_command(cmdname)
File "/usr/lib/python3.8/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/usr/local/lib/python3.8/dist-packages/setuptools/command/install_lib.py", line 11, in run
self.build()
File "/usr/lib/python3.8/distutils/command/install_lib.py", line 109, in build
self.run_command('build_ext')
File "/usr/lib/python3.8/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/usr/local/lib/python3.8/dist-packages/setuptools/command/build_ext.py", line 79, in run
_build_ext.run(self)
File "/usr/local/lib/python3.8/dist-packages/Cython/Distutils/old_build_ext.py", line 186, in run
_build_ext.build_ext.run(self)
File "/usr/lib/python3.8/distutils/command/build_ext.py", line 340, in run
self.build_extensions()
File "/usr/local/lib/python3.8/dist-packages/torch/utils/cpp_extension.py", line 843, in build_extensions
build_ext.build_extensions(self)
File "/usr/local/lib/python3.8/dist-packages/Cython/Distutils/old_build_ext.py", line 195, in build_extensions
_build_ext.build_ext.build_extensions(self)
File "/usr/lib/python3.8/distutils/command/build_ext.py", line 449, in build_extensions
self._build_extensions_serial()
File "/usr/lib/python3.8/distutils/command/build_ext.py", line 474, in _build_extensions_serial
self.build_extension(ext)
File "/usr/local/lib/python3.8/dist-packages/setuptools/command/build_ext.py", line 202, in build_extension
_build_ext.build_extension(self, ext)
File "/usr/lib/python3.8/distutils/command/build_ext.py", line 528, in build_extension
objects = self.compiler.compile(sources,
File "/usr/local/lib/python3.8/dist-packages/torch/utils/cpp_extension.py", line 658, in unix_wrap_ninja_compile
_write_ninja_file_and_compile_objects(
File "/usr/local/lib/python3.8/dist-packages/torch/utils/cpp_extension.py", line 1574, in _write_ninja_file_and_compile_objects
_run_ninja_build(
File "/usr/local/lib/python3.8/dist-packages/torch/utils/cpp_extension.py", line 1909, in _run_ninja_build
raise RuntimeError(message) from e
RuntimeError: Error compiling objects for extension

exactly cuda 11.8 and compiled successfullly on another branch w8a8.
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Sep_21_10:33:58_PDT_2022
Cuda compilation tools, release 11.8, V11.8.89
Build cuda_11.8.r11.8/compiler.31833905_0

@AniZpZ exactly cuda 11.8 and compiled successfullly on another branch w8a8. nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2022 NVIDIA Corporation Built on Wed_Sep_21_10:33:58_PDT_2022 Cuda compilation tools, release 11.8, V11.8.89 Build cuda_11.8.r11.8/compiler.31833905_0

@AniZpZ exactly cuda 11.8 and compiled successfullly on another branch w8a8. nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2022 NVIDIA Corporation Built on Wed_Sep_21_10:33:58_PDT_2022 Cuda compilation tools, release 11.8, V11.8.89 Build cuda_11.8.r11.8/compiler.31833905_0

Could you please check your environment variable TORCH_CUDA_ARCH_LIST? The cuda arch should >= 8.0.

@AniZpZ exactly cuda 11.8 and compiled successfullly on another branch w8a8. nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2022 NVIDIA Corporation Built on Wed_Sep_21_10:33:58_PDT_2022 Cuda compilation tools, release 11.8, V11.8.89 Build cuda_11.8.r11.8/compiler.31833905_0

It looks like a problem occured when GPU arch does not match the requirement. Please check you GPU arch.

TORCH_CUDA_ARCH_LIST

@AniZpZ exactly cuda 11.8 and compiled successfullly on another branch w8a8. nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2022 NVIDIA Corporation Built on Wed_Sep_21_10:33:58_PDT_2022 Cuda compilation tools, release 11.8, V11.8.89 Build cuda_11.8.r11.8/compiler.31833905_0

@AniZpZ exactly cuda 11.8 and compiled successfullly on another branch w8a8. nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2022 NVIDIA Corporation Built on Wed_Sep_21_10:33:58_PDT_2022 Cuda compilation tools, release 11.8, V11.8.89 Build cuda_11.8.r11.8/compiler.31833905_0

Could you please check your environment variable TORCH_CUDA_ARCH_LIST? The cuda arch should >= 8.0.

@AniZpZ @HandH1998
['sm_37', 'sm_50', 'sm_60', 'sm_70', 'sm_75', 'sm_80', 'sm_86']
Here is the cuda arch support list. So is there a probelm? Thanks!

TORCH_CUDA_ARCH_LIST

@AniZpZ exactly cuda 11.8 and compiled successfullly on another branch w8a8. nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2022 NVIDIA Corporation Built on Wed_Sep_21_10:33:58_PDT_2022 Cuda compilation tools, release 11.8, V11.8.89 Build cuda_11.8.r11.8/compiler.31833905_0

@AniZpZ exactly cuda 11.8 and compiled successfullly on another branch w8a8. nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2022 NVIDIA Corporation Built on Wed_Sep_21_10:33:58_PDT_2022 Cuda compilation tools, release 11.8, V11.8.89 Build cuda_11.8.r11.8/compiler.31833905_0

Could you please check your environment variable TORCH_CUDA_ARCH_LIST? The cuda arch should >= 8.0.

@AniZpZ @HandH1998 ['sm_37', 'sm_50', 'sm_60', 'sm_70', 'sm_75', 'sm_80', 'sm_86'] Here is the cuda arch support list. So is there a probelm? Thanks!

export TORCH_CUDA_ARCH_LIST=80;86

We have recently implemented and tested int8 KV-Cache quantization and W8A8 inference in vLLM. We find out that our quantization implementation can increase the throughput over 20% and reduce the first token latency under heavy load. In contrast, W4A16 quant methods(eg. AWQ-based method) provided in vllm cannot improve throughput according to pr1032 because it can not benefit from int8 tensor core. So we propose this PR as an alternative quantization method.
Updates!!! We have made some more progress in #1112 (comment)
More Updates!!! If you want to properly eval mmlu dataset with vllm, some modify on sampler must be done. The code can be found in our mmlu_eval branch
Important message!!! We spilt the pr into two parts for easier review and use. The w8a8 inference part is in #1508 and the kv cache quant part is in #1507
What we have right now:

int8 KV-Cache quantization related works:
a. Quant\Dequant helper functions adapted from Faster Transformer
b. Quantized version CUDA kernels
c. Unit tests for the added kernels
W8A8 inference related works:
a. Int8 Gemm kernels adapted from torch-int
b. W8A8 linear layer modules
c. Support W8A8 inference on Llama model
Test result based on our own dataset

What we plan to do:

[x]  1. Further kernel fusion
[x]  2. Code refactoring and cleaning
[x]  3. Opimize int8 GEMM kernel
[x]  4. Release SmoothQuant for LLaMA
[x]  5. Add code for generating KV-Cache quantization parameters (scales and zero points)
[x]  6. Experiments on more datasets

How to test throughput A. how to enable w8a8 inference ~0. install cutlass because we currently use cutlass gemm kernel. We plan to replace them with cublas gemm kernel soon.~ We support cublas gemm kernel now, you can remove cutlass gemm kernel in setup.py

install smoothquant and torch-int for llama. Use "examples/generate_act_scales.py" to generate act scale, and then use "examples/export_int8_llama.py" to export int8 model. Please note to check and change the 'architectures' field in the model's config.json from 'Int8LlamaForCausalLM' to 'LlamaForCausalLM'.
update vllm and execute

python ./vllm/entrypoints/api_server.py --model=/path/to/quantized/model --tokenizer=/path/to/tokenizer --max-num-batched-tokens=70000 --block-size=16 --swap-space=20 --quantization smoothquant

B. how to enable kv cache quant

use vllm/kv_quant/calibrate.py to genearte scales and use vllm/kv_quant/export_kv_params.py to export kv caches.
exeute

python ./vllm/entrypoints/api_server.py --model=/path/to/quantized/model --tokenizer=/path/to/tokenizer --max-num-batched-tokens=70000 --block-size=16 --swap-space=20 --kv-cache-dtype=int8 --kv-quant-params-path=/path/to/kv_params_dir

And you can use kv cache quant and w8a8 inference together
Experiment Result current test result in our datasets on A100 80G (updates with quant&rms fusion and gemm d2h bug fix)
Throughput of FP16 LLaMA-13B:
Throughput:  4.9945 requests/s, 543.0660 token/s
Average latency: 31.7689 s

Throughput of Int8 LLaMA-13B with int8 KVCacheQuant:
Throughput: 6.1147 requests/s, 664.8646 token/s,
Average latency: 27.4222 s

Throughput of Int8 LLaMA-13B with int8 KVCacheQuant, using cublas gemm kernel:
Throughput: 6.4723 requests/s, 703.7514 token/s,
Average latency: 25.9912 s

How to evalute model performance We add evaluation method of quanted models, currently support mmlu datasets. You can find detail in benchmarks/benchmark_evaluation.py
python benchmark_evaluation.py --model=/path/to/quantized/model --tokenizer=/path/to/tokenizer --dev-data-path=/path/to/mmlu/dev/ --test-data-path=/path/to/mmlu/test/ --kv-cache-dtype=int8 --kv-quant-params-path=/path/to/kv_params_dir --quantization=smoothquant

Updates We have released SmoothQuant for LLaMA in https://github.com/AniZpZ/smoothquant/tree/llama-dev https://github.com/AniZpZ/torch-int/tree/llama-dev
The code for generating KV-Cache quantization parameters is ready, check vllm/kv_quant fold
We replace int8 gemm with cublas version and the increasement of throughput comes to around 30%

Which branch of vllm does kv cache int8 run on? I run it on kv_quant ， but error occurs as follow： ImportError: /home/vllm/vllm-kv_quant/vllm/cuda_utils.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZNSt15__exception_ptr13exception_ptr10_M_releaseEv

@HandH1998
I used the 2 ways  to quant and scored my fined llama 70b model, but not  enable kv cache quant.

model = AutoModelForCausalLM.from_pretrained('xxx',device_map='auto',load_in_8bit=True) , this one got the closed score as the none quanted one.
use the kv-quant-merged branch to compile, the score is bad.
So why this could be so different?

@HandH1998 I used the 2 ways to quant and scored my fined llama 70b model, and not enable kv cache quant.

model = AutoModelForCausalLM.from_pretrained('xxx',device_map='auto',load_in_8bit=True) , this one got the closed score as the none quanted one.
use the kv-quant-merged branch to compile, the score is bad.
So why this could be so different? I used the orginal file "dataset/val.jsonl.zst" as the validation file to export act scales and model, would this be the problem?

method 1 is weight only quantization. Please use our new branch(https://github.com/vllm-project/vllm/pull/1508) to test w8a8 inference.

dequant_scale

This branch mix kv cache quant with w8a8 model quant. Please try kv cache quantization with our new branch(https://github.com/vllm-project/vllm/pull/1507)

dequant_scale

This branch mix kv cache quant with w8a8 model quant. Please try kv cache quantization with our new branch(#1507)

with the #1507， there  is also this issue. I think if not enable kv cache quant, it should not be worse.
I used the 2 ways to quant and scored my fined llama 70b model, and not enable kv cache quant.

model = AutoModelForCausalLM.from_pretrained('xxx',device_map='auto',load_in_8bit=True) , this one got the closed score as the none quanted one.
2.use the kv_quant  branch to compile, the score is bad.
So why this could be so different? I used the orginal file "dataset/val.jsonl.zst" as the validation file to export act scales and model, would this be the problem?

@AniZpZ
When I tried to calibrate, it results an error:

Traceback (most recent call last):
File "/work/vllm_merge/vllm/kv_quant/calibrate.py", line 124, in
fire.Fire(calibrate)
File "/usr/local/lib/python3.10/dist-packages/fire/core.py", line 141, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "/usr/local/lib/python3.10/dist-packages/fire/core.py", line 475, in _Fire
component, remaining_args = _CallAndUpdateTrace(
File "/usr/local/lib/python3.10/dist-packages/fire/core.py", line 691, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "/work/vllm_merge/vllm/kv_quant/calibrate.py", line 115, in calibrate
calib_ctx.calibrate(all_data)
File "/work/vllm_merge/vllm/kv_quant/calibration.py", line 283, in calibrate
_ = model(data.to(self.device))
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/transformers/models/llama/modeling_llama.py", line 1068, in forward
layer_outputs = decoder_layer(
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/work/vllm_merge/vllm/kv_quant/calibration.py", line 167, in _forward
key, value = out.pop(-1)
TypeError: cannot unpack non-iterable NoneType object

My calibration command:
 python3 vllm/kv_quant/calibrate.py /work/llama2-7b-hf c4 128 2048 /work/act-scales
Additional info:
key, value = out.pop(-1)
out is a list [tensor, None]

@warlock135 , hi, you can try transformers==4.34.0. The latest transformers modified the forward code, which make the calibration fail to get key and value.

Any hope for 8 bit quantization to be merged any time soon? 4-bit models doesn't provide great results for complicated cases unfortunately.

Any hope for 8 bit quantization to be merged any time soon? 4-bit models doesn't provide great results for complicated cases unfortunately.

We hope this feature can be merged soon, too.  But it doesn't seem to get a lot of attention of main reviewers. If possible, we   hope you can raise an issue. Thanks.

@warlock135 , hi, you can try transformers==4.34.0. The latest transformers modified the forward code, which makes the calibration fail to get key and value.

Can the latest transformers version be used for quantization currently using the latest AutoSmoothQuant (https://github.com/AniZpZ/AutoSmoothQuant.git)?

@warlock135 , hi, you can try transformers==4.34.0. The latest transformers modified the forward code, which makes the calibration fail to get key and value.

@HandH1998 Can the latest transformers version be used for quantization currently using the latest AutoSmoothQuant (https://github.com/AniZpZ/AutoSmoothQuant.git)?

When I used transformers 4.36.2 to perform quantization in the AutoSmoothQuant project, the following error message appeared:
ImportError: cannot import name 'SiLUActivation' from 'transformers.activations'

@warlock135 , hi, you can try transformers==4.34.0. The latest transformers modified the forward code, which makes the calibration fail to get key and value.

@HandH1998 Can the latest transformers version be used for quantization currently using the latest AutoSmoothQuant (https://github.com/AniZpZ/AutoSmoothQuant.git)?

@HandH1998 When I used transformers 4.36.2 to perform quantization in the AutoSmoothQuant project, the following error message appeared:
ImportError: cannot import name 'SiLUActivation' from 'transformers.activations'

@warlock135 , hi, you can try transformers==4.34.0. The latest transformers modified the forward code, which makes the calibration fail to get key and value.

@HandH1998 Can the latest transformers version be used for quantization currently using the latest AutoSmoothQuant (https://github.com/AniZpZ/AutoSmoothQuant.git)?

@HandH1998 When I used transformers 4.36.2 to perform quantization in the AutoSmoothQuant project, the following error message appeared: ImportError: cannot import name 'SiLUActivation' from 'transformers.activations'

you need to use transformers==4.34.0

@warlock135 , hi, you can try transformers==4.34.0. The latest transformers modified the forward code, which makes the calibration fail to get key and value.

@HandH1998 Can the latest transformers version be used for quantization currently using the latest AutoSmoothQuant (https://github.com/AniZpZ/AutoSmoothQuant.git)?

@HandH1998 When I used transformers 4.36.2 to perform quantization in the AutoSmoothQuant project, the following error message appeared: ImportError: cannot import name 'SiLUActivation' from 'transformers.activations'

you need to use transformers==4.34.0

@HandH1998 After the quantification was completed, I used vllm of the w8a8 branch to deploy, but the following error occurred:
"ValueError: Cannot find any of ['w_bit', 'bits'] in the model's quantization config."
The contents of my quant_config.json are currently:
{
"qkv_proj": "per-tensor",
"o_proj": "per-token",
"gate_up_proj": "per-tensor",
"down_proj": "per-token"
}
How can I modify or add the content in quant_config.json?

what are you think about this kv cache quant technic? https://github.com/jy-yuan/KIVI

what are you think about this kv cache quant technic? https://github.com/jy-yuan/KIVI

what are you think about this kv cache quant technic? https://github.com/jy-yuan/KIVI

Thank you. We will look into the technic.

what are you think about this kv cache quant technic? https://github.com/jy-yuan/KIVI

what are you think about this kv cache quant technic? https://github.com/jy-yuan/KIVI

Thank you. We will look into the technic.

Hi, @AniZpZ , is here any progress on the lower bit quantization?

@AniZpZ Can you provide a docker or environment requirement for both smoothquent and vllm kv-quant-merge branch?

what are you think about this kv cache quant technic? https://github.com/jy-yuan/KIVI

what are you think about this kv cache quant technic? https://github.com/jy-yuan/KIVI

Thank you. We will look into the technic.

Hi, @AniZpZ , is here any progress on the lower bit quantization?

We are working on lower bit quantization.

@AniZpZ Can you provide a docker or environment requirement for both smoothquent and vllm kv-quant-merge branch?

The enviroment requirement should be the similar to original vLLM.

@AniZpZ When I build smoothquant from source, it requires CUDA 11.8. And build kv-quant-merge branch of vllm from source, it requires CUDA 12.0. Is that expected? Can you kindly provide a list of CUDA/torch/transformers/ray dependency for using smoothquant with your vllm branch? Thanks in advance.

Hi, is there any plan to support int4 KVCacheQuant?

Closing as we do have W8A8 and Int8 support nowadays. 🙏

I just found kv_cache fp8 quant, could you provide the related pr that support kv_cache int8 quant?   thanks!


Switch to go modules

Add a common vendor make target as well.
Closes #7

LGTM
@vrothberg PTAL

LGTM, thanks @saschagrunert. (someone else must merge)


Will there be onnx export and inference in the future ?

RT
Just curious

It's easy to use torch.onnx to export the weight file into onnx, you can try it by yourself 😘.


feat(namespace): add namespacing paths

Adds a new command "namespace" to namespace API paths
$ ./kced namespace --help
Namespaces API paths by prefixing it.

By prefixing paths with a specific segment, colliding paths to services can be
namespaced to prevent the colissions. Eg. 2 API definitions that both expose a
'/list' path. By prefixing one with '/addressbook' and the other with '/cookbook'
the resulting paths '/addressbook/list' and '/cookbook/list' can be exposed without
colliding.

'strip-path' settings will be added to the route-config to ensure it is stripped
again from the path before sending it upstream.

NOTE: all paths within a route must match for the route to be updated.

Usage:
  kced namespace [flags]

Flags:
      --format string        output format: json or yaml (default "yaml")
  -h, --help                 help for namespace
      --namespace string     the namespace or updated prefix to apply
  -o, --output-file string   output file to write. Use - to write to stdout. (default "-")
  -p, --prefix string        the existing path-prefix to match. Only matching paths will be namespaced (plain or regex based) (default "/")
  -s, --state string         decK file to process. Use - to read from stdin. (default "-")

$

Codecov Report

:exclamation: No coverage uploaded for pull request base (main@94c4de6). Click here to learn what that means.
Patch has no changes to coverable lines.

Additional details and impacted files
@@           Coverage Diff           @@
##             main      #32   +/-   ##
=======================================
  Coverage        ?   69.44%
=======================================
  Files           ?       12
  Lines           ?     1656
  Branches        ?        0
=======================================
  Hits            ?     1150
  Misses          ?      432
  Partials        ?       74

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

unfortunately not, since it first requires a change to the Gateway.

closing in favor of #129


useCookie state is stored independently

Environment

Operating System: Darwin
Node Version:     v14.17.3
Nuxt Version:     3.0.0
Package Manager<EMAIL_ADDRESS>Bundler:          Vite
User Config:      -
Runtime Modules:  -
Build Modules:    -

Reproduction
<template>

      {{ cookie2 }}

    <button @click="cookie1 = (Math.random() * 100).toFixed()">
      Change cookie
    </button>

</template>

Describe the bug
Multiple invocations of useCookie are not synced, and so setting one will not trigger a change to others. This is not a bug per-se, but it's inconsistent compared to useState, for example...
Additional context
No response
Logs
No response

It seems this workaround does not work anymore 🤔
✅ Nuxt v3.3.3
❌ Nuxt v3.4.0
import { WatchCallback, watch } from 'vue';
import { CookieOptions, CookieRef } from '#app';
import { useCookie, useState } from '#imports';

/**
 * TODO: Remove when is fixed
 * https://github.com/nuxt/nuxt/issues/13020#issuecomment-1397282717
 */
export function useStatefulCookie<T = string | null>(
	name: string,
	options?: CookieOptions<T>,
): CookieRef<T> {
	const cookie = useCookie(name, options);
	const state = useState(name, () => cookie.value);

	const callback: WatchCallback<T, T> = () => {
		cookie.value = state.value;
	};

	watch(state, callback, { deep: true });

	return state;
}

Issue above fixed/works in v3.4.1

May CookieStore somehow improve this? CookieStore allows you to track when value cookie changed.  Theoretically, useCookie may return a ref and add cookie change event listener and change ref value.
function useCookie(name) {
	const cookie = ref()

	// load current cookie value
	const loadCookie = () => ref.value = getCookieCurrentValue(name);
	loadCookie()

	// cookieStore works only in client
	if (process.client) {
		// listen cookie changes and update ref
		cookieStore.addEventListener('change', loadCookie)
		onUnmounted(cookieStore.removeEventListener(loadCookie))
	}

	return cookie;
}


Warning message is not shown when trying to save a reaction with R-group members as .rxn file

Description:
When a user saves a reaction with R-group members as rxn.file, the warning message "ERROR: Unable to save the structure - reactions with R-Groups are not supported at the moment" is not shown and the reaction is not saved.
Steps to reproduce:

Create a reaction.
Create a structure.
Define the structure as R-Group member (R1).
Click "Save As..." button.

Actual result:
When clicking "Save As..." button, the reaction is not saved.
"ERROR: Unable to save the structure - reactions with R-Groups are not supported at the moment" message is not shown.

Expected result:
"ERROR: Unable to save the structure - reactions with R-Groups are not supported at the moment" message is shown when clicking "Save As..." button.

It has been already fixed

Status: PASS
Build: Release 2.1
OS/Browser: Win10/Chrome
Actual result:
'Reactions with r-groups are not supported at the moment' message is shown when clicking "Save As..." button.


Quick Reference Doc

https://github.com/Fechin/reference/blob/main/source/_posts/rust.md

https://github.com/balaji303/learn_rust/commit/d96846c9bfa14b3557f89ed4a42bd81cae3bd38d
fixes this issue


Extract utils into a separate package

Circuit UI exports a number of utility functions for dealing with currencies and numbers. IMO, these should be distributed separately, for the following reasons:

Using the utility functions without Circuit UI or even React, e.g. in a Node backend.
Upgrading the utility functions without having to upgrade components.

Circuit UI is a component library. It should contain/expose as little logic as possible.

I think we could move this to our "i18n" library. It's not uncommon to have something like this. What do you think?

Could make sense. What else does a i18n library usually contain?

I like the idea of grouping it this way 👍

Give flexibility to the consuming application to choose the library of their choice
Maybe there's a third-party library we can use?

Work in progress here: https://github.com/sumup-oss/intl-js/


fix: tiles 638 and 642 values to 70 (#479)

http://www.fwtwr.com/18xx/tiles/tiles.asp

Thanks!

Pull Request Test Coverage Report for Build<PHONE_NUMBER>3
Details

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 3.202%

Totals

Change from base Build<PHONE_NUMBER>9:
0.0%

Covered Lines:
445

Relevant Lines:
15828

💛  - Coveralls


arch/mpfs: Make selection of SBI boot or direct boot run-time configu…

…rable
Allow bootloader to select run-time whether the payload binary is booted with SBI or directly by jumping to entrypoint address.

Use just one bitmask to select sbi or direct boot for each hart
Add mpfs_set_use_sbi function to allow selecting how to boot
Initialize the bitmask by default according to the configuration flags

Signed-off-by: Jukka Laitinen<EMAIL_ADDRESS>Summary
Impact
Testing

I also added a header file for entrypoint functions for easier include in bootloader code

#undef EXTERN
#if defined(__cplusplus)
#define EXTERN extern "C"
extern "C"
{
#else
#define EXTERN extern
#endif

void somefunction(void); // Works in C and C++ both
EXTERN someotherfunction(void); // Works, but is unnecessary

int g_someglobalvalue; // Obviously wrong
extern g_someotherglobalvalue; // Works with C only
EXTERN g_yetanotherglobalvalue; // Works with C and C++ both

#undef EXTERN
#ifdef __cplusplus
}
#endif

So the #define EXTERN extern "C" is still needed if global exported data is included in the header. This was what I meant :)

Yes, absolutely :) but as there is none, isn't it correct as is? Or do you see that you'd need "extern " in front of the function declarations in case of C ?

So the #define EXTERN extern "C" is still needed if global exported data is included in the header. This was what I meant :)

Yes, absolutely :) but as there is none, isn't it correct as is?

Yup, like I said it is correct as is now.

Or do you see that you'd need "extern " in front of the function declarations in case of C ?

No, like I said they are not needed, and I know that if there is an extra EXTERN in front of a function declaration, mr. Scary Monster will want to remove them
Other headers still define the guard like this, even if the header only has function prototypes:
#undef EXTERN
#if defined(__cplusplus)
#define EXTERN extern "C"
extern "C"
{
#else
#define EXTERN extern
#endif

...

#undef EXTERN
#ifdef __cplusplus
}
#endif

I anticipate mr. Scary Monster will want you to add the guards like this.