[Model] Adding support for MiniCPM-V

Adding support for MiniCPM-V-2, please review.
HuggingFace Page: https://huggingface.co/openbmb/MiniCPM-V-2

There's an incompatible pip's dependency error, the questions are listed as follows:

MiniCPM-V need Timm package, where should I add this dependency requirement? I can see many different requirements files in the root dictionary of vllm.
Timm package needs torch==2.1.2, nvidia-nccl-cu12==2.18.1, triton==2.1.0, but these of vllm are torch==2.2.1, nvidia-nccl-cu12==2.19.3, triton==2.20. How can I solve this problem?

seems to be related with @ywang96 RFC https://github.com/vllm-project/vllm/issues/4194 on multi-modality models.

Timm package needs torch==2.1.2, nvidia-nccl-cu12==2.18.1, triton==2.1.0, but these of vllm are torch==2.2.1, nvidia-nccl-cu12==2.19.3, triton==2.20. How can I solve this problem?

We can't do anything until timm has the same dependency as vllm. Or you can try to remove timm dependency.

Sry, we were confused by this situation.
Actually, timm only requires torch >= 1.7 and we've add this dependency in requirements-common.txt.
Please review. @youkaichao @ywang96

@HwwwwwwwH  Thanks for your excellent work, may I ask what is preventing the  progress  of this PR?

Very sry for late!! We've been working on the new VLM MiniCPM-V-2.5 last few days.
I've pushed the new commit according to the reviews. And I see some new features about VLM, is there any requirements for  adapting these features?
Really sry~

ping @ywang96

We should probably get this merged - I will review it, thanks for the ping @jeejeelee

@HwwwwwwwH  Thanks for this great work , by the way, are there any plans to integrate MiniCPM-V-2.5 into vLLM?

@HwwwwwwwH Thanks for this great work , by the way, are there any plans to integrate MiniCPM-V-2.5 into vLLM?

We're working on this.

Very sry for late!! We've been working on the new VLM MiniCPM-V-2.5 last few days.
I've pushed the new commit according to the reviews. And I see some new features about VLM, is there any requirements for adapting these features?
Really sry~

@HwwwwwwwH Very impressive works! I am wondering that could you provide a 'batch' infer example? I have met some errors as following.
File "/xxxxxt/3rd_pkgs/vllm/vllm/model_executor/models/minicpmv.py", line 552, in get_vllm_embedding
image_indices = torch.stack([
RuntimeError: stack expects each tensor to be equal size, but got [0] at entry 0 and [64] at entry 1
2024-05-27 19:53:30,555 - root - INFO - request timeout: 7
2024-05-27 19:58:30,555 - root - INFO - request timeout: 8
Looking forward to your replay & MiniCPM-V-2.5 releases.

Is MiniCPM-Llama3-V 2.5 supported? I tried it and it doesn't seem to work. I used https://github.com/OpenBMB/vllm.

Is MiniCPM-Llama3-V 2.5 supported? I tried it and it doesn't seem to work. I used https://github.com/OpenBMB/vllm.

Not yet, @DarkLight1337 and I have been working on the refactoring and we're about to merge a pretty big PR that would require slightly more work for developers to add their new models. See #4197

@ywang96 @HwwwwwwwH  Sorry to bother you again, but could you please let me know the current progress of this PR?

@ywang96 @HwwwwwwwH Sorry to bother you again, but could you please let me know the current progress of this PR?

Sry, for this... I thought I need to waiting for the last commit to be merged. And then I'll start a new PR for MiniCPM-Llama3-V 2.5. I don't know if things are as I thought.

@ywang96 @HwwwwwwwH Sorry to bother you again, but could you please let me know the current progress of this PR?

Sry, for this... I thought I need to waiting for the last commit to be merged. And then I'll start a new PR for MiniCPM-Llama3-V 2.5. I don't know if things are as I thought.

@ywang96 Sry to bother. Is there anything left I need to do to merge MiniCPMV?

Can you add a test to check the model's consistency with its HuggingFace counterpart? That would be the most straightforward way to verify the correctness of your implementation.
Afterwards, please edit test-pipeline.yaml and multimodal tests in the same way as #5189 to include them in the CI.

By the way, there have been a few changes to multimodal models in the past few weeks, so you should merge main into your branch first.

Almost forgot - please also update vllm.multimodal.get_full_image_text_prompt with your model, otherwise, it won't be available via OpenAI API.

Almost forgot - please also update vllm.multimodal.get_full_image_text_prompt with your model, otherwise, it won't be available via OpenAI API.

Got it! Thank you!

any update?

Almost forgot - please also update vllm.multimodal.get_full_image_text_prompt with your model, otherwise, it won't be available via OpenAI API.

Got it! Thank you! By the way, May I ask one more question? In our MiniCPMV, there's a mechanism to slice the input image. In our previous implementation, we put the slicing into the model. In other words, there's a calculation to get the number of <image> tokens, we wish it could be done in the model. But we found that the length of input_ids can not be changed in the forward process in vLLM. So we have to put a copy code of slicing also in the example file(examples/example_minicpmv.py) to calculate the num of <image>, this make it awkward to use. I've checked the new features of VLM supporting, but still don't to know how to solve it.

Sorry I didn't read your edit. Does your model have dynamic number of image tokens based on the input size? That is not supported yet.

By the way, May I ask one more question? In our MiniCPMV, there's a mechanism to slice the input image. In our previous implementation, we put the slicing into the model. In other words, there's a calculation to get the number of <image> tokens, we wish it could be done in the model. But we found that the length of input_ids can not be changed in the forward process in vLLM. So we have to put a copy code of slicing also in the example file(examples/example_minicpmv.py) to calculate the num of <image>, this make it awkward to use. I've checked the new features of VLM supporting, but still don't to know how to solve it.

Sorry I didn't read your edit. Does your model have dynamic number of image tokens based on the input size? That is not supported yet. For now, you'll have to assume a fixed size just like the case for LLaVA-NeXT.

This is exactly what I want. Hope it will come soon. For now, We use pre-calculate the number of image tokens in example file. But for OpenAI API, maybe it's hard to adapt.

any update?

Sry...I'll the modifications as soon as possible.

@DarkLight1337 @ywang96
These what we've done in the past few days:

Merged the new main branch.
Fixed a bug in minicpmv in slicing image.
Added test for minicpmv in tests.
Tested the consistency for vLLM and its huggingface version.
Added minicpmv in vllm.multimodal.get_full_image_text_prompt.
Added MiniCPM-Llama3-V-2_5[Almost Done]

There are still some questions should be mentioned:

MiniCPM-V will slice image when a image sent to it. So it needs a mechanism to change the length of input tokens. As this is not supported in vLLM for now, we achieve this by calculating the placeholder outside vLLM and slice image in the model(Reference to examples/minicpmv_example.py). But this is difficult and complicated to move to vllm.multimodal.get_full_image_text_prompt. As there's a todo task to move this function to model registry, can we wait for this to be done?
There's some little difference in MiniCPM-V and its HF version. we checked this question and found the differences occurs in MiniCPM(not MiniCPM-V). And it comes from the MLP or Linear implementations. For example, in logits computing, vLLM use the weight of linear layer to multiply hidden_states and the other use forward directly. The results of these two methods are different($10^{-3}~10^{-4}$ for bfloat16). This makes the results of two version different. For validate the correctness of MiniCPM-V in vLLM, we tested it in some benchmarks and it gets the almost the same score compared to its original version in HF. Therefore, I didn't add the test file(test/test_minicpmv.py) to test-pipeline.yaml. Please check this and modify as you want.
The intergration of MiniCPM-Llama3-V-2_5 is almost done. Before that, could we merge this to main branch? It will be easier to update the code that you've reviewed.

At last, It takes a long time since I started this PR. I'm very sorry for the delays that caused by me. Wish the remaining things can go smoothly.~

Thanks for the update!

MiniCPM-V will slice image when a image sent to it. So it needs a mechanism to change the length of input tokens. As this is not supported in vLLM for now, we achieve this by calculating the placeholder outside vLLM and slice image in the model(Reference to examples/minicpmv_example.py). But this is difficult and complicated to move to vllm.multimodal.get_full_image_text_prompt. As there's a todo task to move this function to model registry, can we wait for this to be done?

Can you convert this code into a HuggingFace ImageProcessor, similar to how it's being handled in LLaVA-NeXT? Meanwhile, you can set image_feature_size to be consistent with the image input shape which is also a fixed value according to the config.

There's some little difference in MiniCPM-V and its HF version. we checked this question and found the differences occurs in MiniCPM(not MiniCPM-V). And it comes from the MLP or Linear implementations. For example, in logits computing, vLLM use the weight of linear layer to multiply hidden_states and the other use forward directly. The results of these two methods are different(10−3 10−4 for bfloat16). This makes the results of two version different. For validate the correctness of MiniCPM-V in vLLM, we tested it in some benchmarks and it gets the almost the same score compared to its original version in HF. Therefore, I didn't add the test file(test/test_minicpmv.py) to test-pipeline.yaml. Please check this and modify as you want.

Can you include an example output for the two implementations? If there is only a little divergence between the outputs, you can set max_tokens to a smaller value like the case for Phi-3V so that the test still passes.

The intergration of MiniCPM-Llama3-V-2_5 is almost done. Before that, could we merge this to main branch? It will be easier to update the code that you've reviewed.

Let's address the above two concerns first.

i

Thanks for your excellent work. May I ask when it will be completed?

To use ImageProcessor and Processor of HF, I need more few days to check our repo. (Working on this right now....

Thanks for your excellent work. May I ask when it will be completed?

#5276 has been merged, so you can start to make changes accordingly to support dynamic number of image tokens.

To use ImageProcessor and Processor of HF, I need more few days to check our repo. (Working on this right now....

Thanks for your excellent work. May I ask when it will be completed?

Before next Tuesday? I need a weekend to finish this...

Hi, I've completed image_processor of MiniCPMV and thought this was invoke in model_runner._prepare_model_input_tensors https://github.com/vllm-project/vllm/blob/main/vllm/worker/model_runner.py#L532.
And the new multimodal features add register_input_processor, which in https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/llava_next.py#L169 and https://github.com/vllm-project/vllm/blob/main/vllm/inputs/registry.py#L184. What's the relationship and difference between these two processings?

We have recently released a new guide, see if this can answer your questions.

There are two different processors in play here - the input processor which directly operates on the user input to reserve placeholders for the image features, and the image processor which calls the HuggingFace processor to preprocess the image (e.g. resize),
The overall pipeline is noted here.

There are two different components in play here:

The input processor which directly operates on the user input to reserve placeholders for the image features (registered using INPUT_REGISTRY.register_input_processor and called in LLMEngine).
The input mapper which calls the HuggingFace processor to preprocess the image (registered using MULTIMODAL_REGISTRY.register_image_input_mapper and called in ModelRunner).

For reference, the overall pipeline is noted here.

Thanks for answering, so the INPUT_REGISTRY.register_input_processor doesn't process the images? And I should put the placeholder calculating in there, and leave the image slicing in image_processor. Is this right?

You should place the placeholder calculation in the input processor.
If your model has a HuggingFace processor, it should be unnecessary to define your own input mapper (the default one should work fine already).

In model_runner._prepare_model_input_tensors.
How does it construct multi_modal_kwargs?
Usually I in the image_processor of my implementation for minicpmv, I can get image list shapes of [batch_size, num_images, 3, Hi, Wi], every image may be in different size, so it is like(list[list[tensor(C, H, W)]]). But I get a more dimension in multi_modal_kwargs(64) for image. How should I modify my code?

The image processor only accepts a single example (not a batch), but you should return a dictionary of tensors (each having a batch dimension of size 1). This is consistent with the behaviour of most HuggingFace processors that I've worked with.

Thx for reply.
Another question: in test, https://github.com/vllm-project/vllm/blob/main/tests/conftest.py#L408, it seems that the output consists of the input concatenating with output. But I found that the output from our model only contains the output...

See if this patch fixes the problem:
https://github.com/vllm-project/vllm/pull/5920/files#diff-e52e4ddd58b7ef887ab03c04116e676f6280b824ab7469d5d3080e5cba4f2128

I've modified the code of MiniCPM-V-2 and MiniCPM-Llama3-V-2_5 on huggingface and the codes are still in PR. So the code of vllm can not be run directly right now. But you can check if this is appropriate for vLLM.
If you want to run it locally for checking, you can use code in this repo(https://github.com/HwwwwwwwH/MiniCPM-V-2/tree/main)

I've modified the code of MiniCPM-V-2 and MiniCPM-Llama3-V-2_5 on huggingface and the codes are still in PR(https://huggingface.co/openbmb/MiniCPM-V-2/discussions/14). So the code of vllm can not be run directly right now. But you can check if this is appropriate for vLLM. If you want to run it locally for checking, you can use code in this repo(https://github.com/HwwwwwwwH/MiniCPM-V-2/tree/main)

I've modified the code of MiniCPM-V-2 and MiniCPM-Llama3-V-2_5 on huggingface and the codes are still in PR(https://huggingface.co/openbmb/MiniCPM-V-2/discussions/14). So the code of vllm can not be run directly right now. But you can check if this is appropriate for vLLM. If you want to run it locally for checking, you can use code in this repo(https://github.com/HwwwwwwwH/MiniCPM-V-2/tree/main)

Hi, thx for your codes. I want test MiniCPM-Llama3-V-2_5  with vllm. After merge your pr with vllm and run vllm, I met such error
"""
k0]:   File "/root/mubai/deployment_llm/vllm/vllm/vllm/model_executor/models/minicpmv.py", line 297, in init
[rank0]:     self.llm = MiniCPMForCausalLM(config,
[rank0]:   File "/root/mubai/deployment_llm/vllm/vllm/vllm/model_executor/models/minicpm.py", line 454, in init
[rank0]:     self.scale_width = self.config.hidden_size / self.config.dim_model_base
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/transformers/configuration_utils.py", line 264, in getattribute
[rank0]:     return super().getattribute(key)
[rank0]: AttributeError: 'MiniCPMVConfig' object has no attribute 'dim_model_base'
"""
It seems like I should update the Huggingface files. But you just mentioned the code https://huggingface.co/openbmb/MiniCPM-V-2/discussions/14) and where is the corresponding code for MiniCPM-Llama3-V-2_5?

I've modified the code of MiniCPM-V-2 and MiniCPM-Llama3-V-2_5 on huggingface and the codes are still in PR(https://huggingface.co/openbmb/MiniCPM-V-2/discussions/14). So the code of vllm can not be run directly right now. But you can check if this is appropriate for vLLM. If you want to run it locally for checking, you can use code in this repo(https://github.com/HwwwwwwwH/MiniCPM-V-2/tree/main)

Hi, thx for your codes. I want test MiniCPM-Llama3-V-2_5 with vllm. After merge your pr with vllm and run vllm, I met such error """ k0]: File "/root/mubai/deployment_llm/vllm/vllm/vllm/model_executor/models/minicpmv.py", line 297, in init [rank0]: self.llm = MiniCPMForCausalLM(config, [rank0]: File "/root/mubai/deployment_llm/vllm/vllm/vllm/model_executor/models/minicpm.py", line 454, in init [rank0]: self.scale_width = self.config.hidden_size / self.config.dim_model_base [rank0]: File "/usr/local/lib/python3.10/dist-packages/transformers/configuration_utils.py", line 264, in getattribute [rank0]: return super().getattribute(key) [rank0]: AttributeError: 'MiniCPMVConfig' object has no attribute 'dim_model_base' """
It seems like I should update the Huggingface files. But you just mentioned the code https://huggingface.co/openbmb/MiniCPM-V-2/discussions/14) and where is the corresponding code for MiniCPM-Llama3-V-2_5?

I'll figure it out today.

I pushed a new commit to support MiniCPM-V-2 and MiniCPM-Llama3-V-2_5 simultaneously. The only thing left to do is completing the test of minicpmv.
By the way, I got No module named 'vllm.commit_id'   from vllm.version import __version__ as VLLM_VERSION when I use the latest vllm. How can I solve this?

I pushed a new commit to support MiniCPM-V-2 and MiniCPM-Llama3-V-2_5 simultaneously. The only thing left to do is completing the test of minicpmv. (And formating code hhhhh) By the way, I got No module named 'vllm.commit_id' from vllm.version import __version__ as VLLM_VERSION when I use the latest vllm. How can I solve this?

You can rebuild vLLM from source to get rid of this warning.

I pushed a new commit to support MiniCPM-V-2 and MiniCPM-Llama3-V-2_5 simultaneously. The only thing left to do is completing the test of minicpmv. (And formating code hhhhh) By the way, I got No module named 'vllm.commit_id' from vllm.version import __version__ as VLLM_VERSION when I use the latest vllm. How can I solve this?

I have tested the code from the branch 'minicpmv2.5' found at this GitHub repository and executed the script minicpmv_example.py. My checkpoints were successfully downloaded from Hugging Face. I was able to run the tests correctly in my environment.
However, I encountered an error: AttributeError: 'MiniCPMVConfig' object has no attribute 'im_start_token_id'. Did you mean 'decoder_start_token_id'? while testing the vllm.
Have there been any recent changes to the configuration files?"

I pushed a new commit to support MiniCPM-V-2 and MiniCPM-Llama3-V-2_5 simultaneously. The only thing left to do is completing the test of minicpmv. (And formating code hhhhh) By the way, I got No module named 'vllm.commit_id' from vllm.version import __version__ as VLLM_VERSION when I use the latest vllm. How can I solve this?

I have tested the code from the branch 'minicpmv2.5' found at this GitHub repository and executed the script minicpmv_example.py. My checkpoints were successfully downloaded from Hugging Face. I was able to run the tests correctly in my environment.
However, I encountered an error: AttributeError: 'MiniCPMVConfig' object has no attribute 'im_start_token_id'. Did you mean 'decoder_start_token_id'? while testing the vllm.
Have there been any recent changes to the configuration files?"

The updated code has not be updated to the official repo yet. Try this: https://huggingface.co/HwwwH/MiniCPM-Llama3-V-2_5

I've completed the the tests. Plz review and see what else can I do? @DarkLight1337

I've completed the the tests. Plz review and see what else can I do? @DarkLight1337

Please update the "Supported Models" page with the new models that you have added.

I pushed a new commit to support MiniCPM-V-2 and MiniCPM-Llama3-V-2_5 simultaneously. The only thing left to do is completing the test of minicpmv. (And formating code hhhhh) By the way, I got No module named 'vllm.commit_id' from vllm.version import __version__ as VLLM_VERSION when I use the latest vllm. How can I solve this?

I have tested the code from the branch 'minicpmv2.5' found at this GitHub repository and executed the script minicpmv_example.py. My checkpoints were successfully downloaded from Hugging Face. I was able to run the tests correctly in my environment.
However, I encountered an error: AttributeError: 'MiniCPMVConfig' object has no attribute 'im_start_token_id'. Did you mean 'decoder_start_token_id'? while testing the vllm.
Have there been any recent changes to the configuration files?"

The updated code has not be updated to the official repo yet. Try this: https://huggingface.co/HwwwH/MiniCPM-Llama3-V-2_5

Hi, to deploy the server-side vllm of Minicpm v2.5, should I execute the following command?
git clone https://github.com/HwwwwwwwH/vllm.git
cd vllm
# then switch to the minicpmv2.5 branch
pip install -e .

then run in the cli
vllm serve HwwwH/MiniCPM-Llama3-V-2_5 --dtype auto

is this correct?

I pushed a new commit to support MiniCPM-V-2 and MiniCPM-Llama3-V-2_5 simultaneously. The only thing left to do is completing the test of minicpmv. (And formating code hhhhh) By the way, I got No module named 'vllm.commit_id' from vllm.version import __version__ as VLLM_VERSION when I use the latest vllm. How can I solve this?

I have tested the code from the branch 'minicpmv2.5' found at this GitHub repository and executed the script minicpmv_example.py. My checkpoints were successfully downloaded from Hugging Face. I was able to run the tests correctly in my environment.
However, I encountered an error: AttributeError: 'MiniCPMVConfig' object has no attribute 'im_start_token_id'. Did you mean 'decoder_start_token_id'? while testing the vllm.
Have there been any recent changes to the configuration files?"

The updated code has not be updated to the official repo yet. Try this: https://huggingface.co/HwwwH/MiniCPM-Llama3-V-2_5

Hi, to deploy the server-side vllm of Minicpm v2.5, should I execute the following command?
git clone https://github.com/HwwwwwwwH/vllm.git
cd vllm
# then switch to the minicpmv2.5 branch
pip install -e .

then run in the cli
vllm serve HwwwH/MiniCPM-Llama3-V-2_5 --dtype auto

is this correct?

Right. But the repo is still in modifying. I'm trying to get our code into vllm official repo first(this will be more convenient for using). If there's any problem, maybe you need wait a few days.

I pushed a new commit to support MiniCPM-V-2 and MiniCPM-Llama3-V-2_5 simultaneously. The only thing left to do is completing the test of minicpmv. (And formating code hhhhh) By the way, I got No module named 'vllm.commit_id' from vllm.version import __version__ as VLLM_VERSION when I use the latest vllm. How can I solve this?

I have tested the code from the branch 'minicpmv2.5' found at this GitHub repository and executed the script minicpmv_example.py. My checkpoints were successfully downloaded from Hugging Face. I was able to run the tests correctly in my environment.
However, I encountered an error: AttributeError: 'MiniCPMVConfig' object has no attribute 'im_start_token_id'. Did you mean 'decoder_start_token_id'? while testing the vllm.
Have there been any recent changes to the configuration files?"

The updated code has not be updated to the official repo yet. Try this: https://huggingface.co/HwwwH/MiniCPM-Llama3-V-2_5

Hi, to deploy the server-side vllm of Minicpm v2.5, should I execute the following command?
git clone https://github.com/HwwwwwwwH/vllm.git
cd vllm
# then switch to the minicpmv2.5 branch
pip install -e .

then run in the cli
vllm serve HwwwH/MiniCPM-Llama3-V-2_5 --dtype auto

is this correct?

Right. But the repo is still in modifying. I'm trying to get our code into vllm official repo first(this will be more convenient for using). Of course you can post the question here or MiniCPM-V repo. I'll try my best to help you.

Plz watch the test, it seems that the failures of node tests are caused by the network...

Please check the warnings log for the vision models tests. It appears that the VLLM output format is different from HF's, can you fix it?

Thank you,,
I have one question.
If I want to infer the previous version of the model as vllm, can't I infer as vllm if I don't have the preprocessor_config.json, processing_minicpmv.py, and image_processing_minicpmv.py files?

Thank you,, I have one question. If I want to infer the previous version of the model as vllm, can't I infer as vllm if I don't have the preprocessor_config.json, processing_minicpmv.py, and image_processing_minicpmv.py files?

Sry, I'm afraid not.

idefics2 requires huge GPU memory.
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 5.31 GiB. GPU
[rank0]:   File "/workspace/vllm/vllm/model_executor/models/minicpmv.py", line 616, in forward
[rank0]:     vlm_embeddings, vision_hidden_states = self.get_embedding(inputs)
[rank0]:   File "/workspace/vllm/vllm/model_executor/models/minicpmv.py", line 572, in get_embedding
[rank0]:     vision_hidden_states = self.get_vision_hidden_states(data)
[rank0]:   File "/workspace/vllm/vllm/model_executor/models/minicpmv.py", line 544, in get_vision_hidden_states
[rank0]:     vision_embedding = self.vpm(
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/transformers/models/idefics2/modeling_idefics2.py", line 729, in forward
[rank0]:     encoder_outputs = self.encoder(
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/transformers/models/idefics2/modeling_idefics2.py", line 654, in forward
[rank0]:     layer_outputs = encoder_layer(
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/transformers/models/idefics2/modeling_idefics2.py", line 565, in forward
[rank0]:     hidden_states, attn_weights = self.self_attn(
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/transformers/models/idefics2/modeling_idefics2.py", line 256, in forward
[rank0]:     attn_weights = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query_states.dtype)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/torch/nn/functional.py", line 1887, in softmax
[rank0]:     ret = input.softmax(dim, dtype=dtype)
[rank0]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 5.31 GiB. GPU

@DarkLight1337 Sorry to bother you, but is there any plan to merge and release the minicpmv-2 version into vllm?

@HwwwwwwwH @DarkLight1337 Sorry to bother you, but is there any plan to merge and release the minicpmv-2 version into vllm?

V2 and V2_5 will be supported simultaneously.

@HwwwwwwwH @DarkLight1337 Sorry to bother you, but is there any plan to merge and release the minicpmv-2 version into vllm?

V2 and V2_5 will be supported simultaneously.

I'm eagerly anticipating your work. Could you let me know the exact time it will be available? I'm really excited to give it a try~~ ^_^

@HwwwwwwwH @DarkLight1337 Sorry to bother you, but is there any plan to merge and release the minicpmv-2 version into vllm?

V2 and V2_5 will be supported simultaneously.

I'm eagerly anticipating your work. Could you let me know the exact time it will be available? I'm really excited to give it a try~~ ^_^

I've pushed the last commit right now. Maybe this time we can get it merged smoothly. Let's see...

@DarkLight1337 Thanks for your unwavering help! Could I left the last little things to you?

This is my serving command: vllm serve HwwwH/MiniCPM-Llama3-V-2_5 --dtype half --trust-remote-code
idefics2 requires huge GPU memory cause OOM. My GPU is A30, 24GB GPU memory. torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 5.31 GiB. GPU
[rank0]:   File "/workspace/vllm/vllm/model_executor/models/minicpmv.py", line 616, in forward
[rank0]:     vlm_embeddings, vision_hidden_states = self.get_embedding(inputs)
[rank0]:   File "/workspace/vllm/vllm/model_executor/models/minicpmv.py", line 572, in get_embedding
[rank0]:     vision_hidden_states = self.get_vision_hidden_states(data)
[rank0]:   File "/workspace/vllm/vllm/model_executor/models/minicpmv.py", line 544, in get_vision_hidden_states
[rank0]:     vision_embedding = self.vpm(
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/transformers/models/idefics2/modeling_idefics2.py", line 729, in forward
[rank0]:     encoder_outputs = self.encoder(
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/transformers/models/idefics2/modeling_idefics2.py", line 654, in forward
[rank0]:     layer_outputs = encoder_layer(
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/transformers/models/idefics2/modeling_idefics2.py", line 565, in forward
[rank0]:     hidden_states, attn_weights = self.self_attn(
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/transformers/models/idefics2/modeling_idefics2.py", line 256, in forward
[rank0]:     attn_weights = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query_states.dtype)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/torch/nn/functional.py", line 1887, in softmax
[rank0]:     ret = input.softmax(dim, dtype=dtype)
[rank0]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 5.31 GiB. GPU

Can you show the full stack trace? Also make sure your max_model_len is not set to a larger value than default.

This is my serving command: vllm serve HwwwH/MiniCPM-Llama3-V-2_5 --dtype half --trust-remote-code
idefics2 requires huge GPU memory cause OOM. My GPU is A30, 24GB GPU memory. torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 5.31 GiB. GPU
[rank0]:   File "/workspace/vllm/vllm/model_executor/models/minicpmv.py", line 616, in forward
[rank0]:     vlm_embeddings, vision_hidden_states = self.get_embedding(inputs)
[rank0]:   File "/workspace/vllm/vllm/model_executor/models/minicpmv.py", line 572, in get_embedding
[rank0]:     vision_hidden_states = self.get_vision_hidden_states(data)
[rank0]:   File "/workspace/vllm/vllm/model_executor/models/minicpmv.py", line 544, in get_vision_hidden_states
[rank0]:     vision_embedding = self.vpm(
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/transformers/models/idefics2/modeling_idefics2.py", line 729, in forward
[rank0]:     encoder_outputs = self.encoder(
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/transformers/models/idefics2/modeling_idefics2.py", line 654, in forward
[rank0]:     layer_outputs = encoder_layer(
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/transformers/models/idefics2/modeling_idefics2.py", line 565, in forward
[rank0]:     hidden_states, attn_weights = self.self_attn(
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/transformers/models/idefics2/modeling_idefics2.py", line 256, in forward
[rank0]:     attn_weights = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query_states.dtype)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/torch/nn/functional.py", line 1887, in softmax
[rank0]:     ret = input.softmax(dim, dtype=dtype)
[rank0]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 5.31 GiB. GPU

You can try --max-model-len 2048 when start serving.

Build and install vLLM from the source (merged with minicpm-V), and run the following code
vllm serve HwwwH/MiniCPM-Llama3-V-2_5 --dtype auto --trust-remote-code
lead to the following errors. How to solve this?

INFO 07-25 17:11:57 api_server.py:308] vLLM API server version 0.5.3.post1
INFO 07-25 17:11:57 api_server.py:309] args: Namespace(model_tag='HwwwH/MiniCPM-Llama3-V-2_5', host=None, port=8000, uvicorn_log_level='info', allow_credentials=False, allowed_origins=[''], allowed_methods=[''], allowed_headers=[''], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, model='HwwwH/MiniCPM-Llama3-V-2_5', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=True, download_dir=None, load_format='auto', dtype='auto', kv_cache_dtype='auto', quantization_param_path=None, max_model_len=None, guided_decoding_backend='outlines', distributed_executor_backend=None, worker_use_ray=False, pipeline_parallel_size=1, tensor_parallel_size=1, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=16, enable_prefix_caching=False, disable_sliding_window=False, use_v2_block_manager=False, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.9, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_seqs=256, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, enforce_eager=False, max_context_len_to_capture=None, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, enable_lora=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_model=None, num_speculative_tokens=None, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=None, qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, engine_use_ray=False, disable_log_requests=False, max_log_len=None, dispatch_function=<function serve at 0x7f3f5d388700>)
INFO 07-25 17:11:58 llm_engine.py:176] Initializing an LLM engine (v0.5.3.post1) with config: model='HwwwH/MiniCPM-Llama3-V-2_5', speculative_config=None, tokenizer='HwwwH/MiniCPM-Llama3-V-2_5', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=8192, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None), seed=0, served_model_name=HwwwH/MiniCPM-Llama3-V-2_5, use_v2_block_manager=False, enable_prefix_caching=False)
INFO 07-25 17:11:59 model_runner.py:720] Starting to load model HwwwH/MiniCPM-Llama3-V-2_5...
INFO 07-25 17:12:00 weight_utils.py:224] Using model weights format ['.safetensors']
Loading safetensors checkpoint shards:   0% Completed | 0/7 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  14% Completed | 1/7 [00:00<00:04,  1.33it/s]
Loading safetensors checkpoint shards:  29% Completed | 2/7 [00:01<00:04,  1.23it/s]
Loading safetensors checkpoint shards:  43% Completed | 3/7 [00:02<00:03,  1.20it/s]
Loading safetensors checkpoint shards:  57% Completed | 4/7 [00:03<00:02,  1.20it/s]
Loading safetensors checkpoint shards:  71% Completed | 5/7 [00:04<00:01,  1.20it/s]
Loading safetensors checkpoint shards:  86% Completed | 6/7 [00:04<00:00,  1.19it/s]
Loading safetensors checkpoint shards: 100% Completed | 7/7 [00:05<00:00,  1.19it/s]
Loading safetensors checkpoint shards: 100% Completed | 7/7 [00:05<00:00,  1.20it/s]
INFO 07-25 17:12:07 model_runner.py:732] Loading model weights took 15.9524 GB
/home/jingran_su/miniconda3/envs/llava/lib/python3.10/site-packages/transformers/models/auto/image_processing_auto.py:513: FutureWarning: The image_processor_class argument is deprecated and will be removed in v4.42. Please use slow_image_processor_class, or fast_image_processor_class instead
warnings.warn(
[rank0]: Traceback (most recent call last):
[rank0]:   File "/home/jingran_su/miniconda3/envs/llava/bin/vllm", line 8, in
[rank0]:     sys.exit(main())
[rank0]:   File "/home/jingran_su/workspace/vllm/vllm/scripts.py", line 149, in main
[rank0]:     args.dispatch_function(args)
[rank0]:   File "/home/jingran_su/workspace/vllm/vllm/scripts.py", line 29, in serve
[rank0]:     asyncio.run(run_server(args))
[rank0]:   File "/home/jingran_su/miniconda3/envs/llava/lib/python3.10/asyncio/runners.py", line 44, in run
[rank0]:     return loop.run_until_complete(main)
[rank0]:   File "/home/jingran_su/miniconda3/envs/llava/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
[rank0]:     return future.result()
[rank0]:   File "/home/jingran_su/workspace/vllm/vllm/entrypoints/openai/api_server.py", line 311, in run_server
[rank0]:     server = await build_server(
[rank0]:   File "/home/jingran_su/workspace/vllm/vllm/entrypoints/openai/api_server.py", line 233, in build_server
[rank0]:     if llm_engine is not None else AsyncLLMEngine.from_engine_args(
[rank0]:   File "/home/jingran_su/workspace/vllm/vllm/engine/async_llm_engine.py", line 466, in from_engine_args
[rank0]:     engine = cls(
[rank0]:   File "/home/jingran_su/workspace/vllm/vllm/engine/async_llm_engine.py", line 380, in init
[rank0]:     self.engine = self._init_engine(*args, **kwargs)
[rank0]:   File "/home/jingran_su/workspace/vllm/vllm/engine/async_llm_engine.py", line 547, in _init_engine
[rank0]:     return engine_class(*args, **kwargs)
[rank0]:   File "/home/jingran_su/workspace/vllm/vllm/engine/llm_engine.py", line 265, in init
[rank0]:     self._initialize_kv_caches()
[rank0]:   File "/home/jingran_su/workspace/vllm/vllm/engine/llm_engine.py", line 364, in _initialize_kv_caches
[rank0]:     self.model_executor.determine_num_available_blocks())
[rank0]:   File "/home/jingran_su/workspace/vllm/vllm/executor/gpu_executor.py", line 94, in determine_num_available_blocks
[rank0]:     return self.driver_worker.determine_num_available_blocks()
[rank0]:   File "/home/jingran_su/miniconda3/envs/llava/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
[rank0]:     return func(*args, **kwargs)
[rank0]:   File "/home/jingran_su/workspace/vllm/vllm/worker/worker.py", line 179, in determine_num_available_blocks
[rank0]:     self.model_runner.profile_run()
[rank0]:   File "/home/jingran_su/miniconda3/envs/llava/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
[rank0]:     return func(*args, **kwargs)
[rank0]:   File "/home/jingran_su/workspace/vllm/vllm/worker/model_runner.py", line 935, in profile_run
[rank0]:     self.execute_model(model_input, kv_caches, intermediate_tensors)
[rank0]:   File "/home/jingran_su/miniconda3/envs/llava/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
[rank0]:     return func(*args, **kwargs)
[rank0]:   File "/home/jingran_su/workspace/vllm/vllm/worker/model_runner.py", line 1354, in execute_model
[rank0]:     hidden_or_intermediate_states = model_executable(
[rank0]:   File "/home/jingran_su/miniconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:   File "/home/jingran_su/miniconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:   File "/home/jingran_su/workspace/vllm/vllm/model_executor/models/minicpmv.py", line 619, in forward
[rank0]:     vlm_embeddings, vision_hidden_states = self.get_embedding(inputs)
[rank0]:   File "/home/jingran_su/workspace/vllm/vllm/model_executor/models/minicpmv.py", line 562, in get_embedding
[rank0]:     vision_hidden_states = self.get_vision_hidden_states(data)
[rank0]:   File "/home/jingran_su/workspace/vllm/vllm/model_executor/models/minicpmv.py", line 526, in get_vision_hidden_states
[rank0]:     all_pixel_values = [
[rank0]:   File "/home/jingran_su/workspace/vllm/vllm/model_executor/models/minicpmv.py", line 527, in
[rank0]:     i.flatten(end_dim=1).permute(1, 0) for i in pixel_values
[rank0]: RuntimeError: permute(sparse_coo): number of dimensions in the tensor input does not match the length of the desired ordering of dimensions i.e. input.dim() = 1 is not equal to len(dims) = 2

When I used the latest code of vllm to build the docker image, I got the following error
Is there anything wrong with my request?
curl --location 'http://xxxx/v1/chat/completions' \
--header 'Content-Type: application/json' \
--data '{
  "model": "/models/MiniCPM-Llama3-V-2_5",
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "What’s in this image?"
        },
         {
          "type": "image_url",
          "image_url": {
            "url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"
          }
        }
      ]
    }
  ],
  "max_tokens": 300,
  "temperature":0.1
}'

INFO 07-25 09:32:31 logger.py:36] Received request chat-93e8d4d0900547d4adce1b58a83e6b98: prompt: '<|begin_of_text|><|start_header_id|>user<|end_header_id|>\n\nWhat’s in this image?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n', params: SamplingParams(n=1, best_of=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=0.1, top_p=1.0, top_k=-1, min_p=0.0, seed=None, use_beam_search=False, length_penalty=1.0, early_stopping=False, stop=[], stop_token_ids=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=300, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None), prompt_token_ids: [128000, 128006, 882, 128007, 271, 3923, 753, 304, 420, 2217, 30, 128009, 128006, 78191, 128007, 271], lora_request: None, prompt_adapter_request: None.
INFO:     <IP_ADDRESS>:2070 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/uvicorn/protocols/http/httptools_impl.py", line 399, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "/usr/local/lib/python3.10/dist-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
    return await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/applications.py", line 123, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/cors.py", line 85, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/exceptions.py", line 65, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 756, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 776, in app
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 297, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 77, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 72, in app
    response = await func(request)
  File "/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", line 278, in app
    raw_response = await run_endpoint_function(
  File "/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", line 191, in run_endpoint_function
    return await dependant.call(**values)
  File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/api_server.py", line 130, in create_chat_completion
    generator = await openai_serving_chat.create_chat_completion(
  File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/serving_chat.py", line 197, in create_chat_completion
    return await self.chat_completion_full_generator(
  File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/serving_chat.py", line 448, in chat_completion_full_generator
    async for res in result_generator:
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 772, in generate
    async for output in self._process_request(
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 888, in _process_request
    raise e
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 884, in _process_request
    async for request_output in stream:
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 93, in __anext__
    raise result
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 46, in _log_task_completion
    return_value = task.result()
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 637, in run_engine_loop
    result = task.result()
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 565, in engine_step
    await self.engine.add_request_async(**new_request)
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 323, in add_request_async
    processed_inputs = await self.process_model_inputs_async(
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 305, in process_model_inputs_async
    return self.input_processor(llm_inputs)
  File "/usr/local/lib/python3.10/dist-packages/vllm/inputs/registry.py", line 202, in process_input
    return processor(InputContext(model_config), inputs)
  File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/minicpmv.py", line 366, in input_processor_for_minicpmv
    + text_chunks[1]
IndexError: list index out of range

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/uvicorn/protocols/http/httptools_impl.py", line 399, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "/usr/local/lib/python3.10/dist-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
    return await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/applications.py", line 123, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/cors.py", line 85, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/exceptions.py", line 65, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 756, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 776, in app
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 297, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 77, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 72, in app
    response = await func(request)
  File "/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", line 278, in app
    raw_response = await run_endpoint_function(
  File "/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", line 191, in run_endpoint_function
    return await dependant.call(**values)
  File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/api_server.py", line 130, in create_chat_completion
    generator = await openai_serving_chat.create_chat_completion(
  File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/serving_chat.py", line 197, in create_chat_completion
    return await self.chat_completion_full_generator(
  File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/serving_chat.py", line 448, in chat_completion_full_generator
    async for res in result_generator:
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 772, in generate
    async for output in self._process_request(
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 873, in _process_request
    stream = await self.add_request(
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 676, in add_request
    self.start_background_loop()
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 516, in start_background_loop
    raise AsyncEngineDeadError(
vllm.engine.async_llm_engine.AsyncEngineDeadError: Background loop has errored already.

Build and install vLLM from the source (merged with minicpm-V), and run the following code
vllm serve HwwwH/MiniCPM-Llama3-V-2_5 --dtype auto --trust-remote-code
lead to the following errors. How to solve this? Thanks! @HwwwwwwwH

INFO 07-25 17:11:57 api_server.py:308] vLLM API server version 0.5.3.post1
INFO 07-25 17:11:57 api_server.py:309] args: Namespace(model_tag='HwwwH/MiniCPM-Llama3-V-2_5', host=None, port=8000, uvicorn_log_level='info', allow_credentials=False, allowed_origins=[''], allowed_methods=[''], allowed_headers=[''], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, model='HwwwH/MiniCPM-Llama3-V-2_5', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=True, download_dir=None, load_format='auto', dtype='auto', kv_cache_dtype='auto', quantization_param_path=None, max_model_len=None, guided_decoding_backend='outlines', distributed_executor_backend=None, worker_use_ray=False, pipeline_parallel_size=1, tensor_parallel_size=1, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=16, enable_prefix_caching=False, disable_sliding_window=False, use_v2_block_manager=False, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.9, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_seqs=256, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, enforce_eager=False, max_context_len_to_capture=None, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, enable_lora=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_model=None, num_speculative_tokens=None, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=None, qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, engine_use_ray=False, disable_log_requests=False, max_log_len=None, dispatch_function=<function serve at 0x7f3f5d388700>)
INFO 07-25 17:11:58 llm_engine.py:176] Initializing an LLM engine (v0.5.3.post1) with config: model='HwwwH/MiniCPM-Llama3-V-2_5', speculative_config=None, tokenizer='HwwwH/MiniCPM-Llama3-V-2_5', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=8192, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None), seed=0, served_model_name=HwwwH/MiniCPM-Llama3-V-2_5, use_v2_block_manager=False, enable_prefix_caching=False)
INFO 07-25 17:11:59 model_runner.py:720] Starting to load model HwwwH/MiniCPM-Llama3-V-2_5...
INFO 07-25 17:12:00 weight_utils.py:224] Using model weights format ['.safetensors']
Loading safetensors checkpoint shards:   0% Completed | 0/7 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  14% Completed | 1/7 [00:00<00:04,  1.33it/s]
Loading safetensors checkpoint shards:  29% Completed | 2/7 [00:01<00:04,  1.23it/s]
Loading safetensors checkpoint shards:  43% Completed | 3/7 [00:02<00:03,  1.20it/s]
Loading safetensors checkpoint shards:  57% Completed | 4/7 [00:03<00:02,  1.20it/s]
Loading safetensors checkpoint shards:  71% Completed | 5/7 [00:04<00:01,  1.20it/s]
Loading safetensors checkpoint shards:  86% Completed | 6/7 [00:04<00:00,  1.19it/s]
Loading safetensors checkpoint shards: 100% Completed | 7/7 [00:05<00:00,  1.19it/s]
Loading safetensors checkpoint shards: 100% Completed | 7/7 [00:05<00:00,  1.20it/s]
INFO 07-25 17:12:07 model_runner.py:732] Loading model weights took 15.9524 GB
/home/jingran_su/miniconda3/envs/llava/lib/python3.10/site-packages/transformers/models/auto/image_processing_auto.py:513: FutureWarning: The image_processor_class argument is deprecated and will be removed in v4.42. Please use slow_image_processor_class, or fast_image_processor_class instead
warnings.warn(
[rank0]: Traceback (most recent call last):
[rank0]:   File "/home/jingran_su/miniconda3/envs/llava/bin/vllm", line 8, in
[rank0]:     sys.exit(main())
[rank0]:   File "/home/jingran_su/workspace/vllm/vllm/scripts.py", line 149, in main
[rank0]:     args.dispatch_function(args)
[rank0]:   File "/home/jingran_su/workspace/vllm/vllm/scripts.py", line 29, in serve
[rank0]:     asyncio.run(run_server(args))
[rank0]:   File "/home/jingran_su/miniconda3/envs/llava/lib/python3.10/asyncio/runners.py", line 44, in run
[rank0]:     return loop.run_until_complete(main)
[rank0]:   File "/home/jingran_su/miniconda3/envs/llava/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
[rank0]:     return future.result()
[rank0]:   File "/home/jingran_su/workspace/vllm/vllm/entrypoints/openai/api_server.py", line 311, in run_server
[rank0]:     server = await build_server(
[rank0]:   File "/home/jingran_su/workspace/vllm/vllm/entrypoints/openai/api_server.py", line 233, in build_server
[rank0]:     if llm_engine is not None else AsyncLLMEngine.from_engine_args(
[rank0]:   File "/home/jingran_su/workspace/vllm/vllm/engine/async_llm_engine.py", line 466, in from_engine_args
[rank0]:     engine = cls(
[rank0]:   File "/home/jingran_su/workspace/vllm/vllm/engine/async_llm_engine.py", line 380, in init
[rank0]:     self.engine = self._init_engine(*args, **kwargs)
[rank0]:   File "/home/jingran_su/workspace/vllm/vllm/engine/async_llm_engine.py", line 547, in _init_engine
[rank0]:     return engine_class(*args, **kwargs)
[rank0]:   File "/home/jingran_su/workspace/vllm/vllm/engine/llm_engine.py", line 265, in init
[rank0]:     self._initialize_kv_caches()
[rank0]:   File "/home/jingran_su/workspace/vllm/vllm/engine/llm_engine.py", line 364, in _initialize_kv_caches
[rank0]:     self.model_executor.determine_num_available_blocks())
[rank0]:   File "/home/jingran_su/workspace/vllm/vllm/executor/gpu_executor.py", line 94, in determine_num_available_blocks
[rank0]:     return self.driver_worker.determine_num_available_blocks()
[rank0]:   File "/home/jingran_su/miniconda3/envs/llava/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
[rank0]:     return func(*args, **kwargs)
[rank0]:   File "/home/jingran_su/workspace/vllm/vllm/worker/worker.py", line 179, in determine_num_available_blocks
[rank0]:     self.model_runner.profile_run()
[rank0]:   File "/home/jingran_su/miniconda3/envs/llava/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
[rank0]:     return func(*args, **kwargs)
[rank0]:   File "/home/jingran_su/workspace/vllm/vllm/worker/model_runner.py", line 935, in profile_run
[rank0]:     self.execute_model(model_input, kv_caches, intermediate_tensors)
[rank0]:   File "/home/jingran_su/miniconda3/envs/llava/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
[rank0]:     return func(*args, **kwargs)
[rank0]:   File "/home/jingran_su/workspace/vllm/vllm/worker/model_runner.py", line 1354, in execute_model
[rank0]:     hidden_or_intermediate_states = model_executable(
[rank0]:   File "/home/jingran_su/miniconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:   File "/home/jingran_su/miniconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:   File "/home/jingran_su/workspace/vllm/vllm/model_executor/models/minicpmv.py", line 619, in forward
[rank0]:     vlm_embeddings, vision_hidden_states = self.get_embedding(inputs)
[rank0]:   File "/home/jingran_su/workspace/vllm/vllm/model_executor/models/minicpmv.py", line 562, in get_embedding
[rank0]:     vision_hidden_states = self.get_vision_hidden_states(data)
[rank0]:   File "/home/jingran_su/workspace/vllm/vllm/model_executor/models/minicpmv.py", line 526, in get_vision_hidden_states
[rank0]:     all_pixel_values = [
[rank0]:   File "/home/jingran_su/workspace/vllm/vllm/model_executor/models/minicpmv.py", line 527, in
[rank0]:     i.flatten(end_dim=1).permute(1, 0) for i in pixel_values
[rank0]: RuntimeError: permute(sparse_coo): number of dimensions in the tensor input does not match the length of the desired ordering of dimensions i.e. input.dim() = 1 is not equal to len(dims) = 2

Sry for this inconvenience. I should've delete this repo cause 2.5 has already been merged into this official repo. So there're some updates that I didn't pushed to mine.
Plz try the newest official vllm and MiniCPM-V-Llama3-V-2_5 code(not from mine). I think this will be fine.

When I used the latest code of vllm to build the docker image, I got the following error Is there anything wrong with my request?
curl --location 'http://xxxx/v1/chat/completions' \
--header 'Content-Type: application/json' \
--data '{
  "model": "/models/MiniCPM-Llama3-V-2_5",
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "What’s in this image?"
        },
         {
          "type": "image_url",
          "image_url": {
            "url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"
          }
        }
      ]
    }
  ],
  "max_tokens": 300,
  "temperature":0.1
}'

INFO 07-25 09:32:31 logger.py:36] Received request chat-93e8d4d0900547d4adce1b58a83e6b98: prompt: '<|begin_of_text|><|start_header_id|>user<|end_header_id|>\n\nWhat’s in this image?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n', params: SamplingParams(n=1, best_of=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=0.1, top_p=1.0, top_k=-1, min_p=0.0, seed=None, use_beam_search=False, length_penalty=1.0, early_stopping=False, stop=[], stop_token_ids=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=300, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None), prompt_token_ids: [128000, 128006, 882, 128007, 271, 3923, 753, 304, 420, 2217, 30, 128009, 128006, 78191, 128007, 271], lora_request: None, prompt_adapter_request: None.
INFO:     <IP_ADDRESS>:2070 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/uvicorn/protocols/http/httptools_impl.py", line 399, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "/usr/local/lib/python3.10/dist-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
    return await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/applications.py", line 123, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/cors.py", line 85, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/exceptions.py", line 65, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 756, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 776, in app
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 297, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 77, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 72, in app
    response = await func(request)
  File "/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", line 278, in app
    raw_response = await run_endpoint_function(
  File "/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", line 191, in run_endpoint_function
    return await dependant.call(**values)
  File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/api_server.py", line 130, in create_chat_completion
    generator = await openai_serving_chat.create_chat_completion(
  File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/serving_chat.py", line 197, in create_chat_completion
    return await self.chat_completion_full_generator(
  File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/serving_chat.py", line 448, in chat_completion_full_generator
    async for res in result_generator:
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 772, in generate
    async for output in self._process_request(
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 888, in _process_request
    raise e
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 884, in _process_request
    async for request_output in stream:
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 93, in __anext__
    raise result
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 46, in _log_task_completion
    return_value = task.result()
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 637, in run_engine_loop
    result = task.result()
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 565, in engine_step
    await self.engine.add_request_async(**new_request)
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 323, in add_request_async
    processed_inputs = await self.process_model_inputs_async(
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 305, in process_model_inputs_async
    return self.input_processor(llm_inputs)
  File "/usr/local/lib/python3.10/dist-packages/vllm/inputs/registry.py", line 202, in process_input
    return processor(InputContext(model_config), inputs)
  File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/minicpmv.py", line 366, in input_processor_for_minicpmv
    + text_chunks[1]
IndexError: list index out of range

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/uvicorn/protocols/http/httptools_impl.py", line 399, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "/usr/local/lib/python3.10/dist-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
    return await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/applications.py", line 123, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/cors.py", line 85, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/exceptions.py", line 65, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 756, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 776, in app
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 297, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 77, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 72, in app
    response = await func(request)
  File "/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", line 278, in app
    raw_response = await run_endpoint_function(
  File "/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", line 191, in run_endpoint_function
    return await dependant.call(**values)
  File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/api_server.py", line 130, in create_chat_completion
    generator = await openai_serving_chat.create_chat_completion(
  File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/serving_chat.py", line 197, in create_chat_completion
    return await self.chat_completion_full_generator(
  File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/serving_chat.py", line 448, in chat_completion_full_generator
    async for res in result_generator:
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 772, in generate
    async for output in self._process_request(
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 873, in _process_request
    stream = await self.add_request(
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 676, in add_request
    self.start_background_loop()
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 516, in start_background_loop
    raise AsyncEngineDeadError(
vllm.engine.async_llm_engine.AsyncEngineDeadError: Background loop has errored already.

Add image placeholder to message like examples/minicpmv_example.py
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "(<image>./</image>)\nWhat’s in this image?"
        },
         {
          "type": "image_url",
          "image_url": {
            "url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"
          }
        }
      ]
    }
  ],

@DarkLight1337 check if I forget to add this image placeholder feature to OpenAI Compatible Serving?

It doesn't seem that you added this placeholder.

Actually, there's a weird thing for users I need to state here. I'm sry for not getting MiniCPM-V-2

It doesn't seem that you added this placeholder.

It doesn't seem that you added this placeholder.

Then where should I add this?

Please update this function to handle your case properly.
https://github.com/vllm-project/vllm/blob/316a41ac1de4e6e46933cadb39b9b7af65b01abd/vllm/entrypoints/chat_utils.py#L95

(./)\n

I start a new PR to fix this bug. See #6787

I need to say sry for users waiting for MiniCPM-V-2 and MiniCPM-Llama3-V-2_5.

For now, I've got it ready for vllm and MiniCPM-Llama3-V-2_5.
As for MiniCPM-V-2, my PR to the official code on HF is still under reviewing. There are some necessary tests before it get to be done. I'll complete this soon, wishing it could be merged in next weekdays.
And before that, I create a HF repo that can support vllm. If you want to try it temporarily, you can see https://huggingface.co/HwwwH/MiniCPM-V-2. This code won't have huge differences with the final version.

Hi! The usage of MiniCPMv2_5 with openai compatible API works well, but it seems that some API parameters are not supported currently. For example, when I want to get the logprobs according to the issue:
from openai import OpenAI
  client = OpenAI(
      base_url="http://localhost:8000/v1",
      api_key="token-abc123",
  )

  completion = client.chat.completions.create(
  model="openbmb/MiniCPM-Llama3-V-2_5",
  messages=[
      {"role": "user", "content": "Do you think 2 is larger than 1? Answer yes or no."}
  ],
  extra_body={
      "stop": ['<|eot_id|>'],
      "echo": True,
      "max_tokens": 1,
      "logprobs": True,
  }
  )

The output only contains the logprobs of output (e.g., "yes", logprob = "-0.0065"), not input prompts.
Then how to solve this? @HwwwwwwwH

Hi! The usage of MiniCPMv2_5 with openai compatible API works well, but it seems that some API parameters are not supported currently. For example, when I want to get the logprobs according to the issue:
from openai import OpenAI
  client = OpenAI(
      base_url="http://localhost:8000/v1",
      api_key="token-abc123",
  )

  completion = client.chat.completions.create(
  model="openbmb/MiniCPM-Llama3-V-2_5",
  messages=[
      {"role": "user", "content": "Do you think 2 is larger than 1? Answer yes or no."}
  ],
  extra_body={
      "stop": ['<|eot_id|>'],
      "echo": True,
      "max_tokens": 1,
      "logprobs": True,
  }
  )

The output only contains the logprobs of output (e.g., "yes", logprob = "-0.0065"), not input prompts. Then how to solve this? @HwwwwwwwH

Have you disabled prompt caching as mentioned in the referenced issue? Please show the command which you've used to server the model.

Hi! The usage of MiniCPMv2_5 with openai compatible API works well, but it seems that some API parameters are not supported currently. For example, when I want to get the logprobs according to the issue:
from openai import OpenAI
  client = OpenAI(
      base_url="http://localhost:8000/v1",
      api_key="token-abc123",
  )

  completion = client.chat.completions.create(
  model="openbmb/MiniCPM-Llama3-V-2_5",
  messages=[
      {"role": "user", "content": "Do you think 2 is larger than 1? Answer yes or no."}
  ],
  extra_body={
      "stop": ['<|eot_id|>'],
      "echo": True,
      "max_tokens": 1,
      "logprobs": True,
  }
  )

The output only contains the logprobs of output (e.g., "yes", logprob = "-0.0065"), not input prompts. Then how to solve this? @HwwwwwwwH

Have you disabled prompt caching as mentioned in the referenced issue? Please show the command which you've used to serve the model.

The following is the command to serve the model without using --enable-prefix-caching
CUDA_VISIBLE_DEVICES=0 vllm serve openbmb/MiniCPM-Llama3-V-2_5 --dtype auto --trust-remote-code

I need to say sry for users waiting for MiniCPM-V-2 and MiniCPM-Llama3-V-2_5. There're something I have to state.

For now, I've got it ready for vllm and MiniCPM-Llama3-V-2_5.
As for MiniCPM-V-2, my PR to the official code on HF is still under reviewing. There are some necessary tests before it get to be done. I'll complete this soon, wishing it could be merged in next weekdays.
And before that, I create a HF repo that can support vllm. If you want to try it temporarily, you can see https://huggingface.co/HwwwH/MiniCPM-V-2. This code won't have huge differences with the final version.

Does it support MiniCPM-Llama3-V-2_5 now? I will get the following error when using MiniCPM-Llama3-V-2_5:
[rank0]: Traceback (most recent call last):
[rank0]:   File "/mnt/dolphinfs/ssd_pool/docker/user/hadoop-platcv/yanghao/code/13_optim/minicpm/vllm_infer.py", line 57, in <module>
[rank0]:     main()
[rank0]:   File "/mnt/dolphinfs/ssd_pool/docker/user/hadoop-platcv/yanghao/code/13_optim/minicpm/vllm_infer.py", line 33, in main
[rank0]:     llm, prompt = run_minicpmv(question)
[rank0]:   File "/mnt/dolphinfs/ssd_pool/docker/user/hadoop-platcv/yanghao/code/13_optim/minicpm/vllm_infer.py", line 16, in run_minicpmv
[rank0]:     llm = LLM(
[rank0]:   File "/home/hadoop-platcv/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/entrypoints/llm.py", line 155, in __init__
[rank0]:     self.llm_engine = LLMEngine.from_engine_args(
[rank0]:   File "/home/hadoop-platcv/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 441, in from_engine_args
[rank0]:     engine = cls(
[rank0]:   File "/home/hadoop-platcv/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 251, in __init__
[rank0]:     self.model_executor = executor_class(
[rank0]:   File "/home/hadoop-platcv/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/executor/executor_base.py", line 47, in __init__
[rank0]:     self._init_executor()
[rank0]:   File "/home/hadoop-platcv/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/executor/gpu_executor.py", line 36, in _init_executor
[rank0]:     self.driver_worker.load_model()
[rank0]:   File "/home/hadoop-platcv/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/worker/worker.py", line 139, in load_model
[rank0]:     self.model_runner.load_model()
[rank0]:   File "/home/hadoop-platcv/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 682, in load_model
[rank0]:     self.model = get_model(model_config=self.model_config,
[rank0]:   File "/home/hadoop-platcv/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/model_executor/model_loader/__init__.py", line 21, in get_model
[rank0]:     return loader.load_model(model_config=model_config,
[rank0]:   File "/home/hadoop-platcv/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/model_executor/model_loader/loader.py", line 280, in load_model
[rank0]:     model = _initialize_model(model_config, self.load_config,
[rank0]:   File "/home/hadoop-platcv/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/model_executor/model_loader/loader.py", line 111, in _initialize_model
[rank0]:     return model_class(config=model_config.hf_config,
[rank0]:   File "/home/hadoop-platcv/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/model_executor/models/minicpmv.py", line 395, in __init__
[rank0]:     self.version = float(self.config.version)
[rank0]:   File "/home/hadoop-platcv/miniconda3/envs/vllm/lib/python3.10/site-packages/transformers/configuration_utils.py", line 264, in __getattribute__
[rank0]:     return super().__getattribute__(key)
[rank0]: AttributeError: 'MiniCPMVConfig' object has no attribute 'version'

It should have been fixed last night. Please update to the latest main branch.

@HwwwwwwwH I find Qwen2Model in init_llm, Are there any plans to release maybe minicpmv3-Qwen2 in the future? v

It should have been fixed last night. Please update to the latest main branch.

thanks for your reply! The latest code can run normally, but there is a problem with Chinese output inference(contains some '<|eot_id|><|eot_id|>' characters), which is not available in English. What is the reason for this?
English output:

Chinese output:

Could you show the input prompt for each case?

It should have been fixed last night. Please update to the latest main branch.

thanks for your reply! The latest code can run normally, but there is a problem with Chinese output inference(contains some '<|eot_id|><|eot_id|>' characters), which is not available in English. What is the reason for this? English output:  Chinese output:
@Howe-Young
perhaps you need add stop_tokens.

stop_token_ids = [tokenizer.eos_id, tokenizer.eot_id]
sampling_params = SamplingParams(
stop_token_ids=stop_token_ids,
)

show the input prompt for each case?

English prompt:
question = "please describe the image in detail"
messages = [{
    'role': 'user',
    'content': f'(<image>./</image>)\n{question}'
}]
prompt = tokenizer.apply_chat_template(messages,
                                           tokenize=False,
                                           add_generation_prompt=True)

Chinese prompt:
question = "详细描述图片内容"
messages = [{
    'role': 'user',
    'content': f'(<image>./</image>)\n{question}'
}]
prompt = tokenizer.apply_chat_template(messages,
                                           tokenize=False,
                                           add_generation_prompt=True)

stop_token_ids = [tokenizer.eos_id, tokenizer.eot_id]
sampling_params = SamplingParams(
stop_token_ids=stop_token_ids,
)

thanks, add stop_token_ids=stop_token_ids it works!

你好！MiniCPMv2_5与 openai 兼容 API 一起使用效果很好，但目前似乎不支持某些 API 参数。例如，当我想根据问题获取 logprobs 时：
from openai import OpenAI
  client = OpenAI(
      base_url="http://localhost:8000/v1",
      api_key="token-abc123",
  )

  completion = client.chat.completions.create(
  model="openbmb/MiniCPM-Llama3-V-2_5",
  messages=[
      {"role": "user", "content": "Do you think 2 is larger than 1? Answer yes or no."}
  ],
  extra_body={
      "stop": ['<|eot_id|>'],
      "echo": True,
      "max_tokens": 1,
      "logprobs": True,
  }
  )

输出仅包含输出的 logprobs（例如，“yes”， logprob = “-0.0065”），不包含输入提示。那么如何解决这个问题呢？

请问一下，我使用pip install vllm 安装的vllm版本是0.5.3.post1，为啥还是不能使用python -m vllm.entrypoints.openai.api_server \ --model /home/nlp/xc/NLP/LLM/openLLM/MiniCPM-Llama3-V-2_5 \，他提醒我说不支持这个模型

你好！MiniCPMv2_5与 openai 兼容 API 一起使用效果很好，但目前似乎不支持某些 API 参数。例如，当我想根据问题获取 logprobs 时：
from openai import OpenAI
  client = OpenAI(
      base_url="http://localhost:8000/v1",
      api_key="token-abc123",
  )

  completion = client.chat.completions.create(
  model="openbmb/MiniCPM-Llama3-V-2_5",
  messages=[
      {"role": "user", "content": "Do you think 2 is larger than 1? Answer yes or no."}
  ],
  extra_body={
      "stop": ['<|eot_id|>'],
      "echo": True,
      "max_tokens": 1,
      "logprobs": True,
  }
  )

输出仅包含输出的 logprobs（例如，“yes”， logprob = “-0.0065”），不包含输入提示。那么如何解决这个问题呢？

请问一下，我使用pip install vllm 安装的vllm版本是0.5.3.post1，为啥还是不能使用python -m vllm.entrypoints.openai.api_server \ --model /home/nlp/xc/NLP/LLM/openLLM/MiniCPM-Llama3-V-2_5 \，他提醒我说不支持这个模型

This model was added very recently, after the release of 0.5.3.post1, so it'll only be included in the next release (e.g. 0.5.4). If you want to use it now, please install vLLM from source (i.e. main branch).

你好！MiniCPMv2_5与 openai 兼容 API 一起使用效果很好，但目前似乎不支持某些 API 参数。例如，当我想根据问题获取 logprobs 时：
from openai import OpenAI
  client = OpenAI(
      base_url="http://localhost:8000/v1",
      api_key="token-abc123",
  )

  completion = client.chat.completions.create(
  model="openbmb/MiniCPM-Llama3-V-2_5",
  messages=[
      {"role": "user", "content": "Do you think 2 is larger than 1? Answer yes or no."}
  ],
  extra_body={
      "stop": ['<|eot_id|>'],
      "echo": True,
      "max_tokens": 1,
      "logprobs": True,
  }
  )

输出仅包含输出的 logprobs（例如，“yes”， logprob = “-0.0065”），不包含输入提示。那么如何解决这个问题呢？

请问一下，我使用pip install vllm 安装的vllm版本是0.5.3.post1，为啥还是不能使用python -m vllm.entrypoints.openai.api_server \ --model /home/nlp/xc/NLP/LLM/openLLM/MiniCPM-Llama3-V-2_5 \，他提醒我说不支持这个模型

This model was added after the release of 0.5.3.post1, so it'll only be included in the next release (e.g. 0.5.4). If you want to use it now, please install vLLM from source (i.e. main branch).

@DarkLight1337 I'm updating this PR description to link to this comment from you given how many times we had to answer the same question :P

offline vllm推理 minicpmv2-6 会出现推理结果一直重复输出 某段文字。
vllm ==0.5.4
推理代码：
messages = [{
'role': 'user',
'content': f'(./)\n{question}'
}]
prompt = tokenizer.apply_chat_template(messages,
tokenize=False,
add_generation_prompt=True)
stop_token_ids = [tokenizer.eos_id, tokenizer.eot_id]
stop_token_ids = ['<|eot_id|>']
We set temperature to 0.2 so that outputs can be different
even when all prompts are identical when running batch inference.
sampling_params = SamplingParams(temperature=0.7, max_tokens=8192, stop_token_ids=stop_token_ids)
Single inference
inputs = {
"prompt": prompt,
"multi_modal_data": {
"image": image
},
}
# Batch inference
inputs = [{
"prompt": prompt,
"multi_modal_data": {
"image": image
},
} for _ in range(args.num_prompts)]
outputs = llm.generate(inputs, sampling_params=sampling_params)
for o in outputs:
generated_text = o.outputs[0].text
print(generated_text)

offline vllm推理 minicpmv2-6 会出现推理结果一直重复输出 某段文字。 vllm ==0.5.4 推理代码：
messages = [{
    'role': 'user',
    'content': f'(<image>./</image>)\n{question}'
}]
prompt = tokenizer.apply_chat_template(messages,
                                       tokenize=False,
                                       add_generation_prompt=True)

stop_token_ids = ['<|eot_id|>']
sampling_params = SamplingParams(temperature=0.7, max_tokens=8192, stop_token_ids=stop_token_ids)

inputs = {
    "prompt": prompt,
    "multi_modal_data": {
        "image": image
    },
}

outputs = llm.generate(inputs, sampling_params=sampling_params)

for o in outputs:
    generated_text = o.outputs[0].text
    print(generated_text)

辛苦帮忙看下，感谢~ @ywang96

Could you share a sample input/output with the repetitive generation?

offline vllm推理 minicpmv2-6 会出现推理结果一直重复输出 某段文字。 vllm ==0.5.4 推理代码：
messages = [{
    'role': 'user',
    'content': f'(<image>./</image>)\n{question}'
}]
prompt = tokenizer.apply_chat_template(messages,
                                       tokenize=False,
                                       add_generation_prompt=True)

stop_token_ids = ['<|eot_id|>']
sampling_params = SamplingParams(temperature=0.7, max_tokens=8192, stop_token_ids=stop_token_ids)

inputs = {
    "prompt": prompt,
    "multi_modal_data": {
        "image": image
    },
}

outputs = llm.generate(inputs, sampling_params=sampling_params)

for o in outputs:
    generated_text = o.outputs[0].text
    print(generated_text)

辛苦帮忙看下，感谢~ @ywang96
try it.

stop_tokens = ['<|im_end|>', '<|endoftext|>']
stop_token_ids = [tokenizer.convert_tokens_to_ids(i) for i in stop_tokens]
sampling_params = SamplingParams(
            stop_token_ids=stop_token_ids,
)

offline vllm推理 minicpmv2-6 会出现推理结果一直重复输出 某段文字。 vllm ==0.5.4 推理代码：
messages = [{
    'role': 'user',
    'content': f'(<image>./</image>)\n{question}'
}]
prompt = tokenizer.apply_chat_template(messages,
                                       tokenize=False,
                                       add_generation_prompt=True)

stop_token_ids = ['<|eot_id|>']
sampling_params = SamplingParams(temperature=0.7, max_tokens=8192, stop_token_ids=stop_token_ids)

inputs = {
    "prompt": prompt,
    "multi_modal_data": {
        "image": image
    },
}

outputs = llm.generate(inputs, sampling_params=sampling_params)

for o in outputs:
    generated_text = o.outputs[0].text
    print(generated_text)

辛苦帮忙看下，感谢~ @ywang96

Could you share a sample input/output with the repetitive generation?

prompt in Chinese, which probably means producing some classic advertising copy

By the way, how can i use minicpmv2-6's fewshot feature wtih VLLM structure.

Here is minicpmv2-6 infer best practice with VLLM :
from PIL import Image
from transformers import AutoTokenizer
from vllm import LLM, SamplingParams

# 图像文件路径列表
IMAGES = [
    "/root/ld/ld_project/MiniCPM-V/assets/airplane.jpeg",  # 本地图片路径
]

# 模型名称或路径
MODEL_NAME = "/root/ld/ld_model_pretrained/Minicpmv2_6"  # 本地模型路径或Hugging Face模型名称

# 打开并转换图像
image = Image.open(IMAGES[0]).convert("RGB")

# 初始化分词器
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME, trust_remote_code=True)

# 初始化语言模型
llm = LLM(model=MODEL_NAME,
           gpu_memory_utilization=1,  # 使用全部GPU内存
           trust_remote_code=True,
           max_model_len=2048)  # 根据内存状况可调整此值

# 构建对话消息
messages = [{'role': 'user', 'content': '(<image>./</image>)\n' + '请描述这张图片'}]

# 应用对话模板到消息
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

# 设置停止符ID
# 2.0
# stop_token_ids = [tokenizer.eos_id]
# 2.5
#stop_token_ids = [tokenizer.eos_id, tokenizer.eot_id]
# 2.6
stop_tokens = ['<|im_end|>', '<|endoftext|>']
stop_token_ids = [tokenizer.convert_tokens_to_ids(i) for i in stop_tokens]

# 设置生成参数
sampling_params = SamplingParams(
    stop_token_ids=stop_token_ids,
    # temperature=0.7,
    # top_p=0.8,
    # top_k=100,
    # seed=3472,
    max_tokens=1024,
    # min_tokens=150,
    temperature=0,
    use_beam_search=True,
    # length_penalty=1.2,
    best_of=3)

# 获取模型输出
outputs = llm.generate({
    "prompt": prompt,
    "multi_modal_data": {
        "image": image
    }
}, sampling_params=sampling_params)
print(outputs[0].outputs[0].text)

offline vllm推理 minicpmv2-6 会出现推理结果一直重复输出 某段文字。 vllm ==0.5.4 推理代码：
messages = [{
    'role': 'user',
    'content': f'(<image>./</image>)\n{question}'
}]
prompt = tokenizer.apply_chat_template(messages,
                                       tokenize=False,
                                       add_generation_prompt=True)

stop_token_ids = ['<|eot_id|>']
sampling_params = SamplingParams(temperature=0.7, max_tokens=8192, stop_token_ids=stop_token_ids)

inputs = {
    "prompt": prompt,
    "multi_modal_data": {
        "image": image
    },
}

outputs = llm.generate(inputs, sampling_params=sampling_params)

for o in outputs:
    generated_text = o.outputs[0].text
    print(generated_text)

辛苦帮忙看下，感谢~ @ywang96
try it.

stop_tokens = ['<|im_end|>', '<|endoftext|>']
stop_token_ids = [tokenizer.convert_tokens_to_ids(i) for i in stop_tokens]
sampling_params = SamplingParams(
            stop_token_ids=stop_token_ids,
)

Thank u very much, I think problem is each version has different stoptoken-ids. These codes will work, I think.

By the way, how can i use minicpmv2-6's fewshot feature wtih VLLM structure.

here is official fewshot feature usage with transformers:
import torch
from PIL import Image
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True,
    attn_implementation='sdpa', torch_dtype=torch.bfloat16) # sdpa or flash_attention_2, no eager
model = model.eval().cuda()
tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True)

question = "production date"
image1 = Image.open('example1.jpg').convert('RGB')
answer1 = "2023.08.04"
image2 = Image.open('example2.jpg').convert('RGB')
answer2 = "2007.04.24"
image_test = Image.open('test.jpg').convert('RGB')

msgs = [
    {'role': 'user', 'content': [image1, question]}, {'role': 'assistant', 'content': [answer1]},
    {'role': 'user', 'content': [image2, question]}, {'role': 'assistant', 'content': [answer2]},
    {'role': 'user', 'content': [image_test, question]}
]

answer = model.chat(
    image=None,
    msgs=msgs,
    tokenizer=tokenizer
)
print(answer)

By the way, how can i use minicpmv2-6's fewshot feature wtih VLLM structure.

Looking forward your reply~Thank u. @ywang96 @whyiug

在5.4.0版本的vllm中以openai api形式部署minicpm-v-2.6，遇到这个报错，请帮忙看下：
Process Process-1:
Traceback (most recent call last):
  File "/app/apps/anaconda3/envs/vllm_054_cu118/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/app/apps/anaconda3/envs/vllm_054_cu118/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/app/apps/anaconda3/envs/vllm_054_cu118/lib/python3.9/site-packages/vllm/entrypoints/openai/rpc/server.py", line 217, in run_rpc_serve
r
    server = AsyncEngineRPCServer(async_engine_args, usage_context, port)
  File "/app/apps/anaconda3/envs/vllm_054_cu118/lib/python3.9/site-packages/vllm/entrypoints/openai/rpc/server.py", line 25, in __init__
    self.engine = AsyncLLMEngine.from_engine_args(async_engine_args,
  File "/app/apps/anaconda3/envs/vllm_054_cu118/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py", line 471, in from_engine_args
    engine = cls(
  File "/app/apps/anaconda3/envs/vllm_054_cu118/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py", line 381, in __init__
    self.engine = self._init_engine(*args, **kwargs)
  File "/app/apps/anaconda3/envs/vllm_054_cu118/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py", line 552, in _init_engine
    return engine_class(*args, **kwargs)
  File "/app/apps/anaconda3/envs/vllm_054_cu118/lib/python3.9/site-packages/vllm/engine/llm_engine.py", line 249, in __init__
    self.model_executor = executor_class(
  File "/app/apps/anaconda3/envs/vllm_054_cu118/lib/python3.9/site-packages/vllm/executor/executor_base.py", line 47, in __init__
    self._init_executor()
  File "/app/apps/anaconda3/envs/vllm_054_cu118/lib/python3.9/site-packages/vllm/executor/gpu_executor.py", line 35, in _init_executor
    self.driver_worker.init_device()
  File "/app/apps/anaconda3/envs/vllm_054_cu118/lib/python3.9/site-packages/vllm/worker/worker.py", line 123, in init_device
    torch.cuda.set_device(self.device)
  File "/app/apps/anaconda3/envs/vllm_054_cu118/lib/python3.9/site-packages/torch/cuda/__init__.py", line 420, in set_device
    torch._C._cuda_setDevice(device)
  File "/app/apps/anaconda3/envs/vllm_054_cu118/lib/python3.9/site-packages/torch/cuda/__init__.py", line 300, in _lazy_init
    raise RuntimeError(
RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method

在0.5.4版本中推理minicpm-V，会出现out of memory的情况，采用OpenAI格式部署
https://github.com/OpenBMB/MiniCPM-V/issues/392

我在一张A100-80G显卡上面做了测试，发现使用vllm加载时，内存会先到16GB（读取模型），读取完毕后的某一个瞬间，内存会达到29GB的峰值，然后又降低到了19GB。原因不明。

How to load the vision model in a separate gpu to avoid oom？

@sfyumi I have a solution. In default vllm's max-num-seqs default to 256 and it's too large for the 3090, just lower the number to 32 for max-num-seqs and raise gpu-memory-utilization to 1.

By the way, how can i use minicpmv2-6's fewshot feature wtih VLLM structure.

here is official fewshot feature usage with transformers:
import torch
from PIL import Image
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True,
    attn_implementation='sdpa', torch_dtype=torch.bfloat16) # sdpa or flash_attention_2, no eager
model = model.eval().cuda()
tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True)

question = "production date"
image1 = Image.open('example1.jpg').convert('RGB')
answer1 = "2023.08.04"
image2 = Image.open('example2.jpg').convert('RGB')
answer2 = "2007.04.24"
image_test = Image.open('test.jpg').convert('RGB')

msgs = [
    {'role': 'user', 'content': [image1, question]}, {'role': 'assistant', 'content': [answer1]},
    {'role': 'user', 'content': [image2, question]}, {'role': 'assistant', 'content': [answer2]},
    {'role': 'user', 'content': [image_test, question]}
]

answer = model.chat(
    image=None,
    msgs=msgs,
    tokenizer=tokenizer
)
print(answer)

I think this could work
msgs = [
    {'role': 'user', 'content': "(<image>./</image>)" + question}, {'role': 'assistant', 'content': answer1},
    {'role': 'user', 'content': "(<image>./</image>)" + question}, {'role': 'assistant', 'content': answer2},
    {'role': 'user', 'content': "(<image>./</image>)" + question}
]
prompt = tokenizer.apply_chat_template(
    msgs,
    tokenize=False,
    add_generation_prompt=True
)
inputs = {
    "prompt": prompt,
    "multi_modal_data": {
        "image": [image1, image2, image_test]
    },
}

我在一张A100-80G显卡上面做了测试，发现使用vllm加载时，内存会先到16GB（读取模型），读取完毕后的某一个瞬间，内存会达到29GB的峰值，然后又降低到了19GB。原因不明。

vLLM will send dummy data(with multiple dummy images) to the model. Since MiniCPM-V has only a few image tokens, there might be a large number of dummy images which could cause OOM. You can add max_model_len=2048 while initializing LLM.