fix llava/llava next issue when working with AutoProcessor

crash if running following script
import torch
from transformers import AutoProcessor, AutoModelForVision2Seq

from optimum.habana.transformers.modeling_utils import adapt_transformers_to_gaudi

adapt_transformers_to_gaudi()

model_id = "llava-hf/llava-1.5-7b-hf"
#model_id = "llava-hf/llama3-llava-next-8b-hf"
model = AutoModelForVision2Seq.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    low_cpu_mem_usage=True,
)
model = model.to("hpu")
model.generation_config.pad_token_id = model.generation_config.eos_token_id
processor = AutoProcessor.from_pretrained(model_id, padding_side="left")

# Define a chat histiry and use `apply_chat_template` to get correctly formatted prompt
# Each value in "content" has to be a list of dicts with types ("text", "image")
conversation = [
    {

      "role": "user",
      "content": [
          {"type": "text", "text": "What are these?"},
          {"type": "image"},
        ],
    },
]
prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)

image_file = "http://images.cocodataset.org/val2017/000000039769.jpg"
raw_image = Image.open(requests.get(image_file, stream=True).raw)
inputs = processor(images=raw_image, text=prompt, return_tensors='pt')
for t in inputs:
    if torch.is_tensor(inputs[t]):
        inputs[t] = inputs[t].to("hpu")
output = model.generate(**inputs, max_new_tokens=200, do_sample=False)
print(processor.decode(output[0][2:], skip_special_tokens=True))

for llava:
Traceback (most recent call last):
File "/workspace/wangyi/optimum-habana/test.py", line 42, in
output = model.generate(**inputs, max_new_tokens=200, do_sample=False)
File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
File "/workspace/wangyi/optimum-habana/optimum/habana/transformers/generation/utils.py", line 1468, in generate
result = self._sample(
File "/workspace/wangyi/optimum-habana/optimum/habana/transformers/generation/utils.py", line 2449, in _sample
outputs = self(
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
File "/workspace/wangyi/optimum-habana/optimum/habana/transformers/models/llava/modeling_llava.py", line 183, in forward
outputs = self.language_model(
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1847, in _call_impl
return inner()
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1793, in inner
result = forward_call(*args, **kwargs)
File "/workspace/wangyi/optimum-habana/optimum/habana/transformers/models/llama/modeling_llama.py", line 1371, in forward
outputs = self.model(
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1847, in _call_impl
return inner()
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1793, in inner
result = forward_call(*args, **kwargs)
File "/workspace/wangyi/optimum-habana/optimum/habana/transformers/models/llama/modeling_llama.py", line 1226, in forward
htcore.mark_step()
File "/usr/local/lib/python3.10/dist-packages/habana_frameworks/torch/utils/internal.py", line 36, in lazy_wrapper
func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/habana_frameworks/torch/core/step_closure.py", line 71, in mark_step
htcore._mark_step(device_str, sync)
RuntimeError: The expanded size of the tensor (331776) must match the existing size (576) at non-singleton dimension 0.  Target sizes: [331776, 4096].  Tensor sizes: [576, 4096]

for llava_next:
Traceback (most recent call last):
File "/workspace/wangyi/optimum-habana/test.py", line 43, in
output = model.generate(**inputs, max_new_tokens=200, do_sample=False)
File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
File "/workspace/wangyi/optimum-habana/optimum/habana/transformers/generation/utils.py", line 1468, in generate
result = self._sample(
File "/workspace/wangyi/optimum-habana/optimum/habana/transformers/generation/utils.py", line 2440, in _sample
model_inputs = self.prepare_inputs_for_generation(input_ids, **model_kwargs)
File "/workspace/wangyi/optimum-habana/optimum/habana/transformers/models/llava_next/modeling_llava_next.py", line 342, in prepare_inputs_for_generation
self._merge_input_ids_with_image_features(
File "/workspace/wangyi/optimum-habana/optimum/habana/transformers/models/llava_next/modeling_llava_next.py", line 213, in _merge_input_ids_with_image_features
raise ValueError(
ValueError: The input provided to the model are wrong. The number of image tokens is 2340 while the number of image given to the model is 1. This prevents correct indexing and breaks batch generation.

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

see https://github.com/huggingface/transformers/commit/a29eabd0eb1bec9d8ca71048fdca5f7e0465e15d#diff-178e0eb55a6cf5582be26cd37dc04bdef1f03fe17a5ce8d73521e19cca43dc9fR417 that move "expand input" to Autoprocessor

@lkk12014402 @yuanwu2017 please help review


StaticInjectorError[ViewContainerRef]

I successfully create a npm angular module using the generator, however:

I imported the module I created in an NG app
Component is loaded successfully, but a *ngFor (it seems) breaks it all, I'm getting the error StaticInjectorError[ViewContainerRef]

Detailed error:
ERROR Error: StaticInjectorError[ViewContainerRef]:    StaticInjectorError[ViewContainerRef]:      NullInjectorError: No provider for ViewContainerRef!     at _NullInjector.get (webpack-internal:///../../../core/esm5/core.js:1118)     at resolveToken (webpack-internal:///../../../core/esm5/core.js:1406)     at tryResolveToken (webpack-internal:///../../../core/esm5/core.js:1348)     at StaticInjector.get (webpack-internal:///../../../core/esm5/core.js:1219)     at resolveToken (webpack-internal:///../../../core/esm5/core.js:1406)     at tryResolveToken (webpack-internal:///../../../core/esm5/core.js:1348)     at StaticInjector.get (webpack-internal:///../../../core/esm5/core.js:1219)     at resolveNgModuleDep (webpack-internal:///../../../core/esm5/core.js:10780)     at NgModuleRef_.get (webpack-internal:///../../../core/esm5/core.js:12001)     at resolveDep (webpack-internal:///../../../core/esm5/core.js:12497)
Any idea ?

Ok, found out.
In .angular-cli.json add/replace:
"defaults": {     "styleExt": "css",     "component": {},     "build": {       "preserveSymlinks": true     }   }

@skol-pro — Thank you for sharing. What exactly was the issue and which setting solved it? Thanks again!

@jvandemo – Well there is an issue on the angular repository where you can find some more informations !

@skol-pro — Thank you for sharing, much appreciated!


fix: metadata persistence

fix overwriting of the metadata on the 'end' event with null. Otherwise i.e. metadata that is sent with langchain isn't stored in the trace.

Hi @nadworny, thanks for that! But the gh actions are not working if they're ran from a fork (we need to fix that). Could you please make a PR directly from a branch of our repo?

I will create the PR myself with your code, and I've created a ticket to sort out a way for users outside our org to make PRs.
Thanks for your contribution!


Sanitize the referer header

Fixes the problem described in #434

@lscharen - thanks for the pull request :)
The PROXY_REFERER is used in multiple places, and I think we only need to adjust it for when it's used to request a token. So line 649 in your PR:
string uri = tokenServiceUri + "?f=json&request=getToken&referer=" + PROXY_REFERER + "&expiration=60&username=" + su.Username + "&password=" + su.Password;

Could you update the code accordingly?
Up to you if you want to keep the RemoveQueryString() function, but since it's only used once, maybe a function isn't needed...

Thanks. The code has been updated to limit the modification of PROXY_REFERER to just the token request.

Oh, the latest change was tested against my app and worked as expected.


Validate key remove key in SchemaCache::class.

Q
A

Is bugfix?
❌

New feature?
✔

Breaks BC?
❌

Will be fixed in file cache.