SpeechLMM v1
Collection
1st generation of SpeechLMM models, capable of ingesting video, audio and text and generate text as output. From the Meetween consortium (meetween.eu) • 12 items • Updated • 1
Model Information
This is the version of meetween/Llama-speechlmm-1.0-l that was fine-tuned for Speech-to-Text Translation.
License: see LICENSE
Model Architecture
Identical to the base model. The model was obtained by training LoRA on the LLM. This repository contains the model weights with LoRA merged into the main weights.
How to Use
Identical to the base model.
Fine-tuning Data
This model has been fine-tuned on the same EuroParl-ST and CoVoST2 speech translation data ({en, fr, it, de, es} → {en, fr, it, de, es}) from the training data of the base model.
Evaluation Results
| DATASET: | CoVoST2 | ACL 60/60 | AVG | |||||
|---|---|---|---|---|---|---|---|---|
| BLEU | en-de | de-en | es-en | fr-en | it-en | en-fr | en-de | |
| SeamlessM4T | - | - | - | - | - | 40.4 | 28.0 | - |
| SpeechLMM_v1.0_L | 31.1 | 36.2 | 41.1 | 39.0 | 32.5 | 29.1 | 27.6 | 33.8 |
| SpeechLMM_v1.0_L_ST | 33.7 | 36.7 | 41.5 | 39.0 | 32.4 | 29.6 | 28.5 | 34.5 |
Framework Versions
- Transformers 4.45.0
- Pytorch 2.3.1+cu124.post2
- Datasets 3.2.0
- Tokenizers 0.20.0
- Downloads last month
- 3
Model tree for meetween/Llama-speechlmm-1.0-l-ST
Base model
meetween/Llama-speechlmm-1.0-l