Loss is not finite and patience is exhausted

Describe the bug
2023-06-14 13:04:41,744 - speechbrain.core - WARNING - Parameter is not finite: Parameter containing:
tensor([nan, nan, nan,  ..., nan, nan, nan], device='cuda:0',
       requires_grad=True)
2023-06-14 13:04:41,744 - speechbrain.core - WARNING - Parameter is not finite: Parameter containing:
tensor([nan, nan, nan,  ..., nan, nan, nan], device='cuda:0',
       requires_grad=True)
2023-06-14 13:04:41,745 - speechbrain.core - WARNING - Parameter is not finite: Parameter containing:
tensor([[nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        ...,
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan]], device='cuda:0',
       requires_grad=True)
2023-06-14 13:04:41,746 - speechbrain.core - WARNING - Parameter is not finite: Parameter containing:
tensor([[nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        ...,
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan]], device='cuda:0',
       requires_grad=True)
2023-06-14 13:04:41,747 - speechbrain.core - WARNING - Parameter is not finite: Parameter containing:
tensor([nan, nan, nan,  ..., nan, nan, nan], device='cuda:0',
       requires_grad=True)
2023-06-14 13:04:41,747 - speechbrain.core - WARNING - Parameter is not finite: Parameter containing:
tensor([nan, nan, nan,  ..., nan, nan, nan], device='cuda:0',
       requires_grad=True)
2023-06-14 13:04:41,748 - speechbrain.core - ERROR - Exception:
Traceback (most recent call last):
  File "train_with_wav2vec2.py", line 374, in <module>
    valid_loader_kwargs=hparams["dataloader_opts"],
  File "/home/zhengbeida/anaconda3/envs/slurp/lib/python3.7/site-packages/speechbrain/core.py", line 1225, in fit
    self._fit_train(train_set=train_set, epoch=epoch, enable=enable)
  File "/home/zhengbeida/anaconda3/envs/slurp/lib/python3.7/site-packages/speechbrain/core.py", line 1078, in _fit_train
    loss = self.fit_batch(batch)
  File "train_with_wav2vec2.py", line 145, in fit_batch
    if self.check_gradients(loss):
  File "/home/zhengbeida/anaconda3/envs/slurp/lib/python3.7/site-packages/speechbrain/core.py", line 1007, in check_gradients
    "Loss is not finite and patience is exhausted. "
ValueError: Loss is not finite and patience is exhausted. To debug, wrap `fit()` with autograd's `detect_anomaly()`, e.g.

Expected behaviour
After 7 epochs, "Loss is not finite and patience is exhausted." appears, and sometimes it also appears “IndexError: Out of range: piece id is out of range.”
How should these two problems be solved?
To Reproduce
I didn't change the code for now, only the path to save the file and load the dataset.
Versions
SpeechBrain system description
==============================
Python version:
3.7.16 (default, Jan 17 2023, 22:20:44)
[GCC 11.2.0]
==============================
Installed Python packages:
appdirs==1.4.4
attrs==23.1.0
black==19.10b0
certifi @ file:///croot/certifi_1671487769961/work/certifi
cfgv==3.3.1
charset-normalizer==3.1.0
click==8.0.4
distlib==0.3.6
entrypoints==0.3
filelock==3.12.0
flake8==3.7.9
fsspec==2023.1.0
huggingface-hub==0.15.1
HyperPyYAML==1.2.1
identify==2.5.24
idna==3.4
importlib-metadata==6.6.0
joblib==1.2.0
jsonlines==3.1.0
mccabe==0.6.1
more-itertools==9.1.0
nodeenv==1.7.0
numpy==1.21.6
packaging==23.1
pandas==1.3.5
pathspec==0.11.1
Pillow==9.5.0
platformdirs==3.5.1
pluggy==0.13.1
pre-commit==2.21.0
py==1.11.0
pycodestyle==2.5.0
pyflakes==2.1.1
pytest==5.4.1
python-dateutil==2.8.2
pytz==2023.3
PyYAML==6.0
regex==2023.5.5
requests==2.31.0
ruamel.yaml==0.17.28
ruamel.yaml.clib==0.2.7
scipy==1.7.3
sentencepiece==0.1.99
six==1.16.0
speechbrain==0.5.14
tokenizers==0.13.3
toml==0.10.2
torch==1.9.0
torchaudio==0.9.0
torchvision==0.10.0
tqdm==4.65.0
transformers==4.29.2
typed-ast==1.5.4
typing_extensions==4.5.0
urllib3==2.0.2
virtualenv==20.23.0
wcwidth==0.2.6
yamllint==1.23.0
zipp==3.15.0
==============================
Could not get git revision==============================
CUDA version:
10.2

Relevant log output
2023-06-14 13:04:41,746 - speechbrain.core - WARNING - Parameter is not finite: Parameter containing:
tensor([[nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        ...,
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan]], device='cuda:0',
       requires_grad=True)
2023-06-14 13:04:41,747 - speechbrain.core - WARNING - Parameter is not finite: Parameter containing:
tensor([nan, nan, nan,  ..., nan, nan, nan], device='cuda:0',
       requires_grad=True)
2023-06-14 13:04:41,747 - speechbrain.core - WARNING - Parameter is not finite: Parameter containing:
tensor([nan, nan, nan,  ..., nan, nan, nan], device='cuda:0',
       requires_grad=True)
2023-06-14 13:04:41,748 - speechbrain.core - ERROR - Exception:
Traceback (most recent call last):
  File "train_with_wav2vec2.py", line 374, in <module>
    valid_loader_kwargs=hparams["dataloader_opts"],
  File "/home/zhengbeida/anaconda3/envs/slurp/lib/python3.7/site-packages/speechbrain/core.py", line 1225, in fit
    self._fit_train(train_set=train_set, epoch=epoch, enable=enable)
  File "/home/zhengbeida/anaconda3/envs/slurp/lib/python3.7/site-packages/speechbrain/core.py", line 1078, in _fit_train
    loss = self.fit_batch(batch)
  File "train_with_wav2vec2.py", line 145, in fit_batch
    if self.check_gradients(loss):
  File "/home/zhengbeida/anaconda3/envs/slurp/lib/python3.7/site-packages/speechbrain/core.py", line 1007, in check_gradients
    "Loss is not finite and patience is exhausted. "
ValueError: Loss is not finite and patience is exhausted. To debug, wrap `fit()` with autograd's `detect_anomaly()`, e.g.

Additional context
Then the 7th epochs are run, and the subsequent ones can be followed.

Assuming this is the recipe in use: https://github.com/speechbrain/speechbrain/blob/develop/recipes/SLURP/direct/hparams/train_with_wav2vec2.yaml
This error + the decode_ids error could hint at some mismatch between the target vocabulary size and (similar to #1589) but I am not sure why that would be here.
The error points to train_with_wav2vec2.py:88, where predicted_tokens is output by the beam searcher. Maybe @Adel-Moumen has a better idea (feel free to redirect again).
Depending on how this goes we might want to close #2032 as a duplicate as the two are likely to be the same underlying issue.

thanks

Hi @zhengqing187, could you please confirm that the issue is related to this recipe?
https://github.com/speechbrain/speechbrain/blob/develop/recipes/SLURP/direct/hparams/train_with_wav2vec2.yaml
I have yet to be able to reproduce the issue. Currently at epoch 9th, no problem so far:

epoch: 1, lr: 3.00e-04, wave2vec_lr: 1.00e-05 - train loss: 8.40e-01 - valid loss: 7.95e-01, valid CER: 27.19, valid WER: 32.75
epoch: 2, lr: 3.00e-04, wave2vec_lr: 1.00e-05 - train loss: 7.65e-01 - valid loss: 7.79e-01, valid CER: 20.08, valid WER: 24.68
epoch: 3, lr: 3.00e-04, wave2vec_lr: 1.00e-05 - train loss: 7.52e-01 - valid loss: 7.72e-01, valid CER: 16.58, valid WER: 21.19
epoch: 4, lr: 3.00e-04, wave2vec_lr: 1.00e-05 - train loss: 7.46e-01 - valid loss: 7.69e-01, valid CER: 15.97, valid WER: 20.35
epoch: 5, lr: 3.00e-04, wave2vec_lr: 1.00e-05 - train loss: 7.41e-01 - valid loss: 7.67e-01, valid CER: 15.38, valid WER: 19.57
epoch: 6, lr: 3.00e-04, wave2vec_lr: 1.00e-05 - train loss: 7.39e-01 - valid loss: 7.65e-01, valid CER: 15.22, valid WER: 19.00
epoch: 7, lr: 3.00e-04, wave2vec_lr: 1.00e-05 - train loss: 7.37e-01 - valid loss: 7.64e-01, valid CER: 13.32, valid WER: 17.30
epoch: 8, lr: 3.00e-04, wave2vec_lr: 1.00e-05 - train loss: 7.35e-01 - valid loss: 7.65e-01, valid CER: 13.81, valid WER: 17.63

hello
Generated 2023-06-09 from:
/home/.../SLU/24/speechbrain-develop/speechbrain-develop/recipes/SLURP/direct/hparams/train_with_wav2vec2.yaml
yamllint disable

Model: Direct SLU
Encoder: Pre-trained wav2vec2 -> LSTM
Decoder: GRU + beamsearch
Tokens: BPE with unigram
losses: NLL
Training: SLURP
Authors:  Loren Lugosch, Mirco Ravanelli 2020 Boumadane Abdelmoumene, AbdelWahab Heba, Yingzhi Wang 2021
For more wav2vec2/HuBERT results, please see https://arxiv.org/pdf/2111.02735.pdf

Seed needs to be set at top of yaml, before objects with parameters are made
seed: 1986
__set_seed: !apply:torch.manual_seed [1986]
output_folder: /data_home/SLU_path/slurp/results/lstm/1986
save_folder: /data_home/SLU_path/slurp/results/lstm/1986/save
train_log: /data_home/SLU_path/slurp/results/lstm/1986/train_log.txt
Data files
The SLURP dataset will be automatically downloaded in the specified data_folder
data_folder: /data_home/SLU_path/slurp
data_folder_rirs: !ref <data_folder>
train_splits: [train_synthetic, train_real]
csv_train: /data_home/SLU_path/slurp/results/lstm/1986/train-type=direct.csv
csv_valid: /data_home/SLU_path/slurp/results/lstm/1986/devel-type=direct.csv
csv_test: /data_home/SLU_path/slurp/results/lstm/1986/test-type=direct.csv
tokenizer_file: speechbrain/SLU-direct-SLURP-hubert-enc
skip_prep: false
URL for the wav2vec2 model, you can change to benchmark diffrenet models
wav2vec2_hub: facebook/hubert-base-ls960
Training parameters
number_of_epochs: 35
batch_size: 4
lr: 0.0003
lr_wav2vec2: 0.00001
token_type: unigram # ["unigram", "bpe", "char"]
sorting: random
ckpt_interval_minutes: 15 # save checkpoint every N min
#freeze all wav2vec2
freeze_wav2vec2: false
#set to true to freeze the CONV part of the wav2vec2 model
freeze_wav2vec2_conv: true
Model parameters
sample_rate: 16000
emb_size: 128
dec_neurons: 512
output_neurons: 58 # index(eos/bos) = 0
encoder_dim: 768
Decoding parameters
bos_index: 0
eos_index: 0
min_decode_ratio: 0.0
max_decode_ratio: 10.0
slu_beam_size: 80
eos_threshold: 1.5
temperature: 1.25
dataloader_opts:
batch_size: 4
shuffle: true
epoch_counter: &id009 !new:speechbrain.utils.epoch_loop.EpochCounter
limit: 35
Models
wav2vec2: &id001 !new:speechbrain.lobes.models.huggingface_wav2vec.HuggingFaceWav2Vec2
source: facebook/hubert-base-ls960
output_norm: true
freeze: false
save_path: /data_home/SLU_path/slurp/results/lstm/1986/save/wav2vec2_checkpoint
output_emb: &id002 !new:speechbrain.nnet.embedding.Embedding
num_embeddings: 58
embedding_dim: 128
dec: &id003 !new:speechbrain.nnet.RNN.AttentionalRNNDecoder
enc_dim: 768
input_size: 128
rnn_type: lstm
attn_type: content
hidden_size: 512
attn_dim: 512
num_layers: 3
scaling: 1.0
dropout: 0.0
seq_lin: &id004 !new:speechbrain.nnet.linear.Linear
input_size: 512
n_neurons: 58
augmentation: !new:speechbrain.lobes.augment.TimeDomainSpecAugment
sample_rate: 16000
speeds: [95, 100, 105]
modules:
wav2vec2: *id001
output_emb: *id002
dec: *id003
seq_lin: *id004
model: &id006 !new:torch.nn.ModuleList

[*id002, *id003, *id004]
tokenizer: &id005 !new:sentencepiece.SentencePieceProcessor

pretrainer: !new:speechbrain.utils.parameter_transfer.Pretrainer
collect_in: /data_home/SLU_path/slurp/results/lstm/1986/save/SLURP_tokenizer
loadables:
tokenizer: *id005
paths:
tokenizer: speechbrain/SLU-direct-SLURP-hubert-enc/tokenizer_58_unigram.model
beam_searcher: !new:speechbrain.decoders.S2SRNNBeamSearcher
embedding: *id002
decoder: *id003
linear: *id004
bos_index: 0
eos_index: 0
min_decode_ratio: 0.0
max_decode_ratio: 10.0
beam_size: 80
eos_threshold: 1.5
temperature: 1.25
using_max_attn_shift: false
max_attn_shift: 30
coverage_penalty: 0.
opt_class: !name:torch.optim.Adam
lr: 0.0003
wav2vec2_opt_class: !name:torch.optim.Adam
lr: 0.00001
lr_annealing: &id007 !new:speechbrain.nnet.schedulers.NewBobScheduler
initial_value: 0.0003
improvement_threshold: 0.0025
annealing_factor: 0.8
patient: 0
lr_annealing_wav2vec2: &id008 !new:speechbrain.nnet.schedulers.NewBobScheduler
initial_value: 0.00001
improvement_threshold: 0.0025
annealing_factor: 0.9
checkpointer: !new:speechbrain.utils.checkpoints.Checkpointer
checkpoints_dir: /data_home/SLU_path/slurp/results/lstm/1986/save
recoverables:
model: *id006
wav2vec2: *id001
scheduler: *id007
lr_annealing_wav2vec2: *id008
counter: *id009
log_softmax: !new:speechbrain.nnet.activations.Softmax
apply_log: true
seq_cost: !name:speechbrain.nnet.losses.nll_loss
label_smoothing: 0.1
train_logger: !new:speechbrain.utils.train_logger.FileTrainLogger
save_file: /data_home/SLU_path/slurp/results/lstm/1986/train_log.txt
error_rate_computer: !name:speechbrain.utils.metric_stats.ErrorRateStats
cer_computer: !name:speechbrain.utils.metric_stats.ErrorRateStats
split_tokens: true
This is the train_with_wav3vec.yaml configuration of my reproduction experiment, the device is 3090 and it has this problem somewhat randomly. I'll share my log.txt file with you as well.

hello,@mhn226,This is the log entry generated, it generated IndexError: Out of range: piece id is out of range at epochs 17.
2023-06-07 12:31:04,228 - speechbrain.dataio.dataio - INFO - Skipping merging. Completed in previous run.
2023-06-07 12:31:04,343 - speechbrain.dataio.dataio - INFO - /data_home/SLU_path/slurp/results/lstm/1986/train-type=direct.csv is created.
2023-06-07 12:31:04,881 - speechbrain.utils.parameter_transfer - DEBUG - Collecting files (or symlinks) for pretraining in /data_home/SLU_path/slurp/results/lstm/1986/save/SLURP_tokenizer.
2023-06-07 12:31:04,882 - speechbrain.pretrained.fetching - INFO - Fetch tokenizer_58_unigram.model: Using existing file/symlink in /data_home/SLU_path/slurp/results/lstm/1986/save/SLURP_tokenizer/tokenizer.ckpt.
2023-06-07 12:31:04,882 - speechbrain.utils.parameter_transfer - INFO - Loading pretrained files for: tokenizer
2023-06-07 12:31:05,566 - speechbrain.core - INFO - Info: ckpt_interval_minutes arg from hparam file is used
2023-06-07 12:31:05,571 - speechbrain.core - INFO - 98.3M trainable parameters in SLU
2023-06-07 12:31:05,571 - speechbrain.utils.checkpoints - INFO - Would load a checkpoint here, but none found yet.
2023-06-07 12:31:05,571 - speechbrain.utils.epoch_loop - INFO - Going into epoch 1
2023-06-07 12:46:06,350 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+12-46-05+00
2023-06-07 13:01:06,988 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+13-01-06+00
2023-06-07 13:01:07,043 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+12-46-05+00
2023-06-07 13:16:07,751 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+13-16-07+00
2023-06-07 13:16:07,807 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+13-01-06+00
2023-06-07 13:31:08,526 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+13-31-07+00
2023-06-07 13:31:08,582 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+13-16-07+00
2023-06-07 13:50:48,361 - speechbrain.utils.train_logger - INFO - epoch: 1, lr: 3.00e-04, wave2vec_lr: 1.00e-05 - train loss: 8.29e-01 - valid loss: 7.92e-01, valid CER: 24.83, valid WER: 30.62
2023-06-07 13:50:48,995 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+13-50-48+00
2023-06-07 13:50:49,054 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+13-31-07+00
2023-06-07 13:50:49,054 - speechbrain.utils.epoch_loop - INFO - Going into epoch 2
2023-06-07 14:05:49,737 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+14-05-49+00
2023-06-07 14:20:50,408 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+14-20-49+00
2023-06-07 14:20:50,470 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+14-05-49+00
2023-06-07 14:35:51,167 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+14-35-50+00
2023-06-07 14:35:51,230 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+14-20-49+00
2023-06-07 14:50:51,914 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+14-50-51+00
2023-06-07 14:50:51,978 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+14-35-50+00
2023-06-07 15:10:17,645 - speechbrain.utils.train_logger - INFO - epoch: 2, lr: 3.00e-04, wave2vec_lr: 1.00e-05 - train loss: 7.63e-01 - valid loss: 7.78e-01, valid CER: 18.75, valid WER: 23.41
2023-06-07 15:10:18,334 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+15-10-17+00
2023-06-07 15:10:18,401 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+14-50-51+00
2023-06-07 15:10:18,455 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+13-50-48+00
2023-06-07 15:10:18,455 - speechbrain.utils.epoch_loop - INFO - Going into epoch 3
2023-06-07 15:25:19,245 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+15-25-18+00
2023-06-07 15:40:20,016 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+15-40-19+00
2023-06-07 15:40:20,115 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+15-25-18+00
2023-06-07 15:55:20,868 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+15-55-20+00
2023-06-07 15:55:20,969 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+15-40-19+00
2023-06-07 16:10:21,766 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+16-10-21+00
2023-06-07 16:10:21,859 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+15-55-20+00
2023-06-07 16:25:22,626 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+16-25-21+00
2023-06-07 16:25:22,723 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+16-10-21+00
2023-06-07 16:40:17,788 - speechbrain.utils.train_logger - INFO - epoch: 3, lr: 3.00e-04, wave2vec_lr: 1.00e-05 - train loss: 7.51e-01 - valid loss: 7.71e-01, valid CER: 17.12, valid WER: 21.80
2023-06-07 16:40:18,453 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+16-40-17+00
2023-06-07 16:40:18,559 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+15-10-17+00
2023-06-07 16:40:18,646 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+16-25-21+00
2023-06-07 16:40:18,646 - speechbrain.utils.epoch_loop - INFO - Going into epoch 4
2023-06-07 16:57:39,998 - speechbrain.core - INFO - Beginning experiment!
2023-06-07 16:57:39,999 - speechbrain.core - INFO - Experiment folder: /data_home/SLU_path/slurp/results/lstm/1986
2023-06-07 16:57:40,195 - speechbrain.utils.superpowers - DEBUG - appdirs==1.4.4
attrs==23.1.0
black==19.10b0
certifi==2023.5.7
cfgv==3.3.1
charset-normalizer==3.1.0
click==8.0.4
distlib==0.3.6
entrypoints==0.3
filelock==3.12.0
flake8==3.7.9
fsspec==2023.1.0
huggingface-hub==0.14.1
HyperPyYAML==1.2.0
identify==2.5.24
idna==3.4
importlib-metadata==6.6.0
joblib==1.2.0
jsonlines==3.1.0
mccabe==0.6.1
more-itertools==9.1.0
nodeenv==1.7.0
numpy==1.21.6
nvidia-cublas-cu11==<IP_ADDRESS>
nvidia-cuda-nvrtc-cu11==11.7.99
nvidia-cuda-runtime-cu11==11.7.99
nvidia-cudnn-cu11==<IP_ADDRESS>
packaging==23.1
pandas==1.3.5
pathspec==0.11.1
Pillow==9.5.0
platformdirs==3.5.1
pluggy==0.13.1
pre-commit==2.21.0
py==1.11.0
pycodestyle==2.5.0
pyflakes==2.1.1
pytest==5.4.1
python-dateutil==2.8.2
pytz==2023.3
PyYAML==6.0
regex==2023.5.5
requests==2.30.0
ruamel.yaml==0.17.26
ruamel.yaml.clib==0.2.7
scipy==1.7.3
sentencepiece==0.1.99
six==1.16.0
speechbrain==0.5.14
tokenizers==0.13.3
toml==0.10.2
torch==1.13.1
torchaudio==0.13.1
torchvision==0.14.1
tqdm==4.65.0
transformers==4.29.2
typed-ast==1.5.4
typing_extensions==4.5.0
urllib3==2.0.2
virtualenv==20.23.0
wcwidth==0.2.6
yamllint==1.23.0
zipp==3.15.0
2023-06-07 16:57:40,246 - speechbrain.dataio.dataio - INFO - Skipping merging. Completed in previous run.
2023-06-07 16:57:40,317 - speechbrain.dataio.dataio - INFO - /data_home/SLU_path/slurp/results/lstm/1986/train-type=direct.csv is created.
2023-06-07 16:57:40,847 - speechbrain.utils.parameter_transfer - DEBUG - Collecting files (or symlinks) for pretraining in /data_home/SLU_path/slurp/results/lstm/1986/save/SLURP_tokenizer.
2023-06-07 16:57:40,847 - speechbrain.pretrained.fetching - INFO - Fetch tokenizer_58_unigram.model: Using existing file/symlink in /data_home/SLU_path/slurp/results/lstm/1986/save/SLURP_tokenizer/tokenizer.ckpt.
2023-06-07 16:57:40,847 - speechbrain.utils.parameter_transfer - INFO - Loading pretrained files for: tokenizer
2023-06-07 16:57:41,524 - speechbrain.core - INFO - Info: ckpt_interval_minutes arg from hparam file is used
2023-06-07 16:57:41,528 - speechbrain.core - INFO - 98.3M trainable parameters in SLU
2023-06-07 16:57:41,529 - speechbrain.utils.checkpoints - INFO - Loading a checkpoint from /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+16-40-17+00
2023-06-07 16:57:42,113 - speechbrain.utils.epoch_loop - INFO - Going into epoch 4
2023-06-07 17:12:42,926 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+17-12-42+00
2023-06-07 17:27:43,635 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+17-27-42+00
2023-06-07 17:27:43,722 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+17-12-42+00
2023-06-07 17:42:44,461 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+17-42-43+00
2023-06-07 17:42:44,525 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+17-27-42+00
2023-06-07 17:57:45,219 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+17-57-44+00
2023-06-07 17:57:45,303 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+17-42-43+00
2023-06-07 18:20:07,089 - speechbrain.utils.train_logger - INFO - epoch: 4, lr: 3.00e-04, wave2vec_lr: 1.00e-05 - train loss: 7.46e-01 - valid loss: 7.69e-01, valid CER: 14.98, valid WER: 19.49
2023-06-07 18:20:07,759 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+18-20-07+00
2023-06-07 18:20:07,844 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+16-40-17+00
2023-06-07 18:20:07,923 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+17-57-44+00
2023-06-07 18:20:07,923 - speechbrain.utils.epoch_loop - INFO - Going into epoch 5
2023-06-07 18:35:08,633 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+18-35-07+00
2023-06-07 18:50:09,430 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+18-50-08+00
2023-06-07 18:50:09,523 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+18-35-07+00
2023-06-07 19:05:10,273 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+19-05-09+00
2023-06-07 19:05:10,353 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+18-50-08+00
2023-06-07 19:20:11,076 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+19-20-10+00
2023-06-07 19:20:11,171 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+19-05-09+00
2023-06-07 19:44:51,060 - speechbrain.utils.train_logger - INFO - epoch: 5, lr: 3.00e-04, wave2vec_lr: 1.00e-05 - train loss: 7.42e-01 - valid loss: 7.66e-01, valid CER: 15.05, valid WER: 19.25
2023-06-07 19:44:51,736 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+19-44-51+00
2023-06-07 19:44:51,786 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+18-20-07+00
2023-06-07 19:44:51,857 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+19-20-10+00
2023-06-07 19:44:51,857 - speechbrain.utils.epoch_loop - INFO - Going into epoch 6
2023-06-07 19:59:52,646 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+19-59-51+00
2023-06-07 20:14:53,352 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+20-14-52+00
2023-06-07 20:14:53,448 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+19-59-51+00
2023-06-07 20:29:54,136 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+20-29-53+00
2023-06-07 20:29:54,257 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+20-14-52+00
2023-06-07 20:44:55,046 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+20-44-54+00
2023-06-07 20:44:55,146 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+20-29-53+00
2023-06-07 20:59:55,840 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+20-59-55+00
2023-06-07 20:59:55,936 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+20-44-54+00
2023-06-07 21:12:20,698 - speechbrain.utils.train_logger - INFO - epoch: 6, lr: 3.00e-04, wave2vec_lr: 1.00e-05 - train loss: 7.39e-01 - valid loss: 7.65e-01, valid CER: 15.03, valid WER: 18.93
2023-06-07 21:12:21,376 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+21-12-20+00
2023-06-07 21:12:21,476 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+20-59-55+00
2023-06-07 21:12:21,528 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+19-44-51+00
2023-06-07 21:12:21,528 - speechbrain.utils.epoch_loop - INFO - Going into epoch 7
2023-06-07 21:27:22,256 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+21-27-21+00
2023-06-07 21:42:22,969 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+21-42-22+00
2023-06-07 21:42:23,070 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+21-27-21+00
2023-06-07 21:57:23,800 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+21-57-23+00
2023-06-07 21:57:23,894 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+21-42-22+00
2023-06-07 22:12:24,579 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+22-12-23+00
2023-06-07 22:12:24,683 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+21-57-23+00
2023-06-07 22:35:23,542 - speechbrain.utils.train_logger - INFO - epoch: 7, lr: 3.00e-04, wave2vec_lr: 1.00e-05 - train loss: 7.38e-01 - valid loss: 7.63e-01, valid CER: 13.80, valid WER: 17.84
2023-06-07 22:35:24,212 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+22-35-23+00
2023-06-07 22:35:24,313 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+21-12-20+00
2023-06-07 22:35:24,389 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+22-12-23+00
2023-06-07 22:35:24,389 - speechbrain.utils.epoch_loop - INFO - Going into epoch 8
2023-06-07 22:50:25,181 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+22-50-24+00
2023-06-07 23:05:25,964 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+23-05-25+00
2023-06-07 23:05:26,069 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+22-50-24+00
2023-06-07 23:20:26,864 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+23-20-26+00
2023-06-07 23:20:26,969 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+23-05-25+00
2023-06-07 23:35:27,777 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+23-35-27+00
2023-06-07 23:35:27,884 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+23-20-26+00
2023-06-07 23:59:11,368 - speechbrain.nnet.schedulers - INFO - Changing lr from 0.0003 to 0.00024
2023-06-07 23:59:11,368 - speechbrain.nnet.schedulers - INFO - Changing lr from 1e-05 to 9e-06
2023-06-07 23:59:11,368 - speechbrain.utils.train_logger - INFO - epoch: 8, lr: 3.00e-04, wave2vec_lr: 1.00e-05 - train loss: 7.36e-01 - valid loss: 7.65e-01, valid CER: 14.08, valid WER: 18.07
2023-06-07 23:59:12,031 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+23-59-11+00
2023-06-07 23:59:12,131 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+23-35-27+00
2023-06-07 23:59:12,131 - speechbrain.utils.epoch_loop - INFO - Going into epoch 9
2023-06-08 00:14:12,890 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+00-14-12+00
2023-06-08 00:29:13,644 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+00-29-12+00
2023-06-08 00:29:13,758 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+00-14-12+00
2023-06-08 00:44:14,449 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+00-44-13+00
2023-06-08 00:44:14,572 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+00-29-12+00
2023-06-08 00:59:15,252 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+00-59-14+00
2023-06-08 00:59:15,372 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+00-44-13+00
2023-06-08 01:21:34,391 - speechbrain.utils.train_logger - INFO - epoch: 9, lr: 2.40e-04, wave2vec_lr: 9.00e-06 - train loss: 7.34e-01 - valid loss: 7.63e-01, valid CER: 13.03, valid WER: 17.01
2023-06-08 01:21:35,051 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+01-21-34+00
2023-06-08 01:21:35,172 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+23-59-11+00
2023-06-08 01:21:35,239 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+00-59-14+00
2023-06-08 01:21:35,307 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-07+22-35-23+00
2023-06-08 01:21:35,307 - speechbrain.utils.epoch_loop - INFO - Going into epoch 10
2023-06-08 01:36:36,113 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+01-36-35+00
2023-06-08 01:51:36,894 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+01-51-36+00
2023-06-08 01:51:37,007 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+01-36-35+00
2023-06-08 02:06:37,668 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+02-06-37+00
2023-06-08 02:06:37,786 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+01-51-36+00
2023-06-08 02:21:38,500 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+02-21-37+00
2023-06-08 02:21:38,597 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+02-06-37+00
2023-06-08 02:44:40,519 - speechbrain.utils.train_logger - INFO - epoch: 10, lr: 2.40e-04, wave2vec_lr: 9.00e-06 - train loss: 7.32e-01 - valid loss: 7.61e-01, valid CER: 12.41, valid WER: 16.22
2023-06-08 02:44:41,161 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+02-44-40+00
2023-06-08 02:44:41,260 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+01-21-34+00
2023-06-08 02:44:41,329 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+02-21-37+00
2023-06-08 02:44:41,329 - speechbrain.utils.epoch_loop - INFO - Going into epoch 11
2023-06-08 02:59:42,051 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+02-59-41+00
2023-06-08 03:14:42,771 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+03-14-42+00
2023-06-08 03:14:42,890 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+02-59-41+00
2023-06-08 03:29:43,707 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+03-29-43+00
2023-06-08 03:29:43,813 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+03-14-42+00
2023-06-08 03:44:44,527 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+03-44-43+00
2023-06-08 03:44:44,655 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+03-29-43+00
2023-06-08 04:07:47,588 - speechbrain.nnet.schedulers - INFO - Changing lr from 0.00024 to 0.00019
2023-06-08 04:07:47,588 - speechbrain.nnet.schedulers - INFO - Changing lr from 9e-06 to 8.1e-06
2023-06-08 04:07:47,588 - speechbrain.utils.train_logger - INFO - epoch: 11, lr: 2.40e-04, wave2vec_lr: 9.00e-06 - train loss: 7.32e-01 - valid loss: 7.62e-01, valid CER: 12.43, valid WER: 16.36
2023-06-08 04:07:48,246 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+04-07-47+00
2023-06-08 04:07:48,349 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+03-44-43+00
2023-06-08 04:07:48,350 - speechbrain.utils.epoch_loop - INFO - Going into epoch 12
2023-06-08 04:22:49,092 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+04-22-48+00
2023-06-08 04:37:49,798 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+04-37-49+00
2023-06-08 04:37:49,920 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+04-22-48+00
2023-06-08 04:52:50,586 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+04-52-49+00
2023-06-08 04:52:50,715 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+04-37-49+00
2023-06-08 05:07:51,520 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+05-07-50+00
2023-06-08 05:07:51,645 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+04-52-49+00
2023-06-08 05:31:27,664 - speechbrain.utils.train_logger - INFO - epoch: 12, lr: 1.92e-04, wave2vec_lr: 8.10e-06 - train loss: 7.30e-01 - valid loss: 7.59e-01, valid CER: 11.75, valid WER: 15.66
2023-06-08 05:31:28,312 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+05-31-27+00
2023-06-08 05:31:28,445 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+05-07-50+00
2023-06-08 05:31:28,465 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+02-44-40+00
2023-06-08 05:31:28,535 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+04-07-47+00
2023-06-08 05:31:28,536 - speechbrain.utils.epoch_loop - INFO - Going into epoch 13
2023-06-08 05:46:29,329 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+05-46-28+00
2023-06-08 06:01:30,099 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+06-01-29+00
2023-06-08 06:01:30,229 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+05-46-28+00
2023-06-08 06:16:31,067 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+06-16-30+00
2023-06-08 06:16:31,187 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+06-01-29+00
2023-06-08 06:31:31,866 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+06-31-31+00
2023-06-08 06:31:31,988 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+06-16-30+00
2023-06-08 06:55:04,249 - speechbrain.utils.train_logger - INFO - epoch: 13, lr: 1.92e-04, wave2vec_lr: 8.10e-06 - train loss: 7.29e-01 - valid loss: 7.59e-01, valid CER: 11.49, valid WER: 15.26
2023-06-08 06:55:04,897 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+06-55-04+00
2023-06-08 06:55:05,020 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+06-31-31+00
2023-06-08 06:55:05,059 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+05-31-27+00
2023-06-08 06:55:05,060 - speechbrain.utils.epoch_loop - INFO - Going into epoch 14
2023-06-08 07:10:05,729 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+07-10-05+00
2023-06-08 07:25:06,491 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+07-25-05+00
2023-06-08 07:25:06,618 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+07-10-05+00
2023-06-08 07:40:07,392 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+07-40-06+00
2023-06-08 07:40:07,509 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+07-25-05+00
2023-06-08 07:55:08,239 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+07-55-07+00
2023-06-08 07:55:08,366 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+07-40-06+00
2023-06-08 08:18:43,902 - speechbrain.nnet.schedulers - INFO - Changing lr from 0.00019 to 0.00015
2023-06-08 08:18:43,916 - speechbrain.nnet.schedulers - INFO - Changing lr from 8.1e-06 to 7.3e-06
2023-06-08 08:18:43,916 - speechbrain.utils.train_logger - INFO - epoch: 14, lr: 1.92e-04, wave2vec_lr: 8.10e-06 - train loss: 7.29e-01 - valid loss: 7.59e-01, valid CER: 11.65, valid WER: 15.44
2023-06-08 08:18:44,573 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+08-18-43+00
2023-06-08 08:18:44,695 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+07-55-07+00
2023-06-08 08:18:44,695 - speechbrain.utils.epoch_loop - INFO - Going into epoch 15
2023-06-08 08:33:45,405 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+08-33-44+00
2023-06-08 08:48:46,355 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+08-48-45+00
2023-06-08 08:48:46,509 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+08-33-44+00
2023-06-08 09:03:47,256 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+09-03-46+00
2023-06-08 09:03:47,415 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+08-48-45+00
2023-06-08 09:18:48,090 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+09-18-47+00
2023-06-08 09:18:48,247 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+09-03-46+00
2023-06-08 09:42:25,733 - speechbrain.utils.train_logger - INFO - epoch: 15, lr: 1.54e-04, wave2vec_lr: 7.29e-06 - train loss: 7.27e-01 - valid loss: 7.58e-01, valid CER: 11.31, valid WER: 14.86
2023-06-08 09:42:26,389 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+09-42-25+00
2023-06-08 09:42:26,554 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+08-18-43+00
2023-06-08 09:42:26,620 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+09-18-47+00
2023-06-08 09:42:26,670 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+06-55-04+00
2023-06-08 09:42:26,670 - speechbrain.utils.epoch_loop - INFO - Going into epoch 16
2023-06-08 09:57:27,403 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+09-57-26+00
2023-06-08 10:12:28,180 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+10-12-27+00
2023-06-08 10:12:28,333 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+09-57-26+00
2023-06-08 10:27:29,023 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+10-27-28+00
2023-06-08 10:27:29,171 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+10-12-27+00
2023-06-08 10:42:29,889 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+10-42-29+00
2023-06-08 10:42:30,046 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+10-27-28+00
2023-06-08 11:04:32,944 - speechbrain.utils.train_logger - INFO - epoch: 16, lr: 1.54e-04, wave2vec_lr: 7.29e-06 - train loss: 7.27e-01 - valid loss: 7.58e-01, valid CER: 11.11, valid WER: 14.75
2023-06-08 11:04:33,623 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+11-04-32+00
2023-06-08 11:04:33,766 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+09-42-25+00
2023-06-08 11:04:33,838 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+10-42-29+00
2023-06-08 11:04:33,838 - speechbrain.utils.epoch_loop - INFO - Going into epoch 17
2023-06-08 11:19:34,629 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+11-19-33+00
2023-06-08 11:34:35,451 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+11-34-34+00
2023-06-08 11:34:35,606 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+11-19-33+00
2023-06-08 11:49:36,393 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+11-49-35+00
2023-06-08 11:49:36,549 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+11-34-34+00
2023-06-08 12:04:37,269 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+12-04-36+00
2023-06-08 12:04:37,427 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+11-49-35+00
2023-06-08 12:27:03,616 - speechbrain.core - ERROR - Exception:
Traceback (most recent call last):
File "train_with_wav2vec2.py", line 379, in
valid_loader_kwargs=hparams["dataloader_opts"],
File "/home/zhaochen/anaconda3/envs/SLU/lib/python3.7/site-packages/speechbrain/core.py", line 1226, in fit
self._fit_valid(valid_set=valid_set, epoch=epoch, enable=enable)
File "/home/zhaochen/anaconda3/envs/SLU/lib/python3.7/site-packages/speechbrain/core.py", line 1129, in _fit_valid
loss = self.evaluate_batch(batch, stage=Stage.VALID)
File "train_with_wav2vec2.py", line 161, in evaluate_batch
loss = self.compute_objectives(predictions, batch, stage=stage)
File "train_with_wav2vec2.py", line 94, in compute_objectives
for utt_seq in predicted_tokens
File "train_with_wav2vec2.py", line 94, in
for utt_seq in predicted_tokens
File "/home/zhaochen/anaconda3/envs/SLU/lib/python3.7/site-packages/sentencepiece/init.py", line 837, in DecodeIds
return self.Decode(input=input, out_type=out_type, **kwargs)
File "/home/zhaochen/anaconda3/envs/SLU/lib/python3.7/site-packages/sentencepiece/init.py", line 780, in Decode
return self._DecodeIds(input)
File "/home/zhaochen/anaconda3/envs/SLU/lib/python3.7/site-packages/sentencepiece/init.py", line 337, in _DecodeIds
return _sentencepiece.SentencePieceProcessor__DecodeIds(self, ids)
IndexError: Out of range: piece id is out of range.
2023-06-08 13:52:20,573 - speechbrain.core - INFO - Beginning experiment!
2023-06-08 13:52:20,573 - speechbrain.core - INFO - Experiment folder: /data_home/SLU_path/slurp/results/lstm/1986
2023-06-08 13:52:20,753 - speechbrain.utils.superpowers - DEBUG - appdirs==1.4.4
attrs==23.1.0
black==19.10b0
certifi==2023.5.7
cfgv==3.3.1
charset-normalizer==3.1.0
click==8.0.4
distlib==0.3.6
entrypoints==0.3
filelock==3.12.0
flake8==3.7.9
fsspec==2023.1.0
huggingface-hub==0.14.1
HyperPyYAML==1.2.0
identify==2.5.24
idna==3.4
importlib-metadata==6.6.0
joblib==1.2.0
jsonlines==3.1.0
mccabe==0.6.1
more-itertools==9.1.0
nodeenv==1.7.0
numpy==1.21.6
nvidia-cublas-cu11==<IP_ADDRESS>
nvidia-cuda-nvrtc-cu11==11.7.99
nvidia-cuda-runtime-cu11==11.7.99
nvidia-cudnn-cu11==<IP_ADDRESS>
packaging==23.1
pandas==1.3.5
pathspec==0.11.1
Pillow==9.5.0
platformdirs==3.5.1
pluggy==0.13.1
pre-commit==2.21.0
py==1.11.0
pycodestyle==2.5.0
pyflakes==2.1.1
pytest==5.4.1
python-dateutil==2.8.2
pytz==2023.3
PyYAML==6.0
regex==2023.5.5
requests==2.30.0
ruamel.yaml==0.17.26
ruamel.yaml.clib==0.2.7
scipy==1.7.3
sentencepiece==0.1.99
six==1.16.0
speechbrain==0.5.14
tokenizers==0.13.3
toml==0.10.2
torch==1.13.1
torchaudio==0.13.1
torchvision==0.14.1
tqdm==4.65.0
transformers==4.29.2
typed-ast==1.5.4
typing_extensions==4.5.0
urllib3==2.0.2
virtualenv==20.23.0
wcwidth==0.2.6
yamllint==1.23.0
zipp==3.15.0
2023-06-08 13:52:20,783 - speechbrain.dataio.dataio - INFO - Skipping merging. Completed in previous run.
2023-06-08 13:52:21,499 - speechbrain.dataio.dataio - INFO - /data_home/SLU_path/slurp/results/lstm/1986/train-type=direct.csv is created.
2023-06-08 13:52:22,095 - speechbrain.utils.parameter_transfer - DEBUG - Collecting files (or symlinks) for pretraining in /data_home/SLU_path/slurp/results/lstm/1986/save/SLURP_tokenizer.
2023-06-08 13:52:22,095 - speechbrain.pretrained.fetching - INFO - Fetch tokenizer_58_unigram.model: Using existing file/symlink in /data_home/SLU_path/slurp/results/lstm/1986/save/SLURP_tokenizer/tokenizer.ckpt.
2023-06-08 13:52:22,095 - speechbrain.utils.parameter_transfer - INFO - Loading pretrained files for: tokenizer
2023-06-08 13:52:22,803 - speechbrain.core - INFO - Info: ckpt_interval_minutes arg from hparam file is used
2023-06-08 13:52:22,808 - speechbrain.core - INFO - 98.3M trainable parameters in SLU
2023-06-08 13:52:22,810 - speechbrain.utils.checkpoints - INFO - Loading a checkpoint from /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+12-04-36+00
2023-06-08 13:52:23,373 - speechbrain.utils.epoch_loop - INFO - Going into epoch 17
2023-06-08 14:14:37,263 - speechbrain.utils.train_logger - INFO - epoch: 17, lr: 1.54e-04, wave2vec_lr: 7.29e-06 - train loss: 7.26e-01 - valid loss: 7.58e-01, valid CER: 11.11, valid WER: 14.69
2023-06-08 14:14:37,879 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+14-14-37+00
2023-06-08 14:14:37,967 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+11-04-32+00
2023-06-08 14:14:38,047 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+12-04-36+00
2023-06-08 14:14:38,047 - speechbrain.utils.epoch_loop - INFO - Going into epoch 18
2023-06-08 14:29:38,837 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+14-29-38+00
2023-06-08 14:44:39,725 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+14-44-39+00
2023-06-08 14:44:39,804 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+14-29-38+00
2023-06-08 14:59:40,554 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+14-59-39+00
2023-06-08 14:59:40,636 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+14-44-39+00
2023-06-08 15:14:41,390 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+15-14-40+00
2023-06-08 15:14:41,476 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+14-59-39+00
2023-06-08 15:36:26,521 - speechbrain.utils.train_logger - INFO - epoch: 18, lr: 1.54e-04, wave2vec_lr: 7.29e-06 - train loss: 7.26e-01 - valid loss: 7.58e-01, valid CER: 10.84, valid WER: 14.31
2023-06-08 15:36:27,190 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+15-36-26+00
2023-06-08 15:36:27,278 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+15-14-40+00
2023-06-08 15:36:27,345 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+14-14-37+00
2023-06-08 15:36:27,345 - speechbrain.utils.epoch_loop - INFO - Going into epoch 19
2023-06-08 15:51:28,080 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+15-51-27+00
2023-06-08 16:06:28,778 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+16-06-28+00
2023-06-08 16:06:28,884 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+15-51-27+00
2023-06-08 16:21:29,717 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+16-21-29+00
2023-06-08 16:21:29,808 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+16-06-28+00
2023-06-08 16:36:30,502 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+16-36-29+00
2023-06-08 16:36:30,596 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+16-21-29+00
2023-06-08 16:58:39,857 - speechbrain.nnet.schedulers - INFO - Changing lr from 0.00015 to 0.00012
2023-06-08 16:58:39,857 - speechbrain.nnet.schedulers - INFO - Changing lr from 7.3e-06 to 6.6e-06
2023-06-08 16:58:39,857 - speechbrain.utils.train_logger - INFO - epoch: 19, lr: 1.54e-04, wave2vec_lr: 7.29e-06 - train loss: 7.26e-01 - valid loss: 7.58e-01, valid CER: 10.85, valid WER: 14.48
2023-06-08 16:58:40,531 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+16-58-39+00
2023-06-08 16:58:40,637 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+16-36-29+00
2023-06-08 16:58:40,637 - speechbrain.utils.epoch_loop - INFO - Going into epoch 20
2023-06-08 17:13:41,337 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+17-13-40+00
2023-06-08 17:28:42,128 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+17-28-41+00
2023-06-08 17:28:42,228 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+17-13-40+00
2023-06-08 17:43:42,947 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+17-43-42+00
2023-06-08 17:43:43,038 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+17-28-41+00
2023-06-08 17:58:43,943 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+17-58-43+00
2023-06-08 17:58:44,045 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+17-43-42+00
2023-06-08 18:14:25,403 - speechbrain.core - ERROR - Exception:
Traceback (most recent call last):
File "train_with_wav2vec2.py", line 379, in
valid_loader_kwargs=hparams["dataloader_opts"],
File "/home/zhaochen/anaconda3/envs/SLU/lib/python3.7/site-packages/speechbrain/core.py", line 1226, in fit
self._fit_valid(valid_set=valid_set, epoch=epoch, enable=enable)
File "/home/zhaochen/anaconda3/envs/SLU/lib/python3.7/site-packages/speechbrain/core.py", line 1129, in _fit_valid
loss = self.evaluate_batch(batch, stage=Stage.VALID)
File "train_with_wav2vec2.py", line 161, in evaluate_batch
loss = self.compute_objectives(predictions, batch, stage=stage)
File "train_with_wav2vec2.py", line 94, in compute_objectives
for utt_seq in predicted_tokens
File "train_with_wav2vec2.py", line 94, in
for utt_seq in predicted_tokens
File "/home/zhaochen/anaconda3/envs/SLU/lib/python3.7/site-packages/sentencepiece/init.py", line 837, in DecodeIds
return self.Decode(input=input, out_type=out_type, **kwargs)
File "/home/zhaochen/anaconda3/envs/SLU/lib/python3.7/site-packages/sentencepiece/init.py", line 780, in Decode
return self._DecodeIds(input)
File "/home/zhaochen/anaconda3/envs/SLU/lib/python3.7/site-packages/sentencepiece/init.py", line 337, in _DecodeIds
return _sentencepiece.SentencePieceProcessor__DecodeIds(self, ids)
IndexError: Out of range: piece id is out of range.
2023-06-08 19:35:21,895 - speechbrain.core - INFO - Beginning experiment!
2023-06-08 19:35:21,895 - speechbrain.core - INFO - Experiment folder: /data_home/SLU_path/slurp/results/lstm/1986
2023-06-08 19:35:22,070 - speechbrain.utils.superpowers - DEBUG - appdirs==1.4.4
attrs==23.1.0
black==19.10b0
certifi==2023.5.7
cfgv==3.3.1
charset-normalizer==3.1.0
click==8.0.4
distlib==0.3.6
entrypoints==0.3
filelock==3.12.0
flake8==3.7.9
fsspec==2023.1.0
huggingface-hub==0.14.1
HyperPyYAML==1.2.0
identify==2.5.24
idna==3.4
importlib-metadata==6.6.0
joblib==1.2.0
jsonlines==3.1.0
mccabe==0.6.1
more-itertools==9.1.0
nodeenv==1.7.0
numpy==1.21.6
nvidia-cublas-cu11==<IP_ADDRESS>
nvidia-cuda-nvrtc-cu11==11.7.99
nvidia-cuda-runtime-cu11==11.7.99
nvidia-cudnn-cu11==<IP_ADDRESS>
packaging==23.1
pandas==1.3.5
pathspec==0.11.1
Pillow==9.5.0
platformdirs==3.5.1
pluggy==0.13.1
pre-commit==2.21.0
py==1.11.0
pycodestyle==2.5.0
pyflakes==2.1.1
pytest==5.4.1
python-dateutil==2.8.2
pytz==2023.3
PyYAML==6.0
regex==2023.5.5
requests==2.30.0
ruamel.yaml==0.17.26
ruamel.yaml.clib==0.2.7
scipy==1.7.3
sentencepiece==0.1.99
six==1.16.0
speechbrain==0.5.14
tokenizers==0.13.3
toml==0.10.2
torch==1.13.1
torchaudio==0.13.1
torchvision==0.14.1
tqdm==4.65.0
transformers==4.29.2
typed-ast==1.5.4
typing_extensions==4.5.0
urllib3==2.0.2
virtualenv==20.23.0
wcwidth==0.2.6
yamllint==1.23.0
zipp==3.15.0
2023-06-08 19:35:22,114 - speechbrain.dataio.dataio - INFO - Skipping merging. Completed in previous run.
2023-06-08 19:35:22,207 - speechbrain.dataio.dataio - INFO - /data_home/SLU_path/slurp/results/lstm/1986/train-type=direct.csv is created.
2023-06-08 19:35:22,729 - speechbrain.utils.parameter_transfer - DEBUG - Collecting files (or symlinks) for pretraining in /data_home/SLU_path/slurp/results/lstm/1986/save/SLURP_tokenizer.
2023-06-08 19:35:22,729 - speechbrain.pretrained.fetching - INFO - Fetch tokenizer_58_unigram.model: Using existing file/symlink in /data_home/SLU_path/slurp/results/lstm/1986/save/SLURP_tokenizer/tokenizer.ckpt.
2023-06-08 19:35:22,730 - speechbrain.utils.parameter_transfer - INFO - Loading pretrained files for: tokenizer
2023-06-08 19:35:23,415 - speechbrain.core - INFO - Info: ckpt_interval_minutes arg from hparam file is used
2023-06-08 19:35:23,419 - speechbrain.core - INFO - 98.3M trainable parameters in SLU
2023-06-08 19:35:23,420 - speechbrain.utils.checkpoints - INFO - Loading a checkpoint from /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+17-58-43+00
2023-06-08 19:35:23,979 - speechbrain.utils.epoch_loop - INFO - Going into epoch 20
2023-06-08 19:56:58,646 - speechbrain.utils.train_logger - INFO - epoch: 20, lr: 1.23e-04, wave2vec_lr: 6.56e-06 - train loss: 7.25e-01 - valid loss: 7.57e-01, valid CER: 10.57, valid WER: 14.00
2023-06-08 19:56:59,323 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+19-56-58+00
2023-06-08 19:56:59,412 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+16-58-39+00
2023-06-08 19:56:59,456 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+15-36-26+00
2023-06-08 19:56:59,550 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+17-58-43+00
2023-06-08 19:56:59,550 - speechbrain.utils.epoch_loop - INFO - Going into epoch 21
2023-06-08 20:12:00,261 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+20-11-59+00
2023-06-08 20:27:00,938 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+20-27-00+00
2023-06-08 20:27:01,031 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+20-11-59+00
2023-06-08 20:42:01,821 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+20-42-01+00
2023-06-08 20:42:01,925 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+20-27-00+00
2023-06-08 20:57:02,679 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+20-57-02+00
2023-06-08 20:57:02,765 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+20-42-01+00
2023-06-08 21:19:03,929 - speechbrain.nnet.schedulers - INFO - Changing lr from 0.00012 to 9.8e-05
2023-06-08 21:19:03,929 - speechbrain.nnet.schedulers - INFO - Changing lr from 6.6e-06 to 5.9e-06
2023-06-08 21:19:03,929 - speechbrain.utils.train_logger - INFO - epoch: 21, lr: 1.23e-04, wave2vec_lr: 6.56e-06 - train loss: 7.25e-01 - valid loss: 7.57e-01, valid CER: 10.53, valid WER: 14.04
2023-06-08 21:19:04,581 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+21-19-03+00
2023-06-08 21:19:04,664 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+20-57-02+00
2023-06-08 21:19:04,664 - speechbrain.utils.epoch_loop - INFO - Going into epoch 22
2023-06-08 21:34:05,408 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+21-34-04+00
2023-06-08 21:49:06,253 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+21-49-05+00
2023-06-08 21:49:06,335 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+21-34-04+00
2023-06-08 22:04:07,101 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+22-04-06+00
2023-06-08 22:04:07,186 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+21-49-05+00
2023-06-08 22:19:07,943 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+22-19-07+00
2023-06-08 22:19:08,034 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+22-04-06+00
2023-06-08 22:41:46,295 - speechbrain.utils.train_logger - INFO - epoch: 22, lr: 9.83e-05, wave2vec_lr: 5.90e-06 - train loss: 7.24e-01 - valid loss: 7.57e-01, valid CER: 10.13, valid WER: 13.57
2023-06-08 22:41:46,959 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+22-41-46+00
2023-06-08 22:41:47,018 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+19-56-58+00
2023-06-08 22:41:47,098 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+22-19-07+00
2023-06-08 22:41:47,166 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+21-19-03+00
2023-06-08 22:41:47,167 - speechbrain.utils.epoch_loop - INFO - Going into epoch 23
2023-06-08 22:56:47,904 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+22-56-47+00
2023-06-08 23:11:48,711 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+23-11-48+00
2023-06-08 23:11:48,808 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+22-56-47+00
2023-06-08 23:26:49,518 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+23-26-48+00
2023-06-08 23:26:49,607 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+23-11-48+00
2023-06-08 23:41:50,381 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+23-41-49+00
2023-06-08 23:41:50,479 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+23-26-48+00
2023-06-09 00:04:09,155 - speechbrain.utils.train_logger - INFO - epoch: 23, lr: 9.83e-05, wave2vec_lr: 5.90e-06 - train loss: 7.24e-01 - valid loss: 7.56e-01, valid CER: 10.17, valid WER: 13.53
2023-06-09 00:04:09,831 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+00-04-09+00
2023-06-09 00:04:09,926 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+23-41-49+00
2023-06-09 00:04:10,001 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-08+22-41-46+00
2023-06-09 00:04:10,001 - speechbrain.utils.epoch_loop - INFO - Going into epoch 24
2023-06-09 00:19:10,848 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+00-19-10+00
2023-06-09 00:34:11,619 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+00-34-10+00
2023-06-09 00:34:11,722 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+00-19-10+00
2023-06-09 00:49:12,428 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+00-49-11+00
2023-06-09 00:49:12,523 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+00-34-10+00
2023-06-09 01:04:13,273 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+01-04-12+00
2023-06-09 01:04:13,374 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+00-49-11+00
2023-06-09 01:26:40,640 - speechbrain.nnet.schedulers - INFO - Changing lr from 9.8e-05 to 7.9e-05
2023-06-09 01:26:40,661 - speechbrain.nnet.schedulers - INFO - Changing lr from 5.9e-06 to 5.3e-06
2023-06-09 01:26:40,661 - speechbrain.utils.train_logger - INFO - epoch: 24, lr: 9.83e-05, wave2vec_lr: 5.90e-06 - train loss: 7.23e-01 - valid loss: 7.57e-01, valid CER: 10.31, valid WER: 13.68
2023-06-09 01:26:41,339 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+01-26-40+00
2023-06-09 01:26:41,439 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+01-04-12+00
2023-06-09 01:26:41,440 - speechbrain.utils.epoch_loop - INFO - Going into epoch 25
2023-06-09 01:41:42,136 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+01-41-41+00
2023-06-09 01:56:42,998 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+01-56-42+00
2023-06-09 01:56:43,119 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+01-41-41+00
2023-06-09 02:11:43,837 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+02-11-43+00
2023-06-09 02:11:43,958 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+01-56-42+00
2023-06-09 02:26:44,659 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+02-26-43+00
2023-06-09 02:26:44,773 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+02-11-43+00
2023-06-09 02:45:36,022 - speechbrain.core - ERROR - Exception:
Traceback (most recent call last):
File "train_with_wav2vec2.py", line 379, in
valid_loader_kwargs=hparams["dataloader_opts"],
File "/home/zhaochen/anaconda3/envs/SLU/lib/python3.7/site-packages/speechbrain/core.py", line 1226, in fit
self._fit_valid(valid_set=valid_set, epoch=epoch, enable=enable)
File "/home/zhaochen/anaconda3/envs/SLU/lib/python3.7/site-packages/speechbrain/core.py", line 1129, in _fit_valid
loss = self.evaluate_batch(batch, stage=Stage.VALID)
File "train_with_wav2vec2.py", line 161, in evaluate_batch
loss = self.compute_objectives(predictions, batch, stage=stage)
File "train_with_wav2vec2.py", line 94, in compute_objectives
for utt_seq in predicted_tokens
File "train_with_wav2vec2.py", line 94, in
for utt_seq in predicted_tokens
File "/home/zhaochen/anaconda3/envs/SLU/lib/python3.7/site-packages/sentencepiece/init.py", line 837, in DecodeIds
return self.Decode(input=input, out_type=out_type, **kwargs)
File "/home/zhaochen/anaconda3/envs/SLU/lib/python3.7/site-packages/sentencepiece/init.py", line 780, in Decode
return self._DecodeIds(input)
File "/home/zhaochen/anaconda3/envs/SLU/lib/python3.7/site-packages/sentencepiece/init.py", line 337, in _DecodeIds
return _sentencepiece.SentencePieceProcessor__DecodeIds(self, ids)
IndexError: Out of range: piece id is out of range.
2023-06-09 11:08:57,850 - speechbrain.core - INFO - Beginning experiment!
2023-06-09 11:08:57,851 - speechbrain.core - INFO - Experiment folder: /data_home/SLU_path/slurp/results/lstm/1986
2023-06-09 11:08:58,028 - speechbrain.utils.superpowers - DEBUG - appdirs==1.4.4
attrs==23.1.0
black==19.10b0
certifi==2023.5.7
cfgv==3.3.1
charset-normalizer==3.1.0
click==8.0.4
distlib==0.3.6
entrypoints==0.3
filelock==3.12.0
flake8==3.7.9
fsspec==2023.1.0
huggingface-hub==0.14.1
HyperPyYAML==1.2.0
identify==2.5.24
idna==3.4
importlib-metadata==6.6.0
joblib==1.2.0
jsonlines==3.1.0
mccabe==0.6.1
more-itertools==9.1.0
nodeenv==1.7.0
numpy==1.21.6
nvidia-cublas-cu11==<IP_ADDRESS>
nvidia-cuda-nvrtc-cu11==11.7.99
nvidia-cuda-runtime-cu11==11.7.99
nvidia-cudnn-cu11==<IP_ADDRESS>
packaging==23.1
pandas==1.3.5
pathspec==0.11.1
Pillow==9.5.0
platformdirs==3.5.1
pluggy==0.13.1
pre-commit==2.21.0
py==1.11.0
pycodestyle==2.5.0
pyflakes==2.1.1
pytest==5.4.1
python-dateutil==2.8.2
pytz==2023.3
PyYAML==6.0
regex==2023.5.5
requests==2.30.0
ruamel.yaml==0.17.26
ruamel.yaml.clib==0.2.7
scipy==1.7.3
sentencepiece==0.1.99
six==1.16.0
speechbrain==0.5.14
tokenizers==0.13.3
toml==0.10.2
torch==1.13.1
torchaudio==0.13.1
torchvision==0.14.1
tqdm==4.65.0
transformers==4.29.2
typed-ast==1.5.4
typing_extensions==4.5.0
urllib3==2.0.2
virtualenv==20.23.0
wcwidth==0.2.6
yamllint==1.23.0
zipp==3.15.0
2023-06-09 11:08:58,073 - speechbrain.dataio.dataio - INFO - Skipping merging. Completed in previous run.
2023-06-09 11:08:58,195 - speechbrain.dataio.dataio - INFO - /data_home/SLU_path/slurp/results/lstm/1986/train-type=direct.csv is created.
2023-06-09 11:08:58,725 - speechbrain.utils.parameter_transfer - DEBUG - Collecting files (or symlinks) for pretraining in /data_home/SLU_path/slurp/results/lstm/1986/save/SLURP_tokenizer.
2023-06-09 11:08:58,725 - speechbrain.pretrained.fetching - INFO - Fetch tokenizer_58_unigram.model: Using existing file/symlink in /data_home/SLU_path/slurp/results/lstm/1986/save/SLURP_tokenizer/tokenizer.ckpt.
2023-06-09 11:08:58,725 - speechbrain.utils.parameter_transfer - INFO - Loading pretrained files for: tokenizer
2023-06-09 11:08:59,452 - speechbrain.core - INFO - Info: ckpt_interval_minutes arg from hparam file is used
2023-06-09 11:08:59,457 - speechbrain.core - INFO - 98.3M trainable parameters in SLU
2023-06-09 11:08:59,458 - speechbrain.utils.checkpoints - INFO - Loading a checkpoint from /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+02-26-43+00
2023-06-09 11:09:00,012 - speechbrain.utils.epoch_loop - INFO - Going into epoch 25
2023-06-09 11:30:57,886 - speechbrain.utils.train_logger - INFO - epoch: 25, lr: 7.86e-05, wave2vec_lr: 5.31e-06 - train loss: 7.23e-01 - valid loss: 7.56e-01, valid CER: 10.00, valid WER: 13.30
2023-06-09 11:30:58,530 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+11-30-57+00
2023-06-09 11:30:58,620 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+02-26-43+00
2023-06-09 11:30:58,706 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+01-26-40+00
2023-06-09 11:30:58,777 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+00-04-09+00
2023-06-09 11:30:58,777 - speechbrain.utils.epoch_loop - INFO - Going into epoch 26
2023-06-09 11:45:59,558 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+11-45-58+00
2023-06-09 12:01:00,252 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+12-00-59+00
2023-06-09 12:01:00,336 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+11-45-58+00
2023-06-09 12:16:01,027 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+12-16-00+00
2023-06-09 12:16:01,113 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+12-00-59+00
2023-06-09 12:31:01,795 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+12-31-01+00
2023-06-09 12:31:01,883 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+12-16-00+00
2023-06-09 12:52:39,021 - speechbrain.utils.train_logger - INFO - epoch: 26, lr: 7.86e-05, wave2vec_lr: 5.31e-06 - train loss: 7.23e-01 - valid loss: 7.56e-01, valid CER: 9.83, valid WER: 13.17
2023-06-09 12:52:39,693 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+12-52-39+00
2023-06-09 12:52:39,777 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+11-30-57+00
2023-06-09 12:52:39,854 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+12-31-01+00
2023-06-09 12:52:39,854 - speechbrain.utils.epoch_loop - INFO - Going into epoch 27
2023-06-09 13:07:40,553 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+13-07-39+00
2023-06-09 13:22:41,265 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+13-22-40+00
2023-06-09 13:22:41,354 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+13-07-39+00
2023-06-09 13:37:42,115 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+13-37-41+00
2023-06-09 13:37:42,212 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+13-22-40+00
2023-06-09 13:52:42,925 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+13-52-42+00
2023-06-09 13:52:43,016 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+13-37-41+00
2023-06-09 14:14:53,485 - speechbrain.utils.train_logger - INFO - epoch: 27, lr: 7.86e-05, wave2vec_lr: 5.31e-06 - train loss: 7.23e-01 - valid loss: 7.56e-01, valid CER: 9.81, valid WER: 13.09
2023-06-09 14:14:54,154 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+14-14-53+00
2023-06-09 14:14:54,245 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+13-52-42+00
2023-06-09 14:14:54,320 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+12-52-39+00
2023-06-09 14:14:54,320 - speechbrain.utils.epoch_loop - INFO - Going into epoch 28
2023-06-09 14:29:55,081 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+14-29-54+00
2023-06-09 14:44:55,769 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+14-44-55+00
2023-06-09 14:44:55,872 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+14-29-54+00
2023-06-09 14:59:56,626 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+14-59-55+00
2023-06-09 14:59:56,718 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+14-44-55+00
2023-06-09 15:14:57,428 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+15-14-56+00
2023-06-09 15:14:57,526 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+14-59-55+00
2023-06-09 15:37:05,023 - speechbrain.utils.train_logger - INFO - epoch: 28, lr: 7.86e-05, wave2vec_lr: 5.31e-06 - train loss: 7.23e-01 - valid loss: 7.56e-01, valid CER: 9.87, valid WER: 13.05
2023-06-09 15:37:05,689 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+15-37-05+00
2023-06-09 15:37:05,791 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+14-14-53+00
2023-06-09 15:37:05,868 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+15-14-56+00
2023-06-09 15:37:05,868 - speechbrain.utils.epoch_loop - INFO - Going into epoch 29
2023-06-09 15:52:06,743 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+15-52-06+00
2023-06-09 16:07:07,549 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+16-07-06+00
2023-06-09 16:07:07,650 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+15-52-06+00
2023-06-09 16:22:08,482 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+16-22-07+00
2023-06-09 16:22:08,574 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+16-07-06+00
2023-06-09 16:37:09,362 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+16-37-08+00
2023-06-09 16:37:09,462 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+16-22-07+00
2023-06-09 16:59:33,396 - speechbrain.nnet.schedulers - INFO - Changing lr from 7.9e-05 to 6.3e-05
2023-06-09 16:59:33,396 - speechbrain.nnet.schedulers - INFO - Changing lr from 5.3e-06 to 4.8e-06
2023-06-09 16:59:33,396 - speechbrain.utils.train_logger - INFO - epoch: 29, lr: 7.86e-05, wave2vec_lr: 5.31e-06 - train loss: 7.22e-01 - valid loss: 7.57e-01, valid CER: 9.88, valid WER: 13.21
2023-06-09 16:59:34,080 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+16-59-33+00
2023-06-09 16:59:34,183 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+16-37-08+00
2023-06-09 16:59:34,183 - speechbrain.utils.epoch_loop - INFO - Going into epoch 30
2023-06-09 17:14:34,920 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+17-14-34+00
2023-06-09 17:29:35,744 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+17-29-35+00
2023-06-09 17:29:35,855 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+17-14-34+00
2023-06-09 17:44:36,649 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+17-44-35+00
2023-06-09 17:44:36,761 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+17-29-35+00
2023-06-09 17:59:37,506 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+17-59-36+00
2023-06-09 17:59:37,624 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+17-44-35+00
2023-06-09 18:21:51,902 - speechbrain.utils.train_logger - INFO - epoch: 30, lr: 6.29e-05, wave2vec_lr: 4.78e-06 - train loss: 7.22e-01 - valid loss: 7.56e-01, valid CER: 9.85, valid WER: 13.01
2023-06-09 18:21:52,554 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+18-21-51+00
2023-06-09 18:21:52,669 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+15-37-05+00
2023-06-09 18:21:52,742 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+16-59-33+00
2023-06-09 18:21:52,814 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+17-59-36+00
2023-06-09 18:21:52,814 - speechbrain.utils.epoch_loop - INFO - Going into epoch 31
2023-06-09 18:36:53,599 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+18-36-52+00
2023-06-09 18:51:54,391 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+18-51-53+00
2023-06-09 18:51:54,500 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+18-36-52+00
2023-06-09 19:06:55,211 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+19-06-54+00
2023-06-09 19:06:55,321 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+18-51-53+00
2023-06-09 19:21:55,997 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+19-21-55+00
2023-06-09 19:21:56,112 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+19-06-54+00
2023-06-09 19:43:56,908 - speechbrain.nnet.schedulers - INFO - Changing lr from 6.3e-05 to 5e-05
2023-06-09 19:43:56,908 - speechbrain.nnet.schedulers - INFO - Changing lr from 4.8e-06 to 4.3e-06
2023-06-09 19:43:56,909 - speechbrain.utils.train_logger - INFO - epoch: 31, lr: 6.29e-05, wave2vec_lr: 4.78e-06 - train loss: 7.22e-01 - valid loss: 7.57e-01, valid CER: 9.89, valid WER: 13.09
2023-06-09 19:43:57,580 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+19-43-56+00
2023-06-09 19:43:57,697 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+19-21-55+00
2023-06-09 19:43:57,697 - speechbrain.utils.epoch_loop - INFO - Going into epoch 32
2023-06-09 19:58:58,420 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+19-58-57+00
2023-06-09 20:13:59,152 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+20-13-58+00
2023-06-09 20:13:59,280 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+19-58-57+00
2023-06-09 20:29:00,047 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+20-28-59+00
2023-06-09 20:29:00,185 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+20-13-58+00
2023-06-09 20:44:00,963 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+20-44-00+00
2023-06-09 20:44:01,098 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+20-28-59+00
2023-06-09 21:06:28,854 - speechbrain.utils.train_logger - INFO - epoch: 32, lr: 5.03e-05, wave2vec_lr: 4.30e-06 - train loss: 7.22e-01 - valid loss: 7.56e-01, valid CER: 9.65, valid WER: 12.76
2023-06-09 21:06:29,547 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+21-06-28+00
2023-06-09 21:06:29,686 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+20-44-00+00
2023-06-09 21:06:29,765 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+19-43-56+00
2023-06-09 21:06:29,836 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+18-21-51+00
2023-06-09 21:06:29,836 - speechbrain.utils.epoch_loop - INFO - Going into epoch 33
2023-06-09 21:21:30,569 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+21-21-29+00
2023-06-09 21:36:31,347 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+21-36-30+00
2023-06-09 21:36:31,478 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+21-21-29+00
2023-06-09 21:51:32,286 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+21-51-31+00
2023-06-09 21:51:32,405 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+21-36-30+00
2023-06-09 22:06:33,257 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+22-06-32+00
2023-06-09 22:06:33,384 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+21-51-31+00
2023-06-09 22:28:28,649 - speechbrain.nnet.schedulers - INFO - Changing lr from 5e-05 to 4e-05
2023-06-09 22:28:28,650 - speechbrain.nnet.schedulers - INFO - Changing lr from 4.3e-06 to 3.9e-06
2023-06-09 22:28:28,650 - speechbrain.utils.train_logger - INFO - epoch: 33, lr: 5.03e-05, wave2vec_lr: 4.30e-06 - train loss: 7.22e-01 - valid loss: 7.56e-01, valid CER: 9.66, valid WER: 12.76
2023-06-09 22:28:29,323 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+22-28-28+00
2023-06-09 22:28:29,456 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+22-06-32+00
2023-06-09 22:28:29,533 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+21-06-28+00
2023-06-09 22:28:29,533 - speechbrain.utils.epoch_loop - INFO - Going into epoch 34
2023-06-09 22:43:30,226 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+22-43-29+00
2023-06-09 22:58:31,035 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+22-58-30+00
2023-06-09 22:58:31,169 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+22-43-29+00
2023-06-09 23:13:31,859 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+23-13-31+00
2023-06-09 23:13:31,985 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+22-58-30+00
2023-06-09 23:28:32,674 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+23-28-32+00
2023-06-09 23:28:32,800 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+23-13-31+00
2023-06-09 23:50:35,609 - speechbrain.utils.train_logger - INFO - epoch: 34, lr: 4.03e-05, wave2vec_lr: 3.87e-06 - train loss: 7.21e-01 - valid loss: 7.56e-01, valid CER: 9.22, valid WER: 12.28
2023-06-09 23:50:36,277 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+23-50-35+00
2023-06-09 23:50:36,415 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+22-28-28+00
2023-06-09 23:50:36,495 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-09+23-28-32+00
2023-06-09 23:50:36,495 - speechbrain.utils.epoch_loop - INFO - Going into epoch 35
2023-06-10 00:05:37,192 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-10+00-05-36+00
2023-06-10 00:20:37,972 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-10+00-20-37+00
2023-06-10 00:20:38,113 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-10+00-05-36+00
2023-06-10 00:35:38,854 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-10+00-35-38+00
2023-06-10 00:35:38,985 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-10+00-20-37+00
2023-06-10 00:50:39,754 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-10+00-50-39+00
2023-06-10 00:50:39,891 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-10+00-35-38+00
2023-06-10 01:12:39,269 - speechbrain.nnet.schedulers - INFO - Changing lr from 4e-05 to 3.2e-05
2023-06-10 01:12:39,269 - speechbrain.nnet.schedulers - INFO - Changing lr from 3.9e-06 to 3.5e-06
2023-06-10 01:12:39,269 - speechbrain.utils.train_logger - INFO - epoch: 35, lr: 4.03e-05, wave2vec_lr: 3.87e-06 - train loss: 7.21e-01 - valid loss: 7.56e-01, valid CER: 9.57, valid WER: 12.59
2023-06-10 01:12:39,933 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-10+01-12-39+00
2023-06-10 01:12:40,075 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-10+00-50-39+00
2023-06-10 01:12:40,171 - speechbrain.utils.checkpoints - INFO - Loading a checkpoint from /data_home/SLU_path/slurp/results/lstm/1986/save/CKPT+2023-06-10+01-12-39+00
2023-06-10 01:12:40,288 - root - DEBUG - SaveableDataLoader was requested to load a checkpoint, but the DataLoader has already been iterated. The DataLoader file will be ignored. This is normal in evaluation, when a checkpoint is loaded just to retrieve the best model.
2023-06-10 01:30:01,349 - speechbrain.utils.train_logger - INFO - Epoch loaded: 35 - test loss: 7.61e-01, test CER: 10.13, test WER: 13.61

hello ,  This is the experiment I tried to reproduce in rtx5000, with batch size set to 2 and attn_type set to keyvalue, and ValueError: Loss is not finite and patience is exhausted in the 10th epochs.
Generated 2023-06-27 from:
/home/zhengbeida/code/25/speechbrain-develop/speechbrain-develop/recipes/SLURP/direct/hparams/train_with_wav2vec2.yaml
yamllint disable

Model: Direct SLU
Encoder: Pre-trained wav2vec2 -> LSTM
Decoder: GRU + beamsearch
Tokens: BPE with unigram
losses: NLL
Training: SLURP
Authors:  Loren Lugosch, Mirco Ravanelli 2020 Boumadane Abdelmoumene, AbdelWahab Heba, Yingzhi Wang 2021
For more wav2vec2/HuBERT results, please see https://arxiv.org/pdf/2111.02735.pdf

Seed needs to be set at top of yaml, before objects with parameters are made
seed: 1986
__set_seed: !apply:torch.manual_seed [1986]
output_folder: /mnt/home/zhengbeida/path/slurp/results/lstm/1986
save_folder: /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save
train_log: /mnt/home/zhengbeida/path/slurp/results/lstm/1986/train_log.txt
Data files
The SLURP dataset will be automatically downloaded in the specified data_folder
data_folder: /mnt/home/zhengbeida/path/slurp
data_folder_rirs: !ref <data_folder>
train_splits: [train_real, train_synthetic]
csv_train: /mnt/home/zhengbeida/path/slurp/results/lstm/1986/train-type=direct.csv
csv_valid: /mnt/home/zhengbeida/path/slurp/results/lstm/1986/devel-type=direct.csv
csv_test: /mnt/home/zhengbeida/path/slurp/results/lstm/1986/test-type=direct.csv
tokenizer_file: speechbrain/SLU-direct-SLURP-hubert-enc
skip_prep: false
URL for the wav2vec2 model, you can change to benchmark diffrenet models
wav2vec2_hub: facebook/hubert-base-ls960
Training parameters
number_of_epochs: 35
batch_size: 2
lr: 0.0003
lr_wav2vec2: 0.00001
token_type: unigram # ["unigram", "bpe", "char"]
sorting: random
ckpt_interval_minutes: 15 # save checkpoint every N min
#freeze all wav2vec2
freeze_wav2vec2: false
#set to true to freeze the CONV part of the wav2vec2 model
freeze_wav2vec2_conv: true
Model parameters
sample_rate: 16000
emb_size: 128
dec_neurons: 512
output_neurons: 58 # index(eos/bos) = 0
encoder_dim: 768
Decoding parameters
bos_index: 0
eos_index: 0
min_decode_ratio: 0.0
max_decode_ratio: 10.0
slu_beam_size: 80
eos_threshold: 1.5
temperature: 1.25
dataloader_opts:
batch_size: 2
shuffle: true
epoch_counter: &id009 !new:speechbrain.utils.epoch_loop.EpochCounter
limit: 35
Models
wav2vec2: &id001 !new:speechbrain.lobes.models.huggingface_wav2vec.HuggingFaceWav2Vec2
source: facebook/hubert-base-ls960
output_norm: true
freeze: false
save_path: /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/wav2vec2_checkpoint
output_emb: &id002 !new:speechbrain.nnet.embedding.Embedding
num_embeddings: 58
embedding_dim: 128
dec: &id003 !new:speechbrain.nnet.RNN.AttentionalRNNDecoder
enc_dim: 768
input_size: 128
rnn_type: lstm
attn_type: keyvalue
hidden_size: 512
attn_dim: 512
num_layers: 3
scaling: 1.0
dropout: 0.0
seq_lin: &id004 !new:speechbrain.nnet.linear.Linear
input_size: 512
n_neurons: 58
augmentation: !new:speechbrain.lobes.augment.TimeDomainSpecAugment
sample_rate: 16000
speeds: [95, 100, 105]
modules:
wav2vec2: *id001
output_emb: *id002
dec: *id003
seq_lin: *id004
model: &id006 !new:torch.nn.ModuleList

[*id002, *id003, *id004]
tokenizer: &id005 !new:sentencepiece.SentencePieceProcessor

pretrainer: !new:speechbrain.utils.parameter_transfer.Pretrainer
collect_in: /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/SLURP_tokenizer
loadables:
tokenizer: *id005
paths:
tokenizer: speechbrain/SLU-direct-SLURP-hubert-enc/tokenizer_58_unigram.model
beam_searcher: !new:speechbrain.decoders.S2SRNNBeamSearcher
embedding: *id002
decoder: *id003
linear: *id004
bos_index: 0
eos_index: 0
min_decode_ratio: 0.0
max_decode_ratio: 10.0
beam_size: 80
eos_threshold: 1.5
temperature: 1.25
using_max_attn_shift: false
max_attn_shift: 30
coverage_penalty: 0.
opt_class: !name:torch.optim.Adam
lr: 0.0003
wav2vec2_opt_class: !name:torch.optim.Adam
lr: 0.00001
lr_annealing: &id007 !new:speechbrain.nnet.schedulers.NewBobScheduler
initial_value: 0.0003
improvement_threshold: 0.0025
annealing_factor: 0.8
patient: 0
lr_annealing_wav2vec2: &id008 !new:speechbrain.nnet.schedulers.NewBobScheduler
initial_value: 0.00001
improvement_threshold: 0.0025
annealing_factor: 0.9
checkpointer: !new:speechbrain.utils.checkpoints.Checkpointer
checkpoints_dir: /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save
recoverables:
model: *id006
wav2vec2: *id001
scheduler: *id007
lr_annealing_wav2vec2: *id008
counter: *id009
log_softmax: !new:speechbrain.nnet.activations.Softmax
apply_log: true
seq_cost: !name:speechbrain.nnet.losses.nll_loss
label_smoothing: 0.1
train_logger: !new:speechbrain.utils.train_logger.FileTrainLogger
save_file: /mnt/home/zhengbeida/path/slurp/results/lstm/1986/train_log.txt
error_rate_computer: !name:speechbrain.utils.metric_stats.ErrorRateStats
cer_computer: !name:speechbrain.utils.metric_stats.ErrorRateStats
split_tokens: true

hello,This is the log entry @mhn226
2023-06-20 11:20:58,393 - speechbrain.dataio.dataio - INFO - Skipping merging. Completed in previous run.
2023-06-20 11:20:58,486 - speechbrain.dataio.dataio - INFO - /mnt/home/zhengbeida/path/slurp/results/lstm/1986/train-type=direct.csv is created.
2023-06-20 11:20:59,498 - speechbrain.utils.parameter_transfer - DEBUG - Collecting files (or symlinks) for pretraining in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/SLURP_tokenizer.
2023-06-20 11:20:59,498 - speechbrain.pretrained.fetching - INFO - Fetch tokenizer_58_unigram.model: Using existing file/symlink in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/SLURP_tokenizer/tokenizer.ckpt.
2023-06-20 11:20:59,498 - speechbrain.utils.parameter_transfer - INFO - Loading pretrained files for: tokenizer
2023-06-20 11:21:02,232 - speechbrain.core - INFO - Info: ckpt_interval_minutes arg from hparam file is used
2023-06-20 11:21:02,240 - speechbrain.core - INFO - 98.3M trainable parameters in SLU
2023-06-20 11:21:02,241 - speechbrain.utils.checkpoints - INFO - Would load a checkpoint here, but none found yet.
2023-06-20 11:21:02,242 - speechbrain.utils.epoch_loop - INFO - Going into epoch 1
2023-06-20 11:21:03,549 - speechbrain.core - ERROR - Exception:
Traceback (most recent call last):
File "train_with_wav2vec2.py", line 374, in
valid_loader_kwargs=hparams["dataloader_opts"],
File "/home/zhengbeida/anaconda3/envs/slurp/lib/python3.7/site-packages/speechbrain/core.py", line 1225, in fit
self._fit_train(train_set=train_set, epoch=epoch, enable=enable)
File "/home/zhengbeida/anaconda3/envs/slurp/lib/python3.7/site-packages/speechbrain/core.py", line 1078, in _fit_train
loss = self.fit_batch(batch)
File "train_with_wav2vec2.py", line 143, in fit_batch
loss = self.compute_objectives(predictions, batch, sb.Stage.TRAIN)
File "train_with_wav2vec2.py", line 89, in compute_objectives
for utt_seq in predicted_tokens
File "train_with_wav2vec2.py", line 89, in
for utt_seq in predicted_tokens
File "/home/zhengbeida/anaconda3/envs/slurp/lib/python3.7/site-packages/sentencepiece/init.py", line 837, in DecodeIds
return self.Decode(input=input, out_type=out_type, **kwargs)
File "/home/zhengbeida/anaconda3/envs/slurp/lib/python3.7/site-packages/sentencepiece/init.py", line 780, in Decode
return self._DecodeIds(input)
File "/home/zhengbeida/anaconda3/envs/slurp/lib/python3.7/site-packages/sentencepiece/init.py", line 337, in _DecodeIds
return _sentencepiece.SentencePieceProcessor__DecodeIds(self, ids)
IndexError: Out of range: piece id is out of range.
2023-06-20 11:21:43,082 - speechbrain.core - INFO - Beginning experiment!
2023-06-20 11:21:43,082 - speechbrain.core - INFO - Experiment folder: /mnt/home/zhengbeida/path/slurp/results/lstm/1986
2023-06-20 11:21:43,448 - speechbrain.utils.superpowers - DEBUG - appdirs==1.4.4
attrs==23.1.0
black==19.10b0
certifi @ file:///croot/certifi_1671487769961/work/certifi
cfgv==3.3.1
charset-normalizer==3.1.0
click==8.0.4
distlib==0.3.6
entrypoints==0.3
filelock==3.12.0
flake8==3.7.9
fsspec==2023.1.0
huggingface-hub==0.15.1
HyperPyYAML==1.2.1
identify==2.5.24
idna==3.4
importlib-metadata==6.6.0
joblib==1.2.0
jsonlines==3.1.0
mccabe==0.6.1
more-itertools==9.1.0
nodeenv==1.7.0
numpy==1.21.6
packaging==23.1
pandas==1.3.5
pathspec==0.11.1
Pillow==9.5.0
platformdirs==3.5.1
pluggy==0.13.1
pre-commit==2.21.0
py==1.11.0
pycodestyle==2.5.0
pyflakes==2.1.1
pytest==5.4.1
python-dateutil==2.8.2
pytz==2023.3
PyYAML==6.0
regex==2023.5.5
requests==2.31.0
ruamel.yaml==0.17.28
ruamel.yaml.clib==0.2.7
scipy==1.7.3
sentencepiece==0.1.99
six==1.16.0
speechbrain==0.5.14
tokenizers==0.13.3
toml==0.10.2
torch==1.9.0
torchaudio==0.9.0
torchvision==0.10.0
tqdm==4.65.0
transformers==4.29.2
typed-ast==1.5.4
typing_extensions==4.5.0
urllib3==2.0.2
virtualenv==20.23.0
wcwidth==0.2.6
yamllint==1.23.0
zipp==3.15.0
2023-06-20 11:21:44,391 - speechbrain.dataio.dataio - INFO - Skipping merging. Completed in previous run.
2023-06-20 11:21:44,488 - speechbrain.dataio.dataio - INFO - /mnt/home/zhengbeida/path/slurp/results/lstm/1986/train-type=direct.csv is created.
2023-06-20 11:21:45,489 - speechbrain.utils.parameter_transfer - DEBUG - Collecting files (or symlinks) for pretraining in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/SLURP_tokenizer.
2023-06-20 11:21:45,489 - speechbrain.pretrained.fetching - INFO - Fetch tokenizer_58_unigram.model: Using existing file/symlink in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/SLURP_tokenizer/tokenizer.ckpt.
2023-06-20 11:21:45,489 - speechbrain.utils.parameter_transfer - INFO - Loading pretrained files for: tokenizer
2023-06-20 11:21:48,237 - speechbrain.core - INFO - Info: ckpt_interval_minutes arg from hparam file is used
2023-06-20 11:21:48,245 - speechbrain.core - INFO - 98.3M trainable parameters in SLU
2023-06-20 11:21:48,246 - speechbrain.utils.checkpoints - INFO - Would load a checkpoint here, but none found yet.
2023-06-20 11:21:48,246 - speechbrain.utils.epoch_loop - INFO - Going into epoch 1
2023-06-20 11:36:49,322 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+11-36-48+00
2023-06-20 11:51:50,483 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+11-51-49+00
2023-06-20 11:51:50,625 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+11-36-48+00
2023-06-20 12:06:51,805 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+12-06-50+00
2023-06-20 12:06:51,942 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+11-51-49+00
2023-06-20 12:21:52,999 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+12-21-51+00
2023-06-20 12:21:53,145 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+12-06-50+00
2023-06-20 12:36:54,195 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+12-36-53+00
2023-06-20 12:36:54,338 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+12-21-51+00
2023-06-20 12:51:55,376 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+12-51-54+00
2023-06-20 12:51:55,531 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+12-36-53+00
2023-06-20 13:06:56,686 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+13-06-55+00
2023-06-20 13:06:56,830 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+12-51-54+00
2023-06-20 13:21:57,892 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+13-21-56+00
2023-06-20 13:21:58,045 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+13-06-55+00
2023-06-20 13:36:59,228 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+13-36-58+00
2023-06-20 13:36:59,372 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+13-21-56+00
2023-06-20 13:52:00,611 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+13-51-59+00
2023-06-20 13:52:00,769 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+13-36-58+00
2023-06-20 14:07:01,842 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+14-07-00+00
2023-06-20 14:07:01,992 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+13-51-59+00
2023-06-20 14:22:03,163 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+14-22-02+00
2023-06-20 14:22:03,325 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+14-07-00+00
2023-06-20 14:37:04,481 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+14-37-03+00
2023-06-20 14:37:04,715 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+14-22-02+00
2023-06-20 14:52:06,095 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+14-52-05+00
2023-06-20 14:52:06,261 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+14-37-03+00
2023-06-20 15:16:58,943 - speechbrain.utils.train_logger - INFO - epoch: 1, lr: 3.00e-04, wave2vec_lr: 1.00e-05 - train loss: 8.19e-01 - valid loss: 7.91e-01, valid CER: 23.99, valid WER: 29.62
2023-06-20 15:16:59,972 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+15-16-58+00
2023-06-20 15:17:00,133 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+14-52-05+00
2023-06-20 15:17:00,133 - speechbrain.utils.epoch_loop - INFO - Going into epoch 2
2023-06-20 15:32:01,225 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+15-32-00+00
2023-06-20 15:47:02,364 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+15-47-01+00
2023-06-20 15:47:02,541 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+15-32-00+00
2023-06-20 16:02:03,726 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+16-02-02+00
2023-06-20 16:02:03,904 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+15-47-01+00
2023-06-20 16:17:04,944 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+16-17-03+00
2023-06-20 16:17:05,128 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+16-02-02+00
2023-06-20 16:32:06,252 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+16-32-05+00
2023-06-20 16:32:06,431 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+16-17-03+00
2023-06-20 16:47:07,587 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+16-47-06+00
2023-06-20 16:47:07,770 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+16-32-05+00
2023-06-20 17:02:08,829 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+17-02-07+00
2023-06-20 17:02:09,010 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+16-47-06+00
2023-06-20 17:17:10,142 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+17-17-09+00
2023-06-20 17:17:10,334 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+17-02-07+00
2023-06-20 17:32:11,414 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+17-32-10+00
2023-06-20 17:32:11,599 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+17-17-09+00
2023-06-20 17:47:12,722 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+17-47-11+00
2023-06-20 17:47:12,924 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+17-32-10+00
2023-06-20 18:02:14,359 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+18-02-13+00
2023-06-20 18:02:14,554 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+17-47-11+00
2023-06-20 18:17:15,857 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+18-17-14+00
2023-06-20 18:17:16,059 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+18-02-13+00
2023-06-20 18:32:17,084 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+18-32-16+00
2023-06-20 18:32:17,282 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+18-17-14+00
2023-06-20 18:47:18,465 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+18-47-17+00
2023-06-20 18:47:18,676 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+18-32-16+00
2023-06-20 19:12:09,835 - speechbrain.utils.train_logger - INFO - epoch: 2, lr: 3.00e-04, wave2vec_lr: 1.00e-05 - train loss: 7.65e-01 - valid loss: 7.80e-01, valid CER: 21.44, valid WER: 26.45
2023-06-20 19:12:10,856 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+19-12-09+00
2023-06-20 19:12:11,073 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+15-16-58+00
2023-06-20 19:12:11,204 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+18-47-17+00
2023-06-20 19:12:11,204 - speechbrain.utils.epoch_loop - INFO - Going into epoch 3
2023-06-20 19:27:12,235 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+19-27-11+00
2023-06-20 19:42:13,555 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+19-42-12+00
2023-06-20 19:42:13,775 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+19-27-11+00
2023-06-20 19:57:14,794 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+19-57-13+00
2023-06-20 19:57:15,003 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+19-42-12+00
2023-06-20 20:12:16,231 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+20-12-15+00
2023-06-20 20:12:16,455 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+19-57-13+00
2023-06-20 20:27:17,655 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+20-27-16+00
2023-06-20 20:27:17,871 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+20-12-15+00
2023-06-20 20:42:19,139 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+20-42-18+00
2023-06-20 20:42:19,369 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+20-27-16+00
2023-06-20 20:57:20,596 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+20-57-19+00
2023-06-20 20:57:20,819 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+20-42-18+00
2023-06-20 21:12:21,941 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+21-12-20+00
2023-06-20 21:12:22,177 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+20-57-19+00
2023-06-20 21:27:23,343 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+21-27-22+00
2023-06-20 21:27:23,572 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+21-12-20+00
2023-06-20 21:42:24,843 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+21-42-23+00
2023-06-20 21:42:25,086 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+21-27-22+00
2023-06-20 21:57:26,240 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+21-57-25+00
2023-06-20 21:57:26,475 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+21-42-23+00
2023-06-20 22:12:27,713 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+22-12-26+00
2023-06-20 22:12:27,961 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+21-57-25+00
2023-06-20 22:27:29,044 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+22-27-28+00
2023-06-20 22:27:29,283 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+22-12-26+00
2023-06-20 22:42:30,339 - speechbrain.utils.checkpoints - DEBUG - Saved an intra-epoch checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+22-42-29+00
2023-06-20 22:42:30,587 - speechbrain.utils.checkpoints - DEBUG - Deleted checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+22-27-28+00
2023-06-20 23:03:13,097 - speechbrain.utils.train_logger - INFO - epoch: 3, lr: 3.00e-04, wave2vec_lr: 1.00e-05 - train loss: 7.56e-01 - valid loss: 7.76e-01, valid CER: 18.60, valid WER: 23.18
2023-06-20 23:03:14,127 - speechbrain.utils.checkpoints - INFO - Saved an end-of-epoch checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+23-03-13+00
2023-06-20 23:03:14,378 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+19-12-09+00
2023-06-20 23:03:14,503 - speechbrain.utils.checkpoints - INFO - Deleted checkpoint in /mnt/home/zhengbeida/path/slurp/results/lstm/1986/save/CKPT+2023-06-20+22-42-29+00
2023-06-20 23:03:14,503 - speechbrain.utils.epoch_loop - INFO - Going into epoch 4