I want to finetune this model with my own audio files, how can that be done? I've nearly 10 hours of data .

I want to finetune wav2vec 2.0 model with custom audio data. As there's no fine steps given how to implement that, I can't proceed. And then I need the model for audio-streaming and using it for Text-To-Speech recognition.

use wav2vec_manifest to build a manifest of your audio data
create a parallel file containing labels (see libri-labels.py for example format. in particular if using letters, you  need a word ending token (we use |) that is appended to the end of every word, and each symbol separated by space)
modify the example finetuning command to be more suitable to 10h. you can use the example command for 100h in the readme file and modify the parameters to what was used for 10h in the paper

Thanks for your insight @alexeib , will try this.
One more thing I need to ask is that is it possible to train my own unsupervised model with this wa2vec2.0 model. If I've like 10k hours of my audio files without any labels, can you suggest how can use your model to train from scratch with my own data

you can use the example pre-training command from the wav2vec readme to do your own pre-training

Even for pretraining or finetuning what I'm seeing that transcript file of all the audio files are needed, like which is available for Librispeech dataset. But the audio files which are with me, I don't have transcript for them, is it possible to train or finetune the model with audio files whose transcript file is not present

I have a similar question. Using three pretrained models, can I use those embeddings to further train some downstream task?
Specifically, I have raw audio that I want to convert to the embeddings. I want to then use those embeddings in a downstream task. How would I accomplish that?

use wav2vec_manifest to build a manifest of your audio data
create a parallel file containing labels (see libri-labels.py for example format. in particular if using letters, you  need a word ending token (we use |) that is appended to the end of every word, and each symbol separated by space)
modify the example finetuning command to be more suitable to 10h. you can use the example command for 100h in the readme file and modify the parameters to what was used for 10h in the paper

Don't you need a vocabulary file as well ? It is mentioned in the fine tuning section of Readme.

Even for pretraining or finetuning what I'm seeing that transcript file of all the audio files are needed, like which is available for Librispeech dataset. But the audio files which are with me, I don't have transcript for them, is it possible to train or finetune the model with audio files whose transcript file is not present

the step in create wav2vec pre trainning model not need transcription.
what you do just make sure the wav file in format 16k, 1 channel and already splitting.
base on wav2vec 2.0 README.md, they suggest wavfile lenght 15s to 30s. but in my experience, maximum 30s for the super computer spec because really consume GPU memory.
for me, i make the wav file 2s to 15s, it can train in google colab. and 2s to 6s in my pc with 4GB GPU memory.
i download the audio data from youtube, with ytmp3, chunk it base silence with tracehold setting that make sure the maximum lenght audio is can fit in GPU memory, convert mp3 to wav, with set freq to 16k and  1 channel.

CREATE PRE-TRAINED MODEL

step to do it:

put the wav that already have spec like i mention above in a specific directory, eg: wav_file.
create a new directory for save the manifest, eg: wav_manifest.
create a new directory to save the result, eg: w2v2_pre_train_model
run the wav2vec_manifest.pythat inside fairseq/examples/wav2vec directory. with this command (base wav2vec 2.0 README.md):
python3 'examples/wav2vec/wav2vec_manifest.py' '/path/to/wav_file' --dest 'path/to/wav_manifest' --ext wav
it will create the train.tsv and valid.tsv in your wav_manifest directory.
the start train to make pre-trainned model, by use the command in wav2vec README.md. i choose used the base model on 1 GPU, with command:

python3 fairseq/train.py path/to/wav_manifest \
--save-dir path/to/w2v2_pre_train_model --fp16 --num-workers 128 --task audio_pretraining --criterion wav2vec --arch wav2vec2 \
--log-keys '["prob_perplexity","code_perplexity","temp"]' --quantize-targets --extractor-mode default \
--conv-feature-layers '[(512, 10, 5)] + [(512, 3, 2)] * 4 + [(512,2,2)] * 2' --final-dim 256 --latent-vars 320 \
--latent-groups 2 --latent-temp '(2,0.5,0.999995)' --infonce --optimizer adam \
--adam-betas '(0.9,0.98)' --adam-eps 1e-06 --lr-scheduler polynomial_decay --total-num-update 400000 \
--lr 0.0005 --warmup-updates 32000 --mask-length 10 --mask-prob 0.65 --mask-selection static --mask-other 0 \
--encoder-layerdrop 0.05 --dropout-input 0.1 --dropout-features 0.1 --feature-grad-mult 0.1 \
--loss-weights '[0.1, 10]' --conv-pos 128 --conv-pos-groups 16 --num-negatives 100 --cross-sample-negatives 0 \
--max-sample-size 250000 --min-sample-size 32000 --dropout 0.1 --attention-dropout 0.1 --weight-decay 0.01 \
--max-tokens 1400000 --max-update 400000 --skip-invalid-size-inputs-valid-test --ddp-backend no_c10d

i choose number worker 128 because i train in google colab. if you use a home pc, reduce is base on your processor number - 3, eg: processor number 8, if we reduce 2 or 3, will be 5 or 6 worker. but a little warning here. it's make your pc really quite slow or laggy. if you want use it while working something else, use google colab or something similar, like kaggle or AWS. or reduce the number worker to 2 or 3.
in trainning to create pre-trainned model, in my experience, it stop automaticly when it reach max result. i train it 1 weeks in google colab, to do that i use 2 account that continues used in 12 hrs to exceed maximum used of GPU (12 hrs/day).
so i can still use my pc to do something else, without laggy.

after pre-trainned model create, the next step is to train it again (fine tuning method) with labeled wav.

FINE TUNING

i use dataset from mozila commonvoice, that have 8 hrs labelled wav. so start here you must prepare the wav audio that the number or the condition is base on your cpabilities to work on it. if you have 10k hrs, audio unlabelled. yes.. it's a hardwork if you want to make all audio labelled. but the benefit of using wav2vec 2.0, you not need have labelled audio that much, in research paper they used 58k hrs unlabelled audio to create pre-tainned model and use just 10 minutes labelled audio data to use in fine tuning.
in my assumption 58k hrs labelled data is already have all data that needed to understanding speech features and with 10 minutes labelled to fine-tuning, it's enough to map the features to label. if we not have that much unlabelled data, for another language, i think the labelled data percentage should be raised. if you use english, just use the model that already share by FB.
so, the step to prepare fine-tuning is:

make sure have labelled audio data in wav format, 16k, 1 channel.
put the label or transcripton file in the same folder in wav file. the transcription file format is:

file_name1.wav HI HOW ARE YOU
file_name1.wav THIS IS JUST A SAMPLE OF TRANSCRIPTION FORMAT
file_name3.wav THAT YOU SHOULD BUILD

save the transcription with format folder_name.trans.txt, illustration: i save the wav file in folder / directory named labelled_wav_file so the name of transcription file is labelled_wav_file.trans.txt and the content of the file like the example above.
you can use uppercase or lower case letter. but don't use both of it. just pick one type of it.

run the wav2vec_manifest.py again to produce train.tsv and valid.tsv file from labeled audio data.
python3 examples/wav2vec/wav2vec_manifest.py /path/to/labeled_wav_file --dest /labbeled_manifest/path --ext wav

after that, run twice the file libri_labels.py in fairseq/examples/wav2vec/ directory with comand:

python3 libri_labels.py /path/to/file/labelled_wav_file/train.tsv --output-dir /path/to/file/labelled_wav_file/ --output-name train
python3 libri_labels.py /path/to/file/labelled_wav_file/valid.tsv --output-dir /path/to/file/labelled_wav_file/ --output-name valid

that both command resulting file train.ltr, train.wrd, valid.ltr and valid.wrd. there is some error in run libri_labels.py because of the key in dictionary not match with the key that used in open the dictionay. but i already fix the code few hours ago. so better you download again the libri_labels.py file.

edit the file train.ltr, train.wrd, valid.ltr and valid.wrd to make sure the file only contains  A to Z character (if you use UPPERCASE letter) or a to z character (if you use lower case letter), space and '|' character.

create a new file named dict.ltr.txt. open it in text editor. then:

use find features in that text editor, I use sublime text, then I search the A character because I use uppercase letter. the text editor will show the number of 'A' character. record or write it.
do the same thing until all character count. i mean A to Z character and '|' character. then write it base the number of the character (ascending base number). example:

A 90280
| 78809
N 38268
I 33692
E 30160
K 24305
U 23955
M 20958
T 20551
S 18850
R 18418
D 15972
G 15711
L 13876
B 11768
P 11503
H 10845
Y 9570
O 6758
J 3993
C 2171
W 1118
F 463
V 187
Z 46
X 16
Q 6

then save it. remember the file should named as dict.ltr.txt.

create the lexicon.txt file. i use the train.wrd and valid.wrd to make the lexicon.txt. script i use and write by my self:

import os, codecs, re, pandas as pd
a = 'train.wrd'
b = 'valid.wrd'

df1 = pd.read_csv(a, header=None)
df2 = pd.read_csv(b, header=None)

df1.columns = ['raw']
df2.columns = ['raw']

df1 = df1.drop_duplicates('raw',keep='last')
df2 = df2.drop_duplicates('raw',keep='last')

sentence1 = df1['raw'].to_list()
sentence2 = df2['raw'].to_list()
sentence = sentence1 + sentence2

word = []
for x in sentence:
    tmp = x.split(' ')
    for y in tmp:
        if y not in word:
            word.append(y)

lexicon = []
for x in range(len(word)):
    wrd = word[x]
    temp = []
    for y in wrd:
        temp.append(y)
    result = ' '.join(temp) + ' |'
    lexicon.append(wrd + ' ' + result)

file_to_save = 'lexicon.txt'
f=codecs.open(file,'a+','utf8')
for x in lexicon:
    f.write(x+'\n')
f.close()

it will create file lexicon.txt that have format, something like this:
EVERY E V E R Y |
WORD W O R D |
THAT T H A T |
EXISTS E X I S T S |
IN I N |
YOUR Y O U R |
LABEL L A B E L |
OR O R |
TRANSCRIPTION T R A N S C R I P T I O N |
FILE F I L E |
WILL W I L L |
WRITE W R I T E |
DOWN D O W N |
LIKE L I K E |
THIS T H I S |

if you use lower case letter in transcription, that make the train.wrd and train.wrd, contains lower case letter., so it become:
every e v e r y |
word w o r d |
that t h a t |
etc ...

I use kenlm, so need .bin file, create it by follow the instruction in here, just follow the instruction from 19.50 until 20:49.

now you ready to fine-tuning...the command is base fine-tuning command README.md

valid_subset=train
python train.py --distributed-world-size 24 --distributed-port $PORT /path/to/training_data --save-dir /model/path --fp16 \
--wer-args '("/path/to/lm/4-gram.bin","/path/to/lexicon",2,-1)' \
--post-process letter --valid-subset $valid_subset --no-epoch-checkpoints --best-checkpoint-metric wer --num-workers 4 \
--max-update 80000 --sentence-avg --task audio_pretraining --arch wav2vec_ctc --w2v-path /path/to/pretrained/model \
--labels ltr --apply-mask --mask-selection static --mask-other 0 --mask-length 10 --mask-prob 0.5 --layerdrop 0.1 \
--mask-channel-selection static --mask-channel-other 0 --mask-channel-length 64 --mask-channel-prob 0.5 --zero-infinity \
--feature-grad-mult 0.0 --freeze-finetune-updates 10000 --validate-after-updates 10000 --optimizer adam \
--adam-betas '(0.9, 0.98)' --adam-eps 1e-08 --lr 2e-05 --lr-scheduler tri_stage --warmup-steps 8000 --hold-steps 32000 \
--decay-steps 40000 --final-lr-scale 0.05 --final-dropout 0.0 --dropout 0.0 --activation-dropout 0.1 --criterion ctc \
--attention-dropout 0.0 --max-tokens 1280000 --seed 2337 --log-format json --log-interval 500 --ddp-backend no_c10d

now, i want to asking here... because resulting error in run number 8. the error is:
File "/home/bram/Documents/coding/Speech/fairseq/train.py", line 14, in <module>
    cli_main()
  File "/home/bram/Documents/coding/Speech/fairseq/fairseq_cli/train.py", line 345, in cli_main
    distributed_utils.call_main(args, main)
  File "/home/bram/Documents/coding/Speech/fairseq/fairseq/distributed_utils.py", line 268, in call_main
    main(args, **kwargs)
  File "/home/bram/Documents/coding/Speech/fairseq/fairseq_cli/train.py", line 61, in main
    model = task.build_model(args)
  File "/home/bram/Documents/coding/Speech/fairseq/fairseq/tasks/fairseq_task.py", line 546, in build_model
    model = models.build_model(args, self)
  File "/home/bram/Documents/coding/Speech/fairseq/fairseq/models/__init__.py", line 57, in build_model
    return ARCH_MODEL_REGISTRY[model_cfg.arch].build_model(model_cfg, task)
  File "/home/bram/Documents/coding/Speech/fairseq/fairseq/models/wav2vec/wav2vec2_asr.py", line 168, in build_model
    w2v_encoder = Wav2VecEncoder(args, task.target_dictionary)
  File "/home/bram/Documents/coding/Speech/fairseq/fairseq/models/wav2vec/wav2vec2_asr.py", line 331, in __init__
    args.w2v_path, arg_overrides
  File "/home/bram/Documents/coding/Speech/fairseq/fairseq/checkpoint_utils.py", line 211, in load_checkpoint_to_cpu
    setattr(args, arg_name, arg_val)
AttributeError: 'NoneType' object has no attribute 'dropout'

the command i use is:
python3 train.py '/home/bram/Documents/coding/speech/traindata/text_label' \
--save-dir '/home/bram/Documents/coding/speech/traindata/model_finetuning_wav2vec' --fp16
--wer-args '("/home/bram/Documents/coding/speech/traindata/text_label/lm.bin","/home/bram/Documents/coding/speech/traindata/text_label/lexicon.txt",2,-1)' \
--post-process letter --valid-subset valid --no-epoch-checkpoints --best-checkpoint-metric wer --num-workers 128 \
--max-update 400000 --sentence-avg --task audio_pretraining --arch wav2vec_ctc \
--w2v-path '/home/bram/Documents/coding/speech/traindata/w2v2_pre_traned_model/checkpoint_best.pt' \
--labels ltr --apply-mask --mask-selection static --mask-other 0 --mask-length 10 --mask-prob 0.5 --layerdrop 0.1 \
--mask-channel-selection static --mask-channel-other 0 --mask-channel-length 64 --mask-channel-prob 0.5 --zero-infinity \
--feature-grad-mult 0.0 --freeze-finetune-updates 10000 --validate-after-updates 10000 --optimizer adam \
--adam-betas '(0.9, 0.98)' --adam-eps 1e-08 --lr 2e-05 --lr-scheduler tri_stage --warmup-steps 8000 --hold-steps 32000 \
--decay-steps 40000 --final-lr-scale 0.05 --final-dropout 0.0 --dropout 0.0 --activation-dropout 0.1 --criterion ctc \
--attention-dropout 0.0 --max-tokens 1280000 --seed 2337 --log-format json --log-interval 500 --ddp-backend no_c10d

the folder /home/bram/Documents/coding/speech/traindata/text_label contains:
1. dict.ltr.txt
2. lexicon.txt
3. lm.bin
4. train.tsv
5. train.wrd
6. train.ltr
7. valid.tsv
8. valid.wrd
9. valid.ltr

the folder /home/bram/Documents/coding/speech/traindata/model_finetuning_wav2vec is empty means to save the finetuning model that resulting in finetuning process
the folder /home/bram/Documents/coding/speech/traindata/w2v2_pre_traned_model/ is contained:
1. checkpoint_best.pt
2. checkpoint_last.pt

this file resulting from pre-trainned process..
I try to debugging, by folllow the process, step by step, and found that error is happened:
fairseq/train.py -> call the file fairseq/fairseq_cli/train.py
the error happends in line 61 file fairseq/fairseq_cli/train.py, when the script try to create variabel model ,with command
model = task.build_model(args)
the args variable self are the argsparser which contains:
Namespace(activation_dropout=0.1, adam_betas='(0.9, 0.98)', adam_eps=1e-08, all_gather_list_size=16384, apply_mask=True, arch='wav2vec_ctc', attention_dropout=0.0, batch_size=None, batch_size_valid=None, best_checkpoint_metric='wer', bf16=False, bpe=None, broadcast_buffers=False, bucket_cap_mb=25, checkpoint_shard_count=1, checkpoint_suffix='', clip_norm=25.0, cpu=False, criterion='ctc', curriculum=0, data='/home/bram/Documents/coding/speech/traindata/sukubase/text/hasil', data_buffer_size=10, dataset_impl=None, ddp_backend='no_c10d', decay_steps=40000, device_id=0, disable_validation=False, distributed_backend='nccl', distributed_init_method=None, distributed_no_spawn=False, distributed_port=-1, distributed_rank=0, distributed_world_size=1, distributed_wrapper='DDP', dropout=0.0, dropout_input=0, empty_cache_freq=0, enable_padding=False, fast_stat_sync=False, feature_grad_mult=0.0, final_dropout=0.0, final_lr_scale=0.05, find_unused_parameters=False, finetune_from_model=None, fix_batches_to_gpus=False, fixed_validation_seed=None, fp16=True, fp16_init_scale=128, fp16_no_flatten_grads=False, fp16_scale_tolerance=0.0, fp16_scale_window=None, freeze_finetune_updates=10000, gen_subset='test', hold_steps=32000, init_lr_scale=0.01, keep_best_checkpoints=-1, keep_interval_updates=-1, keep_last_epochs=-1, labels='ltr', layerdrop=0.1, local_rank=0, localsgd_frequency=3, log_format='json', log_interval=500, lr=[2e-05], lr_scheduler='tri_stage', mask_channel_length=64, mask_channel_other=0.0, mask_channel_prob=0.5, mask_channel_selection='static', mask_length=10, mask_other=0.0, mask_prob=0.5, mask_selection='static', max_epoch=0, max_sample_size=None, max_tokens=1280000, max_tokens_valid=1280000, max_update=400000, maximize_best_checkpoint_metric=False, memory_efficient_bf16=False, memory_efficient_fp16=False, min_loss_scale=0.0001, min_lr=-1.0, min_sample_size=None, model_parallel_size=1, no_epoch_checkpoints=True, no_last_checkpoints=False, no_mask_channel_overlap=False, no_mask_overlap=False, no_pretrained_weights=False, no_progress_bar=False, no_save=False, no_save_optimizer_state=False, no_seed_provided=False, normalize=False, nprocs_per_node=1, num_shards=1, num_workers=4, optimizer='adam', optimizer_overrides='{}', patience=-1, pipeline_balance=None, pipeline_checkpoint='never', pipeline_chunks=0, pipeline_decoder_balance=None, pipeline_decoder_devices=None, pipeline_devices=None, pipeline_encoder_balance=None, pipeline_encoder_devices=None, pipeline_model_parallel=False, profile=False, quantization_config_path=None, remove_bpe='letter', required_batch_size_multiple=8, required_seq_len_multiple=1, reset_dataloader=False, reset_lr_scheduler=False, reset_meters=False, reset_optimizer=False, restore_file='checkpoint_last.pt', sample_rate=16000, save_dir='/home/bram/Documents/coding/speech/traindata/cvdata/ori2/model/finetuning_wav2vec', save_interval=1, save_interval_updates=0, scoring='bleu', seed=2337, sentence_avg=True, shard_id=0, skip_invalid_size_inputs_valid_test=False, slowmo_algorithm='LocalSGD', slowmo_momentum=None, stop_time_hours=0, task='audio_pretraining', tensorboard_logdir=None, threshold_loss_scale=None, tokenizer=None, tpu=False, train_subset='train', update_freq=[1], use_bmuf=False, use_old_adam=False, user_dir=None, valid_subset='valid', validate_after_updates=10000, validate_interval=1, validate_interval_updates=0, w2v_path='/home/bram/Documents/coding/speech/traindata/cvdata/ori2/model/wav2vec2/checkpoint_best.pt', warmup_steps=8000, weight_decay=0.0, wer_args='("/home/bram/Documents/coding/speech/traindata/sukubase/text/hasil/lm.bin","/home/bram/Documents/coding/speech/traindata/sukubase/text/hasil/lexicon.txt",2,-1)', zero_infinity=True, zero_sharding='none')
just wondering, is the command that i used, which it come from README.md is false or not?
in my logic, if the command  task.build_model(args), it should be have several parameter that can recognize, which in my case, i think, it not recognize.
can somebody help? if I can fix this I will continues the tutorial.

@wahyubram82 , thanks for the explanatory steps. I have one doubt that, while you were finetuning, was your whole data was labeled or you used both labeled and unlabelled data together for finetuning.

Hey @wahyubram82
I followed your tutorial, it is very useful. Thank you!
I managed to start fine-tuning with pretrained checkpoint. Actually I didn't encounter any problems loading checkpoint. Maybe you can try, according to pytorch documentation, just state = torch.load(/path/to/checkpoint.pt, map_location=lambda s, l: default_restore_location(s, "cpu")) ?

default_restore_location

the error because the train.py in fairseq do not have order to write the arg that we use if we build custom dataset, so we must add it by our self. I already write the script to do that in my answer above....to help any other people tht have similar problem. now I'm stuck in decoding process with kenlm. in inserting word to trie.

I am in the same problem like @wahyubram82.
@alexeib Can you help me?

i am afraid i cant help with wav2letter decoder problems. might be better to ask in wav2letter github
to unblock, you can try making your own viterbi decoder (look in ctc criterion for an example of how to do arg-max decoding) and maybe try to integrate with kaldi decoder if you want (PRs welcome)

i am afraid i cant help with wav2letter decoder problems. might be better to ask in wav2letter github
to unblock, you can try making your own viterbi decoder (look in ctc criterion for an example of how to do arg-max decoding) and maybe try to integrate with kaldi decoder if you want (PRs welcome)

@alexeib I solved this problem. This problem may cause by kenlm install command.
Here is my install command.
sudo apt install -y build-essential cmake libboost-system-dev libboost-thread-dev libboost-program-options-dev libboost-test-dev libeigen3-dev zlib1g-dev libbz2-dev liblzma-dev
git clone https://github.com/kpu/kenlm.git
cd kenlm
mkdir -p build
cd build
cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_POSITION_INDEPENDENT_CODE=ON
make -j 16
export KENLM_ROOT_DIR=$ABSOLUTE_PATH'/kenlm/'
cd ../..

# Install Additional Dependencies (ATLAS, OpenBLAS, Accelerate, Intel MKL)
apt-get install -y libsndfile1-dev libopenblas-dev libfftw3-dev libgflags-dev libgoogle-glog-dev

# Install wav2letter
git clone -b v0.2 https://github.com/facebookresearch/wav2letter.git
cd wav2letter/bindings/python
pip install -e .
cd ../../..

@alexeib I solved this problem. This problem may cause by kenlm install command.
Here is my install command.
sudo apt install -y build-essential cmake libboost-system-dev libboost-thread-dev libboost-program-options-dev libboost-test-dev libeigen3-dev zlib1g-dev libbz2-dev liblzma-dev
git clone https://github.com/kpu/kenlm.git
cd kenlm
mkdir -p build
cd build
cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_POSITION_INDEPENDENT_CODE=ON
make -j 16
export KENLM_ROOT_DIR=$ABSOLUTE_PATH'/kenlm/'
cd ../..

# Install Additional Dependencies (ATLAS, OpenBLAS, Accelerate, Intel MKL)
apt-get install -y libsndfile1-dev libopenblas-dev libfftw3-dev libgflags-dev libgoogle-glog-dev

# Install wav2letter
git clone -b v0.2 https://github.com/facebookresearch/wav2letter.git
cd wav2letter/bindings/python
pip install -e .
cd ../../..

@sooftware thank you for your share.
I've managed to overcome corrupted double-linked list and Segmentation fault (core dumped) just by remove kenlm build option -DKENLM_MAX_ORDER=20.

@sooftware thank you for your share.
I've managed to overcome corrupted double-linked list and Segmentation fault (core dumped) just by remove kenlm build option -DKENLM_MAX_ORDER=20.

@osddeitf Can you explain more detail??

@osddeitf Can you explain more detail??

Previously, I build wav2letter following the instructions at its Wiki, specifically in Dependencies wiki.
The command for building KenLM are:
cmake .. -DCMAKE_BUILD_TYPE=Release -DKENLM_MAX_ORDER=20 -DCMAKE_POSITION_INDEPENDENT_CODE=ON

Now, I simplified it just a little bit:
cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_POSITION_INDEPENDENT_CODE=ON

I've used 4-gram.arpa.gz and it did work after I adjusted the KenML build command.

Previously, I build wav2letter following the instructions at its Wiki, specifically in Dependencies wiki.
The command for building KenLM are:
cmake .. -DCMAKE_BUILD_TYPE=Release -DKENLM_MAX_ORDER=20 -DCMAKE_POSITION_INDEPENDENT_CODE=ON

Now, I simplified it just a little bit:
cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_POSITION_INDEPENDENT_CODE=ON

I've used 4-gram.arpa.gz and it did work after I adjusted the KenML build command.

@sooftware btw, have you successfully finetuned a model?
I have tried follow the instructions at README.md, it use Hydra.
I'm finetuned using LibriSpeech and generate train.tsv, train.wrd, train.ltr, as well as valid.* files.
In first attempts, I'm not using KenLM, because the configurations in Hydra are not.
And now, I've managed to finetuning using KenLM, but still got weird results.
Using checkpoint10.pt:
$ python recognize.hydra.py --target_dict_path /dataset/LibriSpeech/dict.ltr.txt /outputs/2021-01-05/21-13-51/checkpoints/checkpoint10.pt /dataset/LibriSpeech/test-clean/1089/134686/1089-134686-0001.flac
/usr/local/lib/python3.8/site-packages/torch/cuda/__init__.py:52: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx (Triggered internally at  /pytorch/c10/cuda/CUDAFunctions.cpp:100.)
  return torch._C._cuda_getDeviceCount() > 0
O

Using checkpoint20.pt, or checkpoint30.pt, and so on,...
$ python recognize.hydra.py --target_dict_path /dataset/LibriSpeech/dict.ltr.txt /outputs/2021-01-05/21-13-51/checkpoints/checkpoint110.pt /dataset/LibriSpeech/test-clean/1089/134686/1089-134686-0001.flac
/usr/local/lib/python3.8/site-packages/torch/cuda/__init__.py:52: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx (Triggered internally at  /pytorch/c10/cuda/CUDAFunctions.cpp:100.)
  return torch._C._cuda_getDeviceCount() > 0

Can you see the empty line? It's just predict empty string.
May I've not train enough? I guessed so but with checkpoint110.pt above, it was not work either.
May I missed something.
But I've trying run diff through old command line --argument and new Hydra configs, see nothing special changed, but new configs not using KenLM.
Information

Dataset: LibriSpeech - dev-clean.tar.gz and test-clean.tar.gz.
recognize.hydra.py: source
Configurations: base_1h.yaml  with only checkpoint and distribution_training changed.
Commands: python fairseq_cli/hydra_train.py      task.labels=ltr         task.data=/dataset/LibriSpeech         model.w2v_path=/dataset/libri960_big.pt         --config-dir /config/         --config-name base_1h &

    Train logs

```
2021-01-05 21:15:37 | INFO | fairseq_cli.train | end of epoch 1 (average epoch stats below)
2021-01-05 21:15:37 | INFO | train | {"epoch": 1, "train_loss": "1266.85", "train_ntokens": "9291.54", "train_nsentences": "90.8333", "train_nll_loss": "12.385", "train_wps": "3218.5", "train_ups": "0.35", "train_wpb": "9291.5", "train_bsz": "90.8", "train_num_updates": "24", "train_lr": "1.41385e-06", "train_gnorm": "1157.04", "train_loss_scale": "1", "train_train_wall": "88", "train_wall": "89"}
2021-01-05 21:15:37 | INFO | fairseq.trainer | begin training epoch 2
2021-01-05 21:17:13 | INFO | fairseq_cli.train | end of epoch 2 (average epoch stats below)
2021-01-05 21:17:13 | INFO | train | {"epoch": 2, "train_loss": "1322.8", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "12.28", "train_wps": "3031.4", "train_ups": "0.32", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "55", "train_lr": "2.59423e-06", "train_gnorm": "1261.85", "train_loss_scale": "1", "train_train_wall": "95", "train_wall": "185"}
2021-01-05 21:17:13 | INFO | fairseq.trainer | begin training epoch 3
2021-01-05 21:18:52 | INFO | fairseq_cli.train | end of epoch 3 (average epoch stats below)
2021-01-05 21:18:52 | INFO | train | {"epoch": 3, "train_loss": "1315.98", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "12.217", "train_wps": "2941.3", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "86", "train_lr": "3.77462e-06", "train_gnorm": "1417.01", "train_loss_scale": "1", "train_train_wall": "98", "train_wall": "284"}
2021-01-05 21:18:52 | INFO | fairseq.trainer | begin training epoch 4
2021-01-05 21:20:30 | INFO | fairseq_cli.train | end of epoch 4 (average epoch stats below)
2021-01-05 21:20:30 | INFO | train | {"epoch": 4, "train_loss": "1303.25", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "12.099", "train_wps": "2947.6", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "117", "train_lr": "4.955e-06", "train_gnorm": "1388.31", "train_loss_scale": "1", "train_train_wall": "98", "train_wall": "383"}
2021-01-05 21:20:30 | INFO | fairseq.trainer | begin training epoch 5
2021-01-05 21:22:09 | INFO | fairseq_cli.train | end of epoch 5 (average epoch stats below)
2021-01-05 21:22:09 | INFO | train | {"epoch": 5, "train_loss": "1289.98", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "11.976", "train_wps": "2938.7", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "148", "train_lr": "6.13538e-06", "train_gnorm": "1369.71", "train_loss_scale": "1", "train_train_wall": "98", "train_wall": "482"}
2021-01-05 21:22:09 | INFO | fairseq.trainer | begin training epoch 6
2021-01-05 21:23:49 | INFO | fairseq_cli.train | end of epoch 6 (average epoch stats below)
2021-01-05 21:23:49 | INFO | train | {"epoch": 6, "train_loss": "1275.11", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "11.838", "train_wps": "2927.6", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "179", "train_lr": "7.31577e-06", "train_gnorm": "1282.1", "train_loss_scale": "1", "train_train_wall": "99", "train_wall": "581"}
2021-01-05 21:23:49 | INFO | fairseq.trainer | begin training epoch 7
2021-01-05 21:24:57 | INFO | train_inner | {"epoch": 7, "update": 6.677, "loss": "1290.3", "ntokens": "9370.92", "nsentences": "87.75", "nll_loss": "12.082", "wps": "2974.2", "ups": "0.32", "wpb": "9370.9", "bsz": "87.8", "num_updates": "200", "lr": "8.11538e-06", "gnorm": "1328.48", "loss_scale": "1", "train_wall": "644", "wall": "650"}
2021-01-05 21:25:29 | INFO | fairseq_cli.train | end of epoch 7 (average epoch stats below)
2021-01-05 21:25:29 | INFO | train | {"epoch": 7, "train_loss": "1251.74", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "11.621", "train_wps": "2916.1", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "210", "train_lr": "8.49615e-06", "train_gnorm": "1423.09", "train_loss_scale": "1", "train_train_wall": "99", "train_wall": "681"}
2021-01-05 21:25:29 | INFO | fairseq.trainer | begin training epoch 8
2021-01-05 21:26:37 | INFO | fairseq.trainer | NOTE: overflow detected, setting loss scale to: 0.5
2021-01-05 21:27:08 | INFO | fairseq_cli.train | end of epoch 8 (average epoch stats below)
2021-01-05 21:27:08 | INFO | train | {"epoch": 8, "train_loss": "1268.35", "train_ntokens": "9364.87", "train_nsentences": "84.2333", "train_nll_loss": "11.408", "train_wps": "2822.7", "train_ups": "0.3", "train_wpb": "9364.9", "train_bsz": "84.2", "train_num_updates": "240", "train_lr": "9.63846e-06", "train_gnorm": "1432.09", "train_loss_scale": "0.5", "train_train_wall": "99", "train_wall": "781"}
2021-01-05 21:27:08 | INFO | fairseq.trainer | begin training epoch 9
2021-01-05 21:28:49 | INFO | fairseq_cli.train | end of epoch 9 (average epoch stats below)
2021-01-05 21:28:49 | INFO | train | {"epoch": 9, "train_loss": "1207.49", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "11.21", "train_wps": "2903", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "271", "train_lr": "1.08188e-05", "train_gnorm": "1339.77", "train_loss_scale": "0.5", "train_train_wall": "100", "train_wall": "881"}
2021-01-05 21:28:49 | INFO | fairseq.trainer | begin training epoch 10
2021-01-05 21:30:27 | INFO | fairseq_cli.train | begin validation on "valid" subset
2021-01-05 21:32:31 | INFO | valid | {"epoch": 10, "valid_loss": "986.911", "valid_ntokens": "2273.2", "valid_nsentences": "20.96", "valid_nll_loss": "9.1", "valid_uer": "98.972", "valid_wer": "99.981", "valid_raw_wer": "99.981", "valid_wps": "2310.5", "valid_wpb": "2273.2", "valid_bsz": "21", "valid_num_updates": "302"}
2021-01-05 21:32:31 | INFO | fairseq.checkpoint_utils | Preparing to save checkpoint for epoch 10 @ 302 updates
2021-01-05 21:32:52 | INFO | fairseq.trainer | Finished saving checkpoint to checkpoints/checkpoint10.pt
2021-01-05 21:33:53 | INFO | fairseq.checkpoint_utils | Saved checkpoint checkpoints/checkpoint10.pt (epoch 10 @ 302 updates, score 99.981) (writing took 81.33670563600003 seconds)
2021-01-05 21:33:53 | INFO | fairseq_cli.train | end of epoch 10 (average epoch stats below)
2021-01-05 21:33:53 | INFO | train | {"epoch": 10, "train_loss": "1184.48", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "10.996", "train_wps": "956.9", "train_ups": "0.1", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "302", "train_lr": "1.19992e-05", "train_gnorm": "1266.52", "train_loss_scale": "0.5", "train_train_wall": "98", "train_wall": "1185"}
2021-01-05 21:33:53 | INFO | fairseq.trainer | begin training epoch 11
2021-01-05 21:35:31 | INFO | fairseq_cli.train | end of epoch 11 (average epoch stats below)
2021-01-05 21:35:31 | INFO | train | {"epoch": 11, "train_loss": "1143.88", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "10.619", "train_wps": "2958.4", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "333", "train_lr": "1.31796e-05", "train_gnorm": "1387.51", "train_loss_scale": "0.5", "train_train_wall": "97", "train_wall": "1284"}
2021-01-05 21:35:31 | INFO | fairseq.trainer | begin training epoch 12
2021-01-05 21:37:10 | INFO | fairseq_cli.train | end of epoch 12 (average epoch stats below)
2021-01-05 21:37:10 | INFO | train | {"epoch": 12, "train_loss": "1115.62", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "10.357", "train_wps": "2950.5", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "364", "train_lr": "1.436e-05", "train_gnorm": "1375.05", "train_loss_scale": "0.5", "train_train_wall": "98", "train_wall": "1382"}
2021-01-05 21:37:10 | INFO | fairseq.trainer | begin training epoch 13
2021-01-05 21:38:48 | INFO | fairseq_cli.train | end of epoch 13 (average epoch stats below)
2021-01-05 21:38:48 | INFO | train | {"epoch": 13, "train_loss": "1096.54", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "10.18", "train_wps": "2965.8", "train_ups": "0.32", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "395", "train_lr": "1.55404e-05", "train_gnorm": "1254.28", "train_loss_scale": "0.5", "train_train_wall": "97", "train_wall": "1481"}
2021-01-05 21:38:48 | INFO | fairseq.trainer | begin training epoch 14
2021-01-05 21:39:04 | INFO | train_inner | {"epoch": 14, "update": 13.161, "loss": "1174.06", "ntokens": "9396.51", "nsentences": "86.41", "nll_loss": "10.797", "wps": "2219.4", "ups": "0.24", "wpb": "9396.5", "bsz": "86.4", "num_updates": "400", "lr": "1.57308e-05", "gnorm": "1346.79", "loss_scale": "0.5", "train_wall": "636", "wall": "1496"}
2021-01-05 21:40:25 | INFO | fairseq_cli.train | end of epoch 14 (average epoch stats below)
2021-01-05 21:40:25 | INFO | train | {"epoch": 14, "train_loss": "1057.17", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "9.814", "train_wps": "2988.9", "train_ups": "0.32", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "426", "train_lr": "1.67208e-05", "train_gnorm": "1320.55", "train_loss_scale": "0.5", "train_train_wall": "97", "train_wall": "1578"}
2021-01-05 21:40:26 | INFO | fairseq.trainer | begin training epoch 15
2021-01-05 21:42:04 | INFO | fairseq_cli.train | end of epoch 15 (average epoch stats below)
2021-01-05 21:42:04 | INFO | train | {"epoch": 15, "train_loss": "1049.59", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "9.744", "train_wps": "2969", "train_ups": "0.32", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "457", "train_lr": "1.79012e-05", "train_gnorm": "1282.37", "train_loss_scale": "0.5", "train_train_wall": "97", "train_wall": "1676"}
2021-01-05 21:42:04 | INFO | fairseq.trainer | begin training epoch 16
2021-01-05 21:43:41 | INFO | fairseq_cli.train | end of epoch 16 (average epoch stats below)
2021-01-05 21:43:41 | INFO | train | {"epoch": 16, "train_loss": "982.195", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "9.118", "train_wps": "2975.1", "train_ups": "0.32", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "488", "train_lr": "1.90815e-05", "train_gnorm": "1583.89", "train_loss_scale": "0.5", "train_train_wall": "97", "train_wall": "1774"}
2021-01-05 21:43:42 | INFO | fairseq.trainer | begin training epoch 17
2021-01-05 21:45:19 | INFO | fairseq_cli.train | end of epoch 17 (average epoch stats below)
2021-01-05 21:45:19 | INFO | train | {"epoch": 17, "train_loss": "949.922", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "8.819", "train_wps": "2973.3", "train_ups": "0.32", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "519", "train_lr": "2.02619e-05", "train_gnorm": "1651.61", "train_loss_scale": "0.5", "train_train_wall": "97", "train_wall": "1872"}
2021-01-05 21:45:19 | INFO | fairseq.trainer | begin training epoch 18
2021-01-05 21:46:57 | INFO | fairseq_cli.train | end of epoch 18 (average epoch stats below)
2021-01-05 21:46:57 | INFO | train | {"epoch": 18, "train_loss": "904.52", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "8.397", "train_wps": "2970.1", "train_ups": "0.32", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "550", "train_lr": "2.14423e-05", "train_gnorm": "1727.15", "train_loss_scale": "0.5", "train_train_wall": "97", "train_wall": "1970"}
2021-01-05 21:46:57 | INFO | fairseq.trainer | begin training epoch 19
2021-01-05 21:48:35 | INFO | fairseq_cli.train | end of epoch 19 (average epoch stats below)
2021-01-05 21:48:35 | INFO | train | {"epoch": 19, "train_loss": "869.769", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "8.075", "train_wps": "2975", "train_ups": "0.32", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "581", "train_lr": "2.26227e-05", "train_gnorm": "1444.59", "train_loss_scale": "0.5", "train_train_wall": "97", "train_wall": "2068"}
2021-01-05 21:48:35 | INFO | fairseq.trainer | begin training epoch 20
2021-01-05 21:49:35 | INFO | train_inner | {"epoch": 20, "update": 19.613, "loss": "956.214", "ntokens": "9375.11", "nsentences": "86.935", "nll_loss": "8.867", "wps": "2973", "ups": "0.32", "wpb": "9375.1", "bsz": "86.9", "num_updates": "600", "lr": "2.33462e-05", "gnorm": "1493.15", "loss_scale": "0.5", "train_wall": "626", "wall": "2127"}
2021-01-05 21:50:13 | INFO | fairseq_cli.train | begin validation on "valid" subset
2021-01-05 21:52:17 | INFO | valid | {"epoch": 20, "valid_loss": "551.863", "valid_ntokens": "2273.2", "valid_nsentences": "20.96", "valid_nll_loss": "5.088", "valid_uer": "99.965", "valid_wer": "99.996", "valid_raw_wer": "99.996", "valid_wps": "2319.7", "valid_wpb": "2273.2", "valid_bsz": "21", "valid_num_updates": "612", "valid_best_wer": "99.981"}
2021-01-05 21:52:17 | INFO | fairseq.checkpoint_utils | Preparing to save checkpoint for epoch 20 @ 612 updates
2021-01-05 21:52:37 | INFO | fairseq.trainer | Finished saving checkpoint to checkpoints/checkpoint20.pt
2021-01-05 21:53:18 | INFO | fairseq.checkpoint_utils | Saved checkpoint checkpoints/checkpoint20.pt (epoch 20 @ 612 updates, score 99.996) (writing took 61.534299623999686 seconds)
2021-01-05 21:53:18 | INFO | fairseq_cli.train | end of epoch 20 (average epoch stats below)
2021-01-05 21:53:18 | INFO | train | {"epoch": 20, "train_loss": "829.235", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "7.698", "train_wps": "1028.8", "train_ups": "0.11", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "612", "train_lr": "2.38031e-05", "train_gnorm": "1294.73", "train_loss_scale": "0.5", "train_train_wall": "97", "train_wall": "2351"}
2021-01-05 21:53:18 | INFO | fairseq.trainer | begin training epoch 21
2021-01-05 21:54:56 | INFO | fairseq_cli.train | end of epoch 21 (average epoch stats below)
2021-01-05 21:54:56 | INFO | train | {"epoch": 21, "train_loss": "816.933", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "7.584", "train_wps": "2990.8", "train_ups": "0.32", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "643", "train_lr": "2.49835e-05", "train_gnorm": "1000.72", "train_loss_scale": "0.5", "train_train_wall": "97", "train_wall": "2448"}
2021-01-05 21:54:56 | INFO | fairseq.trainer | begin training epoch 22
2021-01-05 21:56:35 | INFO | fairseq_cli.train | end of epoch 22 (average epoch stats below)
2021-01-05 21:56:35 | INFO | train | {"epoch": 22, "train_loss": "788.127", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "7.317", "train_wps": "2919.9", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "674", "train_lr": "2.61638e-05", "train_gnorm": "855.085", "train_loss_scale": "0.5", "train_train_wall": "99", "train_wall": "2548"}
2021-01-05 21:56:35 | INFO | fairseq.trainer | begin training epoch 23
2021-01-05 21:58:15 | INFO | fairseq_cli.train | end of epoch 23 (average epoch stats below)
2021-01-05 21:58:15 | INFO | train | {"epoch": 23, "train_loss": "766.352", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "7.114", "train_wps": "2917.6", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "705", "train_lr": "2.73442e-05", "train_gnorm": "678.209", "train_loss_scale": "0.5", "train_train_wall": "99", "train_wall": "2648"}
2021-01-05 21:58:15 | INFO | fairseq.trainer | begin training epoch 24
2021-01-05 21:59:54 | INFO | fairseq_cli.train | end of epoch 24 (average epoch stats below)
2021-01-05 21:59:54 | INFO | train | {"epoch": 24, "train_loss": "738.264", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "6.854", "train_wps": "2943.2", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "736", "train_lr": "2.85246e-05", "train_gnorm": "525.761", "train_loss_scale": "0.5", "train_train_wall": "98", "train_wall": "2747"}
2021-01-05 21:59:54 | INFO | fairseq.trainer | begin training epoch 25
2021-01-05 22:01:34 | INFO | fairseq_cli.train | end of epoch 25 (average epoch stats below)
2021-01-05 22:01:34 | INFO | train | {"epoch": 25, "train_loss": "706.879", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "6.562", "train_wps": "2920.5", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "767", "train_lr": "2.9705e-05", "train_gnorm": "443.124", "train_loss_scale": "0.5", "train_train_wall": "99", "train_wall": "2846"}
2021-01-05 22:01:34 | INFO | fairseq.trainer | begin training epoch 26
2021-01-05 22:03:13 | INFO | fairseq_cli.train | end of epoch 26 (average epoch stats below)
2021-01-05 22:03:13 | INFO | train | {"epoch": 26, "train_loss": "708.982", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "6.582", "train_wps": "2922.1", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "798", "train_lr": "3.08854e-05", "train_gnorm": "366.901", "train_loss_scale": "0.5", "train_train_wall": "99", "train_wall": "2946"}
2021-01-05 22:03:13 | INFO | fairseq.trainer | begin training epoch 27
2021-01-05 22:03:20 | INFO | train_inner | {"epoch": 27, "update": 26.065, "loss": "757.738", "ntokens": "9411.24", "nsentences": "87.51", "nll_loss": "7.046", "wps": "2279.2", "ups": "0.24", "wpb": "9411.2", "bsz": "87.5", "num_updates": "800", "lr": "3.09615e-05", "gnorm": "673.919", "loss_scale": "0.5", "train_wall": "635", "wall": "2953"}
2021-01-05 22:04:53 | INFO | fairseq_cli.train | end of epoch 27 (average epoch stats below)
2021-01-05 22:04:53 | INFO | train | {"epoch": 27, "train_loss": "740.851", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "6.878", "train_wps": "2926.6", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "829", "train_lr": "3.20658e-05", "train_gnorm": "255.979", "train_loss_scale": "0.5", "train_train_wall": "99", "train_wall": "3045"}
2021-01-05 22:04:53 | INFO | fairseq.trainer | begin training epoch 28
2021-01-05 22:06:32 | INFO | fairseq_cli.train | end of epoch 28 (average epoch stats below)
2021-01-05 22:06:32 | INFO | train | {"epoch": 28, "train_loss": "676.575", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "6.281", "train_wps": "2936.7", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "860", "train_lr": "3.32462e-05", "train_gnorm": "231.511", "train_loss_scale": "0.5", "train_train_wall": "98", "train_wall": "3145"}
2021-01-05 22:06:32 | INFO | fairseq.trainer | begin training epoch 29
2021-01-05 22:08:11 | INFO | fairseq_cli.train | end of epoch 29 (average epoch stats below)
2021-01-05 22:08:11 | INFO | train | {"epoch": 29, "train_loss": "680.784", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "6.32", "train_wps": "2937.9", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "891", "train_lr": "3.44265e-05", "train_gnorm": "212.375", "train_loss_scale": "0.5", "train_train_wall": "98", "train_wall": "3244"}
2021-01-05 22:08:11 | INFO | fairseq.trainer | begin training epoch 30
2021-01-05 22:09:50 | INFO | fairseq_cli.train | begin validation on "valid" subset
2021-01-05 22:11:55 | INFO | valid | {"epoch": 30, "valid_loss": "726.905", "valid_ntokens": "2273.2", "valid_nsentences": "20.96", "valid_nll_loss": "6.702", "valid_uer": "99.982", "valid_wer": "100", "valid_raw_wer": "100", "valid_wps": "2300.2", "valid_wpb": "2273.2", "valid_bsz": "21", "valid_num_updates": "922", "valid_best_wer": "99.981"}
2021-01-05 22:11:55 | INFO | fairseq.checkpoint_utils | Preparing to save checkpoint for epoch 30 @ 922 updates
2021-01-05 22:12:16 | INFO | fairseq.trainer | Finished saving checkpoint to checkpoints/checkpoint30.pt
2021-01-05 22:12:57 | INFO | fairseq.checkpoint_utils | Saved checkpoint checkpoints/checkpoint30.pt (epoch 30 @ 922 updates, score 100.0) (writing took 61.680051468999864 seconds)
2021-01-05 22:12:57 | INFO | fairseq_cli.train | end of epoch 30 (average epoch stats below)
2021-01-05 22:12:57 | INFO | train | {"epoch": 30, "train_loss": "671.817", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "6.237", "train_wps": "1019.7", "train_ups": "0.11", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "922", "train_lr": "3.56069e-05", "train_gnorm": "179.072", "train_loss_scale": "0.5", "train_train_wall": "99", "train_wall": "3529"}
2021-01-05 22:12:57 | INFO | fairseq.trainer | begin training epoch 31
2021-01-05 22:14:34 | INFO | fairseq_cli.train | end of epoch 31 (average epoch stats below)
2021-01-05 22:14:34 | INFO | train | {"epoch": 31, "train_loss": "685.864", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "6.367", "train_wps": "2977.1", "train_ups": "0.32", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "953", "train_lr": "3.67873e-05", "train_gnorm": "164.397", "train_loss_scale": "0.5", "train_train_wall": "97", "train_wall": "3627"}
2021-01-05 22:14:35 | INFO | fairseq.trainer | begin training epoch 32
2021-01-05 22:16:14 | INFO | fairseq_cli.train | end of epoch 32 (average epoch stats below)
2021-01-05 22:16:14 | INFO | train | {"epoch": 32, "train_loss": "676.758", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "6.283", "train_wps": "2929.6", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "984", "train_lr": "3.79677e-05", "train_gnorm": "178.224", "train_loss_scale": "0.5", "train_train_wall": "99", "train_wall": "3726"}
2021-01-05 22:16:14 | INFO | fairseq.trainer | begin training epoch 33
2021-01-05 22:17:06 | INFO | train_inner | {"epoch": 33, "update": 32.516, "loss": "687.414", "ntokens": "9400.45", "nsentences": "86.865", "nll_loss": "6.352", "wps": "2278", "ups": "0.24", "wpb": "9400.5", "bsz": "86.9", "num_updates": "1000", "lr": "3.85769e-05", "gnorm": "199.831", "loss_scale": "0.5", "train_wall": "634", "wall": "3778"}
2021-01-05 22:17:53 | INFO | fairseq_cli.train | end of epoch 33 (average epoch stats below)
2021-01-05 22:17:53 | INFO | train | {"epoch": 33, "train_loss": "647.675", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "6.013", "train_wps": "2935.4", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "1015", "train_lr": "3.91481e-05", "train_gnorm": "145.465", "train_loss_scale": "0.5", "train_train_wall": "98", "train_wall": "3826"}
2021-01-05 22:17:53 | INFO | fairseq.trainer | begin training epoch 34
2021-01-05 22:19:33 | INFO | fairseq_cli.train | end of epoch 34 (average epoch stats below)
2021-01-05 22:19:33 | INFO | train | {"epoch": 34, "train_loss": "646.912", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "6.006", "train_wps": "2921.2", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "1046", "train_lr": "4.03285e-05", "train_gnorm": "150.794", "train_loss_scale": "0.5", "train_train_wall": "99", "train_wall": "3925"}
2021-01-05 22:19:33 | INFO | fairseq.trainer | begin training epoch 35
2021-01-05 22:21:13 | INFO | fairseq_cli.train | end of epoch 35 (average epoch stats below)
2021-01-05 22:21:13 | INFO | train | {"epoch": 35, "train_loss": "649.566", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "6.03", "train_wps": "2916.7", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "1077", "train_lr": "4.15088e-05", "train_gnorm": "134.768", "train_loss_scale": "0.5", "train_train_wall": "99", "train_wall": "4025"}
2021-01-05 22:21:13 | INFO | fairseq.trainer | begin training epoch 36
2021-01-05 22:22:52 | INFO | fairseq_cli.train | end of epoch 36 (average epoch stats below)
2021-01-05 22:22:52 | INFO | train | {"epoch": 36, "train_loss": "631.119", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "5.859", "train_wps": "2933.3", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "1108", "train_lr": "4.26892e-05", "train_gnorm": "140.811", "train_loss_scale": "0.5", "train_train_wall": "98", "train_wall": "4124"}
2021-01-05 22:22:52 | INFO | fairseq.trainer | begin training epoch 37
2021-01-05 22:24:32 | INFO | fairseq_cli.train | end of epoch 37 (average epoch stats below)
2021-01-05 22:24:32 | INFO | train | {"epoch": 37, "train_loss": "663.29", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "6.158", "train_wps": "2906.9", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "1139", "train_lr": "4.38696e-05", "train_gnorm": "120.334", "train_loss_scale": "0.5", "train_train_wall": "99", "train_wall": "4225"}
2021-01-05 22:24:32 | INFO | fairseq.trainer | begin training epoch 38
2021-01-05 22:26:11 | INFO | fairseq_cli.train | end of epoch 38 (average epoch stats below)
2021-01-05 22:26:11 | INFO | train | {"epoch": 38, "train_loss": "621.615", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "5.771", "train_wps": "2932.6", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "1170", "train_lr": "4.505e-05", "train_gnorm": "134.248", "train_loss_scale": "0.5", "train_train_wall": "99", "train_wall": "4324"}
2021-01-05 22:26:11 | INFO | fairseq.trainer | begin training epoch 39
2021-01-05 22:27:47 | INFO | train_inner | {"epoch": 39, "update": 38.968, "loss": "631.701", "ntokens": "9395.9", "nsentences": "87.88", "nll_loss": "5.908", "wps": "2931.6", "ups": "0.31", "wpb": "9395.9", "bsz": "87.9", "num_updates": "1200", "lr": "4.61923e-05", "gnorm": "134.301", "loss_scale": "0.5", "train_wall": "636", "wall": "4419"}
2021-01-05 22:27:49 | INFO | fairseq_cli.train | end of epoch 39 (average epoch stats below)
2021-01-05 22:27:49 | INFO | train | {"epoch": 39, "train_loss": "610.167", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "5.665", "train_wps": "2973.4", "train_ups": "0.32", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "1201", "train_lr": "4.62304e-05", "train_gnorm": "133.635", "train_loss_scale": "0.5", "train_train_wall": "97", "train_wall": "4422"}
2021-01-05 22:27:49 | INFO | fairseq.trainer | begin training epoch 40
2021-01-05 22:29:28 | INFO | fairseq_cli.train | begin validation on "valid" subset
2021-01-05 22:31:32 | INFO | valid | {"epoch": 40, "valid_loss": "987.414", "valid_ntokens": "2273.2", "valid_nsentences": "20.96", "valid_nll_loss": "9.104", "valid_uer": "99.988", "valid_wer": "100", "valid_raw_wer": "100", "valid_wps": "2312.3", "valid_wpb": "2273.2", "valid_bsz": "21", "valid_num_updates": "1232", "valid_best_wer": "99.981"}
2021-01-05 22:31:32 | INFO | fairseq.checkpoint_utils | Preparing to save checkpoint for epoch 40 @ 1232 updates
2021-01-05 22:31:52 | INFO | fairseq.trainer | Finished saving checkpoint to checkpoints/checkpoint40.pt
2021-01-05 22:32:33 | INFO | fairseq.checkpoint_utils | Saved checkpoint checkpoints/checkpoint40.pt (epoch 40 @ 1232 updates, score 100.0) (writing took 61.353146653000294 seconds)
2021-01-05 22:32:33 | INFO | fairseq_cli.train | end of epoch 40 (average epoch stats below)
2021-01-05 22:32:33 | INFO | train | {"epoch": 40, "train_loss": "603.149", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "5.599", "train_wps": "1024.1", "train_ups": "0.11", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "1232", "train_lr": "4.74108e-05", "train_gnorm": "113.494", "train_loss_scale": "0.5", "train_train_wall": "98", "train_wall": "4706"}
2021-01-05 22:32:34 | INFO | fairseq.trainer | begin training epoch 41
2021-01-05 22:34:12 | INFO | fairseq_cli.train | end of epoch 41 (average epoch stats below)
2021-01-05 22:34:12 | INFO | train | {"epoch": 41, "train_loss": "607.114", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "5.636", "train_wps": "2963.2", "train_ups": "0.32", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "1263", "train_lr": "4.85912e-05", "train_gnorm": "104.019", "train_loss_scale": "0.5", "train_train_wall": "98", "train_wall": "4804"}
2021-01-05 22:34:12 | INFO | fairseq.trainer | begin training epoch 42
2021-01-05 22:35:52 | INFO | fairseq_cli.train | end of epoch 42 (average epoch stats below)
2021-01-05 22:35:52 | INFO | train | {"epoch": 42, "train_loss": "627.448", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "5.825", "train_wps": "2913.1", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "1294", "train_lr": "4.97715e-05", "train_gnorm": "123.072", "train_loss_scale": "0.5", "train_train_wall": "99", "train_wall": "4904"}
2021-01-05 22:35:52 | INFO | fairseq.trainer | begin training epoch 43
2021-01-05 22:37:31 | INFO | fairseq_cli.train | end of epoch 43 (average epoch stats below)
2021-01-05 22:37:31 | INFO | train | {"epoch": 43, "train_loss": "616.264", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "5.721", "train_wps": "2919.5", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "1325", "train_lr": "5e-05", "train_gnorm": "115.164", "train_loss_scale": "0.5", "train_train_wall": "99", "train_wall": "5004"}
2021-01-05 22:37:31 | INFO | fairseq.trainer | begin training epoch 44
2021-01-05 22:39:11 | INFO | fairseq_cli.train | end of epoch 44 (average epoch stats below)
2021-01-05 22:39:11 | INFO | train | {"epoch": 44, "train_loss": "609.722", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "5.66", "train_wps": "2920", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "1356", "train_lr": "5e-05", "train_gnorm": "104.88", "train_loss_scale": "0.5", "train_train_wall": "99", "train_wall": "5104"}
2021-01-05 22:39:11 | INFO | fairseq.trainer | begin training epoch 45
2021-01-05 22:40:50 | INFO | fairseq_cli.train | end of epoch 45 (average epoch stats below)
2021-01-05 22:40:50 | INFO | train | {"epoch": 45, "train_loss": "615.961", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "5.718", "train_wps": "2937", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "1387", "train_lr": "5e-05", "train_gnorm": "118.525", "train_loss_scale": "0.5", "train_train_wall": "98", "train_wall": "5203"}
2021-01-05 22:40:50 | INFO | fairseq.trainer | begin training epoch 46
2021-01-05 22:41:34 | INFO | train_inner | {"epoch": 46, "update": 45.419, "loss": "609.299", "ntokens": "9393.33", "nsentences": "87.305", "nll_loss": "5.663", "wps": "2270.8", "ups": "0.24", "wpb": "9393.3", "bsz": "87.3", "num_updates": "1400", "lr": "5e-05", "gnorm": "112.028", "loss_scale": "0.5", "train_wall": "637", "wall": "5247"}
2021-01-05 22:42:30 | INFO | fairseq_cli.train | end of epoch 46 (average epoch stats below)
2021-01-05 22:42:30 | INFO | train | {"epoch": 46, "train_loss": "569.018", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "5.283", "train_wps": "2926.8", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "1418", "train_lr": "5e-05", "train_gnorm": "98.132", "train_loss_scale": "0.5", "train_train_wall": "99", "train_wall": "5302"}
2021-01-05 22:42:30 | INFO | fairseq.trainer | begin training epoch 47
2021-01-05 22:44:09 | INFO | fairseq_cli.train | end of epoch 47 (average epoch stats below)
2021-01-05 22:44:09 | INFO | train | {"epoch": 47, "train_loss": "584.963", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "5.431", "train_wps": "2944.9", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "1449", "train_lr": "5e-05", "train_gnorm": "106.598", "train_loss_scale": "0.5", "train_train_wall": "98", "train_wall": "5401"}
2021-01-05 22:44:09 | INFO | fairseq.trainer | begin training epoch 48
2021-01-05 22:45:48 | INFO | fairseq_cli.train | end of epoch 48 (average epoch stats below)
2021-01-05 22:45:48 | INFO | train | {"epoch": 48, "train_loss": "582.6", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "5.409", "train_wps": "2942", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "1480", "train_lr": "5e-05", "train_gnorm": "98.681", "train_loss_scale": "0.5", "train_train_wall": "98", "train_wall": "5500"}
2021-01-05 22:45:48 | INFO | fairseq.trainer | begin training epoch 49
2021-01-05 22:47:25 | INFO | fairseq_cli.train | end of epoch 49 (average epoch stats below)
2021-01-05 22:47:25 | INFO | train | {"epoch": 49, "train_loss": "582.185", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "5.405", "train_wps": "2974.8", "train_ups": "0.32", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "1511", "train_lr": "5e-05", "train_gnorm": "117.352", "train_loss_scale": "0.5", "train_train_wall": "97", "train_wall": "5598"}
2021-01-05 22:47:26 | INFO | fairseq.trainer | begin training epoch 50
2021-01-05 22:49:05 | INFO | fairseq_cli.train | begin validation on "valid" subset
2021-01-05 22:51:09 | INFO | valid | {"epoch": 50, "valid_loss": "1167.53", "valid_ntokens": "2273.2", "valid_nsentences": "20.96", "valid_nll_loss": "10.765", "valid_uer": "99.993", "valid_wer": "100", "valid_raw_wer": "100", "valid_wps": "2306.1", "valid_wpb": "2273.2", "valid_bsz": "21", "valid_num_updates": "1542", "valid_best_wer": "99.981"}
2021-01-05 22:51:09 | INFO | fairseq.checkpoint_utils | Preparing to save checkpoint for epoch 50 @ 1542 updates
2021-01-05 22:51:30 | INFO | fairseq.trainer | Finished saving checkpoint to checkpoints/checkpoint50.pt
2021-01-05 22:52:11 | INFO | fairseq.checkpoint_utils | Saved checkpoint checkpoints/checkpoint50.pt (epoch 50 @ 1542 updates, score 100.0) (writing took 61.49140399699991 seconds)
2021-01-05 22:52:11 | INFO | fairseq_cli.train | end of epoch 50 (average epoch stats below)
2021-01-05 22:52:11 | INFO | train | {"epoch": 50, "train_loss": "563.006", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "5.227", "train_wps": "1021.1", "train_ups": "0.11", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "1542", "train_lr": "5e-05", "train_gnorm": "97.68", "train_loss_scale": "0.5", "train_train_wall": "99", "train_wall": "5883"}
2021-01-05 22:52:11 | INFO | fairseq.trainer | begin training epoch 51
2021-01-05 22:53:49 | INFO | fairseq_cli.train | end of epoch 51 (average epoch stats below)
2021-01-05 22:53:49 | INFO | train | {"epoch": 51, "train_loss": "564.213", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "5.238", "train_wps": "2952.6", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "1573", "train_lr": "5e-05", "train_gnorm": "107.238", "train_loss_scale": "0.5", "train_train_wall": "98", "train_wall": "5982"}
2021-01-05 22:53:49 | INFO | fairseq.trainer | begin training epoch 52
2021-01-05 22:55:17 | INFO | train_inner | {"epoch": 52, "update": 51.871, "loss": "578.654", "ntokens": "9400.48", "nsentences": "86.99", "nll_loss": "5.355", "wps": "2284.1", "ups": "0.24", "wpb": "9400.5", "bsz": "87", "num_updates": "1600", "lr": "5e-05", "gnorm": "104.341", "loss_scale": "0.5", "train_wall": "632", "wall": "6070"}
2021-01-05 22:55:28 | INFO | fairseq_cli.train | end of epoch 52 (average epoch stats below)
2021-01-05 22:55:28 | INFO | train | {"epoch": 52, "train_loss": "584.582", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "5.427", "train_wps": "2938.7", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "1604", "train_lr": "5e-05", "train_gnorm": "95.464", "train_loss_scale": "0.5", "train_train_wall": "98", "train_wall": "6081"}
2021-01-05 22:55:28 | INFO | fairseq.trainer | begin training epoch 53
2021-01-05 22:57:07 | INFO | fairseq_cli.train | end of epoch 53 (average epoch stats below)
2021-01-05 22:57:07 | INFO | train | {"epoch": 53, "train_loss": "569.604", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "5.288", "train_wps": "2942.3", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "1635", "train_lr": "5e-05", "train_gnorm": "92.402", "train_loss_scale": "0.5", "train_train_wall": "98", "train_wall": "6180"}
2021-01-05 22:57:07 | INFO | fairseq.trainer | begin training epoch 54
2021-01-05 22:58:46 | INFO | fairseq_cli.train | end of epoch 54 (average epoch stats below)
2021-01-05 22:58:46 | INFO | train | {"epoch": 54, "train_loss": "547.871", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "5.086", "train_wps": "2934.5", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "1666", "train_lr": "5e-05", "train_gnorm": "84.481", "train_loss_scale": "0.5", "train_train_wall": "98", "train_wall": "6279"}
2021-01-05 22:58:47 | INFO | fairseq.trainer | begin training epoch 55
2021-01-05 23:00:26 | INFO | fairseq_cli.train | end of epoch 55 (average epoch stats below)
2021-01-05 23:00:26 | INFO | train | {"epoch": 55, "train_loss": "577.575", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "5.362", "train_wps": "2928.9", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "1697", "train_lr": "5e-05", "train_gnorm": "89.587", "train_loss_scale": "0.5", "train_train_wall": "99", "train_wall": "6378"}
2021-01-05 23:00:26 | INFO | fairseq.trainer | begin training epoch 56
2021-01-05 23:02:05 | INFO | fairseq_cli.train | end of epoch 56 (average epoch stats below)
2021-01-05 23:02:05 | INFO | train | {"epoch": 56, "train_loss": "539.614", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "5.01", "train_wps": "2924", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "1728", "train_lr": "5e-05", "train_gnorm": "80.805", "train_loss_scale": "0.5", "train_train_wall": "99", "train_wall": "6478"}
2021-01-05 23:02:06 | INFO | fairseq.trainer | begin training epoch 57
2021-01-05 23:03:45 | INFO | fairseq_cli.train | end of epoch 57 (average epoch stats below)
2021-01-05 23:03:45 | INFO | train | {"epoch": 57, "train_loss": "521.92", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "4.845", "train_wps": "2927.5", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "1759", "train_lr": "5e-05", "train_gnorm": "70.819", "train_loss_scale": "0.5", "train_train_wall": "99", "train_wall": "6577"}
2021-01-05 23:03:45 | INFO | fairseq.trainer | begin training epoch 58
2021-01-05 23:05:24 | INFO | fairseq_cli.train | end of epoch 58 (average epoch stats below)
2021-01-05 23:05:24 | INFO | train | {"epoch": 58, "train_loss": "550.839", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "5.114", "train_wps": "2924.7", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "1790", "train_lr": "5e-05", "train_gnorm": "81.773", "train_loss_scale": "0.5", "train_train_wall": "99", "train_wall": "6677"}
2021-01-05 23:05:25 | INFO | fairseq.trainer | begin training epoch 59
2021-01-05 23:05:57 | INFO | train_inner | {"epoch": 59, "update": 58.323, "loss": "553.94", "ntokens": "9367.43", "nsentences": "86.475", "nll_loss": "5.114", "wps": "2929.2", "ups": "0.31", "wpb": "9367.4", "bsz": "86.5", "num_updates": "1800", "lr": "5e-05", "gnorm": "83.389", "loss_scale": "0.5", "train_wall": "634", "wall": "6709"}
2021-01-05 23:07:03 | INFO | fairseq_cli.train | end of epoch 59 (average epoch stats below)
2021-01-05 23:07:03 | INFO | train | {"epoch": 59, "train_loss": "553.562", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "5.139", "train_wps": "2960.4", "train_ups": "0.32", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "1821", "train_lr": "5e-05", "train_gnorm": "83.904", "train_loss_scale": "0.5", "train_train_wall": "98", "train_wall": "6775"}
2021-01-05 23:07:03 | INFO | fairseq.trainer | begin training epoch 60
2021-01-05 23:08:42 | INFO | fairseq_cli.train | begin validation on "valid" subset
2021-01-05 23:10:47 | INFO | valid | {"epoch": 60, "valid_loss": "1250.56", "valid_ntokens": "2273.2", "valid_nsentences": "20.96", "valid_nll_loss": "11.531", "valid_uer": "99.996", "valid_wer": "100", "valid_raw_wer": "100", "valid_wps": "2304.3", "valid_wpb": "2273.2", "valid_bsz": "21", "valid_num_updates": "1852", "valid_best_wer": "99.981"}
2021-01-05 23:10:47 | INFO | fairseq.checkpoint_utils | Preparing to save checkpoint for epoch 60 @ 1852 updates
2021-01-05 23:11:07 | INFO | fairseq.trainer | Finished saving checkpoint to checkpoints/checkpoint60.pt
2021-01-05 23:11:48 | INFO | fairseq.checkpoint_utils | Saved checkpoint checkpoints/checkpoint60.pt (epoch 60 @ 1852 updates, score 100.0) (writing took 61.40998200899958 seconds)
2021-01-05 23:11:48 | INFO | fairseq_cli.train | end of epoch 60 (average epoch stats below)
2021-01-05 23:11:48 | INFO | train | {"epoch": 60, "train_loss": "549.439", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "5.101", "train_wps": "1020.5", "train_ups": "0.11", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "1852", "train_lr": "5e-05", "train_gnorm": "77.24", "train_loss_scale": "0.5", "train_train_wall": "99", "train_wall": "7061"}
2021-01-05 23:11:48 | INFO | fairseq.trainer | begin training epoch 61
2021-01-05 23:13:26 | INFO | fairseq_cli.train | end of epoch 61 (average epoch stats below)
2021-01-05 23:13:26 | INFO | train | {"epoch": 61, "train_loss": "531.445", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "4.934", "train_wps": "2965.4", "train_ups": "0.32", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "1883", "train_lr": "5e-05", "train_gnorm": "73.892", "train_loss_scale": "0.5", "train_train_wall": "97", "train_wall": "7159"}
2021-01-05 23:13:26 | INFO | fairseq.trainer | begin training epoch 62
2021-01-05 23:15:06 | INFO | fairseq_cli.train | end of epoch 62 (average epoch stats below)
2021-01-05 23:15:06 | INFO | train | {"epoch": 62, "train_loss": "531.664", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "4.936", "train_wps": "2933.6", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "1914", "train_lr": "5e-05", "train_gnorm": "71.722", "train_loss_scale": "0.5", "train_train_wall": "99", "train_wall": "7258"}
2021-01-05 23:15:06 | INFO | fairseq.trainer | begin training epoch 63
2021-01-05 23:16:45 | INFO | fairseq_cli.train | end of epoch 63 (average epoch stats below)
2021-01-05 23:16:45 | INFO | train | {"epoch": 63, "train_loss": "572.34", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "5.313", "train_wps": "2942.4", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "1945", "train_lr": "5e-05", "train_gnorm": "97.248", "train_loss_scale": "0.5", "train_train_wall": "98", "train_wall": "7357"}
2021-01-05 23:16:45 | INFO | fairseq.trainer | begin training epoch 64
2021-01-05 23:18:23 | INFO | fairseq_cli.train | end of epoch 64 (average epoch stats below)
2021-01-05 23:18:23 | INFO | train | {"epoch": 64, "train_loss": "572.354", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "5.314", "train_wps": "2953.1", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "1976", "train_lr": "5e-05", "train_gnorm": "85.478", "train_loss_scale": "0.5", "train_train_wall": "98", "train_wall": "7456"}
2021-01-05 23:18:23 | INFO | fairseq.trainer | begin training epoch 65
2021-01-05 23:19:41 | INFO | train_inner | {"epoch": 65, "update": 64.774, "loss": "548.932", "ntokens": "9400.25", "nsentences": "87.31", "nll_loss": "5.099", "wps": "2282", "ups": "0.24", "wpb": "9400.3", "bsz": "87.3", "num_updates": "2000", "lr": "5e-05", "gnorm": "79.699", "loss_scale": "0.5", "train_wall": "633", "wall": "7533"}
2021-01-05 23:20:02 | INFO | fairseq_cli.train | end of epoch 65 (average epoch stats below)
2021-01-05 23:20:02 | INFO | train | {"epoch": 65, "train_loss": "530.244", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "4.923", "train_wps": "2936.7", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "2007", "train_lr": "5e-05", "train_gnorm": "68.687", "train_loss_scale": "0.5", "train_train_wall": "98", "train_wall": "7555"}
2021-01-05 23:20:02 | INFO | fairseq.trainer | begin training epoch 66
2021-01-05 23:21:41 | INFO | fairseq_cli.train | end of epoch 66 (average epoch stats below)
2021-01-05 23:21:41 | INFO | train | {"epoch": 66, "train_loss": "571.174", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "5.303", "train_wps": "2940.5", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "2038", "train_lr": "5e-05", "train_gnorm": "92.478", "train_loss_scale": "0.5", "train_train_wall": "98", "train_wall": "7654"}
2021-01-05 23:21:41 | INFO | fairseq.trainer | begin training epoch 67
2021-01-05 23:23:21 | INFO | fairseq_cli.train | end of epoch 67 (average epoch stats below)
2021-01-05 23:23:21 | INFO | train | {"epoch": 67, "train_loss": "548.91", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "5.096", "train_wps": "2920.3", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "2069", "train_lr": "5e-05", "train_gnorm": "77.402", "train_loss_scale": "0.5", "train_train_wall": "99", "train_wall": "7754"}
2021-01-05 23:23:21 | INFO | fairseq.trainer | begin training epoch 68
2021-01-05 23:25:00 | INFO | fairseq_cli.train | end of epoch 68 (average epoch stats below)
2021-01-05 23:25:00 | INFO | train | {"epoch": 68, "train_loss": "562.776", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "5.225", "train_wps": "2935.8", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "2100", "train_lr": "5e-05", "train_gnorm": "89.023", "train_loss_scale": "0.5", "train_train_wall": "98", "train_wall": "7853"}
2021-01-05 23:25:00 | INFO | fairseq.trainer | begin training epoch 69
2021-01-05 23:26:39 | INFO | fairseq_cli.train | end of epoch 69 (average epoch stats below)
2021-01-05 23:26:39 | INFO | train | {"epoch": 69, "train_loss": "536.586", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "4.981", "train_wps": "2954.8", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "2131", "train_lr": "5e-05", "train_gnorm": "74.711", "train_loss_scale": "0.5", "train_train_wall": "98", "train_wall": "7951"}
2021-01-05 23:26:39 | INFO | fairseq.trainer | begin training epoch 70
2021-01-05 23:28:18 | INFO | fairseq_cli.train | begin validation on "valid" subset
2021-01-05 23:30:23 | INFO | valid | {"epoch": 70, "valid_loss": "1285.08", "valid_ntokens": "2273.2", "valid_nsentences": "20.96", "valid_nll_loss": "11.849", "valid_uer": "99.998", "valid_wer": "100", "valid_raw_wer": "100", "valid_wps": "2302.5", "valid_wpb": "2273.2", "valid_bsz": "21", "valid_num_updates": "2162", "valid_best_wer": "99.981"}
2021-01-05 23:30:23 | INFO | fairseq.checkpoint_utils | Preparing to save checkpoint for epoch 70 @ 2162 updates
2021-01-05 23:30:42 | INFO | fairseq.trainer | Finished saving checkpoint to checkpoints/checkpoint70.pt
2021-01-05 23:31:24 | INFO | fairseq.checkpoint_utils | Saved checkpoint checkpoints/checkpoint70.pt (epoch 70 @ 2162 updates, score 100.0) (writing took 61.31149206700138 seconds)
2021-01-05 23:31:24 | INFO | fairseq_cli.train | end of epoch 70 (average epoch stats below)
2021-01-05 23:31:24 | INFO | train | {"epoch": 70, "train_loss": "534.887", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "4.966", "train_wps": "1021.2", "train_ups": "0.11", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "2162", "train_lr": "5e-05", "train_gnorm": "71.26", "train_loss_scale": "0.5", "train_train_wall": "99", "train_wall": "8236"}
2021-01-05 23:31:24 | INFO | fairseq.trainer | begin training epoch 71
2021-01-05 23:33:02 | INFO | fairseq_cli.train | end of epoch 71 (average epoch stats below)
2021-01-05 23:33:02 | INFO | train | {"epoch": 71, "train_loss": "567.262", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "5.266", "train_wps": "2971.8", "train_ups": "0.32", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "2193", "train_lr": "5e-05", "train_gnorm": "96.195", "train_loss_scale": "0.5", "train_train_wall": "97", "train_wall": "8334"}
2021-01-05 23:33:02 | INFO | fairseq.trainer | begin training epoch 72
2021-01-05 23:33:24 | INFO | train_inner | {"epoch": 72, "update": 71.226, "loss": "551.509", "ntokens": "9391.76", "nsentences": "87.36", "nll_loss": "5.13", "wps": "2279.9", "ups": "0.24", "wpb": "9391.8", "bsz": "87.4", "num_updates": "2200", "lr": "5e-05", "gnorm": "82.492", "loss_scale": "0.5", "train_wall": "633", "wall": "8357"}
2021-01-05 23:34:41 | INFO | fairseq_cli.train | end of epoch 72 (average epoch stats below)
2021-01-05 23:34:41 | INFO | train | {"epoch": 72, "train_loss": "554.962", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "5.152", "train_wps": "2947.9", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "2224", "train_lr": "5e-05", "train_gnorm": "89.213", "train_loss_scale": "0.5", "train_train_wall": "98", "train_wall": "8433"}
2021-01-05 23:34:41 | INFO | fairseq.trainer | begin training epoch 73
2021-01-05 23:36:20 | INFO | fairseq_cli.train | end of epoch 73 (average epoch stats below)
2021-01-05 23:36:20 | INFO | train | {"epoch": 73, "train_loss": "512.233", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "4.755", "train_wps": "2923.6", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "2255", "train_lr": "5e-05", "train_gnorm": "59.815", "train_loss_scale": "0.5", "train_train_wall": "99", "train_wall": "8533"}
2021-01-05 23:36:20 | INFO | fairseq.trainer | begin training epoch 74
2021-01-05 23:37:59 | INFO | fairseq_cli.train | end of epoch 74 (average epoch stats below)
2021-01-05 23:37:59 | INFO | train | {"epoch": 74, "train_loss": "555.761", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "5.159", "train_wps": "2950.7", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "2286", "train_lr": "5e-05", "train_gnorm": "85.041", "train_loss_scale": "0.5", "train_train_wall": "98", "train_wall": "8631"}
2021-01-05 23:37:59 | INFO | fairseq.trainer | begin training epoch 75
2021-01-05 23:39:38 | INFO | fairseq_cli.train | end of epoch 75 (average epoch stats below)
2021-01-05 23:39:38 | INFO | train | {"epoch": 75, "train_loss": "522.61", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "4.852", "train_wps": "2944.4", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "2317", "train_lr": "5e-05", "train_gnorm": "72.64", "train_loss_scale": "0.5", "train_train_wall": "98", "train_wall": "8730"}
2021-01-05 23:39:38 | INFO | fairseq.trainer | begin training epoch 76
2021-01-05 23:41:17 | INFO | fairseq_cli.train | end of epoch 76 (average epoch stats below)
2021-01-05 23:41:17 | INFO | train | {"epoch": 76, "train_loss": "571.697", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "5.307", "train_wps": "2942.5", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "2348", "train_lr": "5e-05", "train_gnorm": "98.696", "train_loss_scale": "0.5", "train_train_wall": "98", "train_wall": "8829"}
2021-01-05 23:41:17 | INFO | fairseq.trainer | begin training epoch 77
2021-01-05 23:42:56 | INFO | fairseq_cli.train | end of epoch 77 (average epoch stats below)
2021-01-05 23:42:56 | INFO | train | {"epoch": 77, "train_loss": "524.495", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "4.869", "train_wps": "2933.6", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "2379", "train_lr": "5e-05", "train_gnorm": "69.443", "train_loss_scale": "0.5", "train_train_wall": "99", "train_wall": "8929"}
2021-01-05 23:42:56 | INFO | fairseq.trainer | begin training epoch 78
2021-01-05 23:44:04 | INFO | train_inner | {"epoch": 78, "update": 77.677, "loss": "539.643", "ntokens": "9391.18", "nsentences": "87.745", "nll_loss": "5.042", "wps": "2935", "ups": "0.31", "wpb": "9391.2", "bsz": "87.7", "num_updates": "2400", "lr": "5e-05", "gnorm": "80.041", "loss_scale": "0.5", "train_wall": "635", "wall": "8997"}
2021-01-05 23:44:35 | INFO | fairseq_cli.train | end of epoch 78 (average epoch stats below)
2021-01-05 23:44:35 | INFO | train | {"epoch": 78, "train_loss": "533.159", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "4.95", "train_wps": "2936", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "2410", "train_lr": "5e-05", "train_gnorm": "69.747", "train_loss_scale": "0.5", "train_train_wall": "98", "train_wall": "9028"}
2021-01-05 23:44:35 | INFO | fairseq.trainer | begin training epoch 79
2021-01-05 23:46:14 | INFO | fairseq_cli.train | end of epoch 79 (average epoch stats below)
2021-01-05 23:46:14 | INFO | train | {"epoch": 79, "train_loss": "528.853", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "4.91", "train_wps": "2942.5", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "2441", "train_lr": "5e-05", "train_gnorm": "77.279", "train_loss_scale": "0.5", "train_train_wall": "98", "train_wall": "9127"}
2021-01-05 23:46:14 | INFO | fairseq.trainer | begin training epoch 80
2021-01-05 23:47:53 | INFO | fairseq_cli.train | begin validation on "valid" subset
2021-01-05 23:49:58 | INFO | valid | {"epoch": 80, "valid_loss": "1218.33", "valid_ntokens": "2273.2", "valid_nsentences": "20.96", "valid_nll_loss": "11.234", "valid_uer": "99.998", "valid_wer": "100", "valid_raw_wer": "100", "valid_wps": "2303.9", "valid_wpb": "2273.2", "valid_bsz": "21", "valid_num_updates": "2472", "valid_best_wer": "99.981"}
2021-01-05 23:49:58 | INFO | fairseq.checkpoint_utils | Preparing to save checkpoint for epoch 80 @ 2472 updates
2021-01-05 23:50:18 | INFO | fairseq.trainer | Finished saving checkpoint to checkpoints/checkpoint80.pt
2021-01-05 23:50:59 | INFO | fairseq.checkpoint_utils | Saved checkpoint checkpoints/checkpoint80.pt (epoch 80 @ 2472 updates, score 100.0) (writing took 61.367504372999974 seconds)
2021-01-05 23:50:59 | INFO | fairseq_cli.train | end of epoch 80 (average epoch stats below)
2021-01-05 23:50:59 | INFO | train | {"epoch": 80, "train_loss": "520.73", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "4.834", "train_wps": "1021.5", "train_ups": "0.11", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "2472", "train_lr": "5e-05", "train_gnorm": "74.089", "train_loss_scale": "0.5", "train_train_wall": "98", "train_wall": "9412"}
2021-01-05 23:50:59 | INFO | fairseq.trainer | begin training epoch 81
2021-01-05 23:52:38 | INFO | fairseq_cli.train | end of epoch 81 (average epoch stats below)
2021-01-05 23:52:38 | INFO | train | {"epoch": 81, "train_loss": "510.004", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "4.735", "train_wps": "2938", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "2503", "train_lr": "5e-05", "train_gnorm": "61.139", "train_loss_scale": "0.5", "train_train_wall": "98", "train_wall": "9511"}
2021-01-05 23:52:38 | INFO | fairseq.trainer | begin training epoch 82
2021-01-05 23:54:18 | INFO | fairseq_cli.train | end of epoch 82 (average epoch stats below)
2021-01-05 23:54:18 | INFO | train | {"epoch": 82, "train_loss": "506.439", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "4.702", "train_wps": "2931.8", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "2534", "train_lr": "5e-05", "train_gnorm": "74.993", "train_loss_scale": "0.5", "train_train_wall": "99", "train_wall": "9610"}
2021-01-05 23:54:18 | INFO | fairseq.trainer | begin training epoch 83
2021-01-05 23:55:57 | INFO | fairseq_cli.train | end of epoch 83 (average epoch stats below)
2021-01-05 23:55:57 | INFO | train | {"epoch": 83, "train_loss": "523.631", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "4.861", "train_wps": "2928.8", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "2565", "train_lr": "5e-05", "train_gnorm": "73.29", "train_loss_scale": "0.5", "train_train_wall": "99", "train_wall": "9710"}
2021-01-05 23:55:57 | INFO | fairseq.trainer | begin training epoch 84
2021-01-05 23:57:35 | INFO | fairseq_cli.train | end of epoch 84 (average epoch stats below)
2021-01-05 23:57:35 | INFO | train | {"epoch": 84, "train_loss": "551.704", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "5.122", "train_wps": "2963.4", "train_ups": "0.32", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "2596", "train_lr": "5e-05", "train_gnorm": "88.712", "train_loss_scale": "0.5", "train_train_wall": "98", "train_wall": "9808"}
2021-01-05 23:57:35 | INFO | fairseq.trainer | begin training epoch 85
2021-01-05 23:57:49 | INFO | train_inner | {"epoch": 85, "update": 84.129, "loss": "522.103", "ntokens": "9398.18", "nsentences": "87.06", "nll_loss": "4.836", "wps": "2280.1", "ups": "0.24", "wpb": "9398.2", "bsz": "87.1", "num_updates": "2600", "lr": "5e-05", "gnorm": "72.976", "loss_scale": "0.5", "train_wall": "633", "wall": "9821"}
2021-01-05 23:59:15 | INFO | fairseq_cli.train | end of epoch 85 (average epoch stats below)
2021-01-05 23:59:15 | INFO | train | {"epoch": 85, "train_loss": "490.982", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "4.558", "train_wps": "2931.2", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "2627", "train_lr": "5e-05", "train_gnorm": "51.35", "train_loss_scale": "0.5", "train_train_wall": "99", "train_wall": "9907"}
2021-01-05 23:59:15 | INFO | fairseq.trainer | begin training epoch 86
2021-01-06 00:00:54 | INFO | fairseq_cli.train | end of epoch 86 (average epoch stats below)
2021-01-06 00:00:54 | INFO | train | {"epoch": 86, "train_loss": "509.883", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "4.734", "train_wps": "2913.8", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "2658", "train_lr": "5e-05", "train_gnorm": "66.954", "train_loss_scale": "0.5", "train_train_wall": "99", "train_wall": "10007"}
2021-01-06 00:00:55 | INFO | fairseq.trainer | begin training epoch 87
2021-01-06 00:02:34 | INFO | fairseq_cli.train | end of epoch 87 (average epoch stats below)
2021-01-06 00:02:34 | INFO | train | {"epoch": 87, "train_loss": "537.689", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "4.992", "train_wps": "2918.6", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "2689", "train_lr": "5e-05", "train_gnorm": "93.117", "train_loss_scale": "0.5", "train_train_wall": "99", "train_wall": "10107"}
2021-01-06 00:02:34 | INFO | fairseq.trainer | begin training epoch 88
2021-01-06 00:04:14 | INFO | fairseq_cli.train | end of epoch 88 (average epoch stats below)
2021-01-06 00:04:14 | INFO | train | {"epoch": 88, "train_loss": "516.539", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "4.795", "train_wps": "2918.1", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "2720", "train_lr": "5e-05", "train_gnorm": "75.21", "train_loss_scale": "0.5", "train_train_wall": "99", "train_wall": "10207"}
2021-01-06 00:04:14 | INFO | fairseq.trainer | begin training epoch 89
2021-01-06 00:05:54 | INFO | fairseq_cli.train | end of epoch 89 (average epoch stats below)
2021-01-06 00:05:54 | INFO | train | {"epoch": 89, "train_loss": "536.224", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "4.978", "train_wps": "2914.4", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "2751", "train_lr": "5e-05", "train_gnorm": "96.132", "train_loss_scale": "0.5", "train_train_wall": "99", "train_wall": "10306"}
2021-01-06 00:05:54 | INFO | fairseq.trainer | begin training epoch 90
2021-01-06 00:07:34 | INFO | fairseq_cli.train | begin validation on "valid" subset
2021-01-06 00:09:39 | INFO | valid | {"epoch": 90, "valid_loss": "1065.54", "valid_ntokens": "2273.2", "valid_nsentences": "20.96", "valid_nll_loss": "9.825", "valid_uer": "99.998", "valid_wer": "100", "valid_raw_wer": "100", "valid_wps": "2301.8", "valid_wpb": "2273.2", "valid_bsz": "21", "valid_num_updates": "2782", "valid_best_wer": "99.981"}
2021-01-06 00:09:39 | INFO | fairseq.checkpoint_utils | Preparing to save checkpoint for epoch 90 @ 2782 updates
2021-01-06 00:09:59 | INFO | fairseq.trainer | Finished saving checkpoint to checkpoints/checkpoint90.pt
2021-01-06 00:10:40 | INFO | fairseq.checkpoint_utils | Saved checkpoint checkpoints/checkpoint90.pt (epoch 90 @ 2782 updates, score 100.0) (writing took 61.52324476600006 seconds)
2021-01-06 00:10:40 | INFO | fairseq_cli.train | end of epoch 90 (average epoch stats below)
2021-01-06 00:10:40 | INFO | train | {"epoch": 90, "train_loss": "514.343", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "4.775", "train_wps": "1016.4", "train_ups": "0.11", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "2782", "train_lr": "5e-05", "train_gnorm": "69.756", "train_loss_scale": "0.5", "train_train_wall": "100", "train_wall": "10593"}
2021-01-06 00:10:40 | INFO | fairseq.trainer | begin training epoch 91
2021-01-06 00:11:38 | INFO | train_inner | {"epoch": 91, "update": 90.581, "loss": "517.658", "ntokens": "9388.66", "nsentences": "87.315", "nll_loss": "4.814", "wps": "2265", "ups": "0.24", "wpb": "9388.7", "bsz": "87.3", "num_updates": "2800", "lr": "5e-05", "gnorm": "76.341", "loss_scale": "0.5", "train_wall": "638", "wall": "10650"}
2021-01-06 00:12:19 | INFO | fairseq_cli.train | end of epoch 91 (average epoch stats below)
2021-01-06 00:12:19 | INFO | train | {"epoch": 91, "train_loss": "510.921", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "4.743", "train_wps": "2961.5", "train_ups": "0.32", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "2813", "train_lr": "5e-05", "train_gnorm": "81.982", "train_loss_scale": "0.5", "train_train_wall": "98", "train_wall": "10691"}
2021-01-06 00:12:19 | INFO | fairseq.trainer | begin training epoch 92
2021-01-06 00:13:58 | INFO | fairseq_cli.train | end of epoch 92 (average epoch stats below)
2021-01-06 00:13:58 | INFO | train | {"epoch": 92, "train_loss": "508.281", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "4.719", "train_wps": "2941.4", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "2844", "train_lr": "5e-05", "train_gnorm": "71.282", "train_loss_scale": "0.5", "train_train_wall": "98", "train_wall": "10790"}
2021-01-06 00:13:58 | INFO | fairseq.trainer | begin training epoch 93
2021-01-06 00:15:36 | INFO | fairseq_cli.train | end of epoch 93 (average epoch stats below)
2021-01-06 00:15:36 | INFO | train | {"epoch": 93, "train_loss": "490.378", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "4.552", "train_wps": "2945.9", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "2875", "train_lr": "5e-05", "train_gnorm": "52.688", "train_loss_scale": "0.5", "train_train_wall": "98", "train_wall": "10889"}
2021-01-06 00:15:37 | INFO | fairseq.trainer | begin training epoch 94
2021-01-06 00:17:17 | INFO | fairseq_cli.train | end of epoch 94 (average epoch stats below)
2021-01-06 00:17:17 | INFO | train | {"epoch": 94, "train_loss": "510.747", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "4.742", "train_wps": "2911", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "2906", "train_lr": "5e-05", "train_gnorm": "77.897", "train_loss_scale": "0.5", "train_train_wall": "99", "train_wall": "10989"}
2021-01-06 00:17:17 | INFO | fairseq.trainer | begin training epoch 95
2021-01-06 00:18:56 | INFO | fairseq_cli.train | end of epoch 95 (average epoch stats below)
2021-01-06 00:18:56 | INFO | train | {"epoch": 95, "train_loss": "503.735", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "4.676", "train_wps": "2923.6", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "2937", "train_lr": "5e-05", "train_gnorm": "79.732", "train_loss_scale": "0.5", "train_train_wall": "99", "train_wall": "11089"}
2021-01-06 00:18:56 | INFO | fairseq.trainer | begin training epoch 96
2021-01-06 00:20:36 | INFO | fairseq_cli.train | end of epoch 96 (average epoch stats below)
2021-01-06 00:20:36 | INFO | train | {"epoch": 96, "train_loss": "502.331", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "4.663", "train_wps": "2928.5", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "2968", "train_lr": "5e-05", "train_gnorm": "80.289", "train_loss_scale": "0.5", "train_train_wall": "99", "train_wall": "11188"}
2021-01-06 00:20:36 | INFO | fairseq.trainer | begin training epoch 97
2021-01-06 00:22:14 | INFO | fairseq_cli.train | end of epoch 97 (average epoch stats below)
2021-01-06 00:22:14 | INFO | train | {"epoch": 97, "train_loss": "501.981", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "4.66", "train_wps": "2954.3", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "2999", "train_lr": "5e-05", "train_gnorm": "72.969", "train_loss_scale": "0.5", "train_train_wall": "98", "train_wall": "11287"}
2021-01-06 00:22:14 | INFO | fairseq.trainer | begin training epoch 98
2021-01-06 00:22:18 | INFO | train_inner | {"epoch": 98, "update": 97.032, "loss": "503.766", "ntokens": "9385.78", "nsentences": "86.91", "nll_loss": "4.665", "wps": "2933.5", "ups": "0.31", "wpb": "9385.8", "bsz": "86.9", "num_updates": "3000", "lr": "5e-05", "gnorm": "73.346", "loss_scale": "0.5", "train_wall": "635", "wall": "11290"}
2021-01-06 00:23:54 | INFO | fairseq_cli.train | end of epoch 98 (average epoch stats below)
2021-01-06 00:23:54 | INFO | train | {"epoch": 98, "train_loss": "483.335", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "4.487", "train_wps": "2918.6", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "3030", "train_lr": "5e-05", "train_gnorm": "56.826", "train_loss_scale": "0.5", "train_train_wall": "99", "train_wall": "11386"}
2021-01-06 00:23:54 | INFO | fairseq.trainer | begin training epoch 99
2021-01-06 00:25:34 | INFO | fairseq_cli.train | end of epoch 99 (average epoch stats below)
2021-01-06 00:25:34 | INFO | train | {"epoch": 99, "train_loss": "489.271", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "4.542", "train_wps": "2916", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "3061", "train_lr": "5e-05", "train_gnorm": "76.848", "train_loss_scale": "0.5", "train_train_wall": "99", "train_wall": "11486"}
2021-01-06 00:25:34 | INFO | fairseq.trainer | begin training epoch 100
2021-01-06 00:27:12 | INFO | fairseq_cli.train | begin validation on "valid" subset
2021-01-06 00:29:17 | INFO | valid | {"epoch": 100, "valid_loss": "891.226", "valid_ntokens": "2273.2", "valid_nsentences": "20.96", "valid_nll_loss": "8.218", "valid_uer": "99.998", "valid_wer": "100", "valid_raw_wer": "100", "valid_wps": "2300.3", "valid_wpb": "2273.2", "valid_bsz": "21", "valid_num_updates": "3092", "valid_best_wer": "99.981"}
2021-01-06 00:29:17 | INFO | fairseq.checkpoint_utils | Preparing to save checkpoint for epoch 100 @ 3092 updates
2021-01-06 00:29:37 | INFO | fairseq.trainer | Finished saving checkpoint to checkpoints/checkpoint100.pt
2021-01-06 00:30:17 | INFO | fairseq.checkpoint_utils | Saved checkpoint checkpoints/checkpoint100.pt (epoch 100 @ 3092 updates, score 100.0) (writing took 60.38254927700109 seconds)
2021-01-06 00:30:17 | INFO | fairseq_cli.train | end of epoch 100 (average epoch stats below)
2021-01-06 00:30:17 | INFO | train | {"epoch": 100, "train_loss": "508.679", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "4.722", "train_wps": "1026", "train_ups": "0.11", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "3092", "train_lr": "5e-05", "train_gnorm": "81.956", "train_loss_scale": "0.5", "train_train_wall": "98", "train_wall": "11770"}
2021-01-06 00:30:18 | INFO | fairseq.trainer | begin training epoch 101
2021-01-06 00:31:56 | INFO | fairseq_cli.train | end of epoch 101 (average epoch stats below)
2021-01-06 00:31:56 | INFO | train | {"epoch": 101, "train_loss": "503.318", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "4.673", "train_wps": "2956.3", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "3123", "train_lr": "5e-05", "train_gnorm": "83.707", "train_loss_scale": "0.5", "train_train_wall": "98", "train_wall": "11869"}
2021-01-06 00:31:56 | INFO | fairseq.trainer | begin training epoch 102
2021-01-06 00:33:35 | INFO | fairseq_cli.train | end of epoch 102 (average epoch stats below)
2021-01-06 00:33:35 | INFO | train | {"epoch": 102, "train_loss": "486.917", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "4.52", "train_wps": "2935.2", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "3154", "train_lr": "5e-05", "train_gnorm": "64.893", "train_loss_scale": "0.5", "train_train_wall": "98", "train_wall": "11968"}
2021-01-06 00:33:35 | INFO | fairseq.trainer | begin training epoch 103
2021-01-06 00:35:15 | INFO | fairseq_cli.train | end of epoch 103 (average epoch stats below)
2021-01-06 00:35:15 | INFO | train | {"epoch": 103, "train_loss": "506.144", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "4.699", "train_wps": "2930.5", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "3185", "train_lr": "5e-05", "train_gnorm": "90.607", "train_loss_scale": "0.5", "train_train_wall": "99", "train_wall": "12067"}
2021-01-06 00:35:15 | INFO | fairseq.trainer | begin training epoch 104
2021-01-06 00:36:04 | INFO | train_inner | {"epoch": 104, "update": 103.484, "loss": "495.169", "ntokens": "9417.17", "nsentences": "87.325", "nll_loss": "4.592", "wps": "2279.1", "ups": "0.24", "wpb": "9417.2", "bsz": "87.3", "num_updates": "3200", "lr": "5e-05", "gnorm": "74.703", "loss_scale": "0.5", "train_wall": "637", "wall": "12117"}
2021-01-06 00:36:54 | INFO | fairseq_cli.train | end of epoch 104 (average epoch stats below)
2021-01-06 00:36:54 | INFO | train | {"epoch": 104, "train_loss": "495.303", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "4.598", "train_wps": "2912.7", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "3216", "train_lr": "5e-05", "train_gnorm": "78.655", "train_loss_scale": "0.5", "train_train_wall": "99", "train_wall": "12167"}
2021-01-06 00:36:55 | INFO | fairseq.trainer | begin training epoch 105
2021-01-06 00:38:33 | INFO | fairseq_cli.train | end of epoch 105 (average epoch stats below)
2021-01-06 00:38:33 | INFO | train | {"epoch": 105, "train_loss": "493.397", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "4.58", "train_wps": "2943.8", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "3247", "train_lr": "5e-05", "train_gnorm": "77.089", "train_loss_scale": "0.5", "train_train_wall": "98", "train_wall": "12266"}
2021-01-06 00:38:33 | INFO | fairseq.trainer | begin training epoch 106
2021-01-06 00:40:13 | INFO | fairseq_cli.train | end of epoch 106 (average epoch stats below)
2021-01-06 00:40:13 | INFO | train | {"epoch": 106, "train_loss": "488.702", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "4.537", "train_wps": "2921.1", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "3278", "train_lr": "5e-05", "train_gnorm": "78.73", "train_loss_scale": "0.5", "train_train_wall": "99", "train_wall": "12366"}
2021-01-06 00:40:13 | INFO | fairseq.trainer | begin training epoch 107
2021-01-06 00:41:52 | INFO | fairseq_cli.train | end of epoch 107 (average epoch stats below)
2021-01-06 00:41:52 | INFO | train | {"epoch": 107, "train_loss": "467.717", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "4.342", "train_wps": "2948.1", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "3309", "train_lr": "5e-05", "train_gnorm": "62.371", "train_loss_scale": "0.5", "train_train_wall": "98", "train_wall": "12464"}
2021-01-06 00:41:52 | INFO | fairseq.trainer | begin training epoch 108
2021-01-06 00:43:31 | INFO | fairseq_cli.train | end of epoch 108 (average epoch stats below)
2021-01-06 00:43:31 | INFO | train | {"epoch": 108, "train_loss": "499.564", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "4.638", "train_wps": "2936.8", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "3340", "train_lr": "5e-05", "train_gnorm": "82.863", "train_loss_scale": "0.5", "train_train_wall": "98", "train_wall": "12564"}
2021-01-06 00:43:31 | INFO | fairseq.trainer | begin training epoch 109
2021-01-06 00:45:10 | INFO | fairseq_cli.train | end of epoch 109 (average epoch stats below)
2021-01-06 00:45:10 | INFO | train | {"epoch": 109, "train_loss": "490.99", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "4.558", "train_wps": "2942.1", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "3371", "train_lr": "5e-05", "train_gnorm": "91.344", "train_loss_scale": "0.5", "train_train_wall": "98", "train_wall": "12662"}
2021-01-06 00:45:10 | INFO | fairseq.trainer | begin training epoch 110
2021-01-06 00:46:44 | INFO | train_inner | {"epoch": 110, "update": 109.935, "loss": "489.479", "ntokens": "9386.89", "nsentences": "87.02", "nll_loss": "4.538", "wps": "2932.3", "ups": "0.31", "wpb": "9386.9", "bsz": "87", "num_updates": "3400", "lr": "5e-05", "gnorm": "79.622", "loss_scale": "0.5", "train_wall": "635", "wall": "12757"}
2021-01-06 00:46:49 | INFO | fairseq_cli.train | begin validation on "valid" subset
2021-01-06 00:48:54 | INFO | valid | {"epoch": 110, "valid_loss": "743.089", "valid_ntokens": "2273.2", "valid_nsentences": "20.96", "valid_nll_loss": "6.852", "valid_uer": "99.999", "valid_wer": "100", "valid_raw_wer": "100", "valid_wps": "2301.5", "valid_wpb": "2273.2", "valid_bsz": "21", "valid_num_updates": "3402", "valid_best_wer": "99.981"}
2021-01-06 00:48:54 | INFO | fairseq.checkpoint_utils | Preparing to save checkpoint for epoch 110 @ 3402 updates
2021-01-06 00:49:14 | INFO | fairseq.trainer | Finished saving checkpoint to checkpoints/checkpoint110.pt
2021-01-06 00:49:55 | INFO | fairseq.checkpoint_utils | Saved checkpoint checkpoints/checkpoint110.pt (epoch 110 @ 3402 updates, score 100.0) (writing took 61.41470164299972 seconds)
2021-01-06 00:49:55 | INFO | fairseq_cli.train | end of epoch 110 (average epoch stats below)
2021-01-06 00:49:55 | INFO | train | {"epoch": 110, "train_loss": "482.931", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "4.483", "train_wps": "1020.2", "train_ups": "0.11", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "3402", "train_lr": "5e-05", "train_gnorm": "77.424", "train_loss_scale": "0.5", "train_train_wall": "99", "train_wall": "12948"}
2021-01-06 00:49:55 | INFO | fairseq.trainer | begin training epoch 111
2021-01-06 00:51:34 | INFO | fairseq_cli.train | end of epoch 111 (average epoch stats below)
2021-01-06 00:51:34 | INFO | train | {"epoch": 111, "train_loss": "502.462", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "4.665", "train_wps": "2942.9", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "3433", "train_lr": "5e-05", "train_gnorm": "103.143", "train_loss_scale": "0.5", "train_train_wall": "98", "train_wall": "13047"}
2021-01-06 00:51:34 | INFO | fairseq.trainer | begin training epoch 112
2021-01-06 00:53:13 | INFO | fairseq_cli.train | end of epoch 112 (average epoch stats below)
2021-01-06 00:53:13 | INFO | train | {"epoch": 112, "train_loss": "494.874", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "4.594", "train_wps": "2935.1", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "3464", "train_lr": "5e-05", "train_gnorm": "93.196", "train_loss_scale": "0.5", "train_train_wall": "98", "train_wall": "13146"}
2021-01-06 00:53:14 | INFO | fairseq.trainer | begin training epoch 113
2021-01-06 00:54:53 | INFO | fairseq_cli.train | end of epoch 113 (average epoch stats below)
2021-01-06 00:54:53 | INFO | train | {"epoch": 113, "train_loss": "473.194", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "4.393", "train_wps": "2920.8", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "3495", "train_lr": "5e-05", "train_gnorm": "72.592", "train_loss_scale": "0.5", "train_train_wall": "99", "train_wall": "13246"}
2021-01-06 00:54:53 | INFO | fairseq.trainer | begin training epoch 114
2021-01-06 00:56:33 | INFO | fairseq_cli.train | end of epoch 114 (average epoch stats below)
2021-01-06 00:56:33 | INFO | train | {"epoch": 114, "train_loss": "478.389", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "4.441", "train_wps": "2919.9", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "3526", "train_lr": "5e-05", "train_gnorm": "81.775", "train_loss_scale": "0.5", "train_train_wall": "99", "train_wall": "13345"}
2021-01-06 00:56:33 | INFO | fairseq.trainer | begin training epoch 115
2021-01-06 00:58:12 | INFO | fairseq_cli.train | end of epoch 115 (average epoch stats below)
2021-01-06 00:58:12 | INFO | train | {"epoch": 115, "train_loss": "498.786", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "4.631", "train_wps": "2935.2", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "3557", "train_lr": "5e-05", "train_gnorm": "90.693", "train_loss_scale": "0.5", "train_train_wall": "99", "train_wall": "13445"}
2021-01-06 00:58:12 | INFO | fairseq.trainer | begin training epoch 116
2021-01-06 00:59:51 | INFO | fairseq_cli.train | end of epoch 116 (average epoch stats below)
2021-01-06 00:59:51 | INFO | train | {"epoch": 116, "train_loss": "479.033", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "4.447", "train_wps": "2935.7", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "3588", "train_lr": "5e-05", "train_gnorm": "79.427", "train_loss_scale": "0.5", "train_train_wall": "98", "train_wall": "13544"}
2021-01-06 00:59:51 | INFO | fairseq.trainer | begin training epoch 117
2021-01-06 01:00:30 | INFO | train_inner | {"epoch": 117, "update": 116.387, "loss": "488.245", "ntokens": "9373.03", "nsentences": "86.8", "nll_loss": "4.521", "wps": "2270.5", "ups": "0.24", "wpb": "9373", "bsz": "86.8", "num_updates": "3600", "lr": "5e-05", "gnorm": "85.637", "loss_scale": "0.5", "train_wall": "635", "wall": "13583"}
2021-01-06 01:01:31 | INFO | fairseq_cli.train | end of epoch 117 (average epoch stats below)
2021-01-06 01:01:31 | INFO | train | {"epoch": 117, "train_loss": "478.248", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "4.44", "train_wps": "2929", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "3619", "train_lr": "5e-05", "train_gnorm": "81.797", "train_loss_scale": "0.5", "train_train_wall": "99", "train_wall": "13643"}
2021-01-06 01:01:31 | INFO | fairseq.trainer | begin training epoch 118
2021-01-06 01:03:11 | INFO | fairseq_cli.train | end of epoch 118 (average epoch stats below)
2021-01-06 01:03:11 | INFO | train | {"epoch": 118, "train_loss": "511.795", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "4.751", "train_wps": "2910.2", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "3650", "train_lr": "5e-05", "train_gnorm": "118.112", "train_loss_scale": "0.5", "train_train_wall": "99", "train_wall": "13743"}
2021-01-06 01:03:11 | INFO | fairseq.trainer | begin training epoch 119
2021-01-06 01:04:50 | INFO | fairseq_cli.train | end of epoch 119 (average epoch stats below)
2021-01-06 01:04:50 | INFO | train | {"epoch": 119, "train_loss": "462.111", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "4.29", "train_wps": "2941.7", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "3681", "train_lr": "5e-05", "train_gnorm": "73.645", "train_loss_scale": "0.5", "train_train_wall": "98", "train_wall": "13842"}
2021-01-06 01:04:50 | INFO | fairseq.trainer | begin training epoch 120
2021-01-06 01:06:29 | INFO | fairseq_cli.train | begin validation on "valid" subset
2021-01-06 01:08:34 | INFO | valid | {"epoch": 120, "valid_loss": "648.13", "valid_ntokens": "2273.2", "valid_nsentences": "20.96", "valid_nll_loss": "5.976", "valid_uer": "99.999", "valid_wer": "100", "valid_raw_wer": "100", "valid_wps": "2299.5", "valid_wpb": "2273.2", "valid_bsz": "21", "valid_num_updates": "3712", "valid_best_wer": "99.981"}
2021-01-06 01:08:34 | INFO | fairseq.checkpoint_utils | Preparing to save checkpoint for epoch 120 @ 3712 updates
2021-01-06 01:08:54 | INFO | fairseq.trainer | Finished saving checkpoint to checkpoints/checkpoint120.pt
2021-01-06 01:09:36 | INFO | fairseq.checkpoint_utils | Saved checkpoint checkpoints/checkpoint120.pt (epoch 120 @ 3712 updates, score 100.0) (writing took 62.29323269300039 seconds)
2021-01-06 01:09:36 | INFO | fairseq_cli.train | end of epoch 120 (average epoch stats below)
2021-01-06 01:09:36 | INFO | train | {"epoch": 120, "train_loss": "494.045", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "4.587", "train_wps": "1017.4", "train_ups": "0.11", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "3712", "train_lr": "5e-05", "train_gnorm": "87.601", "train_loss_scale": "0.5", "train_train_wall": "98", "train_wall": "14128"}
2021-01-06 01:09:36 | INFO | fairseq.trainer | begin training epoch 121
```

@sooftware btw, have you successfully finetuned a model?
I have tried follow the instructions at README.md, it use Hydra.
I'm finetuned using LibriSpeech and generate train.tsv, train.wrd, train.ltr, as well as valid.* files.
In first attempts, I'm not using KenLM, because the configurations in Hydra are not.
And now, I've managed to finetuning using KenLM, but still got weird results.
Using checkpoint10.pt:
$ python recognize.hydra.py --target_dict_path /dataset/LibriSpeech/dict.ltr.txt /outputs/2021-01-05/21-13-51/checkpoints/checkpoint10.pt /dataset/LibriSpeech/test-clean/1089/134686/1089-134686-0001.flac
/usr/local/lib/python3.8/site-packages/torch/cuda/__init__.py:52: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx (Triggered internally at  /pytorch/c10/cuda/CUDAFunctions.cpp:100.)
  return torch._C._cuda_getDeviceCount() > 0
O

Using checkpoint20.pt, or checkpoint30.pt, and so on,...
$ python recognize.hydra.py --target_dict_path /dataset/LibriSpeech/dict.ltr.txt /outputs/2021-01-05/21-13-51/checkpoints/checkpoint110.pt /dataset/LibriSpeech/test-clean/1089/134686/1089-134686-0001.flac
/usr/local/lib/python3.8/site-packages/torch/cuda/__init__.py:52: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx (Triggered internally at  /pytorch/c10/cuda/CUDAFunctions.cpp:100.)
  return torch._C._cuda_getDeviceCount() > 0

Can you see the empty line? It's just predict empty string.
May I've not train enough? I guessed so but with checkpoint110.pt above, it was not work either.
May I missed something.
But I've trying run diff through old command line --argument and new Hydra configs, see nothing special changed, but new configs not using KenLM.
Information

Dataset: LibriSpeech - dev-clean.tar.gz and test-clean.tar.gz.
recognize.hydra.py: source
Configurations: base_1h.yaml  with only checkpoint and distribution_training changed.
Commands: python fairseq_cli/hydra_train.py      task.labels=ltr         task.data=/dataset/LibriSpeech         model.w2v_path=/dataset/libri960_big.pt         --config-dir /config/         --config-name base_1h &

    Train logs

```
2021-01-05 21:15:37 | INFO | fairseq_cli.train | end of epoch 1 (average epoch stats below)
2021-01-05 21:15:37 | INFO | train | {"epoch": 1, "train_loss": "1266.85", "train_ntokens": "9291.54", "train_nsentences": "90.8333", "train_nll_loss": "12.385", "train_wps": "3218.5", "train_ups": "0.35", "train_wpb": "9291.5", "train_bsz": "90.8", "train_num_updates": "24", "train_lr": "1.41385e-06", "train_gnorm": "1157.04", "train_loss_scale": "1", "train_train_wall": "88", "train_wall": "89"}
2021-01-05 21:15:37 | INFO | fairseq.trainer | begin training epoch 2
2021-01-05 21:17:13 | INFO | fairseq_cli.train | end of epoch 2 (average epoch stats below)
2021-01-05 21:17:13 | INFO | train | {"epoch": 2, "train_loss": "1322.8", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "12.28", "train_wps": "3031.4", "train_ups": "0.32", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "55", "train_lr": "2.59423e-06", "train_gnorm": "1261.85", "train_loss_scale": "1", "train_train_wall": "95", "train_wall": "185"}
2021-01-05 21:17:13 | INFO | fairseq.trainer | begin training epoch 3
2021-01-05 21:18:52 | INFO | fairseq_cli.train | end of epoch 3 (average epoch stats below)
2021-01-05 21:18:52 | INFO | train | {"epoch": 3, "train_loss": "1315.98", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "12.217", "train_wps": "2941.3", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "86", "train_lr": "3.77462e-06", "train_gnorm": "1417.01", "train_loss_scale": "1", "train_train_wall": "98", "train_wall": "284"}
2021-01-05 21:18:52 | INFO | fairseq.trainer | begin training epoch 4
2021-01-05 21:20:30 | INFO | fairseq_cli.train | end of epoch 4 (average epoch stats below)
2021-01-05 21:20:30 | INFO | train | {"epoch": 4, "train_loss": "1303.25", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "12.099", "train_wps": "2947.6", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "117", "train_lr": "4.955e-06", "train_gnorm": "1388.31", "train_loss_scale": "1", "train_train_wall": "98", "train_wall": "383"}
2021-01-05 21:20:30 | INFO | fairseq.trainer | begin training epoch 5
2021-01-05 21:22:09 | INFO | fairseq_cli.train | end of epoch 5 (average epoch stats below)
2021-01-05 21:22:09 | INFO | train | {"epoch": 5, "train_loss": "1289.98", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "11.976", "train_wps": "2938.7", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "148", "train_lr": "6.13538e-06", "train_gnorm": "1369.71", "train_loss_scale": "1", "train_train_wall": "98", "train_wall": "482"}
2021-01-05 21:22:09 | INFO | fairseq.trainer | begin training epoch 6
2021-01-05 21:23:49 | INFO | fairseq_cli.train | end of epoch 6 (average epoch stats below)
2021-01-05 21:23:49 | INFO | train | {"epoch": 6, "train_loss": "1275.11", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "11.838", "train_wps": "2927.6", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "179", "train_lr": "7.31577e-06", "train_gnorm": "1282.1", "train_loss_scale": "1", "train_train_wall": "99", "train_wall": "581"}
2021-01-05 21:23:49 | INFO | fairseq.trainer | begin training epoch 7
2021-01-05 21:24:57 | INFO | train_inner | {"epoch": 7, "update": 6.677, "loss": "1290.3", "ntokens": "9370.92", "nsentences": "87.75", "nll_loss": "12.082", "wps": "2974.2", "ups": "0.32", "wpb": "9370.9", "bsz": "87.8", "num_updates": "200", "lr": "8.11538e-06", "gnorm": "1328.48", "loss_scale": "1", "train_wall": "644", "wall": "650"}
2021-01-05 21:25:29 | INFO | fairseq_cli.train | end of epoch 7 (average epoch stats below)
2021-01-05 21:25:29 | INFO | train | {"epoch": 7, "train_loss": "1251.74", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "11.621", "train_wps": "2916.1", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "210", "train_lr": "8.49615e-06", "train_gnorm": "1423.09", "train_loss_scale": "1", "train_train_wall": "99", "train_wall": "681"}
2021-01-05 21:25:29 | INFO | fairseq.trainer | begin training epoch 8
2021-01-05 21:26:37 | INFO | fairseq.trainer | NOTE: overflow detected, setting loss scale to: 0.5
2021-01-05 21:27:08 | INFO | fairseq_cli.train | end of epoch 8 (average epoch stats below)
2021-01-05 21:27:08 | INFO | train | {"epoch": 8, "train_loss": "1268.35", "train_ntokens": "9364.87", "train_nsentences": "84.2333", "train_nll_loss": "11.408", "train_wps": "2822.7", "train_ups": "0.3", "train_wpb": "9364.9", "train_bsz": "84.2", "train_num_updates": "240", "train_lr": "9.63846e-06", "train_gnorm": "1432.09", "train_loss_scale": "0.5", "train_train_wall": "99", "train_wall": "781"}
2021-01-05 21:27:08 | INFO | fairseq.trainer | begin training epoch 9
2021-01-05 21:28:49 | INFO | fairseq_cli.train | end of epoch 9 (average epoch stats below)
2021-01-05 21:28:49 | INFO | train | {"epoch": 9, "train_loss": "1207.49", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "11.21", "train_wps": "2903", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "271", "train_lr": "1.08188e-05", "train_gnorm": "1339.77", "train_loss_scale": "0.5", "train_train_wall": "100", "train_wall": "881"}
2021-01-05 21:28:49 | INFO | fairseq.trainer | begin training epoch 10
2021-01-05 21:30:27 | INFO | fairseq_cli.train | begin validation on "valid" subset
2021-01-05 21:32:31 | INFO | valid | {"epoch": 10, "valid_loss": "986.911", "valid_ntokens": "2273.2", "valid_nsentences": "20.96", "valid_nll_loss": "9.1", "valid_uer": "98.972", "valid_wer": "99.981", "valid_raw_wer": "99.981", "valid_wps": "2310.5", "valid_wpb": "2273.2", "valid_bsz": "21", "valid_num_updates": "302"}
2021-01-05 21:32:31 | INFO | fairseq.checkpoint_utils | Preparing to save checkpoint for epoch 10 @ 302 updates
2021-01-05 21:32:52 | INFO | fairseq.trainer | Finished saving checkpoint to checkpoints/checkpoint10.pt
2021-01-05 21:33:53 | INFO | fairseq.checkpoint_utils | Saved checkpoint checkpoints/checkpoint10.pt (epoch 10 @ 302 updates, score 99.981) (writing took 81.33670563600003 seconds)
2021-01-05 21:33:53 | INFO | fairseq_cli.train | end of epoch 10 (average epoch stats below)
2021-01-05 21:33:53 | INFO | train | {"epoch": 10, "train_loss": "1184.48", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "10.996", "train_wps": "956.9", "train_ups": "0.1", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "302", "train_lr": "1.19992e-05", "train_gnorm": "1266.52", "train_loss_scale": "0.5", "train_train_wall": "98", "train_wall": "1185"}
2021-01-05 21:33:53 | INFO | fairseq.trainer | begin training epoch 11
2021-01-05 21:35:31 | INFO | fairseq_cli.train | end of epoch 11 (average epoch stats below)
2021-01-05 21:35:31 | INFO | train | {"epoch": 11, "train_loss": "1143.88", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "10.619", "train_wps": "2958.4", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "333", "train_lr": "1.31796e-05", "train_gnorm": "1387.51", "train_loss_scale": "0.5", "train_train_wall": "97", "train_wall": "1284"}
2021-01-05 21:35:31 | INFO | fairseq.trainer | begin training epoch 12
2021-01-05 21:37:10 | INFO | fairseq_cli.train | end of epoch 12 (average epoch stats below)
2021-01-05 21:37:10 | INFO | train | {"epoch": 12, "train_loss": "1115.62", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "10.357", "train_wps": "2950.5", "train_ups": "0.31", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "364", "train_lr": "1.436e-05", "train_gnorm": "1375.05", "train_loss_scale": "0.5", "train_train_wall": "98", "train_wall": "1382"}
2021-01-05 21:37:10 | INFO | fairseq.trainer | begin training epoch 13
2021-01-05 21:38:48 | INFO | fairseq_cli.train | end of epoch 13 (average epoch stats below)
2021-01-05 21:38:48 | INFO | train | {"epoch": 13, "train_loss": "1096.54", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "10.18", "train_wps": "2965.8", "train_ups": "0.32", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "395", "train_lr": "1.55404e-05", "train_gnorm": "1254.28", "train_loss_scale": "0.5", "train_train_wall": "97", "train_wall": "1481"}
2021-01-05 21:38:48 | INFO | fairseq.trainer | begin training epoch 14
2021-01-05 21:39:04 | INFO | train_inner | {"epoch": 14, "update": 13.161, "loss": "1174.06", "ntokens": "9396.51", "nsentences": "86.41", "nll_loss": "10.797", "wps": "2219.4", "ups": "0.24", "wpb": "9396.5", "bsz": "86.4", "num_updates": "400", "lr": "1.57308e-05", "gnorm": "1346.79", "loss_scale": "0.5", "train_wall": "636", "wall": "1496"}
2021-01-05 21:40:25 | INFO | fairseq_cli.train | end of epoch 14 (average epoch stats below)
2021-01-05 21:40:25 | INFO | train | {"epoch": 14, "train_loss": "1057.17", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "9.814", "train_wps": "2988.9", "train_ups": "0.32", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "426", "train_lr": "1.67208e-05", "train_gnorm": "1320.55", "train_loss_scale": "0.5", "train_train_wall": "97", "train_wall": "1578"}
2021-01-05 21:40:26 | INFO | fairseq.trainer | begin training epoch 15
2021-01-05 21:42:04 | INFO | fairseq_cli.train | end of epoch 15 (average epoch stats below)
2021-01-05 21:42:04 | INFO | train | {"epoch": 15, "train_loss": "1049.59", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "9.744", "train_wps": "2969", "train_ups": "0.32", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "457", "train_lr": "1.79012e-05", "train_gnorm": "1282.37", "train_loss_scale": "0.5", "train_train_wall": "97", "train_wall": "1676"}
2021-01-05 21:42:04 | INFO | fairseq.trainer | begin training epoch 16
2021-01-05 21:43:41 | INFO | fairseq_cli.train | end of epoch 16 (average epoch stats below)
2021-01-05 21:43:41 | INFO | train | {"epoch": 16, "train_loss": "982.195", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "9.118", "train_wps": "2975.1", "train_ups": "0.32", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "488", "train_lr": "1.90815e-05", "train_gnorm": "1583.89", "train_loss_scale": "0.5", "train_train_wall": "97", "train_wall": "1774"}
2021-01-05 21:43:42 | INFO | fairseq.trainer | begin training epoch 17
2021-01-05 21:45:19 | INFO | fairseq_cli.train | end of epoch 17 (average epoch stats below)
2021-01-05 21:45:19 | INFO | train | {"epoch": 17, "train_loss": "949.922", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "8.819", "train_wps": "2973.3", "train_ups": "0.32", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "519", "train_lr": "2.02619e-05", "train_gnorm": "1651.61", "train_loss_scale": "0.5", "train_train_wall": "97", "train_wall": "1872"}
2021-01-05 21:45:19 | INFO | fairseq.trainer | begin training epoch 18
2021-01-05 21:46:57 | INFO | fairseq_cli.train | end of epoch 18 (average epoch stats below)
2021-01-05 21:46:57 | INFO | train | {"epoch": 18, "train_loss": "904.52", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "8.397", "train_wps": "2970.1", "train_ups": "0.32", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "550", "train_lr": "2.14423e-05", "train_gnorm": "1727.15", "train_loss_scale": "0.5", "train_train_wall": "97", "train_wall": "1970"}
2021-01-05 21:46:57 | INFO | fairseq.trainer | begin training epoch 19
2021-01-05 21:48:35 | INFO | fairseq_cli.train | end of epoch 19 (average epoch stats below)
2021-01-05 21:48:35 | INFO | train | {"epoch": 19, "train_loss": "869.769", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "8.075", "train_wps": "2975", "train_ups": "0.32", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "581", "train_lr": "2.26227e-05", "train_gnorm": "1444.59", "train_loss_scale": "0.5", "train_train_wall": "97", "train_wall": "2068"}
2021-01-05 21:48:35 | INFO | fairseq.trainer | begin training epoch 20
2021-01-05 21:49:35 | INFO | train_inner | {"epoch": 20, "update": 19.613, "loss": "956.214", "ntokens": "9375.11", "nsentences": "86.935", "nll_loss": "8.867", "wps": "2973", "ups": "0.32", "wpb": "9375.1", "bsz": "86.9", "num_updates": "600", "lr": "2.33462e-05", "gnorm": "1493.15", "loss_scale": "0.5", "train_wall": "626", "wall": "2127"}
2021-01-05 21:50:13 | INFO | fairseq_cli.train | begin validation on "valid" subset
2021-01-05 21:52:17 | INFO | valid | {"epoch": 20, "valid_loss": "551.863", "valid_ntokens": "2273.2", "valid_nsentences": "20.96", "valid_nll_loss": "5.088", "valid_uer": "99.965", "valid_wer": "99.996", "valid_raw_wer": "99.996", "valid_wps": "2319.7", "valid_wpb": "2273.2", "valid_bsz": "21", "valid_num_updates": "612", "valid_best_wer": "99.981"}
2021-01-05 21:52:17 | INFO | fairseq.checkpoint_utils | Preparing to save checkpoint for epoch 20 @ 612 updates
2021-01-05 21:52:37 | INFO | fairseq.trainer | Finished saving checkpoint to checkpoints/checkpoint20.pt
2021-01-05 21:53:18 | INFO | fairseq.checkpoint_utils | Saved checkpoint checkpoints/checkpoint20.pt (epoch 20 @ 612 updates, score 99.996) (writing took 61.534299623999686 seconds)
2021-01-05 21:53:18 | INFO | fairseq_cli.train | end of epoch 20 (average epoch stats below)
2021-01-05 21:53:18 | INFO | train | {"epoch": 20, "train_loss": "829.235", "train_ntokens": "9392.23", "train_nsentences": "87.1935", "train_nll_loss": "7.698", "train_wps": "1028.8", "train_ups": "0.11", "train_wpb": "9392.2", "train_bsz": "87.2", "train_num_updates": "612", "train_lr": "2.38031e-05", "train_gnorm": "1294.73", "train_loss_scale": "0.5", "train_train_wall": "97", "train_wall": "2351"}
2021-01-05 21:53:18 | INFO | fairseq.trainer | begin training epoch 21
2021-01-05 21:54:56 | INFO | fairseq_cli.train | end of epoch 21 (average epoch stats below)