MSID: 88538 Version: 2 DOI: 2<IP_ADDRESS>84540

[PLACE MANUSCRIPT AND EDITOR DETAILS HERE WHEN AVAILABLE] See step 3
[PLACE PDF URL HERE WHEN AVAILABLE] See step 7
Step 1. Inform bioRxiv
Who can help: @QueenKraken, @nlisgo, @scottaubrey

[x] Manuscript is in data hub index (https://data-hub-api.elifesciences.org/enhanced-preprints/docmaps/v1/by-publisher/elife/get-by-manuscript-id?manuscript_id=88538)

or (only one should be ticked. remove other from description.)

[ ] Confirmed with Ted at bioRxiv that MECA archive is available

Send the following email to Ted and wait for his reply.
Hi Ted,

Please can you prepare the preprint MECA for 10.1101/2<IP_ADDRESS>84540

Thanks

Step 2. Create preview of manuscript
Who can help: @fred-atherden, @nlisgo, @scottaubrey

[ ] Manuscript is available for preview (https://prod--epp.elifesciences.org/preview/88538/v2)

[ ] Trigger import for staging: https://prod--epp.elifesciences.org/import
[ ] Sync data folder with s3://prod-elife-epp-data/data
[ ] Trigger import for staging: https://staging--epp.elifesciences.org/import
[ ] Sync data folder with s3://staging-elife-epp-data/data
[ ] Pull request created to enhanced-preprint-data
[ ] Post link to pull request below

Pull request: [PLACE LINK TO PULL REQUEST HERE]
Detailed instructions: https://github.com/elifesciences/enhanced-preprints-data#add-a-manuscript
Step 3: Awaiting public reviews and QC
Who can help: Production team

[ ] QC passed or QC not needed

Who can help: Editorial team

[x] Public reviews available on sciety (https://sciety.org/articles/activity/10.1101/2<IP_ADDRESS>84540)
[ ] Manuscript and editor details at the top of this issue (Supplied by Editorial team)
[ ] Reviews returned by EPP - https://prod--epp.elifesciences.org/api/reviewed-preprints/88538/v2/reviews

Example
"msas": "Genetics and Genomics", "Neuroscience"
"msid": "84628"
"version": "1"
"preprintDoi": "10.1101/2<IP_ADDRESS>4241"
"articleType": "Reviewed Preprint"
"status": "Published from the original preprint after peer review and assessment by eLife."

"Reviewed Preprint posted": "2023-01-02"
"Sent for peer review": "2022-10-28"
"Posted to bioRxiv": "2022-11-21" (link: "Go to bioRxiv": "https://www.biorxiv.org/content/10.1101/2<IP_ADDRESS>4241v1")

Editors:
Reviewing Editor
Michael B Eisen
University of California, Berkeley, United States
Senior Editor
Michael B Eisen
University of California, Berkeley, United States

Step 4: Modify manuscripts.json (no PDF)

[ ] Reviewed preprint is published to EPP (https://elifesciences.org/reviewed-preprints/88538)

[ ] Pull request created to enhanced-preprint-client
[ ] Pull request switched to "Ready for review"
[ ] Post link to pull request below

[ ] Request that a doi be registered

[ ] Post link to #enhanced-preprint comment below

Pull request: [PLACE LINK TO PULL REQUEST HERE]
#enhanced-preprint comment thread: [PLACE LINK TO COMMENT HERE]

Instructions to modify manuscripts.json

Visit: https://github.com/elifesciences/enhanced-preprints-client/actions/workflows/publish-manuscript.yaml
Click: Run workflow
Complete the form and click "Run workflow"
A successful run should result in a new pull request at https://github.com/elifesciences/enhanced-preprints-client/pulls
Open the pull request and click the "Ready for review" button to trigger tests
Once the tests pass and you are happy with the changes the PR can be merged

Example pull request: https://github.com/elifesciences/enhanced-preprints-client/pull/334/files
Once the pull request is merged in it should be available a few minutes later.

Request that a doi
Post the following in #enhanced-preprint:
@Fred can you register a doi for https://elifesciences.org/reviewed-preprints/88538

Step 5: Awaiting search reindex

[ ] Reviewed preprint is avaliable on journal homepage (https://elifesciences.org)

The search reindex is triggered once an hour. We need the reviewed preprint to be indexed as the search application serves the journal homepage.

Additional info
If needed, the jenkins pipeline to reindex search can be triggered sooner.
https://alfred.elifesciences.org/job/process/job/process-reindex-reviewed-preprints/

Step 6: Published! Request PDF generation

[ ] Post the link to the PDF url at the top of the issue

[ ] Post link to #enhanced-preprint comment below

#enhanced-preprint comment thread: [PLACE LINK TO COMMENT HERE]
Post the following to the #enhanced-preprint on slack:
@Ryan Dix-Peek please can you generate a PDF for https://elifesciences.org/reviewed-preprints/88538

Step 7: Introduce PDF to data folder and git repo

[ ] PDF is avaliable at https://github.com/elifesciences/enhanced-preprints-data/raw/master/data/88538/v2/88538-v2.pdf

Detailed instructions: https://github.com/elifesciences/enhanced-preprints-data#add-a-pdf
Step 8: Add PDF url to manuscripts.json

[ ] Reviewed preprint PDF is available for download (https://prod--epp.elifesciences.org/reviewed-preprints/88538)

[ ] Pull request created to enhanced-preprint-client
[ ] Pull request switched to "Ready for review"
[ ] Post link to pull request below

[PLACE LINK TO PULL REQUEST HERE]

Instructions to add PDF url to manuscripts.json

Visit: https://github.com/elifesciences/enhanced-preprints-client/actions/workflows/add-pdf-url-to-manuscript.yaml
Click: Run workflow
Complete the form and click "Run workflow"
A successful run should result in a new pull request at https://github.com/elifesciences/enhanced-preprints-client/pulls
Open the pull request and click the "Ready for review" button to trigger tests
Once the tests pass and you are happy with the changes the PR can be merged

Example pull request: https://github.com/elifesciences/enhanced-preprints-client/pull/397/files
Once the pull request is merged in it should be available a few minutes later.

Step 9: Done!

[ ] Kettle is on!

The V2 preprint has a footnote section:

In the first EPP version, it looks like these were converted to sections but that hasn't happened here. I assume this needs redoing?
eLocations missing from references.

Well spotted @JGilbert-eLife. I think that content is taken from notes (in front):

XML snippet
<notes>
<notes notes-type="competing-interest-statement">
<title>Competing Interest Statement</title>
The authors have declared no competing interest.
</notes>
<notes notes-type="financial-disclosure">
<title>Funding Statement</title>
This study was supported by GM120523 (Q.F.), R01HL163854 (Q.F.), R35GM131770 (C.M.S.), HL133786 (W.Q.W.) and Vanderbilt Faculty Research Scholar Fund (Q.F.). The dataset(s) used for the analyses described were obtained from Vanderbilt University Medical Center's BioVU which is supported by institutional funding, the 1S10RR025141-01 instrumentation award, and by the CTSA grant UL1TR0004from NCATS/NIH. Additional funding provided by the NIH through grants P50GM115305 and U19HL065962. The authors wish to acknowledge the expert technical support of the VANTAGE and VANGARD core facilities, supported in part by the Vanderbilt-Ingram Cancer Center (P30 CA068485) and Vanderbilt Vision Center (P30 EY08126).
Role of the Funder/Sponsor: The funders had no role in design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.
Contributor Statement: L.J., G.L., A.O., A.I., A.L.D., L.L.D., C.P.C., N.C, W.Q.W., C.M.S. and Q.F. wrote the manuscript; L.J., C.M.S., and Q.F. designed the research; L.J., G.L., A.O., A.I., A.L.D., L.L.D., C.P.C., W.Q.W., C.M.S. and Q.F. performed the research; L.J., G.L., C.M.S and Q.F. analyzed the data.
</notes>
<notes notes-type="disclosures">
<title>Author Declarations</title>
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.

Yes

The details of the IRB/oversight body that provided approval or exemption for the research described are given below:

This study was reviewed by the VUMC Institutional Review Board; given the study's retrospective design and use of deidentified data only, informed consent was waived.

I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.

Yes

I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).

Yes

I have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.

Yes
</notes>
<fn-group content-type="summary-of-updates">
<title>Summary of Updates:</title>
<fn fn-type="update">
We have revised the title of the manuscript in response to reviewer's comments. Additionally, we have updated the manuscript with analyses among patients with pre-existing renal disease alone as well as other items suggested by the reviewers. The Tables have been renumbered to accommodate these revisions.
</fn>
</fn-group>
</notes>

which I assume is derived from forms filled out during submission.
We're not rendering notes anywhere as far as I'm aware.
Looking at the v1 I see there was also a notes section, but what was rendered was probably a more complete acknowledgements with much of the same type of information which is not as complete in the v2.
So I think we're OK to proceed here as is - I think long term there is work we/bioRxiv need to do to ensure this information is captured in a machine readable way in the XML and rendered on EPP appropriately. I don't think there's a simple solution since this content seems to be duplicated in some cases (where the authors have also included it in their PDF file, for example).
Let me know if you feel differently.


Errors running prepare_text.sh from wav2vec-u in fresh environment

My Question:
How can I get prepare_text.sh running correctly in a fresh Ubuntu Jupyterlab environment? What needs to be installed, what variables set, etc.?
I've run into various issues attempting to run the script prepare_text.sh, from https://github.com/pytorch/fairseq/blob/master/examples/wav2vec/unsupervised/scripts/prepare_text.sh.
Right now, I'm stuck on preprocess.py: error: unrecognized arguments: --dict-only, but I've run into some other errors that I've had to workaround, detailed below.
Full current output:
After getting through all the other issues I detail below, currently this is what I see when I attempt to run the script.
I cloned the https://github.com/pytorch/fairseq.git repo, and navigated to the scripts folder: https://github.com/pytorch/fairseq/tree/master/examples/wav2vec/unsupervised/scripts before running this.
(wav2vecu_pre) jovyan@user-ofmghcmafhv-jtfbeefyexclusive-0:~/work/fairseq/examples/wav2vec/unsupervised/scripts$ zsh prepare_text.sh sw /home/jovyan/work/WikiDumps/wiki_sw_head.txt /home/jovyan/work/WikiDumps/wiki_sw_head_wav2vecu_prepared.out
sw
sw
/home/jovyan/work/WikiDumps/wiki_sw_head.txt
/home/jovyan/work/WikiDumps/wiki_sw_head_wav2vecu_prepared.out
Warning : `load_model` does not return WordVectorModel or SupervisedModel any more, but a `FastText` object which is very similar.
usage: preprocess.py [-h] [--no-progress-bar] [--log-interval LOG_INTERVAL] [--log-format LOG_FORMAT] [--tensorboard-logdir TENSORBOARD_LOGDIR] [--seed SEED] [--cpu]
                     [--tpu] [--bf16] [--memory-efficient-bf16] [--fp16] [--memory-efficient-fp16] [--fp16-no-flatten-grads] [--fp16-init-scale FP16_INIT_SCALE]
                     [--fp16-scale-window FP16_SCALE_WINDOW] [--fp16-scale-tolerance FP16_SCALE_TOLERANCE] [--min-loss-scale MIN_LOSS_SCALE]
                     [--threshold-loss-scale THRESHOLD_LOSS_SCALE] [--user-dir USER_DIR] [--empty-cache-freq EMPTY_CACHE_FREQ] [--all-gather-list-size ALL_GATHER_LIST_SIZE]
                     [--model-parallel-size MODEL_PARALLEL_SIZE] [--checkpoint-suffix CHECKPOINT_SUFFIX] [--checkpoint-shard-count CHECKPOINT_SHARD_COUNT]
                     [--quantization-config-path QUANTIZATION_CONFIG_PATH] [--profile]
                     [--criterion {masked_lm,nat_loss,sentence_ranking,ctc,composite_loss,cross_entropy,legacy_masked_lm_loss,sentence_prediction,adaptive_loss,label_smoothed_cross_entropy,wav2vec,label_smoothed_cross_entropy_with_alignment,vocab_parallel_cross_entropy}]
                     [--tokenizer {moses,nltk,space}] [--bpe {sentencepiece,bytes,characters,byte_bpe,gpt2,hf_byte_bpe,fastbpe,subword_nmt,bert}]
                     [--optimizer {adam,adamax,adagrad,adafactor,adadelta,lamb,sgd,nag}]
                     [--lr-scheduler {triangular,fixed,reduce_lr_on_plateau,cosine,polynomial_decay,tri_stage,inverse_sqrt}] [--scoring {sacrebleu,bleu,wer,chrf}]
                     [--task TASK] [-s SRC] [-t TARGET] [--trainpref FP] [--validpref FP] [--testpref FP] [--align-suffix FP] [--destdir DIR] [--thresholdtgt N]
                     [--thresholdsrc N] [--tgtdict FP] [--srcdict FP] [--nwordstgt N] [--nwordssrc N] [--alignfile ALIGN] [--dataset-impl FORMAT] [--joined-dictionary]
                     [--only-source] [--padding-factor N] [--workers N]
preprocess.py: error: unrecognized arguments: --dict-only
cut: /home/jovyan/work/WikiDumps/wiki_sw_head_wav2vecu_prepared.out/dict.txt: No such file or directory
fatal error: PHONEMIZER_ESPEAK_PATH=espeak not found is not an executable file
fatal error: PHONEMIZER_ESPEAK_PATH=espeak not found is not an executable file
one is
sed: can't read /home/jovyan/work/WikiDumps/wiki_sw_head_wav2vecu_prepared.out/phones.txt: No such file or directory
paste: /home/jovyan/work/WikiDumps/wiki_sw_head_wav2vecu_prepared.out/phones.txt: No such file or directory
usage: preprocess.py [-h] [--no-progress-bar] [--log-interval LOG_INTERVAL] [--log-format LOG_FORMAT] [--tensorboard-logdir TENSORBOARD_LOGDIR] [--seed SEED] [--cpu]
                     [--tpu] [--bf16] [--memory-efficient-bf16] [--fp16] [--memory-efficient-fp16] [--fp16-no-flatten-grads] [--fp16-init-scale FP16_INIT_SCALE]
                     [--fp16-scale-window FP16_SCALE_WINDOW] [--fp16-scale-tolerance FP16_SCALE_TOLERANCE] [--min-loss-scale MIN_LOSS_SCALE]
                     [--threshold-loss-scale THRESHOLD_LOSS_SCALE] [--user-dir USER_DIR] [--empty-cache-freq EMPTY_CACHE_FREQ] [--all-gather-list-size ALL_GATHER_LIST_SIZE]
                     [--model-parallel-size MODEL_PARALLEL_SIZE] [--checkpoint-suffix CHECKPOINT_SUFFIX] [--checkpoint-shard-count CHECKPOINT_SHARD_COUNT]
                     [--quantization-config-path QUANTIZATION_CONFIG_PATH] [--profile]
                     [--criterion {masked_lm,nat_loss,sentence_ranking,ctc,composite_loss,cross_entropy,legacy_masked_lm_loss,sentence_prediction,adaptive_loss,label_smoothed_cross_entropy,wav2vec,label_smoothed_cross_entropy_with_alignment,vocab_parallel_cross_entropy}]
                     [--tokenizer {moses,nltk,space}] [--bpe {sentencepiece,bytes,characters,byte_bpe,gpt2,hf_byte_bpe,fastbpe,subword_nmt,bert}]
                     [--optimizer {adam,adamax,adagrad,adafactor,adadelta,lamb,sgd,nag}]
                     [--lr-scheduler {triangular,fixed,reduce_lr_on_plateau,cosine,polynomial_decay,tri_stage,inverse_sqrt}] [--scoring {sacrebleu,bleu,wer,chrf}]
                     [--task TASK] [-s SRC] [-t TARGET] [--trainpref FP] [--validpref FP] [--testpref FP] [--align-suffix FP] [--destdir DIR] [--thresholdtgt N]
                     [--thresholdsrc N] [--tgtdict FP] [--srcdict FP] [--nwordstgt N] [--nwordssrc N] [--alignfile ALIGN] [--dataset-impl FORMAT] [--joined-dictionary]
                     [--only-source] [--padding-factor N] [--workers N]
preprocess.py: error: unrecognized arguments: --dict-only
2021-06-03 16:39:42 | INFO | fairseq_cli.preprocess | Namespace(no_progress_bar=False, log_interval=100, log_format=None, tensorboard_logdir=None, seed=1, cpu=False, tpu=False, bf16=False, memory_efficient_bf16=False, fp16=False, memory_efficient_fp16=False, fp16_no_flatten_grads=False, fp16_init_scale=128, fp16_scale_window=None, fp16_scale_tolerance=0.0, min_loss_scale=0.0001, threshold_loss_scale=None, user_dir=None, empty_cache_freq=0, all_gather_list_size=16384, model_parallel_size=1, checkpoint_suffix='', checkpoint_shard_count=1, quantization_config_path=None, profile=False, criterion='cross_entropy', tokenizer=None, bpe=None, optimizer=None, lr_scheduler='fixed', scoring='bleu', task='translation', source_lang=None, target_lang=None, trainpref='/home/jovyan/work/WikiDumps/wiki_sw_head_wav2vecu_prepared.out/phones/lm.phones.filtered.txt', validpref=None, testpref=None, align_suffix=None, destdir='/home/jovyan/work/WikiDumps/wiki_sw_head_wav2vecu_prepared.out/phones', thresholdtgt=0, thresholdsrc=0, tgtdict=None, srcdict='/home/jovyan/work/WikiDumps/wiki_sw_head_wav2vecu_prepared.out/phones/dict.phn.txt', nwordstgt=-1, nwordssrc=-1, alignfile=None, dataset_impl='mmap', joined_dictionary=False, only_source=True, padding_factor=8, workers=70)
Traceback (most recent call last):
  File "/home/jovyan/work/fairseq//fairseq_cli/preprocess.py", line 401, in <module>
    cli_main()
  File "/home/jovyan/work/fairseq//fairseq_cli/preprocess.py", line 397, in cli_main
    main(args)
  File "/home/jovyan/work/fairseq//fairseq_cli/preprocess.py", line 98, in main
    src_dict = task.load_dictionary(args.srcdict)
  File "/opt/conda/envs/wav2vecu_pre/lib/python3.9/site-packages/fairseq/tasks/fairseq_task.py", line 54, in load_dictionary
    return Dictionary.load(filename)
  File "/opt/conda/envs/wav2vecu_pre/lib/python3.9/site-packages/fairseq/data/dictionary.py", line 214, in load
    d.add_from_file(f)
  File "/opt/conda/envs/wav2vecu_pre/lib/python3.9/site-packages/fairseq/data/dictionary.py", line 225, in add_from_file
    self.add_from_file(fd)
  File "/opt/conda/envs/wav2vecu_pre/lib/python3.9/site-packages/fairseq/data/dictionary.py", line 249, in add_from_file
    raise RuntimeError(
RuntimeError: Duplicate word found when loading Dictionary: '<SIL>'. Duplicate words can overwrite earlier ones by adding the #fairseq:overwrite flag at the end of the corresponding row in the dictionary file. If using the Camembert model, please download an updated copy of the model file.
prepare_text.sh:49: command not found: lmplz
prepare_text.sh:50: command not found: build_binary
python: can't open file '/home/jovyan/work/fairseq/examples/wav2vec/unsupervised/scripts/examples/speech_recognition/kaldi/kaldi_initializer.py': [Errno 2] No such file or directory
python: can't open file '/home/jovyan/work/fairseq/examples/wav2vec/unsupervised/scripts/examples/speech_recognition/kaldi/kaldi_initializer.py': [Errno 2] No such file or directory
prepare_text.sh:54: command not found: lmplz
prepare_text.sh:55: command not found: build_binary
prepare_text.sh:56: command not found: lmplz
prepare_text.sh:57: command not found: build_binary
Primary config directory not found.
Check that the config directory '/home/jovyan/work/fairseq/examples/speech_recognition/kaldi/config' exists and readable

Fixed (?) Problem: Can't seem to run it from the same folder as the README (workaround: run from scripts folder)
First, I can't run it from the same folder as the README at https://github.com/pytorch/fairseq/tree/master/examples/wav2vec/unsupervised#preparation-of-speech-and-text-data says to.  If you try doing so, you get errors with, e.g. path not found to other scripts.
zsh scripts/prepare_text.sh sw /home/jovyan/work/WikiDumps/wiki_sw_head.txt /home/jovyan/work/WikiDumps/wiki_sw_head_wav2vecu_prepared.out
sw
sw
/home/jovyan/work/WikiDumps/wiki_sw_head.txt
/home/jovyan/work/WikiDumps/wiki_sw_head_wav2vecu_prepared.out
python: can't open file '/home/jovyan/work/fairseq/examples/wav2vec/unsupervised/normalize_and_filter_text.py': [Errno 2] No such file or directory

Fixed (?) Problem: "ValueError: lid.187.bin cannot be opened for loading!" (workaround: use lid.176.bin instead)
Solution: download a different language ID model, and edit the code to use it.
https://fasttext.cc/docs/en/language-identification.html has a different model,  lid.176.bin
wget https://dl.fbaipublicfiles.com/fasttext/supervised-models/lid.176.bin

and edit this portion of normalize_and_filter_text.py:
    parser.add_argument(
        "--fasttext-model",
        help="path to fasttext model",
        default="lid.176.bin",
    )

Fixed (?) Problem: dependencies needed (phonemizer, fasttext, fairseq)
The script does not list which dependencies are needed. So far I've determined that phonemizer, fasttext are needed, and I think fairseq too. Any more I'm missing?
Fixed (?) Problem: can't find files in fairseq_cli: (solution: iYou need to set an environment variable, FAIRSEQ_ROOT).
I set this to point to the top level of the cloned repo. not sure if that's right.
(I cloned the repo to ~/work/fairseq/)
export FAIRSEQ_ROOT=~/work/fairseq/

Fixed (?) Problem: Not sure what language code to use. (guessed sw)
I've got Swahili data. Not sure whether to use sw, or swahili or what, I assume I should pick from https://github.com/espeak-ng/espeak-ng/blob/master/docs/languages.md
Code
Here's the command I use to invoke the script. Other than editing the default langid model, I haven't edited anything else in the repo, should be the same as https://github.com/pytorch/fairseq/tree/master/examples/wav2vec/unsupervised/scripts. git log shows c47a9b2eef0f41b0564c8daf52cb82ea97fc6548 as the commit.
zsh prepare_text.sh language /home/jovyan/work/WikiDumps/wiki_sw_head.txt /home/jovyan/work/WikiDumps/wiki_sw_head_wav2vecu_prepared.out

What have you tried?

Tried reading https://github.com/pytorch/fairseq/tree/master/examples/wav2vec/unsupervised#preparation-of-speech-and-text-data
Tried reading https://github.com/pytorch/fairseq/issues/3581 and https://github.com/pytorch/fairseq/issues/3586
Googling for various keywords such as "fairseq preprocess dict-only"

What's your environment?
I'm in a Jupyterlab in a Docker container, running Ubuntu.
OS is Ubuntu 20.04.2:
cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.2 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.2 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

pip list:
pip listPackage                Version
---------------------- -------------------
antlr4-python3-runtime 4.8
attrs                  21.2.0
certifi                2021.5.30
cffi                   1.14.5
clldutils              3.9.0
colorlog               5.0.1
csvw                   1.11.0
Cython                 0.29.23
dataclasses            0.6
editdistance           0.5.3
fairseq                0.10.0
fasttext               0.9.2
hydra-core             1.0.6
isodate                0.6.0
joblib                 1.0.1
numpy                  1.20.3
omegaconf              2.0.6
phonemizer             2.2.2
pip                    21.1.2
portalocker            2.0.0
pybind11               2.6.2
pycparser              2.20
python-dateutil        2.8.1
PyYAML                 5.4.1
regex                  2021.4.4
rfc3986                1.5.0
sacrebleu              1.5.1
segments               2.2.0
setuptools             49.6.0.post20210108
six                    1.16.0
tabulate               0.8.9
torch                  1.8.1
tqdm                   4.61.0
typing-extensions      <IP_ADDRESS>
uritemplate            3.0.1
wheel                  0.36.2

conda list:
conda list
# packages in environment at /opt/conda/envs/wav2vecu_pre:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       1_gnu    conda-forge
antlr4-python3-runtime    4.8                      pypi_0    pypi
attrs                     21.2.0                   pypi_0    pypi
ca-certificates           2021.5.30            ha878542_0    conda-forge
certifi                   2021.5.30        py39hf3d152e_0    conda-forge
cffi                      1.14.5                   pypi_0    pypi
clldutils                 3.9.0                    pypi_0    pypi
colorlog                  5.0.1                    pypi_0    pypi
csvw                      1.11.0                   pypi_0    pypi
cython                    0.29.23                  pypi_0    pypi
dataclasses               0.6                      pypi_0    pypi
editdistance              0.5.3                    pypi_0    pypi
fairseq                   0.10.0                   pypi_0    pypi
fasttext                  0.9.2                    pypi_0    pypi
hydra-core                1.0.6                    pypi_0    pypi
isodate                   0.6.0                    pypi_0    pypi
joblib                    1.0.1                    pypi_0    pypi
ld_impl_linux-64          2.35.1               hea4e1c9_2    conda-forge
libffi                    3.3                  h58526e2_2    conda-forge
libgcc-ng                 9.3.0               h2828fa1_19    conda-forge
libgomp                   9.3.0               h2828fa1_19    conda-forge
libstdcxx-ng              9.3.0               h6de172a_19    conda-forge
ncurses                   6.2                  h58526e2_4    conda-forge
numpy                     1.20.3                   pypi_0    pypi
omegaconf                 2.0.6                    pypi_0    pypi
openssl                   1.1.1k               h7f98852_0    conda-forge
phonemizer                2.2.2                    pypi_0    pypi
pip                       21.1.2             pyhd8ed1ab_0    conda-forge
portalocker               2.0.0                    pypi_0    pypi
pybind11                  2.6.2                    pypi_0    pypi
pycparser                 2.20                     pypi_0    pypi
python                    3.9.4           hffdb5ce_0_cpython    conda-forge
python-dateutil           2.8.1                    pypi_0    pypi
python_abi                3.9                      1_cp39    conda-forge
pyyaml                    5.4.1                    pypi_0    pypi
readline                  8.1                  h46c0cb4_0    conda-forge
regex                     2021.4.4                 pypi_0    pypi
rfc3986                   1.5.0                    pypi_0    pypi
sacrebleu                 1.5.1                    pypi_0    pypi
segments                  2.2.0                    pypi_0    pypi
setuptools                49.6.0           py39hf3d152e_3    conda-forge
six                       1.16.0                   pypi_0    pypi
sqlite                    3.35.5               h74cdb3f_0    conda-forge
tabulate                  0.8.9                    pypi_0    pypi
tk                        8.6.10               h21135ba_1    conda-forge
torch                     1.8.1                    pypi_0    pypi
tqdm                      4.61.0                   pypi_0    pypi
typing-extensions         <IP_ADDRESS>                 pypi_0    pypi
tzdata                    2021a                he74cb21_0    conda-forge
uritemplate               3.0.1                    pypi_0    pypi
wheel                     0.36.2             pyhd3deb0d_0    conda-forge
xz                        5.2.5                h516909a_1    conda-forge
zlib                      1.2.11            h516909a_1010    conda-forge

I also apt-installed phonemizer dependencies:
sudo apt-get install festival espeak-ng mbrola

And finally, here's what I get from apt list|grep installed
apt-list.txt

You have to install fairseq from git for a version of preprocess.py with --dict-only
Also:
prepare_text.sh:54: command not found: lmplz
prepare_text.sh:55: command not found: build_binary

You're missing kenlm

You have to install fairseq from git for a version of preprocess.py with --dict-only
Also:
prepare_text.sh:54: command not found: lmplz
prepare_text.sh:55: command not found: build_binary

You're missing kenlm

OK, that first bit worked fine:
pip uninstall fairseq
# (navigate to the top level of the repo)
pip install --editable ./

But now I'm not sure how to install kenlm. I tried pip install https://github.com/kpu/kenlm/archive/master.zip, but the error persists. I will try cloning and building the repo with make, etc

Yeah; you'll need cmake for that, Also:
apt-get -y install libeigen3-dev liblzma-dev zlib1g-dev libbz2-dev

I've been running make, then setup.py, but I'm not sure if that's strictly necessary (maybe just running setup.py is enough)

Followed instructions at https://github.com/kpu/kenlm/blob/master/BUILDING to install dependencies for kenlm. What they don't mention is that you need to take the resulting binaries from kenlm/build/bin/ and copy them to /usr/bin

That seems to have fixed those errors, now trying to figure out fatal error: PHONEMIZER_ESPEAK_PATH=espeak not found is not an executable file
Warning : `load_model` does not return WordVectorModel or SupervisedModel any more, but a `FastText` object which is very similar.
2021-06-03 18:37:41 | INFO | fairseq_cli.preprocess | Namespace(no_progress_bar=False, log_interval=100, log_format=None, log_file=None, tensorboard_logdir=None, wandb_project=None, azureml_logging=False, seed=1, cpu=False, tpu=False, bf16=False, memory_efficient_bf16=False, fp16=False, memory_efficient_fp16=False, fp16_no_flatten_grads=False, fp16_init_scale=128, fp16_scale_window=None, fp16_scale_tolerance=0.0, min_loss_scale=0.0001, threshold_loss_scale=None, amp=False, amp_batch_retries=2, amp_init_scale=128, amp_scale_window=None, user_dir=None, empty_cache_freq=0, all_gather_list_size=16384, model_parallel_size=1, quantization_config_path=None, profile=False, reset_logging=False, suppress_crashes=False, use_plasma_view=False, plasma_path='/tmp/plasma', criterion='cross_entropy', tokenizer=None, bpe=None, optimizer=None, lr_scheduler='fixed', simul_type=None, scoring='bleu', task='translation', source_lang=None, target_lang=None, trainpref='/home/jovyan/work/WikiDumps/wiki_sw_head_wav2vecu_prepared/lm.upper.lid.txt', validpref=None, testpref=None, align_suffix=None, destdir='/home/jovyan/work/WikiDumps/wiki_sw_head_wav2vecu_prepared', thresholdtgt=0, thresholdsrc=2, tgtdict=None, srcdict=None, nwordstgt=-1, nwordssrc=-1, alignfile=None, dataset_impl='mmap', joined_dictionary=False, only_source=True, padding_factor=1, workers=1, dict_only=True)
fatal error: PHONEMIZER_ESPEAK_PATH=espeak not found is not an executable file
fatal error: PHONEMIZER_ESPEAK_PATH=espeak not found is not an executable file

Above error seems to be coming from
sed 's/$/ 1/' $target_dir/words.txt | PHONEMIZER_ESPEAK_PATH=$(which espeak) phonemize -o $target_dir/phones.txt -p ' ' -w '' -l $ph_lg -j 70 --language-switch remove-flags

If you replace each instance of  which espeak with which espeak-ng, that fixes that

had some errors with kaldi_initializer. Lines 51 and 52 lacked $FAIRSEQ_ROOT/

Followed instructions at https://github.com/kpu/kenlm/blob/master/BUILDING to install dependencies for kenlm. What they don't mention is that you need to take the resulting binaries from kenlm/build/bin/ and copy them to /usr/bin

Oh yeah. kenlm was intended to be embedded in other projects, so it doesn't have any sort of install mechanism

Sorry to "hijack" this issue, but it seems that the kaldi_initializer shows the following issue for the two lines after building the word lm:

Traceback (most recent call last):
File "/nobackup/users/junruin2/fairseq//examples/speech_recognition/kaldi/kaldi_initializer.py", line 677, in cli_main
initalize_kaldi(cfg)
File "/nobackup/users/junruin2/fairseq//examples/speech_recognition/kaldi/kaldi_initializer.py", line 616, in initalize_kaldi
cfg.out_labels = cfg.in_labels
omegaconf.errors.MissingMandatoryValue: Missing mandatory value: in_labels
full_key: in_labels
reference_type=Optional[Dict[Union[str, Enum], Any]]
object_type=dict

What argument should be passed to it?

in_labels is phn for this script, out_labels is copied from in_labels if omitted

I've got most of the errors figured out, only one left:
Traceback (most recent call last):
  File "/home/jovyan/work/fairseq//fairseq_cli/preprocess.py", line 401, in <module>
    cli_main()
  File "/home/jovyan/work/fairseq//fairseq_cli/preprocess.py", line 397, in cli_main
    main(args)
  File "/home/jovyan/work/fairseq//fairseq_cli/preprocess.py", line 287, in main
    make_all(args.source_lang, src_dict)
  File "/home/jovyan/work/fairseq//fairseq_cli/preprocess.py", line 255, in make_all
    make_dataset(vocab, args.trainpref, "train", lang, num_workers=args.workers)
  File "/home/jovyan/work/fairseq//fairseq_cli/preprocess.py", line 251, in make_dataset
    make_binary_dataset(vocab, input_prefix, output_prefix, lang, num_workers)
  File "/home/jovyan/work/fairseq//fairseq_cli/preprocess.py", line 184, in make_binary_dataset
    100 * sum(replaced.values()) / n_seq_tok[1],
ZeroDivisionError: division by zero

Which comes from this line
python $FAIRSEQ_ROOT/fairseq_cli/preprocess.py --dataset-impl mmap --trainpref $target_dir/phones/lm.phones.filtered.txt --workers 70 --only-source --destdir $target_dir/phones --srcdict $target_dir/phones/dict.phn.txt

Try adding --thresholdsrc 2 or something similarly low; I didn't use this
for making the phone lm data, but setting the threshold was needed for
later calls to preprocess.py
Ar Déar 3 Meith 2021 ag 21:32, scríobh cdleong @.***>:

I've got most of the errors figured out, only one left:
Traceback (most recent call last):
File "/home/jovyan/work/fairseq//fairseq_cli/preprocess.py", line 401, in
cli_main()
File "/home/jovyan/work/fairseq//fairseq_cli/preprocess.py", line 397, in cli_main
main(args)
File "/home/jovyan/work/fairseq//fairseq_cli/preprocess.py", line 287, in main
make_all(args.source_lang, src_dict)
File "/home/jovyan/work/fairseq//fairseq_cli/preprocess.py", line 255, in make_all
make_dataset(vocab, args.trainpref, "train", lang, num_workers=args.workers)
File "/home/jovyan/work/fairseq//fairseq_cli/preprocess.py", line 251, in make_dataset
make_binary_dataset(vocab, input_prefix, output_prefix, lang, num_workers)
File "/home/jovyan/work/fairseq//fairseq_cli/preprocess.py", line 184, in make_binary_dataset
100 * sum(replaced.values()) / n_seq_tok[1],
ZeroDivisionError: division by zero
Which comes from this line
python $FAIRSEQ_ROOT/fairseq_cli/preprocess.py --dataset-impl mmap --trainpref $target_dir/phones/lm.phones.filtered.txt --workers 70 --only-source --destdir $target_dir/phones --srcdict $target_dir/phones/dict.phn.txt
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/pytorch/fairseq/issues/3591#issuecomment-854158699,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AABXQFX5G74F3YV73NY43FLTQ7RDXANCNFSM46BE3UTA
.

I'll give it a try.

Something went wrong, there should be output in those files
Ar Déar 3 Meith 2021 ag 21:44, scríobh cdleong @.***>:

I'll give it a try.

Reply to this email directly, view it on GitHub
https://github.com/pytorch/fairseq/issues/3591#issuecomment-854166098,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AABXQFWS5IM3NJIDV3REUQLTQ7ST5ANCNFSM46BE3UTA
.

Specifically the divide-by-zero is happening on this line, which means something wrong before then, I think: https://github.com/pytorch/fairseq/blob/c47a9b2eef0f41b0564c8daf52cb82ea97fc6548/examples/wav2vec/unsupervised/scripts/prepare_text.sh#L44

Specifically the divide-by-zero is happening on this line, which means something wrong before then, I think:

Nah, I was getting the same thing and the input was fine. The same divide-by-zero later was happening because the threshold was set too high, I'm going to assume the same is happening here.

Added the threshold, still getting divide-by-zero.
I'm still curious, not sure why lexicon_filtered.lst is empty.
python filter_lexicon.py -d $target_dir/phones/dict.txt < $target_dir/lexicon.lst >! $target_dir/lexicon_filtered.lst is the line that creates lexicon_filtered.lst,

$target_dir/lexicon.lst doesn't seem to be empty.
dict.txt in  $target_dir/phones/ has only <SIL> 0 in it
dict.xt in $target_dir/ has plenty of stuff in it.

Hmmmmmm...

Specifically I added the threshold to line 44, btw

I note that https://github.com/pytorch/fairseq/blob/c47a9b2eef0f41b0564c8daf52cb82ea97fc6548/examples/wav2vec/unsupervised/scripts/prepare_text.sh#L38 already has a threshold param, perhaps that's where I was supposed to change it.

OK, even with threshold 2 on every instance of preprocess.py, still getting divide by zero

Also, I missed this, but I'm getting this error as well:
Primary config directory not found.
Check that the config directory '/home/jovyan/work/fairseq/examples/speech_recognition/kaldi/config' exists and readable

Also, I missed this, but I'm getting this error as well:
Primary config directory not found.
Check that the config directory '/home/jovyan/work/fairseq/examples/speech_recognition/kaldi/config' exists and readable

I created an empty config directory there to bypass the test and ran with
lg=$lg python $FAIRSEQ_ROOT/examples/speech_recognition/kaldi/kaldi_initializer.py kaldi_root=/path/to/kaldi in_labels=phn out_labels=wrd fst_dir=$target_dir/fst/phn_to_words_sil lm_arpa=$target_dir/kenlm.wrd.o40003.arpa wav2letter_lexicon=$target_dir/lexicon_filtered.lst data_dir=$target_dir/phones "blank_symbol='<SIL>'" lg=$lg python $FAIRSEQ_ROOT/examples/speech_recognition/kaldi/kaldi_initializer.py kaldi_root=/path/to/kaldi in_labels=phn out_labels=wrd fst_dir=$target_dir/fst/phn_to_words lm_arpa=$target_dir/kenlm.wrd.o40003.arpa wav2letter_lexicon=$target_dir/lexicon_filtered.lst data_dir=$target_dir/phones

Also, I missed this, but I'm getting this error as well:
Primary config directory not found.
Check that the config directory '/home/jovyan/work/fairseq/examples/speech_recognition/kaldi/config' exists and readable

I created an empty config directory there to bypass the test and ran with
lg=$lg python $FAIRSEQ_ROOT/examples/speech_recognition/kaldi/kaldi_initializer.py kaldi_root=/path/to/kaldi in_labels=phn fst_dir=$target_dir/fst/phn_to_words_sil lm_arpa=$target_dir/kenlm.wrd.o40003.arpa wav2letter_lexicon=$target_dir/lexicon_filtered.lst data_dir=$target_dir/phones "blank_symbol='<SIL>'" lg=$lg python $FAIRSEQ_ROOT/examples/speech_recognition/kaldi/kaldi_initializer.py kaldi_root=/path/to/kaldi in_labels=phn fst_dir=$target_dir/fst/phn_to_words lm_arpa=$target_dir/kenlm.wrd.o40003.arpa wav2letter_lexicon=$target_dir/lexicon_filtered.lst data_dir=$target_dir/phones

I tried actually creating the yaml for it, and it ignores it completely 🤷

i am working on more comprehensive instructions on how to run the pipeline - should have something by next week - stay tuned. meanwhile i can answer questions here if need be

Also, I missed this, but I'm getting this error as well:
Primary config directory not found.
Check that the config directory '/home/jovyan/work/fairseq/examples/speech_recognition/kaldi/config' exists and readable

I created an empty config directory there to bypass the test and ran with
lg=$lg python $FAIRSEQ_ROOT/examples/speech_recognition/kaldi/kaldi_initializer.py kaldi_root=/path/to/kaldi in_labels=phn fst_dir=$target_dir/fst/phn_to_words_sil lm_arpa=$target_dir/kenlm.wrd.o40003.arpa wav2letter_lexicon=$target_dir/lexicon_filtered.lst data_dir=$target_dir/phones "blank_symbol='<SIL>'" lg=$lg python $FAIRSEQ_ROOT/examples/speech_recognition/kaldi/kaldi_initializer.py kaldi_root=/path/to/kaldi in_labels=phn fst_dir=$target_dir/fst/phn_to_words lm_arpa=$target_dir/kenlm.wrd.o40003.arpa wav2letter_lexicon=$target_dir/lexicon_filtered.lst data_dir=$target_dir/phones

I tried actually creating the yaml for it, and it ignores it completely shrug

I think I celebrated too early; when running kaldi_initializer it seems the two lines just quit in the middle before finishing building the final decoding graph. The text corpus was the LibriSpeech LM corpus (https://www.openslr.org/11; using librispeech-lm-norm.txt.gz as is before feeding to prepare_text.sh)
For the first line:

[2021-06-03 15:55:15,071][main][INFO] - Creating /nobackup/users/junruin2/fairseq/examples/wav2vec/unsupervised/librispeech_files/unpaired_text/fst/phn_to_words_sil/LG.phn.kenlm.wrd.o40003.fst
[2021-06-03 18:19:26,628][main][ERROR] - cmd: [PosixPath('/nobackup/users/junruin2/kaldi/src/fstbin/fstpushspecial')], err: /nobackup/users/junruin2/kaldi/src/fstbin/fstpushspecial

Traceback (most recent call last):
File "/nobackup/users/junruin2/fairseq//examples/speech_recognition/kaldi/kaldi_initializer.py", line 677, in cli_main
initalize_kaldi(cfg)
File "/nobackup/users/junruin2/fairseq//examples/speech_recognition/kaldi/kaldi_initializer.py", line 657, in initalize_kaldi
kaldi_root, fst_dir, unique_label, lexicon_graph, grammar_graph
File "/nobackup/users/junruin2/fairseq//examples/speech_recognition/kaldi/kaldi_initializer.py", line 273, in create_LG
check=True,
File "/nobackup/users/junruin2/anaconda3/envs/espnet-pt1.7.1/lib/python3.7/subprocess.py", line 512, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '[PosixPath('/nobackup/users/junruin2/kaldi/src/fstbin/fstpushspecial')]' died with <Signals.SIGKILL: 9>.
For the second line:

/kaldi/kaldi_initializer.py", line 677, in cli_main
initalize_kaldi(cfg)
File "/nobackup/users/junruin2/fairseq//examples/speech_recognition/kaldi/kaldi_initializer.py", line 660, in initalize_kaldi
kaldi_root, fst_dir, unique_label, h_graph, lg_graph, disambig_in_units_file_int
File "/nobackup/users/junruin2/fairseq//examples/speech_recognition/kaldi/kaldi_initializer.py", line 458, in create_HLGa
check=True,
File "/nobackup/users/junruin2/anaconda3/envs/espnet-pt1.7.1/lib/python3.7/subprocess.py", line 512, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '[PosixPath('/nobackup/users/junruin2/kaldi/src/fstbin/fsttablecompose'), PosixPath('/nobackup/users/junruin2/fairseq/examples/wav2vec/unsupervised/librispeech_files/unpaired_text/fst/phn_to_words/H.phn.fst'), PosixPath('/nobackup/users/junruin2/fairseq/examples/wav2vec/unsupervised/librispeech_files/unpaired_text/fst/phn_to_words/LG.phn.kenlm.wrd.o40003.fst')]' died with <Signals.SIGKILL: 9>.

Any idea why it would be the case? Also, in which stage are the fsts built by kaldi_initializer.py used?
Thanks!

most likely you ran out of cpu memory. try to prune the lm a bit more before building fst with it

Hello,
I have another problem with prepare_text.sh, words.txt is not created although i have dict.txt.

Here's what I have for dependencies so far, @alexeib does it look right to you?
pip
pip install phonemizer fasttext
git clone https://github.com/pytorch/fairseq
cd fairseq
pip install --editable ./

Apt
# phonemizer dependencies
sudo apt-get install festival espeak-ng mbrola

#kenlm dependencies from official website
sudo apt-get install build-essential libboost-all-dev cmake zlib1g-dev libbz2-dev liblzma-dev

Other:
Gotta build kenlm as shown here and copy the built binaries to /usr/bin. You need to apt-install the dependencies first.

Sorry for jumping into this issue again, but I assumed the people involved in the discussion here would be interested in at least replicating some results, so here it goes. Has anyone here successfully trained the GAN model on any of the corpora used by the original paper (https://ai.facebook.com/research/publications/unsupervised-speech-recognition/) and achieved somewhere close to the error rate there? If so, what modifications/hyper-parameters have you modified in the currently published code to achieve so? My attempts have been less than successful (https://github.com/pytorch/fairseq/issues/3581) for the past a week and a half, and I just cannot figure out why...

@alexeib
Hi, when I tried to use kaldi decoder for wav2vec_generate.py, I found that for some reason, the final HLG graphs were not successfully built when running kaldi_initializers.py in prepare_text.sh. I kept getting the following error, which occurred when executing the very last create_HLG() function within kaldi_initializers.py.
The kaldi version I have is from the latest pykaldi compatible fork:
https://github.com/pykaldi/kaldi/tree/pykaldi_02

In file included from /home/software/spack/gcc/8.3.0-xdjkb2mmftikxvoeeyaxtxcjtpltcgiz/include/c++/8.3.0/random:51,
from /nobackup/users/junruin2/pykaldi-py3.7.9/tools/kaldi/tools/openfst-1.6.7/include/fst/randgen.h:14,
from /nobackup/users/junruin2/pykaldi-py3.7.9/tools/kaldi/tools/openfst-1.6.7/include/fst/randequivalent.h:15,
from /nobackup/users/junruin2/pykaldi-py3.7.9/tools/kaldi/tools/openfst-1.6.7/include/fst/fstlib.h:61,
from /nobackup/users/junruin2/pykaldi-py3.7.9/tools/kaldi/src/fstext/fstext-lib.h:22,
from /nobackup/users/junruin2/fairseq/examples/speech_recognition/kaldi/add-self-loop-simple.cc:9:
/home/software/spack/gcc/8.3.0-xdjkb2mmftikxvoeeyaxtxcjtpltcgiz/include/c++/8.3.0/bits/random.tcc:2737:5: note: candidate: 'template<class _IntType, class _CharT, class _Traits> std::basic_ostream<_CharT, _Traits>& std::operator<<(std::basic_ostream<_CharT, _Traits>&, const std::discrete_distribution<_IntType>&)'
operator<<(std::basic_ostream<_CharT, _Traits>& __os,
^~~~~~~~
/home/software/spack/gcc/8.3.0-xdjkb2mmftikxvoeeyaxtxcjtpltcgiz/include/c++/8.3.0/bits/random.tcc:2737:5: note:   template argument deduction/substitution failed:
/nobackup/users/junruin2/fairseq/examples/speech_recognition/kaldi/add-self-loop-simple.cc:91:52: note:   'kaldi::MessageLogger' is not derived from 'std::basic_ostream<_CharT, _Traits>'
KALDI_LOG << "Writing FST to " << output << std::endl;

Hi, when I tried to use kaldi decoder for wav2vec_generate.py, I found that for some reason, the final HLG graphs were not successfully built when running kaldi_initializers.py in prepare_text.sh. I kept getting the following error, which occurred when executing the very last create_HLG() function within kaldi_initializers.py.

In file included from /home/software/spack/gcc/8.3.0-xdjkb2mmftikxvoeeyaxtxcjtpltcgiz/include/c++/8.3.0/random:51,
from /nobackup/users/junruin2/pykaldi-py3.7.9/tools/kaldi/tools/openfst-1.6.7/include/fst/randgen.h:14,
from /nobackup/users/junruin2/pykaldi-py3.7.9/tools/kaldi/tools/openfst-1.6.7/include/fst/randequivalent.h:15,
from /nobackup/users/junruin2/pykaldi-py3.7.9/tools/kaldi/tools/openfst-1.6.7/include/fst/fstlib.h:61,
from /nobackup/users/junruin2/pykaldi-py3.7.9/tools/kaldi/src/fstext/fstext-lib.h:22,
from /nobackup/users/junruin2/fairseq/examples/speech_recognition/kaldi/add-self-loop-simple.cc:9:
/home/software/spack/gcc/8.3.0-xdjkb2mmftikxvoeeyaxtxcjtpltcgiz/include/c++/8.3.0/bits/random.tcc:2737:5: note: candidate: 'template<class _IntType, class _CharT, class _Traits> std::basic_ostream<_CharT, _Traits>& std::operator<<(std::basic_ostream<_CharT, _Traits>&, const std::discrete_distribution<_IntType>&)'
operator<<(std::basic_ostream<_CharT, _Traits>& __os,
^~~~~~~~
/home/software/spack/gcc/8.3.0-xdjkb2mmftikxvoeeyaxtxcjtpltcgiz/include/c++/8.3.0/bits/random.tcc:2737:5: note:   template argument deduction/substitution failed:
/nobackup/users/junruin2/fairseq/examples/speech_recognition/kaldi/add-self-loop-simple.cc:91:52: note:   'kaldi::MessageLogger' is not derived from 'std::basic_ostream<_CharT, _Traits>'
KALDI_LOG << "Writing FST to " << output << std::endl;

The kaldi version I have is from the latest pykaldi compatible fork:
https://github.com/pykaldi/kaldi/tree/pykaldi_02
However, the code seems to be running for now as long as I remove all lines that has an ostream regarding  KALDI_LOG. Really don't think it will affect anything other than logging though.

@JeromeNi I met the seem problem, too. I just ignored them by removing the related lines as you. There is another problem when I run create_HLG() . The following codes in kaldi_initializers.py need to find the libkaldi-base.so and libkaldi-fstext.so.  I have installed pykaldi, but the libraies are still missing.

if not add_self_loop.exists():
fst_include = kaldi_root / "tools/openfst-1.6.7/include"
add_self_loop_src = script_dir / "add-self-loop-simple.cc"
             subprocess.run(
                 [
                     "c++",
                     f"-I{kaldi_src}",
                     f"-I{fst_include}",
                     f"-L{kaldi_lib}",
                     add_self_loop_src,
                     "-lkaldi-base",
                     "-lkaldi-fstext",
                     "-o",
                     add_self_loop,
                 ],
                 check=True,
             )

The error messages are below. Did you meet this?

[2021-06-17 10:01:11,763][main][INFO] - Creating /home/exp_fairseq/w2vu/librispeech_10h/prep_text_0.1_all/fst/phn_to_words_sil/HLG.phn.kenlm.wrd.o40003.fst
/usr/bin/ld: cannot find -lkaldi-base
/usr/bin/ld: cannot find -lkaldi-fstext
collect2: error: ld returned 1 exit status
Traceback (most recent call last):
File "/home/share/fairseq_u/examples/speech_recognition/kaldi/kaldi_initializer.py", line 589, in create_HLG
check=True,
File "/home/share/anaconda3_share/envs/zj-w2vu-w2l/lib/python3.6/subprocess.py", line 418, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['c++', '-I/home/share/kaldi/src', '-I/home/share/kaldi/tools/openfst-1.6.7/include', '-L/home/share/kaldi/src/lib', PosixPath('/home/share/fairseq_u/examples/speech_recognition/kaldi/add-self-loop-simple.cc'), '-lkaldi-base', '-lkaldi-fstext', '-o', PosixPath('/home/share/fairseq_u/examples/speech_recognition/kaldi/add-self-loop-simple')]' returned non-zero exit status 1.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/share/fairseq_u/examples/speech_recognition/kaldi/kaldi_initializer.py", line 677, in cli_main
initalize_kaldi(cfg)
File "/home/share/fairseq_u/examples/speech_recognition/kaldi/kaldi_initializer.py", line 662, in initalize_kaldi
hlg_graph = create_HLG(kaldi_root, fst_dir, unique_label, hlga_graph)
File "/home/share/fairseq_u/examples/speech_recognition/kaldi/kaldi_initializer.py", line 606, in create_HLG
logger.error(f"cmd: {e.cmd}, err: {e.stderr.decode('utf-8')}")
AttributeError: 'NoneType' object has no attribute 'decode'

i am working on more comprehensive instructions on how to run the pipeline - should have something by next week - stay tuned. meanwhile i can answer questions here if need be

is it finished?

yes, instructions should be good now.
regarding building the binary for adding self loops - i have plans to rewrite this using pykaldi api instead of c++ but it may take some time. meanwhile, you probably need to build the kaldi toolkit in your pykaldi dir (they have a script for that I believe)

we will have working timit instructions up next week
for librispeech, you can get it to work with as little as 10h of audio, but depending on what you use for text you may need to adjust the 1k threshold when building phone dict

Hello, I have a problem prepare_text.sh. lexicon.lst file is not created by prepare_text.sh but it wants to use it. What am I supposed to do about this?
Another question of mine is language. Can I finetune for my language? @alexeib

it should be created by this line in the script:
paste $target_dir/words.txt $target_dir/phones.txt >! $target_dir/lexicon.lst
maybe some intermediate step failed?
sorry i dont understand the language question - what exactly do you want to finetune ?

I saw that line should work but it did not. Thus I created a manual lexicon.lst. Then filtered_lexicon.lst is absent and again failed.
preparing text is challenging to me.I don't know the exact reason for this situation. Some intermediate steps may be failed but I did not get any error about it. Seeing a working prepared step outputs would be great.  @alexeib

Hello, I can't run script prepare_text.sh, I get a error
[2021-06-22 19:44:15,455][main][INFO] - Creating /hdd/conda_kaldi/exp_unsup_asr/prepare_text/fst/phn_to_phn_sil/H.phn.fst
[2021-06-22 19:44:15,556][main][INFO] - Creating /hdd/conda_kaldi/exp_unsup_asr/prepare_text/fst/phn_to_phn_sil/L.phn.lm.phones.filtered.06.fst (in units: /hdd/conda_kaldi/exp_unsup_asr/prepare_text/fst/phn_to_phn_sil/kaldi_dict.phn_disambig.txt)
[2021-06-22 19:44:15,686][main][INFO] - Creating /hdd/conda_kaldi/exp_unsup_asr/prepare_text/fst/phn_to_phn_sil/LG.phn.lm.phones.filtered.06.fst
[2021-06-22 19:44:16,007][main][INFO] - Creating /hdd/conda_kaldi/exp_unsup_asr/prepare_text/fst/phn_to_phn_sil/HLGa.phn.lm.phones.filtered.06.fst
[2021-06-22 19:44:16,500][main][INFO] - Creating /hdd/conda_kaldi/exp_unsup_asr/prepare_text/fst/phn_to_phn_sil/HLG.phn.lm.phones.filtered.06.fst
[2021-06-22 19:44:17,414][main][ERROR] - cmd: [PosixPath('/hdd/conda_kaldi/rnd_ds/fairseq/examples/speech_recognition/kaldi/add-self-loop-simple'), PosixPath('/hdd/conda_kaldi/exp_unsup_asr/prepare_text/fst/phn_to_phn_sil/HLGa.phn.lm.phones.filtered.06.fst'), PosixPath('/hdd/conda_kaldi/exp_unsup_asr/prepare_text/fst/phn_to_phn_sil/HLG.phn.lm.phones.filtered.06.fst')], err: b''
Traceback (most recent call last):
File "/hdd/conda_kaldi/rnd_ds/fairseq/examples/speech_recognition/kaldi/kaldi_initializer.py", line 677, in cli_main
initalize_kaldi(cfg)
File "/hdd/conda_kaldi/rnd_ds/fairseq/examples/speech_recognition/kaldi/kaldi_initializer.py", line 662, in initalize_kaldi
hlg_graph = create_HLG(kaldi_root, fst_dir, unique_label, hlga_graph)
File "/hdd/conda_kaldi/rnd_ds/fairseq/examples/speech_recognition/kaldi/kaldi_initializer.py", line 595, in create_HLG
subprocess.run(
File "/hdd/conda_kaldi/rnd_ds/lib/python3.8/subprocess.py", line 516, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '[PosixPath('/hdd/conda_kaldi/rnd_ds/fairseq/examples/speech_recognition/kaldi/add-self-loop-simple'), PosixPath('/hdd/conda_kaldi/exp_unsup_asr/prepare_text/fst/phn_to_phn_sil/HLGa.phn.lm.phones.filtered.06.fst'), PosixPath('/hdd/conda_kaldi/exp_unsup_asr/prepare_text/fst/phn_to_phn_sil/HLG.phn.lm.phones.filtered.06.fst')]' died with <Signals.SIGSEGV: 11>.
Do you have any idea why? @alexeib

you probably ran out of memory because your corpus/num of phonemes is too big. you can change the script to prune the lm or create a 4 or 5gram phone lm instead of 6gram, for use in the WFST

I saw that line should work but it did not. Thus I created a manual lexicon.lst. Then filtered_lexicon.lst is absent and again failed.
preparing text is challenging to me.I don't know the exact reason for this situation. Some intermediate steps may be failed but I did not get any error about it. Seeing a working prepared step outputs would be great. @alexeib
I managed to create a lexicon filtered and now I got zero division error.

maybe your corpus is too small and you need to use a smaller phone cutoff threshold? (the example in the readme uses 1000 which is suitable for medium to large corpuses, but not small)

I saw that line should work but it did not. Thus I created a manual lexicon.lst. Then filtered_lexicon.lst is absent and again failed.
preparing text is challenging to me.I don't know the exact reason for this situation. Some intermediate steps may be failed but I did not get any error about it. Seeing a working prepared step outputs would be great. @alexeib
I managed to create a lexicon filtered and now I got zero division error.

maybe your corpus is too small and you need to use a smaller phone cutoff threshold? (the example in the readme uses 1000 which is suitable for medium to large corpuses, but not small)

I tried with even 2 as threshold but again it gives an error. What should be the size of corpus ?

@alexeib thank you for updating the instructions! They look much improved!
I've finally gotten around to looking at them again, and I have a few things I ran into that I wanted to suggest changes about.

## suggestion: add $FAIRSEQ_ROOT to commands
In https://github.com/pytorch/fairseq/tree/master/examples/wav2vec/unsupervised#preparation-of-speech-and-text-data, there is this code block:

create a manifest file for the set original of audio files
python $FAIRSEQ_ROOT/examples/wav2vec/wav2vec_manifest.py /dir/to/save/audio/files --ext wav --dest /path/to/new/train.tsv --valid-percent 0
python scripts/vads.py -r $RVAD_ROOT < /path/to/train.tsv > train.vads
python scripts/remove_silence.py --tsv /path/to/train.tsv --vads train.vads --out /dir/to/save/audio/files
python $FAIRSEQ_ROOT/examples/wav2vec/wav2vec_manifest.py /dir/to/save/audio/files --ext wav --dest /path/to/new/train.tsv --valid-percent 0.01
I think that the middle two lines could be updated to include $FAIRSEQ_ROOT as well, like so:
python $FAIRSEQ_ROOT/examples/wav2vec/unsupervised/scripts/vads.py -r $RVAD_ROOT < /path/to/train.tsv > train.vads
python $FAIRSEQ_ROOT/examples/wav2vec/unsupervised/scripts/remove_silence.py --tsv /path/to/train.tsv --vads train.vads --out /dir/to/save/audio/files

suggestion: clarify "/path/to/new/train.tsv"
The --dest arg on this command may be misleading. It actually doesn't want the path to a new file, it wants the path to a directory in which it will create the new train.tsv.
python $FAIRSEQ_ROOT/examples/wav2vec/wav2vec_manifest.py /dir/to/save/audio/files --ext wav --dest /path/to/new/train.tsv --valid-percent 0

When I gave it a the path to the directory ./foo/, it created ./foo/train.tsv in that directory.
On the other hand, when I gave it the path ./bar/train.tsv, it created the directory ./bar/train.tsv/, and created train.tsv inside that, so I ended up with ./bar/train.tsv/train.tsv
suggestion: add quotes around all the variables.
It's often helpful to avoid issues with spaces in the paths, to just wrap all bash variables in quotes.
"$FAIRSEQ_ROOT"

I think these might help the instructions be even easier for people to follow along with. Any thoughts?

those are all great suggestions. if you would like to submit a PR to improve docs, you are most welcome to do so! otherwise i will keep this in mind when i next touch w2v-u code

Roger! I'm still going through and taking notes, but I think I might be able to contribute some stuff.
Here's a new note, I just discovered that faiss and npy-append-array are also dependencies for the preprocessing. They're used in prepare_audio.sh

Another thing I just ran across: IndexError: list index out of range in wav2vec_cluster_faiss.py
I think it's caused by the fact that I was using https://dl.fbaipublicfiles.com/fairseq/wav2vec/wav2vec_small.pt, which may simply just have 12 layers. I edited the script to print out the length of the relevant variables...
            print(f"res length: " + str(len(res["layer_results"])))
            print(f"self.layer: {self.layer}")

and I get an output like so.
res length: 12
self.layer: 14
  0%|                                                                                | 0/1497 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "/home/jovyan/fairseq/examples/wav2vec/unsupervised/scripts/wav2vec_cluster_faiss.py", line 219, in <module>
    main()
  File "/home/jovyan/fairseq/examples/wav2vec/unsupervised/scripts/wav2vec_cluster_faiss.py", line 153, in main
    for f in tqdm.tqdm(iterator, total=num):
  File "/opt/conda/envs/wav2vecu/lib/python3.7/site-packages/tqdm/std.py", line 1178, in __iter__
    for obj in iterable:
  File "/home/jovyan/fairseq/examples/wav2vec/unsupervised/scripts/wav2vec_cluster_faiss.py", line 132, in iterate
    feats = reader.get_feats(fname)
  File "/home/jovyan/fairseq/examples/wav2vec/unsupervised/scripts/wav2vec_cluster_faiss.py", line 110, in get_feats
    res_layer = res["layer_results"][self.layer]
IndexError: list index out of range

Gonna try again with the larger model, https://dl.fbaipublicfiles.com/fairseq/wav2vec/libri960_big.pt

Alas, when I try that, it fails in the extract_features step:
Traceback (most recent call last):
  File "/home/jovyan/fairseq/examples/wav2vec/unsupervised/scripts/wav2vec_extract_features.py", line 119, in <module>
    main()
  File "/home/jovyan/fairseq/examples/wav2vec/unsupervised/scripts/wav2vec_extract_features.py", line 107, in main
    generator, num = get_iterator(args)
  File "/home/jovyan/fairseq/examples/wav2vec/unsupervised/scripts/wav2vec_extract_features.py", line 76, in get_iterator
    reader = Wav2VecFeatureReader(args.checkpoint, args.layer)
  File "/home/jovyan/fairseq/examples/wav2vec/unsupervised/scripts/wav2vec_extract_features.py", line 39, in __init__
    [cp_file]
  File "/home/jovyan/fairseq/fairseq/checkpoint_utils.py", line 446, in load_model_ensemble_and_task
    model = task.build_model(cfg.model)
  File "/home/jovyan/fairseq/fairseq/tasks/audio_pretraining.py", line 294, in build_model
    model = super().build_model(model_cfg)
  File "/home/jovyan/fairseq/fairseq/tasks/fairseq_task.py", line 324, in build_model
    model = models.build_model(cfg, self)
  File "/home/jovyan/fairseq/fairseq/models/__init__.py", line 96, in build_model
    return model.build_model(cfg, task)
  File "/home/jovyan/fairseq/fairseq/models/wav2vec/wav2vec2_asr.py", line 176, in build_model
    w2v_encoder = Wav2VecEncoder(cfg, len(task.target_dictionary))
  File "/home/jovyan/fairseq/fairseq/tasks/audio_pretraining.py", line 267, in target_dictionary
    return self.state.target_dictionary
  File "/home/jovyan/fairseq/fairseq/tasks/fairseq_task.py", line 41, in __getattr__
    self._state[name] = self._factories[name]()
  File "/home/jovyan/fairseq/fairseq/tasks/audio_pretraining.py", line 178, in load_target_dictionary
    return Dictionary.load(dict_path)
  File "/home/jovyan/fairseq/fairseq/data/dictionary.py", line 225, in load
    d.add_from_file(f)
  File "/home/jovyan/fairseq/fairseq/data/dictionary.py", line 238, in add_from_file
    raise fnfe
  File "/home/jovyan/fairseq/fairseq/data/dictionary.py", line 235, in add_from_file
    with open(PathManager.get_local_path(f), "r", encoding="utf-8") as fd:
FileNotFoundError: [Errno 2] No such file or directory: '/checkpoint/abaevski/data/speech/libri/960h/wav2vec/raw/dict.ltr.txt

Tried again with xlsr_53_56k.pt, don't get FileNotFoundError.

Tried tracing the FileNotFoundError back, and it seems that when loading in https://dl.fbaipublicfiles.com/fairseq/wav2vec/libri960_big.pt, it actually contains the following key/value pair within it:
'data': '/checkpoint/abaevski/data/speech/libri/960h/wav2vec/raw/'

whereas https://dl.fbaipublicfiles.com/fairseq/wav2vec/xlsr_53_56k.pt does not.

I went to load_checkpoint_to_cpu() in checkpoint_utils, and added some print statements to see what's in there, right before the return statement.
When I load the XLSR 53 pretrained model and look at state["cfg"]["task"] I see
{'_name': 'audio_pretraining', 'data': '/private/home/aconneau/projects/XLSR/MLS/53bis/', 'labels': None, 'sample_rate': 16000, 'normalize': True, 'enable_padding': False, 'max_sample_size': 320000, 'min_sample_size': 32000, 'eval_wer': False, 'eval_wer_config': {'_name': None, 'beam': 5, 'nbest': 1, 'max_len_a': 0.0, 'max_len_b': 200, 'min_len': 1, 'match_source_len': False, 'unnormalized': False, 'no_early_stop': False, 'no_beamable_mm': False, 'lenpen': 1.0, 'unkpen': 0.0, 'replace_unk': None, 'sacrebleu': False, 'score_reference': False, 'prefix_size': 0, 'no_repeat_ngram_size': 0, 'sampling': False, 'sampling_topk': -1, 'sampling_topp': -1.0, 'constraints': None, 'temperature': 1.0, 'diverse_beam_groups': -1, 'diverse_beam_strength': 0.5, 'diversity_rate': -1.0, 'print_alignment': 'hard', 'print_step': False, 'lm_path': None, 'lm_weight': 0.0, 'iter_decode_eos_penalty': 0.0, 'iter_decode_max_iter': 10, 'iter_decode_force_max_iter': False, 'iter_decode_with_beam': 1, 'iter_decode_with_external_reranker': False, 'retain_iter_history': False, 'retain_dropout': False, 'retain_dropout_modules': None, 'decoding_format': None, 'no_seed_provided': False}, 'eval_wer_tokenizer': None, 'eval_wer_post_process': 'letter', 'autoregressive': False}

whereas for 960h, we get
{'_name': 'audio_pretraining', 'data': '/checkpoint/abaevski/data/speech/libri/960h/wav2vec/raw/', 'labels': 'ltr', 'binarized_dataset': False, 'sample_rate': 16000, 'normalize': False, 'enable_padding': False, 'max_sample_size': None, 'min_sample_size': None, 'eval_wer': False, 'eval_wer_config': {'_name': None, 'beam': 5, 'nbest': 1, 'max_len_a': 0.0, 'max_len_b': 200, 'min_len': 1, 'match_source_len': False, 'unnormalized': False, 'no_early_stop': False, 'no_beamable_mm': False, 'lenpen': 1.0, 'unkpen': 0.0, 'replace_unk': None, 'sacrebleu': False, 'score_reference': False, 'prefix_size': 0, 'no_repeat_ngram_size': 0, 'sampling': False, 'sampling_topk': -1, 'sampling_topp': -1.0, 'constraints': None, 'temperature': 1.0, 'diverse_beam_groups': -1, 'diverse_beam_strength': 0.5, 'diversity_rate': -1.0, 'print_alignment': None, 'print_step': False, 'lm_path': None, 'lm_weight': 0.0, 'iter_decode_eos_penalty': 0.0, 'iter_decode_max_iter': 10, 'iter_decode_force_max_iter': False, 'iter_decode_with_beam': 1, 'iter_decode_with_external_reranker': False, 'retain_iter_history': False, 'retain_dropout': False, 'retain_dropout_modules': None, 'decoding_format': None, 'no_seed_provided': False}, 'eval_wer_tokenizer': None, 'eval_wer_post_process': 'letter', 'autoregressive': False, 'num_batch_buckets': 0, 'precompute_mask_indices': False, 'inferred_w2v_config': None, 'tpu': True}

which look very similar! So why does one succeed, and the other fail?

Ah, one has "labels", the other does not. So this is the divergence point: https://github.com/pytorch/fairseq/blob/cdc1a553eb2af4fac720880aff4ee2566a28ad21/fairseq/tasks/audio_pretraining.py#L176

And one of them gets set with state["args"] and the other gets set with state["cfg"} in checkpoint_utils. Hm.

So, something I just discovered. prepare_text.sh doesn't seem to like it when you give it a local path for the output in the following command:
zsh scripts/prepare_text.sh language /path/to/text/file /output/dir 1000 espeak /path/to/fasttext/lid/model

I set the output dir to ./prepared_text/, and got an error on https://github.com/pytorch/fairseq/blob/d18e44a28994a1558e9f8dc988a23bd6f77a55d5/examples/speech_recognition/kaldi/kaldi_initializer.py#L626, "No such file". But the error message was looking for prepared_output/phones/dict.phn.txt, without the ./ portion.

So, something I just discovered. prepare_text.sh doesn't seem to like it when you give it a local path for the output in the following command:
zsh scripts/prepare_text.sh language /path/to/text/file /output/dir 1000 espeak /path/to/fasttext/lid/model

I set the output dir to ./prepared_text/, and got an error on
https://github.com/pytorch/fairseq/blob/d18e44a28994a1558e9f8dc988a23bd6f77a55d5/examples/speech_recognition/kaldi/kaldi_initializer.py#L626
, "No such file". But the error message was looking for prepared_output/phones/dict.phn.txt, without the ./ portion.

specifically the above error happens on this line of prepare_text.sh:
lg=$lg python $FAIRSEQ_ROOT/examples/speech_recognition/kaldi/kaldi_initializer.py kaldi_root=$KALDI_ROOT fst_dir=$target_dir/fst/phn_to_words_sil lm_arpa=$target_dir/kenlm.wrd.o40003.arpa wav2letter_lexicon=$target_dir/lexicon_filtered.lst data_dir=$target_dir/phones in_labels=phn "blank_symbol='<SIL>'"

Latest install script: after running the following, I can get through prepare_audio.sh and about halfway through prepare_text.sh. on a clean docker image Jupyter Docker with Ubuntu 20.04.
#!/bin/bash
set -o errexit
set -o pipefail
set -o nounset

install_folder="$HOME"

# from https://erictleung.com/conda-in-subshell-script, lets you run conda activate.
eval "$(conda shell.bash hook)"

# phonemizer dependencies
sudo apt update
sudo apt-get install -y festival espeak-ng mbrola g++ zsh

# wav2vecu preprocessing
conda create -n wav2vecu pip python=3.7 --yes
conda activate wav2vecu
which pip

# install pykaldi
conda install -c pykaldi pykaldi --yes

# needed for prepare_audio.sh later:
conda install -c pytorch faiss-gpu --yes
pip install npy-append-array

# install FAIRSEQ
cd "$install_folder"
git clone https://github.com/pytorch/fairseq
cd fairseq
pip install --editable ./
echo "export FAIRSEQ_ROOT=$PWD" >> ~/.bashrc

# RVADFast
cd "$install_folder"
git clone https://github.com/zhenghuatan/rVADfast.git && cd rVADfast
echo "export RVAD_ROOT=$PWD" >> ~/.bashrc

# KenLM
cd "$install_folder" && git clone https://github.com/kpu/kenlm.git && cd kenlm

# KenLM dependencies from official website
sudo apt-get install -y build-essential libboost-all-dev cmake zlib1g-dev libbz2-dev liblzma-dev

mkdir -p build
cd build
cmake ..
make -j 4

cd bin
echo "export KENLM_ROOT=$PWD" >> ~/.bashrc

# we're in kenlm/build/bin, need to go up thrice to get back to the base folder.
cd ../../..

# Kaldi
sudo apt install -y sox subversion intel-mkl-full make zlib1g-dev patch
cd "$install_folder"
git clone https://github.com/kaldi-asr/kaldi.git
cd kaldi
echo "export KALDI_ROOT=$PWD" >> ~/.bashrc
cd tools
make -j 32
cd ../src
./configure --shared
make depend -j 32
make -j 32

pip install soundfile scipy tqdm torchaudio fasttext phonemizer

Currently stuck on this error:
ERROR (arpa2fst[5.5.951~1-579c9]:Read():arpa-file-parser.cc:127) line 1 []: \data\ section missing or empty.

which I presume comes from the fact that kenlm.wrd.o40003.arpa is... empty. Hmmm.

UPDATE: Voidful created a minimal working example (from https://huggingface.slack.com/archives/C01QZ90Q83Z/p1626102538142300)!

if anyone trying to run wav2vec Unsupervised (wav2vec-U) but you stuck on building its environment
I just made a docker file and an example for that. :partyparrot:
https://github.com/voidful/wav2vec-u-exp

Currently stuck on this error:
ERROR (arpa2fst[5.5.951~1-579c9]:Read():arpa-file-parser.cc:127) line 1 []: \data\ section missing or empty.

which I presume comes from the fact that kenlm.wrd.o40003.arpa is... empty. Hmmm.
TIL learned that KenLM's lmplz takes a -S flag to specify how much memory to use. If you don't specify it, and you're running on a VM whose memory is limited, the process might be killed! And then your output file will be blank.

Also, because of the >! redirect, I somehow didn't see "Killed" until I ran that line outside of prepare_text.sh manually.
Not sure what the solution within prepare_text.sh ought to be, perhaps specify it at some relatively low number of G, and add a comment?
https://kheafield.com/code/kenlm/estimation/

sfdsfvdfdb


Fix ruleset for mdrpi:PublicationInfo outside extensions element

Testing this is noisy, because the rest of the ruleset fails on numerous mdrpi element rules.

I suggested doing this then the larger work associated with getting this ruleset working across the environments. I suppose the other option would be to do that latter work first instead, just omitting the unit tests around this bug, then adding this on to the end.
Your choice. Let me know when you have something you want me to review.

OK. Let me reverse this by adding in the overall fix first.

OK. I've re-ordered. You should be able to go through each commit to see different things failing and then being fixed. Of note, as soon as that ruleset is added (the first commit) the non-xalan validators will fail on elements in the other tests. So only on commit 77c8514ba480e4126595ab9dc77453db54080019 will things look clean (before I add the test that highlights the namespace prefix bug).

Modulo the questions we discussed about cross-ruleset validity and schema validity in particular, I'm happy with this and will merge it.


Error when load model

When I run this command:
splitter = NNSplit.load('en')
This error occured:
thread '<unnamed>' panicked at 'Once instance has previously been poisoned', library/std/src/sync/once.rs:394:21
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
pyo3_runtime.PanicException: Once instance has previously been poisoned

Hi, thanks for reporting this.
Which OS, Python version and nnsplit version do you use?

Closing this as stale.


FW-219 Bug was in shouldComponentUpdate check

https://firstvoices.atlassian.net/browse/FW-219
A shouldComponentUpdate check was inspecting state to determine if a rerender should proceed (believe this was added to suppress a bug where modals weren't being opened on click).
The value prop is referenced to determine if the modal should load an edit screen. The previous version of shouldComponentUpdate wasn't looking at props and so it wouldn't rerender after an item was selected which also meant the button text wasn't updated.

Can one of the admins verify this patch?

Going to merge this PR.
The most recent changes fixed the Cypress tests (which were failing due to changes in other branches/merges).
There are still 3 tests that fail:
DialectFilterList-Words.js > DialectFilterList
TestLanguageSix has no categories
SearchDialect-Words-Public.js > FW-936
TestLanguageTwo has no parts of speech (other than 'Any')
SearchStrategy.js > Approximate search
Fails due to server/endpoint error


New HttpSubprovider

Simple subprovider to wrap Web3.privders.HttpProvider objects. Makes inserting an existing web3 object (from Mist or Metamask, for instance) into the provider chain.
Here's how my app uses it:
https://gist.github.com/tcoulter/7fc48cf4a8107281ad5e

hmm im confused i thought be8a874 was merged. i must have messed that up.

Ya, it's weird. I'm not sure if you did or I did. I think I just removed a comma here in this PR.

This is redundant with our rpc subprovider. Ours is better because it doesnt trigger CORS pre-flight headers  on every request. Ours should be renamed http tho.

It's not redundant in that:

HttpProvider takes in a web3 provider as its input, which makes it easy to use and compatible with any web3 provider that uses that interface;
It can interface with previous providers built for that interface (i.e., no rewriting of the hooked web3 provider or the blockapps web3 provider)
Doesn't introduce another dependency (xhr) that must be bundled when using the provider engine in the browser.

If not this, allow me to edit the rpc subprovider to take in a web3 provider as input and call out to that instead.

Ours is better because it doesnt trigger CORS pre-flight headers on every request
Because you're using the request module. This benefit wouldn't change in the browser.

Reopening to force you to take a look at this again. :)

ok just rename it to generically wrap web3 providers, it doesnt really have anything to do with http

Renamed. Should be ready for inclusion.

:+1:


Logging toggle

Hello!
We're using this gem to track down slow and unnecessary callbacks in a large project. Unfortunately, most team members do not need to see the debug messages.
To avoid having to maintain separate branches with the gem enabled and disabled, I added a simple config layer that allows toggling the gem on and off with an environmental variable.
Hope you accept this PR.

To avoid having to maintain separate branches with the gem enabled and disabled, I added a simple config layer that allows toggling the gem on and off with an environmental variable.

Hi Augusts, sounds like a good idea.
We will have two configuration options now, so I'd like them to work the same way. For example:
   FILTER = ENV["RAILS_CALLBACK_LOG_FILTER"].present?.freeze
+  ENABLED = (ENV.fetch("RAILS_CALLBACK_LOG_ENABLED", "true") == "true").freeze

This would be simple.
But, a Configuration class is OK too, as long as it handles both options.
What do you think?

I think we can bring in the existing option into the Config class no problem.
I'll break backwards compatibility by moving to config reading, rather than the env variable reading in gem, OK?

I've mede the changes, but I have to say the "make it so" example for filtering does not cut it, nor are there any specs for it. What is the filter string supposed to be? Some pathname, a portion or the callback message?

I'll break backwards compatibility by moving to config reading, rather than the env variable reading in gem, OK?

That's fine. gem is still pre-1.0.

What is the filter string supposed to be? Some pathname, a portion of the callback message or something else?

Doesn't matter. FILTER is boolean.
Let me know when you get the tests passing and I'll review, thanks.

You can do this without changing the gem by doing
Gemfile
gem 'rails-callback_log', require: false

config/initializers/callback_log.rb
require "rails-callback_log" if ENV["LOG_RAILS_CALLBACKS"] == "true"

You can do this without changing the gem ..

Oh! That's a good point, Kevin. I'd be happy to keep it simple like that, as long as we add that to the docs. Does that work for you, Augusts?

Works like a charm! I'll change the PR to a simple update to the docs.

That's that.

Looks like the tests are failing due to bundler 2.0 being installed by default but the tests expecting a 1.x version. I'll try to take a look at that later today.


Add MakeChannelArgumentOption() for void*

It looks the sanity check failure is from src/core/lib/event_engine/cf_engine/cf_engine.cc and is unrelated.


TestServer sends remaining content in pipe writer on complete

Change TestServer to match Kestrel's behavior by sending unflushed content in pipe writer when completing request.

Recommend closing this PR. We realized this is an issue with the PipeWriter polyfill, not with the server. I'll file an issue.

Fix is either https://github.com/aspnet/AspNetCore/issues/11305 or changing TestServer to use pipes instead of Streams.


commit on link

User Snap #244 #245  edit and update.

Tugboat has finished building the preview for this pull request!
Link:

https://pr164-75wykzc8hlga8al83ks3rpco6kgto9fu.tugboat.qa

Dashboard:

https://dashboard.tugboat.qa/60e5bc2857072dc73aedaf5d

Tugboat has finished building the preview for this pull request!
Link:

https://pr164-75wykzc8hlga8al83ks3rpco6kgto9fu.tugboat.qa

Dashboard:

https://dashboard.tugboat.qa/60e5bc2857072dc73aedaf5d

Tugboat has finished building the preview for this pull request!
Link:

https://pr164-75wykzc8hlga8al83ks3rpco6kgto9fu.tugboat.qa

Dashboard:

https://dashboard.tugboat.qa/60e5bc2857072dc73aedaf5d

Tugboat has finished building the preview for this pull request!
Link:

https://pr164-75wykzc8hlga8al83ks3rpco6kgto9fu.tugboat.qa

Dashboard:

https://dashboard.tugboat.qa/60e5bc2857072dc73aedaf5d

Tugboat has finished building the preview for this pull request!
Link:

https://pr164-75wykzc8hlga8al83ks3rpco6kgto9fu.tugboat.qa

Dashboard:

https://dashboard.tugboat.qa/60e5bc2857072dc73aedaf5d

Tugboat has finished building the preview for this pull request!
Link:

https://pr164-75wykzc8hlga8al83ks3rpco6kgto9fu.tugboat.qa

Dashboard:

https://dashboard.tugboat.qa/60e5bc2857072dc73aedaf5d

Hey, noticing that this has been here for a few days. @ronniemcbride did you need @coreyvickery to review as well?

Hey, noticing that this has been here for a few days. @ronniemcbride did you need @coreyvickery to review as well?

The update that was made @coreyvickery  had assigned to himself prior to his time off, but I went ahead and took care of them. I think Corey should have a chance to review them before I publish this page.

Tugboat has finished building the preview for this pull request!
Link:

https://pr164-75wykzc8hlga8al83ks3rpco6kgto9fu.tugboat.qa

Dashboard:

https://dashboard.tugboat.qa/60e5bc2857072dc73aedaf5d


Doc enhancements for on-prem deployments

Some additional enhancements which might have helped me to get started.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Thanks @LarsAC. We'll review this asap.


4K full screen toggle

Hello,
I'm not sure if this as issue or just me..
When using 'SadConsole.Settings.ToggleFullScreen();' with a desktop set as  3840x2160 the screen goes blank for a few seconds and then the window will be minimized onto the taskbar.
After that there is nothing you can do to get the screen back up.
If I alter the desktop resolution to a smaller resolution (1920x1200) full screen toggle works fine.
Using the window maximize button works fine also at the high resolution.
Kind regards,
Twoshrubs

@Twoshrubs Hi! Thank you for filing this issue. I'm not sure why or how that would happen. I haven't hard coded any sort of resolution limits or anything. SadConsole just gets the resize information from MonoGame when the resize event happens. I don't have a 4k monitor to try and test and debug this, so I would need you or someone else to do that.
This is the code that goes to full screen. As you can see I'm querying MonoGame for how big the screen can be and then setting that size.
After the screen is toggled, the MonoGame game event code path will trigger Global.ResetRendering().
Perhaps you can try the free version of https://marketplace.visualstudio.com/items?itemName=RevDeBug.RecordReplayforNET-RevDeBug or something? Or we can arrange for us to use the Visual Studio Live Share feature. This way I can easily and securely connect to your machine and watch and control the debugging.

Hello :)
Yes, I will give the recorder a try this weekend, I'm a contract automation engineer.. so I'm away all week commissioning. If we get no joy with the recorder, I have no issues with doing a shared screen session one weekend.

@SadConsole Hello, I just remembered my lad had simular issues with Spiral Knights at the weekend, and could only get it working in Window mode. I had a google and it seems it uses XNA/Monogame.
Then came across this.. http://community.monogame.net/t/fullscreen-game-not-rendering-to-proper-size-for-resolutions-besides-1920-x-1080/8255 It looks like a Monogame issue?


HBASE-26942 cache region locations when getAllRegionLocations

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
3m 42s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s
No case conflicting files found.

+1 :green_heart:
hbaseanti
0m  0s
Patch does not have any anti-patterns.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

_ branch-2 Compile Tests _

+0 :ok:
mvndep
0m 14s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
3m 20s
branch-2 passed

+1 :green_heart:
compile
2m 54s
branch-2 passed

+1 :green_heart:
checkstyle
0m 52s
branch-2 passed

+1 :green_heart:
spotbugs
1m 54s
branch-2 passed

_ Patch Compile Tests _

+0 :ok:
mvndep
0m  9s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
2m  0s
the patch passed

+1 :green_heart:
compile
2m 54s
the patch passed

+1 :green_heart:
javac
2m 54s
the patch passed

+1 :green_heart:
checkstyle
0m 52s
the patch passed

+1 :green_heart:
whitespace
0m  0s
The patch has no whitespace issues.

+1 :green_heart:
hadoopcheck
7m 51s
Patch does not cause any errors with Hadoop 3.1.2 3.2.1.

+1 :green_heart:
spotbugs
2m  1s
the patch passed

_ Other Tests _

+1 :green_heart:
asflicense
0m 16s
The patch does not generate ASF License warnings.

34m  4s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4364/1/artifact/yetus-general-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/4364

Optional Tests
dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile

uname
Linux 37bc381ec20c 5.4.0-1071-aws #76~18.04.1-Ubuntu SMP Mon Mar 28 17:49:57 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
branch-2 /<PHONE_NUMBER>

Default Java
AdoptOpenJDK-1.8.0_282-b08

Max. process+thread count
64 (vs. ulimit of 12500)

modules
C: hbase-client hbase-server U: .

Console output
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4364/1/console

versions
git=2.17.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 45s
Docker mode activated.

-0 :warning:
yetus
0m  6s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ branch-2 Compile Tests _

+0 :ok:
mvndep
0m 14s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
3m 25s
branch-2 passed

+1 :green_heart:
compile
0m 53s
branch-2 passed

+1 :green_heart:
shadedjars
3m 52s
branch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
0m 39s
branch-2 passed

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 10s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
2m  4s
the patch passed

+1 :green_heart:
compile
0m 56s
the patch passed

+1 :green_heart:
javac
0m 56s
the patch passed

+1 :green_heart:
shadedjars
4m 23s
patch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
0m 56s
the patch passed

_ Other Tests _

+1 :green_heart:
unit
2m 58s
hbase-client in the patch passed.

+1 :green_heart:
unit
183m 43s
hbase-server in the patch passed.

206m 17s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4364/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/4364

Optional Tests
javac javadoc unit shadedjars compile

uname
Linux 4849f712e37a 5.4.0-1071-aws #76~18.04.1-Ubuntu SMP Mon Mar 28 17:49:57 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
branch-2 /<PHONE_NUMBER>

Default Java
AdoptOpenJDK-1.8.0_282-b08

Test Results
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4364/1/testReport/

Max. process+thread count
2557 (vs. ulimit of 12500)

modules
C: hbase-client hbase-server U: .

Console output
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4364/1/console

versions
git=2.17.1 maven=3.6.3

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
1m 13s
Docker mode activated.

-0 :warning:
yetus
0m  7s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ branch-2 Compile Tests _

+0 :ok:
mvndep
0m 12s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
2m 44s
branch-2 passed

+1 :green_heart:
compile
1m  7s
branch-2 passed

+1 :green_heart:
shadedjars
3m 49s
branch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
0m 47s
branch-2 passed

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 18s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
2m 31s
the patch passed

+1 :green_heart:
compile
1m 10s
the patch passed

+1 :green_heart:
javac
1m 10s
the patch passed

+1 :green_heart:
shadedjars
3m 47s
patch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
0m 45s
the patch passed

_ Other Tests _

+1 :green_heart:
unit
2m 52s
hbase-client in the patch passed.

+1 :green_heart:
unit
187m 58s
hbase-server in the patch passed.

211m 16s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4364/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/4364

Optional Tests
javac javadoc unit shadedjars compile

uname
Linux 24b402a8360a 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
branch-2 /<PHONE_NUMBER>

Default Java
AdoptOpenJDK-11.0.10+9

Test Results
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4364/1/testReport/

Max. process+thread count
2516 (vs. ulimit of 12500)

modules
C: hbase-client hbase-server U: .

Console output
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4364/1/console

versions
git=2.17.1 maven=3.6.3

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.


element.style.height returns no value

I am trying to dynamically adjust the size of component by using the height some other component. It looks like the scalatags element.style.height property does not return any value (the element is rendered and added to the dom)?! When I use jQuery $(element).height I do get the correct value.

This isn't a Scalatags problem; this is just how javascript works


styles: header not centered

.... or content not centered:

closed by https://github.com/inveniosoftware/invenio-theme/pull/198 and https://github.com/inveniosoftware/invenio-theme/pull/199


package/search with a version 1 response should not return objects that don't have marathon

One solution is to create a v2 search request that takes a optional parameter. The optional parameter should indicate which packages type to return in the search. We need to investigate how to best express this filtering condition.

Closing this issue after an offline discussion. Adding a filtering option to search requests was deemed too much of an ad-hoc solution without more information from Cosmos clients/users. The decision hinged on the following user outcomes:

If this issue is implemented: users of DC/OS 1.8 that request v1 search responses will have CLI-only packages silently dropped from search results; the incomplete output would be inaccurate and could cause confusion, with nothing to inform the user that they should ask for v2 responses instead.
If this issue is not implemented: users of DC/OS 1.8 will always see CLI-only packages; if they try to describe or install them when requesting v1 responses, an explicit, specific error response will be sent. Users can then search for or ask for help on that error, which would hopefully result in them switching to v2 responses.

In both cases, the scenario requires the use of an older CLI or web UI that does not ask for v2 responses on a 1.8 DC/OS cluster. Since that is expected to be rare, it would be better to take option (2) above and see if customers actually run into trouble, before deciding to write code.
Please comment if I've forgotten to include or account for some information!