[tf_clean] swb1/v1-tf WFST decoding - checking on assumptions

Decided a new thread would be good for this issue.
Right now the SWB tf code as checked in seems to have a discrepancy, and I'm writing down some of my assumptions as I work through cleaning up the WFST decode.
It looks like to me that run_ctc_phn.sh creates a set of training/cv labels that ignores noises (and overwrites units.txt, removing spn and npn).  However, utils/ctc_compile_dict_token.sh assumes that units.txt and lexicon.txt are synchronized, resulting in the lovely error:
FATAL: FstCompiler: Symbol "spn" is not mapped to any integer arc ilabel, symbol table = data/lang_phn/tokens.txt, source = standard input, line = 1171

The fix is pretty simple (synchronizing the lexicon) but I'm trying to figure out how much to modify the utils/ctc_compile_dict_token.sh script vs. correcting the prep script to do the appropriate correction.  I'm thinking that I'll correct the prep script, but if anyone has any thoughts on that let me know.

great yes thanks @efosler we really couldn't get a full recipe for tf + wfst. I have some spare scripts but nothing clean and oficial...
This is correct yes:

It looks like to me that run_ctc_phn.sh creates a set of training/cv labels that ignores noises (and overwrites units.txt, removing spn and npn). However, utils/ctc_compile_dict_token.sh assumes that units.txt and lexicon.txt are synchronized, resulting in the lovely error:

I was thinking if is there any improved (or more simple way) to do the data preparation for characters and phonemes separately. Do you have any thought? I can try to help. Otherwise we can reuse the preparation of the master branch.
Also, another random thing that I observed with swbd: I tried to prepare the char set up substituting numbers by written words and removing noises but at the end it did not work out...
I am working in integrating CharRNN decoding recipe that we have (it doesn't perfrom better than wfst but we allow open vocabulary) https://arxiv.org/abs/1708.04469.
Please let me know if I can help you somehow I am will be very happy to!
Thanks again!

Let me think about it as I play with the scripts.  I just created a _tf version of swbd1_decode_graph.sh which gets rid of the -unk option, but that feels like there could be a better factorization.

So, an update: the good news is that I was able to get a decode to run all the way through.  There does seem to be a bit of underperformance w.r.t. Yajie's runs on the non-tf version.  Currently, I'm seeing 24.7% WER on eval2000 (vs. 21.0) using SWB + Fisher LM.  I think there are a few differences:

4 layer BiLSTM vs 5 layer BiLSTM
I'm not sure that the default tf recipe currently checked in has speaker adaptation

I'm sure that there is some other possible set of differences in parameters as well.
Just to check: what I did was just work with the output of ./steps/decode_ctc_am_tf.sh and feed the logprobs through latgen-faster.  NB this just runs test.py in ctc-am rather than nnet.py or anything else (not sure if this is the right thing to do, but it's what's checked in).
Any thoughts on diffs between the tf and old versions that might be causing the discrepancy?

Hi Eric,
Thank you very much for that. Can you let me know which token error rate
were you getting in the acoustic model? I have some experiments that
achieved way below this WER. Did you use the prior probabilities of each
character during wfst decoding? Can you share the complete log of the
acoustic model training?
Thanks!
Best,
Ramon
2018-06-26 12:24 GMT-04:00 Eric Fosler-Lussier<EMAIL_ADDRESS>
So, an update: the good news is that I was able to get a decode to run all
the way through. There does seem to be a bit of underperformance w.r.t.
Yajie's runs on the non-tf version. Currently, I'm seeing 24.7% WER on
eval2000 (vs. 21.0) using SWB + Fisher LM. I think there are a few
differences:

4 layer BiLSTM vs 5 layer BiLSTM
I'm not sure that the default tf recipe currently checked in has
speaker adaptation

I'm sure that there is some other possible set of differences in
parameters as well.
Just to check: what I did was just work with the output of
./steps/decode_ctc_am_tf.sh and feed the logprobs through latgen-faster. NB
this just runs test.py in ctc-am rather than nnet.py or anything else (not
sure if this is the right thing to do, but it's what's checked in).
Any thoughts on diffs between the tf and old versions that might be
causing the discrepancy?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/srvk/eesen/issues/193#issuecomment-400375570, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AMlwPRMl704W1O4LrlphOdE-Xoarzhciks5uAmA4gaJpZM4U2hdg
.

Let me re-run it - I just started up the non-tf version and I think it blew away the tf version (hmpf).  I am pretty sure that it isn't using the prior probabilities of each phone (not character) but I'm not sure.  (I don't see where that would have been injected into the system).

@efosler I would also like to integrate the tf acoustic model into a WFST for decoding. As I understand this thread you have managed to do that. Is any of your code in the repo ?
I pulled tf_clean and asr_egs/swbd/v1-tf/run_ctc_phn.sh only does acoustic decoding.
Would be great if I could avoid starting from scratch :)

Sorry for the delay - I had a few other things pop up.  The non-tf run didn't finish before we had a massive server shutdown because of a planned power outage (sigh).  So @ericbolo let me try to run the v1-tf branch again and I can check in against my copy of the repository.  I think that @ramonsanabria has had a better outcome than I have.
Basically, the things I had to do were slight modifications to the building the TLG graph followed by calling latgen-faster and score_sclite.sh.  I'm sure that the decoding parameters aren't right and I have to investigate whether I do have the priors involved or not before decoding.

@efosler, thank you !
On 29 June 2018 at 18:28, Eric Fosler-Lussier<EMAIL_ADDRESS>wrote:

Sorry for the delay - I had a few other things pop up. The non-tf run
didn't finish before we had a massive server shutdown because of a planned
power outage (sigh). So @ericbolo https://github.com/ericbolo let me
try to run the v1-tf branch again and I can check in against my copy of the
repository. I think that @ramonsanabria https://github.com/ramonsanabria
has had a better outcome than I have.
Basically, the things I had to do were slight modifications to the
building the TLG graph followed by calling latgen-faster and
score_sclite.sh. I'm sure that the decoding parameters aren't right and I
have to investigate whether I do have the priors involved or not before
decoding.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/srvk/eesen/issues/193#issuecomment-401406139, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AEXQ_KvFiDQlFEOibqPA0fxwQrwXJqzqks5uBlXJgaJpZM4U2hdg
.

--
Eric Bolo
CTO
tel: 06 29 72 19 80 http://06 58 79 25 37/

Batvoice Technologies
10, rue Coquillière - 75001 Paris
www.batvoice.com

Hi all,
Yesterday I was able to re run one of our complete pipeline of EESEN + WFST
using BPE units (https://arxiv.org/abs/1708.04469) in SWBD. I hit the 16.5
without fine tunning. I feel I can maybe get some extra points by just
playing a bit with WFST parameters.
The BPE pipeline made me think that in the case you need a little bit more
speed during decoding you can also play use BPE (bigger units means less
steps in the decoding process) and the accuracy is not that bad.
PS: Also we have a recipe to train the acoustic model with the whole fisher
swbd corpus set in case you needed for the your real time implementation
that you have in mind.
Thanks!
Best,
Ramon
2018-07-02 4:52 GMT-04:00 ericbolo<EMAIL_ADDRESS>
@efosler, thank you !
On 29 June 2018 at 18:28, Eric Fosler-Lussier<EMAIL_ADDRESS>wrote:

Sorry for the delay - I had a few other things pop up. The non-tf run
didn't finish before we had a massive server shutdown because of a
planned
power outage (sigh). So @ericbolo https://github.com/ericbolo let me
try to run the v1-tf branch again and I can check in against my copy of
the
repository. I think that @ramonsanabria <https://github.com/
ramonsanabria>
has had a better outcome than I have.
Basically, the things I had to do were slight modifications to the
building the TLG graph followed by calling latgen-faster and
score_sclite.sh. I'm sure that the decoding parameters aren't right and I
have to investigate whether I do have the priors involved or not before
decoding.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/srvk/eesen/issues/193#issuecomment-401406139, or
mute
the thread
<https://github.com/notifications/unsubscribe-auth/AEXQ_
KvFiDQlFEOibqPA0fxwQrwXJqzqks5uBlXJgaJpZM4U2hdg>
.

--
Eric Bolo
CTO
tel: 06 29 72 19 80 http://06 58 79 25 37/

Batvoice Technologies
10, rue Coquillière - 75001 Paris
https://maps.google.com/?q=10,+rue+Coquillière+-+75001+Paris&entry=gmail&source=g
www.batvoice.com
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/srvk/eesen/issues/193#issuecomment-401719312, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AMlwPavmS82ay9tCjDizst6ZUnw4DkXqks5uCd9YgaJpZM4U2hdg
.

sorry wrong paper. Correction, the BPE paper is this one:
https://arxiv.org/pdf/1712.06855.pdf
2018-07-02 9:07 GMT-04:00 Ramon Sanabria<EMAIL_ADDRESS>:

Hi all,
Yesterday I was able to re run one of our complete pipeline of EESEN +
WFST using BPE units (https://arxiv.org/abs/1708.04469) in SWBD. I hit
the 16.5 without fine tunning. I feel I can maybe get some extra points by
just playing a bit with WFST parameters.
The BPE pipeline made me think that in the case you need a little bit more
speed during decoding you can also play use BPE (bigger units means less
steps in the decoding process) and the accuracy is not that bad.
PS: Also we have a recipe to train the acoustic model with the whole
fisher swbd corpus set in case you needed for the your real time
implementation that you have in mind.
Thanks!
Best,
Ramon
2018-07-02 4:52 GMT-04:00 ericbolo<EMAIL_ADDRESS>
@efosler, thank you !
On 29 June 2018 at 18:28, Eric Fosler-Lussier<EMAIL_ADDRESS>wrote:

Sorry for the delay - I had a few other things pop up. The non-tf run
didn't finish before we had a massive server shutdown because of a
planned
power outage (sigh). So @ericbolo https://github.com/ericbolo let me
try to run the v1-tf branch again and I can check in against my copy of
the
repository. I think that @ramonsanabria <https://github.com/ramonsanab
ria>
has had a better outcome than I have.
Basically, the things I had to do were slight modifications to the
building the TLG graph followed by calling latgen-faster and
score_sclite.sh. I'm sure that the decoding parameters aren't right and
I
have to investigate whether I do have the priors involved or not before
decoding.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/srvk/eesen/issues/193#issuecomment-401406139, or
mute
the thread
<https://github.com/notifications/unsubscribe-auth/AEXQ_KvFi
DQlFEOibqPA0fxwQrwXJqzqks5uBlXJgaJpZM4U2hdg>
.

--
Eric Bolo
CTO
tel: 06 29 72 19 80 http://06 58 79 25 37/

Batvoice Technologies
10, rue Coquillière - 75001 Paris
https://maps.google.com/?q=10,+rue+Coquillière+-+75001+Paris&entry=gmail&source=g
www.batvoice.com
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/srvk/eesen/issues/193#issuecomment-401719312, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AMlwPavmS82ay9tCjDizst6ZUnw4DkXqks5uCd9YgaJpZM4U2hdg
.

@ramonsanabria thanks!  We've been having some NFS issues here so I haven't gotten my full pass to run.  It would be great to have this recipe in the mix.  Does this want to go into v1-tf or should there be a v2-tf?

We finally got the NFS issues resolved so I should have the training done by tomorrowish.  @ramonsanabria, two questions -

I noticed that the default forward pass used epoch 14 (rather than the full model) - was there a reason for that or is that something that I should clean up.  (This would be part of the reason for substandard results, but more likely #2 below...)
I do not believe that the decoding is using priors.  I see a start on that for ctc-am/tf/tf_test.py but it doesn't seem to do anything with the priors, nor is there a model.priors file built during training (as far as I can tell).  Am I missing something?

Hi Eric, Sorry for not responding the last message. We were also a little bit busy in JSALT.
Regarding  tf-v2. Yes we can do that, cool idea. @fmetze what do you think? I am still fine tunning everything (WFST and AM) and I should include some code (mostly for the BPE generation) but it would be a good idea. We are preparing a second publication for SLT, after the acceptance we can actually release the whole recipie.
Ok let me send all the parameters the I am using. Can you share your TER results with your configuration? You might find some parameters that are currently not implemented in the master branch (dropout etc.). But with the intersected parameters you should be fine. With this configuration on swbd I remember that @fmetze achieved something close to 11% TER.
target_scheme {'no_name_language': {'no_target_name': 47}} drop_out 0.0 sat_conf {'num_sat_layers': 2, 'continue_ckpt_sat': False, 'sat_stage': 'fine_tune', 'sat _type': 'non_adapted'} init_nproj 80 clip 0.1 nlayer 5 nhidden 320 data_dir /tmp/tmp.EKt4xyU6eX min_lr_rate 0.0005 half_rate 0.5 do_shuf True nepoch 30 grad_opt grad random_seed 15213 model_dir exp/fmetze_test_43j26e/model input_feats_dim 129 batch_size 16 kl_weight 0.0 lstm_type cudnn lr_rate 0.05 model deepbilstm nproj 340 final_nproj 0 half_after 8 train_dir exp/fmetze_test_43j26e/ online_augment_conf {'window': 3, 'subsampling': 3, 'roll': True} clip_norm False l2 0.001 store_model True debug False continue_ckpt half_period 4 force_lr_epoch False batch_norm True
Let me take a look to the bullet point 2 later in the day. This you should use https://github.com/srvk/eesen/blob/tf_clean/tf/ctc-am/test.py in order to perform testing. It will generate varios versions of the forward pass (log_probs, logits, probs, etc.) with blank in the 0 position. I will need to clean this up so that the script only outputs what is really needed.
After having the log_probs then is when you can just apply normal eesen c++ recipie (i.e. apply wfst to log_probs). I am not sure why my character based WFST is not working. I could make it work with bpe300 and other units but not in characters. I will try to get back to you later on this.
Thanks!

Just for the record, here are diffs on config:
nlayer: 4 (vs 5)
input_feats_dim: 120 (vs 129)
batch_size: 32 (vs 16)
lr_rate: 0.005 (vs 0.05)
nproj: 60 (vs 340)
online_augment_conf.roll = False (vs True)
l2: 0.0 (vs 0.001)
batch_norm: False (vs True)
So it's pretty clear that there are some significant differences, and I'd believe the sum total of them could result in a 4% difference in TER (particularly layers, l2, batch norm, lr_rate, and maybe nproj).  The really interesting question is what the extra 9 features are - it looks like one additional base feature which has deltas/double-deltas and windowing applied.
{'continue_ckpt': '', 'diff_num_target_ckpt': False, 'force_lr_epoch': False, 'random_seed': 15213, 'debug': False, 'store_model': True, 'data_dir': '/scratch/tmp/fosler/tmp.xJUz4scH4T', 'train_dir': 'exp/train_phn_l4_c320_mdeepbilstm_w3_ntrue_p60_ip80_fp80', 'batch_size': 32, 'do_shuf': True, 'nepoch': 30, 'lr_rate': 0.005, 'min_lr_rate': 0.0005, 'half_period': 4, 'half_rate': 0.5, 'half_after': 8, 'drop_out': 0.0, 'clip_norm': False, 'kl_weight': 0.0, 'model': 'deepbilstm', 'lstm_type': 'cudnn', 'nproj': 60, 'final_nproj': 100, 'init_nproj': 80, 'l2': 0.0, 'nlayer': 4, 'nhidden': 320, 'clip': 0.1, 'batch_norm': False, 'grad_opt': 'grad', 'sat_conf': {'sat_type': 'non_adapted', 'sat_stage': 'fine_tune', 'num_sat_layers': 2, 'continue_ckpt_sat': False}, 'online_augment_conf': {'window': 3, 'subsampling': 3, 'roll': False}, 'input_feats_dim': 120, 'target_scheme': {'no_name_language': {'no_target_name': 43}}, 'model_dir': 'exp/train_phn_l4_c320_mdeepbilstm_w3_ntrue_p60_ip80_fp80/model'}

Perfect yes, so with this parameters you should see an improvement
according to my experience. For input_feats_dim: 120 (vs 129) just
extract fbank_pitch features (we should also change this in the recipie of
v1-tf). I assume that 'subsampling': 3 is correct right? (this is also
important).
Also those ones that I think are implemented: 'nproj': 60, 'final_nproj': 100, 'init_nproj': 80. Otherwise I will push code to have them.
Thanks!
2018-07-03 17:08 GMT-04:00 Eric Fosler-Lussier<EMAIL_ADDRESS>
Just for the record, here are diffs on config:
nlayer: 4 (vs 5)
input_feats_dim: 120 (vs 129)
batch_size: 32 (vs 16)
lr_rate: 0.005 (vs 0.05)
nproj: 60 (vs 340)
online_augment_conf.roll = False (vs True)
l2: 0.0 (vs 0.001)
batch_norm: False (vs True)
So it's pretty clear that there are some significant differences, and I'd
believe the sum total of them could result in a 4% difference in TER
(particularly layers, l2, batch norm, lr_rate, and maybe nproj). The really
interesting question is what the extra 9 features are - it looks like one
additional base feature which has deltas/double-deltas and windowing
applied.
{'continue_ckpt': '', 'diff_num_target_ckpt': False, 'force_lr_epoch':
False, 'random_seed': 15213, 'debug': False, 'store_model': True,
'data_dir': '/scratch/tmp/fosler/tmp.xJUz4scH4T', 'train_dir':
'exp/train_phn_l4_c320_mdeepbilstm_w3_ntrue_p60_ip80_fp80', 'batch_size':
32, 'do_shuf': True, 'nepoch': 30, 'lr_rate': 0.005, 'min_lr_rate': 0.0005,
'half_period': 4, 'half_rate': 0.5, 'half_after': 8, 'drop_out': 0.0,
'clip_norm': False, 'kl_weight': 0.0, 'model': 'deepbilstm', 'lstm_type':
'cudnn', 'nproj': 60, 'final_nproj': 100, 'init_nproj': 80, 'l2': 0.0,
'nlayer': 4, 'nhidden': 320, 'clip': 0.1, 'batch_norm': False, 'grad_opt':
'grad', 'sat_conf': {'sat_type': 'non_adapted', 'sat_stage': 'fine_tune',
'num_sat_layers': 2, 'continue_ckpt_sat': False}, 'online_augment_conf':
{'window': 3, 'subsampling': 3, 'roll': False}, 'input_feats_dim': 120,
'target_scheme': {'no_name_language': {'no_target_name': 43}}, 'model_dir':
'exp/train_phn_l4_c320_mdeepbilstm_w3_ntrue_p60_ip80_fp80/model'}
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/srvk/eesen/issues/193#issuecomment-402292233, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AMlwPbFOxV0nyRRyx7rdsXBvhPvB1-xYks5uC90xgaJpZM4U2hdg
.

Hello all,
@efosler: yes, all I need is a running pipeline, not the best performing
one, so that I can look at all the pieces of an online decoding system with
tensorflow + wfst.
On 4 July 2018 at 00:22, ramonsanabria<EMAIL_ADDRESS>wrote:

Perfect yes, so with this parameters you should see an improvement
according to my experience. For input_feats_dim: 120 (vs 129) just
extract fbank_pitch features (we should also change this in the recipie of
v1-tf). I assume that 'subsampling': 3 is correct right? (this is also
important).
Also those ones that I think are implemented: 'nproj': 60, 'final_nproj': 100, 'init_nproj': 80. Otherwise I will push code to have them.
Thanks!
2018-07-03 17:08 GMT-04:00 Eric Fosler-Lussier<EMAIL_ADDRESS>
Just for the record, here are diffs on config:
nlayer: 4 (vs 5)
input_feats_dim: 120 (vs 129)
batch_size: 32 (vs 16)
lr_rate: 0.005 (vs 0.05)
nproj: 60 (vs 340)
online_augment_conf.roll = False (vs True)
l2: 0.0 (vs 0.001)
batch_norm: False (vs True)
So it's pretty clear that there are some significant differences, and I'd
believe the sum total of them could result in a 4% difference in TER
(particularly layers, l2, batch norm, lr_rate, and maybe nproj). The
really
interesting question is what the extra 9 features are - it looks like one
additional base feature which has deltas/double-deltas and windowing
applied.
{'continue_ckpt': '', 'diff_num_target_ckpt': False, 'force_lr_epoch':
False, 'random_seed': 15213, 'debug': False, 'store_model': True,
'data_dir': '/scratch/tmp/fosler/tmp.xJUz4scH4T', 'train_dir':
'exp/train_phn_l4_c320_mdeepbilstm_w3_ntrue_p60_ip80_fp80',
'batch_size':
32, 'do_shuf': True, 'nepoch': 30, 'lr_rate': 0.005, 'min_lr_rate':
0.0005,
'half_period': 4, 'half_rate': 0.5, 'half_after': 8, 'drop_out': 0.0,
'clip_norm': False, 'kl_weight': 0.0, 'model': 'deepbilstm', 'lstm_type':
'cudnn', 'nproj': 60, 'final_nproj': 100, 'init_nproj': 80, 'l2': 0.0,
'nlayer': 4, 'nhidden': 320, 'clip': 0.1, 'batch_norm': False,
'grad_opt':
'grad', 'sat_conf': {'sat_type': 'non_adapted', 'sat_stage': 'fine_tune',
'num_sat_layers': 2, 'continue_ckpt_sat': False}, 'online_augment_conf':
{'window': 3, 'subsampling': 3, 'roll': False}, 'input_feats_dim': 120,
'target_scheme': {'no_name_language': {'no_target_name': 43}},
'model_dir':
'exp/train_phn_l4_c320_mdeepbilstm_w3_ntrue_p60_ip80_fp80/model'}
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/srvk/eesen/issues/193#issuecomment-402292233, or
mute
the thread
<https://github.com/notifications/unsubscribe-auth/
AMlwPbFOxV0nyRRyx7rdsXBvhPvB1-xYks5uC90xgaJpZM4U2hdg>
.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/srvk/eesen/issues/193#issuecomment-402308973, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AEXQ_JHFGkUzWdTdWwYVqrrxN6k8GQbAks5uC-6PgaJpZM4U2hdg
.

--
Eric Bolo
CTO
tel: 06 29 72 19 80 http://06 58 79 25 37/

Batvoice Technologies
10, rue Coquillière - 75001 Paris
www.batvoice.com

@ericbolo I've uploaded my changes to efosler/eesen so you can grab the newest copy.  This should work - there are a few diffs with the graph prep scripts.  Here's a list of the files that I changed so that you can just grab them if you want:

asr_egs/swbd/v1-tf/local/swbd1_data_prep.sh
asr_egs/swbd/v1-tf/local/swbd1_decode_graph_tf.sh
asr_egs/swbd/v1-tf/run_ctc_phn.sh
asr_egs/swbd/v1/local/swbd1_data_prep.sh
[cosmetic changes only to these]
asr_egs/wsj/steps/decode_ctc_lat_tf.sh
asr_egs/wsj/steps/train_ctc_tf.sh
[python 3 compatibility]
asr_egs/wsj/utils/ctc_token_fst.py
asr_egs/wsj/utils/model_topo.py

My next step will be to try to rework the recipe so that it matches the parameters sent by @ramonsanabria .  Once I've got that done and confirmed I'll send a pull request.

NB: the decode script is woefully non-parallel (needs to be fixed), but for the online stuff this won't matter.

@efosler: wonderful, thanks !
I don't have the swbd db but I can adapt it for, say, tedlium.
On Fri, Jul 6, 2018, 12:06 AM Eric Fosler-Lussier<EMAIL_ADDRESS>wrote:

@ericbolo https://github.com/ericbolo I've uploaded my changes to
efosler/eesen so you can grab the newest copy. This should work - there
are a few diffs with the graph prep scripts. Here's a list of the files
that I changed so that you can just grab them if you want:

asr_egs/swbd/v1-tf/local/swbd1_data_prep.sh
asr_egs/swbd/v1-tf/local/swbd1_decode_graph_tf.sh
asr_egs/swbd/v1-tf/run_ctc_phn.sh
asr_egs/swbd/v1/local/swbd1_data_prep.sh
[cosmetic changes only to these]
asr_egs/wsj/steps/decode_ctc_lat_tf.sh
asr_egs/wsj/steps/train_ctc_tf.sh
[python 3 compatibility]
asr_egs/wsj/utils/ctc_token_fst.py
asr_egs/wsj/utils/model_topo.py

My next step will be to try to rework the recipe so that it matches the
parameters sent by @ramonsanabria https://github.com/ramonsanabria .
Once I've got that done and confirmed I'll send a pull request.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/srvk/eesen/issues/193#issuecomment-402866670, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AEXQ_EbYf7FANRUU4OBuQuUkC0MBrQRUks5uDo3QgaJpZM4U2hdg
.

Hey @ramonsanabria , quick question:   you said...

Also those ones that I think are implemented: 'nproj': 60, 'final_nproj': 100, 'init_nproj': 80. Otherwise I will push code to have them.

Looking through the code base, it seems like these are passed as parameters - will it not do the right thing if those parameters are set?

About to go offline for a bit, so I won't be able to report on the full run, but training with the parameters above (same as @fmetze 's run but with nproj=60, final_nproj=100, init_nproj=80) does get down to 11.7% TER, so I will make those the default with the script going forward.  Decoding hasn't happened yet.

@efosler, a quick update: I was able to run the full pipeline with
tensorflow + language model decoding on a dummy dataset. Thanks again !
Next steps re:online decoding (#141): implementing a forward-only LSTM, and
the loss function for student-teacher learning.
On 7 July 2018 at 19:49, Eric Fosler-Lussier<EMAIL_ADDRESS>wrote:

About to go offline for a bit, so I won't be able to report on the full
run, but training with the parameters above (same as @fmetze
https://github.com/fmetze 's run but with nproj=60, final_nproj=100,
init_nproj=80) does get down to 11.7% TER, so I will make those the default
with the script going forward. Decoding hasn't happened yet.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/srvk/eesen/issues/193#issuecomment-403232508, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AEXQ_DAAIxwLWmIlWqqyQVnLlV6rkaVKks5uEPTAgaJpZM4U2hdg
.

--
Eric Bolo
CTO
tel: 06 29 72 19 80 http://06 58 79 25 37/

Batvoice Technologies
10, rue Coquillière - 75001 Paris
www.batvoice.com

re: priors. As @efosler noted, it seems the priors are not used in the current decoding code.
in tf_test.py:
if(config[constants.CONFIG_TAGS_TEST.USE_PRIORS]):
            #TODO we need to add priors also:
            #feed_priors={i: y for i, y in zip(model.priors, config["prior"])}
            print(config[constants.CONFIG_TAGS_TEST.PRIORS_SCHEME])

model.priors doesn't seem to be generated anywhere, but we can use label.counts to generate it.
@fmetze , are priors used in the original (c++) implementation ?

Hi all,
Priors are generated by:
`labels=$dir_am/label.counts
gunzip -c $dir_am/labels.tr.gz cat | awk '{line=$0; gsub(" "," 0 ",line);
print line " 0";}' |
/data/ASR5/fmetze/eesen-block-copy/src/decoderbin/analyze-counts
--verbose=1 --binary=false ark:- $labels`
Then you can use nnet.py as:
$decode_cmd JOB=1:$nj $mdl/log/decode.JOB.log \   cat $PWD/$mdl/split$nj/JOB/feats.scp \| sort -k 1 \| python utils/nnet.py --label-counts $labels --temperature $temperature --blank-scale $bkscale \| \   latgen-faster  --max-active=$max_active --max-mem=$max_mem --beam=$beam --lattice-beam=$lattice_beam \   --acoustic-scale=$acwt --allow-partial=true --word-symbol-table=$graphdir/words.txt \   $graphdir/TLG.fst ark:- "ark:|gzip -c > $mdl/lat/lat.JOB.gz" || exit 1;
This nnet.py is currently using tensorflow, I have a version that doesn't
rely on that. I will push it now. Will keep you posted.
Thanks!
2018-07-24 6:20 GMT-04:00 ericbolo<EMAIL_ADDRESS>
re: priors. As @efosler https://github.com/efosler noted, it seems the
priors are not used in the current decoding code.
in tf_test.py:
if(config[constants.CONFIG_TAGS_TEST.USE_PRIORS]):
#TODO we need to add priors also:
#feed_priors={i: y for i, y in zip(model.priors, config["prior"])}
print(config[constants.CONFIG_TAGS_TEST.PRIORS_SCHEME])
model.priors doesn't seem to be generated anywhere, but we can use
label.counts to generate it.
@fmetze https://github.com/fmetze , are priors used in the original
(c++) implementation ?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/srvk/eesen/issues/193#issuecomment-407357122, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AMlwPR_eyqQT-me5-dBRyiyqStYX0Fe4ks5uJvTdgaJpZM4U2hdg
.

So, an update on my progress with SWB (now that I'm getting back to this).  I haven't tried out @ramonsanabria 's code above yet.
I'm able to train a SWB system getting 11.8% TER on the CV set (much better than before).  However, decoding with this (again not with priors) gives me a 40+% WER - much worse than the previous setup.  I'm trying to debug this to understand where things are going wrong.
One thing I tried to do was turn on TER calculation during the forward pass.  Had to do some modifications to steps/decode_ctc_am_tf.sh to make it pass the right flags to the test module.  However, that was a non-starter it seems - the forward pass just hangs with no errors.
Seems like the next best step would be to just try to switch to @ramonsanabria 's decode strategy and abandon steps/decode_ctc_am_tf.sh?

@ramonsanabria what's a good (rough) value for blank_scale?

@ramonsanabria Now looking through nnet.py (and non-tf version) - this actually takes the output of the net and does the smoothing and priors as a filter, right?  The code snippet you have above doesn't actually run the net forward, it seems to me, but would do something funky on the features in feats.scp.

Hi all,
How is it going. A good value for blank scale should be 0.9 and 1.1. But is
something that we should play with. Exactly, the nnet.py script will only
take the posteriors from the eesen, modify them slightly (add blank
scaling, put blank in index-zero so WFST can read it, add temperature to
the whole distribution, add priors which will certainly boost WER scores)
and finally pipe it to the next script, which I believe it is the WFST
decoding.
Will you guys be in India for Interspeech? Would be great to meet :)
2018-08-10 17:40 GMT+01:00 Eric Fosler-Lussier<EMAIL_ADDRESS>
@ramonsanabria https://github.com/ramonsanabria Now looking through
nnet.py (and non-tf version) - this actually takes the output of the net
and does the smoothing and priors as a filter, right? The code snippet you
have above doesn't actually run the net forward, it seems to me, but would
do something funky on the features in feats.scp.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/srvk/eesen/issues/193#issuecomment-412138534, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AMlwPWRcr8L4FFltzhLFlUa1kbv-ZW0yks5uPbeOgaJpZM4U2hdg
.

India, I wish ! But no...
On Fri, Aug 10, 2018, 8:43 PM ramonsanabria<EMAIL_ADDRESS>wrote:

Hi all,
How is it going. A good value for blank scale should be 0.9 and 1.1. But is
something that we should play with. Exactly, the nnet.py script will only
take the posteriors from the eesen, modify them slightly (add blank
scaling, put blank in index-zero so WFST can read it, add temperature to
the whole distribution, add priors which will certainly boost WER scores)
and finally pipe it to the next script, which I believe it is the WFST
decoding.
Will you guys be in India for Interspeech? Would be great to meet :)
2018-08-10 17:40 GMT+01:00 Eric Fosler-Lussier<EMAIL_ADDRESS>
@ramonsanabria https://github.com/ramonsanabria Now looking through
nnet.py (and non-tf version) - this actually takes the output of the net
and does the smoothing and priors as a filter, right? The code snippet
you
have above doesn't actually run the net forward, it seems to me, but
would
do something funky on the features in feats.scp.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/srvk/eesen/issues/193#issuecomment-412138534, or
mute
the thread
<
https://github.com/notifications/unsubscribe-auth/AMlwPWRcr8L4FFltzhLFlUa1kbv-ZW0yks5uPbeOgaJpZM4U2hdg
.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/srvk/eesen/issues/193#issuecomment-412171233, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AEXQ_K1PrDJkJz6OEpop29iaR5KFcWJoks5uPdRVgaJpZM4U2hdg
.

Alas, I won't be in India either.
Update on progress: I wrote a small script to do greedy decoding on a logit/posterior stream and calculate the TER.  (Will post this to my repo soonish and then send a pull request.)  Found that on the SWB eval2000 test set I was getting 30% TER (this after priors; without priors it is worse).  I was slightly puzzled by that, so I decided to calculate the TER on the train_dev set for SWB - I'm getting roughly 21-22% TER.  This was a system that was reporting 11.8% TER on the cv set (which I think is the same?) during training.  So something is rather hinky.  Still digging, but if anyone has ideas, let me know.

I think I've enabled tf_train to dump out the forward pass on cv to see what's going on - if there is a difference in the output.  Took me a good chunk of the evening. One thing I did run across is that the fp on subsampled data gets averaged in tf_test - it's not clear to me if the TER reported in tf_train is over averaged or (as I suspect) all variants.  I don't think this could account for a factor of two in TER though.
FWIW, I think the code would be cleaner if tf_train and tf_test were factorized some - I had to copy a lot of code over and I worry about inconsistencies between them (although they are hooked together through the model class).

Update from yesterday (now that the swbd system has some time to train): the dumped cv ark files do not show the same CTC error rate as the system claims.  I am suspecting that the averaging might be doing something weird.  Writing down assumptions here and someone can pick this apart:

The implementation of the greedy decoder is just to take the max at each frame, remove duplicates, and then remove blanks.  Shortest decoder I've ever written:

def greedy_decode(logits):     return [i for i,_ in itertools.groupby(logits.argmax(1)) if i>0]

The averaging code (taken from tf_test.py) checks if there is augmented data.  If so it computes the average logit stream from all examples.  But if the network has radically different placement of the tokens it will likely give different decodings.

(Now this is making me wonder if the test set was augmented... hmmm...)
Anyway, just to give a sample of the difference in TER:
Reported by tf during training:

             Validate cost: 40.4, ter: 27.6%, #example: 11190
            Validate cost: 32.9, ter: 22.2%, #example: 11190
            Validate cost: 30.2, ter: 21.2%, #example: 11190
            Validate cost: 27.8, ter: 19.2%, #example: 11190
            Validate cost: 26.8, ter: 18.3%, #example: 11190
            Validate cost: 35.0, ter: 23.7%, #example: 11190
            Validate cost: 28.4, ter: 19.4%, #example: 11190
            Validate cost: 24.8, ter: 17.1%, #example: 11190

Decoding on the averaged stream:

TER = 76690 / 152641 = 50.2
TER = 69108 / 152380 = 45.4
TER = 68611 / 152380 = 45.0
TER = 62259 / 152380 = 40.9
TER = 59838 / 152380 = 39.3
TER = 72821 / 152380 = 47.8
TER = 61498 / 152380 = 40.4
TER = 59800 / 152380 = 39.2

@ramonsanabria and @fmetze can you confirm what the online feature augmentation is doing?  I think I misunderstood it in my comments above.  (I had visions of other types of augmentation going on but reading the code I think it's simpler than I thought.)
Looking through the code it seems like when you have the subsample and window set to 3, what it's doing is stacking three frames on the input, and making the input 3 times as small.  Is it also creating three variants with different shifts?  I'm trying to figure out where the averaging would come in later.

Created new issue for this particular bug.  #194

Latest update:
Decoding with sw+fish LM, incorporating priors, and fixing the averaging bug leads to 19.2% WER on eval 2000, with the swbd subset getting 13.4% WER (the kaldi triphone based system gets 13.3% WER on the same set, although this may be a more involved model).  I think that this is close enough for a baseline to declare victory.  I'll clean stuff up and then make a pull request.

Successful full train and decode; I also tested out a run with a slightly larger net (with a bit of improvement).  Adding these baselines to the README file.
# CTC Phonemes on the Complete set (with 5 BiLSTM layers) with WFST decode
%WER 12.5 | 1831 21395 | 88.9 7.7 3.4 1.5 12.5 49.6 | exp/train_phn_fbank_pitch_l5_c320_mdeepbilstm_w3_ntrue_p60_ip80_fp80/results_epoch22_bs5.0_sw1_fsh_tgpr/score_8/eval2000.ctm.swbd.filt.sys
%WER 18.3 | 4459 42989 | 83.9 11.7 4.4 2.2 18.3 57.3 | exp/train_phn_fbank_pitch_l5_c320_mdeepbilstm_w3_ntrue_p60_ip80_fp80/results_epoch22_bs4.0_sw1_fsh_tgpr/score_8/eval2000.ctm.filt.sys
%WER 23.9 | 2628 21594 | 79.0 15.5 5.6 2.8 23.9 62.5 | exp/train_phn_fbank_pitch_l5_c320_mdeepbilstm_w3_ntrue_p60_ip80_fp80/results_epoch22_bs4.0_sw1_fsh_tgpr/score_8/eval2000.ctm.callhm.filt.sys

# Slightly larger model (400 units, 80 internal projections) with WFST decode
%WER 12.2 | 1831 21395 | 89.2 7.7 3.1 1.4 12.2 49.7 | exp/train_phn_fbank_pitch_l5_c400_mdeepbilstm_w3_ntrue_p80_ip80_fp80/results_epoch23_bs7.0_sw1_fsh_tgpr/score_10/eval2000.ctm.swbd.filt.sys
%WER 17.8 | 4459 42989 | 84.1 11.1 4.8 1.9 17.8 57.1 | exp/train_phn_fbank_pitch_l5_c400_mdeepbilstm_w3_ntrue_p80_ip80_fp80/results_epoch23_bs7.0_sw1_fsh_tgpr/score_9/eval2000.ctm.filt.sys
%WER 23.4 | 2628 21594 | 79.3 14.8 5.9 2.7 23.4 62.1 | exp/train_phn_fbank_pitch_l5_c400_mdeepbilstm_w3_ntrue_p80_ip80_fp80/results_epoch23_bs7.0_sw1_fsh_tgpr/score_10/eval2000.ctm.callhm.filt.sys

Awesome thank you so much Eric! the numbers looks great. Can you share the full training configuration?
Thank you again!

Just submitted the pull request (#196).

Once we decide that #196 is all good, I think we can close this particular thread!!!


Warning when using GridEvenShare with unsigned offsets

The following warning occurs when DispatchRadixSort is instantiated with an unsigned offset type.
cuda-11.0/bin/../targets/x86_64-linux/include/cub/device/dispatch/../../grid/grid_even_share.cuh: In instantiation of ‘void cub::GridEvenShare<OffsetT>::DispatchInit(OffsetT, int, int) [with OffsetT = unsigned int]’:
required from here
cuda-11.0/bin/../targets/x86_64-linux/include/cub/device/dispatch/../../grid/grid_even_share.cuh:133:37: warning: comparison of integer expressions of different signedness: ‘int’ and ‘unsigned int’ [-Wsign-compare]
  133 |         this->grid_size             = CUB_MIN(total_tiles, max_grid_size);

Minimal reproduction code:
// nvcc -std=c++14 -Xcompiler "-Wall" file.cu
// Cuda compilation tools, release 11.0, V11.0.221
// Build cuda_11.0_bu.TC445_37.28845127_0

#include <cub/cub.cuh>

int main(){
    cub::GridEvenShare<unsigned int> a;
    a.DispatchInit(1,1,1);
}

Is it safe to ignore this warning?

Heh, I literally just fixed that on my local branch for #249 yesterday :) This shouldn't be an issue after the next release.

Is it safe to ignore this warning?

Yes.

Heh, I literally just fixed that on my local branch for #249 yesterday :) This shouldn't be an issue after the next release.

Is it safe to ignore this warning?

Yes.


Remove storage classes and default storage requests for cloud providers

…

@tybritten added a bunch more docs on recommended storage size and storage classes

perfect


New aka.ms link

Aka.ms link name
SecureWorkstation
Category (Optional. Select None to auto-select)
None

Thank you for submitting aka.ms/SecureWorkstation. Your link already exists at akaSearch.net. 🙏✅


Update for ReSharper 2017.2

Updated to wave 9 for ReSharper 2017.2 support.
Incremented version number to 5.2.0

Any ETA on when this can be expected to be merged in?

Any plans to merge this?


Unable to set document-quality using PutMarkLogic processor

I'm using PutMarkLogic <IP_ADDRESS> and as per https://marklogic.github.io/nifi/nifi-features#putmarklogic-processor setting quality is not supported.
Is there any work-around for this?

Closing as this was resolved via #121 in <IP_ADDRESS>


python38Packages.scikits-odes: 2.6.1 -> 2.6.2

Automatic update generated by nixpkgs-update tools. This update was made based on information from passthru.updateScript.
meta.description for python38Packages.scikits-odes is: "A scikit offering extra ode/dae solvers, as an extension to what is available in scipy"
meta.homepage for python38Packages.scikits-odes is: "https://github.com/bmcage/odes"
meta.changelog for python38Packages.scikits-odes is: ""
Updates performed

Ran passthru.UpdateScript

To inspect upstream changes
Impact

Checks done (click to expand)

built on NixOS

Warning: a test defined in passthru.tests did not pass

0 of 0 passed binary check by having a zero exit code.

0 of 0 passed binary check by having the new version present in output.

found 2.6.2 with grep in /nix/store/d88z0rnda69bvfzqfnd8l9rvrn5sni2y-python3.8-scikits.odes-2.6.2

found 2.6.2 in filename of file in /nix/store/d88z0rnda69bvfzqfnd8l9rvrn5sni2y-python3.8-scikits.odes-2.6.2

directory tree listing: https://gist.github.com/f3e3fb91344c6252f760f10c1f82a1ed

du listing: https://gist.github.com/c90e478279549d8a0969057e0330ed33

Rebuild report (if merged into master) (click to expand)

6 total rebuild path(s)

2 package rebuild(s)

2 x86_64-linux rebuild(s)
0 i686-linux rebuild(s)
2 x86_64-darwin rebuild(s)
2 aarch64-linux rebuild(s)

First fifty rebuilds by attrpath
python38Packages.scikits-odes
python39Packages.scikits-odes

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/d88z0rnda69bvfzqfnd8l9rvrn5sni2y-python3.8-scikits.odes-2.6.2 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A python38Packages.scikits-odes https://github.com/r-ryantm/nixpkgs/archive/ed2eda8f72793a7bb22aa84c4565e00c87c99251.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/d88z0rnda69bvfzqfnd8l9rvrn5sni2y-python3.8-scikits.odes-2.6.2
ls -la /nix/store/d88z0rnda69bvfzqfnd8l9rvrn5sni2y-python3.8-scikits.odes-2.6.2/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

2 packages built:

python38Packages.scikits-odes
python39Packages.scikits-odes

Maintainer pings
cc @idontgetoutmuch for testing.

Result of nixpkgs-review pr 138246 at ed2eda8f run on aarch64-linux 1

2 packages built successfully:

python38Packages.scikits-odes
python39Packages.scikits-odes

1 suggestion:

warning: unused-argument
Unused argument: fetchurl.
Near pkgs/development/python-modules/scikits-odes/default.nix:4:3:
  |
4 | , fetchurl
  |   ^


Missing "two digit hour" and "hide menu button" options in v3

Per Prithibi Biswas via Twitter, the options for a two digit hour (regardless of 12 hour setting) and the option to hide the menu button are missing from v3.

Both options have been re-added and will be available in version 3.2.0


Feature/cubed sphere constr xylatlon

Description
Note that I have not implemented the FV3EquiDistant projection
This change deals with the main issue that Willem had with the original app - that the projections need explicitly the tile index to work.
What I have done for equidistant and equiangular projections. In the grid we go from (xyt)(indices) <-> xy (degrees) <-> alpha beta (degrees) <-> longitude/latitude (degrees).
There is a wrinkle in the design at present.  There is at duplicate routine in the grid constructor and in the CubedSphereProjectionBase, namely tileFromXY.

@odlomax - are you o.k. with approving this?

@odlomax - I plan to create a separate PR to update the base to the latest ecmwf develop.


Crash when jumping to error

Whenever i jump to error Atom crashes, I am using cl on windows.

Crashes how?
Can you bring up dev tools and post any errors from there?

how do i do that?

and crashing as in stops responding

I'm gonna need more information if I should be able to do something here... Steps to reproduce, crashlog, error outputs, etc.

.atom-build.json:
{
  "cmd": "{PROJECT_PATH}/code/build.bat",
  "args": [],
  "sh": false,
  "cwd": "{PROJECT_PATH}",
  "env": {

  },
  "errorMatch": "(?<file>.+)\\((?<line>\\d+)\\)"
}

build.bat:
@echo off

call "c:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\vcvarsall.bat" x64
cd w:\build

cl -DHANDMADE_WIN32=1 -FC -Zi ..\handmade\code\win32_handmade.cpp user32.lib gdi32.lib

steps to crash are

Make an error in your code
Use errorMarcher to jump to the error
You get a crash, or at least i got one.

I don't know where to get the crashlog, so if you would be so nice to tell where to find it, I could give it to you.

Does Atom terminate, or freeze?
I have never experienced a crash so I don't know where to find crash logs. I would have to look it up, same as you - and since I'm not on windows chances are it won't help you.

Seems atom-build doesn't like subst at all. Just tried to go to where it was on the actual drive, and build it from there, and it works like a charm.

subst?

virtual drive

the command on windows is subst

So it works well inside atom, unless the project root is via a virtual drive?

yes

then it freezes and asks to close.


module 'ark.phenotyping.post_cluster_utils' has no attribute 'generate_new_cluster_resolution'

Hello!
I am running 4_Post_Clustering, and when importing packages, I was unable to import 'ark.phenotyping.post_cluster_utils.generate_new_cluster_resoltuion'. I was looking at the documentation for post_cluster_utils and there is no generate_new_cluster_utils there.
I have tried running on both v0_5_2 and v0_6_4. Let me know if I am doing something wrong, thank you so much in advance!

Hi @c0urtneyt! That function is here: https://github.com/angelolab/ark-analysis/blob/main/src/ark/phenotyping/post_cluster_utils.py#L174-L208 and was added in this pull request: https://github.com/angelolab/ark-analysis/pull/1095.
A new version of ark was recently released (v0.7.1) - try installing this version and running it again. Hope this helps!


Add ArangoDB Certified Professional by ArangoDB

[!NOTE]
Thanks to @ediazhu for opening the issue and @karentamrazyan for requesting the PR.

This PR adds the ArangoDB Certified Professional exam to the list.
Closes #83

@codeTun Nice suggestion. Please create for it an issue or add it yourself via a PR. Thanks.


Current instructions for launch options constantly crashes SE for me

I navigated to the most recent release, downloaded the most recent release, followed directions to add the given files to Bin64 of Space Engineers, and I put this string in the game launcher window: -plugin PluginLoader.dll.
I think my problems are the launch option currently.
Every time I try to open the game it crashes
I like the released plugins and using plugin loader - any help would be appreciated

Hi, If you can join the Plugin Loader Discord and post your full game and loader logs, then we can help you right away. If not, then please attach your log files here.
Latest SpaceEngineers_*.log file by date from the %AppData%\SpaceEngineers directory.
Loader log should be in C:\Program Files (x86)\Steam\steamapps\common\SpaceEngineers\Bin64\Plugins, unless you moved the game files to somewhere else.

Resolved by using the Unblock button on the DLLs


Repeated force install fails

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
BUG REPORT
Environment (plugin version, maven version, OS, ...):
plugin version: 6.13.0
mvn version: 3.8.1
helm version: 3.8.0
What happened:

Plugin configuration:

<configuration>
    <chartDirectory>helm/</chartDirectory>
    <chartVersion>0.1.0</chartVersion>
    <useLocalHelmBinary>true</useLocalHelmBinary>
    <installForce>true</installForce>
</configuration>

helm:install done successfylly, created resource has annotations:

meta.helm.sh/release-name: my-app
meta.helm.sh/release-namespace: my-namespace

helm:dry-run fail with error

[ERROR] Error: INSTALLATION FAILED: rendered manifests contain a resource that already exists. Unable to continue with install: ServiceAccount "my-app" in namespace "my-namespace" exists and cannot be imported into the current release: invalid ownership metadata; annotation validation error: key "meta.helm.sh/release-name" must equal "my-app-1706159052": current value is "my-app"

helm:install fail with error:

[ERROR] Error: unknown flag: --force

helm:uninstall done successfylly

What you expected to happen:
3) I'm using installForce=true In my configuration and expect repeated instalation (without uninstall) will work fine.
So i do not understend why there is inconsistency between install command (creates meta.helm.sh/release-name without uid) and dry-run command (expects meta.helm.sh/release-name with uid).
4) Expect Helm version: 3.8.0 - supports install force flag.
How to reproduce it (as minimally and precisely as possible):
Test resource:
apiVersion: v2
name: my-app
description: A Helm chart for Kubernetes
type: application
version: 0.1.0
appVersion: "1.16.0"

apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-app
  namespace: my-namespace

@tipame  Name for install was fixed with https://github.com/kokuwaio/helm-maven-plugin/pull/353. --force is not supported by 3.8.0 and was added with 3.11.0. Can you check with updated helm/plugin?

Closing due missing feedback


Do no longer force-install a default nginx.conf

As far as I can tell, a default nginx.conf does no longer need to be force-installed. Let me know if I forgot something.

The force install was introduced here.

Seems like tests fail?

Fixed.

Going to close this. A real fix needs to take into account the fact that we don't use the PPA for nginx with the debian solution :( I'll take care of this in #1467.

Scratch that. Seems this can go in after all :)

Closing in favor of #1468. Thanks for your work @SonicHedgehog!

Whoa! We want both I think, yea?

Right I'm a nub. Reopened.


don't see alert on dashboard

Hi Folk,
I'm having several issues with cyphon, I think that my config is fine, my site is:

Source sent to udp port 5044 to Cyphon (cyphondock).
Logs Cyphon: cyphon_1         | Running Production Server

Logstash conf like input, filter and output
Logs Logstash:
logstash_1       | [2018-04-02T14:50:31,359][INFO ][logstash.inputs.beats    ] Beats inputs: Starting input listener {:address=>"<IP_ADDRESS>:5044"}
logstash_1       | [2018-04-02T14:50:31,444][WARN ][logstash.inputs.beats    ] WARNING! - Multiline codec with beats input has been deprecated. Support for this configuration will be removed in a future version. Please refer to the beats documentation for how to best manage multiline data. See https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html
logstash_1       | [2018-04-02T14:50:31,452][INFO ][logstash.inputs.beats    ] Beats inputs: Starting input listener {:address=>"<IP_ADDRESS>:5045"}
logstash_1       | [2018-04-02T14:50:31,473][INFO ][logstash.inputs.udp      ] Starting UDP listener {:address=>"<IP_ADDRESS>:5046"}
logstash_1       | [2018-04-02T14:50:31,475][INFO ][org.logstash.beats.Server] Starting server on port: 5045
logstash_1       | [2018-04-02T14:50:31,478][INFO ][logstash.inputs.udp      ] Starting UDP listener {:address=>"<IP_ADDRESS>:5000"}
logstash_1       | [2018-04-02T14:50:31,477][INFO ][org.logstash.beats.Server] Starting server on port: 5044
logstash_1       | [2018-04-02T14:50:31,491][INFO ][logstash.inputs.udp      ] UDP listener started {:address=>"<IP_ADDRESS>:5046", :receive_buffer_bytes=>"106496", :queue_size=>"2000"}
logstash_1       | [2018-04-02T14:50:31,491][INFO ][logstash.inputs.udp      ] UDP listener started {:address=>"<IP_ADDRESS>:5000", :receive_buffer_bytes=>"106496", :queue_size=>"2000"}
logstash_1       | [2018-04-02T14:50:31,518][INFO ][logstash.pipeline        ] Pipeline main started

Elasticsearach and kibana are up.

I see with tcpdump on Cyphon machine the events and good port.
Do I have restart the services but show the events on Cyphon and Kibana????
What I must do but that it is work?
Regards

Did you setup your data structure in Cyphon? Documentation on that is found here: http://cyphon.readthedocs.io/en/latest/tutorials/logstash-tutorial.html#step-3-cyphon

Yes, I set this tutorial but I don't see nothing. Do you think to must restart the machine?
Regards

alerts are not stored in elasticsearch, and thus not visible via Kibana. Alerts are stored in the postgres database.
What beat clients do you have running pointed at port 5044? 5044 is used for the Elastic Beats agents, ie. filebeat, winlogbeat, metricbeat, etc. Any other data types being pointed at port 5044 will be dropped due to not being properly formatted.

Hi, I'm seending logs vi another port like 5046. I have pharser the logs on logstash (can see on kibana properly). Do you say me that Cyphon not lisen on another port ????
I don't see the alerts on Cyphon, I've followed this FAQ "http://cyphon.readthedocs.io/en/latest/tutorials/logstash-tutorial.html#"
Thanks

What i can do what I see the logs into Cyphon???
The logs of Cyphon is:
user@Cyphon02:/opt/cyphon/cyphondock$ docker-compose logs -f cyphon
Attaching to cyphondock_cyphon_1
cyphon_1         | Running Production Server
Cyphon seems is working fine because I see the dashboard and I've created a new alert and all is ok.
Any help for me, please!

Please read through the tutorial again. The tutorial is specifically designed with ports in mind, changing the ports or pieces of the tutorial require a more thorough understanding of the underlying concepts of setup.
The alerts to Cyphon are fed through rabbitmq, which are consumed by the watchdog into Cyphon. If you don't have all the components setup, you won't be seeing alerts.
Cyphon is not for Log Management, you keep mentioning "I don't see the logs". The only items that you should be feeding into Cyphon are the events that require alerting. In the example, snort alerts are being sent into logstash, which then parses the alert events, and feeds them through to RabbitMQ.


Debugging of HIP executable with rocgdb is broken as of rocm 5.4.2

Tried to hit a breakpoint in the kernel function defined by the following example:
> cat rocgdb_test.hip
#include <hip/hip_runtime.h>

__global__ void kernel(int* i)
{
	(*i)++;
}

int main(int argc, char* argv[])
{
	int* val;
	hipMalloc(&val, sizeof(int));
	hipMemcpy(val, &argc, sizeof(int), hipMemcpyHostToDevice);
	kernel<<<dim3(1), dim3(1)>>>(val);
	hipDeviceSynchronize();
	hipMemcpy(&argc, val, sizeof(int), hipMemcpyDeviceToHost);
	printf("result = %d\n", argc);
	return 0;
}

Compiling with host and device debugging:
> /opt/rocm/hip/bin/hipcc --version
HIP version: 5.4.22803-474e8620
AMD clang version 15.0.0 (https://github.com/RadeonOpenCompute/llvm-project roc-5.4.2 22506 5cddd31fe35b165caadc4409b1d79d0d377c89be)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/rocm-5.4.2/llvm/bin

/opt/rocm/hip/bin/hipcc -ggdb -fstandalone-debug -Xclang -O0 -Xclang -gcodeview rocgdb_test.hip -o rocgdb_test

The rocgdb is unable to pick up a source location within the kernel code, when attempting breakpoint at line 5:
rocgdb ./rocgdb_test
Reading symbols from ./rocgdb_test...
(gdb) b 5
No compiled code for line 5 in the current file.
Make breakpoint pending on future shared library load? (y or [n]) n
(gdb) q
marcusmae@vinet ~/a/rocgdb_test> rocgdb ./rocgdb_test
Reading symbols from ./rocgdb_test...
(gdb) start
Temporary breakpoint 1 at 0x214ab6: file rocgdb_test.hip, line 11.
Starting program: /home/marcusmae/amd/rocgdb_test/rocgdb_test
warning: AMD GPU driver's version 1.1 not supported (version must be >= 1.6 and < 2.0)
warning: amd-dbgapi: unable to enable GPU debugging due to a restriction error
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Temporary breakpoint 1, main (argc=1, argv=0x7fffffffe4f8) at rocgdb_test.hip:11
11		hipMalloc(&val, sizeof(int));
(gdb) b 5
No compiled code for line 5 in the current file.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 2 (5) pending.
(gdb) li 5
1	#include <hip/hip_runtime.h>
2
3	__global__ void kernel(int* i)
4	{
5		(*i)++;
6	}
7
8	int main(int argc, char* argv[])
9	{
10		int* val;

Therefore, as of ROCm 5.4.2 HIP code debugging is unusable.
Also note the following warning messages:
warning: AMD GPU driver's version 1.1 not supported (version must be >= 1.6 and < 2.0)
warning: amd-dbgapi: unable to enable GPU debugging due to a restriction error

@dmikushin Apologies for the lack of response. Can you please test with latest ROCm 6.0.2 (HIP 6.0.32831)? If resolved, please close ticket. Thanks!

@dmikushin Closing ticket. Please re-open if you still see the issue with the latest ROCm 6.1.2. Thanks!


make runtime/incremental generator something that can be iterated on locally without packagereferences

This was keyed off by our conversation today, @KathleenDollard.
The idea is that in the released version, people do have to use PackageReferences, but when doing a simple dotnet build of the Example project, the defaults use a combination of ProjectReferences and manual wiring up of the Analyzer MSBuild item to ensure the analyzer generates as you expect.
The test matrix becomes:

clone empty and dotnet build Example.csproj -> should compile out of the box
make the local nuget source, pack and push runtime, pack and push incremental with -p:TestRelease=true set, dotnet build Example.csproj -p:TestRelease=true -> should also compile

Closed in favor of Kathleen's PR.


Remove @Primary annotations

Coverage remained the same at 69.841% when pulling e844ebc45bca059a499cfc7cd4e4310f4eb51691 on scottmo/removePrimaryAnnotations into 78e7152f4fd783f95d87c20d48c83738a28b56a7 on master.

Coverage remained the same at 69.841% when pulling e844ebc45bca059a499cfc7cd4e4310f4eb51691 on scottmo/removePrimaryAnnotations into 78e7152f4fd783f95d87c20d48c83738a28b56a7 on master.


fix TypeScript error TS4023 "using name 'SchemaObjectInstance'"

error TS4023: Exported variable 'MyVariableName' has or is using name 'SchemaObjectInstance' from external module 'schema-object' but cannot be named.
This is because the interface is not exported from the module and is not globally accessible

thank you for the PR!

No worries, can you let me know when you publish a release to npm so that I can update.
Cheers 😄