microRNAseq analysis using bcbio for non model organisms

Hi,
Do you expect that the microRNAseq analysis capability provided by bcbio would make sense for analysis of microRNAseq data of non model organisms?
I am trying to see if I can process the Capsicum annuum microRNAseq data generated in this project using bcbio:
https://www.ncbi.nlm.nih.gov/bioproject/PRJNA177852
I am new to microRNAseq analysis so I am not really sure how to run this analysis and I am also not sure what kind of output I should expect.
The following is the yaml file that I am using:
upload:
  dir: ../final
details:
  - analysis: smallRNA-seq
    algorithm:
      aligner: star # any other aligner is supported.
      # change adapter according project
      # adapters: ["TGGAATTCTCGGGTGC"]
      expression_caller: [ seqcluster, mirdeep2]
      # expression_caller: [trna, seqcluster, mirdeep2, mirge] Read docs to know how to use
      # miRge tools: https://bcbio-nextgen.readthedocs.io/en/latest/contents/pipelines.html#smallrna-seq
      # species: hsa
    genome_build: my_ref
#resources:
#  atropos:
#    options: ["-u 4", "-u -4"]
#  mirge:
#    options: ["-lib $PATH_TO_LIBS_FOLDER"]

This is the log file produced by the analysis.
[2018-06-27T18:55Z] grid_controller: System YAML configuration: /workspace/my_user/tmp_bcbio_1.1.0_development/data_dir/galaxy/bcbio_system.yaml
[2018-06-27T18:56Z] grid_controller: Timing: organize samples
[2018-06-27T18:56Z] grid_controller: ipython: organize_samples
[2018-06-27T18:56Z] exeuction_node_20: Using input YAML configuration: /leading_dir/config/DA_1164_samples-merged.
yaml
[2018-06-27T18:56Z] exeuction_node_20: Checking sample YAML configuration: /leading_dir/config/DA_1164_samples-mer
ged.yaml
[2018-06-27T18:56Z] exeuction_node_20: Testing minimum versions of installed programs
[2018-06-27T18:56Z] grid_controller: ipython: prepare_sample
[2018-06-27T18:56Z] grid_controller: Timing: adapter trimming
[2018-06-27T18:56Z] grid_controller: ipython: trim_srna_sample
[2018-06-27T19:26Z] exeuction_node_20: Collapsing /leading_dir/work/trimmed/DA_1164_01/DA_1164_01.clean.fastq.gz
 with --min_size 16 --min 1
[2018-06-27T20:17Z] exeuction_node_20: Collapsing /leading_dir/work/trimmed/DA_1164_02/DA_1164_02.clean.fastq.gz
 with --min_size 16 --min 1
[2018-06-27T21:00Z] exeuction_node_20: Collapsing /leading_dir/work/trimmed/DA_1164_03/DA_1164_03.clean.fastq.gz
 with --min_size 16 --min 1
[2018-06-27T21:42Z] exeuction_node_20: Collapsing /leading_dir/work/trimmed/DA_1164_04/DA_1164_04.clean.fastq.gz
 with --min_size 16 --min 1
[2018-06-27T22:20Z] exeuction_node_20: Collapsing /leading_dir/work/trimmed/DA_1164_05/DA_1164_05.clean.fastq.gz
 with --min_size 16 --min 1
[2018-06-27T22:55Z] exeuction_node_20: Collapsing /leading_dir/work/trimmed/DA_1164_06/DA_1164_06.clean.fastq.gz
 with --min_size 16 --min 1
[2018-06-27T23:31Z] exeuction_node_20: Collapsing /leading_dir/work/trimmed/DA_1164_07/DA_1164_07.clean.fastq.gz
 with --min_size 16 --min 1
[2018-06-28T00:28Z] exeuction_node_20: Collapsing /leading_dir/work/trimmed/DA_1164_08/DA_1164_08.clean.fastq.gz
 with --min_size 16 --min 1
[2018-06-28T01:11Z] exeuction_node_20: Collapsing /leading_dir/work/trimmed/DA_1164_09/DA_1164_09.clean.fastq.gz
 with --min_size 16 --min 1
[2018-06-28T01:54Z] exeuction_node_20: Collapsing /leading_dir/work/trimmed/DA_1164_10/DA_1164_10.clean.fastq.gz
 with --min_size 16 --min 1
[2018-06-28T02:13Z] grid_controller: Timing: prepare
[2018-06-28T02:13Z] grid_controller: ipython: seqcluster_prepare
[2018-06-28T03:05Z] exeuction_node_24: Prepare seqs.fastq with -minl 17 -maxl 40 -minc 2 --min_shared 0.1
[2018-06-28T03:08Z] grid_controller: Timing: alignment
[2018-06-28T03:08Z] grid_controller: ipython: srna_alignment
[2018-06-28T03:08Z] exeuction_node_24: Aligning lane DA_1164_01 with star aligner
[2018-06-28T03:11Z] exeuction_node_24: mirdeep2 Rfam file not instaled. Skipping...
[2018-06-28T03:11Z] grid_controller: Timing: small RNA annotation
[2018-06-28T03:11Z] grid_controller: ipython: srna_annotation
[2018-06-28T03:12Z] grid_controller: Timing: cluster
[2018-06-28T03:12Z] grid_controller: ipython: seqcluster_cluster
[2018-06-28T04:59Z] grid_controller: Timing: quality control
[2018-06-28T04:59Z] grid_controller: ipython: pipeline_summary
[2018-06-28T04:59Z] exeuction_node_20: QC: DA_1164_01 fastqc

I am not sure how to specify that dnapi should be run for de-novo adapter detection followed by adapter trimming.  As far as I can tell dnapi  was not used for adapter trimming.  The fastqc part of the multiqQC report shows that of the 50bp reads the last 25 bp  is almost 100% adapters.
As far as I can tell Capsicum annuum is not in mirbase. Therefore I did not enter a 3 letter species code.  I am not sure if it makes sense to just enter the species code of a somewhat related species
http://www.mirbase.org/cgi-bin/mirna_summary.pl?org=sly
Or that I better just don't provide a species code.
The analysis did not seem to produce much results. See the file list at the bottom of this comment. Then again I am also not sure what to expect.
The lack of output might in part be because mirdeep2 Rfam not being installed/found. Should I have done that myself?
[2018-06-28T03:08Z] exeuction_node_24: Aligning lane DA_1164_01 with star aligner
[2018-06-28T03:11Z] exeuction_node_24: mirdeep2 Rfam file not instaled. Skipping...

What I kind of expect as output for an microRNAseq analysis is:

identification/ filtering of known/discovered non microRNA sequences (either biological (e.g. other RNA's) or adapters)
identification of know mircroRNA sequences from mirbase or similar
identification of new microRNA's
per sample alignment BAM files of the microRNA sequences (not sure if this should run against the genome or transcriptome (or both). And I am not sure if  these alignments identify target loci/mRNAs or microRNA precursur loci/mRNAs (or both))
microRNA target mRNA/gene prediction
microRNA quantification

Do you think it is possible to get the above results using bcbio for microRNAseq data of a non model organism?
How would I then do that using bcbio? Is the yaml that I use correct? Should I add tRNA as an expression caller?
Since I am new to microRNAseq the bcbio microRNAseq documentation is also a bit short me.
I would also very much appreciate it if you can point to me a recent sort of best practice method  / review paper that describes the method(s) that bcbio in general tries to provide for microRNAseq analysis.
Thank you very much!
final/
final/DA_1164_05
final/DA_1164_05/qc
final/DA_1164_05/qc/fastqc
final/DA_1164_05/qc/fastqc/fastqc_report.html
final/DA_1164_05/qc/fastqc/fastqc_data.txt
final/DA_1164_05/qc/fastqc/DA_1164_05.zip
final/DA_1164_05/qc/fastqc/Per_base_sequence_quality.tsv
final/DA_1164_05/qc/fastqc/Per_tile_sequence_quality.tsv
final/DA_1164_05/qc/fastqc/Per_sequence_quality_scores.tsv
final/DA_1164_05/qc/fastqc/Per_base_sequence_content.tsv
final/DA_1164_05/qc/fastqc/Per_sequence_GC_content.tsv
final/DA_1164_05/qc/fastqc/Per_base_N_content.tsv
final/DA_1164_05/qc/fastqc/Sequence_Length_Distribution.tsv
final/DA_1164_05/DA_1164_05-ready.trimming_stats
final/DA_1164_04
final/DA_1164_04/qc
final/DA_1164_04/qc/fastqc
final/DA_1164_04/qc/fastqc/fastqc_report.html
final/DA_1164_04/qc/fastqc/fastqc_data.txt
final/DA_1164_04/qc/fastqc/DA_1164_04.zip
final/DA_1164_04/qc/fastqc/Per_base_sequence_quality.tsv
final/DA_1164_04/qc/fastqc/Per_tile_sequence_quality.tsv
final/DA_1164_04/qc/fastqc/Per_sequence_quality_scores.tsv
final/DA_1164_04/qc/fastqc/Per_base_sequence_content.tsv
final/DA_1164_04/qc/fastqc/Per_sequence_GC_content.tsv
final/DA_1164_04/qc/fastqc/Per_base_N_content.tsv
final/DA_1164_04/qc/fastqc/Sequence_Length_Distribution.tsv
final/DA_1164_04/DA_1164_04-ready.trimming_stats
final/DA_1164_09
final/DA_1164_09/qc
final/DA_1164_09/qc/fastqc
final/DA_1164_09/qc/fastqc/fastqc_report.html
final/DA_1164_09/qc/fastqc/fastqc_data.txt
final/DA_1164_09/qc/fastqc/DA_1164_09.zip
final/DA_1164_09/qc/fastqc/Per_base_sequence_quality.tsv
final/DA_1164_09/qc/fastqc/Per_tile_sequence_quality.tsv
final/DA_1164_09/qc/fastqc/Per_sequence_quality_scores.tsv
final/DA_1164_09/qc/fastqc/Per_base_sequence_content.tsv
final/DA_1164_09/qc/fastqc/Per_sequence_GC_content.tsv
final/DA_1164_09/qc/fastqc/Per_base_N_content.tsv
final/DA_1164_09/qc/fastqc/Sequence_Length_Distribution.tsv
final/DA_1164_09/DA_1164_09-ready.trimming_stats
final/DA_1164_08
final/DA_1164_08/qc
final/DA_1164_08/qc/fastqc
final/DA_1164_08/qc/fastqc/fastqc_report.html
final/DA_1164_08/qc/fastqc/fastqc_data.txt
final/DA_1164_08/qc/fastqc/DA_1164_08.zip
final/DA_1164_08/qc/fastqc/Per_base_sequence_quality.tsv
final/DA_1164_08/qc/fastqc/Per_tile_sequence_quality.tsv
final/DA_1164_08/qc/fastqc/Per_sequence_quality_scores.tsv
final/DA_1164_08/qc/fastqc/Per_base_sequence_content.tsv
final/DA_1164_08/qc/fastqc/Per_sequence_GC_content.tsv
final/DA_1164_08/qc/fastqc/Per_base_N_content.tsv
final/DA_1164_08/qc/fastqc/Sequence_Length_Distribution.tsv
final/DA_1164_08/DA_1164_08-ready.trimming_stats
final/DA_1164_07
final/DA_1164_07/qc
final/DA_1164_07/qc/fastqc
final/DA_1164_07/qc/fastqc/fastqc_report.html
final/DA_1164_07/qc/fastqc/fastqc_data.txt
final/DA_1164_07/qc/fastqc/DA_1164_07.zip
final/DA_1164_07/qc/fastqc/Per_base_sequence_quality.tsv
final/DA_1164_07/qc/fastqc/Per_tile_sequence_quality.tsv
final/DA_1164_07/qc/fastqc/Per_sequence_quality_scores.tsv
final/DA_1164_07/qc/fastqc/Per_base_sequence_content.tsv
final/DA_1164_07/qc/fastqc/Per_sequence_GC_content.tsv
final/DA_1164_07/qc/fastqc/Per_base_N_content.tsv
final/DA_1164_07/qc/fastqc/Sequence_Length_Distribution.tsv
final/DA_1164_07/DA_1164_07-ready.trimming_stats
final/DA_1164_06
final/DA_1164_06/qc
final/DA_1164_06/qc/fastqc
final/DA_1164_06/qc/fastqc/fastqc_report.html
final/DA_1164_06/qc/fastqc/fastqc_data.txt
final/DA_1164_06/qc/fastqc/DA_1164_06.zip
final/DA_1164_06/qc/fastqc/Per_base_sequence_quality.tsv
final/DA_1164_06/qc/fastqc/Per_tile_sequence_quality.tsv
final/DA_1164_06/qc/fastqc/Per_sequence_quality_scores.tsv
final/DA_1164_06/qc/fastqc/Per_base_sequence_content.tsv
final/DA_1164_06/qc/fastqc/Per_sequence_GC_content.tsv
final/DA_1164_06/qc/fastqc/Per_base_N_content.tsv
final/DA_1164_06/qc/fastqc/Sequence_Length_Distribution.tsv
final/DA_1164_06/DA_1164_06-ready.trimming_stats
final/DA_1164_01
final/DA_1164_01/qc
final/DA_1164_01/qc/fastqc
final/DA_1164_01/qc/fastqc/fastqc_report.html
final/DA_1164_01/qc/fastqc/fastqc_data.txt
final/DA_1164_01/qc/fastqc/DA_1164_01.zip
final/DA_1164_01/qc/fastqc/Per_base_sequence_quality.tsv
final/DA_1164_01/qc/fastqc/Per_tile_sequence_quality.tsv
final/DA_1164_01/qc/fastqc/Per_sequence_quality_scores.tsv
final/DA_1164_01/qc/fastqc/Per_base_sequence_content.tsv
final/DA_1164_01/qc/fastqc/Per_sequence_GC_content.tsv
final/DA_1164_01/qc/fastqc/Per_base_N_content.tsv
final/DA_1164_01/qc/fastqc/Sequence_Length_Distribution.tsv
final/DA_1164_01/qc/small-rna
final/DA_1164_01/qc/small-rna/DA_1164_01.txt
final/DA_1164_01/DA_1164_01-ready.trimming_stats
final/DA_1164_03
final/DA_1164_03/qc
final/DA_1164_03/qc/fastqc
final/DA_1164_03/qc/fastqc/fastqc_report.html
final/DA_1164_03/qc/fastqc/fastqc_data.txt
final/DA_1164_03/qc/fastqc/DA_1164_03.zip
final/DA_1164_03/qc/fastqc/Per_base_sequence_quality.tsv
final/DA_1164_03/qc/fastqc/Per_tile_sequence_quality.tsv
final/DA_1164_03/qc/fastqc/Per_sequence_quality_scores.tsv
final/DA_1164_03/qc/fastqc/Per_base_sequence_content.tsv
final/DA_1164_03/qc/fastqc/Per_sequence_GC_content.tsv
final/DA_1164_03/qc/fastqc/Per_base_N_content.tsv
final/DA_1164_03/qc/fastqc/Sequence_Length_Distribution.tsv
final/DA_1164_03/DA_1164_03-ready.trimming_stats
final/DA_1164_02
final/DA_1164_02/qc
final/DA_1164_02/qc/fastqc
final/DA_1164_02/qc/fastqc/fastqc_report.html
final/DA_1164_02/qc/fastqc/fastqc_data.txt
final/DA_1164_02/qc/fastqc/DA_1164_02.zip
final/DA_1164_02/qc/fastqc/Per_base_sequence_quality.tsv
final/DA_1164_02/qc/fastqc/Per_tile_sequence_quality.tsv
final/DA_1164_02/qc/fastqc/Per_sequence_quality_scores.tsv
final/DA_1164_02/qc/fastqc/Per_base_sequence_content.tsv
final/DA_1164_02/qc/fastqc/Per_sequence_GC_content.tsv
final/DA_1164_02/qc/fastqc/Per_base_N_content.tsv
final/DA_1164_02/qc/fastqc/Sequence_Length_Distribution.tsv
final/DA_1164_02/DA_1164_02-ready.trimming_stats
final/DA_1164_10
final/DA_1164_10/qc
final/DA_1164_10/qc/fastqc
final/DA_1164_10/qc/fastqc/fastqc_report.html
final/DA_1164_10/qc/fastqc/fastqc_data.txt
final/DA_1164_10/qc/fastqc/DA_1164_10.zip
final/DA_1164_10/qc/fastqc/Per_base_sequence_quality.tsv
final/DA_1164_10/qc/fastqc/Per_tile_sequence_quality.tsv
final/DA_1164_10/qc/fastqc/Per_sequence_quality_scores.tsv
final/DA_1164_10/qc/fastqc/Per_base_sequence_content.tsv
final/DA_1164_10/qc/fastqc/Per_sequence_GC_content.tsv
final/DA_1164_10/qc/fastqc/Per_base_N_content.tsv
final/DA_1164_10/qc/fastqc/Sequence_Length_Distribution.tsv
final/DA_1164_10/DA_1164_10-ready.trimming_stats
final/2018-06-28_DA_1164_samples-merged
final/2018-06-28_DA_1164_samples-merged/programs.txt
final/2018-06-28_DA_1164_samples-merged/bcbio-nextgen.log
final/2018-06-28_DA_1164_samples-merged/bcbio-nextgen-commands.log
final/2018-06-28_DA_1164_samples-merged/project-summary.yaml
final/2018-06-28_DA_1164_samples-merged/report
final/2018-06-28_DA_1164_samples-merged/report/srna_report.rmd
final/2018-06-28_DA_1164_samples-merged/report/summary.csv
final/2018-06-28_DA_1164_samples-merged/multiqc
final/2018-06-28_DA_1164_samples-merged/multiqc/multiqc_report.html
final/2018-06-28_DA_1164_samples-merged/multiqc/report
final/2018-06-28_DA_1164_samples-merged/multiqc/report/metrics
final/2018-06-28_DA_1164_samples-merged/multiqc/report/metrics/DA_1164_08_bcbio.txt
final/2018-06-28_DA_1164_samples-merged/multiqc/report/metrics/DA_1164_04_bcbio.txt
final/2018-06-28_DA_1164_samples-merged/multiqc/report/metrics/DA_1164_07_bcbio.txt
final/2018-06-28_DA_1164_samples-merged/multiqc/report/metrics/DA_1164_06_bcbio.txt
final/2018-06-28_DA_1164_samples-merged/multiqc/report/metrics/DA_1164_02_bcbio.txt
final/2018-06-28_DA_1164_samples-merged/multiqc/report/metrics/DA_1164_05_bcbio.txt
final/2018-06-28_DA_1164_samples-merged/multiqc/report/metrics/DA_1164_10_bcbio.txt
final/2018-06-28_DA_1164_samples-merged/multiqc/report/metrics/DA_1164_01_bcbio.txt
final/2018-06-28_DA_1164_samples-merged/multiqc/report/metrics/DA_1164_09_bcbio.txt
final/2018-06-28_DA_1164_samples-merged/multiqc/report/metrics/DA_1164_03_bcbio.txt
final/2018-06-28_DA_1164_samples-merged/multiqc/multiqc_config.yaml
final/2018-06-28_DA_1164_samples-merged/multiqc/multiqc_data
final/2018-06-28_DA_1164_samples-merged/multiqc/multiqc_data/multiqc_data_final.json
final/2018-06-28_DA_1164_samples-merged/multiqc/list_files_final.txt
final/2018-06-28_DA_1164_samples-merged/seqcluster
final/2018-06-28_DA_1164_samples-merged/seqcluster/log
final/2018-06-28_DA_1164_samples-merged/seqcluster/log/run.log
final/2018-06-28_DA_1164_samples-merged/seqcluster/log/trace.log
final/2018-06-28_DA_1164_samples-merged/seqcluster/seqs_rmlw.bam_cov.tsv
final/2018-06-28_DA_1164_samples-merged/seqcluster/read_stats.tsv
final/2018-06-28_DA_1164_samples-merged/seqcluster/cluster.bed
final/2018-06-28_DA_1164_samples-merged/seqcluster/list_obj.pk
final/2018-06-28_DA_1164_samples-merged/seqcluster/list_obj_red.pk
final/2018-06-28_DA_1164_samples-merged/seqcluster/counts.tsv
final/2018-06-28_DA_1164_samples-merged/seqcluster/size_counts.tsv
final/2018-06-28_DA_1164_samples-merged/seqcluster/positions.bed
final/2018-06-28_DA_1164_samples-merged/seqcluster/counts_sequence.tsv
final/2018-06-28_DA_1164_samples-merged/seqcluster/seqcluster.json
final/2018-06-28_DA_1164_samples-merged/seqclusterViz
final/2018-06-28_DA_1164_samples-merged/seqclusterViz/log
final/2018-06-28_DA_1164_samples-merged/seqclusterViz/log/run.log
final/2018-06-28_DA_1164_samples-merged/seqclusterViz/log/trace.log
final/2018-06-28_DA_1164_samples-merged/seqclusterViz/profiles
final/2018-06-28_DA_1164_samples-merged/seqclusterViz/profiles/344
final/2018-06-28_DA_1164_samples-merged/seqclusterViz/profiles/5
final/2018-06-28_DA_1164_samples-merged/seqclusterViz/seqcluster.db

Hi,
Thanks for the questions.
Sadly, for non-model organism, the only analysis that will run is seqcluster that will generate small RNA loci over the genome and the expression of them (seqcluster/counts.tsv) that you can use with DESeq2.  As well you can visualize this data with: https://github.com/lpantano/seqclusterViz https://github.com/lpantano/seqclusterViz, you’ll need to download the repo, open the index.html file and load the seqclusterViz/seqcluster.db file.
Mirdeep won’t work on plants, for that we’ll need a plant prediction tool, I know of one but is not integrated, I can work on that, but it will take a month or so. When you got the trimming happens, maybe mirdeep runs successfully and you get some prediction, not sure if I’ll trust so much that.
To make the trimming happen you need to activate trim_reads : True under algorithm in the yaml file.
If you know there is a similar species in mirbase, let me know and I can help with set in up the files for that.
Thanks for trying bcbio, happy to help to get the trimming working and the annotation with mirbase if you know of a similar species.
Cheers

On Jun 29, 2018, at 12:41 AM, WimSpee<EMAIL_ADDRESS>wrote:
Hi,
Do you expect that the microRNAseq analysis capability provided by bcbio would make sense for analysis of microRNAseq data of non model organisms?
I am trying to see if I can process the Capsicum annuum microRNAseq data generated in this project using bcbio:
https://www.ncbi.nlm.nih.gov/bioproject/PRJNA177852 https://www.ncbi.nlm.nih.gov/bioproject/PRJNA177852
I am new to microRNAseq analysis so I am not really sure how to run this analysis and I am also not sure what kind of output I should expect.
The following is the yaml file that I am using:
upload:
dir: ../final
details:

analysis: smallRNA-seq
algorithm:
aligner: star # any other aligner is supported.
change adapter according project
adapters: ["TGGAATTCTCGGGTGC"]
expression_caller: [ seqcluster, mirdeep2]
expression_caller: [trna, seqcluster, mirdeep2, mirge] Read docs to know how to use
miRge tools: https://bcbio-nextgen.readthedocs.io/en/latest/contents/pipelines.html#smallrna-seq
species: hsa
genome_build: my_ref
#resources:

atropos:
options: ["-u 4", "-u -4"]
mirge:
options: ["-lib $PATH_TO_LIBS_FOLDER"]
This is the log file produced by the analysis.
[2018-06-27T18:55Z] grid_controller: System YAML configuration: /workspace/my_user/tmp_bcbio_1.1.0_development/data_dir/galaxy/bcbio_system.yaml
[2018-06-27T18:56Z] grid_controller: Timing: organize samples
[2018-06-27T18:56Z] grid_controller: ipython: organize_samples
[2018-06-27T18:56Z] exeuction_node_20: Using input YAML configuration: /leading_dir/config/DA_1164_samples-merged.
yaml
[2018-06-27T18:56Z] exeuction_node_20: Checking sample YAML configuration: /leading_dir/config/DA_1164_samples-mer
ged.yaml
[2018-06-27T18:56Z] exeuction_node_20: Testing minimum versions of installed programs
[2018-06-27T18:56Z] grid_controller: ipython: prepare_sample
[2018-06-27T18:56Z] grid_controller: Timing: adapter trimming
[2018-06-27T18:56Z] grid_controller: ipython: trim_srna_sample
[2018-06-27T19:26Z] exeuction_node_20: Collapsing /leading_dir/work/trimmed/DA_1164_01/DA_1164_01.clean.fastq.gz
with --min_size 16 --min 1
[2018-06-27T20:17Z] exeuction_node_20: Collapsing /leading_dir/work/trimmed/DA_1164_02/DA_1164_02.clean.fastq.gz
with --min_size 16 --min 1
[2018-06-27T21:00Z] exeuction_node_20: Collapsing /leading_dir/work/trimmed/DA_1164_03/DA_1164_03.clean.fastq.gz
with --min_size 16 --min 1
[2018-06-27T21:42Z] exeuction_node_20: Collapsing /leading_dir/work/trimmed/DA_1164_04/DA_1164_04.clean.fastq.gz
with --min_size 16 --min 1
[2018-06-27T22:20Z] exeuction_node_20: Collapsing /leading_dir/work/trimmed/DA_1164_05/DA_1164_05.clean.fastq.gz
with --min_size 16 --min 1
[2018-06-27T22:55Z] exeuction_node_20: Collapsing /leading_dir/work/trimmed/DA_1164_06/DA_1164_06.clean.fastq.gz
with --min_size 16 --min 1
[2018-06-27T23:31Z] exeuction_node_20: Collapsing /leading_dir/work/trimmed/DA_1164_07/DA_1164_07.clean.fastq.gz
with --min_size 16 --min 1
[2018-06-28T00:28Z] exeuction_node_20: Collapsing /leading_dir/work/trimmed/DA_1164_08/DA_1164_08.clean.fastq.gz
with --min_size 16 --min 1
[2018-06-28T01:11Z] exeuction_node_20: Collapsing /leading_dir/work/trimmed/DA_1164_09/DA_1164_09.clean.fastq.gz
with --min_size 16 --min 1
[2018-06-28T01:54Z] exeuction_node_20: Collapsing /leading_dir/work/trimmed/DA_1164_10/DA_1164_10.clean.fastq.gz
with --min_size 16 --min 1
[2018-06-28T02:13Z] grid_controller: Timing: prepare
[2018-06-28T02:13Z] grid_controller: ipython: seqcluster_prepare
[2018-06-28T03:05Z] exeuction_node_24: Prepare seqs.fastq with -minl 17 -maxl 40 -minc 2 --min_shared 0.1
[2018-06-28T03:08Z] grid_controller: Timing: alignment
[2018-06-28T03:08Z] grid_controller: ipython: srna_alignment
[2018-06-28T03:08Z] exeuction_node_24: Aligning lane DA_1164_01 with star aligner
[2018-06-28T03:11Z] exeuction_node_24: mirdeep2 Rfam file not instaled. Skipping...
[2018-06-28T03:11Z] grid_controller: Timing: small RNA annotation
[2018-06-28T03:11Z] grid_controller: ipython: srna_annotation
[2018-06-28T03:12Z] grid_controller: Timing: cluster
[2018-06-28T03:12Z] grid_controller: ipython: seqcluster_cluster
[2018-06-28T04:59Z] grid_controller: Timing: quality control
[2018-06-28T04:59Z] grid_controller: ipython: pipeline_summary
[2018-06-28T04:59Z] exeuction_node_20: QC: DA_1164_01 fastqc
I am not sure how to specify that dnapi should be run for de-novo adapter detection followed by adapter trimming. As far as I can tell dnapi was not used for adapter trimming. The fastqc part of the multiqQC report shows that of the 50bp reads the last 25 bp is almost 100% adapters.
As far as I can tell Capsicum annuum is not in mirbase. Therefore I did not enter a 3 letter species code. I am not sure if it makes sense to just enter the species code of a somewhat related species
http://www.mirbase.org/cgi-bin/mirna_summary.pl?org=sly http://www.mirbase.org/cgi-bin/mirna_summary.pl?org=sly
Or that I better just don't provide a species code.
The analysis did not seem to produce much results. See the file list at the bottom of this comment. Then again I am also not sure what to expect.
The lack of output might in part be because mirdeep2 Rfam not being installed/found. Should I have done that myself?
[2018-06-28T03:08Z] exeuction_node_24: Aligning lane DA_1164_01 with star aligner
[2018-06-28T03:11Z] exeuction_node_24: mirdeep2 Rfam file not instaled. Skipping...
What I kind of expect as output for an microRNAseq analysis is:
identification/ filtering of known/discovered non microRNA sequences (either biological (e.g. other RNA's) or adapters)
identification of know mircroRNA sequences from mirbase or similar
per sample alignment BAM files of the microRNA sequences (not sure if this should run against the genome or transcriptome (or both). And I am not sure if these alignments identify target loci/mRNAs or microRNA precursur loci/mRNAs (or both))
microRNA target mRNA/gene prediction
microRNA quantification
Do you think it is possible to get the above results using bcbio for microRNAseq data of a non model organism?
How would I then do that using bcbio? Is the yaml that I use correct? Should I add tRNA as an expression caller?
Since I am new to microRNAseq the bcbio microRNAseq documentation is also a bit short me.
I would also very much appreciate it if you can point to me a recent sort of best practice method / review paper that describes the method(s) that bcbio in general tries to provide for microRNAseq analysis.
Thank you very much!
final/
final/DA_1164_05
final/DA_1164_05/qc
final/DA_1164_05/qc/fastqc
final/DA_1164_05/qc/fastqc/fastqc_report.html
final/DA_1164_05/qc/fastqc/fastqc_data.txt
final/DA_1164_05/qc/fastqc/DA_1164_05.zip
final/DA_1164_05/qc/fastqc/Per_base_sequence_quality.tsv
final/DA_1164_05/qc/fastqc/Per_tile_sequence_quality.tsv
final/DA_1164_05/qc/fastqc/Per_sequence_quality_scores.tsv
final/DA_1164_05/qc/fastqc/Per_base_sequence_content.tsv
final/DA_1164_05/qc/fastqc/Per_sequence_GC_content.tsv
final/DA_1164_05/qc/fastqc/Per_base_N_content.tsv
final/DA_1164_05/qc/fastqc/Sequence_Length_Distribution.tsv
final/DA_1164_05/DA_1164_05-ready.trimming_stats
final/DA_1164_04
final/DA_1164_04/qc
final/DA_1164_04/qc/fastqc
final/DA_1164_04/qc/fastqc/fastqc_report.html
final/DA_1164_04/qc/fastqc/fastqc_data.txt
final/DA_1164_04/qc/fastqc/DA_1164_04.zip
final/DA_1164_04/qc/fastqc/Per_base_sequence_quality.tsv
final/DA_1164_04/qc/fastqc/Per_tile_sequence_quality.tsv
final/DA_1164_04/qc/fastqc/Per_sequence_quality_scores.tsv
final/DA_1164_04/qc/fastqc/Per_base_sequence_content.tsv
final/DA_1164_04/qc/fastqc/Per_sequence_GC_content.tsv
final/DA_1164_04/qc/fastqc/Per_base_N_content.tsv
final/DA_1164_04/qc/fastqc/Sequence_Length_Distribution.tsv
final/DA_1164_04/DA_1164_04-ready.trimming_stats
final/DA_1164_09
final/DA_1164_09/qc
final/DA_1164_09/qc/fastqc
final/DA_1164_09/qc/fastqc/fastqc_report.html
final/DA_1164_09/qc/fastqc/fastqc_data.txt
final/DA_1164_09/qc/fastqc/DA_1164_09.zip
final/DA_1164_09/qc/fastqc/Per_base_sequence_quality.tsv
final/DA_1164_09/qc/fastqc/Per_tile_sequence_quality.tsv
final/DA_1164_09/qc/fastqc/Per_sequence_quality_scores.tsv
final/DA_1164_09/qc/fastqc/Per_base_sequence_content.tsv
final/DA_1164_09/qc/fastqc/Per_sequence_GC_content.tsv
final/DA_1164_09/qc/fastqc/Per_base_N_content.tsv
final/DA_1164_09/qc/fastqc/Sequence_Length_Distribution.tsv
final/DA_1164_09/DA_1164_09-ready.trimming_stats
final/DA_1164_08
final/DA_1164_08/qc
final/DA_1164_08/qc/fastqc
final/DA_1164_08/qc/fastqc/fastqc_report.html
final/DA_1164_08/qc/fastqc/fastqc_data.txt
final/DA_1164_08/qc/fastqc/DA_1164_08.zip
final/DA_1164_08/qc/fastqc/Per_base_sequence_quality.tsv
final/DA_1164_08/qc/fastqc/Per_tile_sequence_quality.tsv
final/DA_1164_08/qc/fastqc/Per_sequence_quality_scores.tsv
final/DA_1164_08/qc/fastqc/Per_base_sequence_content.tsv
final/DA_1164_08/qc/fastqc/Per_sequence_GC_content.tsv
final/DA_1164_08/qc/fastqc/Per_base_N_content.tsv
final/DA_1164_08/qc/fastqc/Sequence_Length_Distribution.tsv
final/DA_1164_08/DA_1164_08-ready.trimming_stats
final/DA_1164_07
final/DA_1164_07/qc
final/DA_1164_07/qc/fastqc
final/DA_1164_07/qc/fastqc/fastqc_report.html
final/DA_1164_07/qc/fastqc/fastqc_data.txt
final/DA_1164_07/qc/fastqc/DA_1164_07.zip
final/DA_1164_07/qc/fastqc/Per_base_sequence_quality.tsv
final/DA_1164_07/qc/fastqc/Per_tile_sequence_quality.tsv
final/DA_1164_07/qc/fastqc/Per_sequence_quality_scores.tsv
final/DA_1164_07/qc/fastqc/Per_base_sequence_content.tsv
final/DA_1164_07/qc/fastqc/Per_sequence_GC_content.tsv
final/DA_1164_07/qc/fastqc/Per_base_N_content.tsv
final/DA_1164_07/qc/fastqc/Sequence_Length_Distribution.tsv
final/DA_1164_07/DA_1164_07-ready.trimming_stats
final/DA_1164_06
final/DA_1164_06/qc
final/DA_1164_06/qc/fastqc
final/DA_1164_06/qc/fastqc/fastqc_report.html
final/DA_1164_06/qc/fastqc/fastqc_data.txt
final/DA_1164_06/qc/fastqc/DA_1164_06.zip
final/DA_1164_06/qc/fastqc/Per_base_sequence_quality.tsv
final/DA_1164_06/qc/fastqc/Per_tile_sequence_quality.tsv
final/DA_1164_06/qc/fastqc/Per_sequence_quality_scores.tsv
final/DA_1164_06/qc/fastqc/Per_base_sequence_content.tsv
final/DA_1164_06/qc/fastqc/Per_sequence_GC_content.tsv
final/DA_1164_06/qc/fastqc/Per_base_N_content.tsv
final/DA_1164_06/qc/fastqc/Sequence_Length_Distribution.tsv
final/DA_1164_06/DA_1164_06-ready.trimming_stats
final/DA_1164_01
final/DA_1164_01/qc
final/DA_1164_01/qc/fastqc
final/DA_1164_01/qc/fastqc/fastqc_report.html
final/DA_1164_01/qc/fastqc/fastqc_data.txt
final/DA_1164_01/qc/fastqc/DA_1164_01.zip
final/DA_1164_01/qc/fastqc/Per_base_sequence_quality.tsv
final/DA_1164_01/qc/fastqc/Per_tile_sequence_quality.tsv
final/DA_1164_01/qc/fastqc/Per_sequence_quality_scores.tsv
final/DA_1164_01/qc/fastqc/Per_base_sequence_content.tsv
final/DA_1164_01/qc/fastqc/Per_sequence_GC_content.tsv
final/DA_1164_01/qc/fastqc/Per_base_N_content.tsv
final/DA_1164_01/qc/fastqc/Sequence_Length_Distribution.tsv
final/DA_1164_01/qc/small-rna
final/DA_1164_01/qc/small-rna/DA_1164_01.txt
final/DA_1164_01/DA_1164_01-ready.trimming_stats
final/DA_1164_03
final/DA_1164_03/qc
final/DA_1164_03/qc/fastqc
final/DA_1164_03/qc/fastqc/fastqc_report.html
final/DA_1164_03/qc/fastqc/fastqc_data.txt
final/DA_1164_03/qc/fastqc/DA_1164_03.zip
final/DA_1164_03/qc/fastqc/Per_base_sequence_quality.tsv
final/DA_1164_03/qc/fastqc/Per_tile_sequence_quality.tsv
final/DA_1164_03/qc/fastqc/Per_sequence_quality_scores.tsv
final/DA_1164_03/qc/fastqc/Per_base_sequence_content.tsv
final/DA_1164_03/qc/fastqc/Per_sequence_GC_content.tsv
final/DA_1164_03/qc/fastqc/Per_base_N_content.tsv
final/DA_1164_03/qc/fastqc/Sequence_Length_Distribution.tsv
final/DA_1164_03/DA_1164_03-ready.trimming_stats
final/DA_1164_02
final/DA_1164_02/qc
final/DA_1164_02/qc/fastqc
final/DA_1164_02/qc/fastqc/fastqc_report.html
final/DA_1164_02/qc/fastqc/fastqc_data.txt
final/DA_1164_02/qc/fastqc/DA_1164_02.zip
final/DA_1164_02/qc/fastqc/Per_base_sequence_quality.tsv
final/DA_1164_02/qc/fastqc/Per_tile_sequence_quality.tsv
final/DA_1164_02/qc/fastqc/Per_sequence_quality_scores.tsv
final/DA_1164_02/qc/fastqc/Per_base_sequence_content.tsv
final/DA_1164_02/qc/fastqc/Per_sequence_GC_content.tsv
final/DA_1164_02/qc/fastqc/Per_base_N_content.tsv
final/DA_1164_02/qc/fastqc/Sequence_Length_Distribution.tsv
final/DA_1164_02/DA_1164_02-ready.trimming_stats
final/DA_1164_10
final/DA_1164_10/qc
final/DA_1164_10/qc/fastqc
final/DA_1164_10/qc/fastqc/fastqc_report.html
final/DA_1164_10/qc/fastqc/fastqc_data.txt
final/DA_1164_10/qc/fastqc/DA_1164_10.zip
final/DA_1164_10/qc/fastqc/Per_base_sequence_quality.tsv
final/DA_1164_10/qc/fastqc/Per_tile_sequence_quality.tsv
final/DA_1164_10/qc/fastqc/Per_sequence_quality_scores.tsv
final/DA_1164_10/qc/fastqc/Per_base_sequence_content.tsv
final/DA_1164_10/qc/fastqc/Per_sequence_GC_content.tsv
final/DA_1164_10/qc/fastqc/Per_base_N_content.tsv
final/DA_1164_10/qc/fastqc/Sequence_Length_Distribution.tsv
final/DA_1164_10/DA_1164_10-ready.trimming_stats
final/2018-06-28_DA_1164_samples-merged
final/2018-06-28_DA_1164_samples-merged/programs.txt
final/2018-06-28_DA_1164_samples-merged/bcbio-nextgen.log
final/2018-06-28_DA_1164_samples-merged/bcbio-nextgen-commands.log
final/2018-06-28_DA_1164_samples-merged/project-summary.yaml
final/2018-06-28_DA_1164_samples-merged/report
final/2018-06-28_DA_1164_samples-merged/report/srna_report.rmd
final/2018-06-28_DA_1164_samples-merged/report/summary.csv
final/2018-06-28_DA_1164_samples-merged/multiqc
final/2018-06-28_DA_1164_samples-merged/multiqc/multiqc_report.html
final/2018-06-28_DA_1164_samples-merged/multiqc/report
final/2018-06-28_DA_1164_samples-merged/multiqc/report/metrics
final/2018-06-28_DA_1164_samples-merged/multiqc/report/metrics/DA_1164_08_bcbio.txt
final/2018-06-28_DA_1164_samples-merged/multiqc/report/metrics/DA_1164_04_bcbio.txt
final/2018-06-28_DA_1164_samples-merged/multiqc/report/metrics/DA_1164_07_bcbio.txt
final/2018-06-28_DA_1164_samples-merged/multiqc/report/metrics/DA_1164_06_bcbio.txt
final/2018-06-28_DA_1164_samples-merged/multiqc/report/metrics/DA_1164_02_bcbio.txt
final/2018-06-28_DA_1164_samples-merged/multiqc/report/metrics/DA_1164_05_bcbio.txt
final/2018-06-28_DA_1164_samples-merged/multiqc/report/metrics/DA_1164_10_bcbio.txt
final/2018-06-28_DA_1164_samples-merged/multiqc/report/metrics/DA_1164_01_bcbio.txt
final/2018-06-28_DA_1164_samples-merged/multiqc/report/metrics/DA_1164_09_bcbio.txt
final/2018-06-28_DA_1164_samples-merged/multiqc/report/metrics/DA_1164_03_bcbio.txt
final/2018-06-28_DA_1164_samples-merged/multiqc/multiqc_config.yaml
final/2018-06-28_DA_1164_samples-merged/multiqc/multiqc_data
final/2018-06-28_DA_1164_samples-merged/multiqc/multiqc_data/multiqc_data_final.json
final/2018-06-28_DA_1164_samples-merged/multiqc/list_files_final.txt
final/2018-06-28_DA_1164_samples-merged/seqcluster
final/2018-06-28_DA_1164_samples-merged/seqcluster/log
final/2018-06-28_DA_1164_samples-merged/seqcluster/log/run.log
final/2018-06-28_DA_1164_samples-merged/seqcluster/log/trace.log
final/2018-06-28_DA_1164_samples-merged/seqcluster/seqs_rmlw.bam_cov.tsv
final/2018-06-28_DA_1164_samples-merged/seqcluster/read_stats.tsv
final/2018-06-28_DA_1164_samples-merged/seqcluster/cluster.bed
final/2018-06-28_DA_1164_samples-merged/seqcluster/list_obj.pk
final/2018-06-28_DA_1164_samples-merged/seqcluster/list_obj_red.pk
final/2018-06-28_DA_1164_samples-merged/seqcluster/counts.tsv
final/2018-06-28_DA_1164_samples-merged/seqcluster/size_counts.tsv
final/2018-06-28_DA_1164_samples-merged/seqcluster/positions.bed
final/2018-06-28_DA_1164_samples-merged/seqcluster/counts_sequence.tsv
final/2018-06-28_DA_1164_samples-merged/seqcluster/seqcluster.json
final/2018-06-28_DA_1164_samples-merged/seqclusterViz
final/2018-06-28_DA_1164_samples-merged/seqclusterViz/log
final/2018-06-28_DA_1164_samples-merged/seqclusterViz/log/run.log
final/2018-06-28_DA_1164_samples-merged/seqclusterViz/log/trace.log
final/2018-06-28_DA_1164_samples-merged/seqclusterViz/profiles
final/2018-06-28_DA_1164_samples-merged/seqclusterViz/profiles/344
final/2018-06-28_DA_1164_samples-merged/seqclusterViz/profiles/5
final/2018-06-28_DA_1164_samples-merged/seqclusterViz/seqcluster.db
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub https://github.com/bcbio/bcbio-nextgen/issues/2427, or mute the thread https://github.com/notifications/unsubscribe-auth/ABi_HPT3EnjSkz6UUAtjeWebu5pIAmf-ks5uBdodgaJpZM4U8lIW.

Hi,
Yes, those are godo examples.
I would be happy to add this to bcbio. What it would help is if you use some of these tools and tell us the command line you use. We try to implement tools that we can test with some data and we know kind of give good results. If you do that and find some, I’ll be happy to add it, for sure.
About using this other species with bcbio, that should work. If you have installed the hg19 and mm10 genome you could generate the same files and put  it in the right location.
I am assuming you have the genome set up, and you can locate genome_name/build_name/seq folder used by bcbio when you ran the small RNA seq pipeline.
You need to add the following to the genome-resources.yaml file:
srnaseq:
srna_transcripts: ../srnaseq/srna-transcripts.gtf
mirbase_hairpin: ../srnaseq/hairpin.fa
mirbase_mature: ../srnaseq/mature.fa
In the srnaseq folder, that is at the same level than the seq folder,  you need these files:
From mirbase you can download the mature.fa and hairpin.fa and the miRNA.str (ftp://mirbase.org/pub/mirbase/CURRENT/ ftp://mirbase.org/pub/mirbase/CURRENT/)
You’ll need to prepare you’ll files like this:
zcat hairpin.fa.gz |  awk '{if ($0~/>sly/){name=$0; print name} else if ($0~/^>/){name=0};if (name!=0 && $0!~/^>/){print $0;}}' | sed 's/U/T/g'  > hairpin.fa
zcat mature.fa.gz |  awk '{if ($0~/>sly/){name=$0; print name} else if ($0~/^>/){name=0};if (name!=0 && $0!~/^>/){print $0;}}' | sed 's/U/T/g'  > mature.fa
zcat miRNA.str.gz | awk '{if ($0~/sly/)print $0}' > miRNA.str
You can use a custom gtf from your species, so if you have a genome ensemble and there is a gene annotation (ftp://ftp.ensemblgenomes.org/pub/release-39/plants/gtf/solanum_lycopersicum), you can put that one there as the srna-transcripts.gtf. That file is used by seqcluster to annotate the clusters found. I think it should work from the ensembl database if you are using that genome.
If you get the mirna part and the cluster working you can use this package to load all the data with this package:
https://lpantano.github.io/bcbioSmallRna/reference/loadSmallRnaRun.html
Beside you have this kind of template for a quick QC analysis and how to get the count data and annotation:
https://github.com/lpantano/bcbioSmallRna/blob/master/inst/rmarkdown/templates/srnaseq/skeleton/skeleton.Rmd https://github.com/lpantano/bcbioSmallRna/blob/master/inst/rmarkdown/templates/srnaseq/skeleton/skeleton.Rmd
I am working on this currently, so It is a good time to start using it.
I hope this helps.
Cheers

On Jul 4, 2018, at 10:25 AM, WimSpee<EMAIL_ADDRESS>wrote:
Hi Lorena Pantano.
Thank you for the information. I did not know that plant specific tools were needed. Do you mean any of these two tools?
miRPlant: https://bmcbioinformatics.biomedcentral.com/articles/10.1186/1471-2105-15-275 https://bmcbioinformatics.biomedcentral.com/articles/10.1186/1471-2105-15-275
miRDeep-P: https://academic.oup.com/bioinformatics/article/27/18/2614/181153 https://academic.oup.com/bioinformatics/article/27/18/2614/181153
The first paper mentions that different tools are needed because of that the miRNA precursors are different / longer in plants than in animals.
The most challenging problem in identifying novel plant miRNA is to find a suitable genomic region as a miRNA precursor candidate (to test whether it forms hairpins) because the majority of precursor miRNA in plants are between 100-200 bp [4], which is much longer than those in animals.
Do you know if there are other reasons plant specific miRNA tools are needed?
I will try to use / look at the seqcluster results.
I will try with trim_reads : True .
Also I will try the analysis with Solanum lycopersicum (mirbase SLY) as the known miRNA data set. That species is some what close (also in the nightshade family), and the miRNA seqeunces are conserved in plants according to one of the above papers.
http://www.mirbase.org/cgi-bin/mirna_summary.pl?org=sly http://www.mirbase.org/cgi-bin/mirna_summary.pl?org=sly
Do I need to do anything to make use of the the SLY miRNA known sequences?
Do you know how and by who the sequences for a species get added in mirbase?
It would be nice if the microRNA seq functionality of bcbio works for plants. I kind of hoped it would / did not expect a need plant specific tools. At the same time I understand your primary focus is on other species, thus it would only make sense to me if it's not to much work or we could do part of it.
Thank again for the information.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub https://github.com/bcbio/bcbio-nextgen/issues/2427#issuecomment-402494179, or mute the thread https://github.com/notifications/unsubscribe-auth/ABi_HLRgQ3rv51mlkTjyxlXivd10lzhzks5uDNA_gaJpZM4U8lIW.

Hi @lpantano  . I tried to run the same analysis with trim_reads : true and species: sly.
This resulted in the following error during adapter removal:
[2018-07-10T16:51Z] execution_machine_x2: 2018-07-10 18:51:49,585 INFO: This is Atropos 1.1.18 with Python 3.6.5
[2018-07-10T16:51Z] execution_machine_x2: 2018-07-10 18:51:49,590 INFO: Trimming 0 adapter with at most 10.0% errors in single-end mode ...
[2018-07-10T16:52Z] execution_machine_x2: =======
[2018-07-10T16:52Z] execution_machine_x2: Atropos
[2018-07-10T16:52Z] execution_machine_x2: =======
[2018-07-10T16:52Z] execution_machine_x2: Atropos version: 1.1.18
[2018-07-10T16:52Z] execution_machine_x2: Python version: 3.6.5
[2018-07-10T16:52Z] execution_machine_x2: Command line parameters: trim --max-reads 500000 -u 22 -se /data/run/Projects/DA-1164/input_fastq/concat/DA_1164_10.fastq.gz -o /data/run/Projects/DA-1164/DA_1164_samples-merged/work/bcbiotx/tmpHUH_oP/DA_1164_10end.fastq.gz
[2018-07-10T16:52Z] execution_machine_x2: Sample ID: DA_1164_10
[2018-07-10T16:52Z] execution_machine_x2: Input format: FASTQ, Read 1, w/ Qualities
[2018-07-10T16:52Z] execution_machine_x2: Input files:
[2018-07-10T16:52Z] execution_machine_x2:   /data/run/Projects/DA-1164/input_fastq/concat/DA_1164_10.fastq.gz
[2018-07-10T16:52Z] execution_machine_x2: Start time: 2018-07-10T18:51:49.589469
[2018-07-10T16:52Z] execution_machine_x2: Wallclock time: 15.54 s (31 us/read; 1.93 M reads/minute)
[2018-07-10T16:52Z] execution_machine_x2: CPU time (main process): 11.20 s
[2018-07-10T16:52Z] execution_machine_x2: --------
[2018-07-10T16:52Z] execution_machine_x2: Trimming
[2018-07-10T16:52Z] execution_machine_x2: --------
[2018-07-10T16:52Z] execution_machine_x2: Reads                                  records   fraction
[2018-07-10T16:52Z] execution_machine_x2: ----------------------------------- ---------- ----------
[2018-07-10T16:52Z] execution_machine_x2: Total reads processed:                 500,000
[2018-07-10T16:52Z] execution_machine_x2: Reads written (passing filters):       500,000     100.0%
[2018-07-10T16:52Z] execution_machine_x2: Base pairs                                  bp   fraction
[2018-07-10T16:52Z] execution_machine_x2: ----------------------------------- ---------- ----------
[2018-07-10T16:52Z] execution_machine_x2: Total bp processed:                 25,500,000
[2018-07-10T16:52Z] execution_machine_x2: Cut unconditionally                 11,000,000      43.1%
[2018-07-10T16:52Z] execution_machine_x2: Total bp written (filtered):        14,500,000      56.9%
[2018-07-10T16:52Z] execution_machine_x2: Unexpected error
Traceback (most recent call last):
  File "/home/my_user/workspace/tmp_bcbio_1.1.0_development/data_dir/anaconda/lib/python2.7/site-packages/bcbio/distributed/ipythontasks.py", line 51, in _setup_logging
    yield config
  File "/home/my_user/workspace/tmp_bcbio_1.1.0_development/data_dir/anaconda/lib/python2.7/site-packages/bcbio/distributed/ipythontasks.py", line 92, in trim_srna_sample
    return ipython.zip_args(apply(srna.trim_srna_sample, *args))
  File "/home/my_user/workspace/tmp_bcbio_1.1.0_development/data_dir/anaconda/lib/python2.7/site-packages/bcbio/srna/sample.py", line 61, in trim_srna_sample
    adapters = adapter if adapter else _dnapi_prediction(in_file, out_dir)
  File "/home/my_user/workspace/tmp_bcbio_1.1.0_development/data_dir/anaconda/lib/python2.7/site-packages/bcbio/srna/sample.py", line 157, in _dnapi_prediction
    max_score = iterative_result[1][1]
IndexError: list index out of range

This seems to be sample specific. Other samples seem not to run into this error.
[2018-07-10T12:54Z] execution_machine_x2: 2018-07-10 14:54:13,644 INFO: This is Atropos 1.1.18 with Python 3.6.5
[2018-07-10T12:54Z] execution_machine_x2: 2018-07-10 14:54:13,655 INFO: Trimming 0 adapter with at most 10.0% errors in single-end mode ...
[2018-07-10T12:54Z] execution_machine_x2: =======
[2018-07-10T12:54Z] execution_machine_x2: Atropos
[2018-07-10T12:54Z] execution_machine_x2: =======
[2018-07-10T12:54Z] execution_machine_x2: Atropos version: 1.1.18
[2018-07-10T12:54Z] execution_machine_x2: Python version: 3.6.5
[2018-07-10T12:54Z] execution_machine_x2: Command line parameters: trim --max-reads 500000 -u 22 -se /data/run/Projects/DA-1164/input_fastq/
concat/DA_1164_01.fastq.gz -o /data/run/Projects/DA-1164/DA_1164_samples-merged/work/bcbiotx/tmpEtG7q8/DA_1164_01end.fastq.gz
[2018-07-10T12:54Z] execution_machine_x2: Sample ID: DA_1164_01
[2018-07-10T12:54Z] execution_machine_x2: Input format: FASTQ, Read 1, w/ Qualities
[2018-07-10T12:54Z] execution_machine_x2: Input files:
[2018-07-10T12:54Z] execution_machine_x2:   /data/run/Projects/DA-1164/input_fastq/concat/DA_1164_01.fastq.gz
[2018-07-10T12:54Z] execution_machine_x2: Start time: 2018-07-10T14:54:13.654689
[2018-07-10T12:54Z] execution_machine_x2: Wallclock time: 15.33 s (31 us/read; 1.96 M reads/minute)
[2018-07-10T12:54Z] execution_machine_x2: CPU time (main process): 11.04 s
[2018-07-10T12:54Z] execution_machine_x2: --------
[2018-07-10T12:54Z] execution_machine_x2: Trimming
[2018-07-10T12:54Z] execution_machine_x2: --------
[2018-07-10T12:54Z] execution_machine_x2: Reads                                  records   fraction
[2018-07-10T12:54Z] execution_machine_x2: ----------------------------------- ---------- ----------
[2018-07-10T12:54Z] execution_machine_x2: Total reads processed:                 500,000
[2018-07-10T12:54Z] execution_machine_x2: Reads written (passing filters):       500,000     100.0%
[2018-07-10T12:54Z] execution_machine_x2: Base pairs                                  bp   fraction
[2018-07-10T12:54Z] execution_machine_x2: ----------------------------------- ---------- ----------
[2018-07-10T12:54Z] execution_machine_x2: Total bp processed:                 25,500,000
[2018-07-10T12:54Z] execution_machine_x2: Cut unconditionally                 11,000,000      43.1%
[2018-07-10T12:54Z] execution_machine_x2: Total bp written (filtered):        14,500,000      56.9%
[2018-07-10T12:54Z] execution_machine_x2: Adding adapter to the list: TGGAATTCTCGGG with score 282.8354
[2018-07-10T12:54Z] execution_machine_x2: Adding adapter to the list: GGTGCCAAGGAA with score 78.588
[2018-07-10T12:54Z] execution_machine_x2: remove adapter for DA_1164_01
[2018-07-10T14:02Z] execution_machine_x2: Collapsing /data/run/Projects/DA-1164/DA_1164_samples-merged/work/trimmed/DA_1164_01/DA_1164
_01.clean.fastq.gz with --min_size 16 --min 1

For this sample I am also not sure why Atropos is run before _dnapi_prediction and where  Atropos  get's the TGGAATTCTCGGG  and GGTGCCAAGGAA  adapters from.

Hi,
sorry about this. It seems that the tool we used to predict the adapter is not working there. If you know the 3' adapter, I'll suggest to add the adapter to the adapters: [] to the config file:
https://github.com/bcbio/bcbio-nextgen/blob/master/config/templates/illumina-srnaseq.yaml#L8
If you don't know you can ask the sequencing core for that.
In this case, I'll suggest to start from scratch the analysis.
Let me know if that helps.

Thanks, closing this as it seems like its been answered.


Question: using undefined base object

I'm trying to use immer.js to set a property in an object which is possible undefined - as are it's children. Can I ensure the object is initialized somehow?
type Config = {
  personal?: {
    navigate?: boolean;
  }
};

const myFunc = (configuration?: Config): void => {
  const mergedConfig: Config = produce(configuration, (draft): void => {
    draft.personal.navigate = true;
  });
};

Using this code, I receive a typescript error saying Object 'draft' is possibly 'undefined'.
Anyway I can solve this?
Thanks.

If you search a bit in the closed issues, I'm pretty confident you'll find
a duplicate. Let me know if you don't.
Op vr 21 jun. 2019 11:09 schreef Nick Sevens<EMAIL_ADDRESS>
I'm trying to use immer.js to set a property in an object which is
possible undefined - as are it's children. Can I ensure the object is
initialized somehow?
type Config = {
personal?: {
navigate?: boolean;
}
};
const myFunc = (configuration?: Config): void => {
const mergedConfig: Config = produce(configuration, (draft): void => {
draft.personal.navigate = true;
});
};
Using this code, I receive a typescript error saying Object 'draft' is
possibly 'undefined'.
Anyway I can solve this?
Thanks.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/immerjs/immer/issues/390?email_source=notifications&email_token=AAN4NBDKA5MPLHE63YHXGX3P3SLDZA5CNFSM4H2PFNCKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4G25HLEA,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAN4NBCOFDX55OM52ZODTVTP3SLDZANCNFSM4H2PFNCA
.

I've searched quite a bit but couldn't find a solution for me. If you could point me in the right direction would be much appreciated!

See https://github.com/immerjs/immer/issues/292
P.S. optional chaining moved to stage3 last week!

This is not Immer's problem. You just need to learn more TypeScript.
-const myFunc = (configuration?: Config): void => {
+const myFunc = (configuration: Config = {}): void => {
   const mergedConfig: Config = produce(configuration, (draft): void => {
     draft.personal.navigate = true;
   });
 };

Duplicate of #337, but that issue never had a proper solution given.


[CircleCI] Optimize CI Tests (6X speed up on test suite)

Motivation and Context

This speed up the CI by: Making it More Parallel
Removing typeguard which gives a 50X speedup on some tests.
Fix a bug the CircleCI cache loading (will now build a lot faster for subsequent builds on PR branches).
Total testing cut by more than half! Build times from 4 minutes -> 2 minutes on CI Cache. PyTest suite speed up from 30 minutes to just 5 minutes!

CI Tests were very slow (taking 30 minutes to run the whole suite)

How Has This Been Tested
CI

Types of changes

[X] Docs change / refactoring / dependency upgrade
[X] Bug fix (non-breaking change which fixes an issue)

Checklist

[X] My code follows the code style of this project.
[X] I have read the CONTRIBUTING document.
[X] I have completed my CLA (see CONTRIBUTING)
[X] I have added tests to cover my changes.
[X] All new and existing tests passed.

@mathfac Next PR should be to move CONDA building to its own process workflow. If we do that, we can get the whole CI Build down to half an hour.


chore(deps): upgrade jenkins-x/jx3-pipeline-catalog to version 3.10.116

from: https://github.com/jenkins-x/jx

/retest


Update dependency: deps/k_release

Outputs of the integration tests test/imp/max-symbolic.imp.out.golden and test/smt-none/program.test.out.golden neede to be updated because of the commit 8200fdb41f268a875044c18dda5dd206e5435d16 to K


[Test] Add unit test for virtualmachine mutator

Is your feature request related to a problem? Please describe.

Add unit test cases to test patchAffinity in virtualmachine mutator.
https://github.com/harvester/harvester/pull/3000#discussion_r1001552711
Describe the solution you'd like

Describe alternatives you've considered

Additional context

Need to backport to v1.1 branch as well.

Close as the unit test has been added to both master https://github.com/harvester/harvester/pull/3405 and the v1.1 branch.


Unable to serialize value

When testing the tutorial on this  page I encountered the following error:
Detecting the anomaly status of the latest data point.
Unable to serialize value: Timestamp('2018-03-01 00:00:00+0000', tz='UTC') as type: 'iso-8601'., TypeError: an integer is required

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 884b5e88-48eb-c3ad-1995-7cd87339783b
Version Independent ID: e06cfd58-46d0-200a-5967-c6a1a7276997
Content: Quickstart: Anomaly detection using the Anomaly Detector client library - Azure Cognitive Services
Content Source: articles/cognitive-services/Anomaly-Detector/quickstarts/client-libraries.md
Service: cognitive-services
Sub-service: anomaly-detector
GitHub Login: @mrbullwinkle
Microsoft Alias: mbullwin

@Aoloa-dev Thanks for the question. We are investigating the issue and will update you shortly.

@Aoloa-dev Thanks, Play with your data using the online interactive demo :https://algoevaluation.azurewebsites.net/#/

Why was this issue closed? Same error still happening 6 months later...

This is addressed in this PR: https://github.com/MicrosoftDocs/azure-docs-pr/pull/201367 The change is merged and will go-live during the next publishing run.


undefined method `[]' for nil:NilClass during omniauth callback

I think there's some kind of problem with this line of code in omniauth_callbacks_controller (line 14):
...

  devise_mapping = [request.env['omniauth.params']['namespace_name'],
                    request.env['omniauth.params']['resource_class'].underscore.gsub('/', '_')].compact.join('_')
...

After successfully being prompted by facebook, the 'omniauth' object seems to be empty raising the following error:
undefined method `[]' for nil:NilClass

C:/RailsInstaller/Ruby2.2.0/lib/ruby/gems/2.2.0/bundler/gems/devise_token_auth-162967970c95/app/controllers/devise_token_auth/omniauth_callbacks_controller.rb:14:in `redirect_callbacks'

I think this issue has been addressed before, but I think it wasn't solved and it has now, resurfaced.

A more detailed description of what is going on can be found here

update: the request.env['omniauth.params'] is coming in null from the callback, as well as request.env['omniauth.auth'] for that matter...
would really appreciate some help in this @lynndylanhurley ! thank you in advance

I'm also running into this issue

Hi there @franciscoml67    ,
In an effort to cleanup this project and prioritize a bit, we're marking issues that haven't had any activity in a while with a "close-in-7-days" label. If we don't hear from you in about a week, we'll be closing this issue. Obviously feel free to re-open it at any time if it's the right time or this was done in error!
If you are still having the issue (especially if it's a bug report) please refer to our new Issue Template to provide some more details to help us solve it.
Hope all is well.

hey @franciscoml67 @zachfeldman I'm having this same issue!

hey @franciscoml67 @zachfeldman I'm having this same issue!

@franciscoml67 did you find any solution ? Getting the same problem ...

Yes, I stopped using devise_token_auth and omniauth :)

Yes, I stopped using devise_token_auth and omniauth :)
I did it from scratch by myself. It is not this complicated and at least you have no issue!

Ahaha maybe I will do the same in a few hours 🤯🤯🤯

Trying a different version of omniauth-google-oauth2 gem solved the issue for me.

Error during Google OAuth callback: undefined method `persisted?' for nil:NilClass
how to solve this problem?


Missing @login_required decorator

The views reg_begin and reg_complete both reference request.user without having the @login_required decorator. Opening these URLs leads to unwanted Internal Server Errors, if the user is not logged in.
https://github.com/mkalioby/django-passkeys/blob/310b4f497bb793646eba0c6dc0b00329a98fc240/passkeys/FIDO2.py#L60-L99
(Technically, it wouldn't be needed on reg_complete, but I think for the sake of completeness, it should be added nonetheless.)

Nice catch

@mkalioby Do you intend to fix this as part of upcoming merges or may I make a PR together with fixes for #21?

Let's do this in a single PR if you can, as this applies to version v1.2 as with the template naming, we will move to v2.0 as it is a breaking change


set_access_token() should validate the token immediately

Currently, the token is not validated immediately when the token is set. A test API call can be made immediately after setting the token to validate it.

This is how it will look when this feature will be added.
The test api call that is used here is for getting all strategies.


Inline math no longer rendering

Hi @jakevdp, the inline math on your blog pages no longer seems to be rendering. Updating the MathJax link in base.html to:
<script src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/MathJax.js?config=TeX-AMS_HTML" type="text/javascript"></script>

seems to fix the issue.

Strange, it works just fine for me. What browser are you using?

I now see it as an intermittent problem - just refreshing my browser (both Chrome on my laptop and on my phone) a few times seems to show inline math rendering properly about half the time. The non-inline stuff between $$ $$ seems to always renders.
Oddly, I just built a site very heavily based on yours here http://mattpitkin.github.io/samplers-demo/blog/2018/02/01/samplers-samplers-everywhere/ and the inline math never renders - although of be interested to see if you can see it they properly? For that site, when I made this fix it fixed the rendering issue, at least when I built the page and viewed it locally using make serve.

In that screen shot it looks like the inline math isn't rendering for you either.

Oh, I see, I misunderstood what you were referring to. I saw the rendered math and thought it was OK

Perhaps this is the reason? https://github.com/mathjax/MathJax/issues/1138#issuecomment-86378114


[FEATURE] Keeping state of the table

Is it a way to keep the state of the table? Use case: I reload the page and have the search / sorting applied as I left the table.

Hi @fabarea, sorry for lately reply.

reload the page

You means you refresh the page in browser?

Yes.

@fabarea, I think it is not a important feature for now and I think in react world, you should just keep the search text and sorting(field and order) and cache all in somewhere where should not be clear when reload. If reloading, just set the default sort the call handleSearch will restore the status.
BTW, #540 has asked a default search for this component and I'll implement it in recently. So, maybe in the future, you just set the default sort and default search when reload trigger!

Thanks for your reply.
handleSearch could do the trick but I am not sure how to use it. Could you give me a direction (like a snippet) how to proceed it or shall I wait #540 to have it implemented? Here is my use case for a user which is fairly common:

route /customers - list view of customers (bootstrap table). We are dealing with ~ 2000 entries. Let assume the user filters with term "foo".
route customers/edit/xyz - edit view of a customer, the user has clicked on the edit icon in a row and is now editing the data.
route /customers - the user is back on the list view, data is refreshed - however the state of the table is "lost". => I wish to have search term "foo" re-applied against the list

Hope it shades some light about what I would like to achieve - which is a very ordinary / expected behaviour.
For instance, in the DataTables jQuery plugin (I know not to compare with in the approch!) they have an option stateSave: true which does the job.

@fabarea, sorry for lately reply, I thinks handleSearch is what you wanted. check docs
following is example:
//in somewhere, you can do a dynamic search
this.refs.table. handleSearch('foo');
///

render() {
  return(
    <BootstrapTable ref='table'>
    </BootstrapTable>
  );
}

I'll open it if necessary, feel free to discuss with me.

Thanks for the pointer. It will look quiet clumsy but I resolved the case as that for the time being:
	/**
	 * @return void
	 */
	componentDidUpdate() {
		const searchTerm = localStorage.getItem('react-bootstrap-table-search-term');
		if (this.refs.table && searchTerm) {

			// Trigger a search on the table
			this.refs.table.handleSearch(searchTerm);

			// Re-inject value into search text field.
			$('.react-bs-table-search-form input').val(searchTerm);
		}
	}

And even more awful (I barely dare copy / paste here) to persist the search term value - I could not use onSearchChange as I don't do server side processing

	/**
	 * @return void
	 */
	componentWillMount() {
		$(document).on("keyup", ".react-bs-table-search-form input", e => {
			localStorage.setItem('react-bootstrap-table-search-term', $('.react-bs-table-search-form input').val());
		});
	}


[BUG] Failed to train index

At the end of training it says:
Failed to train index: need at least one array to concatenate
If you are running this code in a virtual environment, make sure you have enough GPU available to generate the Index file.

I also noticed that it says Unfortunately, there is no compatible GPU available to support your training. in the advanced gpu settings. I have RTX 3060. I updated my "game ready" drivers.
At the start of the training there is this warning:
Starting training...
C:\Users\ZeusLT\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\autograd\__init__.py:251: UserWarning: Grad strides do not match bucket view strides. This may indicate grad was not created according to the gradient layout contract, or that the param's strides changed since DDP was constructed.  This is not an error, but may impair performance.
grad.sizes() = [64, 1, 4], strides() = [4, 1, 1]
bucket_view.sizes() = [64, 1, 4], strides() = [4, 4, 1] (Triggered internally at ..\torch\csrc\distributed\c10d\reducer.cpp:334.)
  Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass

Failed to train index: need at least one array to concatenate
If you are running this code in a virtual environment, make sure you have enough GPU available to generate the Index file.
Linux Debian 12,  python-3.10.12
I am getting the same error, Running on CPU only
Everything else appears to be working correctly on this install of Applio,,,although am new to this,,May be other underlaying problems,have not hit yet?
Thanks,,This looks like amazing for voice cloning

It seems that "extract features" doesn't generate any files into the folder "3_feature768".

3_feature256 and 3_feature768 folders are empty, when I use "extract features" I get a bunch of these:
"slow_conv2d_cpu" not implemented for 'Half'
"slow_conv2d_cpu" not implemented for 'Half'
"slow_conv2d_cpu" not implemented for 'Half'
"slow_conv2d_cpu" not implemented for 'Half'
"slow_conv2d_cpu" not implemented for 'Half'

Does this mean that the files failed to generate, hence the "need at least one array to concatenate"?

Okay I fixed it. It was passing 16bit float where it was supposed to be 32bit. I'm unsure what is the best solution, but converting it too 32bit works.
Inside \Applio\rvc\train\extract\extract_feature_print.py line 86
Before:
                    inputs = {
                        "source": feats.to(device),
                        "padding_mask": padding_mask.to(device),
                        "output_layer": 9 if version == "v1" else 12,
                    }

After:
                    inputs = {
                        "source": feats.to(device).float(),
                        "padding_mask": padding_mask.to(device),
                        "output_layer": 9 if version == "v1" else 12,
                    }

Okay it seems like it was not a proper solution.
I've noticed my torch was not a cuda version.
"pip list" before:

Running this pip command fixed it for me:
pip install --force-reinstall torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
"pip list" after:

Adding:
On the Google Colab of Applio when training a voice the training always went well and generated a pth file,and then post process to makle the index file I got the above mentioned error.
This also happened if i tried to simply generate the index without doing the training button itself.
ERROR:
Failed to train index: need at least one array to concatenate
If you are running this code in a virtual environment, make sure you have enough GPU available to generate the Index file.
This was using the T4 GPU. Never did try it will the cpu runtime.

@ZeusLT
Are you getting any saved .pth files?
When i click on 'start traning' end up with two .index files,,,and no .pth file?
Ideas?
Am on Debian 12 Linux , Python=3.10.12
Did just do a 'git pull,',and pulled down 6 new py files,,,but made no difference.
Thanks

@ZeusLT
Are you getting any saved .pth files? When i click on 'start traning' end up with two .index files,,,and no .pth file?
Ideas? Am on Debian 12 Linux , Python=3.10.12
Did just do a 'git pull,',and pulled down 6 new py files,,,but made no difference. If i run the Google Colab it does generate the 'pht files.
Thanks

Without any output log it would be hard to tell.

`Process dataset{
No supported Nvidia GPU found
Theme Loaded: Applio
Running on local URL:  http://<IP_ADDRESS>:6969
To create a public link, set share=True in launch().
Starting preprocessing...
Preprocessing completed!
Extract features: (rmvpe)
No supported Nvidia GPU found
Theme Loaded: Applio
Running on local URL:  http://<IP_ADDRESS>:6969
To create a public link, set share=True in launch().
Starting preprocessing...
Preprocessing completed!
Start training:  2 - 10 - 100 , settings ,,,check only Save only latest
rocess Process-1:
Traceback (most recent call last):
File "/home/brcisna/.pyenv/versions/3.10.12/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/home/brcisna/.pyenv/versions/3.10.12/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/brcisna/MyPrograms/applio1/Applio1/rvc/train/train.py", line 143, in run
train_sampler = DistributedBucketSampler(
File "/home/brcisna/MyPrograms/applio1/Applio1/rvc/train/data_utils.py", line 363, in init
self.buckets, self.num_samples_per_bucket = self._create_buckets()
File "/home/brcisna/MyPrograms/applio1/Applio1/rvc/train/data_utils.py", line 385, in _create_buckets
total_batch_size - (len_bucket % total_batch_size)
ZeroDivisionError: integer division or modulo by zero
Saved index file '/home/brcisna/MyPrograms/applio1/Applio1/logs/linneal32k/added_IVF591_Flat_nprobe_1_v2.index'
`

total_batch_size - (len_bucket % total_batch_size)
ZeroDivisionError: integer division or modulo by zero
This would indicate that "total_batch_size" ended up a zero. Check if there are files in:
/logs/linneal32k/0_gt_wavs
/logs/linneal32k/1_16k_wavs
/logs/linneal32k/2a_f0
/logs/linneal32k/2b-f0nsf
/logs/linneal32k/3_feature768

How long is your voice data set?

Thanks for trying to help @ZeusLT
All i have is the first two directories you mention in your post and a preprocessor.txt file,,thats all.
Would it have anything to doi with running only on CPU,,, I have tried learning about the fp16 thing,,that keeps coming up in discussions.
I also read many many posts froma year ago,,,whisper,,,had similar errors,,as what spews out here,,in my post,,,and was that would run fine in VScode and on windows but failed in linux,,,was something to the GCC versions...
Am in the process of rigging up a decent GPU machine as we speak,

It seems that it fails to extract features. Applio don't support cpu only and I can't help much. Goodluck


Make pfr more friendly for integrating into other libraries?

Description of issue
Now i'm working with Boost.Fusion extension for Boost.PFR, i decomposed it into 3 PR's and opened these PR's here: https://github.com/boostorg/fusion/pull/241, here https://github.com/boostorg/fusion/pull/242 and here https://github.com/boostorg/fusion/pull/243.
Review is moving on, there is some wishes from reviewers(more on that later).
First, I would like to talk about such a problem like 3 PR's is too much code and the review takes too long.
I suggest to make pfr more adaptable and more friendly to adapt into other libraries(not only fusion).
Suggested solution

To implement two functions:

boost::pfr::view
boost::pfr::view_fields

To implement BOOST_PFR_REGISTRATE_REFLECTABLE macro.
To add boost::pfr::is_reflectable trait. Example of using trait:

struct A {};
struct B {};
struct C {};
BOOST_PFR_REGISTRATE_REFLECTABLE(A)
using boost::pfr::is_reflectable;
is_reflectable<A>::value;                                       //< true
is_reflectable<decltype(boost::pfr::view_fields(B{}))>::value;  //< true
is_reflectable<C>::value;                                       //< false

To use boost::pfr::is_reflectable trait for create overloading of compare/io operators and specialization of std::hash function. Examples of using these overloads:

#include <boost/pfr/ops.hpp>

template <class T>
struct uniform_comparator_less {
    bool operator()(const T& lhs, const T& rhs) const noexcept {
        // If T has operator< or conversion operator then it is used.
        return boost::pfr::view(lhs) < boost::pfr::view(rhs);
    }
};

#include <boost/pfr/functions_for.hpp>

struct pair_like {
    int first;
    short second;
};

BOOST_PFR_REGISTRATE_REFLECTABLE(pair_like)   // Added pair_like's specialization for is_reflectable trait.

// ...

assert(pair_like{1, 2} < pair_like{1, 3});

#include <boost/pfr/io_fields.hpp>

struct pair_like {
    int first;
    std::string second;
};

inline std::ostream& operator<<(std::ostream& os, const pair_like& x) {
    return os <<  bost::pfr::view_fields(x);
}

To declare io, eq, ne, lt, gt, le, ge, hash_value, io_fields, eq_fields, ne_fields, lt_fields, gt_fields, le_fields, ge_fields, hash_fields, BOOST_PFR_FUNCTIONS_FOR as deprecated, and to actualize https://www.boost.org/doc/libs/1_75_0/doc/html/boost_pfr/tutorial.html#boost_pfr.tutorial.three_ways_of_getting_operators.

Conclusion
If everything planned can be done, then the PFR will be adapted as easily as the std::tuple, it will be just one small PR.

What further prospects will we get if we implement this approach:

To extend is_reflectable trait, to implement it's code for auto detect that user defined type is reflectable.
It discussed here: https://github.com/boostorg/pfr/issues/56 and reviewers from Fusion wish it.
To give for PFR user a fallback, in case the automation fails:
-BOOST_PFR_FORCE_REFLECTABLE macro (just alias for BOOST_PFR_REGISTRATE_REFLECTABLE)
-BOOST_PFR_FORCE_NON_REFLECTABLE macro
-boost::pfr::force_reflectable function (just alias for boost::pfr::view_fields)
-boost::pfr::force_non_reflectable function

struct A {};
struct B {};
struct C {};
BOOST_PFR_FORCE_REFLECTABLE(A)
BOOST_PFR_FORCE_NON_REFLECTABLE(B)
using boost::pfr::is_reflectable;
using boost::pfr::force_reflectable;
using boost::pfr::force_non_reflectable;
is_reflectable<A>::value;                                     //< true
is_reflectable<B>::value;                                     //< false
is_reflectable<decltype(force_reflectable(C{}))>::value;      //< true
is_reflectable<decltype(force_non_reflectable(C{}))>::value;  //< false

To move it all into separate experimental include section.

I'd appreciate help with all the above points

I wrote a draft code for this functionality. The whole PR with docs and tests will be in a nearest future. I suppose that these changes will be super clean and won't get stuck on review.
Draft: https://godbolt.org/z/Y98xhajb8
Not all of the planned was done, some points turned out to be impracticable (and inexpedient), I will describe in more detail about deviations from the plan in the PR.
How can we use modified PFR to make other libraries extensions? Like these:
Boost Json extension sample: https://godbolt.org/z/GYY8e9jMs
Boost Fusion extension sample: https://godbolt.org/z/Y6713Ej4a
Boost Spirit with Fusions extension sample: https://godbolt.org/z/9cYvea3b7
Also, i want to say that the most difficult part of the future PR is the implementation of pfr::view. It is an wrap-object, that can be a reference to underlying value or can be an owner, can be forwarded and can be implicitly casted to its underlying type.
I didn't find a solution in all C++ libraries that i know, so i just reinvented the wheel.
View: https://godbolt.org/z/rYhdnv9MW
I suppose that this wrap-object can slow down the review for us.

We decided to select yet more simple strategy for integrating. In the Fusion side PFR will be fallback, and will be selected when no other reflection not found for T.
It can be solved using pfr::is_implicitly_reflectable trait, no need for difficult pfr::view_t.
Draft: https://godbolt.org/z/P1481orTM
The whole PR in progress, also sample for Fusion and Spirit. It all will be available in the nearest future.

We decided to select yet more simple strategy for integrating. In the Fusion side PFR will be fallback, and will be selected when no other reflection not found for T.
It can be solved using pfr::is_implicitly_reflectable trait, no need for difficult pfr::view_t.
Draft: https://godbolt.org/z/P1481orTM
The whole PR in progress, also sample for Fusion and Spirit. It all will be available in the nearest future.

We also need to apply patch on the Fusion side, described here: https://github.com/boostorg/fusion/issues/231

We decided to select yet more simple strategy for integrating. In the Fusion side PFR will be fallback, and will be selected when no other reflection not found for T.
It can be solved using pfr::is_implicitly_reflectable trait, no need for difficult pfr::view_t.
Draft: https://godbolt.org/z/P1481orTM
The whole PR in progress, also sample for Fusion and Spirit. It all will be available in the nearest future.

Look at this: https://github.com/boostorg/pfr/pull/111

Looks like everything is done here. Closing. Please create new issue if there's anything to improve


Add a Redirect Block

Describe the bug
If we build a content structure like this:
/Plone/communications-department
/Plone/communications-department/news
/Plone/communications-department/events
/Plone/communications-department/press-releases
In Plone we can add a "Link" item, set it as a "default page" for communications-department folder and get a redirect to, say, /Plone/communications-department/news, this way we can group several folders under a folder and get the users redirected.
In plain Volto this is not possible.
A "Redirect Block" could help here. If we add a "Redirect Block" to the communications-department then it will work like in Classic Plone.

Why not use a folderish Document instead of a Folder? Just to keep the url with /news?

Yes indeed we can, I was trying to bring to Volto what we currently have in Plone with "links-as-default-pages" which can't be done right now in Volto.


[FormRecognizer] Re-enable StartAnalyzeDocumentPopulatesExtractedReceiptJpg

Fixes https://github.com/azure/azure-sdk-for-net/issues/27083.

API change check for Azure.AI.FormRecognizer
API changes are not detected in this pull request for Azure.AI.FormRecognizer


Test whether the recent build errors come from multiprocessing or not

We haven't merged any PR where the CI build did not pass. So I don't think the recent failures are linked to any change on our end (at least not in isolation).
My guess is that it may be that Travis made a change that resulted in builds timing out after ~20-25 min (exact timing is non-deterministic). TF tests have been taking that much time, or more, for a long time, but that only started being a problem a few days ago.
This view is reinforced by the fact that the failures aren't completely deterministic, sometimes one of the two TF builds pass (when it happens to take less time).
Proposed fix: temporarily disable very large TF tests to bring TF build time under 20 min.


Characteristic.WriteAsync never returns on iOS

Steps to reproduce

Connect to device
Discover services
Discover characteristic
Write to characteristic async

My code does the following:
_bleAdapter.ConnectToDevice(device);

In the DeviceConnected event I do the following:
private async void BleAdapterOnDeviceConnected(object sender, DeviceConnectionEventArgs args)
{
    var device = args.Device;

    var deviceViewModel = Devices.FirstOrDefault(d => d.Id == device.ID);
    deviceViewModel.Connected = true;
    deviceViewModel.DeviceName = device.Name;

    Mvx.TaggedTrace("HomeViewModel::DeviceConnected", $"Connected to {deviceViewModel.DeviceName} - UUID {deviceViewModel.Uuid}");

    try
    {
        var service =
            await device.GetServiceAsync(new Guid("13333333333333333333333333333337")).ConfigureAwait(false);
        if (service != null)
        {
            Mvx.TaggedTrace("HomeViewModel::DeviceConnected", "Found WiFi configuration service");

            var characteristic =
                await
                    service.GetCharacteristicAsync(new Guid("13333333333333333333333333330001"))
                        .ConfigureAwait(false);

            if (characteristic != null && characteristic.CanWrite)
            {
                Mvx.TaggedTrace("HomeViewModel::DeviceConnected", "Found SSID configuration characteristic");

                var ssidBytes = Encoding.UTF8.GetBytes(HotspotViewModel.Ssid);
                var writeOk = await characteristic.WriteAsync(ssidBytes).ConfigureAwait(false);
                if (writeOk)
                {
                    Mvx.TaggedTrace("HomeViewModel::DeviceConnected", "Wrote SSID successful!");
                    deviceViewModel.Ssid = HotspotViewModel.Ssid;
                }
            }
        }

        _bleAdapter.DisconnectDevice(device);
        Mvx.TaggedTrace("HomeViewModel",
            $"Disconnecting device  {deviceViewModel.DeviceName} - UUID {deviceViewModel.Uuid}");
    }
    catch (Exception e)
    {
        Mvx.TaggedTrace("HomeViewModel::DeviceConnected", $"Got exception: {e}");
    }
}

The code never reaches the if (writeOk) and does not throw any exceptions either. The bytes are written to the device. So must be something wrong with the TaskCompletionSource.
This same construction works fine on Android.
Expected behavior
WriteAsync should eventually return
Actual behavior
WriteAsync never returns
Configuration
Version: MvvmCross.Plugin.BLE 0.9.2
Platform: iOS
Device: iPhone 4s, iPhone 6+

Maybe moving the declaration of the event handler writtenHandler to the local space is enough
https://github.com/xabre/xamarin-bluetooth-le/blob/master/Source/Plugin.BLE.iOS/Characteristic.cs#L72

I tried with latest 1.0.0-beta and the issue is still there.

@Cheesebaron  Unfortunately, we have no test device with a WriteWithResponse characteristic. Could you please use the sample app, connect and try to write a value?

I have seen a similar issue on iOS. We were trying to have our peripheral device support both WriteWithResponse and WriteWithoutResponse (which is legal in BLE, from what I can tell), and iOS would report WriteWithResponse on the characteristic, but not actually ever send a response. It's possible that the outgoing write didn't identify as trying to write with response, but I didn't look into it much further.
We got around the issue by only enabling WriteWithoutResponse on our peripheral device, but that's not a solution for most people dealing with existing peripherals which the do not control.

Off topic: @adolgov thanks for your recent comments. Are you at https://xamarinchat.slack.com/ or can you send me an message on twitter?


LUCENE-10495: Fix return statement of siblingsLoaded() in TaxonomyFacets

Description
Found a bug in TaxonomyFacets when trying to use the siblingsLoaded function. Currently, siblingsLoaded() returns children!= null, but it should return siblings != null;
Solution
Fixed the return statement: return siblings != null.
Tests
Added test case testSiblingsLoaded() in TestTaxonomyFacetSumValueSource where siblingsLoaded() and childrenLoaded() can return different boolean values.
Checklist
Please review the following and check all that apply:

[X] I have reviewed the guidelines for How to Contribute and my code conforms to the standards described there to the best of my ability.
[X] I have created a Jira issue and added the issue ID to my pull request title.
[X] I have given Lucene maintainers access to contribute to my PR branch. (optional but recommended)
[X] I have developed this patch against the main branch.
[X] I have run ./gradlew check.
[X] I have added tests for my changes.

@Yuti-G looks like a conflict needs to be resolved when you get a chance. Also, could you please add an entry to CHANGES.txt noting the bug fix please? We'll backport this to 9.2, so it would make sense to add it in the 9.2 section. Thanks again for finding this and fixing it!

Thanks @gsmiller! I resolved the conflicts and added an entry to CHANGES.txt.

Thanks @zhaih !


Initial CORS implementation

Pretty small package. Just does the job.

The code looks good to me. Would like to see more tests though :wink:

Also isn't travis setup?

Yes, Travis is active now.


sl-date-range-picker: "endDateValue" cannot be less than than the "startDateValue"

Provide a fix for "endDateValue" is allowed to be less than the "startDateValue" and then verify the fix by turning the skipped test below back on.
This test fails:
test( '"endDateValue" cannot be less than "startDateValue"', function( assert ) {
this.render( hbs`
    {{sl-date-range-picker
        startDateValue="09/25/2015"
        endDateValue="09/20/2015"
    }}
` );

assert.notEqual(
    this.$( '>:first-child' ).find( '.sl-daterange-end-date' ).find( 'input' ).val(),
    '09/20/2015',
    'The "endDateValue" is not more than the "startDateValue"'
);

});

I'm working on this. addressed by #870 (will cause merge conflicts otherwise)

Should be resolved by #1458


Removal of Part-Time Application Questions For Partner Supporting Users

Related to #2172
For PT applications, remove the following questions relating to Spouses/Partner information (these questions appear both in the 'supporting users' tab AND in the student's own application if they are self-reporting on behalf of their spouse - they should be removed in both places
Acceptance Criteria
Remove the following questions from supporting users form and student form 'self-report' feature

[ ]   "Will you be employed full-time or part-time during your partner's study period"
[ ]   "Will you be at home caring for eligible dependent children on a full-time basis during your partner's entire study period?"
[ ]  "Will you be living with your partner during the study period"
[ ]  "Will you be a full-time post-secondary student for some or all of your partner's study period (remove the follow-up questions capturing weeks of study as well)"
[ ]  "Will you be paying a Canada Student Loan.."
[ ]  "Will you be paying for Child support and/or spousal support during your Partner's study period"
[ ]  "Will you be taking care of eligible dependents during your Partner's study period"
[ ]  Add to the ticket comments the removed keys and mention the workflow impact points.

Removing the following questions from PT applications will NOT have workflow impact points.

Will you/your partner be employed full-time or part-time during your Partner's study period
Will you/your partner be at home caring for eligible dependant children on a full-time basis during your Partner's entire study period?
Will you/your partner be living with your Partner during your Partner's study period?
Will you/your partner be taking care of eligible dependants during your Partner's Study Period?

The following questions have related workflow impact points in Camunda.

Will you/your partner be a full-time post-secondary student for some or all of your Partner's study period?

Supporting-user input: How many weeks will you be a full-time post-secondary student during your Partner's study period?

FormIO API: partnerStudentStudyWeeks

Self-reporting input: If your partner will be a full-time student, how many weeks of your study period will they also be in studies?

FormIO API: ifYourPartnerWillBeAFullTimeStudentHowManyWeeksOfYourStudyPeriodWillTheyAlsoBeInStudies (doesn't have a key in Camunda)

Will you/your partner be paying a Canada Student Loan and/or a provincial student loan regular scheduled payments during your Partner's study period?

Supporting-user input: Enter amount that you will pay during your Partner's study period.

FormIO API: partnerTotalStudentLoan

Self-reporting input: Enter amount that your partner will pay during your study period.

FormIO API: partnerstudentloanCosts

Will you/your partner be paying for child support and/or spousal support during your Partner's study period?

Supporting-user input: Enter amount that you will pay during your Partner's study period.

FormIO API: partnerTotalSupport

Self-reporting input: Enter amount that your partner will pay during your study period.

FormIO API: partnerchildsupportCosts

FomIO
Camunda
Calculated Data
Workflow Data

partnerStudentStudyWeeks
partner1StudentStudyWeeks
calculatedDataPartnerStudentStudyWeeks
calculatedData.partnerStudentStudyWeeks

partnerTotalStudentLoan
partner1TotalStudentLoan
None
None

partnerstudentloanCosts
no key
None
None

partnerTotalSupport
no key
None
None

partnerchildsupportCosts
no key
None
None

The following questions have related workflow impact points in ier12.

Will you/your partner be a full-time post-secondary student for some or all of your Partner's study period?

Supporting-user input: How many weeks will you be a full-time post-secondary student during your Partner's study period?

FormIO API: partnerStudentStudyWeeks
A new ier12 record will have partnerStudentStudyWeeks: workflowData.calculatedData.partnerStudentStudyWeeks or 0

Conclusion:

No immediate impact on the current workflows for removing the questions for PT applications
Potential impact later on:

In case the question "Will you/your partner be a full-time post-secondary student" becomes available. When the applicant enters a value for
the self-reporting input -- "If your partner will be a full-time student..." -- the value will not be captured in workflows and
a new ier12 record will have partnerStudentStudyWeeks === 0 as the key is missing.

@lewischen-aot the possible missing key partnerStudentStudyWeeks will always be present for Full-time applications, right?
If it helps, IER12 is only generated for full-time applications.

That's correct @andrewsignori-aot. For FT applications, the key partnerStudentStudyWeeks will be present in the supporting user form, but it is missing in the self-report form. We may want to address this later, but it has no impact for PT applications for now.


Is it compatible with react-native v0.57.4?

I'm in process of upgrading my app which is using react-native-progress-circle v2.0.1, will it be compatible if I upgrade the react-native version to 5.7?

I guess so, it's just JavaScript and unless they heavily reworked the yoga layout engine, I don't see why not.
Just try it.

Hi @MrToph.
Actually, not so much. There is a small graphical bug with the latest version. The progress circle is not a circle anymore, it seems that it's a rectangle or something similar.
If you can test it to confirm, I'll appreciate

React Native 0.57.4 and the issue still happens

Much to my regret  ！React Native 0.57.7 and the issue still happens

Is this issues resolved. I am having issues with 0.59.8?


GCS client for front

We need a GCS interface in typescript for the upsert queue that we will likely use as well for content fragment v2.
We already have the SERVICE_ACCOUNT that has access to our GCS in front secrets.
We'll create 1 new bucket and make sure we add access to it to the service account (production and test):

[ ] Create bucket dust-upsert-queue
[ ] Make sure service account have access to it (prod and dev)
[ ] Add import for GCS library in typescript and test it

Actually we already have the Storage import available.

Created https://console.cloud.google.com/storage/browser/dust-upsert-queue;tab=objects?hl=en&project=or1g1n-186209&prefix=&forceOnObjectsSortingFiltering=false with no retention, no versioning, unifrom access, multi region, public access prevention

gsutil iam ch serviceAccount:dust-api-svc@or1g1n-186209.iam.gserviceaccount.com:roles/storage.objectAdmin gs://dust-upsert-queue-prod

gsutil iam ch serviceAccount:dust-api-svc-test@or1g1n-186209.iam.gserviceaccount.com:roles/storage.objectAdmin gs://dust-upsert-queue-test

Added export DUST_UPSERT_QUEUE_BUCKET=dust-upsert-queue-test  to dev setup

Thanks.


SSO support

Attempting to try out an SSO profile and receiving the following error:
Missing credentials in config, if using AWS_CONFIG_FILE, set AWS_SDK_LOAD_CONFIG=1
The connections code here doesn't suggest support for SSO which leads me to believe the above error isn't actually providing a path to resolution.
Is SSO support planned?

I've been using the SSO token to create temporary credentials. Here's a very simple python script.
import json
import glob
import os
import boto3
from pathlib import Path

print(
    "Before running this script, make sure you have created an AWS SSO token:\naws sso login --profile my-profile"
)

AWS_CONFIG_PATH = f"{Path.home()}/.aws/config"
AWS_CREDENTIAL_PATH = f"{Path.home()}/.aws/credentials"
AWS_SSO_CACHE_PATH = f"{Path.home()}/.aws/sso/cache"
ROLE_NAME = "my-admin-role"
ACCOUNT_PREFIX = ""  # to be stripped

# fetching all credential files and selecting youngest
# this is assuming that the youngest file was created by the SSO request
list_of_files = glob.glob(f"{AWS_SSO_CACHE_PATH}/*.json")
newest_file = max(list_of_files, key=os.path.getctime)

# loading SSO token from file
with open(newest_file, "r") as f:
    credentials = json.load(f)
region = credentials["region"]
access_token = credentials["accessToken"]

# creating session, finding all accounts, ...
session = boto3.Session(region_name=region)
sso = session.client("sso")
paginator = sso.get_paginator("list_accounts")
results = paginator.paginate(accessToken=access_token)
account_list = results.build_full_result()["accountList"]

credentials_string = []

# iterating over accounts, retrieving credentials
for account in account_list:
    response = sso.get_role_credentials(
        roleName=ROLE_NAME,
        accountId=account["accountId"],
        accessToken=access_token,
    )

    credentials_string.append(
        f"""[{account['accountName'].replace(ACCOUNT_PREFIX,'')}]
aws_access_key_id={response['roleCredentials']['accessKeyId']}
aws_secret_access_key={response['roleCredentials']['secretAccessKey']}
aws_session_token={response['roleCredentials']['sessionToken']}
"""
    )

# writing credentials to file
with open(AWS_CREDENTIAL_PATH, "w") as f:
    f.write("".join(credentials_string))

Unless I am missing something obvious then this issue also results in failure to get credentials from instance metadata.
I think it would probably still work if the instance role was able to assume another role configured in a profile. eg. a profile like
[admin-role]
source_profile = default
role_arn = arn:...

For my case where there is no role switching happening you can get the details like this.
export ROLE=role_name
curl http://<IP_ADDRESS>/latest/meta-data/iam/security-credentials/$role_name | jq -r '"aws_access_key_id = " + (.AccessKeyId) + "\naws_secret_access_key = " + (.SecretAccessKey) + "\naws_session_token = " + (.Token)'

Hi,
I am facing a similar issue. I have VS Code running on an EC2 instance. The EC2 instance has an IAM role attached which allows using Athena so any application running on the EC2 should be able to use Athena without requiring IAM keys.  However I can't figure out how to avoid reconfiguring the plugin each time I need to use Athena.
+1 for SSO


NestedProgram tutorials update

Summary
Fix #386 , #342
Details and comments

Merged #347
Updated references from Program to NestedProgram

Before merge @Tansito it can be great if you can double check that all tutorials/guide continue working properly, just in case...

Reviewing the code I just sat what I forgot to rename run_program in the Provider and in the Job. I just update the references and the documentation. As soon as I finish reviewing the guides and the tutorials I merge it.

We need to use QuantumFunction name now :)

@pacomf we will go with "Program" in place of "Nested Program"

@pacomf we will go with "Program" in place of "Nested Program"

Yay! Program is much better!

@IceKhan13 @pacomf I didn't change in this PR run_program. With the change of Program do we want to continue change it or...?