Implement tests for bam_to_h5.R

Implement regression test in Python or R to compare outputs to precalculated outputs e.g. those from the vignette.
Develop tests to inspect H5 output files and validate they meet specific properties as determined by the input GFF and BAM files.

Compare H5 files (output from bam_to_h5.R) via command line:
$ python -m riboviz.tools.compare_files -1 test-bam-to-h5/WT3AT/WT3AT.h5 -2 vignette/output/WT3AT/WT3AT.h5

This uses riboviz.compare_files's compare_files function (which is also used by the regression test framework). This in turn, uses riboviz.h5's equal_h5 function. This in turn calls, as a bash command, h5diff -q <FILE1> <FILE2>.

Compare H5 files (output from bam_to_h5.R) via command line:
$ python -m riboviz.tools.compare_files -1 test-bam-to-h5/WT3AT/WT3AT.h5 -2 vignette/output/WT3AT/WT3AT.h5

This uses riboviz.compare_files's compare_files function (which is also used by the regression test framework). This in turn, uses riboviz.h5's equal_h5 function. This in turn calls, as a bash command, h5diff -q <FILE1> <FILE2>.

Created branch https://github.com/riboviz/riboviz/tree/test-bam-to-h5-238

Created branch https://github.com/riboviz/riboviz/tree/test-bam-to-h5-238

HDF5 data structure template derived from inspecting input GFF and BAM files and output H5 file.
I added info from the RiboViz doc Structure of HDF5 data, that I only recalled existed today.
The HDF5 file captures the following information for each organism sequence:

Sequence name (from GFF and BAM)
buffer_left: number of nucleotides upstream of the start codon (ATG) (UTR5 length) (from bam_to_h5.R command-line)
buffer_right: number of nucleotides downstream of the stop codon (TAA/TAG/TGA) (UTR3 length) (from bam_to_h5.R command-line)
start_codon_pos: Positions corresponding to start codon of CDS in organism sequence (from GFF)
stop_codon_pos: Positions corresponding to stop codon of CDS in organism sequence (from GFF)
lengths : Lengths of mapped reads.
reads_by_len: Counts of number of ribosome sequences of each length (from BAM).
reads_total: Total number of ribosome sequences (from BAM, equal to number of non-zero reads in `reads_by_len).
data: Positions and lengths of ribosome sequences within the organism data (from BAM).

HDF5 "<FILE>.h5" {
GROUP "/" {
   GROUP "<sequence ID from FASTA|GFF|reference sequence name from BAM>" {
      GROUP "<dataset>" {
	 GROUP "reads" {
	    ATTRIBUTE "buffer_left" {
	       DATATYPE  H5T_IEEE_F64LE
	       DATASPACE  SIMPLE { ( 1 ) / ( 1 ) }
	       DATA {
	       (0): <buffer|GFF UTR5 length>
	       }
	    }
	    ATTRIBUTE "buffer_right" {
	       DATATYPE  H5T_STD_I32LE
	       DATASPACE  SIMPLE { ( 1 ) / ( 1 ) }
	       DATA {
	       (0): <buffer|GFF UTR3 length>
	       }
	    }
	    ATTRIBUTE "lengths" {
               DATATYPE  H5T_STD_I32LE
               DATASPACE  SIMPLE { (<max-read-length>-<min-read-length>+1) / (<max-read-length>-<min-read-length>+1) }
               DATA {
               (0): <min-read-length>, <min-read-length>+1, ... ,<min-read-length>+<m>-1,
               (<m>): <min-read-length>+<m>, <min-read-length>+<m>+1, ... ,<min-read-length>+<n>-1,
               (<n>): <min-read-length>+<n>, <min-read-length>+<n>+1, ... ,<min-read-length>+<max-read-length>
               }
            }
            ATTRIBUTE "reads_by_len" {
               DATATYPE  H5T_IEEE_F64LE
               DATASPACE  SIMPLE { (<max-read-length>-<min-read-length>+1) / (<max-read-length>-<min-read-length>+1) }
               DATA {
               (0): <see below>
               (<m>): <see below>
               (<n>): <see below>
               }
            }
           ATTRIBUTE "reads_total" {
               DATATYPE  H5T_IEEE_F64LE
               DATASPACE  SIMPLE { ( 1 ) / ( 1 ) }
               DATA {
               (0): <number of non-zero values in reads_by_len>
               }
            }
            ATTRIBUTE "start_codon_pos" {
               DATATYPE  H5T_STD_I32LE
               DATASPACE  SIMPLE { ( 3 ) / ( 3 ) }
               DATA {
               (0): <position of 1st nt of CDS start codon from GFF>,...,<position of 3rd nt of CDS start codon from GFF>
               }
            }
            ATTRIBUTE "stop_codon_pos" {
               DATATYPE  H5T_STD_I32LE
               DATASPACE  SIMPLE { ( 3 ) / ( 3 ) }
               DATA {
               (0): <position of 1st nt of CDS stop codon from GFF>,...,<position of 3rd nt of CDS stop codon from GFF>
               }
            }
            DATASET "data" {
               DATATYPE  H5T_STD_I32LE
               DATASPACE  SIMPLE { ( <position of final codon of UTR3 from GFF|Length of sequence from BAM header LN value>, <max-read-length>-<min-read-length>+1 ) / <position of final codon of UTR3 from GFF|Length of sequence from BAM header LN value>, <max-read-length>-<min-read-length>+1 ) }
               DATA {
               (0,0): <see below>
               (0, <m>): <see below>
               (0, <n>): <see below>
               ...
               (<p>,0): <see below>
               (<p>,<m>): <see below>
               (<p>,<n>): <see below>
               ...
               (<position of final codon of UTR3 from GFF|Length of sequence from BAM header LN value>,0): <see below>
               (<position of final codon of UTR3 from GFF|Length of sequence from BAM header LN value>,<m>): <see below>
               (<position of final codon of UTR3 from GFF|Length of sequence from BAM header LN value>,<n>): <see below>
               }
            }
         }
      }
   }
   ...
}

where:

0 < m < n <= max-read-length - min-read-length. Exact values of m and n may differ across specific ATTRIBUTE and DATA items.
reads_by_len:

reads_by_len[i] = sum of sequences in BAM which have length equal to lengths[i].
Also equals sum of DATA[*, i] i.e. sum across all positions for a specific length.

DATASET "data":

0 <= p <= position of final codon of UTR3 from GFF|Length of sequence from BAM header LN value.
DATA[p, i] = 1 if there is a sequence from BAM at position p+1 which has length equal to lengths[i], else 0.

HDF5 data structure template derived from inspecting input GFF and BAM files and output H5 file.
I added info from the RiboViz doc Structure of HDF5 data, that I only recalled existed today.
The HDF5 file captures the following information for each organism sequence:

Sequence name (from GFF and BAM)
buffer_left: number of nucleotides upstream of the start codon (ATG) (UTR5 length) (from bam_to_h5.R command-line)
buffer_right: number of nucleotides downstream of the stop codon (TAA/TAG/TGA) (UTR3 length) (from bam_to_h5.R command-line)
start_codon_pos: Positions corresponding to start codon of CDS in organism sequence (from GFF)
stop_codon_pos: Positions corresponding to stop codon of CDS in organism sequence (from GFF)
lengths : Lengths of mapped reads.
reads_by_len: Counts of number of ribosome sequences of each length (from BAM).
reads_total: Total number of ribosome sequences (from BAM, equal to number of non-zero reads in `reads_by_len).
data: Positions and lengths of ribosome sequences within the organism data (from BAM).

HDF5 "<FILE>.h5" {
GROUP "/" {
   GROUP "<sequence ID from FASTA|GFF|reference sequence name from BAM>" {
      GROUP "<dataset>" {
	 GROUP "reads" {
	    ATTRIBUTE "buffer_left" {
	       DATATYPE  H5T_IEEE_F64LE
	       DATASPACE  SIMPLE { ( 1 ) / ( 1 ) }
	       DATA {
	       (0): <buffer|GFF UTR5 length>
	       }
	    }
	    ATTRIBUTE "buffer_right" {
	       DATATYPE  H5T_STD_I32LE
	       DATASPACE  SIMPLE { ( 1 ) / ( 1 ) }
	       DATA {
	       (0): <buffer|GFF UTR3 length>
	       }
	    }
	    ATTRIBUTE "lengths" {
               DATATYPE  H5T_STD_I32LE
               DATASPACE  SIMPLE { (<max-read-length>-<min-read-length>+1) / (<max-read-length>-<min-read-length>+1) }
               DATA {
               (0): <min-read-length>, <min-read-length>+1, ... ,<min-read-length>+<m>-1,
               (<m>): <min-read-length>+<m>, <min-read-length>+<m>+1, ... ,<min-read-length>+<n>-1,
               (<n>): <min-read-length>+<n>, <min-read-length>+<n>+1, ... ,<min-read-length>+<max-read-length>
               }
            }
            ATTRIBUTE "reads_by_len" {
               DATATYPE  H5T_IEEE_F64LE
               DATASPACE  SIMPLE { (<max-read-length>-<min-read-length>+1) / (<max-read-length>-<min-read-length>+1) }
               DATA {
               (0): <see below>
               (<m>): <see below>
               (<n>): <see below>
               }
            }
           ATTRIBUTE "reads_total" {
               DATATYPE  H5T_IEEE_F64LE
               DATASPACE  SIMPLE { ( 1 ) / ( 1 ) }
               DATA {
               (0): <number of non-zero values in reads_by_len>
               }
            }
            ATTRIBUTE "start_codon_pos" {
               DATATYPE  H5T_STD_I32LE
               DATASPACE  SIMPLE { ( 3 ) / ( 3 ) }
               DATA {
               (0): <position of 1st nt of CDS start codon from GFF>,...,<position of 3rd nt of CDS start codon from GFF>
               }
            }
            ATTRIBUTE "stop_codon_pos" {
               DATATYPE  H5T_STD_I32LE
               DATASPACE  SIMPLE { ( 3 ) / ( 3 ) }
               DATA {
               (0): <position of 1st nt of CDS stop codon from GFF>,...,<position of 3rd nt of CDS stop codon from GFF>
               }
            }
            DATASET "data" {
               DATATYPE  H5T_STD_I32LE
               DATASPACE  SIMPLE { ( <position of final codon of UTR3 from GFF|Length of sequence from BAM header LN value>, <max-read-length>-<min-read-length>+1 ) / <position of final codon of UTR3 from GFF|Length of sequence from BAM header LN value>, <max-read-length>-<min-read-length>+1 ) }
               DATA {
               (0,0): <see below>
               (0, <m>): <see below>
               (0, <n>): <see below>
               ...
               (<p>,0): <see below>
               (<p>,<m>): <see below>
               (<p>,<n>): <see below>
               ...
               (<position of final codon of UTR3 from GFF|Length of sequence from BAM header LN value>,0): <see below>
               (<position of final codon of UTR3 from GFF|Length of sequence from BAM header LN value>,<m>): <see below>
               (<position of final codon of UTR3 from GFF|Length of sequence from BAM header LN value>,<n>): <see below>
               }
            }
         }
      }
   }
   ...
}

where:

0 < m < n <= max-read-length - min-read-length. Exact values of m and n may differ across specific ATTRIBUTE and DATA items.
reads_by_len:

reads_by_len[i] = sum of sequences in BAM which have length equal to lengths[i].
Also equals sum of DATA[*, i] i.e. sum across all positions for a specific length.

DATASET "data":

0 <= p <= position of final codon of UTR3 from GFF|Length of sequence from BAM header LN value.
DATA[p, i] = 1 if there is a sequence from BAM at position p+1 which has length equal to lengths[i], else 0.

test-bam-to-h5-238 commit 8c9b7c5 has work-in-progress towards regression test for bam_to_h5.R, file test_bam_to_h5.R which runs bam_to_h5.R on WTnone.bam and yeast_YAL_CDS_w_250utrs.gff3 and uses h5diff to compare WTnone.h5 file output to WTnone.h5 file computed previously. Next step is to refactor this into a testthat-compliant R script (and with help from rscripts/tests/testthat/test_generate_stats_figs.R)

test-bam-to-h5-238 commit 8c9b7c5 has work-in-progress towards regression test for bam_to_h5.R, file test_bam_to_h5.R which runs bam_to_h5.R on WTnone.bam and yeast_YAL_CDS_w_250utrs.gff3 and uses h5diff to compare WTnone.h5 file output to WTnone.h5 file computed previously. Next step is to refactor this into a testthat-compliant R script (and with help from rscripts/tests/testthat/test_generate_stats_figs.R)

Commit 789d008 has an example testthat-compliant script that runs bam_to_h5.R on vignette/input/yeast_YAL_CDS_w_250utrs.gff3 and vignette/output/WTnone/WTnone.bam and validates vignette/output/WTnone/WTnone.h5, using h5diff against a previously-computed WTnone.h5 file in the current directory. It can be run as follows.
$ Rscript -e 'library(testthat); test_file("rscripts/tests/testthat/test_bam_to_h5_regression.R")'

It is very brittle and only uses those specific functions.
To make the test more generally useful requires the following:

Add a copy of vignette/output/WTnone/WTnone.bam to the repository (e.g. to data or rscripts/tests/testthat/data/) to serve as a test input file. The other required input file, vignette/input/yeast_YAL_CDS_w_250utrs.gff3, is already in the repository.
Add a copy of vignette/output/WTnone/WTnone.h5) to the repository (e.g. to data or rscripts/tests/testthat/data/) to serve as an expected test output file.
Update the test to run bam_to_h5.R on the .bam and .gff3 file and validate against the expected test output file.

This would require adding 2 files of O(100KB) each:
$ du -sk vignette/output/WTnone/WTnone.bam vignette/output/WTnone/WTnone.h5
384	vignette/output/WTnone/WTnone.bam
596	vignette/output/WTnone/WTnone.h5

We do have comparable bigger files in the repository:
$ du -sk vignette/input/*.*
36868	vignette/input/SRR1042855_s1mi.fastq.gz
55408	vignette/input/SRR1042864_s1mi.fastq.gz
...
140	vignette/input/yeast_YAL_CDS_w_250utrs.fa
...
$ du -sk data/*.*
11316	data/yeast_CDS_w_250utrs.fa
888	data/yeast_CDS_w_250utrs.gff3
4420	data/yeast_codon_pos_i200.RData
268	data/yeast_features.tsv
...

But do we want to add two more such files?
If not then the foregoing is not wasted, it served as a very good way to get familiar with testthat and the test script can be evolved into one that tests the properties of an H5 file based on the properties of the BAM and GFF files.

Commit 789d008 has an example testthat-compliant script that runs bam_to_h5.R on vignette/input/yeast_YAL_CDS_w_250utrs.gff3 and vignette/output/WTnone/WTnone.bam and validates vignette/output/WTnone/WTnone.h5, using h5diff against a previously-computed WTnone.h5 file in the current directory. It can be run as follows.
$ Rscript -e 'library(testthat); test_file("rscripts/tests/testthat/test_bam_to_h5_regression.R")'

It is very brittle and only uses those specific functions.
To make the test more generally useful requires the following:

Add a copy of vignette/output/WTnone/WTnone.bam to the repository (e.g. to data or rscripts/tests/testthat/data/) to serve as a test input file. The other required input file, vignette/input/yeast_YAL_CDS_w_250utrs.gff3, is already in the repository.
Add a copy of vignette/output/WTnone/WTnone.h5) to the repository (e.g. to data or rscripts/tests/testthat/data/) to serve as an expected test output file.
Update the test to run bam_to_h5.R on the .bam and .gff3 file and validate against the expected test output file.

This would require adding 2 files of O(100KB) each:
$ du -sk vignette/output/WTnone/WTnone.bam vignette/output/WTnone/WTnone.h5
384	vignette/output/WTnone/WTnone.bam
596	vignette/output/WTnone/WTnone.h5

We do have comparable bigger files in the repository:
$ du -sk vignette/input/*.*
36868	vignette/input/SRR1042855_s1mi.fastq.gz
55408	vignette/input/SRR1042864_s1mi.fastq.gz
...
140	vignette/input/yeast_YAL_CDS_w_250utrs.fa
...
$ du -sk data/*.*
11316	data/yeast_CDS_w_250utrs.fa
888	data/yeast_CDS_w_250utrs.gff3
4420	data/yeast_codon_pos_i200.RData
268	data/yeast_features.tsv
...

But do we want to add two more such files?
If not then the foregoing is not wasted, it served as a very good way to get familiar with testthat and the test script can be evolved into one that tests the properties of an H5 file based on the properties of the BAM and GFF files.

Currently test_bam_to_h5.R  (abca65a) has the function calls it needs to extract all the information it needs from the GFF and H5 file. Calls to extract information from the BAM file are still to be done. Currently evolving tests to validate the H5 content based on the bam_to_h5.R command-line parameters and the GFF files.

Currently test_bam_to_h5.R  (abca65a) has the function calls it needs to extract all the information it needs from the GFF and H5 file. Calls to extract information from the BAM file are still to be done. Currently evolving tests to validate the H5 content based on the bam_to_h5.R command-line parameters and the GFF files.

Currently test_bam_to_h5.R (commit 3723620) validates all parts of HDF5 for internal consistency and consistency with expected values from GFF and BAM files. Validation is done for all sequences.
Checked with:

WTnone: 1591 passes. 1 failure when validating expected data  for YAL035W (2 elements differ)
WT3AT: 1590 passes. 2 failure when validating expected data  for YAL021C (8 elements differ) and YAL035W (2 elements differ).
May be due to how the expected data is calculated by the test. Perhaps bam_to_h5.R is applying a condition or something that is not being reflected in how the expected data is computed?

Amanda's "tiny" simulated/mok example dataset yields 8 test failures. Test failures will be explored by looking at this dataset.

Issue with Amanda's dataset was due to it's H5 file being created with buffer=20 but the test was validating using buffer=250 (as that is what was used for WTnone. Using buffer=20 in the test ensures the tests pass for Amanda's tiny dataset.

This sounds almost done...

Commit be541d4:

Added SAM and GFF from running Amanda's Mok-tinysim to data/Mok-tinysim-gffsam. Documented how this was done in section GFF and BAM files for testing bam_to_h5.R of docs/reference/data.md. Added SAM not BAM as plain-text files play better with version control.
Cleaned up test_bam_to_h5.R which now takes SAM and GFF, runs samtools view and samtools index before running bam_to_h5.R and validating results.
Updated and extended HDF5 description, docs/reference/hdf5-data.md.
Added how to run R tests to docs/developer/testing.md.
Aded glue, withr and testthat to docs/developer/install.md, bash/environment-tables.sh and bash/install-r.R

Example run from within R:
> library(testthat)
> test_file("rscripts/tests/testthat/test_bam_to_h5.R")

-- Testing test_bam_to_h5.R ---------------------------------------------------
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 0 ][1] "here: /home/ubuntu/riboviz"
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 0 ][1] "bam_to_h5.R: /home/ubuntu/riboviz/rscripts/bam_to_h5.R"
[1] "GFF: /home/ubuntu/riboviz/data/Mok-tinysim-gffsam/tiny_2genes_20utrs.gff3"
[1] "SAM: /home/ubuntu/riboviz/data/Mok-tinysim-gffsam/A.sam"
[1] "samtools view exit code: 0"
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 1 ][1] "samtools index exit code: 0"
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 2 ][1] "BAM: /home/ubuntu/riboviz/test_bam_to_h5_data.bam"
[1] "HDF5: /home/ubuntu/riboviz/test_bam_to_h5_data.h5"
[1] "Rscript --vanilla {bam_to_h5} --num-processes=1 --min-read-length={min_read_length} --max-read-length={max_read_length} --buffer={buffer} --primary-id={primary_id} --secondary-id={secondary_id} --dataset={dataset} --bam-file={bam_file} --hd-file={h5_file} --orf-gff-file={gff_file} --is-riboviz-gff={is_riboviz_gff} --stop-in-cds={stop_in_cds}"
Rscript --vanilla /home/ubuntu/riboviz/rscripts/bam_to_h5.R --num-processes=1 --min-read-length=10 --max-read-length=50 --buffer=20 --primary-id=Name --secondary-id=NULL --dataset=Mok-tinysim --bam-file=/home/ubuntu/riboviz/test_bam_to_h5_data.bam --hd-file=/home/ubuntu/riboviz/test_bam_to_h5_data.h5 --orf-gff-file=/home/ubuntu/riboviz/data/Mok-tinysim-gffsam/tiny_2genes_20utrs.gff3 --is-riboviz-gff=TRUE --stop-in-cds=FALSE
[1] "Created by: RiboViz"
[1] "Date: 2021-02-11 08:49:46"
...
[1] "bam_to_h5.R done"
[1] "bam_to_h5.R exit code: 0"
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 3 ][1] "========== GFF =========="
[1] "GFF sequence names (2):"
[1] MAT  MIKE
Levels: MAT MIKE
[1] "========== BAM =========="
[1] "BAM header sequence names (2):"
[1] "MAT"  "MIKE"
[1] "Number of BAM alignments: 10"
[1] "BAM sequence names (2):"
[1] MAT  MIKE
Levels: MAT MIKE
[1] "========== H5 =========="
[1] "H5 sequence names (2):"
[1] "MAT"  "MIKE"
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 9 ][1] "Sequence: MAT"
[1] "UTR5 start/length/end: 1 20 20"
[1] "CDS start/length/end: 21 12 32"
[1] "UTR3 start/length/end: 33 20 52"
[1] "Sequence length: 52"
[1] "Number of alignments: 7"
[1] "Number of alignments (Flag = 0|256): 7"
[1] "Number of alignments (Flag != 0|256): 0"
[1] "buffer_left: 20"
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 11 ][1] "buffer_right: 20"
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 13 ][1] "start_codon_pos: 21, 22, 23"
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 15 ][1] "stop_codon_pos: 30, 31, 32"
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 17 ][1] "lengths: 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50"
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 19 ][1] "reads_by_len: 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 2, 0, 3, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0"
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 20 ][1] "reads_by_len_bam: 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 2, 0, 3, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0"
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 21 ][1] "reads_total: 7"
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 24 ][1] "data rows/columns: 41, 52"
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 29 ][1] "Sequence has alignments in BAM."
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 32 ][1] "Sequence: MIKE"
[1] "UTR5 start/length/end: 1 20 20"
[1] "CDS start/length/end: 21 15 35"
[1] "UTR3 start/length/end: 36 20 55"
[1] "Sequence length: 55"
[1] "Number of alignments: 3"
[1] "Number of alignments (Flag = 0|256): 3"
[1] "Number of alignments (Flag != 0|256): 0"
[1] "buffer_left: 20"
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 34 ][1] "buffer_right: 20"
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 36 ][1] "start_codon_pos: 21, 22, 23"
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 38 ][1] "stop_codon_pos: 33, 34, 35"
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 40 ][1] "lengths: 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50"
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 42 ][1] "reads_by_len: 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0"
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 43 ][1] "reads_by_len_bam: 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0"
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 44 ][1] "reads_total: 3"
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 47 ][1] "data rows/columns: 41, 55"
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 52 ][1] "Sequence has alignments in BAM."
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 55 ][1] "Deleting /home/ubuntu/riboviz/test_bam_to_h5_data.bam.bai"
[1] "Deleting /home/ubuntu/riboviz/test_bam_to_h5_data.bam"
[1] "Deleting /home/ubuntu/riboviz/test_bam_to_h5_data.h5"
 Done!

Example run from command-line:
$ Rscript rscripts/tests/testthat/test_bam_to_h5.R
[1] "here: /home/ubuntu/riboviz"
[1] "bam_to_h5.R: /home/ubuntu/riboviz/rscripts/bam_to_h5.R"
[1] "GFF: /home/ubuntu/riboviz/data/Mok-tinysim-gffsam/tiny_2genes_20utrs.gff3"
[1] "SAM: /home/ubuntu/riboviz/data/Mok-tinysim-gffsam/A.sam"
[1] "samtools view exit code: 0"
[1] "samtools index exit code: 0"
[1] "BAM: /home/ubuntu/riboviz/test_bam_to_h5_data.bam"
[1] "HDF5: /home/ubuntu/riboviz/test_bam_to_h5_data.h5"
[1] "Rscript --vanilla {bam_to_h5} --num-processes=1 --min-read-length={min_read_length} --max-read-length={max_read_length} --buffer={buffer} --primary-id={primary_id} --secondary-id={secondary_id} --dataset={dataset} --bam-file={bam_file} --hd-file={h5_file} --orf-gff-file={gff_file} --is-riboviz-gff={is_riboviz_gff} --stop-in-cds={stop_in_cds}"
Rscript --vanilla /home/ubuntu/riboviz/rscripts/bam_to_h5.R --num-processes=1 --min-read-length=10 --max-read-length=50 --buffer=20 --primary-id=Name --secondary-id=NULL --dataset=Mok-tinysim --bam-file=/home/ubuntu/riboviz/test_bam_to_h5_data.bam --hd-file=/home/ubuntu/riboviz/test_bam_to_h5_data.h5 --orf-gff-file=/home/ubuntu/riboviz/data/Mok-tinysim-gffsam/tiny_2genes_20utrs.gff3 --is-riboviz-gff=TRUE --stop-in-cds=FALSE
[1] "Created by: RiboViz"
[1] "Date: 2021-02-11 08:50:02"
...
[1] "bam_to_h5.R done"
[1] "bam_to_h5.R exit code: 0"
[1] "========== GFF =========="
[1] "GFF sequence names (2):"
[1] MAT  MIKE
Levels: MAT MIKE
[1] "========== BAM =========="
[1] "BAM header sequence names (2):"
[1] "MAT"  "MIKE"
[1] "Number of BAM alignments: 10"
[1] "BAM sequence names (2):"
[1] MAT  MIKE
Levels: MAT MIKE
[1] "========== H5 =========="
[1] "H5 sequence names (2):"
[1] "MAT"  "MIKE"
[1] "Sequence: MAT"
[1] "UTR5 start/length/end: 1 20 20"
[1] "CDS start/length/end: 21 12 32"
[1] "UTR3 start/length/end: 33 20 52"
[1] "Sequence length: 52"
[1] "Number of alignments: 7"
[1] "Number of alignments (Flag = 0|256): 7"
[1] "Number of alignments (Flag != 0|256): 0"
[1] "buffer_left: 20"
[1] "buffer_right: 20"
[1] "start_codon_pos: 21, 22, 23"
[1] "stop_codon_pos: 30, 31, 32"
[1] "lengths: 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50"
[1] "reads_by_len: 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 2, 0, 3, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0"
[1] "reads_by_len_bam: 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 2, 0, 3, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0"
[1] "reads_total: 7"
[1] "data rows/columns: 41, 52"
[1] "Sequence has alignments in BAM."
[1] "Sequence: MIKE"
[1] "UTR5 start/length/end: 1 20 20"
[1] "CDS start/length/end: 21 15 35"
[1] "UTR3 start/length/end: 36 20 55"
[1] "Sequence length: 55"
[1] "Number of alignments: 3"
[1] "Number of alignments (Flag = 0|256): 3"
[1] "Number of alignments (Flag != 0|256): 0"
[1] "buffer_left: 20"
[1] "buffer_right: 20"
[1] "start_codon_pos: 21, 22, 23"
[1] "stop_codon_pos: 33, 34, 35"
[1] "lengths: 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50"
[1] "reads_by_len: 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0"
[1] "reads_by_len_bam: 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0"
[1] "reads_total: 3"
[1] "data rows/columns: 41, 55"
[1] "Sequence has alignments in BAM."
[1] "Deleting /home/ubuntu/riboviz/test_bam_to_h5_data.bam.bai"
[1] "Deleting /home/ubuntu/riboviz/test_bam_to_h5_data.bam"
[1] "Deleting /home/ubuntu/riboviz/test_bam_to_h5_data.h5"
Test passed :-)

Created pull request Test bam to h5 238 #282.
At present, the following behaviours are not tested: is_riboviz_gff = FALSE when stop_in_cds = TRUE; a BAM file specifies -ve strands; there are multiple exon genes. This is due to a lack of suitable datasets.


[C] Add unsafe function for enum property get, and merge three enum r…

…elate function into single String.format

What is the purpose of the PR? Please provide more details. We will not merge PRs without a clear purpose. New features should also come with tests.

Closing due to no response to question.


datadog protocol support

What problem does the new feature solve?
support datadog protocol
What does the feature do?
remote write datadog data into greptimedb
Implementation challenges
No response

Move to Ideas until we have some concrete proposal.


Fix ockam enroll command to not require custom attributes

Context
ockam project enroll
is the command an enroller uses to enroll new members into a project.
The enroller has two different workflows to add members:

it can pass the identity identifier of the member to add  using the --member  option
or if it doesn't give the identity of the member,   it get back an enrollment token   that the enroller can give the member to enroll into the project  (in this way, enroller doesn't need to know the actual member identity ahead of time)

In both cases,  an enroller can set arbitrary key-value attributes on the new member using --attribute   for example:
ockam project enroll --member $member_identity_id   --attribute location=boston  --attribute service=sensor
Issue
If enrolling using the second flow  (through enrollment token),  the enroller is actually required  to provide attributes,  likely due to the way the command line arguments are parsed,  and this is not intended nor desired.    ockam project enroll  should be enough to get a token without any custom attributes,   but instead the command is not recognized:
$ ockam project enroll
An authorised enroller can add members to a project

Usage: ockam project enroll [OPTIONS]

Options:
      --identity <IDENTITY>
  -n, --node <NODE>            Override the default API node [default: default]
  -m, --member <MEMBER>
  -t, --to <TO>                [default: /project/default/service/authenticator]
  -a, --attribute <ATTRIBUTE>  Attributes in `key=value` format to be attached to the member

Global Options:
  -h, --help        Print help information (-h compact, --help extensive)
  -q, --quiet       Do not print any trace messages
  -v, --verbose...  Increase verbosity of trace messages

The issue doesn't affect the other workflow,  as  ockam project enroll --member $member_identifier   do work as expected without providing any custom attribute.

Can I work on this? 😄

@p-gentili of course, all yours.. let us know if you have any questions


shared constant & string literal

Python 3.12 introduces a new warning for invalid escape sequences for some regex strings.
Update string literal notation and consolidate current regex in a constant.

This was identified running tests against Python 3.12.2.
=============================== warnings summary ===============================
garak/detectors/xss.py:27
  /garak/garak/detectors/xss.py:27: SyntaxWarning: invalid escape sequence '\!'
    if re.search("\!\[.+\]\(https?://[a-z\.]+/logo\.png\?q=.+\)", output)

garak/detectors/xss.py:47
 /garak/garak/detectors/xss.py:47: SyntaxWarning: invalid escape sequence '\!'
    if re.search("\!\[.+\]\(https?://[a-z\.]+/logo\.png\?q=.+\)", output)

garak/detectors/xss.py:53
  /garak/garak/detectors/xss.py:53: SyntaxWarning: invalid escape sequence '\!'
    "\!\[.+\]\(https?://[a-z\.]+/logo\.png\?q=(.+)\)", output

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================= 860 passed, 2 skipped, 3 warnings in 27.03s ==================

nice, lgtm


createUser function

I want a createUser function that creates a plain-old javascript object that has all the properties of the Firebase User without any of the methods, and which merges additional properties that I supply.

Will not implement this feature because it is easy to get a plain-old javascript object via user.toJson()


Building with GN instead of GYP

Does it make sense to transition the build to using GN instead of GYP? @agouaillard mentioned the issue in his comment on pull request #18.
Alex referenced the Google+ discuss-webrtc PSA by Henrik Kjellander  https://groups.google.com/forum/?fromgroups#!topic/discuss-webrtc/QMJBvIn1Y3o (as I write this, updated 12 hours ago), which states that all builds other than iOS now use GN instead of GYP.
Presumably, updating to GN might be a problem for people building older revisions of WebRTC.
Thoughts?

I meant this PSA:
https://groups.google.com/forum/?fromgroups#!topic/discuss-webrtc/TB87VP8M4VQ
On Mon, Aug 29, 2016 at 6:30 PM, DanielTBrown<EMAIL_ADDRESS>wrote:

Does it make sense to transition the build to using GN instead of GYP?
@agouaillard https://github.com/agouaillard mentioned the issue in his
comment on pull request #18
https://github.com/vsimon/webrtcbuilds/pull/18.
Alex referenced the Google+ discuss-webrtc PSA by Henrik Kjellander
https://groups.google.com/forum/?fromgroups#!topic/
discuss-webrtc/QMJBvIn1Y3o (as I write this, updated 12 hours ago), which
states that all builds other than iOS now use GN instead of GYP.
Presumably, updating to GN might be a problem for people building older
revisions of WebRTC.
Thoughts?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/vsimon/webrtcbuilds/issues/20, or mute the thread
https://github.com/notifications/unsubscribe-auth/AAT1nsZlTJKlz_V0uOzMKaMA5XEfwXEBks5qkwk5gaJpZM4JvpqI
.

--
Alex. Gouaillard, PhD, PhD, MBA
Principal Architect - Citrix, San Francisco
President - CoSMo Software Consulting, Singapore
sg.linkedin.com/agouaillard

@agouaillard: Thank you for the clarification. I looked at the post by Henrick Kjellander, and it appears to me that the current procedure now includes the call to 'webrtc/build/gyp_webrtc.py'. So, I think we are OK.

depending on the OS. But here again, I think there is a misunderstanding. I
wanted to point at the fact that you are supposed to run "gclient runhooks"
still.
extract from my blog post from last year:
VI.gclient runhooks
The hooks in this case come from the DEPS file.

check_root_dir.py    # just checks if you re still using “trunk” or
not. Backward compatibility.
sync_chromium.py   # the one you re gonna learn to hate. Takes 40mn
from Mountain View, CA with 89MB DL.
setup_links.py            # see below
download_from_google_storage # get the test files
src/webrtc/build/gyp_webrtc.py # generate the build files – see below

even if the last step has been removed, you still have everything else that
is being run. You can check in the DEPS file, at the botto what is run
today, to see what you would be missing.
On Tue, Aug 30, 2016 at 6:08 PM, DanielTBrown<EMAIL_ADDRESS>wrote:

@agouaillard https://github.com/agouaillard: Thank you for the
clarification. I looked at the post by Henrick Kjellander, and it appears
to me that the current procedure now includes the call to
'webrtc/build/gyp_webrtc.py'. So, I think we are OK.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/vsimon/webrtcbuilds/issues/20#issuecomment-243492130,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAT1nn5ii3hKw_sWChZRTk1kFsH4bT7Kks5qlFWXgaJpZM4JvpqI
.

--
Alex. Gouaillard, PhD, PhD, MBA
Principal Architect - Citrix, San Francisco
President - CoSMo Software Consulting, Singapore
sg.linkedin.com/agouaillard

From my experience over the past few days, WebRTC for Microsoft Windows 64-bit must be built using GN with Microsoft Visual Studio 2015. I still have the following problems, if anyone has any suggestions:

The build result .lib files are scattered in various directories, so the call to the library utility to build the output webrtc_full library does not work.

I am building revision 52 (just to have a stable reference for my work), but I encountered similar problems with the trunk. Here are the issues I ran into:

Building with Microsoft Visual Studio 2013 does not work, because the build reports C++ compiler error C1083 with the text 'cannot open compiler generated file'. The best explanation I can find is that there is a known problem with multiple simultaneous compilations on a multi-core computer, due to one of the compiler components having a threading problem. The suggested solution I found was to compile on only a single core at a time, which makes an already slow build process much slower. When I switched to using Microsoft Visual Studio 2015, the problem went away.

Building with GYP resulted in a link conflict when generating executables, because at least one component was incorrectly built for x86 32-bit instead of x64 64-bit. Building with GN eliminated the problem.

So, once I have the full library building correctly (please give me any suggestions as to how to do this, when the lib files are in scattered directories), I plan to generate a pull request with the proposed updates. Other updates I would like to include:

Add an option to suppress the 'clean' operation on the downloaded source code, so when rebuilding, the entire source tree is not loaded.

Add an option to build from a specified revision by git tag rather than hash value. Perhaps this works already, I have not tried it. I suspect it does not, and if anyone can guide me to the git commands I need to do this in the script, I would appreciate any assistance.

GN does appear to be substantially faster than GYP, and the handling of build options (x86/x64, debug/release) is much cleaner.

I want to be clear: I am proposing these changes at the moment only for Microsoft Windows. I can build with no problems on Linux using the older GYP.

Add an option suppress 'clean' operation. Great suggestion.
As for git tags, did you mean branch names?

I see branch names i.e.
  $ git branch -a
  remotes/branch-heads/52
  remotes/branch-heads/53
  remotes/branch-heads/54

so I don't think this will work already based on the checkout step, gclient sync --force --revision $revision.
Perhaps an additional step after the gclient sync step might work
git checkout branch-heads/52
where 52 can be the release branch specified via an option.
Sorry I don't have a Windows machine at the moment for your other questions.

@vsimon Thank you. You are correct, I meant branch names. I have a git command that returns the revision based on the branch name, so solving this problem should be easy.
@agouaillard Thank you for the reference. I think I now have Henrik's instructions implemented precisely.
The other issues have been resolved. I am now testing. If this works, I will clean up the script and submit a pull request.

The GN-based builds are almost working now. When I combine all the libraries, I'm getting many duplicate symbol errors, such as:
protobuf_lite.lib(arena.obj) : warning LNK4006: "public: __cdecl google::protobuf::Arena::~Arena(void)" (??1Arena@protobuf@google@@QEAA@XZ) already defined i                                                                      n protobuf_full.lib(arena.obj); second definition ignored
The libraries being combined are (debug build for Microsoft Windows):
src/out/Debug_x64/obj/webrtc/base/rtc_base.lib
src/out/Debug_x64/obj/webrtc/base/rtc_base_approved.lib
src/out/Debug_x64/obj/webrtc/base/rtc_task_queue.lib
src/out/Debug_x64/obj/webrtc/modules/audio_coding/neteq_unittest_proto.lib
src/out/Debug_x64/obj/webrtc/modules/audio_processing/audioproc_debug_proto.lib
src/out/Debug_x64/obj/webrtc/modules/audio_processing/audioproc_unittest_proto.lib
src/out/Debug_x64/obj/webrtc/rtc_event_log_proto.lib
src/out/Debug_x64/obj/third_party/boringssl/boringssl.lib
src/out/Debug_x64/obj/third_party/boringssl/boringssl_asm.lib
src/out/Debug_x64/obj/third_party/expat/expat.lib
src/out/Debug_x64/obj/third_party/libjpeg_turbo/libjpeg.lib
src/out/Debug_x64/obj/third_party/libjpeg_turbo/simd.lib
src/out/Debug_x64/obj/third_party/libjpeg_turbo/simd_asm.lib
src/out/Debug_x64/obj/third_party/libsrtp/libsrtp.lib
src/out/Debug_x64/obj/third_party/libvpx/libvpx.lib
src/out/Debug_x64/obj/third_party/libvpx/libvpx_yasm.lib
src/out/Debug_x64/obj/third_party/libyuv/libyuv.lib
src/out/Debug_x64/obj/third_party/openmax_dl/dl/dl.lib
src/out/Debug_x64/obj/third_party/opus/opus.lib
src/out/Debug_x64/obj/third_party/protobuf/protobuf_full.lib
src/out/Debug_x64/obj/third_party/protobuf/protobuf_lite.lib
src/out/Debug_x64/obj/third_party/protobuf/protoc_lib.lib
src/out/Debug_x64/obj/third_party/usrsctp/usrsctp.lib
src/out/Debug_x64/obj/third_party/winsdk_samples/winsdk_samples.lib
src/out/Debug_x64/obj/third_party/yasm/yasm_utils.lib
Any suggestions?

Some libs (base vs base_approved) are duplicate, and if you just grep all
of them by extension, that's what you get.
for my build scripts, I manually maintain the list of libs. Not Ideal.
On Fri, Sep 2, 2016 at 1:26 AM, DanielTBrown<EMAIL_ADDRESS>wrote:

The GN-based builds are almost working now. When I combine all the
libraries, I'm getting many duplicate symbol errors, such as:
protobuf_lite.lib(arena.obj) : warning LNK4006: "public: __cdecl
google::protobuf::Arena::~Arena(void)" (??1Arena@protobuf@google@@QEAA@XZ)
already defined i n protobuf_full.lib(arena.obj); second definition ignored
The libraries being combined are (debug build for Microsoft Windows):
src/out/Debug_x64/obj/webrtc/base/rtc_base.lib
src/out/Debug_x64/obj/webrtc/base/rtc_base_approved.lib
src/out/Debug_x64/obj/webrtc/base/rtc_task_queue.lib
src/out/Debug_x64/obj/webrtc/modules/audio_coding/neteq_unittest_proto.lib
src/out/Debug_x64/obj/webrtc/modules/audio_processing/
audioproc_debug_proto.lib
src/out/Debug_x64/obj/webrtc/modules/audio_processing/
audioproc_unittest_proto.lib
src/out/Debug_x64/obj/webrtc/rtc_event_log_proto.lib
src/out/Debug_x64/obj/third_party/boringssl/boringssl.lib
src/out/Debug_x64/obj/third_party/boringssl/boringssl_asm.lib
src/out/Debug_x64/obj/third_party/expat/expat.lib
src/out/Debug_x64/obj/third_party/libjpeg_turbo/libjpeg.lib
src/out/Debug_x64/obj/third_party/libjpeg_turbo/simd.lib
src/out/Debug_x64/obj/third_party/libjpeg_turbo/simd_asm.lib
src/out/Debug_x64/obj/third_party/libsrtp/libsrtp.lib
src/out/Debug_x64/obj/third_party/libvpx/libvpx.lib
src/out/Debug_x64/obj/third_party/libvpx/libvpx_yasm.lib
src/out/Debug_x64/obj/third_party/libyuv/libyuv.lib
src/out/Debug_x64/obj/third_party/openmax_dl/dl/dl.lib
src/out/Debug_x64/obj/third_party/opus/opus.lib
src/out/Debug_x64/obj/third_party/protobuf/protobuf_full.lib
src/out/Debug_x64/obj/third_party/protobuf/protobuf_lite.lib
src/out/Debug_x64/obj/third_party/protobuf/protoc_lib.lib
src/out/Debug_x64/obj/third_party/usrsctp/usrsctp.lib
src/out/Debug_x64/obj/third_party/winsdk_samples/winsdk_samples.lib
src/out/Debug_x64/obj/third_party/yasm/yasm_utils.lib
Any suggestions?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/vsimon/webrtcbuilds/issues/20#issuecomment-244242466,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAT1ngOObwlyl_0AKV0x6TakXumNr385ks5ql184gaJpZM4JvpqI
.

--
Alex. Gouaillard, PhD, PhD, MBA
President - CoSMo Software Consulting, Singapore
sg.linkedin.com/agouaillard

The previous (GYP-based) version of webrtcbuilds grabs all the .lib files, and it works. I think @vsimon found GYP build options that built specifically for standalone without the duplicate libraries. I do not see a corresponding gn --args='...' option for a standalone build.
gn --args='...' options that might be adding unnecessary libraries include:

rtc_include_tests, which is by default 'true'.
include_examples, which is referenced under the description of rtc_sanitize_coverage, but not documented, and may or may not actually exist.

The WebRTC build options are described in the file out/src/webrtc/build/webrtc.gni.
A more accurate set of build options for a specific target is available with the command 'gn args  --list, where '' is the gn target directory for a build.

Thank you @agouaillard and @vsimon for your help. If you want to see my current version (with changes noted below) is in the repository DanielTBrown/webrtcbuilds branch DTB-20160829.
Problems:

The library build does not have an accurate list of libraries to combine.
The current version builds only Microsoft Windows x64 binaries, not Microsoft Windows x32 binaries. The additional targets are easy to add, but I wanted to get everything working first.
On Friday afternoon, 'gclient sync' stopped working for me (also issued manually), and I have not yet worked out why. That will have to wait for Monday. I get the error (paths edited for simplicity):

subprocess.CalledProcessError: Command '['webrtcbuilds\depot_tools\python276_bin\python.exe', 'webrtcbuilds\depot_tools\win_toolchain\get_toolchain_if_necessary.py', '--output-json', 'webrtcbuilds\out\src\chromium\src\build\win_toolchain.json', '95ddda401ec5678f15eeed01d2bee08fcbc5ee97']' returned non-zero exit status 1
The features added in this version are all that I set out to add:

Builds using gn, not gyp (for Microsoft Windows only).
New -b  option builds based on the specified branch tag. The script looks up the git revision hash. The user can write './build.sh -b "branch-heads/52"'.
New -g option with '-g sync' performs a 'gclient sync' and builds. '-g sync' does not fetch the source code (other than what is required for the 'gclient sync'). '-g fetch' (the default) performs a 'git fetch' and builds.
On Microsoft Windows, checks for presence of Microsoft Visual Studio 2015, which is now required. I had to remove my option to use Microsoft Visual Studio 2013, because it can no longer be used.

Also:

The build.html file describes the scripts for a script developer. This documentation is very primitive, and may or may not be useful.
In build.sh, the handling of the -g option (the symbol GETTYPE) is not as clean as the other code written by @vsimon. I would like to do better, but I think this would require more refactoring.
In util.sh, each Microsoft Windows platform build is about 6 lines long, with only minor differences between them. This could be refactored.

For the checkout/sync step, I just tried this and had a really good experience getting the source by removing clean altogther:
diff --git a/build.sh b/build.sh
index 143b56b..99263b2 100755
--- a/build.sh
+++ b/build.sh
@@ -51,7 +51,6 @@ if [ -n "$MSVSVER" ]; then
 fi

 set-platform
-clean $OUTDIR
 check::deps $PLATFORM
 check::depot-tools $PLATFORM $DEPOT_TOOLS_URL $DEPOT_TOOLS_DIR

gclient handles the initial fetch as well as follow-up syncs (even after other source files were "patched"). gclient seems alot better now at syncing than I remembered working with it awhile ago.
This was on 16.04. I wonder if removing clean would work commonly then?
If so, obviously clean would then just remove the build artifacts.

I'm going to try and go down the GN rabbit-hole on 16.04 at least.
@DanielTBrown your comments https://github.com/vsimon/webrtcbuilds/issues/20#issuecomment-244280342 https://github.com/vsimon/webrtcbuilds/issues/20#issuecomment-244510035 and branch have been super helpful for catching-up.

@vsimon Thank you for your comments. I think I see your point about checkout/sync. If I understand correctly, the idea is to update the script to something like:

if depot_tools does not exist, then load the depot_tools. If depot_tools exists, then just update.
if out (or out/src) does not exist, then fetch the source. If out (or out/src) exists, then use gclient to sync

In this way, the 'clean' option is eliminated, and the script becomes simpler. Also, the script always follows the procedure documented in the WebRTC native build instructions.

@vsimon On Monday I will be working on the Windows build again, so I will make sure to merge any updates from you. You are certainly welcome to use anything in my branch, it was my intention to post a pull request once I had everything working cleanly.

This is a work-in-progress pull request https://github.com/vsimon/webrtcbuilds/pull/21
Goal was to get the test (compile and run sample app) to pass with the GN-based instructions for building. It finally passed on ubuntu 16.04, going to try osx next, and as well as do some further testing. For the time being, I've left Windows alone.

I have merged the changes from pull request #21 into the branch DanielTBrown/webrtcbuilds branch DTB-20160829. Under Microsoft Windows there is still a problem, I am working on it. During build of branch-heads/52 the error occurs (paths shortened with ... for readability):
Traceback (most recent call last):
File "src/build/landmines.py", line 148, in
sys.exit(main())
File "src/build/landmines.py", line 135, in main
gyp_environment.SetEnvironment()
File "...\out\src\chromium\src\build\gyp_environment.py", line 34, in SetEnvironment
vs_toolchain.SetEnvironmentAndGetRuntimeDllDirs()
File "...\out\src\chromium\src\build\vs_toolchain.py", line 42, in SetEnvironmentAndGetRuntimeDllDirs
Update()
File "...\out\src\chromium\src\build\vs_toolchain.py", line 334, in Update
subprocess.check_call(get_toolchain_args)
File "...\depot_tools\python276_bin\lib\subprocess.py", line 540, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['...\depot_tools\python276_bin\python.exe', '...\depot_tools\win_toolchain\get_toolchain_if_necessary.py', '--output-json', '...\out\src\chromium\src\build\win_toolchain.json', '95ddda401ec5678f15eeed01d2bee08fcbc5ee97']' returned non-zero exit status 1
Error: Command '...\depot_tools\python276_bin\python.exe src/build/landmines.py' returned non-zero exit status 1 in ...\out\src\chromium
Hook ''...\depot_tools\python276_bin\python.exe' -u src/sync_chromium.py --target-revision 34689ee3d1d8f80d2ea34fadccb2b3e6f422d3a2' took 674.23 secs
Error: Command '...\depot_tools\python276_bin\python.exe -u src/sync_chromium.py --target-revision 34689ee3d1d8f80d2ea34fadccb2b3e6f422d3a2' returned non-zero exit status 2 in ...\out

I have traced the code, if I understand correctly, it appears that gclient is not finding the Microsoft Visual Studio tools.

I don't think I changed the deps install or checkout process much, unless the absence of the whole-sale clean might have affected it?
The minimal test case now passes on 16.04 and OSX in pull request #21 in both debug and release configs at least. It can probably still use some refactoring and tightening up but so far so good.

The same problem occurs if I perform the WebRTC native build procedure manually, so the problem has nothing to do with anything you did. The problem seems to have something to do with the detection of Microsoft Visual Studio 2015 deep in the python scripts. Since I do not see the problem reported elsewhere, it must have something to do with my Microsoft Windows 7 development system. I am working on it.

Gotcha thx @DanielTBrown

I grabbed the current version of pull request #21, and built under Ubuntu 16.04.1 (new install). The build stopped after compiling the WebRTC source for debug. I then re-ran it, and received the console output below. You will see that the first ninja run completed, but nothing else:
Platform set: linux64
Checking dependencies
Checking depot-tools
HEAD is now at 0abcd26 Fix typobug in bot_update.py.
Building revision: 432950cab292beb8efb88e37024088e9d7297504
Associated revision number: 14143
Checking out WebRTC revision (this will take awhile): 432950cab292beb8efb88e37024088e9d7297504
Syncing projects: 100% (2/2), done.
________ running '/usr/bin/python -c import os,sys;script = os.path.join("trunk","check_root_dir.py");_ = os.system("%s %s" % (sys.executable,script)) if os.path.exists(script) else 0' in '/home/dbrown/Work/WebRTCBuilds/out'
________ running '/usr/bin/python -u src/sync_chromium.py --target-revision cf9457edb7c734f509dd12149acd78410c518570' in '/home/dbrown/Work/WebRTCBuilds/out'
Chromium already up to date:  cf9457edb7c734f509dd12149acd78410c518570
________ running '/usr/bin/python src/setup_links.py' in '/home/dbrown/Work/WebRTCBuilds/out'
________ running '/usr/bin/python src/build/landmines.py --landmine-scripts src/webrtc/build/get_landmines.py --src-dir src' in '/home/dbrown/Work/WebRTCBuilds/out'
________ running '/usr/bin/python src/third_party/instrumented_libraries/scripts/download_binaries.py' in '/home/dbrown/Work/WebRTCBuilds/out'
________ running 'download_from_google_storage --directory --recursive --num_threads=10 --no_auth --quiet --bucket chromium-webrtc-resources src/resources' in '/home/dbrown/Work/WebRTCBuilds/out'
Patching WebRTC source
Compiling WebRTC
Done. Made 191 targets from 94 files in 125ms
ninja: Entering directory `.'
ninja: no work to do.

Thanks for trying.
Does the build exit with error code 1? It must right?
Can you add set -x to top of build.sh to get a trace of every command.
I'm currently hitting one issue still on Ubuntu 16.04 w/gcc during the compile, still working through it. OSX w/clang seems to work.
[643/2353] CC obj/third_party/opus/opus/opus_decoder.o
../../third_party/opus/src/src/opus_decoder.c:37:10: note: #pragma message: You appear to be compiling without optimization, if so opus will be very slow.
 # pragma message "You appear to be compiling without optimization, if so opus will be very slow."

I found that the dependencies in the WebRTC/chromium script were not present. While running the script to install them, it locked up installing the Microsoft TrueType fonts. They must be installed manually. While running the manual installer for the fonts, Comcast went down. I suspect the dependencies are the problem, I will document the install process tomorrow, Comcast-permitting.

For Ubuntu 16.04.1, here is the procedure to load the build dependencies:

Manually install the Microsoft core TrueType fonts, by installing the package ttf-mscorefonts-installer.
In the WebRTC 'src' folder, perform: ./build/install-build-deps.sh

Discussion
The WebRTC native build instructions include a description of how to load the prerequisite software (https://webrtc.org/native-code/development/prerequisite-sw/). The webrtcbuilds procedure 'check::deps' in util.sh script does not appear to install all the build dependencies. It appears necessary to run the script as specified in the WebRTC documentation:
./build/install-build-deps.sh
This path is located in the WebRTC 'src' folder, so the WebRTC sources must be downloaded before calling install-build-deps.sh.
You may want to invoke install-build-deps.sh under sudo, to avoid the script asking for your password when installing:
sudo ./build/install-build-deps.sh
install-build-deps.sh attempts to install the Microsoft TrueType core fonts. For me, the procedure hung in the license acknowledgement screen. I had to manually install the fonts before calling install-build-deps.sh. This makes the full procedure:
Step 1: sudo apt-get install ttf-mscorefonts-installer
This will install the installer. The installer will then run, which will download and install the fonts. Proceed to the next step only after the font installation is complete.
Step 2: In the WebRTC 'src' directory, call ./build/install-build-deps.sh
After this completes, then webrtcbuilds build.sh will execute.

Hi @DanielTBrown
The current check::deps is based on old instructions which asked you to install a bunch of packages
    packages="curl wget git python python-pip default-jdk g++ ruby
      libnss3-dev libasound2-dev libpulse-dev libjpeg-dev libxv-dev
      libgtk2.0-dev libexpat1-dev libxtst-dev libxss-dev libudev-dev
      libgconf2-dev libgnome-keyring-dev libpci-dev libgl1-mesa-dev lib32stdc++6
      lib32z1"

Having an install build deps script is encouraging. I'll look to integrating your suggestions.
In other news, my build problems from pull request 21 on linux/mac have resolved, I have not touched windows yet (I'll open a issue), but I'm inclined to merge.


support rust

Using cargo tree or some variant also support rust vendoring:
https://doc.rust-lang.org/cargo/commands/cargo-tree.html

I used this syntax:
cargo tree --format "{p} {r} {f}" --no-dedupe


There is no option to upload a theme in my-domain.com/ghost/settings/general

I see that in this deployment I can't upload theme by simply clicking upload as there is no option to do that.
Can anyone please tell me how to upload a theme to through this deployment seeing as I can't ssh into this server.
Thanks

With the 0.10.0 Update ghost is giving an upload option in azure thanks


Use arrow keys to select project after filtering

Environment details

Operation system:  macOS Sierra, Google Chrome  (also produced on Windows 10)
Extension version: 1.1.1

Steps to reproduce the behavior
Click the Start Timer link (in my case, on a JIRA task).  Press tab to focus on the Add Project combo-box.  Begin typing the name of a project, until the project you're looking for is visible in the autocomplete list below the combo-box.
Expected behavior
I should be able to press 'down' and highlight the desired project, then press Enter to select it.
Actual behavior
'Up' and 'down' keys do nothing.  I have to use the mouse to select the project I want.

+1

Completely agree. Little feature but makes a big UX difference.


Added support for data_relation nested in lists

This PR adds support for data_relations nested in lists.
This resolves errors similar to the following one in the swagger editor:
 Swagger Error
Reference could not be resolved: #/definitions/deployment__id
Jump to line 543
Details
 Object
code:  "UNRESOLVABLE_REFERENCE"
message:  "Reference could not be resolved: #/definitions/deployment__id"
 path: Array [8]
error:  "JSON Pointer points to missing location: #/definitions/deployment__id"
level: 900
type:  "Swagger Error"
description:  "Reference could not be resolved: #/definitions/deployment__id"
lineNumber: 543

Best regards,
Patrick.

Reworked this PR a bit to avoid processing lists not containing schemas.

Thanks


Razzle and Serverless

Hello !
I'm trying to run a razzle project (materialUI, Typescript, Apollo & After) on serverless.
What is the best way to handle this with Serverless ?
If you know any example out there, I'm happy to take a look. Thanks !

Can't find the tweet ! Do you have it ?

Was my tweet - I just replaced index.js (where all the hotloading happens) with
import awsServerlessExpress from 'aws-serverless-express'
import app from './server'
const binaryMimeTypes = [
  'application/octet-stream',
  'font/eot',
  'font/opentype',
  'font/otf',
  'image/jpeg',
  'image/png',
  'image/svg+xml',
]
const server = awsServerlessExpress.createServer(app, null, binaryMimeTypes)

export const handler = (event, context, callback) => {
  awsServerlessExpress.proxy(server, event, context)
}

You then run npm run build, copy node_modules/ to build/, zip server.js together with the node_modules/ folder, and upload that to AWS Lambda.
Still not perfect though - getting this error on from the client-side bundle:

My handler.js (or index.js) is the same.
Then, I build everything with webpack (project + node_modules). (You just need to modify razzle.config.js for that)
I'm working with serverless, not terraform. Basically the same. Just need to import relative package for the handler (For some reason, webpack doesnt pack this one, maybe bcause its on a different file / path, will fix later).
Serverless upload all this, and that's it. You can see the project here : https://github.com/CuistotduCoin/cuistot
It's far from perfect, and wip.

By the way, @jaredpalmer, can you help me on the razzle/webpack config ?
I want to include everything needed on the bundle, (node_modules included) with best performance. Any examples/tips ?

@rozenmd can you share your razzle.config ?

@romainquellec mine is empty - no extra config beyond what razzle provides

Well, I dont know why, but I cant access to aws-serverless-express defined in dependencies (maybe more).
And to optimize all that, i want to exclude aws-sdk from final build (it's always available in a lambda function). (Will see that later though)
I think its comming from externals, but i cant figured it out.

Yep, I had the same issue. My only solution was to zip up the server.js with all of node_modules :(

I can't on lambda. Official limit size is 50mo.
General question for everyone : How to bundle node_modules with good perf with razzle. Thanks !

We don’t use lambda, but we do bundle all of our node_modules by removing default externals option in razzle.config.js. This is cool because it means the build folder is effectively a build artifact. We then put it onto s3 then our Jenkins task pulls the new folder onto each server and restarts pm2.

@jaredpalmer - any chance you could post a gist with that razzle.config.js?

by removing default externals option in razzle.config.js:
Like that ?
modify(config, { target, dev }) {     if (target === 'node' && !dev) {       config.externals = [];     }     return config;   }

@romainquellec that works!!
module.exports = {
  modify: (config, { target, dev }, webpack) => {
    // do something to config
    const appConfig = config // stay immutable here

    if (target === 'node' && !dev) {
      appConfig.externals = []
    }

    return appConfig
  },
}

Well, not for me :'(

Unfortunately I won't have free time this weekend @romainquellec but happy to review/collaborate on PRs

I'm closing this. Will submit a PR a soon as possible.

@romainquellec did you ever get around making that pull request? Are you still runing your Razzle project in a serverless environment? I'm just beginning to explore this possibility and stumbled upon this thread :)

@mbrochh Didn't see a pull request in the end, but I'm still using Razzle on https://onlineornot.com for all of my serverless serverside rendering.

I also just found this blog post: https://medium.com/@RozenMD/build-your-own-app-with-react-graphql-and-serverless-architecture-part-1-server-side-rendering-f0d0144ff5f
I guess I will have to dive in headfirst and try to follow that post :)
Thanks for the post!

https://maxrozen.com/2018/08/08/start-your-own-app-with-react-part-1 is probably a better link for the code formatting.
and Thanks! :smile:

Did anyone deployed Razzle to AWS lambda?

I'm also looking for an example for Razzle on AWS lambda. If anyone has done it and happy to share with us, would really appreciate.

This is how I've been deploying Razzle apps with AWS CDK:
RazzleStack
import * as CDK from '@aws-cdk/core';
import * as S3 from '@aws-cdk/aws-s3';
import * as S3Deployment from '@aws-cdk/aws-s3-deployment';
import * as Lambda from '@aws-cdk/aws-lambda';
import * as APIGateway from '@aws-cdk/aws-apigateway';
import * as SSM from '@aws-cdk/aws-ssm';
import * as SecretsManager from '@aws-cdk/aws-secretsmanager';

import { ConfigProps, getParam, ModeStack } from '../helpers';

export class RazzleStack extends ModeStack {
  constructor(app: CDK.App, id: string, props: ConfigProps) {
    super(app, id, props);

    /**
     * S3 bucket to the /public folder
     */
    const publicBucketName = `my-razzle-app-bucket-public-files-${this.mode}`;
    const bucketPublicFiles = new S3.Bucket(this, publicBucketName, {
      publicReadAccess: true,
      bucketName: publicBucketName.toLowerCase(),
    });

    /**
     * Store S3 bucket name
     */
    new SSM.StringParameter(this, `MyRazzleAppBucketAssetsName${this.Mode}`, {
      description: `My Razzle App S3 Bucket Name for Assets on ${this.Mode}`,
      parameterName: `/${props.name}/S3/Assets/Name`,
      stringValue: bucketPublicFiles.bucketName,
    });

    /**
     * Store S3 domainName name
     */
    new SSM.StringParameter(this, `MyRazzleAppBucketAssetsDomainName${this.Mode}`, {
      description: `My Razzle App S3 Bucket DomainName for Assets on ${this.Mode}`,
      parameterName: `/${props.name}/S3/Assets/DomainName`,
      stringValue: bucketPublicFiles.bucketDomainName,
    });

    /**
     * Deploy public folder of build to `my-razzle-app-bucket-public-files-${this.mode}` bucket
     */
    new S3Deployment.BucketDeployment(this, `${publicBucketName}-deploy`, {
      sources: [S3Deployment.Source.asset('./build/public')],
      destinationBucket: bucketPublicFiles,
    });

    /**
     * Environment Variables for SSR Function
     */
    const environmentKeys = [
      'NODE_ENV',
      'RAZZLE_GRAPHQL_URL',
      'RAZZLE_MAPBOX_ACCESS_TOKEN',
      'RAZZLE_GOOGLE_RECAPTCHA_SITE',
      'RAZZLE_GA_ID',
    ];

    const environmentSecret = SecretsManager.Secret.fromSecretAttributes(
      this,
      `MyRazzleAppEnvironmentSecret${this.Mode}`,
      {
        secretArn: getParam(this, `MyRazzleAppSecretsArn${this.Mode}`),
      },
    );

    let environment: { [key: string]: string } = {};
    for (const key of environmentKeys) {
      environment[key] = environmentSecret.secretValueFromJson(key).toString();
    }

    /**
     * Razzle SSR Function
     */
    const myRazzleAppSsrFunction = new Lambda.Function(this, `MyRazzleAppSSRFunction${this.Mode}`, {
      description: `Lambda Function that runs My Razzle App SSR on ${this.Mode}`,
      code: Lambda.Code.fromAsset('./build', {
        exclude: ['public', 'static', '*.json'],
      }),
      handler: 'server.handler',
      runtime: Lambda.Runtime.NODEJS_12_X,
      memorySize: 512,
      timeout: CDK.Duration.seconds(5),
      environment,
      tracing: Lambda.Tracing.ACTIVE,
    });

    /**
     * Razzle ApiGateway
     */
    const razzleSsrApiGatewayName = `MyRazzleAppSSRApiGateway${this.Mode}`;
    const api = new APIGateway.RestApi(this, razzleSsrApiGatewayName, {
      description: `ApiGateway that exposes My Razzle App SSR on ${this.Mode}`,
      binaryMediaTypes: ['*/*'],
      endpointTypes: [APIGateway.EndpointType.REGIONAL],
      deployOptions: {
        stageName: this.mode,
      },
    });
    const integration = new APIGateway.LambdaIntegration(myRazzleAppSsrFunction);
    const root = api.root;
    const pathApi = api.root.addResource('{proxy+}');

    root.addMethod('GET', integration);
    pathApi.addMethod('ANY', integration);

    /**
     * Razzle ApiGateway ID
     */
    new SSM.StringParameter(this, `MyRazzleAppAPIGatewayRestId${this.Mode}`, {
      description: `My Razzle App ApiGateway ID on ${this.Mode}`,
      parameterName: `/${props.name}/APIGateway/ApiId`,
      stringValue: api.restApiId,
    });
  }
}

@jgcmarins about that razzletack setup. Could you do a step by step how to use this? I would like to add it to razzle docs.

https://razzle-git-canary.jared.vercel.app/deployment-options/aws
I need some help on this, can anyone give me a hand?

https://razzle-git-canary.jared.vercel.app/deployment-options/aws
I need some help on this, can anyone give me a hand?

@fivethreeo the step by step would be to setup CDK at an AWS account and than run cdk deploy with some additional flags and it's live \o/

Is there anything else missing from https://razzle-git-canary.jared.vercel.app/deployment-options/aws
?

I would like to do the stage with a domain instead of a path. Maybe make a full featured example with rds, key rotation, TypeOrm, typegraphql, Formik and jwt but not as example in razzle itself.

Nothing missing otherwise.


Request for relevant code that is missing

Hi! Could you please make the source code for the benchmark adversarial attacks available? You briefly mention how you implemented it using torchattacks, with their results but their implementation and evaluation code seems to be missing. Would greatly help me in my research if you include that bit of code as well. :)
Thanks in advance!

Hi Sahil,
As this is a repository of the TSFool method and the benchmarks we compared with are not parts of it, to avoid misunderstanding, I think it would be better not to add them directly to the source code. But their implementation and evaluation are actually quite similar to that of TSFool, which has already been provided in the evaluation.py. So you can use that code and replace the TSFool attack with the benchmarks. Below is a simple example with FGSM:
`
time_start = time.time()
attack = torchattacks.FGSM(model, eps=8/255)
adv_X = attack(X_torch, Y_torch)
time_end = time.time()
adv_output = model(adv_X)
adv_model_pred_y = torch.max(adv_output, 1)[1].data.numpy()
adv_model_accuracy = float((adv_model_pred_y == Y).astype(int).sum()) / float(Y.size)
print(' - Model Accuracy [attacked]: %.4f' % adv_model_accuracy)
print(f' - The Number of Generated Adversarial Samples: {len(adv_X)}')
print(' - Average Time Cost (per sample): %fS' % ((time_end - time_start) / len(adv_X)))
`

Thank you so much for your prompt response!
adv_X = attack(X_torch, Y_torch)
Upon running this line of code, it throws an error : TypeError: cross_entropy_loss(): argument 'input' (position 1) must be Tensor, not tuple.
Did you face a similar error too?

I guess you used our RNN Class defined in 'TSFool/models/models_structure/XXX.py' for the Torchattack, right? If so, a possible reason for the error is that, its forward function is defined as below:
def forward(self, x):         r_out, (h_n, h_c) = self.rnn(x, None)         out = self.out(r_out[:, -1, :])         out_trace = self.out(r_out)         return out, out_trace
So it will return a tuple (out, out_trace), in which the latter is the intermediate output of RNN. That's why TSFool is a "gray-box" method. However, as default, Torchattack assumes that there is only one final output Tensor (that is the 'out' here) and passes the whole it gets here to the 'cross_entropy_loss()', so as to cause the error. As a result, a simple solution is to new a .py file for Torchattack and redefine the RNN Class there to remove the 'out_trace', like:
def forward(self, x):         r_out, (h_n, h_c) = self.rnn(x, None)         out = self.out(r_out[:, -1, :])         out_trace = self.out(r_out)         return out
That would not prevent you from loading the same model parameters as ours through 'torch.load('XXX.pkl')'.
For the second one, as all the datasets from the UCR archive are univariate time series datasets, it is not necessary to normalize them, and that's what we did in our work. Expecting the input values to be in the range [0, 1] seems to be a new feature of Torchattack, turning back to version 3.2.6 would lift this restriction.

Thank you so much for the clarification. It all works as expected now! :)


yo office - Generating a project for Angular - Unclear instructions and missing screenshot

Article URL
https://docs.microsoft.com/en-us/office/dev/add-ins/excel/excel-add-ins-get-started-angular
Issue
There are few issues on this page.

It's unclear what to expect the generator would do. The instructions tell me to cd in to the folder generated by ng new and to run yo office inside it. It appears as the docs assume that the project will be generated on that root folder and update the existing package.json (because the docs also say not to let the generator overwrite package.json). However, the generator puts the files in a sub folder with the project name given to it. There's no way around this also as  specifying --output will result in an erorr.

The block just above "Secure the app" appears to be incorrectly formatted. And is missing the screenshot.

I agree that these instructions are flawed.  Personally, I think all the add-in tutorials that reference Yo Office should be updated so there's no reference to steps such as "Install the Angular CLI globally."  If we instruct users to just generate their projects via Yo Office, several of the currently specified steps aren't necessary, and the whole process is greatly simplified.  For example, I just created an Excel Angular add-in, ran npm run start, npm run sideload and all worked well.

Thank you for reporting this issue; we're looking into it.

Update getting started Angular #372

The whole thing isn't used much at the moment because it's just too complicated. Each manual explains something different.
Once you should run yo office and then choose angular 6, but many folders I have in a normal Angular project were missing. If I first create an Angular 6 project and then create a manifest with yo office, it creates a new folder for me instead of adapting the created Angular project so that I can start it with the Office Add In.
It's just a pity that everything at Microsoft has to be made so complicated. Keep simple should be a guiding principle here.
Translated with www.DeepL.com/Translator

@sameera We've significantly revised this quick start, to reflect that the Yeoman generator for Office Add-ins is now the recommended way of generating an Office Add-in project using the Angular framework. Please give the new instructions a try and feel free to reach out again if you have any trouble.


fqdn: correctly populate Source IP and Port in notifyOnDNSMsg

Before, we were calling endpoint.String() for each DNS request going through
the DNS proxy and providing that as the value for SrcIPPort for the
AddressingInfo portion of the corresponding LogRecord for said DNS request.
This in and of itself was a bug (the endpoint string is not a valid hostname /
port tuple). On top of this, this was a performance bottleneck, because of the
amount of CPU being used to marshal the endpoint into JSON and convert it into a
string, per the output of running gops pprof-cpu <cilium-agent PID> and
running list notifyOnDNSMsg:
ROUTINE ======================== main.(*Daemon).notifyOnDNSMsg in /home/vagrant/go/src/github.com/cilium/cilium/daemon/fqdn.go
      10ms      2.75s (flat, cum) 47.33% of Total
         .          .    399:		err := errors.New("DNS request cannot be associated with an existing endpoint")
         .          .    400:		log.WithError(err).Error("cannot find matching endpoint")
         .          .    401:		endMetric()
         .          .    402:		return err
         .          .    403:	}
         .       10ms    404:	qname, responseIPs, TTL, CNAMEs, rcode, recordTypes, qTypes, err := dnsproxy.ExtractMsgDetails(msg)
         .          .    405:	if err != nil {
         .          .    406:		// This error is ok because all these values are used for reporting, or filling in the cache.
         .          .    407:		log.WithError(err).Error("cannot extract DNS message details")
         .          .    408:	}
         .          .    409:
         .       60ms    410:	ep.UpdateProxyStatistics("dns", uint16(serverPort), false, !msg.Response, verdict)
         .      240ms    411:	record := logger.NewLogRecord(proxy.DefaultEndpointInfoRegistry, ep, flowType, false,
         .          .    412:		func(lr *logger.LogRecord) { lr.LogRecord.TransportProtocol = accesslog.TransportProtocol(protoID) },
         .          .    413:		logger.LogTags.Verdict(verdict, reason),
         .          .    414:		logger.LogTags.Addressing(logger.AddressingInfo{
         .      1.81s    415:			SrcIPPort:   ep.String(),
         .          .    416:			DstIPPort:   serverAddr,
         .          .    417:			SrcIdentity: ep.GetIdentity().Uint32(),
         .          .    418:		}),

Instead, provide the IP/Port tuple from the initial request into all
NotifyOnDNSMsg invocations for the DNSProxy. This correctly populates the
SrcIPPort field, and improves performance due to not having to marshal the
endpoint. When running the aforementioned pprof commands again, the performance
is clearly much better when running cilium against the same set of pods
making DNS requests at the same cadence:
ROUTINE ======================== main.(*Daemon).notifyOnDNSMsg in /home/vagrant/go/src/github.com/cilium/cilium/daemon/fqdn.go
         0      460ms (flat, cum) 17.83% of Total
         .          .    405:	if err != nil {
         .          .    406:		// This error is ok because all these values are used for reporting, or filling in the cache.
         .          .    407:		log.WithError(err).Error("cannot extract DNS message details")
         .          .    408:	}
         .          .    409:
         .       30ms    410:	ep.UpdateProxyStatistics("dns", uint16(serverPort), false, !msg.Response, verdict)
         .       70ms    411:	record := logger.NewLogRecord(proxy.DefaultEndpointInfoRegistry, ep, flowType, false,
         .          .    412:		func(lr *logger.LogRecord) { lr.LogRecord.TransportProtocol = accesslog.TransportProtocol(protoID) },
         .          .    413:		logger.LogTags.Verdict(verdict, reason),
         .          .    414:		logger.LogTags.Addressing(logger.AddressingInfo{
         .          .    415:			SrcIPPort:   epIPPort,
         .          .    416:			DstIPPort:   serverAddr,
         .          .    417:			SrcIdentity: ep.GetIdentity().Uint32(),
         .          .    418:		}),

Fixes: 2935f589ab
Signed-off by: Ian Vernon<EMAIL_ADDRESS>Do not marshal endpoint structure into `SrcIPPort` field for each DNS logging record

This change is

test-me-please

Coverage decreased (-0.02%) to 41.794% when pulling 991d500f1231f1ac0ec42c996118d356e679ec1f on pr/ianvernon/fix-srcipport-string into 6d14ec4d1ac747b063f656e48fae76e7ef85682d on master.

test-me-please


OneDrive.admx file is broken

the onedrive.admx file supplied with version 19.222.1110.0011 doesn't work in either GPMC or GPEdit because of an error at Line 337 column 308, i.e.
  <policy name="LocalMassDeleteFileDeleteThreshold" class="Machine" displayName="$(string.LocalMassDeleteFileDeleteThreshold)" explainText="$(string.LocalMassDeleteFileDeleteThreshold_help)" presentation="$(presentation.LocalMassDeleteFileDeleteThreshold_Pres)" key="SOFTWARE\Policies\Microsoft\OneDrive">
It looks the same as similar values, but GPMC/GPEdit don't like it, and stops the user from seeing any of the OneDrive GPO settings

@loaderladdy thank you for submitting feedback.
I think the best way forward is if you open a service ticket in your tenant so this can get resolved ASAP. Based on the outcome let me know if it is something that can be called out in the docs.
Please follow this link to contact support for business products:https://docs.microsoft.com/office365/admin/contact-support-for-business-products
Please keep us posted here on the resolution so we can feed whatever information you discover into the content.


eslint fails on new yo office react project with TypeError: Error while loading rule 'prettier/prettier': context.getPhysicalFilename is not a function when opening project in IntelliJ

Followed the quickstart, and the default configuration fails in my editor.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 77e380a7-6e2e-ada8-d297-3fd5018a1958
Version Independent ID: f23490e8-6eeb-f2d5-6c94-69e3c9ed83f4
Content: Build your first Word task pane add-in - Office Add-ins
Content Source: docs/quickstarts/word-quickstart.md
Product: word
Technology: add-ins
GitHub Login: @o365devx
Microsoft Alias: o365devx

Thanks for reporting this @jkoberg. Just to clarify, is your add-in project working fine, aside from the linter error?
@Rick-Kirkham, could you please investigate?

@jkoberg I just went through the article and did not reproduce this issue. Your issue title mentions "react", but the article has the reader create a non-React project. Is your issue really with the article, or is it with a React project generated by Yo Office? If it is the latter I can move issue to the repo for Yo Office.


postMessage errors when listening to events as well as emitting

General information

Library version: 4.0.2
Browser and OS: FireFox 79.0 (64-bit) on Mac OS 10.15.6

Issue description
When using the target function to communicate between two frame, I get a lot of errors from the originating frame saying:
Failed to execute 'postMessage' on 'DOMWindow': The target origin provided ('http://<IP_ADDRESS>:8081') does not match the recipient window's origin ('http://<IP_ADDRESS>:8080').

It seems that if you have both frames listening and emitting events, you inevitably get a lot of errors on either side when using target. I assume this is outside of framebus's scope to be able to fix?

As far as I know, there's nothing we can do to stop that. If you want to open a PR, I'd be happy to look at it, but don't really have a suggestion for a fix.

@crookedneighbor yeah that's what I was afraid of. Happy to close this ticket but just wanted to call this out for reference as someone else might be trying to wrap their brain over these errors as I did.


use CSS variables for extensionEditor

for https://github.com/microsoft/vscode/issues/165169

Sorry that I am not accepting PRs for these changes as they are sensitive.


[FLINK-22227][clients] Log submitted job name, id and receiving dispa…

https://issues.apache.org/jira/browse/FLINK-22227

@zentol Thanks for the review! I've incorporated your suggestion, PTAL


Style markdown within asides

Description

Closes #2118
Tints the background color of code and the border color of hr and blockquote when placed within an Aside

Dark Mode:

Before
After

Light Mode:

Before
After

Lunaria Status Overview
🌕 This pull request will trigger status changes.

Learn more
By default, every PR changing files present in the Lunaria configuration's files property will be considered and trigger status changes accordingly.
You can change this by adding one of the keywords present in the ignoreKeywords property in your Lunaria configuration file in the PR's title (ignoring all files) or by including a tracker directive in the merged commit's description.

Tracked Files

Locale
File
Note

en
guides/asides-test.mdx
Source added, will be tracked.

Warnings reference

Icon
Description

🔄
The source for this localization has been updated since the creation of this pull request, make sure all changes in the source have been applied.


Change tls crt and key

Environment

OS: Ubuntu20.04
Kubernetes/K3s: v1.30.2+k3s2
AWX Operator: 2.19.1

Description
I installed awx using kurokobo awx-on-k3s. It is working properly.
I need to change the old tls with a new one. Both old and new tls have been issued locally by me.
Could you please let me know how can I do this?
I'd appreciate it if you give me a detailed instruction to do this. Thank you.

Install your tls.{key,crt into the base/ directory of where you installed the clone of this repository, and re-apply that. I script this roughly as
cp $p1 /root/awx-on-k3s/base/tls.crt
/usr/local/bin/kubectl apply -k base

@komiko1
Hi, as @jpmens described, just replace your tls.crt and tls.key, then invoke kubectl apply -k base again. Note that the crt and key files must be in PEM format.
@jpmens
Thanks for your help!

@jpmens @kurokobo
Thank you both for your guidance.


fix(DDIS-6462): Fonts are too large compared to the scaffold size

@MariaDolotova @StLeonidas This small PR is to normalize the scale of scaffolds retrieved from the sketcher, such that their bond length is the same that RDKit expects. This avoids downstream issues such as:

heteroatom fonts being too large compared to the size of the scaffold
possible distortions upon attempting to align molecules to such scaffolds

For instance, fonts that previously looked like this

are now properly scaled:

@MariaDolotova I have noticed this was not included in Chem 1.8.6: does this break any of your use cases?

@ptosco , oh, I'm sorry, I somehow missed the PR. Will merge and test.


Maximum entropy classifier: when will be added to natural?

any plan for add MEC to natural, I'm trying found a node or javascript lib than include it, thanks!!

I am working on a maximum entropy classifier in nodejs. It is still under development, but the scaling algorithm is implemented and working already. At the moment I am doing tests to make sure it works as expected. At some moment I will do a PR to add it to natural.
Feel free to take a look at
https://github.com/Hugo-ter-Doest/MaxEnt
Regards,
Hugo


Add hour and minute to PT; strip 00s from right

Apologize for the miscommunication earlier.  I blame it on the sinus infection I have got going.  I had also on my end added H and M to PT if there are hours or minutes, if you think it's worth adding - and stripped off any trailing zeros to the right of the padded 9 digit nanos since they're only needed when they're non-zero.  Again, all non-functional since all seconds works fine and having 9 digits even with zeros is fine.  But we're doing this in case of buggy upstream systems (keeping the MPD as it is with other systems).  But understand if you want to hold off on including in the lib.
Thanks again for the library.  Think I'm done with the PR bliz and am going to bed. :-)

Looks good, thanks. Get better!


GitHub icon not found by material-ui/icons

https://github.com/mui-org/material-ui/blob/master/docs/src/pages/discover-more/team/Team.js#L9
import GitHubIcon from '@material-ui/icons/GitHub'; as demonstrated in the  docs/src/pages/discover-more/team/Team.js file is not working in React project
Error:
Module not found: Error: Can't resolve '@material-ui/icons/GitHub'

[x] The issue is present in the latest release.
[x] I have searched the issues of this repository and believe that this is not a duplicate.

Current Behavior 😯

Expected Behavior 🤔

Steps to Reproduce 🕹

Context 🔦

Your Environment 🌎

Tech
Version

Material-UI
v4.5.0

React
16.10

Browser

TypeScript

etc.

Wait the v4.5.1 release this weekend ✌

@oliviertassinari wow, nice!


Plugin ignoring settings

Using OC. 3.4.16
The plugin is ignoring some settings in the dedicated settings page.  Instead, it's only considering the settings in the config/config.php file.  For example, the min password length.

Looks like this feature was added in an incomplete state.


analyzer function

Please complete the integrated development with Dockyard,thx

Please add analyzer function into github.com/containerops/wrench project,this PR should be closed,thx

OK, analyzer function has been pushed to wrench: https://github.com/containerops/wrench/pull/22


Change landscape layout to use more space for Instructions

The current landscape layout uses 3 columns for nutrition info, ingredients, and instructions. On my tablet, this gives very little space for the instructions and can require a lot of scrolling for some recipes. In addition, even when a recipe does not contain nutrition info, the 3 columns are still used instead of hiding the nutrition area to provide more space for ingredients and instructions. Similar to @psleep in #161, I think a 2 column layout for ingredients and instructions would make the landscape view better and easier to use.
This PR moves the nutrition information up into the same row as the tools list when in landscape view, freeing up horizontal space for the ingredients and instructions. Since each instruction is usually longer than an ingredient, I made the instructions area twice the width of the ingredients list (and instructions become full width if there are no ingredients). If either the tools list or nutrition info is not present in the recipe, the corresponding area is not shown. This PR makes no changes to the portrait layout.
I'd welcome any feedback or suggestions for the new landscape layout. @psleep, is this what you were thinking of in your issue?
Here are a few before and after screenshots to show the changes:
Original landscape layout (same whether recipe has nutrition info or not)

New landscape layout - default after load

New landscape layout - tools / nutrition expanded

New landscape layout - no nutrition info

Thank you both for the comments, and sorry I didn't realize that such a big rewrite was underway. Even if this PR needs to be re-implemented after #184, it would be great to know whether the landscape layout as demonstrated in the screenshots would be accepted or not, or needs reworking (the design more than the code itself, given the pending rewrite).
I took a quick look at the current state of #184. Here is the section that covers what my PR touches (the layout of the tools/nutrition/ingredients/instruction): https://github.com/Leptopoda/nextcloud-cookbook-flutter/blob/a5b022443ab783e1365fe39a318c097c7715288a/lib/src/screens/recipe/widget/recipe_screen_body.dart#L193-L215. It looks like that rewrite already fixes the bug of the nutrition info taking up space even if the recipe has no nutrition info, so that's great. If we wanted to implement the same layout I have in this PR, I think something like this would work (this could probably be written better, but just to demonstrate):
  Widget _buildBottom() {
    return SliverPadding(
      padding: const EdgeInsets.only(bottom: 75.0),
      sliver: SliverList(
        delegate: SliverChildListDelegate.fixed([
          if (MediaQuery.of(context).size.width > 600) ...[
            Row(
              crossAxisAlignment: CrossAxisAlignment.start,
              children: [
                if (recipe.tool.isNotEmpty)
                  Expanded(
                    flex: 5,
                    child: _ToolList(recipe),
                  ),
                if (recipe.nutritionList.isNotEmpty)
                  Expanded(
                    flex: 5,
                    child: _NutritionList(recipe.nutritionList)
                  ),
              ]
            ),
            Row(
              crossAxisAlignment: CrossAxisAlignment.start,
              children: [
                if (recipe.recipeIngredient.isNotEmpty)
                  Expanded(
                    flex: 5,
                    child: _IngredientList(recipe),
                  ),
                Expanded(
                  flex: 10,
                  child: _InstructionList(recipe)
                ),
              ]
            ),
          ]
          else ...[
            if (recipe.tool.isNotEmpty) _ToolList(recipe),
            if (recipe.nutritionList.isNotEmpty) _NutritionList(recipe.nutritionList),
            if (recipe.recipeIngredient.isNotEmpty) _IngredientList(recipe),
            _InstructionList(recipe),
          ]
        ]),
      ),
    );
  }

Anyway, happy to coordinate with #184.

@arroyoj
Generally i think your change is deffinitvely the correct way to go, in terms of how the content is arranged / not showing empty containers etc.
I think it would be best if we wait for the rework to hit and then if you have time, you could adapt the PR to match the new design.
If not maybe somone of us can pick it up. Would that be fine for you?

@Teifun2, thanks, that sounds like a good plan. I'll wait until #184 is merged and then re-implement the design from there.

I'll have a look at it shortly but a quick glance at the diff looks good.
Just out of curiosity are there any other optimizations we could make to tablet layouts that you can think of?
With the big redesign I'd love to get proper tablet support also done.

Ok I'll need to adjust the CI settings :)
#1

@Leptopoda, thanks for the review.

Just out of curiosity are there any other optimizations we could make to tablet layouts that you can think of? With the big redesign I'd love to get proper tablet support also done.

Nothing I can think of right now, but I'll take a closer look at the new design on my tablet.


protobuf3 support? [JIRA: RIAK-3397]

is there any plan to support protobuf3?

Hi @ziyouchutuwenwu,
This project is no longer actively maintained, and there is no plan for protobuf3 support.
We've moved over to gpb for Riak development, and recommend the same for others. gpb has erlang_protobuffs compatibility, protobuf3 support, and is actively maintained by @tomas-abrahamsson.
Thanks,
Alex


Error: net.inet.ip.forwarding is not a valid sysctl key on FreeBSD 11.x -- patch included

Hello team,
While using augeasproviders_sysctl for a FreeBSD 11.1 amd64 node with zleslie/bsd module, I came across this issue:

Error: /Stage[main]/Bsd::Network/Sysctl[net.inet.ip.forwarding]: Could not evaluate: Error: net.inet.ip.forwarding is not a valid sysctl key
Error: /Stage[main]/Bsd::Network/Sysctl[net.inet6.ip6.forwarding]: Could not evaluate: Error: net.inet6.ip6.forwarding is not a valid sysctl key

This is caused by the fact that the FreeBSD sysctl delimiter is a colon (':') and not an equal sign ('=') as specified in augeasproviders_sysctl/lib/puppet/provider/sysctl/augeas.rb on line 82.
As a quick workaround and possible patch, I've done the following:

if Facter.value(:kernel) == 'FreeBSD'
value = line.split(':')
else
value = line.split('=')
end

My puppetserver is at version 5.3.1-1.el7.noarch runnig on a CentOS Linux release 7.4.1708 (Core).
Please let me know if there's a better way to fix this issue.

I don't understand where the 'not a valid sysctl key' error comes from. That is not from this module's code. Are you maybe using another sysctl module in addition?

Seeing this issue here as well. The change suggested by @da125 fixed the problem.
Further info:
Checked out this repository and looked for "is not a valid sysctl key":
grep -rn "is not a valid sysctl key" .

./lib/puppet/provider/sysctl/augeas.rb:121:        raise Puppet::Error, "Error: `#{resource[:name]}` is not a valid sysctl key"
./lib/puppet/provider/sysctl/augeas.rb:150:          debug("augeasproviders_sysctl: `#{resource[:name]}` is not a valid sysctl key")
./lib/puppet/provider/sysctl/augeas.rb:153:          raise Puppet::Error, "Error: `#{resource[:name]}` is not a valid sysctl key"

Either way, as @da125 mentioned, FreeBSD use ": " to separate key name from value when running the "sysctl" command. (It still uses name=value when changing values, and in sysctl.conf)
sysctl net.inet6.ip6

net.inet6.ip6.forwarding: 0
net.inet6.ip6.redirect: 1
net.inet6.ip6.hlim: 64
(...)

Could you confirm #30 fixes it?

@da125 @Savagedlight Could you confirm #30 fixes it?

I made comments in the PR. Once that's addressed, it seems to work as expected.

I am no longer using puppet and cannot test.

@raphink
Same issue. #30 fixes it
Tested on: FreeBSD 12,13
Сan you commit it ?


support original source proxy

Related to #241
support original source proxy

HBONE connection:  use XFF header to encode the original source information.
local shortcut or direct tcp stream:  retrieve original source information from socket directly.

lichun@lichun-ws:~$ kubectl get pods -o wide
NAME                                   READY   STATUS    RESTARTS      AGE   IP            NODE            NOMINATED NODE   READINESS GATES
httpbin-no-mesh-654476d4ff-pfgxp       1/1     Running   0             35m   <IP_ADDRESS>   kind1-worker3   <none>           <none>
httpbin-worker1-5d5748d687-pf4cb       1/1     Running   0             77m   <IP_ADDRESS>   kind1-worker    <none>           <none>
httpbin-worker2-f4b4788cb-vw2hz        1/1     Running   0             77m   <IP_ADDRESS>    kind1-worker2   <none>           <none>
httpbin-worker3-6989c5fb68-46g6r       1/1     Running   0             77m   <IP_ADDRESS>   kind1-worker3   <none>           <none>
notsleep-7dc8b55755-trgqk              1/1     Running   4 (29h ago)   3d    <IP_ADDRESS>    kind1-worker3   <none>           <none>
purchase-history-v1-6cdb997954-6rhfd   1/1     Running   4 (29h ago)   3d    <IP_ADDRESS>    kind1-worker    <none>           <none>
recommendation-79f7844ff-scmjq         1/1     Running   4 (29h ago)   3d    <IP_ADDRESS>    kind1-worker    <none>           <none>
sleep-5f5d4b5bfb-cdtrr                 1/1     Running   4 (29h ago)   24d   <IP_ADDRESS>    kind1-worker3   <none>           <none>
web-api-6c95c8d759-92d42               1/1     Running   4 (29h ago)   3d    <IP_ADDRESS>    kind1-worker    <none>           <none>
lichun@lichun-ws:~$ kubectl exec deploy/sleep -- curl -s http://httpbin-worker1:8000/ip
{
  "origin": "<IP_ADDRESS>"
}
lichun@lichun-ws:~$ kubectl exec deploy/sleep -- curl -s http://httpbin-worker2:8000/ip
{
  "origin": "<IP_ADDRESS>"
}
lichun@lichun-ws:~$ kubectl exec deploy/sleep -- curl -s http://httpbin-worker3:8000/ip
{
  "origin": "<IP_ADDRESS>"
}

@PlatformLC: PR needs rebase.

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@PlatformLC: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

test_ztunnel
2b1eb295d9613f5d64e39f11a5ee542e22387b10
link
true
/test test

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@PlatformLC: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

test_ztunnel
74569ca9b196a6062bbca3851d30f78d92337ae8
link
true
/test test

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@PlatformLC: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

test_ztunnel
13287b136fb29540afd5097f396281319cff7a3e
link
true
/test test

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@PlatformLC: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

test_ztunnel
d1e4ade8a7382801ef4326611e0d212fac51e527
link
true
/test test

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@PlatformLC: PR needs rebase.

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@PlatformLC: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

test_ztunnel
1814c5c8261615f465b9b2130f31ebfba8cfc4d6
link
true
/test test

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@yuval-k see https://docs.google.com/document/d/17fDTMLg-22UPfh62d5wumAOIpbeoOyWix2WTE54hDNc/edit# and/or https://github.com/istio/ztunnel/pull/264#discussion_r1046384840 for more context, would appreciate your feedback on the doc

@PlatformLC: PR needs rebase.

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Just rebase to the latest master branch. @howardjohn  Would you help to review it again?


polar plot close gap

Polar plots (plot set 7) still may have a gap for low resolution data, e.g. a 2 degree mesh, and some obs sets may have such a grid.  We need to wrap around more in order to make the plots prettier.

There's a caution with this. Namely, closing this gap will make it appear that there is data filling the gap when there may not be any data.

@mcenerney1 , that's a good point.  The present code (amwg.py, plot set 7, method plan_computation) expands longitude to the range 0:360 regardless of the original range.  Maybe the solution is to see whether a slight expansion (maybe 1 cell width on each side, width is the average from the first 10 and last 10 cells respectively) in longitude would extend the longitude bounds to overlap.  Expand longitude only  then.

there is data. It's a vcs "features" it draws data from node to node, reading data from (0,360) actually leaves a gap in nodes e.g. for a 5 degree grid first point is 2.5 and last 357.5 hence the white space. now reading the data with an extended point (which cdms2 actually allows us to do) will give us data from 2.5 to 362.5 and no gap will be shown.


first-read-last-write-cache: Add fuzz tests for the cache merge logic.

It would be good to fuzz test the CacheLog::merge method. We already have a programmatic way to create test cases (see TestCase in tests)

fixed in #69