Error in ATAqC (ValueError: all the input array dimensions except for the concatenation axis must match exactly)

Thank you for making this pipeline available. I am able to get the pipeline running except when I include the ATAqC step I obtain an error in numpy. I have included the full log file below.
https://gist.github.com/mdurante1/3f51ef55b2fb0758949a1c68cacb6d28
Do you have any insight on how to fix this error?
            Traceback (most recent call last):
              File "/nethome/mdurante/atac_dnase_pipelines/ataqc/run_ataqc.py", line 1598, in <module>
                main()
              File "/nethome/mdurante/atac_dnase_pipelines/ataqc/run_ataqc.py", line 1460, in main
                ROADMAP_META, OUTPUT_PREFIX)
              File "/nethome/mdurante/atac_dnase_pipelines/ataqc/run_ataqc.py", line 905, in compare_to_roadmap
                sample_mean0_col)
              File "/nethome/mdurante/miniconda3/envs/bds_atac/lib/python2.7/site-packages/scipy/stats/stats.py", line 3310, in spearmanr
                rs = np.corrcoef(ar, br, rowvar=axisout)
              File "/nethome/mdurante/miniconda3/envs/bds_atac/lib/python2.7/site-packages/numpy/lib/function_base.py", line 2145, in corrcoef
                c = cov(x, y, rowvar)
              File "/nethome/mdurante/miniconda3/envs/bds_atac/lib/python2.7/site-packages/numpy/lib/function_base.py", line 2024, in cov
                X = np.vstack((X, y))
              File "/nethome/mdurante/miniconda3/envs/bds_atac/lib/python2.7/site-packages/numpy/core/shape_base.py", line 230, in vstack
                return _nx.concatenate([atleast_2d(_m) for _m in tup], 0)
            ValueError: all the input array dimensions except for the concatenation axis must match exactly
    StdOut (100000000 lines)  :
            Default case invoked for:
               opcode  = 153, "LoopLimit"

It looks like your local version of metaseq python package conflicts with numpy in the conda environment. Can you try the following to debug?
$ python
>>> import metaseq

When running the command I obtained:
~/.local/lib/python2.7/site-packages/subprocess32.py
I proceeded to remove the subprocess32 files and removed my local installation of metaseq but I am still receiving the same error after ataqc runs for ~8 hours. Also the TSS plot (*.trim.PE2SE_vplot.png) is being generated so it seems that metaseq is running partly. Any further suggestions?
Here is the log file from the latest run:
https://gist.github.com/mdurante1/0f479edd2beefd54db25da5499a6c04d

Hi,
I see from your comment that you might have an older version of ATAqC (we've renamed *vplot.png to *tss-enrich.png) - could you try uninstall the whole pipeline and then reinstall, making sure to clone recursively to get the package? And also get the latest version of the genome annotations, downloading as described in the instructions.
Thanks!
Daniel

I just uninstalled all dependencies, removed the whole pipeline, and cloned it recursively using "git clone https://github.com/kundajelab/atac_dnase_pipelines --recursive"
I reinstalled the dependencies and then checked the "run_ataqc.py" file in the ~/atac_dnase_pipelines/ataqc/ directory and it seems as if the old ATAqC version is being loaded:
def make_vplot(bam_file, tss, prefix, chromsizes, read_len, bins=400, bp_edge=2000,
processes=8, greenleaf_norm=True):
'''
Take bootstraps, generate V-plots, and get a mean and
standard deviation on the plot. Produces 2 plots. One is the
aggregation plot alone, while the other also shows the signal
at each TSS ordered by strength.
'''
logging.info('Generating vplot...')
vplot_file = '{0}_vplot.png'.format(prefix)
vplot_large_file = '{0}_large_vplot.png'.format(prefix)
Should the new version of ATAqC be loaded automatically or do I have to clone it separately?

Sorry, we were a commit behind in the sync with ATAqC. It's updated now, please try again. Thanks!

Hi Daniel,
After reinstalling the updated pipeline and redownloading the genomes, I ran the samples again and I am obtaining the same error I obtained previously. I checked the qc directory and I now see the properly named TSS plot (.trim.PE2SE_large_tss-enrich.png) and a gc plot (.trim.PE2SE_gcPlot.pdf), which was not there before. If I add -no_ataqc to the script and resubmit the job it completes the analysis so it seems to be a specific ATAqC problem. Any other thoughts on how to fix this error?
Thanks!
Michael

Hi Michael,
Looking more carefully at this, we've actually resolved your issue - this is a new downstream issue, related to comparing your samples to publicly available ATAC. Can you look in your qc folder and look for a test.log file if you still have it? That might have some more information for me to look at to see what's going on.
Daniel

Here is the test.log file from the latest run:
https://gist.github.com/mdurante1/8dc28ce6195b7b54fdfee7054524082c
I have also included the error log file from my latest run to show the error:
https://gist.github.com/mdurante1/a3bf052dd1f8dc86f552c3d13abcf3cb
		Traceback (most recent call last): 		  File "/nethome/mdurante/atac_dnase_pipelines/ataqc/run_ataqc.py", line 1598, in <module> 		    main() 		  File "/nethome/mdurante/atac_dnase_pipelines/ataqc/run_ataqc.py", line 1460, in main 		    ROADMAP_META, OUTPUT_PREFIX) 		  File "/nethome/mdurante/atac_dnase_pipelines/ataqc/run_ataqc.py", line 905, in compare_to_roadmap 		    sample_mean0_col) 		  File "/nethome/mdurante/miniconda3/envs/bds_atac/lib/python2.7/site-packages/scipy/stats/stats.py", line 3310, in spearmanr 		    rs = np.corrcoef(ar, br, rowvar=axisout) 		  File "/nethome/mdurante/miniconda3/envs/bds_atac/lib/python2.7/site-packages/numpy/lib/function_base.py", line 2145, in corrcoef 		    c = cov(x, y, rowvar) 		  File "/nethome/mdurante/miniconda3/envs/bds_atac/lib/python2.7/site-packages/numpy/lib/function_base.py", line 2024, in cov 		    X = np.vstack((X, y)) 		  File "/nethome/mdurante/miniconda3/envs/bds_atac/lib/python2.7/site-packages/numpy/core/shape_base.py", line 230, in vstack 		    return _nx.concatenate([atleast_2d(_m) for _m in tup], 0) 		ValueError: all the input array dimensions except for the concatenation axis must match exactly 	StdOut (100000000 lines)  : 		Default case invoked for:  		   opcode  = 153, "LoopLimit"
Hope this helps!
Michael

Could you do two things for me:

In your error log, it makes a note that your BAM index file (.bai) is older than the BAM file. Could you do a clean re-run (if it doesn't take too long to do the re-run)?
Could you run which conda in your terminal where you run BDS and share the output path?

Daniel

Hi Daniel,

I can do a clean re-run although this will take 3-4 days. I will post the output when the run is complete.
when I do which conda the output is:
~/miniconda3/envs/bds_atac/bin/conda

Michael

Sounds good. I've also received feedback from someone else who ran into this issue:
"I removed miniconda and reinstalled miniconda version 4.0.5. I then removed/re-cloned the bds_atac pipeline. I then used uninstall_dependencies.sh and then install_dependencies.sh"
So basically try a full uninstall, set up miniconda first, then clone the pipeline, then the dependencies. Hope this helps!

Also to do a quick test, do consider subsampling your FASTQ files and just running one of them to make sure that it can make it through ATAqC. This will decrease the time to debug before having to do a full re-run of everything :)

Hi Daniel,
I did a clean run through and I am now getting a different error:
Traceback (most recent call last):
  File "/nethome/mdurante/atac_dnase_pipelines/ataqc/run_ataqc.py", line 1598, in <module>
    main()
  File "/nethome/mdurante/atac_dnase_pipelines/ataqc/run_ataqc.py", line 1413, in main
    [flagstat, mapped_count] = get_samtools_flagstat(ALIGNED_BAM)
  File "/nethome/mdurante/atac_dnase_pipelines/ataqc/run_ataqc.py", line 589, in get_samtools_flagstat
    return flagstat, mapped_reads
UnboundLocalError: local variable 'mapped_reads' referenced before assignment

Here is the log file from the latest run:
https://gist.github.com/mdurante1/ada64d8f842812e984857a7ac90c6069
I have also included the test.log file:
https://gist.github.com/mdurante1/30bf1378c550948a653122fd5c184d3c
Thanks,
Michael

It's a pysam issue - there are different outputs in versions 0.9+ (https://github.com/pysam-developers/pysam/issues/245 in case you're curious).
We've pushed a hot fix, go ahead and pull the pipeline again (recursively to get the ataqc submodule). no need to reinstall anything.

Hi Daniel,
I did another clean run through with all the suggested changes and I am now getting a different error:
                Traceback (most recent call last):                   File "/nethome/mdurante/atac_dnase_pipelines/ataqc/run_ataqc.py", line 1598, in <module>                     main()                   File "/nethome/mdurante/atac_dnase_pipelines/ataqc/run_ataqc.py", line 1413, in main                     [flagstat, mapped_count] = get_samtools_flagstat(ALIGNED_BAM)                   File "/nethome/mdurante/atac_dnase_pipelines/ataqc/run_ataqc.py", line 582, in get_samtools_flagstat                     results = pysam.flagstat(bam_file, split_lines=TRUE)                 NameError: global name 'TRUE' is not defined
Here's the log file of this run:
https://gist.github.com/mdurante1/caee77db53813e0c990f35abf5d3cace
test.log:
https://gist.github.com/mdurante1/bab061d3e306b6f0e9412448c22e851e

Hi Michael,
Thanks - so I made a big goof and used R syntax accidentally. It's fixed and in the latest ataqc module, so go ahead and do the same as before (just pull the pipeline recursively to get the submodule, no reinstallations needed) and that bug should go away.
Daniel

Hi Daniel,
Your fixed worked for that bug but now I am receiving the original error again:
		Traceback (most recent call last):
		  File "/nethome/mdurante/atac_dnase_pipelines/ataqc/run_ataqc.py", line 1598, in <module>
		    main()
		  File "/nethome/mdurante/atac_dnase_pipelines/ataqc/run_ataqc.py", line 1460, in main
		    ROADMAP_META, OUTPUT_PREFIX)
		  File "/nethome/mdurante/atac_dnase_pipelines/ataqc/run_ataqc.py", line 905, in compare_to_roadmap
		    sample_mean0_col)
		  File "/nethome/mdurante/miniconda3/envs/bds_atac/lib/python2.7/site-packages/scipy/stats/stats.py", line 3310, in spearmanr
		    rs = np.corrcoef(ar, br, rowvar=axisout)
		  File "/nethome/mdurante/miniconda3/envs/bds_atac/lib/python2.7/site-packages/numpy/lib/function_base.py", line 2145, in corrcoef
		    c = cov(x, y, rowvar)
		  File "/nethome/mdurante/miniconda3/envs/bds_atac/lib/python2.7/site-packages/numpy/lib/function_base.py", line 2024, in cov
		    X = np.vstack((X, y))
		  File "/nethome/mdurante/miniconda3/envs/bds_atac/lib/python2.7/site-packages/numpy/core/shape_base.py", line 230, in vstack
		    return _nx.concatenate([atleast_2d(_m) for _m in tup], 0)
		ValueError: all the input array dimensions except for the concatenation axis must match exactly

Here are the corresponding log files:
https://gist.github.com/mdurante1/20997f4b9a57f89f8c10464b4910f5c5
https://gist.github.com/mdurante1/a52c7db10f97e1e0748da4de0ce9be3e

@vervacity Hi Daniel,
I am also having the same problem.  I get this error
		# SYS command. line 145

		 TASKTIME=$[$(date +%s)-${STARTTIME}]; echo "Task has finished (${TASKTIME} seconds)."
	StdErr (100000000 lines)  :
		Picked up _JAVA_OPTIONS: -Xms256M -Xmx15G -XX:ParallelGCThreads=1
		ERROR	2017-05-18 23:04:55	ProcessExecutor	Warning messages:
		ERROR	2017-05-18 23:04:55	ProcessExecutor	1: In arrows(metrics$GC, metrics$NORMALIZED_COVERAGE - metrics$ERROR_BAR_WIDTH,  :
		ERROR	2017-05-18 23:04:55	ProcessExecutor	  zero-length arrow is of indeterminate angle and so skipped
		ERROR	2017-05-18 23:04:55	ProcessExecutor	2: In arrows(metrics$GC, metrics$NORMALIZED_COVERAGE - metrics$ERROR_BAR_WIDTH,  :
		ERROR	2017-05-18 23:04:55	ProcessExecutor	  zero-length arrow is of indeterminate angle and so skipped
		ERROR	2017-05-18 23:04:55	ProcessExecutor	3: In arrows(metrics$GC, metrics$NORMALIZED_COVERAGE - metrics$ERROR_BAR_WIDTH,  :
		ERROR	2017-05-18 23:04:55	ProcessExecutor	  zero-length arrow is of indeterminate angle and so skipped
		ERROR	2017-05-18 23:04:55	ProcessExecutor	4: In arrows(metrics$GC, metrics$NORMALIZED_COVERAGE - metrics$ERROR_BAR_WIDTH,  :
		ERROR	2017-05-18 23:04:55	ProcessExecutor	  zero-length arrow is of indeterminate angle and so skipped
		ERROR	2017-05-18 23:04:55	ProcessExecutor	5: In arrows(metrics$GC, metrics$NORMALIZED_COVERAGE - metrics$ERROR_BAR_WIDTH,  :
		ERROR	2017-05-18 23:04:55	ProcessExecutor	  zero-length arrow is of indeterminate angle and so skipped
		ERROR	2017-05-18 23:04:55	ProcessExecutor	6: In arrows(metrics$GC, metrics$NORMALIZED_COVERAGE - metrics$ERROR_BAR_WIDTH,  :
		ERROR	2017-05-18 23:04:55	ProcessExecutor	  zero-length arrow is of indeterminate angle and so skipped
		ERROR	2017-05-18 23:04:55	ProcessExecutor	7: In arrows(metrics$GC, metrics$NORMALIZED_COVERAGE - metrics$ERROR_BAR_WIDTH,  :
		ERROR	2017-05-18 23:04:55	ProcessExecutor	  zero-length arrow is of indeterminate angle and so skipped
		Picked up _JAVA_OPTIONS: -Xms256M -Xmx15G -XX:ParallelGCThreads=1
		[bam_sort_core] merging from 26 files...
		[bam_sort_core] merging from 26 files...
		Picked up _JAVA_OPTIONS: -Xms256M -Xmx15G -XX:ParallelGCThreads=1
		Traceback (most recent call last):
		  File "/hd2/atac_dnase_pipelines/ataqc/run_ataqc.py", line 1598, in <module>
		    main()
		  File "/hd2/atac_dnase_pipelines/ataqc/run_ataqc.py", line 1413, in main
		    [flagstat, mapped_count] = get_samtools_flagstat(ALIGNED_BAM)
		  File "/hd2/atac_dnase_pipelines/ataqc/run_ataqc.py", line 582, in get_samtools_flagstat
		    results = pysam.flagstat(bam_file, split_lines=TRUE)
		NameError: global name 'TRUE' is not defined

Fatal error: /hd2/atac_dnase_pipelines/atac.bds, line 1525, pos 2. Task/s failed.
atac.bds, line 74 :	main()
atac.bds, line 77 :	void main() { // atac pipeline starts here
atac.bds, line 97 :		ataqc()
atac.bds, line 1514 :	void ataqc() {
atac.bds, line 1525 :		wait

Creating checkpoint file: Config or command line option disabled checkpoint file creation, nothing done.

Here is the test. log file in the qc folder.
INFO:root:Reading bowtie alignment log...
INFO:root:18507448 reads; of these:
INFO:root:18507448 (100.00%) were paired; of these:
INFO:root:189484 (1.02%) aligned concordantly 0 times
INFO:root:8274240 (44.71%) aligned concordantly exactly 1 time
INFO:root:10043724 (54.27%) aligned concordantly >1 times
INFO:root:----
INFO:root:189484 pairs aligned concordantly 0 times; of these:
INFO:root:26354 (13.91%) aligned discordantly 1 time
INFO:root:----
INFO:root:163130 pairs aligned 0 times concordantly or discordantly; of these:
INFO:root:326260 mates make up the pairs; of these:
INFO:root:233407 (71.54%) aligned 0 times
INFO:root:21280 (6.52%) aligned exactly 1 time
INFO:root:71573 (21.94%) aligned >1 times
INFO:root:99.37% overall alignment rate
INFO:root:Getting mitochondrial chromosome fraction...
INFO:root:Getting GC bias...
INFO:root:java -Xmx4G -jar /home/rick/anaconda3/envs/bds_atac/bin/../share/picard*/picard.jar CollectGcBiasMetrics R=/hd2/bds_genome_data/mm9/mm9.fa I=/hd2/atacseq_taf/output/ATAC_5lKO/out/align/rep2/ATAC_5_L002_R1_001.trim.PE2SE.nodup.bam O=/hd2/atacseq_taf/output/ATAC_5lKO/out/qc/rep2/ATAC_5_L002_R1_001.trim.PE2SE_gc.txt VERBOSITY=ERROR QUIET=TRUE ASSUME_SORTED=FALSE CHART=/hd2/atacseq_taf/output/ATAC_5lKO/out/qc/rep2/ATAC_5_L002_R1_001.trim.PE2SE_gcPlot.pdf S=/hd2/atacseq_taf/output/ATAC_5lKO/out/qc/rep2/ATAC_5_L002_R1_001.trim.PE2SE_gcSummary.txt
INFO:root:Running preseq...
INFO:root:preseq lc_extrap -P -B -o /hd2/atacseq_taf/output/ATAC_5lKO/out/qc/rep2/ATAC_5_L002_R1_001.trim.PE2SE.preseq.dat /hd2/atacseq_taf/output/ATAC_5lKO/out/qc/rep2/ATAC_5_L002_R1_001.trim.PE2SE.sorted.bam -v 2> /hd2/atacseq_taf/output/ATAC_5lKO/out/qc/rep2/ATAC_5_L002_R1_001.trim.PE2SE.preseq.log
INFO:root:samtools mapq 30...
INFO:root:Running Picard MarkDuplicates...
INFO:root:samtools flagstat...

@rfarouni you probably grabbed the commit right before we corrected it - go ahead and pull the latest version (making sure to do recursive to get the latest ataqc module) and it should work.
@mdurante1 turns out that we had an annotation switch in mm9. this has now been corrected and should run properly. please pull the latest commit and download the latest genome data annotations. (also as an aside, I would personally recommend making the jump to mm10, if it fits with your other analyses!)

@vervacity Thaks. This fixed it, but another error came up several steps down the pipeline. Here is the error
--------------------Stderr--------------------
Picked up _JAVA_OPTIONS: -Xms256M -Xmx20G -XX:ParallelGCThreads=1
ERROR	2017-05-20 17:51:40	ProcessExecutor	Warning messages:
ERROR	2017-05-20 17:51:40	ProcessExecutor	1: In arrows(metrics$GC, metrics$NORMALIZED_COVERAGE - metrics$ERROR_BAR_WIDTH,  :
ERROR	2017-05-20 17:51:40	ProcessExecutor	  zero-length arrow is of indeterminate angle and so skipped
ERROR	2017-05-20 17:51:40	ProcessExecutor	2: In arrows(metrics$GC, metrics$NORMALIZED_COVERAGE - metrics$ERROR_BAR_WIDTH,  :
ERROR	2017-05-20 17:51:40	ProcessExecutor	  zero-length arrow is of indeterminate angle and so skipped
ERROR	2017-05-20 17:51:40	ProcessExecutor	3: In arrows(metrics$GC, metrics$NORMALIZED_COVERAGE - metrics$ERROR_BAR_WIDTH,  :
ERROR	2017-05-20 17:51:40	ProcessExecutor	  zero-length arrow is of indeterminate angle and so skipped
ERROR	2017-05-20 17:51:40	ProcessExecutor	4: In arrows(metrics$GC, metrics$NORMALIZED_COVERAGE - metrics$ERROR_BAR_WIDTH,  :
ERROR	2017-05-20 17:51:40	ProcessExecutor	  zero-length arrow is of indeterminate angle and so skipped
ERROR	2017-05-20 17:51:40	ProcessExecutor	5: In arrows(metrics$GC, metrics$NORMALIZED_COVERAGE - metrics$ERROR_BAR_WIDTH,  :
ERROR	2017-05-20 17:51:40	ProcessExecutor	  zero-length arrow is of indeterminate angle and so skipped
ERROR	2017-05-20 17:51:40	ProcessExecutor	6: In arrows(metrics$GC, metrics$NORMALIZED_COVERAGE - metrics$ERROR_BAR_WIDTH,  :
ERROR	2017-05-20 17:51:40	ProcessExecutor	  zero-length arrow is of indeterminate angle and so skipped
ERROR	2017-05-20 17:51:40	ProcessExecutor	7: In arrows(metrics$GC, metrics$NORMALIZED_COVERAGE - metrics$ERROR_BAR_WIDTH,  :
ERROR	2017-05-20 17:51:40	ProcessExecutor	  zero-length arrow is of indeterminate angle and so skipped
Picked up _JAVA_OPTIONS: -Xms256M -Xmx20G -XX:ParallelGCThreads=1
[bam_sort_core] merging from 26 files...
[bam_sort_core] merging from 26 files...
Picked up _JAVA_OPTIONS: -Xms256M -Xmx20G -XX:ParallelGCThreads=1
Picked up _JAVA_OPTIONS: -Xms256M -Xmx20G -XX:ParallelGCThreads=1
processing chromosomes
Traceback (most recent call last):
  File "/hd2/atac_dnase_pipelines/ataqc/run_ataqc.py", line 1598, in
    main()
  File "/hd2/atac_dnase_pipelines/ataqc/run_ataqc.py", line 1460, in main
    ROADMAP_META, OUTPUT_PREFIX)
  File "/hd2/atac_dnase_pipelines/ataqc/run_ataqc.py", line 905, in compare_to_roadmap
    sample_mean0_col)
  File "/home/rick/anaconda3/envs/bds_atac/lib/python2.7/site-packages/scipy/stats/stats.py", line 3310, in spearmanr
    rs = np.corrcoef(ar, br, rowvar=axisout)
  File "/home/rick/anaconda3/envs/bds_atac/lib/python2.7/site-packages/numpy/lib/function_base.py", line 2145, in corrcoef
    c = cov(x, y, rowvar)
  File "/home/rick/anaconda3/envs/bds_atac/lib/python2.7/site-packages/numpy/lib/function_base.py", line 2024, in cov
    X = np.vstack((X, y))
  File "/home/rick/anaconda3/envs/bds_atac/lib/python2.7/site-packages/numpy/core/shape_base.py", line 230, in vstack
    return _nx.concatenate([atleast_2d(_m) for _m in tup], 0)
ValueError: all the input array dimensions except for the concatenation axis must match exactly

and here is the test.log file
INFO:root:Reading bowtie alignment log...
INFO:root:18507448 reads; of these:
INFO:root:18507448 (100.00%) were paired; of these:
INFO:root:189484 (1.02%) aligned concordantly 0 times
INFO:root:8274240 (44.71%) aligned concordantly exactly 1 time
INFO:root:10043724 (54.27%) aligned concordantly >1 times
INFO:root:----
INFO:root:189484 pairs aligned concordantly 0 times; of these:
INFO:root:26354 (13.91%) aligned discordantly 1 time
INFO:root:----
INFO:root:163130 pairs aligned 0 times concordantly or discordantly; of these:
INFO:root:326260 mates make up the pairs; of these:
INFO:root:233407 (71.54%) aligned 0 times
INFO:root:21280 (6.52%) aligned exactly 1 time
INFO:root:71573 (21.94%) aligned >1 times
INFO:root:99.37% overall alignment rate
INFO:root:Getting mitochondrial chromosome fraction...
INFO:root:Getting GC bias...
INFO:root:java -Xmx4G -jar /home/rick/anaconda3/envs/bds_atac/bin/../share/picard*/picard.jar CollectGcBiasMetrics R=/hd2/bds_genome_data/mm9/mm9.fa I=/hd2/atacseq_taf/output/ATAC_5lKO/out/align/rep2/ATAC_5lKO_TAAGGCGA_L002_R1_001.trim.PE2SE.nodup.bam O=/hd2/atacseq_taf/output/ATAC_5lKO/out/qc/rep2/ATAC_5lKO_TAAGGCGA_L002_R1_001.trim.PE2SE_gc.txt VERBOSITY=ERROR QUIET=TRUE ASSUME_SORTED=FALSE CHART=/hd2/atacseq_taf/output/ATAC_5lKO/out/qc/rep2/ATAC_5lKO_TAAGGCGA_L002_R1_001.trim.PE2SE_gcPlot.pdf S=/hd2/atacseq_taf/output/ATAC_5lKO/out/qc/rep2/ATAC_5lKO_TAAGGCGA_L002_R1_001.trim.PE2SE_gcSummary.txt
INFO:root:Running preseq...
INFO:root:preseq lc_extrap -P -B -o /hd2/atacseq_taf/output/ATAC_5lKO/out/qc/rep2/ATAC_5lKO_TAAGGCGA_L002_R1_001.trim.PE2SE.preseq.dat /hd2/atacseq_taf/output/ATAC_5lKO/out/qc/rep2/ATAC_5lKO_TAAGGCGA_L002_R1_001.trim.PE2SE.sorted.bam -v 2> /hd2/atacseq_taf/output/ATAC_5lKO/out/qc/rep2/ATAC_5lKO_TAAGGCGA_L002_R1_001.trim.PE2SE.preseq.log
INFO:root:samtools mapq 30...
INFO:root:Running Picard MarkDuplicates...
INFO:root:samtools flagstat...
INFO:root:37014896 + 0 in total (QC-passed reads + QC-failed reads)
INFO:root:0 + 0 secondary
INFO:root:0 + 0 supplementary
INFO:root:0 + 0 duplicates
INFO:root:36781489 + 0 mapped (99.37%:-nan%)
INFO:root:37014896 + 0 paired in sequencing
INFO:root:18507448 + 0 read1
INFO:root:18507448 + 0 read2
INFO:root:36635928 + 0 properly paired (98.98%:-nan%)
INFO:root:36758530 + 0 with itself and mate mapped
INFO:root:22959 + 0 singletons (0.06%:-nan%)
INFO:root:33546 + 0 with mate mapped to a different chr
INFO:root:17237 + 0 with mate mapped to a different chr (mapQ>=5)
INFO:root:final read counts...
INFO:root:insert size distribution...
INFO:root:java -Xmx4G -jar /home/rick/anaconda3/envs/bds_atac/bin/../share/picard*/picard.jar CollectInsertSizeMetrics INPUT=/hd2/atacseq_taf/output/ATAC_5lKO/out/align/rep2/ATAC_5lKO_TAAGGCGA_L002_R1_001.trim.PE2SE.nodup.bam OUTPUT=/hd2/atacseq_taf/output/ATAC_5lKO/out/qc/rep2/ATAC_5lKO_TAAGGCGA_L002_R1_001.trim.PE2SE.inserts.hist_data.log H=/hd2/atacseq_taf/output/ATAC_5lKO/out/qc/rep2/ATAC_5lKO_TAAGGCGA_L002_R1_001.trim.PE2SE.inserts.hist_graph.pdf VERBOSITY=ERROR QUIET=TRUE W=1000 STOP_AFTER=5000000
INFO:root:Generating tss plot...
INFO:root:signal to noise...
INFO:root:bigWigAverageOverBed /hd2/atacseq_taf/output/ATAC_5lKO/out/signal/macs2/rep2/ATAC_5lKO_TAAGGCGA_L002_R1_001.trim.PE2SE.nodup.tn5.pf.pval.signal.bigwig /hd2/bds_genome_data/mm9/ataqc/mm9_univ_dhs_ucsc.from_mm10.bed.gz /hd2/atacseq_taf/output/ATAC_5lKO/out/qc/rep2/ATAC_5lKO_TAAGGCGA_L002_R1_001.trim.PE2SE.signal

@rfarouni ok I see. You need to update your species config and files - go ahead and re-download the genome data (https://github.com/kundajelab/atac_dnase_pipelines#genome-data) and make sure in your species file ([SPECIES FILE] as noted in the installation instructions) that you have a line has something like this under [mm9]: reg2map_bed	= /path/to/pipeline_genome_data/mm9/ataqc/mm9_dhs_universal_ucsc_v1.bed.gz

@vervacity I re-downloaded the genome data, but there is not such file in my genome_data folder.  The file mm9/ataqc/mm9_dhs_universal_ucsc_v1.bed.gz is not there. I have this though dnase	= /hd2/bds_genome_data/mm9/ataqc/mm9_univ_dhs_ucsc.from_mm10.bed.gz

@rfarouni so I'm seeing that it's listed in the latest genome data download script (https://github.com/kundajelab/atac_dnase_pipelines/blob/master/install_genome_data.sh#L64) and is also available in our genome data folder (http://mitra.stanford.edu/kundaje/genome_data/mm9/ataqc/) - could you grab the latest genome download script and try again?

I downloaded the file and added the filepath in the species file. It fixed it and the final report gets produced, but I still get one error
# SYS command. line 30

if [[ -f $(which conda) && $(conda env list | grep bds_atac | wc -l) != "0" ]]; then source activate bds_atac; sleep 5; fi;  export PATH=/hd2/atac_dnase_pipelines/.:/hd2/atac_dnase_pipelines/modules:/hd2/atac_dnase_pipelines/utils:${PATH}:/bin:/usr/bin:/usr/local/bin:${HOME}/.bds; set -o pipefail; STARTTIME=$(date +%s)

# SYS command. line 32

zcat /hd2/atacseq_taf/output/ATAC_5lKO/out/peak/macs2/pooled_rep/ATAC_5lKO_TAAGGCGA_L001_R1_001.trim.PE2SE.nodup.tn5_pooled.pf.filt.gappedPeak.gz | sort -k1,1 -k2,2n > /hd2/atacseq_taf/output/ATAC_5lKO/out/peak/macs2/pooled_rep/ATAC_5lKO_TAAGGCGA_L001_R1_001.trim.PE2SE.nodup.tn5_pooled.pf.filt.gappedPeak.bb.tmp

# SYS command. line 33

cat /hd2/bds_genome_data/mm9/mm9/mm9.chrom.sizes | grep -P 'chr[\dXY]+[ \t]' > /hd2/atacseq_taf/output/ATAC_5lKO/out/peak/macs2/pooled_rep/ATAC_5lKO_TAAGGCGA_L001_R1_001.trim.PE2SE.nodup.tn5_pooled.pf.filt.gappedPeak.bb.chrsz.tmp

# SYS command. line 34

bedToBigBed -type=bed12+3 -as=/hd2/atac_dnase_pipelines/etc/gappedPeak.as /hd2/atacseq_taf/output/ATAC_5lKO/out/peak/macs2/pooled_rep/ATAC_5lKO_TAAGGCGA_L001_R1_001.trim.PE2SE.nodup.tn5_pooled.pf.filt.gappedPeak.bb.tmp /hd2/atacseq_taf/output/ATAC_5lKO/out/peak/macs2/pooled_rep/ATAC_5lKO_TAAGGCGA_L001_R1_001.trim.PE2SE.nodup.tn5_pooled.pf.filt.gappedPeak.bb.chrsz.tmp /hd2/atacseq_taf/output/ATAC_5lKO/out/peak/macs2/pooled_rep/ATAC_5lKO_TAAGGCGA_L001_R1_001.trim.PE2SE.nodup.tn5_pooled.pf.filt.gappedPeak.bb

# SYS command. line 35

rm -f /hd2/atacseq_taf/output/ATAC_5lKO/out/peak/macs2/pooled_rep/ATAC_5lKO_TAAGGCGA_L001_R1_001.trim.PE2SE.nodup.tn5_pooled.pf.filt.gappedPeak.bb.tmp /hd2/atacseq_taf/output/ATAC_5lKO/out/peak/macs2/pooled_rep/ATAC_5lKO_TAAGGCGA_L001_R1_001.trim.PE2SE.nodup.tn5_pooled.pf.filt.gappedPeak.bb.chrsz.tmp

# SYS command. line 39

TASKTIME=$[$(date +%s)-${STARTTIME}]; echo "Task has finished (${TASKTIME} seconds)."

End coordinate 61342464 bigger than chr19 size of 61342430 line 180743 of /hd2/atacseq_taf/output/ATAC_5lKO/out/peak/macs2/pooled_rep/ATAC_5lKO_TAAGGCGA_L001_R1_001.trim.PE2SE.nodup.tn5_pooled.pf.filt.gappedPeak.bb.tmp

the test log
INFO:root:Reading bowtie alignment log...
INFO:root:18014559 reads; of these:
INFO:root:18014559 (100.00%) were paired; of these:
INFO:root:226896 (1.26%) aligned concordantly 0 times
INFO:root:8040724 (44.63%) aligned concordantly exactly 1 time
INFO:root:9746939 (54.11%) aligned concordantly >1 times
INFO:root:----
INFO:root:226896 pairs aligned concordantly 0 times; of these:
INFO:root:49707 (21.91%) aligned discordantly 1 time
INFO:root:----
INFO:root:177189 pairs aligned 0 times concordantly or discordantly; of these:
INFO:root:354378 mates make up the pairs; of these:
INFO:root:232838 (65.70%) aligned 0 times
INFO:root:21941 (6.19%) aligned exactly 1 time
INFO:root:99599 (28.11%) aligned >1 times
INFO:root:99.35% overall alignment rate
INFO:root:Getting mitochondrial chromosome fraction...
INFO:root:Getting GC bias...
INFO:root:java -Xmx4G -jar /home/rick/anaconda3/envs/bds_atac/bin/../share/picard*/picard.jar CollectGcBiasMetrics R=/hd2/bds_genome_data/mm9/mm9/mm9.fa I=/hd2/atacseq_taf/output/ATAC_5lKO/out/align/rep1/ATAC_5lKO_TAAGGCGA_L001_R1_001.trim.PE2SE.nodup.bam O=/hd2/atacseq_taf/output/ATAC_5lKO/out/qc/rep1/ATAC_5lKO_TAAGGCGA_L001_R1_001.trim.PE2SE_gc.txt VERBOSITY=ERROR QUIET=TRUE ASSUME_SORTED=FALSE CHART=/hd2/atacseq_taf/output/ATAC_5lKO/out/qc/rep1/ATAC_5lKO_TAAGGCGA_L001_R1_001.trim.PE2SE_gcPlot.pdf S=/hd2/atacseq_taf/output/ATAC_5lKO/out/qc/rep1/ATAC_5lKO_TAAGGCGA_L001_R1_001.trim.PE2SE_gcSummary.txt
INFO:root:Running preseq...
INFO:root:preseq lc_extrap -P -B -o /hd2/atacseq_taf/output/ATAC_5lKO/out/qc/rep1/ATAC_5lKO_TAAGGCGA_L001_R1_001.trim.PE2SE.preseq.dat /hd2/atacseq_taf/output/ATAC_5lKO/out/qc/rep1/ATAC_5lKO_TAAGGCGA_L001_R1_001.trim.PE2SE.sorted.bam -v 2> /hd2/atacseq_taf/output/ATAC_5lKO/out/qc/rep1/ATAC_5lKO_TAAGGCGA_L001_R1_001.trim.PE2SE.preseq.log
INFO:root:samtools mapq 30...
INFO:root:Running Picard MarkDuplicates...
INFO:root:samtools flagstat...
INFO:root:36029118 + 0 in total (QC-passed reads + QC-failed reads)
INFO:root:0 + 0 secondary
INFO:root:0 + 0 supplementary
INFO:root:0 + 0 duplicates
INFO:root:35796280 + 0 mapped (99.35%:-nan%)
INFO:root:36029118 + 0 paired in sequencing
INFO:root:18014559 + 0 read1
INFO:root:18014559 + 0 read2
INFO:root:35575326 + 0 properly paired (98.74%:-nan%)
INFO:root:35772440 + 0 with itself and mate mapped
INFO:root:23840 + 0 singletons (0.07%:-nan%)
INFO:root:40314 + 0 with mate mapped to a different chr
INFO:root:16449 + 0 with mate mapped to a different chr (mapQ>=5)
INFO:root:final read counts...
INFO:root:insert size distribution...
INFO:root:java -Xmx4G -jar /home/rick/anaconda3/envs/bds_atac/bin/../share/picard*/picard.jar CollectInsertSizeMetrics INPUT=/hd2/atacseq_taf/output/ATAC_5lKO/out/align/rep1/ATAC_5lKO_TAAGGCGA_L001_R1_001.trim.PE2SE.nodup.bam OUTPUT=/hd2/atacseq_taf/output/ATAC_5lKO/out/qc/rep1/ATAC_5lKO_TAAGGCGA_L001_R1_001.trim.PE2SE.inserts.hist_data.log H=/hd2/atacseq_taf/output/ATAC_5lKO/out/qc/rep1/ATAC_5lKO_TAAGGCGA_L001_R1_001.trim.PE2SE.inserts.hist_graph.pdf VERBOSITY=ERROR QUIET=TRUE W=1000 STOP_AFTER=5000000
INFO:root:Generating tss plot...
INFO:root:signal to noise...
INFO:root:bigWigAverageOverBed /hd2/atacseq_taf/output/ATAC_5lKO/out/signal/macs2/rep1/ATAC_5lKO_TAAGGCGA_L001_R1_001.trim.PE2SE.nodup.tn5.pf.pval.signal.bigwig /hd2/bds_genome_data/mm9/mm9/ataqc/mm9_dhs_universal_ucsc_v1.bed.gz /hd2/atacseq_taf/output/ATAC_5lKO/out/qc/rep1/ATAC_5lKO_TAAGGCGA_L001_R1_001.trim.PE2SE.signal
INFO:root:ENCFF001PCS	SpearmanrResult(correlation=0.033773658301836137, pvalue=0.0)
INFO:root:ENCFF001PDB	SpearmanrResult(correlation=0.045394639569752246, pvalue=0.0)
INFO:root:ENCFF001PDP	SpearmanrResult(correlation=0.03117305766539559, pvalue=0.0)
INFO:root:ENCFF001PEA	SpearmanrResult(correlation=0.064350783611477114, pvalue=0.0)
INFO:root:ENCFF001QEN	SpearmanrResult(correlation=0.11751503760067165, pvalue=0.0)
INFO:root:ENCFF001QBV	SpearmanrResult(correlation=0.058886984313049347, pvalue=0.0)
INFO:root:ENCFF001PZU	SpearmanrResult(correlation=0.048606934562090573, pvalue=0.0)
INFO:root:ENCFF001PFG	SpearmanrResult(correlation=0.058729132628544053, pvalue=0.0)
INFO:root:ENCFF001PGU	SpearmanrResult(correlation=0.11441127014544508, pvalue=0.0)
INFO:root:ENCFF001PHE	SpearmanrResult(correlation=0.084972152825066191, pvalue=0.0)
INFO:root:ENCFF001PHL	SpearmanrResult(correlation=0.10805913535450433, pvalue=0.0)
INFO:root:ENCFF001PHS	SpearmanrResult(correlation=0.092773761325087106, pvalue=0.0)
INFO:root:ENCFF001PIB	SpearmanrResult(correlation=0.099420586283102103, pvalue=0.0)
INFO:root:ENCFF001PNK	SpearmanrResult(correlation=0.17334718873462551, pvalue=0.0)
INFO:root:ENCFF001PVW	SpearmanrResult(correlation=0.089321528179288359, pvalue=0.0)
INFO:root:ENCFF001PIK	SpearmanrResult(correlation=0.32975026450162248, pvalue=0.0)
INFO:root:ENCFF001PIY	SpearmanrResult(correlation=0.32913102263702887, pvalue=0.0)
INFO:root:ENCFF001PKP	SpearmanrResult(correlation=0.092786146689878099, pvalue=0.0)
INFO:root:ENCFF001PKY	SpearmanrResult(correlation=0.086798762024097684, pvalue=0.0)
INFO:root:ENCFF001PLI	SpearmanrResult(correlation=0.19039392295092475, pvalue=0.0)
INFO:root:ENCFF001LTZ	SpearmanrResult(correlation=0.15161944771329813, pvalue=0.0)
INFO:root:ENCFF001LUI	SpearmanrResult(correlation=0.072168420005018871, pvalue=0.0)
INFO:root:ENCFF001PLW	SpearmanrResult(correlation=0.069049288296783431, pvalue=0.0)
INFO:root:ENCFF001PMI	SpearmanrResult(correlation=0.14039189856903245, pvalue=0.0)
INFO:root:ENCFF001PMR	SpearmanrResult(correlation=0.17890894073017335, pvalue=0.0)
INFO:root:ENCFF001QCS	SpearmanrResult(correlation=0.067259461370709289, pvalue=0.0)
INFO:root:ENCFF001PNU	SpearmanrResult(correlation=0.11656677312462088, pvalue=0.0)
INFO:root:ENCFF001PON	SpearmanrResult(correlation=0.090301961635109354, pvalue=0.0)
INFO:root:ENCFF001PPI	SpearmanrResult(correlation=0.14340252460503231, pvalue=0.0)
INFO:root:ENCFF001PTV	SpearmanrResult(correlation=0.088709205487457948, pvalue=0.0)
INFO:root:ENCFF001PUI	SpearmanrResult(correlation=0.041341040870329318, pvalue=0.0)
INFO:root:ENCFF001PWJ	SpearmanrResult(correlation=0.053202448119509249, pvalue=0.0)
INFO:root:ENCFF001PUV	SpearmanrResult(correlation=0.20143252463051384, pvalue=0.0)
INFO:root:ENCFF001QBE	SpearmanrResult(correlation=0.085697950810573756, pvalue=0.0)
INFO:root:ENCFF001PWW	SpearmanrResult(correlation=0.12324226976911377, pvalue=0.0)
INFO:root:ENCFF001PXH	SpearmanrResult(correlation=0.071244657821574187, pvalue=0.0)
INFO:root:ENCFF001OQF	SpearmanrResult(correlation=0.18337167266240909, pvalue=0.0)
INFO:root:ENCFF001PYR	SpearmanrResult(correlation=0.14473639332147181, pvalue=0.0)
INFO:root:ENCFF001PZE	SpearmanrResult(correlation=0.07972817386565216, pvalue=0.0)
INFO:root:ENCFF001PES	SpearmanrResult(correlation=0.069160517857457471, pvalue=0.0)
INFO:root:ENCFF001PFY	SpearmanrResult(correlation=0.075503324225279431, pvalue=0.0)
INFO:root:ENCFF001QAG	SpearmanrResult(correlation=0.1032940188731358, pvalue=0.0)
INFO:root:ENCFF001PKH	SpearmanrResult(correlation=0.30859281272295885, pvalue=0.0)
INFO:root:ENCFF001QHL	SpearmanrResult(correlation=0.28408478077383448, pvalue=0.0)

@rfarouni can you pass along that gapped Peak file that had the error so that we can replicate it? thanks!

Hi Daniel,
When I downloaded the latest mm9 genome annotations I still received the same concatenation error. I tried running the pipeline with mm10 and did not receive this error. My future analyses will be conducted with mm10 so I will close this issue. Thanks for all your help I really appreciate it!
Best regards,
Michael

Hi Daniel,
I am wondering whether you have fixed mm9 genome annotation problem after previous issue post


Can a button/link on the toolbar show a specific photo?

I saw in an old post for an older v2.x version of photoswipe that the .show function can be used show a specific photo in the list.
I want to be able to add a link to the toolbar that will go to a specific photo.
Is this possible in v3.x ?
Thanks

I'd also like this functionality.
Currently I'm able to use instance.show(0) to start the slide on the first image and can start the slideshow on a different image with  instance.show(5) but show() can't be used once the slideshow has been opened.
It would be nice to have access to other functions like:
this.slideCarousel(point, action)
instance.moveCarousel(point)
I found both of these in the photoswipe code but don't know how I can access them in my script.


Make sure all failures to load module interfaces are diagnosed

...specifically, diagnosed in the parent DiagnosticEngine. This not only provides a better user experience, but makes sure that the compiler exits with a nonzero exit code even if the module goes unused.
rdar://problem/50789839

@swift-ci Please smoke test

@swift-ci Please smoke test


Hot reload in angular 10????

🚀 HMR
Description
 A clear and concise description of the problem or missing capability...
https://levelup.gitconnected.com/hmr-in-angular-cli-project-eb8512ef514b this doesn't work on angular 10, Is there a way we can enable HMR in angular 10?
Describe the solution you'd like
Making a small change reloads the entire application, instead should refresh the specific component.

Duplicate of https://github.com/angular/angular-cli/issues/17324

See the following comments for more context:
https://github.com/angular/angular-cli/issues/17324#issuecomment-686373741
https://github.com/angular/angular-cli/issues/17324#issuecomment-686440970


If you're receiving audio and video from RTSP you will need to split the pipeline and have depayer for h264 as well as for the audio (for example: https://gstreamer.freedesktop.org/documentation/rtp/rtpac3depay.html?gi-language=c), then you need to mux these together to a single stream and then send to kvssink.
Here's and example from someone else who tried to do this, if you follow this it should work for you!

Hi,
Video:
Codec: h264
Resolution: 852 X 480
Bitrate: 1083 kbps VBR
Audio:
Codec: aac stereo
Sample rate: 48000
Bitrate: 67 kbps VBR
I have using these configuration for my rtsp but my audio is not coming on aws kinesis-video-stream. I don't know much about gst-launch-1.0 library, can you please share with me complete command.
Thanks

Having the same issue here. Did anyone solve it?

Hello, this is a pure gstreamer issue unrelated to the internals of kvssink. Your pads aren't connecting between your rtspsrc and rtpPCMAdepay elements.

Something like that should work.
Very similar to this issue someone had with a/v pipelines.
https://github.com/awslabs/amazon-kinesis-video-streams-webrtc-sdk-c/issues/1014


Long play mode and alternative conditions

There is an undocumented option in verifier that allows checking user generated content from consumer devices like iPhone’s profile 8.4 video, named --alternative-conditions or -ac, it supports ugch4_ios14, ugch4_ios15 etc. The ios14 mode requires  scene_reflash_flag  always be 1, and actually it’s expected (but not limited) to be DM2 and only has L1 block. It’s also the behavior of some broadcasting devices, which only analyze frame based L1 values, in realtime. It’s a mode that RPU generated from this tool being considered as fully valid, it’s worth implementing this as an option. See this branch
Btw ugch4 modes have different recommended composer metadata values, and ios15 mode has strictly defined L8, L9, L10 and L11, different values will not pass the verifier. I don’t know if it’s worth implementing generating with ability to specify alternative conditions.
The recent version of Final Cut Pro also generates profile 8.4 video with ios15 condition, but Resolve beta on Windows doesn’t follow that.

Making specific presets for these "alternative conditions" isn't something I'm interested in.
It's all possible to do with the current generate config.
By default, iOS15 (my samples are only labeled iPhone 11 to 13) profile 8.4 look like:

Default L3
L9 with DCI-P3 D65 primaries
Some automatically generated L8 for a 400 nits target
L10 for 400 nits target, not sure what target display 20 is.

Not sure what target primary 2 is, maybe some real device DCI-P3.

L11 with content type 4, UGC.

Do you plan on  creating a PR for the long play mode config? It's almost good enough but I was planning to do it anyways.
Also, I don't think profile 5 with profile 7/8 DM metadata is valid (especially the RGB coeffs).
Although Apple once made HDR10 profile 5 files, which I don't believe are valid.

I didn't use the profile 8 coeffs for profile 5 in that commit, just used the default composer metadata of profile 8.1 (no reshaping), with correct IPT dm_data (coeffs).

You're right, I forgot I had done this already for Profile5::dm_data.

DCI-P3 D65 primaries

That is called P3-D65.


Toast setup

mainly focused on the visual here. in this mock case, we have 2 toasts, a confirmed and a failed. they will appear in order of when we received them based on lastUpdated then display one at a time. after 3 seconds, it's automatically dismissed.

https://github.com/AugurProject/augur/issues/9913

https://github.com/AugurProject/augur/issues/9913


Prompt gets lost in auto-execute mode

In auto-execute mode, the prompt gets lost. It would be great to keep it.
Alternatively, we could remove the auto-execute mode. It was critized for limited reproducibility as the prompts are lost.

I think it is important to make clear that the user of AI must understand, check, and be responsible for the code that is going to be executed. Along those lines I feel removing auto-execution would be good (and also we would need to maintain less code).

Yes, fully agreed. I also had some discussions in this context. I created an issue so that we can keep track of this: https://github.com/haesleinhuepf/bia-bob/issues/164


add ''wait for action" option

like client.wait_for_server()
obviously there is a (small) difference in just listening to goal/status topic and waiting for client.wait_for_server() discovered by @HannesBachter
analog to #117

fixed in https://github.com/floweisshardt/atf/pull/131


--flannel-backend option not recognised when starting agent with it

Version:
k3s version v1.0.0 (18bd921c)
Describe the bug
Not sure it is expected, but when I run k3s in agent mode specifying ipsec as the backend for flannel, I get this:
flag provided but not defined: -flannel-backend

However, the list of options available for the agent shows this:
--no-flannel                        (deprecated) use --flannel-backend=none

suggesting that the --flannel-backend option should be available.
If I start the server with that option, and the agents without it, will the traffic be encrypted or not?
To Reproduce
Start the agent with --flannel-backend=ipsec
Expected behavior
I thought that I would have to specify the flannel backend for the agents as well, and the list of options seems to suggest that this should be possible/required for the traffic to be encrypted.
Actual behavior
The --flannel-backend  option is not recognised when starting the agent with it.

It seems that this flag is only need on the master. When i launch the agent, there is some log about ipsec. I guess it got the right config from the master.

Hi @sebastien-prudhomme I also saw some ipsec lines in the logs so I guess I should be good. Thanks!

Hi,
I ask to reopen this please. I got the issue with K3s v1.17 when using the --flannel-backend=none on K3s agent cause I want to use calico instead.
The service unit fails with:
[root@slave1 ~]# systemctl status k3s-agent -l
● k3s-agent.service - Lightweight Kubernetes
   Loaded: loaded (/etc/systemd/system/k3s-agent.service; enabled; vendor preset: disabled)
   Active: activating (auto-restart) (Result: exit-code) since Wed 2020-06-10 02:49:22 CDT; 3s ago
     Docs: https://k3s.io
  Process: 4907 ExecStart=/usr/local/bin/k3s agent --flannel-backend=none --debug (code=exited, status=1/FAILURE)
  Process: 4905 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
  Process: 4903 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
 Main PID: 4907 (code=exited, status=1/FAILURE)

Jun 10 02:49:22 slave1.k3s.local k3s[4907]: time="2020-06-10T02:49:22-05:00" level=fatal msg="flag provided but not defined: -flannel-backend"
Jun 10 02:49:22 slave1.k3s.local systemd[1]: k3s-agent.service: main process exited, code=exited, status=1/FAILURE
Jun 10 02:49:22 slave1.k3s.local systemd[1]: Unit k3s-agent.service entered failed state.
Jun 10 02:49:22 slave1.k3s.local systemd[1]: k3s-agent.service failed.

But it woks fine if I use the command line. Also the service does not work if using the legacy --no-flanel
JF

So tweaking the service unit for K3s with:
ExecStart=/usr/local/bin/k3s \
    agent \
        '\--flannel-backend=none'

did the trick. Please note the \ before the --

You shouldn't be configuring the flannel backend on the agent, which is why the flag is deprecated. Flannel configuration for agents is controlled by the server and pushed to agents when they start.

That's what I thought initally but I got flannel running on my worker nodes even when I said no flannel CNI at the server/master. This situation messed up with the BGP communication needed by Calico between Server and Agent.
This why I tried to deactivate it at agent level which is not deprecated (using --no-flannel is deprecated, not --flannel-backend=none).
Still there is an issue cause I got the same problem with -- debug. When the service is created, K3s cries cause systemctl asks it to run k3s agent -debug and it does not know -debug. This has to be solved with the same trick: putting '\--debug' in the service unit.

I think this might be unique to your environment. I have multiple flags passed to both the agent and server without needing to muck about with quoting or escaping characters. What does your unit file look like?
I also see the flannel configuration being properly pulled from the server without having to pass it to each individual agent.

So you think it pulls, I think it installs it by default. It does not take it from the serve.
I installed my server/Master with no flannel CNI cause I use Calico there and still my worker , by default, after joining got Flannel. Which messed up with Calico BGP.
The only way to not have Flannel installed on the worker node was to use the non deprecated --flannel-backend=none flag on the K3s installer script:
K3S_URL=https://${K3S_MASTER_HOSTNAME}:6443 K3S_TOKEN=<K3s Master Token> INSTALL_K3S_SKIP_DOWNLOAD=true all-in-one/scripts/Rancher/install.sh --flannel-backend=none

The service unit for K3s seemed to be OK:
ExecStart=/usr/local/bin/k3s \
    agent \
        '--flannel-backend=none'

but when starting I got the following error:
[root@slave1 ~]# systemctl status k3s-agent -l
● k3s-agent.service - Lightweight Kubernetes
   Loaded: loaded (/etc/systemd/system/k3s-agent.service; enabled; vendor preset: disabled)
   Active: activating (auto-restart) (Result: exit-code) since Wed 2020-06-10 02:49:22 CDT; 3s ago
     Docs: https://k3s.io
  Process: 4907 ExecStart=/usr/local/bin/k3s agent --flannel-backend=none --debug (code=exited, status=1/FAILURE)
  Process: 4905 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
  Process: 4903 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
 Main PID: 4907 (code=exited, status=1/FAILURE)

Jun 10 02:49:22 slave1.k3s.local k3s[4907]: time="2020-06-10T02:49:22-05:00" **level=fatal msg="flag provided but not defined: -flannel-backend"**
Jun 10 02:49:22 slave1.k3s.local systemd[1]: k3s-agent.service: main process exited, code=exited, status=1/FAILURE
Jun 10 02:49:22 slave1.k3s.local systemd[1]: Unit k3s-agent.service entered failed state.
Jun 10 02:49:22 slave1.k3s.local systemd[1]: k3s-agent.service failed.

Only escaping the --flannel-backend=none in the service unit allowed me to have the service started without flannel.
I got the same issue with : K3S_URL=https://${K3S_MASTER_HOSTNAME}:6443 K3S_TOKEN=<K3s Master Token> INSTALL_K3S_SKIP_DOWNLOAD=true all-in-one/scripts/Rancher/install.sh --debug
Where I got the corresponding error : level=fatal msg="flag provided but not defined: -debug

You're not making this work by adding backslashes, you're just making it ignore those arguments entirely. I'm not super familiar with the argument parser that k3s is using, but it appears you can pass it anything you want if you prefix it with a backslash:
root@dev01:~# /usr/local/bin/k3s agent '--total-garbage=yes'
FATA[0000] flag provided but not defined: -total-garbage

root@dev01:~# /usr/local/bin/k3s agent '\--total-garbage=yes'
FATA[2020-06-12T14:33:42.655779916-07:00] --token is required

If you want to see the actual valid arguments you can run /usr/local/bin/k3s --help and /usr/local/bin/k3s agent --help. Note that --debug is a global flag, not a flag for the agent command, so you if you want to add it you have to put it before agent: usr/local/bin/k3s --debug agent or set the K3S_DEBUG environment variable.


email mentioned in FAQ as optional

Hi,
email mentioned in FAQ as optional but in form it's required

Ah, I see, thanks for the clarification! :)


Hash cache

I added a hash cache thing which improves the startup time and makes switching between replay files faster. The improvements are more noticable the larger the user's beatmap folder is.
Whenever a replay is picked or on boot, it checks if its hash is in the database and if it is it gets the folder name from the database. If its not in the database it uses the old method to find the correct folder and adds it to the database.
first pull request so its probably bad :D

Caching hashes is a very good idea, and I appreciate that! There is 2 small issues, though. The first one is that you pushed __pycache__ folders by mistake.
The second issue is that if I run main.py from another directory, it will create another hashCache.db in my current directory instead of using the existing hashCache.db in osr2mp4app/.
Having only one .db file is a good idea in my opinion.

I added the database to abspath and got rid of the pycache folders.

👍


Rewrite computed implementation

It needs to be more imperative, like class construction.
({
    [foo]() {},
    get x() {},
    y: 1
})

Translates to something like:
_esdown.object(function(__) {
    __(foo, function() {});
    __.get("x", function() {});
    __("y", 1);
});

626339fad9a53472c8fa2948f2a0560617fbc72b


[Feature request] Allow overwriting file if exists

It would be nice to have overwriting dmg option if file already exists. For now we just have following error:
✖ Error: Target already exists

Yes, but should be opt-in with a flag: --overwrite.


Doesn't work

Now you can boot untrusted images.
[libusb] Waiting for the USB handle with VID: 0x5AC, PID: 0x1227
Found the USB handle.
usb_timeout: 5
[libusb] Waiting for the USB handle with VID: 0x5AC, PID: 0x1227
CPID: 0x8010
Found the USB handle.
Now you can boot untrusted images.
[libusb] Waiting for the USB handle with VID: 0x5AC, PID: 0x1227
Found the USB handle.
main: Starting...
iOS 15 iBoot detected!
getting get_debug_enabled_patch() patch
getting get_sigcheck_patch() patch
applying patch=0x1800c3ecc : 000080d2
applying patch=0x1800c3f14 : 000080d2
applying patch=0x1800c5b04 : 200080d2
main: Writing out patched file to work/iBSS.patched...
main: Quitting...
none
main: Starting...
iOS 15 iBoot detected!
getting get_boot_arg_patch(rd=md0 debug=0x2014e -v wdt=-1 ) patch
getting get_debug_enabled_patch() patch
getting get_unlock_nvram_patch() patch
getting get_sigcheck_patch() patch
applying patch=0x1800c3ecc : 000080d2
applying patch=0x1800c3f14 : 000080d2
applying patch=0x1800b1f64 : 000080d2c0035fd6
applying patch=0x1800b1fbc : 000080d2c0035fd6
applying patch=0x1800eeb30 : 000080d2c0035fd6
applying patch=0x1800c5b04 : 200080d2
applying patch=0x1800c7068 : 58060b50
applying patch=0x1800dd132 : 72643d6d64302064656275673d30783230313465202d76207764743d2d312000
applying patch=0x1800cb3c8 : 1f2003d5
main: Writing out patched file to work/iBEC.patched...
main: Quitting...
none
krnl
main: Starting...
main: Detected fat macho kernel
Kernel: Adding AMFI_get_out_of_my_way patch...
get_amfi_out_of_my_way_patch: Entering ...
get_amfi_out_of_my_way_patch: Kernel-8020 inputted
get_amfi_out_of_my_way_patch: Found Internal Error: No cdhash found. str loc at 0x9a7061
get_amfi_out_of_my_way_patch: Found Internal Error: No cdhash found. str ref at 0x10bcc64
get_amfi_out_of_my_way_patch: Patching AMFI at 0x10b82a8
main: Writing out patched file to work/kcache.patched...
main: Quitting...
krnl
dtre
rtsc
rdsk
Reading work/ramdisk.dmg...
IM4P outputted to: work/ramdisk.im4p
Reading work/ramdisk.im4p...
Reading work/IM4M...
Creating Image4...
Image4 file outputted to: sshramdisk/ramdisk.img4
none
[] Booting ramdisk
usb_timeout: 5
[libusb] Waiting for the USB handle with VID: 0x5AC, PID: 0x1227
CPID: 0x8010
Found the USB handle.
Now you can boot untrusted images.
usb_timeout: 5
[libusb] Waiting for the USB handle with VID: 0x5AC, PID: 0x1227
Found the USB handle.
[==================================================] 100.0%
[==================================================] 100.0%
[==================================================] 100.0%
[==================================================] 100.0%
[==================================================] 100.0%
[==================================================] 100.0%
[==================================================] 100.0%
[] Waiting for the ramdisk to finish booting
libusb: warning [op_get_configuration] device unconfigured
libusb: error [op_get_active_config_descriptor] device unconfigured
[13:30:47.815][3] Could not get old configuration descriptor for device 2-14: LIBUSB_ERROR_NOT_FOUND
[13:30:47.819][3] Connecting to new device on location 0x2000e as ID 1
[13:30:47.920][3] Connected to v2.0 device 1 on location 0x2000e with serial number ramdisk tool Sep 18 2022 20:14:43
[13:30:47.921][2] device_control_input: Got unhandled payload type 5
[13:30:48.008][2] device_control_input: Got unhandled payload type 5
Error connecting to device: No error information
8
[*] Testing for baseband presence
7
8
9
seputil: Gigalocker file (/mnt7/D93C2246-3BEE-53F9-8546-EC66A02BE0C0.gl) exists
seputil: Gigalocker initialization completed
sepi digest (48 bytes): 8b10b8e5b66ed27bc449d9923a4aca57e2c49cccbc2e85bb15f827ca7d77f171cd00341528ad1f3519faf9e041302255
sepi nonce (20 bytes): 0cae5a2c2c152063b005d7cf35d4562e279f5aef
rsep digest (48 bytes): f7deb217718e7422fd14a484e0dfd2d6a5517546094c90834489ee320eeccdb79d85567fd5d5f3da2ed766a5c91d8c03
rsep nonce (20 bytes): 0cae5a2c2c152063b005d7cf35d4562e279f5aef
[13:31:07.511][3] Removed device 1 on location 0x2000e
Connection to localhost closed by remote host.
In the screen of the iPad there's wrote: please go to https://panic.apple.com to report this panic

ios 16 ?

Yes

Yes

that's great bro, you got bootloop your ipad, you are very cool


Feature/cleanup on exit

By default, curator-test-rule leaves behind a lot of data in temp directories. Especially when using LocalZooKeeperRule, this can quickly fill up tmp.

Closing so that the two aren't linked


[SPARK-24068] Propagating DataFrameReader's options to Text datasource on schema inferring

What changes were proposed in this pull request?
While reading CSV or JSON files, DataFrameReader's options are converted to Hadoop's parameters, for example there:
https://github.com/apache/spark/blob/branch-2.3/sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala#L302
but the options are not propagated to Text datasource on schema inferring, for instance:
https://github.com/apache/spark/blob/branch-2.3/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVDataSource.scala#L184-L188
The PR proposes propagation of user's options to Text datasource on scheme inferring in similar way as user's options are converted to Hadoop parameters if schema is specified.
How was this patch tested?
The changes were tested manually by using https://github.com/twitter/hadoop-lzo:
hadoop-lzo> mvn clean package
hadoop-lzo> ln -s ./target/hadoop-lzo-0.4.21-SNAPSHOT.jar ./hadoop-lzo.jar

Create 2 test files in JSON and CSV format and compress them:
$ cat test.csv
col1|col2
a|1
$ lzop test.csv
$ cat test.json
{"col1":"a","col2":1}
$ lzop test.json

Run spark-shell with hadoop-lzo:
bin/spark-shell --jars ~/hadoop-lzo/hadoop-lzo.jar

reading compressed CSV and JSON without schema:
spark.read.option("io.compression.codecs", "com.hadoop.compression.lzo.LzopCodec").option("inferSchema",true).option("header",true).option("sep","|").csv("test.csv.lzo").show()
+----+----+
|col1|col2|
+----+----+
|   a|   1|
+----+----+

spark.read.option("io.compression.codecs", "com.hadoop.compression.lzo.LzopCodec").option("multiLine", true).json("test.json.lzo").printSchema
root
 |-- col1: string (nullable = true)
 |-- col2: long (nullable = true)

Test build #89929 has finished for PR 21182 at commit 8a8ff3f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89929/
Test FAILed.

jenkins, retest this, please

Test build #89942 has started for PR 21182 at commit 8a8ff3f.

Test build #89942 has finished for PR 21182 at commit 8a8ff3f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89942/
Test FAILed.

Test build #89953 has started for PR 21182 at commit 1fa871d.

Test build #89953 has finished for PR 21182 at commit 1fa871d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89953/
Test PASSed.

Test build #90010 has started for PR 21182 at commit 9f55aa8.

jenkins, retest this, please

Test build #90047 has started for PR 21182 at commit 9f55aa8.

Test build #90047 has finished for PR 21182 at commit 9f55aa8.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90047/
Test FAILed.

cc @gengliangwang

Seems fine otherwise.

Test build #90269 has started for PR 21182 at commit cd1ffad.

Test build #90270 has started for PR 21182 at commit d138c44.

Test build #90270 has finished for PR 21182 at commit d138c44.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90270/
Test FAILed.

Test build #90276 has started for PR 21182 at commit 6610826.

Test build #90269 has finished for PR 21182 at commit cd1ffad.

This patch fails from timeout after a configured wait of `300m`.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90269/
Test FAILed.

jenkins, retest this, please

Test build #90284 has started for PR 21182 at commit 6610826.

Test build #90276 has finished for PR 21182 at commit 6610826.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90276/
Test PASSed.

Test build #90284 has finished for PR 21182 at commit 6610826.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90284/
Test FAILed.

jenkins, retest this, please

Test build #90286 has started for PR 21182 at commit 6610826.

Test build #90286 has finished for PR 21182 at commit 6610826.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90286/
Test PASSed.

@HyukjinKwon @viirya @gengliangwang May I ask you to look at the PR again.

Merged to master.

shall we backport it to 2.3?

I tried but it had conflict. @MaxGekk mind opening a backport pr?

@HyukjinKwon Sure, I will prepare a PR

Here is the backport to 2.3: https://github.com/apache/spark/pull/21292


Add unit and widget tests to existing library code

pending #14

remainder:

actor
adapter
tree

maybe no more unit-testable things

#21


added SIGTERM support

I'm not entirely sure why SIGTERM handling isn't the default in ctrlc, but it isn't, so this is required to make a rosrust application respond to that signal gracefully.

Friendly ping!

Didn't know this was feature-gated. Thanks for the fix!


Typewriter dimming does not work on separate window

Hello, if I enable "Toggle typewriter scrolling and dimming on current paragraph" the dimming feature does not work if I'm not on the main obsidian window.

Is "separate window" referring to the window of a different vault? Unfortunately there is no way to exchange information between vaults, because each vault loads a new instance of all plugins. Thus the dimming cannot sync across vault windows.

Is "separate window" referring to the window of a different vault? Unfortunately there is no way to exchange information between vaults, because each vault loads a new instance of all plugins. Thus the dimming cannot sync across vault windows.

Obsidian has a feature that allow to open a note on a separate window (same vault, different window).

I released a (possible) fix as version 0.0.3-beta.2 which you can install using the BRAT plugin (as described in the README). Feel free to check it out and tell me if it works for you, so that I can include this in the next release.


Students cannot see Exam Tab with active Exam

Describe the bug
Students cannot see the Exam Tab in the Course Overview if an exam is visible.
To Reproduce

Create an Exam with a future visible date
Check with Student permissions that exam tab is visible
Wait until the exam is visible
Check with Student permissions that exam tab not visible

Expected behavior
The exam should be visible to the student.

Should be fixed with https://github.com/ls1intum/Artemis/pull/1740 and https://github.com/ls1intum/Artemis/commit/c845c7eee6f99bb4eb2b957b50904e7ab0e5eb4f


Complete html comment remover

complete the code at the remove html function
It must read a html file, ignore the comments, and rewrite the file again without the comments

please assign this issue to me

of course! let me know if you need extra details


Debug in VSCode

I had been playing with create-react-app and was able to debug it using VSCode. I removed it and added in LBD and now I'm not getting the same behavior. Have you had anyone who has used VSCode to debug this or run into this issue?
Here's my launch.json setting:
{ //make sure the Debugger for Chrome extension is installed
            "type": "chrome",
            "request": "launch",
            "name": "Front-End",
            "url": "http://localhost:3000",
            "webRoot": "${workspaceRoot}/Nri.ProposalTracker.Ui/front-end/src",
            "preLaunchTask": "start-devserver",
            "userDataDir": "${workspaceRoot}/.vscode/.chrome",
            "sourceMapPathOverrides": {
                "webpack:///*": "*",
                "webpack:///src/*": "${webRoot}/*",
                "webpack:///./~/*": "${webRoot}/node_modules/*"
            }
        }

And here's my tasks.json setting:
{
            "taskName": "start-devserver",
            "command": "yarn start",
            "type": "shell",
            "options": {
                "cwd": "${workspaceRoot}/Nri.ProposalTracker.Ui/front-end"
            },
            "presentation": {
                "echo": true,
                "reveal": "always",
                "focus": false,
                "panel": "shared"
            }
        }

Disregard! I rebooted Windows and all is well... Not sure what that was about.


tfchain client wallet creation fails

Branch: Development
Reproduce:
c = j.clients.tfchain.new('my_client', network_type='TEST')
w = c.wallets.new("my_wallet")
Gives following error:
JSX> w = c.wallets.new("my_wallet")  # a new seed will be generated
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/sandbox/lib/jumpscale/Jumpscale/core/BASECLASSES/JSBaseConfigs.py", line 59, in new
    self._children[name] = kl(parent=self, data=data, **kwargs)
  File "/sandbox/lib/jumpscale/Jumpscale/core/BASECLASSES/JSBaseConfig.py", line 19, in __init__
    JSBaseDataObj.__init__(self,data=data, parent=parent, topclass=False,**kwargs)
  File "/sandbox/lib/jumpscale/Jumpscale/core/BASECLASSES/JSBaseDataObj.py", line 36, in __init__
    if kwargs=={} and data==None:
  File "/sandbox/lib/jumpscale/Jumpscale/data/schema/DataObjBase.py", line 206, in __eq__
    return self._data == val._data
  File "/sandbox/lib/jumpscale/Jumpscale/data/schema/DataObjBase.py", line 159, in _data
    self._cobj_=self._cobj.as_builder()
  File "/sandbox/var/codegen/schema_jumpscale_tfchain_wallet.py", line 243, in _cobj
    raise RuntimeError(msg)
RuntimeError:
ERROR: could not create capnp message
    {
     "keyCount": 1,
     "keyScanCount": -1,
     "name": "my_wallet"
    }

schema:
    <capnp.lib.capnp._StructModule object at 0x110230ac8>

error was:
capnp/dynamic.c++:1675: failed: expected value >= 0 && T(value) == value; Value out-of-range for requested type.; value = -1
stack: 0x10e0c3980 0x10e0c7f63 0x10df7ab4f 0x10df3bb76 0x10df82b2a 0x10dfdad57 0x10dfca877 0x10dfca3ef 0x10bb0e09f 0x10bb0f3f1 0x10bb0e7a1 0x10bb9d2bc 0x10bb0ea6e 0x10bb14b97 0x10bb3b6b3 0x10bba15bb 0x10bb0ea6e 0x10bb14b97 0x10bb3b6b3 0x10bba15bb 0x10bb0ea6e 0x10bb0f304 0x10bb4c753 0x10bb3adb4 0x10bba169b 0x10bba4c6b 0x10bb0e2c1 0x10bb9d2bc 0x10bba4c6b 0x10bb0e2c1 0x10bb0f3f1

deprecated, works right now


Support for BigFix recipe type

@jgstew, can you point us to an example of a working BigFix recipe? I found this but it seems old.
Which arguments are required for BESUploader (e.g. app name, description), and which should be stored in the com.github.autopkg prefs file (e.g. hostname, username, and password)?
In the TODO, you pointed to Windows recipes, but are those the same thing as BigFix recipes?
(Also see #73.)

Oddly enough, the one you referenced might be the only one currently public. I've asked @hansen-m to provide more of them.
The BESEngine processor for AutoPkg can make both Windows and Mac items. The resulting recipes will not be exactly the same however.
I also just realized that the Windows download recipes aren't quite as generic as the others due to having extra processing in them to pull out versions in some cases. It might be that some of this could be shifted over to the BigFix recipe rather than the download recipe, or it may be that it will need to be a different kind of Download recipe.
Right now I'm just working to make BigFix Mac recipes work with recipe-robot as I think that will be the easier one to tackle at first.
BigFix recipes require AppName, Version, Description, and possibly some more values. The BigFix server URL, UserName, and Password could be configured in the prefs file, but there is also a besapi.conf file option.

Now that I'm thinking about this more, I'm remembering things.
The BESUploader is usually not needed. It is to manually upload the dmg/pkg/etc... to the BigFix root server. This isn't needed when a URL can be referenced instead.
The BESImporter is responsible for importing the final task that AutoPkg generates into the BigFix root server so that it can be deployed. Even if this were to fail or not function, you'd still have a generated task from AutoPkg that is useful that you could import manually. While not ideal, this isn't as essential as the actual task generation itself. In some cases the manual import option instead of using BESImporter may be the only option some people have due to politics and other reasons.
The AutoPkgBESEngine is mandatory. It is what does the magic of turning the plist recipe into an actual task that can be deployed with BigFix. It does not require knowledge of the BigFix server, BigFix user, or password to function. It just knows how to transform the data in the plist into the XML format BigFix requires.

AutoPkgBESEngine Processor Arguments

bes_overrideurl - optional, only needed if BESUploader is used
bes_filename - this is the name the downloaded item will have on endpoints when deployed
bes_version - the version of the thing being installed that will be used in the BigFix relevance
bes_title - the title the generated task will have in the BigFix console
bes_description - the HTML description that will be visible in the BigFix console when the install task is selected.
bes_category - the category the task will have in the BigFix console
bes_relevance - an array of relevance items that will be in the generated BigFix task. The order of elements in the array matters.
bes_actions - a dictionary of the action options for in the BigFix installation task. Only 1 is required. This is the actual "script" that will be run on the endpoint that will actually do the installation of the software.

For all software installed or patched by BigFix, there are basically 2 different types of tasks used to do this.

Install/Upgrade - This will install it if it is not installed, or upgrade it if an older version is installed.
This is the option used for any software that you wish to be installed on all systems, or all systems that meet certain criteria. This is also the option most frequently used in BigFix baselines.
Upgrade/Patch - This will only upgrade/patch existing software of an older version. It will not install it to a system that doesn't already have an older version already installed.
This is the option used for updates/patching. This can be used in BigFix baselines that are specific to patching.

Eventually I hope to have some combination of AutoPkg and/or recipe-robot able to automatically generate both of these.

I'm about 80% complete with this, with another 80% to go :)
I need to generalize the actionscript, then test the recipes created by recipe-robot with AutoPkg and the BES Engine, then import them into BigFix by hand and make sure the generated installation tasks actually work. Once this has been done with many different installer types, then I'll work on adding in the BESImporter process to automate that part. The BESUploader will come last.

This is what happens during an autopkg run with a BigFix recipe when the AutoPkgBESEngine processor is not present.
autopkg run TextWrangler.bigfix.recipe
Processing TextWrangler.bigfix.recipe...
Failed.

The following recipes failed:
    TextWrangler.bigfix.recipe
        Unknown processor 'AutoPkgBESEngine'.

Nothing downloaded, packaged or imported.

@jgstew Does that mean that we need to add the two autopkg install commands above to the [REMINDER] text?

@homebysix Yes, it does mean that. I was documenting the issue before making those changes.

This is a resulting BigFix task that works that was generated by an AutoPkg recipe that was generated with recipe-robot: https://bigfix.me/fixlet/details/6285
I did need to change the parent download recipe, but otherwise it was unchanged. There is definitely some more refinement, improvements, and handling of other cases to be done, but at least one case is working. I was a little worried about potentially needing to deal with escaping characters in a different way that is done automatically, but that seems to be working just fine.

Examples of new cases to handle:

https://bigfix.me/fixlet/details/3944
https://bigfix.me/fixlet/details/3796 (this is more of a AutoPkg case rather than recipe-robot)

I want to get back to this. I can't believe it is coming up on a year.

The skeleton of a function that creates BigFix recipes is still in place, if there's any interest in continuing this work: https://github.com/homebysix/recipe-robot/blob/main/scripts/recipe_robot_lib/recipe_generator.py#L1647


rbd: merge rbd_finish_aio_read into generic and clean up the code

Signed-off-by: Xiubo Li<EMAIL_ADDRESS>

Nice. Thanks.


Admin has no access to root url

Hi,
I downloaded current DSC repo and current Messaging Services repo. I updated Messaging Services dependency to use current repo version (5.0.2).
After I start DSC I access https://localhost:8080/actuator/info and enter admin credentials. Everything works as expected.
On switching to root (https://localhost:8080/) I get a There was an unexpected error (type=Forbidden, status=403).

I also tried this with the current main DSC and Messaging-Services 5.0.2 and it worked. Tried it with both Firefox and Chrome.


rest_api: review logging

Update the docs with info on how to enable it and what the user gets.

To be implemented as part of https://github.com/dlt-hub/dlt/pull/1371

Implemented in https://github.com/dlt-hub/dlt/pull/1371


Fix sidebar indentation and line breaks

Fixes https://github.com/rancher/rancher-docs/issues/227

Closing as Jen has also started working on rearranging the contents of the sidebar. Will revisit this piece once that's done.


Fix invalid format string

Introduced: https://github.com/aspnet/WebSockets/commit/79036ea166f4a1d1efa7ed7cf94fca8f017db4d5

@mikeharder,
Thanks for having already signed the Contribution License Agreement. Your agreement was validated by .NET Foundation. We will now review your pull request.
Thanks,
.NET Foundation Pull Request Bot


[iOS] Incorrect iPhone model

There are entries like iPhone15,2 or iPhone10,6 on the mixpanel dashboard which is incorrect since iPhone 15 is not yet released

Lib version: 2.2.5
RN version: 0.69.9

hi @makarkotlov , the Model number is not the same thing as the generation number, please check this wiki here: https://www.theiphonewiki.com/wiki/Models


validate formio json

how to validate json in formio is there anything to  check form json is valid or not

Hello @AA299,
Thank you for your interest in Form.io!
In the future, please submit your questions to Gitter( located here: https://gitter.im/formio/formio) for our community support.  We do offer Support Packages that are for training, support or development services. The hours can be used in one-hour increments at your convenience over a 90 day period.  We will assign one of our senior developers to work directly with you in online sessions. If you are interested in learning more about our Support Packages, then please contact<EMAIL_ADDRESS>Thank you have a great day!


qrCode.GetGraphic(20) doubt.

I don't understand how exactly the QRCode.GetGraphics works and I've sort of blazed through the available documentation.
I am using the QR Code package to create a qr-code representation of a Public Key that will be photographed by an Android App and later used for secure communication. As such, the image needs to be as accurate as possible as the loss of a single byte on the key sequence will make the whole protocol fail. Mind you that this is experimental at least on my part and I don't know how reliable it would be.
Anywawy, the QR Code image is being done on a Windows Forms .NET Framework Application and I would just like to understand why all examples use:
Bitmap qrCodeImage = qrCode.GetGraphic(20);
And how passing an higher or lower value impacts image quality. Maybe it's a dumb question but I would still like some clarification.

Hi @FranciscoKloganB ,
a QR code consists out of black and white "blocks", the so-called "modules". The more data/information you like to store in a QR code (=the longer your input string is), the more blocks are needed to store the information.
The GetGraphic-functions takes a matrix of modules information (x position, y position, color (black/white)) and create the actual QR Code image out of it. The QR code image is a Bitmap (at least for the "QRCode" class). Bitmaps consist out of pixels. The parameter (e.g. 20) says how much pixels will be used to draw a single module.
For example if you create a QR code which has 20x20 modules and you call GetGraphics(20) then you get a bitmap file with the dimensions of 400x400 px.
You should use the parameter this way, that the outcoming QR code graphic can be shown in its native resolution. Otherwise if you app/picturebox/whatever you use stretches the image oder scales it down, it may become blurry and harder to read for QR code scanners.
In Theory you could also call GetGraphic(1), but then the reader's camera must be able to take such high resolution images that it can capture each an every pixel of your QR code graphic. So it isn't a good idea to use GetGraphic(1). Just try around, how small you can go, while keeping the ability to read the QR code. (In practice it may be a good idea too choose the space which is available in your app. Even if your tests show that you could draw the QR code smaller and it is still readable, maybe not all users/customers have that good hardware like you. ;-) )

As such, the image needs to be as accurate as possible as the loss of a single byte on the key sequence will make the whole protocol fail.

This can't happen with QR codes. QR codes have some kind of checksum feature. Either they can be read (totally and correct) or they can't be read. As explained above - when choosing the image size that fits your needs, just check if the QR code keeps readable.

Great explanation. Thanks alot for the help. It's now clear as water. :)
💯 👍

Thanks for your kind words. So I'll close this issue.

@codebude  so if I increase the pixelsPerModule this will be more easy to reader to read this qrocde?
Why in code do you set 20?
how much "blocks" is there in qrcode? is that constant or depend in the length of data I want to encode?
thank you!


Circuit  assign_voltage_frequency_dependent_excitation_to_ports, Only allows for one frequency dependant source

Before submitting the issue

[X] I have searched among the existing issues
[X] I am using a Python virtual environment

Description of the bug
oDesign.UpdateSources in the "NAME:Data" list allows for multiple voltage frequency dependent sources.  The pyaedt implementation only allows for one voltage source in the data list.  When iterating over model pins and assigning unique spectral data for that pin, this causes any previous assigned FDS data to be overwritten by the new source.
Although the function allows for one fds files to be assigned to multiple pins, it is most likely that there would be multiple fds/pin assignments.
It seems that the function needs to read the sources prior to populating the list.  Or a separate script that reads the source as "arg1" data and have it as a optional command argument.
Steps To Reproduce

Examine circuit.py assign_voltage_frequency_dependent_excitation_to_ports starting at line 1437.
Starting at line 1472 the data list is fixed

Which Operating System are you using?
Windows
Which Python version are you using?
IronPython
Installed packages
aiohttp==3.8.1
aiosignal==1.2.0
appdirs==1.4.4
argon2-cffi==21.3.0
argon2-cffi-bindings==21.2.0
astroid==2.11.2
async-timeout==4.0.2
asynctest==0.13.0
atomicwrites==1.4.0
attrs==21.4.0
autopep8==1.6.0
backcall==0.2.0
beautifulsoup4==4.11.1
bleach==5.0.1
cffi==1.15.1
cfgv==3.3.1
charset-normalizer==2.0.12
colorama==0.4.4
cycler==0.11.0
debugpy==1.6.2
decorator==5.1.1
defusedxml==0.7.1
dill==0.3.4
distlib==0.3.4
entrypoints==0.4
fastjsonschema==2.15.3
filelock==3.6.0
flake8==4.0.1
fonttools==4.29.1
frozenlist==1.3.0
identify==2.5.0
idna==3.3
imageio==2.16.0
importlib-metadata==4.11.3
importlib-resources==5.8.0
iniconfig==1.1.1
ipykernel==6.15.1
ipython==7.31.1
ipython-genutils==0.2.0
ipywidgets==7.7.1
isort==5.10.1
jedi==0.18.1
Jinja2==3.1.2
jsonschema==4.7.2
jupyter==1.0.0
jupyter-client==7.3.4
jupyter-console==6.4.4
jupyter-core==4.11.1
jupyterlab-pygments==0.2.2
jupyterlab-widgets==1.1.1
kiwisolver==1.3.2
lazy-object-proxy==1.7.1
MarkupSafe==2.1.1
matplotlib==3.5.1
matplotlib-inline==0.1.3
mccabe==0.6.1
mistune==0.8.4
multidict==6.0.2
nbclient==0.6.6
nbconvert==6.5.0
nbformat==5.4.0
nest-asyncio==1.5.5
nodeenv==1.6.0
notebook==6.4.12
numpy==1.21.5
packaging==21.3
pandocfilters==1.5.0
parso==0.8.3
pickleshare==0.7.5
Pillow==9.0.1
platformdirs==2.5.1
pluggy==1.0.0
pre-commit==2.18.1
prometheus-client==0.14.1
prompt-toolkit==3.0.28
psutil==5.9.1
py==1.11.0
pycodestyle==2.8.0
pycparser==2.21
pydocstyle==6.1.1
pyflakes==2.4.0
Pygments==2.11.2
pylint==2.13.5
pyparsing==3.0.7
pyrsistent==0.18.1
PySide6==6.2.4
pytest==7.1.2
python-dateutil==2.8.2
python-lsp-jsonrpc==1.0.0
python-lsp-server==1.4.1
pyvista==0.33.2
pywin32==304
pywinpty==2.0.6
PyYAML==6.0
pyzmq==23.2.0
qtconsole==5.3.1
QtPy==2.1.0
rope==1.0.0
scooby==0.5.11
Send2Trash==1.8.0
shiboken6==6.2.4
six==1.16.0
snowballstemmer==2.2.0
soupsieve==2.3.2.post1
terminado==0.15.0
tinycss2==1.1.1
toml==0.10.2
tomli==2.0.1
tornado==6.2
traitlets==5.3.0
typed-ast==1.5.2
typing_extensions==4.1.1
ujson==5.2.0
virtualenv==20.14.1
vtk==9.1.0
wcwidth==0.2.5
webencodings==0.5.1
widgetsnbextension==3.6.1
wrapt==1.14.0
wslink==1.4.1
yapf==0.32.0
yarl==1.7.2
zipp==3.8.0

Hi @cmesibov ,
This is a current limitation in PyAEDT. I submitted an issue months ago (#1537 ) But the solution is tricky. For your specific project, do you need only Frequenecy dependent excitations (with different .fds files)? Or any other type of excitation?

Hi Sam,
I have coded my own version of this function that gets the job done.  I am happy to share it with you.  I just don't have the time now to add it to pyaedt.
It uses 3 data structures as inputs and updates all the ports accordingly.
Chris
From: Samuel Lopez @.>
Sent: Friday, October 28, 2022 10:34 AM
To: pyansys/pyaedt @.>
Cc: Chris Mesibov @.>; Mention @.>
Subject: Re: [pyansys/pyaedt] Circuit assign_voltage_frequency_dependent_excitation_to_ports, Only allows for one frequency dependant source (Issue #1848)
[External Sender]
Hi @cmesibovhttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fcmesibov&data=05|01|chris.mesibov%40ansys.com|0846fc479542406e883408dab8f163f2|34c6ce6715b84eff80e952da8be89706|0|0|638025644168298018|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|3000|||&sdata=Sct1FjR1%2F8kluVPwtSGRu9bubgKUI%2BjjuuBnHyRc5dg%3D&reserved=0 ,
This is a current limitation in PyAEDT. I submitted an issue months ago (#1537https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fpyansys%2Fpyaedt%2Fissues%2F1537&data=05|01|chris.mesibov%40ansys.com|0846fc479542406e883408dab8f163f2|34c6ce6715b84eff80e952da8be89706|0|0|638025644168298018|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|3000|||&sdata=cRAZvSbXSm2mYjeePgSAttigg220c%2FdLY7NEyoTZW%2BU%3D&reserved=0 ) But the solution is tricky. For your specific project, do you need only Frequenecy dependent excitations (with different .fds files)? Or any other type of excitation?

Reply to this email directly, view it on GitHubhttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fpyansys%2Fpyaedt%2Fissues%2F1848%23issuecomment-1295076157&data=05|01|chris.mesibov%40ansys.com|0846fc479542406e883408dab8f163f2|34c6ce6715b84eff80e952da8be89706|0|0|638025644168298018|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|3000|||&sdata=d0viLZzTSTkQGzJTbqIBdk9Z2v4Q0otSfGiAcKq9Zw8%3D&reserved=0, or unsubscribehttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAP5EEVAXPPWWRS3L6VCTJ3TWFPP3ZANCNFSM6AAAAAARIORM24&data=05|01|chris.mesibov%40ansys.com|0846fc479542406e883408dab8f163f2|34c6ce6715b84eff80e952da8be89706|0|0|638025644168298018|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|3000|||&sdata=1F%2BFXAlKs83TAXp6YSO%2FsWfF7%2FUxC6DeHeT6DQh7Mt4%3D&reserved=0.
You are receiving this because you were mentioned.Message ID<EMAIL_ADDRESS>


Add support for storing change event timestamp to resume sending events on server shutdown

When OpenMetadata server is shutdown, a webhook publisher might not have completed sending all the events due to webhook endpoint being slow, or a webhook publisher might be in awaitingRetry state. Add recovery functionality to restart sending events to webhooks where the server left off.

Address the following comments from #1900 by @harshach:

do we need to add .shutdown() in shutdown path to flush any messages pending?


Weird #a0 strings in text

There are several occurrences of the string '#a0' within the text/page. I think this is some kind of invalid markdown issue.

Dokumentdetails
⚠ Bearbeiten Sie diesen Abschnitt nicht. Er ist für die Verknüpfung von docs.microsoft.com zum GitHub-Artikel erforderlich.

ID: 2bdb99e4-289a-c09a-c35c-2774fc52a402
Version Independent ID: 4d325389-c9b8-2601-0ff4-c7b6bf5a6c28
Content: Microsoft Defender Advanced Threat Protection
Content Source: windows/security/threat-protection/microsoft-defender-atp/microsoft-defender-advanced-threat-protection.md
Product: w10
Technology: windows
GitHub Login: @mjcaparas
Microsoft Alias: macapara

This issue is within the German translation of the docs. I don't know why the feedback option of the German docs is partially broken and refers to the English version of the docs. Sorry for posting this to the English docs and I hope someone can investigate this. I wasn't aware the Github integration will post this to the wrong repo.

Thank you for posting in the en-us version page to report an issue, especially if you find that the original language page seems to be malfunctioning in the #feedback area.

Indeed, I checked this with current version of Edge browser and the feedback form does work. I did find out that this is an Adblock Plus issue. I reported the bug to them. Thanks for your comment @illfated!

Affected lines in the originating page:

Threat & Vulnerability Management (beneath first image)
Threat & Vulnerability Management (first section below the Tip blob)
Threat & Vulnerability Management (Note blob in the Secure score section)

Common denominator: the Ampersand character (&)
UNICODE     : U+00026
HEX CODE    : &
HTML CODE   : &
HTML ENTITY : &

https://www.toptal.com/designers/htmlarrows/punctuation/
Unfortunately, there is no indicator on the en-us page showing us that the ampersand character & should be converted to #a0, only & for a handful appearances.
It does not appear to be an issue with the Fr-Fr or Es-Es pages, so I am uncertain why only the De-De page is affected.
@TinaMcN : Do you have any suggestions for this issue?

@officedocsbot assign @jvsam

@GiantCrocodile thanks for reporting this and @illfated thanks for the investigation.   This is an issue caused by the Machine Translation tool.   We switched to a neural MT engine a few months ago which generally produces better quality MT but there are a number of issues like this.   I see this particular issue is occurring on numerous German articles.  I reported it to the MT tool team, it may take them weeks or months to get to it as they have a big backlog of issues.

Oh. I see. Thank you for your very informative reply.
I wish this issue could have been easier to resolve,
but as you indicated, it is probably too difficult to solve
from our side here in the Github repository.

Thank you to both of you for your effort and informative responses here.

@GiantCrocodile this article has been human translated and the issue is fixed.  The underlying issue is not yet fixed so it may occur in other articles but I think we can close this issue now.

Thank you for your great support @TinaMcN. I wish you and your team happy/merry holidays!


Move openstack terraform configs into modules

Based on supporting production clusters, we've found there are too many cases where change in the terraform configurations would destroy the cluster before re-provisioning. Existing openstack tf configurations will be marked for deprecation in a future version of mantl.
By moving to modules we can:

reduce the number of destroy/rebuilds in terraform
attach instances to existing networks and routers
have floating ips only for the edge nodes
roll out new groups of servers without affecting existing instances. This allows us to migrate to new versions of mantl without affecting production.
dynamically create new roles for ansible/terraform.py using role=
adds support for optional user_data to instance groups for cloud_init

This PR also clears out openstack vars in the roles, allowing us to use environment variables, addressing #864.

please test this

I love the inline comments! I was able to correctly configure the sample terraform file in just one go. I'm testing now, is there anything in particular you'd like me to check out (e.g. how might one add nodes to an existing network, etc.)?

@siddharthist yes, you could test using an existing network_uuid, or having another stanza for a separate group of worker nodes.

I was able to deploy this branch as I would any other on Openstack after configuring the sample file, and I was able to add another group of workers after rolling out the first deployment. I'm now rebasing this onto master, removing the old sample terraform config, and adding depreciation warnings to the old modules.


bug:

Describe the bug
Scanning an image for Dockerfile misconfiguration with the command
trivy image --image-config-scanners config ...

with a layered Dockerfile containing a COPY command that copies a file from a builder layer as in:
COPY --from=builder somefile /somepath/somefile

fails with the message
CRITICAL: Slash is expected at the end of COPY command argument '/somepath/somefile'
══════════════════════════════════════════════════════════
When a COPY command has more than two arguments, the last one should end with a slash.

See https://avd.aquasec.com/misconfig/ds

It looks like the rego rule  implemented in copy_with_more_than_two_arguments_not_ending_with_slash.rego does not take into account the COPY command flags
To Reproduce
Write a simple Dockerfile that copies a file from a builder image

Hi @vmarchese !
What are your versions of docker and Trivy?

❯ trivy -v
Version: 0.45.1
Vulnerability DB:
  Version: 2
  UpdatedAt: 2023-10-12 12:16:39.10633075 +0000 UTC
  NextUpdate: 2023-10-12 18:16:39.10633025 +0000 UTC
  DownloadedAt: 2023-10-12 12:19:33.086508 +0000 UTC
Policy Bundle:
  Digest: sha256:1df8ade71efc830877ca3b1130f83e0c6368e3a45b0d4c0f0418955501644054
  DownloadedAt: 2023-10-12 12:22:50.276607 +0000 UTC

❯ docker -v
Docker version 24.0.6, build ed223bc820

@vmarchese Thanks! Can you show the output of the docker image history <your_image> command ?

> docker image history myimage
IMAGE          CREATED        CREATED BY                                      SIZE      COMMENT
7f2f4d6c08b2   25 hours ago   /bin/sh -c #(nop)  ENTRYPOINT ["/app/server"]   0B
<missing>      25 hours ago   /bin/sh -c #(nop)  USER appuser                 0B
<missing>      25 hours ago   /bin/sh -c #(nop) COPY file:e62be48b501e30f7…   782B
<missing>      25 hours ago   /bin/sh -c #(nop) COPY file:dbf1ce21196d2f5a…   27.3MB
<missing>      25 hours ago   /bin/sh -c #(nop) COPY file:5bcbbabf74e2e1db…   1.22kB
<missing>      25 hours ago   /bin/sh -c #(nop) COPY file:cee95daa6036230f…   215kB
<missing>      25 hours ago   /bin/sh -c #(nop) COPY dir:78c3d824fe8276c61…   1.07MB
<missing>      25 hours ago   /bin/sh -c #(nop)  EXPOSE 2112 8080             0B
<missing>      25 hours ago   /bin/sh -c #(nop) WORKDIR /app                  0B

btw, this is a "from scratch" image with a go executable inside. The Dockerfile snippet in question is:
FROM scratch
WORKDIR /app
EXPOSE 8080 2112

# Import from builder.
COPY --from=builder /usr/share/zoneinfo /usr/share/zoneinfo
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=builder /etc/passwd /etc/passwd
COPY --from=builder /workspace/bin/server /app/server
COPY banner.txt /app/banner.txt

# Use an unprivileged user.
USER appuser

ENTRYPOINT ["/app/server"]

I think the count of the COPY args could be fixed by filtering out the flags with something like:
count_args := [ x | x := copy.Value[_]; not startswith(x, "--")]

@vmarchese Strangely, I have not been able to reproduce the problem:
FROM alpine as builder

COPY test.txt /workspace/bin/test.txt

FROM scratch
WORKDIR /app

COPY --from=builder /workspace/bin/test.txt /app/test.txt

trivy image --image-config-scanners config test-img
2023-10-12T21:49:07.988+0700    INFO    Container image config scanners: ["config"]
2023-10-12T21:49:07.989+0700    INFO    Vulnerability scanning is enabled
2023-10-12T21:49:07.989+0700    INFO    Misconfiguration scanning is enabled
2023-10-12T21:49:07.989+0700    INFO    Secret scanning is enabled
2023-10-12T21:49:07.989+0700    INFO    If your scanning is slow, please try '--scanners vuln' to disable secret scanning
2023-10-12T21:49:07.989+0700    INFO    Please see also https://aquasecurity.github.io/trivy/v0.45/docs/scanner/secret/#recommendation for faster secret detection
2023-10-12T21:49:08.100+0700    INFO    Number of language-specific files: 0

I see a difference if I run trivy with -d. In the image built with your Dockerfile there is no config file detected.
With my image I have:
2023-10-12T16:56:25.569+0200	INFO	Detected config files: 1

uhm... I suspect it has something to do with our CI/CD environment (docker  in docker)
As a matter of fact if I rebuild the image on my Mac I do not get the misconfiguration errors.
Inspecting the images I get:
On my Mac:
┃ ● Layers ┣━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Cmp   Size  Command
       0 B  FROM 6dc9925546b0573
    1.1 MB  COPY /usr/share/zoneinfo /usr/share/zoneinfo # buildkit
    214 kB  COPY /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ # buildkit
    1.2 kB  COPY /etc/passwd /etc/passwd # buildkit
     26 MB  COPY /workspace/bin/server /app/server # buildkit
     782 B  COPY banner.txt /app/banner.txt # buildkit

The image built in docker (the one that has the errors)
┃ ● Layers ┣━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Cmp   Size  Command
       0 B  FROM 3fd69bc430c06ea
    1.1 MB  #(nop) COPY dir:78c3d824fe8276c613fe2d144127ffca86629a3c767df24e80ea2313542b0671 in /usr/share/zoneinfo
    214 kB  #(nop) COPY file:cee95daa6036230fcbf45d10edbd5f2df75ec449e633dfcfa3951819c01dec8b in /etc/ssl/certs/
    1.2 kB  #(nop) COPY file:5bcbbabf74e2e1db7d06af9ed221579031de263472cefe2d4064ed5eb7bb19d1 in /etc/passwd
     27 MB  #(nop) COPY file:dbf1ce21196d2f5a7fd9eeac1426fd3201de8a593e7563d9ec253085fd208731 in /app/server
     782 B  #(nop) COPY file:e62be48b501e30f7b698566a4b7ae4bbbe422636e3dff56a7dd0b9e45008cf90 in /app/banner.txt

Nevertheless, I guess the rego rule should be fixed

❯ docker -v
Docker version 24.0.6, build ed223bc820

Our CI/CD system has docker 20.10.17 and I see in the layers that the COPY command  is
COPY file:dbf1ce21196d2f5a7fd9eeac1426fd3201de8a593e7563d9ec253085fd208731 in /app/server

maybe is that ?

As a matter of fact this is the complete message:
CRITICAL: Slash is expected at the end of COPY command argument '/app/banner.txt'
════════════════════════════════════════════════════════════════════════════
When a COPY command has more than two arguments, the last one should end with a slash.

See https://avd.aquasec.com/misconfig/ds011
──────────────────────────────────────────────────────
 postesviluppo.azurecr.io/anpr-step-api:1.1.0-SNAPSHOT:7
──────────────────────────────────────────────────────
   7 [ COPY file:e62be48b501e30f7b698566a4b7ae4bbbe422636e3dff56a7dd0b9e45008cf90 in /app/banner.txt
──────────────────────────────────────────────────────
```

@vmarchese I reproduced this with 20.10.17 docker.


Seekable test is flaky

Error:  Errors:
Error:  io.javalin.TestResponse.seekable - range works and input stream closed
Error:    Run 1: TestResponse.seekable - range works and input stream closed:186 � Runtime java.lang.Exception: Assertion error:
Expecting value to be true but was false
Error:    Run 2: TestResponse.seekable - range works and input stream closed:186 � Runtime java.lang.Exception: Assertion error:
Expecting value to be true but was false
Error:    Run 3: TestResponse.seekable - range works and input stream closed:186 � Runtime java.lang.Exception: Assertion error:
Expecting value to be true but was false

Ever since https://github.com/javalin/javalin/pull/1756, this test has been very flaky. Could you have a look @viche12345 ?

Yeah, I'll take a look. Is it only failing on Windows?

Let me check some runs.

Yes, all windows:
Test all JDKs on all OSes / build (17, windows-latest) (pull_request) Failing after 2m
Test all JDKs on all OSes / build (19, windows-latest) (pull_request) Failing after 2m
Test all JDKs on all OSes / build (17, windows-latest) (pull_request) Failing after 3m
Test all JDKs on all OSes / build (17, windows-latest) (push) Failing after 3m


add runQuery

WIP

What does this do?

Closing in favour of #123 but we should take some of the work from this.