Human WGS data sequenced by BGI and installation of HELLO

I have human WGS data sequenced by BGI. Is my data appropriate for HELLO? If yes, which pipeline (Illumina, PacBio) should I use?

For shorter reads, Illumina models may be used, and for longer reads, the PacBio models may be used. However, the models are trained on the Illumina and PacBio error profiles, and not optimized for other sequencing platforms. I would say if there is a model/algorithm that is trained on BGI reads from another tool, that will likely provide better performance than HELLO.

Thanks for yourquick response. I want to run the Illumina variant calling option on our high performance cluster (HPC) in a singularity container for safety reasons.
However, currently, I am trying to understand how to get the docker container running.
Steps already taken:

I pulled the docker container: docker pull oddjobs/hello_image.x86_64
Looked inside the container: docker run -it oddjobs/hello_image.x86_64 bash
I found two sh files and one directory:
Miniconda3-latest-Linux-x86_64.sh
bazel-0.26.1-installer-linux-x86_64.sh
miniconda3 (directory)
I was also looking for the python/call.py inside the docker container, but without success.

Can you tell me how to use the docker container?
When do I need to build the tool? I git cloned the repository and tried to build the tool (cmake . && make -j 12), but without success.
I am also a little confused by the note "To properly download models, git-lfs needs to be installed. Once installed, please do git lfs pull inside the git repo". Is git-lfs already installed in the docker container?

The docker image only contains the dependencies, not hello itself. Take a look at the following steps to get it built:
# Launch container
docker run -it oddjobs/hello_image.x86_64 /bin/bash

# Inside container
apt-get -y update && apt-get -y install git && git clone https://github.com/anands-repo/hello.git
mkdir git-lfs
cd git-lfs/
wget https://github.com/git-lfs/git-lfs/releases/download/v3.1.2/git-lfs-linux-amd64-v3.1.2.tar.gz  # Or another version of git lfs
tar xvfz git-lfs-linux-amd64-v3.1.2.tar.gz
./install.sh
cd
git lfs install
cd hello/
git lfs pull
cmake . && make -j 12

Thanks for your quick response again. I am getting closer, but I am not there yet.
I executed the steps you recommended. Thereafter, I executed a docker commit to make the changes permanent and afterwards, I tried to convert the docker image into a singularity container using singularity 3.9.2.
Steps:

docker commit c3f279d17e0a hello
singularity build hello.sif docker-daemon://hello:latest

I noticed that in the docker container hello is stored under /root/hello. In the singularity container I do not have access to root. Do you have an idea what I could do because I can only run HELLO in a singularity container on our HPC due to security reasons.

This is a bit of a sticky problem. Does singularity only play well with certain paths like /opt? Currently some dependencies are in /usr etc if I am not mistaken.
I don't know whether the following will work, but I can think of three steps to try:

Move miniconda3 to /opt. After the move, the PATH environment may need to be changed to add the new path of miniconda3/bin. Refer: https://www.anaconda.com/blog/moving-conda-environments
If BOOST is not installed in /opt, install (build from source) BOOST with boost::python support in /opt. Note that boost with boost::python support needs a few changes to the standard boost compilation flow. Change CMakeLists.txt file in hello (see commented lines at the start of the file) accordingly. I do not remember off the top of my head, but I believe I used boost 1.68. This is the step that can lead to some difficulties - it may need some trial and error to get everything working together (boost installation, CMakeLists.txt etc).
Install GNU parallel in /opt, and update PATH appropriately

Thanks for the reply. I also got some IT support at my end now. Hopefully, we will get the tool running based on your suggestions.
I also have some other questions which I would like to have answered while we are working on getting the tool running. How to use the num_threads? Is there a min or max limit? I guess the workdir points to the output file directory. Is that correct?

num_threads is the number of CPU cores to be used in each run. Correct - workdir points to the output directory.

The tools is running, but I currently get the following error message (slurm output):
ETA: 0s Left: 1 AVG: 0.63s  local:1/10985/100%/0.6s
ETA: 0s Left: 0 AVG: 0.63s  local:0/10986/100%/0.6s
INFO:root:Completed runs, checking log files
ERROR:root:File /projects/xm41/robert/hello/GALF003/features_bams___GALF003__bam__/features0.log doesn't have termination string
In the workdir, the following files where created:
features_bams___GALF003__bam__       hotspots_bams___GALF003__bam_chr16_  hotspots_bams___GALF003__bam_chr3_
hotspots_bams___GALF003__bam_chr1_   hotspots_bams___GALF003__bam_chr17_  hotspots_bams___GALF003__bam_chr4_
hotspots_bams___GALF003__bam_chr10_  hotspots_bams___GALF003__bam_chr18_  hotspots_bams___GALF003__bam_chr5_
hotspots_bams___GALF003__bam_chr11_  hotspots_bams___GALF003__bam_chr19_  hotspots_bams___GALF003__bam_chr6_
hotspots_bams___GALF003__bam_chr12_  hotspots_bams___GALF003__bam_chr2_   hotspots_bams___GALF003__bam_chr7_
hotspots_bams___GALF003__bam_chr13_  hotspots_bams___GALF003__bam_chr20_  hotspots_bams___GALF003__bam_chr8_
hotspots_bams___GALF003__bam_chr14_  hotspots_bams___GALF003__bam_chr21_  hotspots_bams___GALF003__bam_chr9_
hotspots_bams___GALF003__bam_chr15_  hotspots_bams___GALF003__bam_chr22_
Do you have an idea what might be wrong and how I can fix that issue? Do you have an example somewhere I could try out? I would just like to make sure that the tools behaves in a way the tool is supposed to behave.

If you wanted to try an Illumina dataset to do sanity checking, you can do
one from Genome In a Bottle. I haven’t directly used them, but I believe
one of those should work directly. However, we are also checking whether
the installation process is correct. If there is an issue with
installation, neither case will provide an output on  the platform you are
employing.
It may be easier to send to me the run command and the contents of
*/projects/xm41/robert/hello/GALF003/features_bams___GALF003__bam__/features0.log
*
My suspicion is that boost::Python isn’t the right version or hasn’t been
installed correctly. Standard boost installation or build from source
doesn’t come with boost::Python enabled. The build files need to be
modified a little for that and it wasn’t documented well when I tried to
install it.
On Thu, Apr 14, 2022 at 9:37 PM robertzeibich @.***>
wrote:

The tools is running, but I currently get the following error message
(slurm output):
ETA: 0s Left: 1 AVG: 0.63s local:1/10985/100%/0.6s
ETA: 0s Left: 0 AVG: 0.63s local:0/10986/100%/0.6s
INFO:root:Completed runs, checking log files
ERROR:root:File
/projects/xm41/robert/hello/GALF003/features_bams___GALF003__bam__/features0.log
doesn't have termination string
In the workdir, the following files where created:
features_bams___GALF003__bam__ hotspots_bams___GALF003__bam_chr16_
hotspots_bams___GALF003__bam_chr3_
hotspots_bams___GALF003__bam_chr1_ hotspots_bams___GALF003__bam_chr17_
hotspots_bams___GALF003__bam_chr4_
hotspots_bams___GALF003__bam_chr10_ hotspots_bams___GALF003__bam_chr18_
hotspots_bams___GALF003__bam_chr5_
hotspots_bams___GALF003__bam_chr11_ hotspots_bams___GALF003__bam_chr19_
hotspots_bams___GALF003__bam_chr6_
hotspots_bams___GALF003__bam_chr12_ hotspots_bams___GALF003__bam_chr2_
hotspots_bams___GALF003__bam_chr7_
hotspots_bams___GALF003__bam_chr13_ hotspots_bams___GALF003__bam_chr20_
hotspots_bams___GALF003__bam_chr8_
hotspots_bams___GALF003__bam_chr14_ hotspots_bams___GALF003__bam_chr21_
hotspots_bams___GALF003__bam_chr9_
hotspots_bams___GALF003__bam_chr15_ hotspots_bams___GALF003__bam_chr22_
Do you have an idea what might be wrong and how I can fix that issue? Do
you have an example somewhere I could try out? I would just like to make
sure that the tools behaves in a way the tool is supposed to behave.
—
Reply to this email directly, view it on GitHub
https://github.com/anands-repo/hello/issues/5#issuecomment-1099843567,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADJJMQEHD3PMV32J5SUIC7TVFDW7HANCNFSM5SIFJ2LQ
.
You are receiving this because you commented.Message ID:
@.***>

Which version of boost::Python do you recommend installing?
I guess my problem has something to do with the following directory: /tmp/vcftemp
If I run the script without /tmp:vcfemp bound to the container, I will receive the following error:
Script:
REF="/scratch/xm41/hg38_resources/resources_broad_hg38_v0_Homo_sapiens_assembly38.fasta"
NUM_THREADS=8
workdir="/projects/xm41/robert/hello"
hello="/usr/local/hello/709b7c3/"
mkdir -p /tmp/vcftemp
module load singularity/3.9.2
#-B /tmp/vcftemp:/tmp/vcftemp
singularity run -B /fs02 -B /fs04 -B /fs03:/fs03 -B /projects:/projects -B /scratch:/scratch ${hello}/hello.sif
python /opt/hello/python/call.py
--ibam GALF003.p0.03.bam
--ref $REF
--network models/illumina_multi_coverage_mapq_threshold_hg002_continue_run16.wrapper.dnn
--num_threads $NUM_THREADS
--workdir $workdir/GALF003_1x_tmp --mapq_threshold 5
Error:
INFO:root:Combining all hotspots and sharding
2022-04-16 10:57:52,214 Directory /projects/xm41/robert/hello/GALF003_1x_tmp/hotspots_hello___GALF003__p0__03__bam_chr22_ exists
2022-04-16 10:57:52,214 Counting hotspots
2022-04-16 10:57:52,214 Sharding with >= 0 items per cluster
INFO:root:Completed sharding, creating caller commands for dumping training data
INFO:root:Created call command for chromosome chr22
INFO:root:Launching all caller commands
Computers / CPU cores / Max jobs to run
1:local / 8 / 1
ETA: 0s Left: 0 AVG: 0.00s  0
INFO:root:Completed runs, checking log files
INFO:root:All commands completed correctly, preparing vcf
sort -k1,1d -k2,2n
sort -k1,1d -k2,2n
Search string /tmp/vcftemp/*expert0.vcf
Found 0 files
Running command ['vcf-sort', '/projects/xm41/robert/hello/GALF003_1x_tmp/results.ngs.tmp.vcf']
Search string /tmp/vcftemp/*expert1.vcf
Found 0 files
Running command ['vcf-sort', '/projects/xm41/robert/hello/GALF003_1x_tmp/results.tgs.tmp.vcf']
Search string /tmp/vcftemp/*expert2.vcf
Found 0 files
Running command ['vcf-sort', '/projects/xm41/robert/hello/GALF003_1x_tmp/results.ngs_tgs.tmp.vcf']
Search string /tmp/vcftemp/*best.vcf
Found 0 files
Running command ['vcf-sort', '/projects/xm41/robert/hello/GALF003_1x_tmp/results.best.tmp.vcf']
Search string /tmp/vcftemp/*mean.vcf
Found 0 files
Running command ['vcf-sort', '/projects/xm41/robert/hello/GALF003_1x_tmp/results.mean.tmp.vcf']
Choice histogram = {0: 0, 1: 0, 2: 0}
If I run the script with /tmp:vcfemp bound to the container, I will receive the following error:
Script:
REF="/scratch/xm41/hg38_resources/resources_broad_hg38_v0_Homo_sapiens_assembly38.fasta"
NUM_THREADS=8
workdir="/projects/xm41/robert/hello"
hello="/usr/local/hello/709b7c3/"
mkdir -p /tmp/vcftemp
module load singularity/3.9.2
singularity run -B /fs02 -B /fs04 -B /fs03:/fs03 -B /projects:/projects -B /scratch:/scratch -B /tmp/vcftemp:/tmp/vcftemp ${hello}/hello.sif
python /opt/hello/python/call.py
--ibam GALF003.p0.03.bam
--ref $REF
--network models/illumina_multi_coverage_mapq_threshold_hg002_continue_run16.wrapper.dnn
--num_threads $NUM_THREADS
--workdir $workdir/GALF003_1x_tmp --mapq_threshold 5
Error:
ETA: 0s Left: 4 AVG: 0.03s  local:4/497/100%/0.0s
ETA: 0s Left: 1 AVG: 0.03s  local:1/500/100%/0.0s
ETA: 0s Left: 0 AVG: 0.03s  local:0/501/100%/0.0s
INFO:root:Combining all hotspots and sharding
2022-04-16 10:45:26,049 Directory /projects/xm41/robert/hello/GALF003_1x_tmp/hotspots_hello___GALF003__p0__03__bam_chr22_ exists
2022-04-16 10:45:26,050 Counting hotspots
2022-04-16 10:45:26,050 Sharding with >= 0 items per cluster
INFO:root:Completed sharding, creating caller commands for dumping training data
INFO:root:Created call command for chromosome chr22
INFO:root:Launching all caller commands
Computers / CPU cores / Max jobs to run
1:local / 8 / 1
ETA: 0s Left: 0 AVG: 0.00s  0
INFO:root:Completed runs, checking log files
INFO:root:All commands completed correctly, preparing vcf
Traceback (most recent call last):
File "/opt/hello/python/prepareVcf.py", line 255, in
shutil.rmtree(args.tmpdir);
File "/opt/miniconda3/lib/python3.7/shutil.py", line 498, in rmtree
onerror(os.rmdir, path, sys.exc_info())
File "/opt/miniconda3/lib/python3.7/shutil.py", line 496, in rmtree
os.rmdir(path)
OSError: [Errno 16] Device or resource busy: '/tmp/vcftemp'
Unfortunately, I cannot create the problem I mentioned in my previous message.

Sorry, the previous mail was sent premature.
For all inputs not in home directory, can we check whether the inputs
exist? For example, run “du -sh filename” inside the singularity container.
Can we direct all outputs to your home directory? I believe singularity
auto mounts your home directory at the same path inside a container, so
there is no need to mount anything for this purpose - paths in your home
directory should work directly inside the singularity container. (I could
be wrong on this based on the cluster setup). Would like the singularity
Mount issues to be resolved, otherwise we could be debugging the wrong
issue. I also see use of both relative and absolute paths. Let’s make all
paths absolute.
On Fri, Apr 15, 2022 at 10:33 PM Anand Ramachandran @.***>
wrote:

Right. I think the similarity commands and the Mount structure needs to be
simplified.
On Fri, Apr 15, 2022 at 6:04 PM robertzeibich @.***>
wrote:

Which version of boost::Python do you recommend installing?
I guess my problem has something to do with the following directory:
/tmp/vcftemp
If I run the script without /tmp:vcfemp bound to the container, I will
receive the following error:
Script:
REF="/scratch/xm41/hg38_resources/resources_broad_hg38_v0_Homo_sapiens_assembly38.fasta"
NUM_THREADS=8
workdir="/projects/xm41/robert/hello"
hello="/usr/local/hello/709b7c3/"
mkdir -p /tmp/vcftemp
module load singularity/3.9.2
#-B /tmp/vcftemp:/tmp/vcftemp
singularity run -B /fs02 -B /fs04 -B /fs03:/fs03 -B /projects:/projects
-B /scratch:/scratch ${hello}/hello.sif
python /opt/hello/python/call.py
--ibam GALF003.p0.03.bam
--ref $REF
--network
models/illumina_multi_coverage_mapq_threshold_hg002_continue_run16.wrapper.dnn
--num_threads $NUM_THREADS
--workdir $workdir/GALF003_1x_tmp --mapq_threshold 5
Error:
INFO:root:Combining all hotspots and sharding
2022-04-16 10:57:52,214 Directory
/projects/xm41/robert/hello/GALF003_1x_tmp/hotspots_hello___GALF003__p0__03__bam_chr22_
exists
2022-04-16 10:57:52,214 Counting hotspots
2022-04-16 10:57:52,214 Sharding with >= 0 items per cluster
INFO:root:Completed sharding, creating caller commands for dumping
training data
INFO:root:Created call command for chromosome chr22
INFO:root:Launching all caller commands
Computers / CPU cores / Max jobs to run
1:local / 8 / 1
ETA: 0s Left: 0 AVG: 0.00s 0
INFO:root:Completed runs, checking log files
INFO:root:All commands completed correctly, preparing vcf
sort -k1,1d -k2,2n
sort -k1,1d -k2,2n
Search string /tmp/vcftemp/*expert0.vcf
Found 0 files
Running command ['vcf-sort',
'/projects/xm41/robert/hello/GALF003_1x_tmp/results.ngs.tmp.vcf']
Search string /tmp/vcftemp/*expert1.vcf
Found 0 files
Running command ['vcf-sort',
'/projects/xm41/robert/hello/GALF003_1x_tmp/results.tgs.tmp.vcf']
Search string /tmp/vcftemp/*expert2.vcf
Found 0 files
Running command ['vcf-sort',
'/projects/xm41/robert/hello/GALF003_1x_tmp/results.ngs_tgs.tmp.vcf']
Search string /tmp/vcftemp/*best.vcf
Found 0 files
Running command ['vcf-sort',
'/projects/xm41/robert/hello/GALF003_1x_tmp/results.best.tmp.vcf']
Search string /tmp/vcftemp/*mean.vcf
Found 0 files
Running command ['vcf-sort',
'/projects/xm41/robert/hello/GALF003_1x_tmp/results.mean.tmp.vcf']
Choice histogram = {0: 0, 1: 0, 2: 0}
If I run the script with /tmp:vcfemp bound to the container, I will
receive the following error:
Script:
REF="/scratch/xm41/hg38_resources/resources_broad_hg38_v0_Homo_sapiens_assembly38.fasta"
NUM_THREADS=8
workdir="/projects/xm41/robert/hello"
hello="/usr/local/hello/709b7c3/"
mkdir -p /tmp/vcftemp
module load singularity/3.9.2
singularity run -B /fs02 -B /fs04 -B /fs03:/fs03 -B /projects:/projects
-B /scratch:/scratch -B /tmp/vcftemp:/tmp/vcftemp ${hello}/hello.sif
python /opt/hello/python/call.py
--ibam GALF003.p0.03.bam
--ref $REF
--network
models/illumina_multi_coverage_mapq_threshold_hg002_continue_run16.wrapper.dnn
--num_threads $NUM_THREADS
--workdir $workdir/GALF003_1x_tmp --mapq_threshold 5
Error:
ETA: 0s Left: 4 AVG: 0.03s local:4/497/100%/0.0s
ETA: 0s Left: 1 AVG: 0.03s local:1/500/100%/0.0s
ETA: 0s Left: 0 AVG: 0.03s local:0/501/100%/0.0s
INFO:root:Combining all hotspots and sharding
2022-04-16 10:45:26,049 Directory
/projects/xm41/robert/hello/GALF003_1x_tmp/hotspots_hello___GALF003__p0__03__bam_chr22_
exists
2022-04-16 10:45:26,050 Counting hotspots
2022-04-16 10:45:26,050 Sharding with >= 0 items per cluster
INFO:root:Completed sharding, creating caller commands for dumping
training data
INFO:root:Created call command for chromosome chr22
INFO:root:Launching all caller commands
Computers / CPU cores / Max jobs to run
1:local / 8 / 1
ETA: 0s Left: 0 AVG: 0.00s 0
INFO:root:Completed runs, checking log files
INFO:root:All commands completed correctly, preparing vcf
Traceback (most recent call last):
File "/opt/hello/python/prepareVcf.py", line 255, in
shutil.rmtree(args.tmpdir);
File "/opt/miniconda3/lib/python3.7/shutil.py", line 498, in rmtree
onerror(os.rmdir, path, sys.exc_info())
File "/opt/miniconda3/lib/python3.7/shutil.py", line 496, in rmtree
os.rmdir(path)
OSError: [Errno 16] Device or resource busy: '/tmp/vcftemp'
Unfortunately, I cannot create the problem I mentioned in my previous
message.
—
Reply to this email directly, view it on GitHub
https://github.com/anands-repo/hello/issues/5#issuecomment-1100493038,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADJJMQHHVJ2I2UWMQOWJ5VLVFIG3BANCNFSM5SIFJ2LQ
.
You are receiving this because you commented.Message ID:
@.***>

The inputs exist inside the singularity container:
3.1G    /scratch/xm41/hg38_resources/resources_broad_hg38_v0_Homo_sapiens_assembly38.fasta
16K     /opt/hello/python/call.py
4.5G    /scratch/xm41/ct/bamsDown/step1_bams1x/GALF003_1x.bam
access to /projects is also given
I cannot use my home directory because I only have 10GB available. When I run call.py with absolute paths + workdir /projects/xm41/robert/hello/GALF003_1x_tmp4/ inside the singularity container, the following files are created:

features_hello___GALF003__p0__03__bam__
hotspots_hello___GALF003__p0__03__bam_chr1_ .. chr22_
results.choices.bed
results.mean.vcf
results.ngs.vcf
results.ngs_tgs.vcf
results.tgs.vcf
results.best.vcf

The bed and vcf files do not have an entry, just a header with no variants listed. The console output is the same as mentioned in my previous post under 1) WITHOUT /tmp:vcfemp Error.
 A whole genome BAM with 4.5GB size and a suffix _1x seem to indicate as much. If it’s indeed 1x coverage, HELLO won’t work and probably would give empty outputs as you are getting (though I never tried using HELLO at 1x coverage, even if it gave a non-zero output, I am not sure what that would mean).
If not, let’s continue - unfortunately, I am not able to do a Zoom for several weeks as I am away on a family matter. What I can do is review some logs etc on my phone for these several weeks. My schedule is no-existent and I pick up any odd 5 minutes available at some point in time during the day.
There are .sh and .log files inside the features and hotspot files. They are numbered, say <some_prefix>0.sh, <some_other_prefix>0.log etc. could you pick one instance of each (say numbered 10) from one of the hotspots* directory and the features* directory and upload it to Dropbox/box/Google drive and share it with me? Ideally, I would want the full working directory for a small chromosome, say, 22, but if that’s too big, one instance of each file would be sufficient.
if I get a chance to get anywhere near a computer in the next few days, I will share with you an example of what to look for more specifically…

I just read your message and I am looking for a bam file currently because I am not allowed to share data from the server. I am following your idea with genome in a bottle:
https://github.com/genome-in-a-bottle/giab_data_indexes/blob/master/ChineseTrio/alignment.index.ChineseTrio_Illumina300X100X_wgs_novoalign_GRCh37_GRCh38_NHGRI_04062016.HG007
I thought of downloading the first one:
ftp://ftp-trace.ncbi.nlm.nih.gov/ReferenceSamples/giab/data/ChineseTrio/HG007_NA24695-hu38168_mother/NA24695_Mother_HiSeq100x/NHGRI_Illumina100X_Chinesetrio_novoalign_bams/HG007.GRCh38_full_plus_hs38d1_analysis_set_minus_alts.100x.bam
Do you think this could work? If you have another file in mind, please let me know.

Take a look at this:
ftp://ftp-trace.ncbi.nlm.nih.gov/ReferenceSamples/giab/data/AshkenazimTrio/HG002_NA24385_son/NIST_HiSeq_HG002_Homogeneity-10953946/NHGRI_Illumina300X_AJtrio_novoalign_bams/
They have 60x BAM files, which must be quite a lot smaller.

HELLO works on a coverage of 30x WGS BGI data and the same data downsampled to a coverage of 1x. The error was not using the correct network path. Take away: Check all paths. Thx for your help again.
Do you know if I need to run a kind of quality control on the VCF files like VQSR or has that already been covered by the tool itself?

Great to hear that!
There is no additional filtration to be run on the VCF. But there is a
small error in the VCF header. Some VCF processing commands work without
fixing it, but some may not. In any case, it’s very simple to fix the
issue and how to fix it is given at the end of the README file, if you
scroll down.
Thanks again, for debugging it. It’s nice to hear it worked out in your use
case.
On Wed, Apr 20, 2022 at 4:28 AM robertzeibich @.***>
wrote:

HELLO works on a coverage of 30x WGS BGI data and the same data
downsampled to a coverage of 1x. The error was not using the correct
network path. Take away: Check all paths. Thx for your help again.
Do you know if I need to run a kind of quality control on the VCF files
like VQSR or has that already been covered by the tool itself?
—
Reply to this email directly, view it on GitHub
https://github.com/anands-repo/hello/issues/5#issuecomment-1103823605,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADJJMQD7ROLR24VC5SL6LLDVF7S4LANCNFSM5SIFJ2LQ
.
 To change this behavior, one would need to pass the —chromosomes option.
Will close the issue in a couple of days, in case you have other questions.

Thx for pointing the VCF fix out and for telling me about the chromosomes option. How do I need to use the chromosomes option?
---chromosomes 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,X,Y?

That works perfectly. I was able to add chrX and chrY. I also have to admit that the tool runs much faster than DeepVariant CPU based and GATK.
What are the differences between those vcf files?
results.best.vcf
results.mean.vcf
results.ngs_tgs.vcf
results.ngs.vcf
results.tgs.vcf
How can I create a VCF file with all variants containing?

This is the correct output VCF: results.mean.vcf
Please reference the last 3 paragraphs or so of the README to see how to use this file.

I believe in your correct_string Number and String must be exchanged. However, I currently have these corrected VCF files for multiple samples. When using GATK for variant calling, I was able to combine the single gVCF files into one gVCF using GenomicsDBImport --> GenotypeGVCFs --> ... I noticed that your tool does not produce gVCF files. Do you have an idea how I could combine the VCF files produced by your tool into one VCF file or generate out of the VCF files produced by your tool gVCF files?
Info: When I use GenomicsDBImport, I receive the following error message: A USER ERROR has occurred: Bad input: GenomicsDBImport does not support GVCFs with MNPs. MNP found at chr2:289039 in VCF. I have also done some digging online, but before starting to figure out a workaround, I thought of asking you for a solution because you might have one.

I guess the problem you are describing is that - you have multiple VCF files from multiple runs. You would like to combine them into one VCF file. The issue with MNP is probably that HELLO lists more than 2 alleles but selects only two of them using the SAMPLE1 column. In such a case, the unlisted allele may be removed from the file.
For example, if REF = A, ALT=C,G,T and SAMPLE1=0/1, then we can rewrite this as REF=A,ALT=C,SAMPLE1=0/1. I do not have a script which does this automatically, but it must not be very hard to write such a script at least in theory.
By the way, I finished a new version of HELLO which may interest you. It incorporates many changes based mainly on issues you faced, so thanks for your feedback. All the algorithms and models remain the same. But the following convenience features are implemented

New docker image that is singularity friendly (directly convert to singularity without having to hack the image)
Better logging to convey exactly what is going on (no GNU parallel, progress bars, uch more concise and readable messages)
Better error handling - avoiding phantom fails which do not reveal what exactly is the problem

I will have to update the repo tomorrow as I am waiting docker image upload overnight (I am in Pacific Time zone, US).

I just went through the files to put an example together:
SAMPLE1.gvcf
#CHROM  POS     ID      REF     ALT     QUAL    FILTER  INFO    FORMAT   SAMPLE1
chr10   57686   .       T       <NON_REF>       .       .       END=57686       GT:DP:GQ:MIN_DP:PL      0/0:43:87:43:0,87,1590
...
SAMPLE2.gvcf
#CHROM  POS     ID      REF     ALT     QUAL    FILTER  INFO    FORMAT   SAMPLE2
chr10   57686   .       T       G,<NON_REF>     40.64   .       BaseQRankSum=-1.958;DP=44;ExcessHet=0.0000;MLEAC=1,0;MLEAF=0.500,0.00;MQRankSum=-3.333;RAW_MQandDP=70798,44;ReadPosRankSum=-1.385    GT:AD:DP:GQ:PGT:PID:PL:PS:SB    0|1:40,4,0:44:48:0|1:57659_A_T:48,0,1668,168,1680,1848:57659:17,23,3,1
...
SAMPLE3.gvcf
#CHROM  POS     ID      REF     ALT     QUAL    FILTER  INFO    FORMAT   SAMPLE3
chr10  57686   .       T       A,<NON_REF>     81.64   .       BaseQRankSum=0.326;DP=60;ExcessHet=0.0000;MLEAC=1,0;MLEAF=0.500,0.00;MQRankSum=-3.425;RAW_MQandDP=95382,60;ReadPosRankSum=-1.194     GT:AD:DP:GQ:PGT:PID:PL:PS:SB    0|1:54,6,0:60:89:0|1:57659_A_T:89,0,2249,252,2267,2519:57659:18,36,1,5
...
COMBINED.gvcf
#CHROM  POS     ID      REF     ALT     QUAL    FILTER  INFO    FORMAT  SAMPLE1 SAMPLE2 SAMPLE3
chr10   57686   .       T       G,A     103.15  PASS    AC=1,1;AF=9.615e-03,9.615e-03;AN=104;BaseQRankSum=0.326;DP=2312;ExcessHet=0.0424;FS=1.120;InbreedingCoeff=-0.0371;MLEAC=1,1;MLEAF=9.615e-03,9.615e-03;MQ=39.68;MQRankSum=-3.333e+00;QD=0.99;ReadPosRankSum=-1.385e+00;SOR=0.425;VQSLOD=-4.762e+00;culprit=QD GT:AD:DP:GQ:PGT:PID:PL:PS    0/0:43,0,0:43:87:.:.:0,87,1590,87,1590,1590     0|1:40,4,0:44:48:0|1:57659_A_T:48,0,1668,168,1680,1848:57659    0|2:54,0,6:60:89:0|1:57659_A_T:89,252,2519,0,2267,2249:57659
When GATK is used, other information is provided too, but I am not entirely sure what the minimum requirements would be to use GATK for combining different gVCF files. I could easily write a script to solve this issue. I could not agree more with you on that, but I can imagine that you would greatly benefit from integrating this function in your tool yourself because then your tool would be compatible with GATK, which is still the gold standard solution for variant calling. Just an idea. Tomorrow, I will see once more if there is a tool out there which might solve my current VCF file combining problem.
Glad to read that you incorporate my feedback. I just used the Illumina variant calling function for BGI WGS data. I know the tool was not trained for this task, but that seems to be the best option for me for now.

Thanks for the feedback regarding usage with GATK. Will look into that for a future release, and hope you are able to find a simple solution to the VCF merging problem.

There is a simple solution: vcf-merge (http://vcftools.sourceforge.net/perl_module.html#vcf-merge).
There is also another thing I noticed when using your tool because when I use other variant callers, they use the sample name given in the BAM file, e.g. XXX003. Your tool just uses SAMPLE1 in the header. Can be easily fixed with a sed command at my end, but I just wanted to let you know.
Thanks for the FYI.

I currently stumbled upon something else. When looking at the VCF files produced by other variant callers, I see that they contain much more contig entries.
##contig=<ID=chr22,length=50818468>
##contig=<ID=chrX,length=156040895>
##contig=<ID=chrY,length=57227415>
##contig=<ID=chrM,length=16569>
##contig=<ID=chr1_KI270706v1_random,length=175055>
##contig=<ID=chr1_KI270707v1_random,length=32032>
##contig=<ID=chr1_KI270708v1_random,length=127682>
##contig=<ID=chr1_KI270709v1_random,length=66860>
##contig=<ID=chr1_KI270710v1_random,length=40176>
##contig=<ID=chr1_KI270711v1_random,length=42210>
##contig=<ID=chr1_KI270712v1_random,length=176043>
##contig=<ID=chr1_KI270713v1_random,length=40745>
##contig=<ID=chr1_KI270714v1_random,length=41717>
##contig=<ID=chr2_KI270715v1_random,length=161471>
##contig=<ID=chr2_KI270716v1_random,length=153799>
##contig=<ID=chr3_GL000221v1_random,length=155397>
...```

Do you know if these other chromosomes/contig entries (e.g., chr3_GL000221v1_random) can be created with your tool as well?

You can ignore my last post for now. I will update what I have written once I have a better understanding about another tool I want to use.

I just used chr1 ... chrY and replaced the contigs in the VCF file. Thereafter, I still had to filter the VCF file because in the REF column, I found strange entries:
AAAGCACAAAN, AN, B, CN, GN, K, M, N, R, RGCGG, S, TGRGCG, TN, W, WG, Y, YS
To be honest, at the beginning, I thought your tool is not working correctly, but when I checked the positions, I found that these alleles/nucleotide and sequences can be found in the FATA file. I am not sure if you want to filter these out in the future. I would.

I see. Thanks for letting me know. Will keep the suggestions in mind for future updates.

Right. I think the similarity commands and the Mount structure needs to be
simplified.
On Fri, Apr 15, 2022 at 6:04 PM robertzeibich @.***>
wrote:

Which version of boost::Python do you recommend installing?
I guess my problem has something to do with the following directory:
/tmp/vcftemp
If I run the script without /tmp:vcfemp bound to the container, I will
receive the following error:
Script:
REF="/scratch/xm41/hg38_resources/resources_broad_hg38_v0_Homo_sapiens_assembly38.fasta"
NUM_THREADS=8
workdir="/projects/xm41/robert/hello"
hello="/usr/local/hello/709b7c3/"
mkdir -p /tmp/vcftemp
module load singularity/3.9.2
#-B /tmp/vcftemp:/tmp/vcftemp
singularity run -B /fs02 -B /fs04 -B /fs03:/fs03 -B /projects:/projects -B
/scratch:/scratch ${hello}/hello.sif
python /opt/hello/python/call.py
--ibam GALF003.p0.03.bam
--ref $REF
--network
models/illumina_multi_coverage_mapq_threshold_hg002_continue_run16.wrapper.dnn
--num_threads $NUM_THREADS
--workdir $workdir/GALF003_1x_tmp --mapq_threshold 5
Error:
INFO:root:Combining all hotspots and sharding
2022-04-16 10:57:52,214 Directory
/projects/xm41/robert/hello/GALF003_1x_tmp/hotspots_hello___GALF003__p0__03__bam_chr22_
exists
2022-04-16 10:57:52,214 Counting hotspots
2022-04-16 10:57:52,214 Sharding with >= 0 items per cluster
INFO:root:Completed sharding, creating caller commands for dumping
training data
INFO:root:Created call command for chromosome chr22
INFO:root:Launching all caller commands
Computers / CPU cores / Max jobs to run
1:local / 8 / 1
ETA: 0s Left: 0 AVG: 0.00s 0
INFO:root:Completed runs, checking log files
INFO:root:All commands completed correctly, preparing vcf
sort -k1,1d -k2,2n
sort -k1,1d -k2,2n
Search string /tmp/vcftemp/*expert0.vcf
Found 0 files
Running command ['vcf-sort',
'/projects/xm41/robert/hello/GALF003_1x_tmp/results.ngs.tmp.vcf']
Search string /tmp/vcftemp/*expert1.vcf
Found 0 files
Running command ['vcf-sort',
'/projects/xm41/robert/hello/GALF003_1x_tmp/results.tgs.tmp.vcf']
Search string /tmp/vcftemp/*expert2.vcf
Found 0 files
Running command ['vcf-sort',
'/projects/xm41/robert/hello/GALF003_1x_tmp/results.ngs_tgs.tmp.vcf']
Search string /tmp/vcftemp/*best.vcf
Found 0 files
Running command ['vcf-sort',
'/projects/xm41/robert/hello/GALF003_1x_tmp/results.best.tmp.vcf']
Search string /tmp/vcftemp/*mean.vcf
Found 0 files
Running command ['vcf-sort',
'/projects/xm41/robert/hello/GALF003_1x_tmp/results.mean.tmp.vcf']
Choice histogram = {0: 0, 1: 0, 2: 0}
If I run the script with /tmp:vcfemp bound to the container, I will
receive the following error:
Script:
REF="/scratch/xm41/hg38_resources/resources_broad_hg38_v0_Homo_sapiens_assembly38.fasta"
NUM_THREADS=8
workdir="/projects/xm41/robert/hello"
hello="/usr/local/hello/709b7c3/"
mkdir -p /tmp/vcftemp
module load singularity/3.9.2
singularity run -B /fs02 -B /fs04 -B /fs03:/fs03 -B /projects:/projects -B
/scratch:/scratch -B /tmp/vcftemp:/tmp/vcftemp ${hello}/hello.sif
python /opt/hello/python/call.py
--ibam GALF003.p0.03.bam
--ref $REF
--network
models/illumina_multi_coverage_mapq_threshold_hg002_continue_run16.wrapper.dnn
--num_threads $NUM_THREADS
--workdir $workdir/GALF003_1x_tmp --mapq_threshold 5
Error:
ETA: 0s Left: 4 AVG: 0.03s local:4/497/100%/0.0s
ETA: 0s Left: 1 AVG: 0.03s local:1/500/100%/0.0s
ETA: 0s Left: 0 AVG: 0.03s local:0/501/100%/0.0s
INFO:root:Combining all hotspots and sharding
2022-04-16 10:45:26,049 Directory
/projects/xm41/robert/hello/GALF003_1x_tmp/hotspots_hello___GALF003__p0__03__bam_chr22_
exists
2022-04-16 10:45:26,050 Counting hotspots
2022-04-16 10:45:26,050 Sharding with >= 0 items per cluster
INFO:root:Completed sharding, creating caller commands for dumping
training data
INFO:root:Created call command for chromosome chr22
INFO:root:Launching all caller commands
Computers / CPU cores / Max jobs to run
1:local / 8 / 1
ETA: 0s Left: 0 AVG: 0.00s 0
INFO:root:Completed runs, checking log files
INFO:root:All commands completed correctly, preparing vcf
Traceback (most recent call last):
File "/opt/hello/python/prepareVcf.py", line 255, in
shutil.rmtree(args.tmpdir);
File "/opt/miniconda3/lib/python3.7/shutil.py", line 498, in rmtree
onerror(os.rmdir, path, sys.exc_info())
File "/opt/miniconda3/lib/python3.7/shutil.py", line 496, in rmtree
os.rmdir(path)
OSError: [Errno 16] Device or resource busy: '/tmp/vcftemp'
Unfortunately, I cannot create the problem I mentioned in my previous
message.
—
Reply to this email directly, view it on GitHub
https://github.com/anands-repo/hello/issues/5#issuecomment-1100493038,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADJJMQHHVJ2I2UWMQOWJ5VLVFIG3BANCNFSM5SIFJ2LQ
.
You are receiving this because you commented.Message ID:
@.***>

.


Clarified the comparison according to jhaals clarification in Issue 10

Hi Johan, thanks for your clarifications in https://github.com/jhaals/ansible-vault/issues/10#issuecomment-232313205
Hoping you're willing to include the revisions in your README for my bad memory in the future, as well as other persons like me who might be similarly confused.
Thanks again.

Will do! Thanks for the PR


Feat 1.1.2

Codecov Report
Base: 100.00% // Head: 100.00% // No change to project coverage :thumbsup:

Coverage data is based on head (b1987ad) compared to base (5e7ce3c).
Patch has no changes to coverable lines.

Additional details and impacted files
@@            Coverage Diff            @@
##            master       #12   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files            8         8
  Lines          116       116
  Branches        23        23
=========================================
  Hits           116       116

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


Remove "Was this page helpful?" form

Unfortunately the form itself was not very helpful due to the amount of spam responses it received.
We can investigate adding this back someday if we can gate it to only employees.

@danoc I just noticed this form is still present on a few pages.
Examples:

https://thumbprint.design/components/action-sheet/ios/
https://thumbprint.design/help/
https://thumbprint.design/updates/notes/


Member profiles do not line up when profile information is missing

Description
On the members page, if there is missing GitHub profile information, the member's profile becomes oout of line with the rest of the members
Solution
Take into account empty member profile information when spacing the member profile information.
Expected Behaviour
All member profiles should line up properly.
Screenshots
Here is a screenshot of the issue.

Resources

Flexbox
CSS Grid

@nickytonline thank you so much for the issue! This is actually handled in the virtualcoffee.io repo
But this also brings to light some improvements we can make to make this more clear to everyone 🙏

Ooops, I thought I was in the right repo. My bad. 🙃


DOC more consistent formatting in docstring

I uniformized the keys order in InterpolationJoiner and Joiner to match the Polars and Pandas API
Small number of cosmetic changes to docstrings

thanks a lot @TheooJ !


Error with Svelte 5.0.0

Describe the bug
✘ [ERROR] No matching export in<EMAIL_ADDRESS>for import "get_current_component"
node_modules/.pnpm/flowbite-svelte@0.44.20_svelte@5.0.0-next.26/node_modules/flowbite-svelte/dist/utils/createEventDispatcher.js:9:9:
  9 │ import { get_current_component } from 'svelte/internal';
    ╵          ~~~~~~~~~~~~~~~~~~~~~

10:43:53 PM [vite] error while updating dependencies:
Error: Build failed with 1 error:
node_modules/.pnpm/flowbite-svelte@0.44.20_svelte@5.0.0-next.26/node_modules/flowbite-svelte/dist/utils/createEventDispatcher.js:9:9: ERROR: No matching export in<EMAIL_ADDRESS>for import "get_current_component"
at failureErrorWithLog (/Users/maitrongtue/Documents/GitHub/odm/node_modules/.pnpm/esbuild@0.19.9/node_modules/esbuild/lib/main.js:1650:15)
at /Users/maitrongtue/Documents/GitHub/odm/node_modules/.pnpm/esbuild@0.19.9/node_modules/esbuild/lib/main.js:1058:25
at /Users/maitrongtue/Documents/GitHub/odm/node_modules/.pnpm/esbuild@0.19.9/node_modules/esbuild/lib/main.js:1526:9
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
Reproduction

Default
Alternative
Flowbite version and System Info
Svelte 5.0.0

"peerDependencies": {
"svelte": "^4.0.0"
},

I thinks error in function utils/createEventDispatcher.js: import "get_current_component"... May be big change event function on Svelte 5.0.0.

This problem is already addressed with PR #1168
It needs the official tag though.

@tuemaytinh1981 new version is released now. Can you please check if issue is corrected?

@tuemaytinh1981 new version is released now. Can you please check if issue is corrected?

I still get this issue @jjagielka , It wont let me install unless I have v4 installed, which doesn't have some features I'm using.

@GreenDiscord the line import { get_current_component } from 'svelte/internal'; does not exist any more in the code base.
Can you please give the details what is blocking you?


torch.ByteTensor and nonzero() method

Can I apply nonzero() method to torch.ByteTensor object? or do I need to convert to torch.FloatTensor at first?

Yes, you can apply nonzero to torch.ByteTensors.

Ok. Can I apply the same method to torch.CudaByteTensor?

There is unfortunately no nonzero() implementation for CUDA yet (just realized the issue was opened in the cutorch repo)

Some time ago @soumith mentioned that Facebook implemented nonzero for CUDA in #70

this is now pushed into master


Updated Makefile to new mains usage

Coverage remained the same at 76.378% when pulling 85ced37a9bf369d70f7c8214a17ace399f9193cb on doc/adaptmakefile into ed02994d37b6d367d1cc0356021c15cb42a6c708 on master.


KDTree for accelerating NMS

This pull request solves issue #39.
After polygon candidates have been reformatted to parallel lists of 3D points and distances (sorted by probability), I construct a KDTree from the points using scipy.spatial.cKDTree. This takes negligible time for all of my tested cases (a few seconds at most). I then call kdtree.query_ball_tree which returns a list of lists - which is also parallel to points/dists. The i^th sublist contains the points which are sufficiently close to point i to merit comparison in NMS. This query is somewhat time consuming, for the largest case tested (6.32e6 points) it took 23 minutes.
This list of lists is passed to c_non_max_suppression_inds using the O& option to PyArg_ParseTuple and a custom python -> c data types converter function lists_to_vectors.
The inner loop of _COMMON_non_maximum_suppression_sparse is modified to only iterate over these candidate pairs. This capitalizes on the investment of the previous bounded neighbors query. For the largest tested case it brought NMS time down from 7.5 hours to 20 minutes, while still finding the same set of polygons.
I evaluated NMS time over a wide number of polygon candidates. Here are the results (note the log_10 scale on the vertical axis):

In all cases except the largest, the number of detected polygons between the baseline method and using the kdtree was exactly the same. For the largest case, the baseline method detected 90535 polygons and the kdtree method detected 90536 polygons. Here is a side by side comparison of a single plane of polygons detected in the largest case for both methods:

and an overlay (baseline method in red, with kdtree in green):

Correspondence is exact through the whole stack - I'm not sure where that extra one is - too hard to find by eye :) !
Some additional notes
The kdtree query time also scales with number of points, but with a worst case complexity of O(N*logN). For very few points (left side of plot) the query time is negligible (milliseconds) - so the kdtree doesn't hurt in those cases.
The O(NlogN) query time for the kdtree could also be reduced to O(ClogN) where C is a constant strictly smaller than N, using a better implementation. Currently, I'm building the kdtree in python, running the bounded neighbors query for all points and then passing the result to the C++ code. Some of these points get suppressed before their bounded neighbors lists are used, and thus computing their bounded neighbors was a waste. If the kdtree data structure was directly accessible in the C++ code, you would instead want to query for bounded neighbors of each individual point at the start of the inner loop of NMS. That way, points which were already suppressed never need to be queried; I think this would cut the query time in half at least. It's a bit more tricky to implement, but definitely possible. I would probably grab the cKDTree implementation from scipy and start from there.
The one extra polygon detected for the largest test case illustrates an interesting aspect of the kdtree.query_ball_tree (bounded neighbors) search. You must provide an upper bound. You can guarantee that any polygon which could be suppressed by a point is included in it's bounded neighbors list by using 2 * np.max(dists) as the upper bound (any polygons further than that can not possibly overlap), but this results in a pretty long query time, though likely still an overall gain compared to NMS w/o the kdtree. The current choice is 2 * (np.mean(dists) + np.std(dists)) which gives acceptable results for my dataset and is a bit quicker than the max - so I've stuck with it.

Oh I also just realized, the approximate times I give for the cKDTree.query_ball_tree query times also include passing the list of lists to the c code and converting it to a vector of vectors. My C++ is really rusty, so I might be doing something really inefficient in this conversion (lists_to_vectors function in stardist3d.cpp), e.g. the push_back function seems suspect to me because it reallocates RAM every time the vector grows. There is likely a better way to write this.

Hi Greg,
thanks a lot for this contribution - it already looks very promising!

I evaluated NMS time over a wide number of polygon candidates. Here are the results (note the log_10 scale on the vertical axis):

Thats a nice speed up!  The 6Mio candidates are for the whole stack, I assume? So using predict_big doesn't have any effect on the prediction time?
I'm slightly worried about the additional memory footprint that a full kdtree of all candidates    might bring - did you try measuring the peak memory usage?  Additionally I suspect there is quite some heavy differences even for different kdtree implementations e.g. there are KDTree and BallTree from scikit-learn. Here's a short test I did with random points in 3D and those different implementation
"ckdtree" -> scipy.spatial.ckdtree
"kdtree" -> sklearn.neighbors.KDTree
"ball" -> sklearn.neighbors.BallTree

shows that i) there are quite some difference esp. for memory, and ii) the overall memory footprint is pretty high (e.g. 18GB for a mere 30k points for ckdtree).
So having a similar plot as above but for memory consumption in your example would be nice and instructive. Additionally moving to BallTree might make things more efficient...

Hi Martin,
Yes, the 6Mio candidates are for prediction on the whole stack. I did try predict_big on this volume. What I noticed was that the memory footprint went way down. Using Uwe's recommended parameters I was able to run the job with a single cpu (# cores requested and maximum amount of RAM available are mixed on our cluster; each core comes with 15GB RAM). However the remaining runtime predicted by tqdm after a few iterations had accumulated was something like 36 hours - so I cancelled it and gave up on it. Uwe reminded me that with one core, I likely had a maximum of 2 threads, but I did not go back and try predict_big with more cores. Do you think predict_big with say 32 cores (64 threads) could finish in less than an hour? If so I will give it another try.
For different kdtree implementations - I did look through this article written by the scikit-learn kdtree and balltree developer, but I didn't think there was enough differences in his graphs to justify introducing a new dependency (I don't think scikit-learn was a dependency before?) and I also preferred something with an underlying C++ implementation, as there are more big speed ups available if the kdtree data structure is available for query directly within the NMS algorithm.
Unfortunately the only RAM measurements I have are the maximum and average usage (over the full run, prediction and NMS) for prediction on the full volumes both w/ and w/o the kdtree. The command in both cases was:
pred, det = model.predict_instances(image_norm, verbose=True, n_tiles=(4, 8, 2))
On a volume with 860x1676x301 voxels and 6.32e6 polygon candidates
w/o kdtree max: 434.439GB
w/   kdtree max: 430.482GB
w/o kdtree avg: 221.672GB
w/   kdtree avg: 297.733GB
One thing that I haven't looked at in the code - it seems like the biggest expansion of data in the method is from raw data --> prediction; e.g. in my case I'm using 128 rays, so I need (129 * 2) more RAM relative to my input image (assuming 32 bit floats for probs and dists and 16 bit int for image). The n_tiles parameter cuts the image up into overlapping tiles for prediction. Does each tile also get separately prob thresholded before stitching back up for NMS? Based on the way RAM usage appears to me, it seems like no; but that would save a lot of space, since the dists of most voxels could be thrown out before the dists of new voxels were predicted/stored. I think if this were implemented, then the additional RAM of the kdtree would become an important factor.

Hi Martin,
I think you are right about scikit-learn kdtree outperforming scipy cKDTree. While working on another change to the package, I was led to trying the scikit-learn kdtree. For the same dataset considered above, the query was a few minutes faster than scipy cKDTree, but more importantly - due to the underlying data types used - the query results consumed significantly less RAM.
For both packages I've found that the tree structure itself takes a negligible amount of RAM - I'm not sure exactly how this is implemented under the hood, but I suppose each node in the tree could be as small as a single pointer to where in the array the points are split. If that's the case you really only need something like 3*log_2(N) references (3 axes and N points). However it's implemented, empirically it's quite small.
The query results however can be quite large, in the abstract it's a list of lists. The outer list is of length N (for N points) and the ith inner list is of length equal to the number of points within radius distance of the ith point. So this could be on the order of ~1000 * N long ints - that's something like 25GB without considering any extra metadata overhead.
Scipy represents this object as a Python list of Python lists of Python ints. Python ints contain a cumbersome amount of metadata, so I don't think they're a great data type here. Scikit-learn on the other hand represents this as a Numpy array with dtype = 'O', that is an array of objects. The objects are 1D Numpy arrays themselves with dtype = np.int64. This seems to be overall more space efficient. I'd love to select np.int32 instead, since that's more than adequate, but it does not seem to be possible without modifying the Scikit-learn.neighbors.kdtree code (maybe still worth doing in the long run).
So - combining the small query speed up and the better space efficiency, I've added code to use scikit-learn kdtree in a new commit.

I've made some significant improvements by switching to the scikit-learn version of KDTree (good suggestion!) and changing the way the KDTree is used within NMS. To keep things clean, I'm going to delete my existing fork, create a new one, and

Hi @GFleishman,
Thanks for all the work! Two things that I think are still worth pondering:

The max_distance cutoff for the spatial data structure should ideally be chosen such that no potential intersecting polygon is missed. Some options are

(Mean) max_distance= 2*(np.mean(dist)+np.std(dist)) what is used in the PR, but might miss some pairs
(Max) max_distance= 2*np.max(dist), preferable as it won't miss any pairs but which leads to order of magnitude more pairs.

There likely won't be any difference between (Mean) and (Max) in practice (as your initial comparison showed), but I am still a bit wary of not having that guarantee.

Memory: I tried  prediction on a small crop (with ~250k candidates) of the test data above:

without kdtree: 15.4 GB, 97s
with kdtree (Mean): 15.5 GB, 98s
with kdtree (Max): 39.2 GB, 121s

So using (Max) will clearly take a hit on memory...

Hi @maweigert,
Right now I'm working on a different strategy from the software standpoint that will solve the extra time cost for using Max and the extra memory from having to store the larger lists of neighbors (potential suppression candidates). My first implementation in the PR was a good demonstration of kdtree, but definitely suboptimal w.r.t. what can be done with the Python C-API, which I'm learning on the fly so things have gone through various stages of kludge.
The idea now is that I can call the sklearn kdtree object, and it's query_radius method, from within the C++ code itself - enabling a separate query for each polygon candidate just before the inner loop of NMS begins. That way, points which are "suppressed early" will never be queried against the tree - reducing total query time even if the distance threshold is large. The second benefit is that the code now only has to store one neighbor list at a time, thus reducing the total RAM used. This is definitely the best way to built it algorithmically but I still don't have the Python C-API down perfectly so implementing this today has been tricky.
I also implemented what I mentioned before - applying the probability threshold for each tile separately during prediction. For the whole image, it cut max RAM down from ~400GB to ~50GB, and that was including the neighbors list from kdtree. So, if I can get the tile based threshold combined with the embedded python approach to the kdtree, I think I can get the whole zebrafish image segmented in like 10-15 minutes for something like 30GB RAM. Of course, the tile based prob threshold breaks a bunch of stuff, so I'll leave it to you guys to decide if you want to move in that direction generally or to just have my commit somewhere accessible on the repo so if others run into RAM issues there's some path to cutting it down without changing so many internal dependencies between functions.
This is all WIP but I'm hoping to finish it soon.
Thanks,
Greg

Hi @GFleishman ,
sounds great!

The idea now is that I can call the sklearn kdtree object, and it's query_radius method, from within the C++ code itself

Yes, that seems to be the preferable way. Having external source code be compiled and linked correctly via setup.py probably the biggest hurdle (it was quite some hassle with qhull).

I also implemented what I mentioned before - applying the probability threshold for each tile separately during prediction

Yes, I have a local branch that does exactly this. We haven't merged that feature yet, as long term we want to use the whole prob/stardist map for further shape refinement. But it might be worth reconsidering it for the pure stardist shapes (as it leads to huge memory reduction).
For this PR though, I would probably keep these two things (kdtree, prob thresholding) separate and first focus on the kdtree.
Great to see this rolling! :)

Hi @maweigert ,

Having external source code be compiled and linked correctly via setup.py probably the biggest hurdle (it was quite some hassle with qhull).

Unfortunately I don't think I can do it this way, since sklearn kdtree is all scientific python stack under the hood, i.e. no C code. AFAIK you can't compile and link this like you would an external C library? My approach was to try and run sklearn kdtree from within the C code using the Python C-API - something like what's going on here.
But it's been tricky to determine exactly what is needed, since this and the linked situation are different. Here we have a Python extension module (Stardist3d) which handles all the Python interfacing then calls implementations from another C++ file (Stardist3d_impl) passing only C style array pointers. In the linked code, they have an embedded Python interpreter which they interact with entirely within C.
Although I see now it kind of mixes up the Python interface vs. implementation separation, I was trying to do things like:
#include <Python.h>
#include "numpy/arrayobject.h"
.
.
.
PyArrayObject* point = (PyArrayObject*) PyArray_SimpleNew(2, dims_point, NPY_FLOAT);

in Stardist3d_impl, which compiles with no warnings, but then the program just hangs on this line when executed - I guess it might eventually SegFault, but I never give it long enough to find out, after hanging for a few minutes I just kill the process, since that line shouldn't take longer than a few milliseconds(?).
After trying multiple ways to initialize a numpy array in Stardist3d_impl and all showing the same failure mode, I'm trying to understand why this doesn't work, and also considering other designs like somehow having the kdtree object live in Stardist3d and just passing a callback function to stardist3d_impl or something. Do you have any ideas here?

Yes, I have a local branch that does exactly this. We haven't merged that feature yet, as long term we want to use the whole prob/stardist map for further shape refinement. But it might be worth reconsidering it for the pure stardist shapes (as it leads to huge memory reduction).

Yeah, I figured that was the strongest pro for keeping the whole image of predictions. I think having a separate branch in the repo for RAM considerations is good though - then users can just switch and recompile to move into "maximum memory efficient" mode if that suits their application better. It could also eventually be part of the Config. So, I'll submit it on a separate branch/PR.
This has been fun to work on so far :)

Hi @GFleishman,

Do you have any ideas here?

Yes, as far as I see incorporating sklearn into the c extension itself will become tricky and messy outright. There is another reason why this might not what we want: Currently, the c++ implementation (stardist3d_impl.cpp) and the python wrapper (stardist3d.cpp) are separated, such to allow all the C++ code to be compiled into a clean native library that can be used e.g. from Java/Fiji and where linking against libpython is not desirable.  There indeed is already a Makefile providing exactly that functionality. So I would opt for something C++ only.
But there might be good news! I just had a look into C++ kdtree implementations and there are some that look suitable for us. A good candidate might be nanoflann, which is BSD and even header only (meaning including a single hpp would be enough). There already is a radius query example which is exactly what we would need and it works just fine for me. Even better, some quick benchmark I did suggests, that query times are a magnitude faster than sklearn!

So looks like a perfect match!

then users can just switch and recompile to move into "maximum memory efficient" mode if that suits their application better.

Yep, that is a sensible strategy.

This has been fun to work on so far :)

It is! :)

Awww - I just got the sklearn integration to work a few hours ago! There is a bit of trickyness involved in using the numpy C-API from multiple compilation units (here it would be stardist3d.cpp and stardist3d_impl.cpp); the answer there helped sort it out and got things working.
I agree that keeping the Python module and C++ implementations separate is best. For now, the integrated sklearn strategy can at least demonstrate what kind of gains we'll get by putting the kdtree properly in the nms loop. I'm running some tests now and will report back. I did switch back to the "max" approach, but quite nicely I'm now using radius = dist_max + local_dist_max where dist_max is the largest distance in the whole array of polygon candidate distances, and local_dist_max is the max over dists for just the current polygon candidate (polygon i within the outer NMS loop).
Great find with nanoflann! I'll have a look and see if I can get it to compile into stardist!

Is it possible that you compared radius = 0.1**2 for nanoflann to radius=0.1 for sklearn? That would definitely cause a speedup (of course it could also just be really fast!)

Yes, that tripped me off at the beginning, but the other way around (using radius^2=0.1 for nanoflann, so massively underestimating the speed up :)

Great, thanks for the demo!
In the meantime, I can confirm the expected speed up and RAM reduction for embedding the kdtree in the NMS loop. These are quick results from the same image I shared before:
The NMS time was the same between embedded-kdtree and external kdtree, ~20 minutes
However, the ~13 minutes required to query the tree for all candidate points is no longer present for the embedded-kdtree (since queries are done individually within the NMS outer loop).
When combined with prob_thresholding on individual tiles, the maximum RAM over prediction and NMS for the embedded-kdtree was 17.378GB.

Hi,
Following on this (great!) discussion , there are now two new branches spare and kdtree
which I would love to see your benchmarks on:

I added the sparse prediction code that I mentioned earlier into a new branch sparse, which provides now an optional parameter
model.predict_instances(... sparse=True)
which will reduce memory footprint considerably for large volumes.
Let me know if that works for you!

I did play around with the nanoflann code a bit and added it to another new branch kdtree. It uses a kdtree radius search with the max_distance following your earlier comment (max+ local_max). There is now an option
model.predict_instances(... use_kdtree=True)
Here's a comparison for crops of your data with and without kdtree:

So while memory footprint is the same, the kdtree implementation is already 3 times faster even for a relatively small subvolume. Pretty good.
Note that this branch kdtree is as of now only a short demo that shows how it could be done with nanoflann -  I would wait for your PR to see would piece all together with your code.

Hi again!
Sorry for the long delay - when I got sklearn KDTree working with tile based thresholding locally I was able to step back and work on some of the datasets that I need results for. I will try out your branches soon, but I expect they're excellent and won't need any modifications from me :) But I will let you know.

Hello @maweigert!
I hope the last 3 months have been good for you and that you're staying safe and sane. I've just gotten back into structural segmentation after a long trip doing functional segmentation. I've used the kdtree branch of the repo with nanoflann and it works very well. I haven't done any actual comparison, but the nms time seems as fast or faster than my implementation calling the sklearn methods using the python-API.
One weird quirk I've noticed though - the NMS loop seems to run about 2x slower when I run model.predict_instances with a gpu available. The prediction itself is 2-3x faster on the gpu (definitely worthwhile), but again the NMS loop is about 2x slower in the gpu case. I have confirmed this with two separate experiments on our gpu/cpu cluster and both times ensured that the loop was running with the same number of threads available. I'm working with our system administrators to see if there is some difference between our cpu exclusive and gpu connected nodes but so far we haven't identified anything that could explain the slowdown - the two different node types use the same OpenMP library for example.
So, I'm wondering where the prob/dist maps are after prediction on the gpu. Are these for sure sent back from gpu memory to system memory before the NMS loop starts? It seems like they for sure must be since they are C++ arrays/pointers but I don't know what I don't know so I thought I would ask before ruling this possibility out and looking further into potential differences on our cluster nodes.

Hi @GFleishman

I've used the kdtree branch of the repo with nanoflann and it works very well. I haven't done any actual comparison, but the nms time seems as fast or faster than my implementation calling the sklearn methods using the python-API.

Good to hear! :)

the NMS loop seems to run about 2x slower when I run model.predict_instances

This is strange and I don't see a reason why that would be the case.

Are these for sure sent back from gpu memory to system memory before the NMS loop starts?

Yes, they are in the CPU (they are returned by keras as numpy arrays). Before the NMS, both prob and dist are additionally converted to continuous c arrays, so they definitely are already on the host memory.
How do you time the NMS loop?

Hi @maweigert :),

Yes, they are in the CPU (they are returned by keras as numpy arrays). Before the NMS, both prob and dist are additionally converted to continuous c arrays, so they definitely are already on the host memory.

This is what I suspected. Ok, I think it must be some difference between the gpu connected nodes and cpu exclusive nodes on our cluster. They are different processor models, so maybe I should try compiling on the gpu connected nodes? Previously I had compiled on one of the cpu exclusive nodes, so maybe it's more optimized for that processor architecture?

How do you time the NMS loop?

I'm running with verbose=True so the end of the NMS loops writes out a total NMS time in seconds. I'm just using that.

They are different processor models, so maybe I should try compiling on the gpu connected nodes?

Yes, that could be. I would suggest comparing CPU vs GPU speed on the very same GPU node and disable tf-gpu via  CUDA_VISIBLE_DEVICES= python ... in one of the two runs.

@GFleishman FYI, the block-wise prediction in the latest release is now also shown here: https://github.com/mpicbg-csbd/stardist/blob/dev/examples/other2D/predict_big_data.ipynb
While the example notebook is for a 2D example, it should also work for 3D images.


[feature] Add "birthday" (aka "block height") option when restoring wallet

[being moved here from daedalus#2416]
When restoring a wallet from seed phrase, the user should have an option to set a so-called "birthday" (block height or date) of the first transaction. The idea is that transactions from older blocks can be ignored unless they have direct bearing on UTXOs owned by this wallet.
The goal is dramatically faster syncing.
I am coming at this from the Bitcoin / Monero space where this feature has been part of the UX for a long time. Perhaps there is something different in Cardano which prevents such a speedup?
Jira ticket
ADP-106

@jonathancross Thanks for your feedback here. Note that we have captured this (i.e. the ability to restore a wallet from a given date / slot number) in a user story some times ago already. As for many other UX improvements that we have in the backlog, what we are missing so far is a bit of time to go through them. As you know, Cardano is evolving at a quite crazy pace and so far, we've been busy in catching up with era changes in the ledger / the node.
We've been discussing internally about making some of our roadmap in the wallet a bit more public to give people visibility about this kind of things. From the Github repository, it may looks like we aren't doing much, but we actually have a quite large backlog of UX items like this one.
Short story short: I totally agree with you and this would be a nice feature.

Transparency and open collaboration are good  :-)

We implemented a feature like that, see "Restoration mode / Restoration from block" in the create wallet endpoint.


fix(API): signals and frames semantics

Because of what these return when iterated, they act more like a
Sequence rather than Mapping, so we're changing them to actually map
that.
Fixes #173
BREAKING CHANGE: session.frames and session.signals no longer have
keys, items, and values.  We do not expect this to impact anyone.

[x] This contribution adheres to CONTRIBUTING.md.
[x] New tests have been created for any new features or regression tests for bugfixes.
[x] tox successfully runs, including unit tests and style checks (see CONTRIBUTING.md).

Coverage remained the same at 64.175% when pulling fbb2a886dbceb309684decb37c4785c427a83072 on epage:mapping into 5590c92682e9a33adc823396a3565fa4dbb47055 on ni:master.

Coverage remained the same at 64.175% when pulling fbb2a886dbceb309684decb37c4785c427a83072 on epage:mapping into 5590c92682e9a33adc823396a3565fa4dbb47055 on ni:master.

Coverage remained the same at 64.175% when pulling e2e8f3eaba54e918193a6816a0d89f3d87441593 on epage:mapping into 5590c92682e9a33adc823396a3565fa4dbb47055 on ni:master.

Coverage remained the same at 64.175% when pulling e2e8f3eaba54e918193a6816a0d89f3d87441593 on epage:mapping into 5590c92682e9a33adc823396a3565fa4dbb47055 on ni:master.

Coverage remained the same at 64.179% when pulling 7c01b4b1f43ebb49082247ad6b2c9410a928e8f5 on epage:mapping into 2b0b3bdef21f1953110a2a23a7fe4a5b60a4d29c on ni:master.


GH-38814: [C++][Parquet] Update parquet.thrift to sync with 2.10.0

Rationale for this change
Apache Parquet has just released its format version 2.10.0: https://parquet.apache.org/blog/2023/11/20/2.10.0/. We need to update parquet.thirft and its generated source files to be in sync.
What changes are included in this PR?
Update parquet.thirft and its generated source files.
Are these changes tested?
Make sure all tests pass.
Are there any user-facing changes?
I don't think so.

Closes: #38814

I just updated the thrift file. cc @pitrou @emkornfield

@github-actions crossbow submit -g cpp

@github-actions crossbow submit -g cpp

Just curious, when should we execute this command? @pitrou

https://github.com/ursacomputing/crossbow/actions/runs/6943255937/job/18887922572 failed with following error. Let me create an issue:
[ RUN      ] TestArrowReaderAdHoc.ReadFloat16Files
/arrow/cpp/src/parquet/arrow/arrow_reader_writer_test.cc:4129: Failure
Failed
'reader->ReadTable(&table)' failed with NotImplemented: Support for codec 'snappy' not built
Google Test trace:
/arrow/cpp/src/parquet/arrow/arrow_reader_writer_test.cc:4123: path = /arrow/cpp/submodules/parquet-testing/data/float16_nonzeros_and_nans.parquet
[  FAILED  ] TestArrowReaderAdHoc.ReadFloat16Files (1 ms)
[----------] 6 tests from TestArrowReaderAdHoc (11 ms total)

@github-actions crossbow submit -g cpp

Just curious, when should we execute this command? @pitrou

There is no rule about this. In this case, I was wondering if the newly-generated Thrift files could fail compiling on some platforms.
Just use your committer's intuition, really :-)

Does it has any rules for these codec in parquet-testing? Seems brotli, lz4/lz4_raw, zstd, snappy are all used there...

Will merge in one day if other doesn't have negative views.


Youtube channel

I was wondering why I don't see my channel featured on your YT channel

It's on there

In which section?

Okay but I'm in school right now

same but its almost done. Take your time when I post it to upload it

Can you subscribe to me for videos and also I don't see my channel on there

I see it. Thanks


Link to Github ASR quickstart was invalid

Updated link to https://github.com/Azure/azure-quickstart-templates/tree/master/asr-automation-recovery/scripts

@eltimmo Thanks! #sign-off

@rayne-wiselman: I'm sorry - only the author of this article, @ponatara, can sign off on your changes. But we do have an exception process - if you are on the Microsoft content or product team for this product area, you can ask the PR review team to review and merge it by sending mail to the techdocprs alias.


Question: How to test code that relies on Duck-typed interface?

Summary
Documentation suggests usage of Duck-typed interfaces
#[wasm_bindgen]
extern "C" {
    pub type Quacks;

    #[wasm_bindgen(structural, method)]
    pub fn quack(this: &Quacks) -> String;
}

/// Next, we can export a function that takes any object that quacks:
#[wasm_bindgen]
pub fn make_em_quack_to_this(duck: &Quacks) {
    let _s = duck.quack();
    // ...
}

But it is quite unobvious how to write unit test for make_em_quack_to_this function in Rust. Is there any way how to build instance of Quacks in Rust for passing to the tested function?

I think the "best" way to test this would be to write a Rust function that calls to JS which calls back into Rust's make_em_quack_to_this function.

@alexcrichton What is good place to write js code in this case? JS snippets have problems with imports and nodejs target.

Ah unfortunately I don't think there's a good place. The tests in this crate itself do some wonky things with NODE_PATH to get things to work, but otherwise it's not super well supported.


how to instantiate client for ledger wallets with no eth_private_key?

hi there,
how to instantiate client for ledger wallets with no eth_private_key?
I am following example here:

SAMPLE 1: onboarding: https://github.com/dydxprotocol/dydx-v3-python/blob/master/examples/onboard.py
SAMPLE 2: create order: https://github.com/dydxprotocol/dydx-v3-python/blob/master/examples/orders.py

I want to simply query account "Equity".
client = Client(
network_id=NETWORK_ID_MAINNET,
host=API_HOST_MAINNET,
default_ethereum_address=ETHEREUM_ADDRESS,
eth_private_key=ETHEREUM_PRIVATE_KEY, <---- I dont have private key as it's ledger wallet, what to do?! (And I didnt use SAMPLE 1 to onboard the account. I done it via dydx website https://trade.dydx.exchange and there's no place from there where I can say export stark_public_key, stark_public_key_y_coordinate. Assuming this is what I need? I dont think "stark_private_key" can ever be exported?)
web3=Web3(Web3.HTTPProvider(WEB_PROVIDER_URL)),
)
stark_private_key = client.onboarding.derive_stark_key()
client.stark_private_key = stark_private_key
account_response = client.private.get_account().data['account']
account_free_collateral_usd = float(account_response['freeCollateral'])
account_equity_usd = float(account_response['equity'])
I checked client constructor:
class Client(object):
def init(
self,
host,
api_timeout=None,
default_ethereum_address=None,
eth_private_key=None,
eth_send_options=None,
network_id=None,
stark_private_key=None,
stark_public_key=None,   <---- This is what I need?
stark_public_key_y_coordinate=None,  <---- This is what I need?
web3=None,
web3_account=None,
web3_provider=None,
api_key_credentials=None,
crypto_c_exports_path=None,
)
Thanks in advance.

I also tried ed by the frontend app to derive the STARK key pair in a way that is recoverable. Programmatic traders may optionally derive their STARK key pair in the same way.
client.onboarding.derive_stark_key(ethereum_address=ETHEREUM_ADDRESS,)
Error was:
raise ValueError(response["error"])
ValueError: {'code': -32601, 'message': 'The method eth_signTypedData does not exist/is not available'}

Solution in above


ente-auth: 4.1.6 -> 4.2.1

Changelogs:

https://github.com/ente-io/ente/releases/tag/auth-v4.2.0
https://github.com/ente-io/ente/releases/tag/auth-v4.2.1

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
25.05 Release Notes (or backporting 24.11 and 25.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Diff LGTM can someone test the binary?

Thanks everyone!


svg started with BOM detected as xml

Hello
If .svg file starts with BOM(EF BB BF), mmmagic detected this as 'xml'.
In attachment test1.svg and test2.svg is equal, but mmmagic return different type
test.zip

Please try the master branch, I have merged in libmagic 5.32 now.

I've published v0.4.6. If this is still a problem, please report the issue upstream to the libmagic/file bug tracker.


[deck arcs] add JS hooks for sourceColor & targetColor

This is just adding in some hooks to make it available through JS. Eventually we probably want 2 color pickers.
@betodealmeida

Codecov Report

Merging #5706 into master will decrease coverage by <.01%.
The diff coverage is 0%.

@@            Coverage Diff             @@
##           master    #5706      +/-   ##
==========================================
- Coverage   63.45%   63.44%   -0.01%
==========================================
  Files         361      361
  Lines       22972    22974       +2
  Branches     2557     2559       +2
==========================================
  Hits        14576    14576
- Misses       8381     8383       +2
  Partials       15       15

Impacted Files
Coverage Δ

...et/assets/src/visualizations/deckgl/layers/arc.jsx
0% <0%> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8992755...66fc941. Read the comment docs.


CLI: Compress / decompress

Add compress and decompress commands in the CLI such that files can be handled

The desired behavior for this issue is that when the user types "compress," all extra files will be compressed, and when the user types "decompress," all files will be decompressed.
Alternatively, does the user need to pass a variable to specify a single file to be compressed or decompressed?

I tried to follow cli.c and management.c to understand how encryption is implemented, but I ended up getting confused and couldn't find the exact code for executing encryption. How do the CLI commands work in our project? Which files should I focus on?
Any ideas would be appreciated. Thanks!

If you want to work on this then you are oon the correct track - it is more or less a copy'n'paste starting from cli.c and through management.c.
So, where ever it says encrypt / decrypt - copy and rename compress / decompress
Do you want to be assigned ?

Okay, I can try this.
However, I haven't found the exact location where the encryption function is called, so I still don't understand how it works.

cli.c -> management.c -> main.c -> aes.c

So, you need to if () / elseif() or switch in main.c and then call the correct function in the configured compression method, so

bzip2_compression.h
gzip_compression.h
lz4_compression.h
zstandard_compression.h

Okay. Thanks for your help.


Resolve test failure under FindRecordCommandTest

Codecov Report

Merging #62 (27c4857) into master (d074939) will increase coverage by 5.06%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master      #62      +/-   ##
============================================
+ Coverage     65.38%   70.44%   +5.06%
- Complexity      446      487      +41
============================================
  Files            86       86
  Lines          1537     1543       +6
  Branches        158      160       +2
============================================
+ Hits           1005     1087      +82
+ Misses          491      411      -80
- Partials         41       45       +4

Impacted Files
Coverage Δ

.../seedu/address/logic/parser/AddressBookParser.java
77.27% <0.00%> (ø)

.../java/seedu/address/storage/JsonAdaptedPerson.java
100.00% <0.00%> (+3.70%)
:arrow_up:

...rc/main/java/seedu/address/model/ModelManager.java
97.14% <0.00%> (+4.28%)
:arrow_up:

...c/main/java/seedu/address/model/person/Person.java
91.22% <0.00%> (+7.01%)
:arrow_up:

...in/java/seedu/address/model/person/RecordList.java
59.09% <0.00%> (+9.09%)
:arrow_up:

...in/java/seedu/address/logic/parser/ParserUtil.java
97.72% <0.00%> (+14.39%)
:arrow_up:

...c/main/java/seedu/address/model/person/Record.java
85.71% <0.00%> (+17.29%)
:arrow_up:

...va/seedu/address/model/person/DisplayedPerson.java
89.47% <0.00%> (+26.31%)
:arrow_up:

.../java/seedu/address/storage/JsonAdaptedRecord.java
84.61% <0.00%> (+39.16%)
:arrow_up:

...seedu/address/logic/commands/AddRecordCommand.java
86.66% <0.00%> (+86.66%)
:arrow_up:

... and 3 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


Allow customization for the routes that are tracked

The user should be able to customize which routes are tracked.

@clayne11, thanks for the submission. Could you give me an example of your usecase, and how you envision it working?

@clayne11, we don't see the usecase where you would need to actively avoid tracking specific routes. Closing this for now.


Small updates to the Simple Managed Tracer

Change constructor to take a trace id and project id instead of a trace proto
Fix a bug where a flush does not remove the already reported spans.

Thanks Mike, merging now


Array Docs and Types

This adds some docs and examples for methods that did not have some and adds some types.  I left some types empty based on my preserved usage.

@wontruefree Is there a particular reason to close this PR? Or do you just don't intend to put any more effort in it?

@wontruefree Is there a particular reason to close this PR? Or do you just don't intend to put any more effort in it?

@straight-shoota I would like to work on these Docs PRs.  There were a lot of comments and some of them I was confused on where they stood.  I figure I would just submit smaller PRs that could be reviewed more easily.

@straight-shoota I would like to work on these Docs PRs.  There were a lot of comments and some of them I was confused on where they stood.  I figure I would just submit smaller PRs that could be reviewed more easily.


fix(npm): move forge related components inside ngx-forge

Moves the following:

forge-errors
forge-exception
spinner
multiple-selection-list
single-selection-dropdown
pipeline-view

adds stylesheets

This PR tracks this user story - https://github.com/openshiftio/openshift.io/issues/1186
Once merged here, changes have to be made on Fabric8-ui to import these components

Some tests would be nice

👍 what do you think @corinnekrych

I think to we should add component testing. As areference, see:

component test example in fabric8-ui / codebase module
Bartosz did this PR using a testing tools pattern

@arunkumars08 here is a PR with my review: https://github.com/arunkumars08/ngx-forge/pull/1


[Bug] Routine load job cause dead lock

Describe the bug
See the stack of FE, all routine load thread blocks like:
"doris-mysql-nio-pool-35" #1037 daemon prio=5 os_prio=0 cpu=152.99ms elapsed=67090.75s tid=0x00007fc360003800 nid=0xac5b waiting on condition  [0x00007fc1fb7f5000]
   java.lang.Thread.State: WAITING (parking)
        at<EMAIL_ADDRESS>Method)
        - parking to wait for  <0x000000068ac10400> (a java.util.concurrent.locks.ReentrantReadWriteLock$FairSync)
        at java.util.concurrent.locks.LockSupport.park(java.base@11.0.10/LockSupport.java:194)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11.0.10/AbstractQueuedSynchronizer.java:885)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(java.base@11.0.10/AbstractQueuedSynchronizer.java:1009)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(java.base@11.0.10/AbstractQueuedSynchronizer.java:1324)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(java.base@11.0.10/ReentrantReadWriteLock.java:738)
        at org.apache.doris.load.routineload.RoutineLoadJob.readLock(RoutineLoadJob.java:387)
        at org.apache.doris.load.routineload.RoutineLoadJob.getShowInfo(RoutineLoadJob.java:1289)
        at org.apache.doris.qe.ShowExecutor.handleShowRoutineLoad(ShowExecutor.java:1185)
        at org.apache.doris.qe.ShowExecutor.execute(ShowExecutor.java:232)
        at org.apache.doris.qe.StmtExecutor.handleShow(StmtExecutor.java:1452)
        at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:388)
        at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:288)
        at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:206)
        at org.apache.doris.qe.ConnectProcessor.dispatch(ConnectProcessor.java:344)
        at org.apache.doris.qe.ConnectProcessor.processOnce(ConnectProcessor.java:545)
        at org.apache.doris.mysql.nio.ReadListener.lambda$handleEvent$0(ReadListener.java:50)
        at org.apache.doris.mysql.nio.ReadListener$$Lambda$676/0x0000000800770c40.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.10/ThreadPoolExecutor.java:1128)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.10/ThreadPoolExecutor.java:628)
        at java.lang.Thread.run(java.base@11.0.10/Thread.java:834)

This is because:
when aborting a task of a routine load job, the txn will do following step:

beforeStateTransform()
abortTxn();
afterStateTransform();

In beforeStateTransform(), it will lock the write lock of the routine load job, and in afterStateTransform(), it will be unlocked.
when locking, the routine load job object is got from TxnStateCallbackFactory via callbackId.
But the routine load job object may be removed from TxnStateCallbackFactory at step 2, eg, job being stopped.
So in step 3, we can not get routine load job object from TxnStateCallbackFactory, so that we can not release the lock of the job.

Can you show case when the callback will be remove from step 2?

Can you show the case when the callback will be remove from step 2?

Updated. In case that job being stopped.

Stop job must get the writeLock first, but step 1 has got the write lock, how could stopping job happen in step 2?

Stop job must get the writeLock first, but step 1 has got the write lock, how could stopping job happen in step 2?

I didn't notice that. Sounds reasonable, I will recheck it later.


Hacking Instructor scripts [🤖]

Add Hacking Instructor tutorial scripts for challenges suitable for step-by-step guidance and no need for interaction outside Angular (which upon leaving would cancel any tutorial) or Browser DevTools:

[ ] Forged Feedback (find hidden input for UserId, make it visible via DevTools and change value)
[ ] Login Bender (find out email from a review then login via<EMAIL_ADDRESS>[ ] Login Jim (exactly same as Bender just for Jim)
[ ] Password Strength (educate on weak passwords, advise to lookup common lists, give guessing hints)
[ ] Privacy Policy (guide towards dropdown menu item)
[ ] View Basket (ask to log in, view own basket, guide to DevTools session storage, wait for changing and view basket)

Todo for Projects Summit Q1 2020


Add nullable modifier to Eloquent casts

I can do a PR for this. Wanted to get any thoughts before creating.
I came across a situation where I was using Eloquent's casting capabilities to cast an attribute to an integer. In this situation, if the column was null in the DB, I didn't actually want an integer to be returned, as it would result in 0. 0 happened to have a different meaning than null.
This is fixable with mutators, but I thought it might be a nice addition to add a modifier, which specifies null should be returned if the value is null, otherwise, cast as defined.
My initial thought was tacking "?" on to the end of the cast name. Some languages use "?" for features that surround null, e.g. C#.
Example:
protected $casts = [
    'data' => 'array',
    'count' => 'int',
    'row' => 'int?' // will return null if the row is null, otherwise it will cast to an int
    'created_at' => 'date?' // null if null, otherwise, the date
];

Thoughts?

https://github.com/laravel/framework/blob/5.2/src/Illuminate/Database/Eloquent/Model.php#L2870-L2872

Thanks @deefour, must have missed that when reviewing.

Did the link change? I cannot see that this is implemented now?


settings.get: 'cast' and 'default' don't seem to work together

dynaconf 1.2.0 on Python 3.6.5
I have a setting that has a default and a cast:
settings.get("x.y", default=DEFAULT_X_Y, cast="@str")
Now, x.y is not defined, but DEFAULT_X_Y is.
When using the debugger, I get (> are debugger inputs)
> settings.get("x.y")
None
> settings.get("x.y", default=DEFAULT_X_Y)
(value of DEFAULT_X_Y)
> settings.get("x.y", default=DEFAULT_X_Y, cast="@str")
{TypeError}'NoneType' object is not callable

I would have expected for DEFAULT_X_Y to be casted to str

There is no @str converter https://github.com/rochacbruno/dynaconf/blob/master/dynaconf/utils/parse_conf.py#L16-L29

There is no @str because env vars are strings by default and that feature is meant to use with env vars.
I recommend you to do
settings.get("x.y", default=str(DEFAULT_X_Y))

to get the same results


Fix geojson-flatten require

The package geojson-flatten has been updated with the modulejs syntax. Need to fix the require.

@tmcw any chance this can be merged soon?

@tmcw any chance this can be merged soon?

I don't have commit or publish access on this repository, someone in the Mapbox org has to do it.

I don't have commit or publish access on this repository, someone in the Mapbox org has to do it.

@tmcw looks like there's a conflict in the package.json between this branch and master

I don't have commit or push access to this repo, this is someone at Mapbox's responsibility.

@tmcw indeed, but I think rectifying the conflict in another commit or commit --amend and force push would make it easier for the maintainer to merge

How exactly am I involved in this? I'm not the author of the PR, the maintainer of the repo, or a reviewer.

@tmcw have opened this to allow you to get a fix out ASAP: https://github.com/mapbox/geojson-coords/pull/8

How exactly am I involved in this? I'm not the author of the PR, the maintainer of the repo, or a reviewer.

Actually, Tom, are you saying you can't actually release this package at all? (you're labelled as a contributor for this repository)

Actually, Tom, are you saying you can't actually release this package at all?

Yes. I worked at Mapbox, built a lot of this stuff, and no longer work at Mapbox. I don't have commit to this repo, the ability to release a version or anything like that, because I don't work there.

@gpanneti can you resolve this merge conflict so the maintainers might have a better time reviewing and merging? thanks for this bugfix!

Thank you for your contribution @gpanneti!
Closing this in favor of @ThisIsMissEm's merge fix here.


bug: Investigate possible issue with pruning logic

What is the bug?
When a recent backfill of all messages was performed, the hubs successfully merged older messages for a test fid 2157 which had been used to test pruning logic. This was inactive for months, and we would've expected re-submitted messages to be rejected by the hubs (either as already merged or because they were immediately prunable with timestamps older than the earliest message).
Investigate what happened.
How can it be reproduced? (optional)
Maybe resubmit all messages for this fid against a local hub?
Additional context (optional)
Relevant logs:
# Filtering out LINK to remove noise from unrelated follow backfill
cat ~/.pm2/logs/hubble-out.log | grep "success.*2157 " | grep -v LINK | head  -n 10
{"level":30,"time":1686626826827,"pid":1914,"hostname":"hubble","component":"Hub","submittedMessage":{"timestamp":1686626827000,"hash":"0xacda08496d4c0922b85fba258524c64aaa1401cd","fid":12157,"type":3},"source":"gossip","msg":"submitMessage success<PHONE_NUMBER>83392: fid 12157 merged REACTION_ADD 0xacda08496d4c0922b85fba258524c64aaa1401cd"}
{"level":30,"time":1686664796716,"pid":1914,"hostname":"hubble","component":"Hub","submittedMessage":{"timestamp":1686664796000,"hash":"0xc9380b238299a93db79943413c58c75681dc9d70","fid":12157,"type":3},"source":"gossip","msg":"submitMessage success<PHONE_NUMBER>48736: fid 12157 merged REACTION_ADD 0xc9380b238299a93db79943413c58c75681dc9d70"}
{"level":30,"time":1687391893060,"pid":1914,"hostname":"hubble","component":"Hub","submittedMessage":{"timestamp":1681651669000,"hash":"0xca53005a5571faf56b612818e4bab81e047e8e39","fid":12157,"type":11},"source":"gossip","msg":"submitMessage success<PHONE_NUMBER>73760: fid 12157 merged USER_DATA_ADD 0xca53005a5571faf56b612818e4bab81e047e8e39"}
{"level":30,"time":1687391932524,"pid":1914,"hostname":"hubble","component":"Hub","submittedMessage":{"timestamp":1679095914000,"hash":"0x77ecb6fe3a42f8239c19e23aa277d2697fcaee1c","fid":2157,"type":2},"source":"gossip","msg":"submitMessage success<PHONE_NUMBER>18304: fid 2157 merged CAST_REMOVE 0x77ecb6fe3a42f8239c19e23aa277d2697fcaee1c"}
{"level":30,"time":1687393476733,"pid":1914,"hostname":"hubble","component":"Hub","submittedMessage":{"timestamp":1681651472000,"hash":"0x939e590b608cb6416cd0561a6a29c481fd349d30","fid":12157,"type":7},"source":"gossip","msg":"submitMessage success<PHONE_NUMBER>94272: fid 12157 merged VERIFICATION_ADD_ETH_ADDRESS0x939e590b608cb6416cd0561a6a29c481fd349d30"}
{"level":30,"time":1687394875941,"pid":1914,"hostname":"hubble","component":"Hub","submittedMessage":{"timestamp":1679094635000,"hash":"0xff9ec5a0197b153f89acd5ff253042a898e22ab6","fid":2157,"type":2},"source":"gossip","msg":"submitMessage success<PHONE_NUMBER>54336: fid 2157 merged CAST_REMOVE 0xff9ec5a0197b153f89acd5ff253042a898e22ab6"}
{"level":30,"time":1687396921368,"pid":1914,"hostname":"hubble","component":"Hub","submittedMessage":{"timestamp":1681651401000,"hash":"0xce917bb6e88ecb68d671bf863166498b223eaabc","fid":12157,"type":11},"source":"gossip","msg":"submitMessage success<PHONE_NUMBER>24320: fid 12157 merged USER_DATA_ADD 0xce917bb6e88ecb68d671bf863166498b223eaabc"}
{"level":30,"time":1687401812170,"pid":142975,"hostname":"hubble","component":"Hub","submittedMessage":{"timestamp":1678830664000,"hash":"0xc90e3da068ce77b76222df759319a5da8e5bc1a1","fid":2157,"type":10},"source":"sync","msg":"submitMessage success<PHONE_NUMBER>40128: fid 2157 merged SIGNER_REMOVE 0xc90e3da068ce77b76222df759319a5da8e5bc1a1"}
{"level":30,"time":1687401812381,"pid":142975,"hostname":"hubble","component":"Hub","submittedMessage":{"timestamp":1678830706000,"hash":"0xaf750c82b8729006b5b2d0b8461120461f9ddf6e","fid":2157,"type":10},"source":"sync","msg":"submitMessage success<PHONE_NUMBER>12576: fid 2157 merged SIGNER_REMOVE 0xaf750c82b8729006b5b2d0b8461120461f9ddf6e"}

# Last messages
cat ~/.pm2/logs/hubble-out.log | grep "submitMessage.*2157 " | tail -n 10
{"level":30,"time":1687419879995,"pid":142975,"hostname":"hubble","component":"Hub","submittedMessage":{"timestamp":1679678880000,"hash":"0x0cdebe81708d6144c3c360842e289ec1eb5e22f8","fid":2157,"type":1},"source":"gossip","msg":"submitMessage success<PHONE_NUMBER>55424: fid 2157 merged CAST_ADD 0x0cdebe81708d6144c3c360842e289ec1eb5e22f8"}
{"level":30,"time":1687419885124,"pid":142975,"hostname":"hubble","component":"Hub","submittedMessage":{"timestamp":1679683419000,"hash":"0x0af8ef3f208b593de70d731026629779cec7effb","fid":2157,"type":1},"source":"gossip","msg":"submitMessage success<PHONE_NUMBER>63808: fid 2157 merged CAST_ADD 0x0af8ef3f208b593de70d731026629779cec7effb"}
{"level":30,"time":1687419886800,"pid":142975,"hostname":"hubble","component":"Hub","submittedMessage":{"timestamp":1679685851000,"hash":"0x2838f76f3a5141f6d673030335ec8caef55a70be","fid":2157,"type":3},"source":"gossip","msg":"submitMessage success<PHONE_NUMBER>32800: fid 2157 merged REACTION_ADD 0x2838f76f3a5141f6d673030335ec8caef55a70be"}
{"level":30,"time":1687419888274,"pid":142975,"hostname":"hubble","component":"Hub","submittedMessage":{"timestamp":1679685801000,"hash":"0x83b56c5fed22a460140bca6e33684757ff513e55","fid":2157,"type":3},"source":"gossip","msg":"submitMessage success<PHONE_NUMBER>66208: fid 2157 merged REACTION_ADD 0x83b56c5fed22a460140bca6e33684757ff513e55"}
{"level":30,"time":1687419888279,"pid":142975,"hostname":"hubble","component":"Hub","submittedMessage":{"timestamp":1679683476000,"hash":"0x83b4fafd5aad834207229c2f18beeadcb1f5e9c5","fid":2157,"type":1},"source":"gossip","msg":"submitMessage success<PHONE_NUMBER>90784: fid 2157 merged CAST_ADD 0x83b4fafd5aad834207229c2f18beeadcb1f5e9c5"}
{"level":30,"time":1687419889229,"pid":142975,"hostname":"hubble","component":"Hub","submittedMessage":{"timestamp":1679684511000,"hash":"0x594e472a0d0302c3d50ce676bf463f3f6c52f028","fid":2157,"type":1},"source":"gossip","msg":"submitMessage success<PHONE_NUMBER>81984: fid 2157 merged CAST_ADD 0x594e472a0d0302c3d50ce676bf463f3f6c52f028"}
{"level":30,"time":1687419890395,"pid":142975,"hostname":"hubble","component":"Hub","submittedMessage":{"timestamp":1679683809000,"hash":"0x77248184ca6fb06df183444186cc7114f94aea4d","fid":2157,"type":1},"source":"gossip","msg":"submitMessage success<PHONE_NUMBER>57920: fid 2157 merged CAST_ADD 0x77248184ca6fb06df183444186cc7114f94aea4d"}
{"level":30,"time":1687419893064,"pid":142975,"hostname":"hubble","component":"Hub","submittedMessage":{"timestamp":1679677600000,"hash":"0x09dbceea17e82844588268ab7f7c9076e5f29bda","fid":2157,"type":1},"source":"gossip","msg":"submitMessage success<PHONE_NUMBER>86048: fid 2157 merged CAST_ADD 0x09dbceea17e82844588268ab7f7c9076e5f29bda"}
{"level":30,"time":1687419893260,"pid":142975,"hostname":"hubble","component":"Hub","submittedMessage":{"timestamp":1679684298000,"hash":"0x09dcc223a6e5ce7deb5afdc4c0e0f70fc5f2a629","fid":2157,"type":1},"source":"gossip","msg":"submitMessage success<PHONE_NUMBER>92960: fid 2157 merged CAST_ADD 0x09dcc223a6e5ce7deb5afdc4c0e0f70fc5f2a629"}
{"level":30,"time":1687419897285,"pid":142975,"hostname":"hubble","component":"Hub","submittedMessage":{"timestamp":1679678757000,"hash":"0x29b7af0591396363669d4f7118224dc89e26284b","fid":2157,"type":1},"source":"gossip","msg":"submitMessage success<PHONE_NUMBER>75264: fid 2157 merged CAST_ADD 0x29b7af0591396363669d4f7118224dc89e26284b"}

cat ~/.pm2/logs/hubble-out.log | grep pruned.*2157
{"level":30,"time":1686873606557,"pid":1914,"hostname":"hubble","component":"Engine","msg":"pruned 5000 reaction messages for fid 2157"}
{"level":30,"time":1687395607571,"pid":1914,"hostname":"hubble","component":"Engine","msg":"pruned 1 cast messages for fid 2157"}
{"level":30,"time":1687402806081,"pid":142975,"hostname":"hubble","component":"Engine","msg":"pruned 3647 cast messages for fid 2157"}
{"level":30,"time":1687406404635,"pid":142975,"hostname":"hubble","component":"Engine","msg":"pruned 1246 cast messages for fid 2157"}
{"level":30,"time":1687410004076,"pid":142975,"hostname":"hubble","component":"Engine","msg":"pruned 1252 cast messages for fid 2157"}
{"level":30,"time":1687413605271,"pid":142975,"hostname":"hubble","component":"Engine","msg":"pruned 2270 cast messages for fid 2157"}
{"level":30,"time":1687417204615,"pid":142975,"hostname":"hubble","component":"Engine","msg":"pruned 1493 cast messages for fid 2157"}
{"level":30,"time":1687420803788,"pid":142975,"hostname":"hubble","component":"Engine","msg":"pruned 995 cast messages for fid 2157"}
{"level":30,"time":1687464005473,"pid":142975,"hostname":"hubble","component":"Engine","msg":"pruned 4414 reaction messages for fid 2157"}
{"level":30,"time":1687467603210,"pid":142975,"hostname":"hubble","component":"Engine","msg":"pruned 180 reaction messages for fid 2157"}

@sanjayprabhu thinks this is a function of the bug where we had incorrect set sizes, and is no longer an issue.


Clean up configuration dependencies

This updates the configuration dependencies to be more accurate to what's actually required by each component. This will prevent us from restarting services when it's not necessary. Currently, we have to restart Router on a Hive configuration change.

LGTM


Missing PetG-HF

Looks like the available filaments need to be updated.
Im using relatively new Bambu PetG Basic, and it shows as unknown in bambu-farm with latest release.
Shows as PetG in bambu app.
Im also getting a printer error, when status's look clean, and last print finished normally. (Could just be the unknown filament)

Will work on doing a PR for additional Filament types.