bad zip file dbNSFPv2.9.zip

Hi, wondering is there any way i fix the following problem?
Archive:  dbNSFPv2.9.zip
error: End-of-centdir-64 signature not where expected (prepended bytes?)
(attempting to process anyway)
warning [dbNSFPv2.9.zip]:<PHONE_NUMBER> extra bytes at beginning or within zipfile
(attempting to process anyway)
skipping: dbNSFP2.9_variant.chr1  need PK compat. v4.5 (can do v2.1)
skipping: dbNSFP2.9_variant.chr2  need PK compat. v4.5 (can do v2.1)
skipping: dbNSFP2.9_variant.chr20  need PK compat. v4.5 (can do v2.1)
skipping: dbNSFP2.9_variant.chr21  need PK compat. v4.5 (can do v2.1)
skipping: dbNSFP2.9_variant.chr22  need PK compat. v4.5 (can do v2.1)
skipping: dbNSFP2.9_variant.chr3  need PK compat. v4.5 (can do v2.1)
skipping: dbNSFP2.9_variant.chr4  need PK compat. v4.5 (can do v2.1)
skipping: dbNSFP2.9_variant.chr5  need PK compat. v4.5 (can do v2.1)
skipping: dbNSFP2.9_variant.chr6  need PK compat. v4.5 (can do v2.1)
skipping: dbNSFP2.9_variant.chr7  need PK compat. v4.5 (can do v2.1)
skipping: dbNSFP2.9_variant.chr8  need PK compat. v4.5 (can do v2.1)
skipping: dbNSFP2.9_variant.chr9  need PK compat. v4.5 (can do v2.1)
skipping: dbNSFP2.9_variant.chrX  need PK compat. v4.5 (can do v2.1)
skipping: dbNSFP2.9_variant.chrY  need PK compat. v4.5 (can do v2.1)
skipping: LICENSE.txt             need PK compat. v4.5 (can do v2.1)
skipping: search_dbNSFP29.class   need PK compat. v4.5 (can do v2.1)
skipping: search_dbNSFP29.java    need PK compat. v4.5 (can do v2.1)
skipping: search_dbNSFP29.readme.pdf  need PK compat. v4.5 (can do v2.1)
skipping: try.vcf                 need PK compat. v4.5 (can do v2.1)
skipping: tryhg18.in              need PK compat. v4.5 (can do v2.1)
skipping: tryhg19.in              need PK compat. v4.5 (can do v2.1)
skipping: tryhg38.in              need PK compat. v4.5 (can do v2.1)
file #1:  bad zipfile offset (local header sig):<PHONE_NUMBER>
(attempting to re-compensate)
replace dbNSFPv2.9/dbNSFP2.9.readme.txt? [y]es, [n]o, [A]ll, [N]one, [r]ename:  NULL
(EOF or read error, treating as "[N]one" ...)
Fatal error: local() encountered an error (return code 2) while executing 'unzip dbNSFPv2.9.zip -d dbNSFPv2.9'

Apologies about the issue. I bumped the download to the latest dbNSFP release and it should hopefully now download and install cleanly. When you restart remove the cached cloudbiolinux (rm -rf tmpbcbio-install) and it will grab the latest version. Hope this gets everything installed for you.

thank you for the quick reply and the solution.
I restarted the installation, and it works....(?!)
the confusions is I still get the same err msg yet followed by the installation complete  msg
don't know if I successfully installed the dbNSFP this time, should I update my bcbio_nextgen.py script and restart again?

forget to paste the installation log....
2015-07-01 09:33:36 (1000 KB/s) - “dbNSFPv3.0b2c.zip” saved [6281431546]
Archive:  /opt/pipelines/bcbio/genomes/Hsapiens/GRCh37/variation/dbNSFPv3.0b2c.zip
error: End-of-centdir-64 signature not where expected (prepended bytes?)
(attempting to process anyway)
warning [/opt/pipelines/bcbio/genomes/Hsapiens/GRCh37/variation/dbNSFPv3.0b2c.zip]:<PHONE_NUMBER> extra bytes at beginning or within zipfile
(attempting to process anyway)
skipping: dbNSFP3.0b2c_variant.chr1  need PK compat. v4.5 (can do v2.1)
skipping: dbNSFP3.0b2c_variant.chr4  need PK compat. v4.5 (can do v2.1)
skipping: dbNSFP3.0b2c_variant.chr5  need PK compat. v4.5 (can do v2.1)
skipping: dbNSFP3.0b2c_variant.chr6  need PK compat. v4.5 (can do v2.1)
skipping: dbNSFP3.0b2c_variant.chr7  need PK compat. v4.5 (can do v2.1)
skipping: dbNSFP3.0b2c_variant.chr8  need PK compat. v4.5 (can do v2.1)
skipping: dbNSFP3.0b2c_variant.chr9  need PK compat. v4.5 (can do v2.1)
skipping: dbNSFP3.0b2c_variant.chrM  need PK compat. v4.5 (can do v2.1)
skipping: dbNSFP3.0b2c_variant.chrX  need PK compat. v4.5 (can do v2.1)
skipping: dbNSFP3.0b2c_variant.chrY  need PK compat. v4.5 (can do v2.1)
skipping: dbNSFP3.0b2_gene        need PK compat. v4.5 (can do v2.1)
skipping: dbNSFP3.0b2_gene.complete  need PK compat. v4.5 (can do v2.1)
skipping: search_dbNSFP30b2c.class  need PK compat. v4.5 (can do v2.1)
skipping: search_dbNSFP30b2c.readme.pdf  need PK compat. v4.5 (can do v2.1)
skipping: try.vcf                 need PK compat. v4.5 (can do v2.1)
skipping: tryhg18.in              need PK compat. v4.5 (can do v2.1)
skipping: tryhg19.in              need PK compat. v4.5 (can do v2.1)
skipping: tryhg38.in              need PK compat. v4.5 (can do v2.1)
file #1:  bad zipfile offset (local header sig):<PHONE_NUMBER>
(attempting to re-compensate)
inflating: dbNSFPv3.0b2c/dbNSFP3.0b2c.readme.txt
inflating: dbNSFPv3.0b2c/dbNSFP3.0b2c_variant.chr10
inflating: dbNSFPv3.0b2c/dbNSFP3.0b2c_variant.chr11
inflating: dbNSFPv3.0b2c/dbNSFP3.0b2c_variant.chr12
inflating: dbNSFPv3.0b2c/dbNSFP3.0b2c_variant.chr13
inflating: dbNSFPv3.0b2c/dbNSFP3.0b2c_variant.chr14
inflating: dbNSFPv3.0b2c/dbNSFP3.0b2c_variant.chr15
inflating: dbNSFPv3.0b2c/dbNSFP3.0b2c_variant.chr16
inflating: dbNSFPv3.0b2c/dbNSFP3.0b2c_variant.chr17
inflating: dbNSFPv3.0b2c/dbNSFP3.0b2c_variant.chr18
inflating: dbNSFPv3.0b2c/dbNSFP3.0b2c_variant.chr19
Fatal error: local() encountered an error (return code 2) while executing 'unzip /opt/pipelines/bcbio/genomes/Hsapiens/GRCh37/variation/dbNSFPv3.0b2c.zip -d dbNSFPv3.0b2c'
Aborting.
nux/install_packages.R)"
[localhost] local: echo '
bioc.pkgs <- c("BiocInstaller", "BubbleTree", "cn.mops", "DEXSeq", "DNAcopy", "GenomicRanges", "IRanges", "rtracklayer")
bioc.installer = repo.installer(biocinstallRepos(), biocLite, NULL)
lapply(bioc.pkgs, bioc.installer)
' >> "$(echo /root/tmp/cloudbiolinux/install_packages.R)"
[localhost] local: echo '
std2.pkgs <- c("PSCBS")
lapply(std2.pkgs, std.installer)
' >> "$(echo /root/tmp/cloudbiolinux/install_packages.R)"
[localhost] local: Rscript /root/tmp/cloudbiolinux/install_packages.R
[localhost] local: rm -f /root/tmp/cloudbiolinux/install_packages.R
[localhost] local: rm -rf /root/tmp/cloudbiolinux
[localhost] local: rm -rf .cpanm
[localhost] local: rm -f /var/crash/*
[localhost] local: rm -f ~/.dot
[localhost] local: rm -f ~/.log
Creating manifest of installed packages in /opt/pipelines/bcbio/manifest
Third party tools upgrade complete.
Installing additional tools
Upgrading bcbio-nextgen data files
Setting up virtual machine
[localhost] local: echo $HOME
[localhost] local: uname -m
[localhost] local: pwd
[localhost] local: mkdir -p '/opt/pipelines/bcbio/genomes/Hsapiens/GRCh37/variation/9e5abf75-1cb7-3e48-84fd-7817f48269a1'
[localhost] local: wget --continue --no-check-certificate -O dbNSFPv3.0b2c.zip<EMAIL_ADDRESS>[localhost] local: mv dbNSFPv3.0b2c.zip /opt/pipelines/bcbio/genomes/Hsapiens/GRCh37/variation
[localhost] local: rm -rf /opt/pipelines/bcbio/genomes/Hsapiens/GRCh37/variation/9e5abf75-1cb7-3e48-84fd-7817f48269a1
[localhost] local: mkdir -p dbNSFPv3.0b2c
[localhost] local: unzip /opt/pipelines/bcbio/genomes/Hsapiens/GRCh37/variation/dbNSFPv3.0b2c.zip -d dbNSFPv3.0b2c

Looks like I'm encountering the same error...
2015-07-01 16:44:25 (994 KB/s) - “dbNSFPv3.0b2c.zip” saved [6281431546]

Archive:  /packages/bcbio/v0.8.8/bcbio/genomes/Hsapiens/GRCh37/variation/dbNSFPv3.0b2c.zip
error: End-of-centdir-64 signature not where expected (prepended bytes?)
  (attempting to process anyway)
warning [/packages/bcbio/v0.8.8/bcbio/genomes/Hsapiens/GRCh37/variation/dbNSFPv3.0b2c.zip]:<PHONE_NUMBER> extra bytes at beginning or within zipfile
  (attempting to process anyway)
   skipping: dbNSFP3.0b2c_variant.chr1  need PK compat. v4.5 (can do v2.1)
   skipping: dbNSFP3.0b2c_variant.chr4  need PK compat. v4.5 (can do v2.1)
   skipping: dbNSFP3.0b2c_variant.chr5  need PK compat. v4.5 (can do v2.1)
   skipping: dbNSFP3.0b2c_variant.chr6  need PK compat. v4.5 (can do v2.1)
   skipping: dbNSFP3.0b2c_variant.chr7  need PK compat. v4.5 (can do v2.1)
   skipping: dbNSFP3.0b2c_variant.chr8  need PK compat. v4.5 (can do v2.1)
   skipping: dbNSFP3.0b2c_variant.chr9  need PK compat. v4.5 (can do v2.1)
   skipping: dbNSFP3.0b2c_variant.chrM  need PK compat. v4.5 (can do v2.1)
   skipping: dbNSFP3.0b2c_variant.chrX  need PK compat. v4.5 (can do v2.1)
   skipping: dbNSFP3.0b2c_variant.chrY  need PK compat. v4.5 (can do v2.1)
   skipping: dbNSFP3.0b2_gene        need PK compat. v4.5 (can do v2.1)
   skipping: dbNSFP3.0b2_gene.complete  need PK compat. v4.5 (can do v2.1)
   skipping: search_dbNSFP30b2c.class  need PK compat. v4.5 (can do v2.1)
   skipping: search_dbNSFP30b2c.readme.pdf  need PK compat. v4.5 (can do v2.1)
   skipping: try.vcf                 need PK compat. v4.5 (can do v2.1)
   skipping: tryhg18.in              need PK compat. v4.5 (can do v2.1)
   skipping: tryhg19.in              need PK compat. v4.5 (can do v2.1)
   skipping: tryhg38.in              need PK compat. v4.5 (can do v2.1)
file #1:  bad zipfile offset (local header sig):<PHONE_NUMBER>
  (attempting to re-compensate)
  inflating: dbNSFPv3.0b2c/dbNSFP3.0b2c.readme.txt
  inflating: dbNSFPv3.0b2c/dbNSFP3.0b2c_variant.chr10
  inflating: dbNSFPv3.0b2c/dbNSFP3.0b2c_variant.chr11
  inflating: dbNSFPv3.0b2c/dbNSFP3.0b2c_variant.chr12
  inflating: dbNSFPv3.0b2c/dbNSFP3.0b2c_variant.chr13
  inflating: dbNSFPv3.0b2c/dbNSFP3.0b2c_variant.chr14
  inflating: dbNSFPv3.0b2c/dbNSFP3.0b2c_variant.chr15
  inflating: dbNSFPv3.0b2c/dbNSFP3.0b2c_variant.chr16
  inflating: dbNSFPv3.0b2c/dbNSFP3.0b2c_variant.chr17
  inflating: dbNSFPv3.0b2c/dbNSFP3.0b2c_variant.chr18
  inflating: dbNSFPv3.0b2c/dbNSFP3.0b2c_variant.chr19
  inflating: dbNSFPv3.0b2c/dbNSFP3.0b2c_variant.chr2
  inflating: dbNSFPv3.0b2c/dbNSFP3.0b2c_variant.chr20
  inflating: dbNSFPv3.0b2c/dbNSFP3.0b2c_variant.chr21
  inflating: dbNSFPv3.0b2c/dbNSFP3.0b2c_variant.chr22
  inflating: dbNSFPv3.0b2c/dbNSFP3.0b2c_variant.chr3

Fatal error: local() encountered an error (return code 2) while executing 'unzip /packages/bcbio/v0.8.8/bcbio/genomes/Hsapiens/GRCh37/variation/dbNSFPv3.0b2c.zip -d dbNSFPv3.0b2c'

Aborting.
ges/bcbio/v0.8.8/tooldir/bin/brew info rna-star
[localhost] local: /packages/bcbio/v0.8.8/tooldir/bin/brew info tophat-binary
[localhost] local: /packages/bcbio/v0.8.8/tooldir/bin/brew info cmake
[localhost] local: /packages/bcbio/v0.8.8/tooldir/bin/brew info pkg-config
[localhost] local: /packages/bcbio/v0.8.8/tooldir/bin/brew link --overwrite pkg-config
[localhost] local: /packages/bcbio/v0.8.8/tooldir/bin/brew info s3gof3r
[localhost] local: /packages/bcbio/v0.8.8/tooldir/bin/brew info cpanminus
[localhost] local: /packages/bcbio/v0.8.8/bcbio/anaconda/bin/conda install -y -c https://conda.binstar.org/bcbio phylowgs pyloh
[localhost] local: mkdir -p '/scratch/mdeboth/tmp/cloudbiolinux'
[localhost] local: touch /scratch/mdeboth/tmp/cloudbiolinux/install_packages.R
[localhost] local: mkdir -p /packages/bcbio/v0.8.8/tooldir/lib/R/site-library
[localhost] local: chown -R mdeboth /packages/bcbio/v0.8.8/tooldir/lib/R/site-library
[localhost] local: echo '
    .libPaths(c("/packages/bcbio/v0.8.8/tooldir/lib/R/site-library"))
    library(methods)
    cran.repos <- getOption("repos")
    cran.repos["CRAN" ] <- "http://cran.fhcrc.org/"
    options(repos=cran.repos)
    source("http://bioconductor.org/biocLite.R")
    ' >> "$(echo /scratch/mdeboth/tmp/cloudbiolinux/install_packages.R)"
[localhost] local: echo '
    repo.installer <- function(repos, install.fn, pkg_name_fn) {

        update.packages(lib.loc="/packages/bcbio/v0.8.8/tooldir/lib/R/site-library", repos=repos, ask=FALSE)

      maybe.install <- function(pname) {
        check_name <- ifelse(is.null(pkg_name_fn), pname, pkg_name_fn(pname))
        if (!(is.element(check_name, installed.packages()[,1])))
          install.fn(pname)
      }
    }
    ' >> "$(echo /scratch/mdeboth/tmp/cloudbiolinux/install_packages.R)"
[localhost] local: echo '
    std.pkgs <- c("cghFLasso", "devtools", "ggplot2", "gsalib", "matrixStats", "snow", "RColorBrewer")
    std.installer = repo.installer(cran.repos, install.packages, NULL)
    lapply(std.pkgs, std.installer)
    ' >> "$(echo /scratch/mdeboth/tmp/cloudbiolinux/install_packages.R)"
[localhost] local: echo '
        bioc.pkgs <- c("BiocInstaller", "BubbleTree", "cn.mops", "DEXSeq", "DNAcopy", "GenomicRanges", "IRanges", "rtracklayer")
        bioc.installer = repo.installer(biocinstallRepos(), biocLite, NULL)
        lapply(bioc.pkgs, bioc.installer)
        ' >> "$(echo /scratch/mdeboth/tmp/cloudbiolinux/install_packages.R)"
[localhost] local: echo '
        std2.pkgs <- c("PSCBS")
        lapply(std2.pkgs, std.installer)
        ' >> "$(echo /scratch/mdeboth/tmp/cloudbiolinux/install_packages.R)"
[localhost] local: Rscript /scratch/mdeboth/tmp/cloudbiolinux/install_packages.R
[localhost] local: rm -f /scratch/mdeboth/tmp/cloudbiolinux/install_packages.R
[localhost] local: rm -rf /scratch/mdeboth/tmp/cloudbiolinux
[localhost] local: rm -rf .cpanm
[localhost] local: rm -f /var/crash/*
[localhost] local: rm -f ~/*.dot
[localhost] local: rm -f ~/*.log
Creating manifest of installed packages in /packages/bcbio/v0.8.8/bcbio/manifest
Third party tools upgrade complete.
Installing additional tools
Upgrading bcbio-nextgen data files
Setting up virtual machine
[localhost] local: echo $HOME
[localhost] local: uname -m
[localhost] local: pwd
[localhost] local: mkdir -p '/packages/bcbio/v0.8.8/bcbio/genomes/Hsapiens/GRCh37/variation/3b1a9efa-2984-30b4-a807-d646ab58eb2e'
[localhost] local: wget --continue --no-check-certificate -O dbNSFPv3.0b2c.zip<EMAIL_ADDRESS>[localhost] local: mv dbNSFPv3.0b2c.zip /packages/bcbio/v0.8.8/bcbio/genomes/Hsapiens/GRCh37/variation
[localhost] local: rm -rf /packages/bcbio/v0.8.8/bcbio/genomes/Hsapiens/GRCh37/variation/3b1a9efa-2984-30b4-a807-d646ab58eb2e
[localhost] local: mkdir -p dbNSFPv3.0b2c
[localhost] local: unzip /packages/bcbio/v0.8.8/bcbio/genomes/Hsapiens/GRCh37/variation/dbNSFPv3.0b2c.zip -d dbNSFPv3.0b2c
local() encountered an error (return code 2) while executing 'unzip /packages/bcbio/v0.8.8/bcbio/genomes/Hsapiens/GRCh37/variation/dbNSFPv3.0b2c.zip -d dbNSFPv3.0b2c'

Sorry about the continued problems. It looks like some older versions of unzip can't correctly handle the new dbNSFP archives, which is causing this error. I pushed a fix to install and use p7zip, which appears to cleanly handle the new zip files. If you remove the temporary cloudbiolinux directory (rm -rf tmpbcbio-install) and previous partial installation (rm -f /path/to/bcbio/genomes/Hsapiens/GRCh37/variation/dbNSFP_v3.0b2c.gz*) and re-run, it should hopefully work cleanly now. Thanks for following up and testing this and hope it works for you both now.

Thanks, Brad. Works for me!

I think the dbNSFP problem solved.
don't know if it's ok to post here, I think I encountered python2/python3 issue this time(?!)
(btw, I use python3.4 for bcbio_nextgen.py and python2.7 for bcbio_nextgen.py)
here's the error msg after I remove and rerun dbNSFP installation.

downloading https://s3.amazonaws.com/gemini-annotations/whole_genome_SNVs.tsv.compressed.gz to /opt/bcbio/gemini_da

Curl failed with non-zero exit code 18. Retrying
Curl failed with non-zero exit code 56. Retrying
Curl failed with non-zero exit code 56. Retrying
Traceback (most recent call last):
File "/opt/bcbio/anaconda/lib/python2.7/site-packages/gemini/install-data.py", line 179, in
install_annotation_files(args.anno_dir, args.dl_files, args.extra)
File "/opt/bcbio/anaconda/lib/python2.7/site-packages/gemini/install-data.py", line 101, in install_annotation_file
to_dl, anno_dir, cur_config)
File "/opt/bcbio/anaconda/lib/python2.7/site-packages/gemini/install-data.py", line 133, in _download_anno_files
cur_config.get("versions", {}).get(orig, 1))
File "/opt/bcbio/anaconda/lib/python2.7/site-packages/gemini/install-data.py", line 160, in _download_to_dir
raise ValueError("Failed to download with curl")
ValueError: Failed to download with curl
Traceback (most recent call last):
File "/opt/bcbio/anaconda/bin/gemini", line 6, in
gemini.gemini_main.main()
File "/opt/bcbio/anaconda/lib/python2.7/site-packages/gemini/gemini_main.py", line 1121, in main
args.func(parser, args)
File "/opt/bcbio/anaconda/lib/python2.7/site-packages/gemini/gemini_main.py", line 996, in update_fn
gemini_update.release(parser, args)
File "/opt/bcbio/anaconda/lib/python2.7/site-packages/gemini/gemini_update.py", line 68, in release
subprocess.check_call([sys.executable, _get_install_script(), config["annotation_dir"]] + extra_args)
File "/opt/bcbio/anaconda/lib/python2.7/subprocess.py", line 540, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/opt/bcbio/anaconda/bin/python', '/opt/bcbio/anaconda/lib/python2.7/site-pa
[localhost] local: touch /root/tmp/cloudbiolinux/install_packages.R
[localhost] local: mkdir -p /opt/bcbio/tools/lib/R/site-library
[localhost] local: chown -R root /opt/bcbio/tools/lib/R/site-library
[localhost] local: echo '
.libPaths(c("/opt/bcbio/tools/lib/R/site-library"))
library(methods)
cran.repos <- getOption("repos")
cran.repos["CRAN" ] <- "http://cran.fhcrc.org/"
options(repos=cran.repos)
source("http://bioconductor.org/biocLite.R")
' >> "$(echo /root/tmp/cloudbiolinux/install_packages.R)"
[localhost] local: echo '
repo.installer <- function(repos, install.fn, pkg_name_fn) {
    update.packages(lib.loc="/opt/bcbio/tools/lib/R/site-library", repos=repos, ask=FALSE)

  maybe.install <- function(pname) {

check_name <- ifelse(is.null(pkg_name_fn), pname, pkg_name_fn(pname))
if (!(is.element(check_name, installed.packages()[,1])))
install.fn(pname)
}
}
' >> "$(echo /root/tmp/cloudbiolinux/install_packages.R)"
[localhost] local: echo '
std.pkgs <- c("cghFLasso", "devtools", "ggplot2", "gsalib", "matrixStats", "snow", "RColorBrewer")
std.installer = repo.installer(cran.repos, install.packages, NULL)
lapply(std.pkgs, std.installer)
' >> "$(echo /root/tmp/cloudbiolinux/install_packages.R)"
[localhost] local: echo '
bioc.pkgs <- c("BiocInstaller", "BubbleTree", "cn.mops", "DEXSeq", "DNAcopy", "GenomicRanges", "IRanges", "rt
bioc.installer = repo.installer(biocinstallRepos(), biocLite, NULL)
lapply(bioc.pkgs, bioc.installer)
' >> "$(echo /root/tmp/cloudbiolinux/install_packages.R)"
[localhost] local: echo '
std2.pkgs <- c("PSCBS")
lapply(std2.pkgs, std.installer)
' >> "$(echo /root/tmp/cloudbiolinux/install_packages.R)"
[localhost] local: Rscript /root/tmp/cloudbiolinux/install_packages.R
[localhost] local: rm -f /root/tmp/cloudbiolinux/install_packages.R
[localhost] local: rm -rf /root/tmp/cloudbiolinux
[localhost] local: rm -rf .cpanm
[root@Rome bcbio]# htop
[root@Rome bcbio]# less -S bcbio.toolplus.install.cadd.dbnsfp.log
[localhost] local: rm -f ~/.dot
[localhost] local: rm -f ~/.log
Creating manifest of installed packages in /opt/bcbio/manifest
Third party tools upgrade complete.
Installing additional tools
Upgrading bcbio-nextgen data files
Setting up virtual machine
[localhost] local: echo $HOME
[localhost] local: uname -m
[localhost] local: pwd
[localhost] local: mkdir -p '/opt/bcbio/genomes/Hsapiens/GRCh37/variation/9e5abf75-1cb7-3e48-84fd-7817f48269a1'
[localhost] local: wget --continue --no-check-certificate -O dbNSFPv3.0b2c.zip<EMAIL_ADDRESS>[localhost] local: mv dbNSFPv3.0b2c.zip /opt/bcbio/genomes/Hsapiens/GRCh37/variation
[localhost] local: rm -rf /opt/bcbio/genomes/Hsapiens/GRCh37/variation/9e5abf75-1cb7-3e48-84fd-7817f48269a1
[localhost] local: mkdir -p dbNSFPv3.0b2c
[localhost] local: 7za x /opt/bcbio/genomes/Hsapiens/GRCh37/variation/dbNSFPv3.0b2c.zip -y -odbNSFPv3.0b2c
[localhost] local: cat dbNSFPv3.0b2c/dbNSFP*_variant.chr* | bgzip -c > dbNSFP_v3.0b2c.gz
[localhost] local: rm -f dbNSFPv3.0b2c/* && rmdir dbNSFPv3.0b2c
[localhost] local: rm -f /opt/bcbio/genomes/Hsapiens/GRCh37/variation/dbNSFPv3.0b2c.zip
[localhost] local: tabix -s 1 -b 2 -e 2 -c '#' dbNSFP_v3.0b2c.gz
Traceback (most recent call last):
File "./tools/bin/bcbio_nextgen.py", line 207, in
install.upgrade_bcbio(kwargs["args"])
File "/opt/bcbio/anaconda/lib/python2.7/site-packages/bcbio/install.py", line 94, in upgrade_bcbio
upgrade_bcbio_data(args, REMOTES)
:

Glad to hear dbNSFP is now working cleanly. It looks like your reported issues is from a failure to download the GEMINI CADD file:
* downloading https://s3.amazonaws.com/gemini-annotations/whole_genome_SNVs.tsv.compressed.gz to /opt/bcbio/gemini_da

Curl failed with non-zero exit code 18. Retrying

This is a huge file (35Gb) so it's likely timing out or having other issues. CADD files are optional since they have such a big download. You'll need to retry or manually download depending on your network connection.
bcbio will only work with Python 2.7 (although the installer should be fine with Python 3.x as well) but that doesn't influence your actual problem here.
Hope this helps.

thank you so much for the quick reply, as always!
quite a relieve to heard that timing out issue (we do have internet problem....)
Other than that, should I rm -r tmpbcbio-install, would it be also remove the installed dbNSFP?
Or would the analysis pipeline be affected?
I am asking that because I have an "samblaster: Unrecognized option: -M" problem now that I didn't see before with the same input files.

The latest release of bcbio handles the new samblaster and should only be passing the -M argument when you have that version:
https://github.com/chapmanb/bcbio-nextgen/blob/fd0bcdeaef2eea59825dcfa3bdfcc8c2561a4114/bcbio/ngsalign/postalign.py#L82
When you run samblaster --version and which samblaster, are you finding version 0.1.22 installed by bcbio?
Right now there appears to be something wrong with your installation where bcbio thinks you have the most recent version but you're getting an older version on your PATH.
Hope this helps.

Oops, I do have a older version on my PATH, already fix this. Big thanks!


TagBot trigger issue

This issue is used to trigger TagBot; feel free to unsubscribe.
If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.
If you'd like for me to do this for you, comment TagBot fix on this issue.
I'll open a PR within a few hours, please be patient!

Triggering TagBot for merged registry pull request: https://github.com/JuliaRegistries/General/pull/49471

Triggering TagBot for merged registry pull request: https://github.com/JuliaRegistries/General/pull/49646

Triggering TagBot for merged registry pull request: https://github.com/JuliaRegistries/General/pull/54682

Triggering TagBot for merged registry pull request: https://github.com/JuliaRegistries/General/pull/54972

Triggering TagBot for merged registry pull request: https://github.com/JuliaRegistries/General/pull/56598


bug(cem-expanded-types): Imported type checker does not account for unions

If I have an imported type MapView:
import type MapView from "@arcgis/core/views/MapView";

and a property in my web component class like this:
  @property() view?: MapView;

The cem-expanded-types library does incorrect comparison here:
https://github.com/break-stuff/cem-tools/blob/691a720dc392b3bb22a3f1661a0cfa9b61d0ca2f/packages/expanded-types/src/cem-plugin.ts#L263
The item.type?.text is MapView | undefined, but i.name is MapView - the comparison fails to account for a union.

I'm not sure I understand. What are you expecting here?

I am sorry, I might be misunderstanding what cem-expanded-types is trying to do, but my assumption was that given this code:
// types.ts
export type SomeUnion = "a" | "b";

// my-component.ts
import type { SomeUnion } from './types';
export class MyComponent extends LitElement {
  @property() someUnion?: SomeUnion;
}

in custom-elements.json I would see something like this:
  "type": {
    "text": "SomeUnion | undefined"
  },
  "expandedType": {
    "text": "'a' | 'b' | undefined"
  }

These should be working fine. I think the issue is that types are in an NPM package and the tool currently does not resolve types in packages.

Duplicate of #101


Fixes silent fail in bigtableio.py by logging ImportError

Log module import error. Avoid silent fail.
Doesn't throw except since existing implementations might not be ready to handle.
R: @pabloem
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[x] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

Could you fix the lint issues?

LGTM. I'll merge once we have PreCommits passing


Page not longer using this script

Hi Dustan, sadly I've redesigned my personal site a while ago and it's not longer using this script. I've seen it's present in the README (jeremenichelli.github.io/site) so I think you should remove it. If you want I can help you build an example page.
Cheers!

Thanks @jeremenichelli. Removed the In The Wild section from the readme.


Babel Loader plugin options not recognised in happypack

Hi @amireh ,
I love using your plugin, its made our build times so much faster! I've just had to update our webpack build to use css-modles-transform which is a babel plugin.
Without using Happypack, it works fine.
The error I'm getting is:
Error: /packages/ctm-app/server/server.js: Configuration 'append[0]' is not a function or a valid module path
This is a css-modules-transform error being thrown, but it only happens when its inside happypack.
HappyPack must be somehow ignoring my require('postcss-next').bind part, and thus not passing it to the babel plugin. Or it's not parsing it correctly or something? I'm not sure the internal workings.
Would there be a way to get this working at all do you think?
This is my HappyPack plugin:
        new HappyPack({
            verbose: debugMode,
            debug: debugMode,
            cache: true,
            cacheContext: {
                env: process.env.NODE_ENV
            },
            id: 'ctmJSX',
            threadPool: happyThreadPool,
            loaders: [{
                loader: 'babel-loader',
                query: {
                    cacheDirectory: true,
                    plugins: [
                        [
                            /**
                             * Css modules transform is a wrapper of css-modules-require-hook, and is run as a babel plugin.
                             *
                             * @see https://github.com/michalkvasnicak/babel-plugin-css-modules-transform
                             * @see https://github.com/css-modules/css-modules-require-hook/
                             */
                            "css-modules-transform", {
                            /**
                             * Short alias for the postcss-modules-scope plugin's option.
                             * Helps you to specify the custom way to build generic names for the class selectors.
                             * You may also use a string pattern similar to the webpack's css-loader.
                             */
                            "generateScopedName": "[hash:base64:5]",
                            /**
                             * Appends custom plugins to the end of the PostCSS pipeline.
                             * Since the require function is synchronous, you should provide synchronous plugins only.
                             */
                            "append": [
                                require('postcss-cssnext').bind(this, {
                                    browsers: ['last 2 versions', 'IE > 10', 'Chrome 44']
                                }),
                                require('postcss-reporter').bind(this, { clearMessages: true })
                            ]
                        }
                        ]
                    ]
                }
            }]
        }),

Thanks!
For clarity, this works:
    module: {
        loaders: [
            {
                test: /\.js$|\.jsx$/,
                exclude: /node_modules/,
                loader: 'babel-loader',
                query: {
                    cacheDirectory: true,
                    plugins: [
                        [
                            "css-modules-transform", {
                            "generateScopedName": "[hash:base64:5]",
                            "append": [
                                cssnext.bind(this, {
                                    browsers: ['last 2 versions', 'IE > 10', 'Chrome 44']
                                }),
                                postcssReporter.bind(this, {clearMessages: true})
                            ]
                        }
                        ]
                    ]
                }
            }
}

package json dependencies:
 "dependencies": {
    "animated-scroll-to": "1.0.1",
    "axios": "0.15.3",
    "babel-plugin-transform-runtime": "6.15.0",
    "classnames": "2.2.5",
    "cross-env": "3.1.4",
    "cuid": "1.3.8",
    "deep-equal": "1.0.1",
    "express": "4.14.0",
    "express-subdomain": "1.0.5",
    "express-winston": "2.1.2",
    "immutable": "3.8.1",
    "intl": "1.2.5",
    "isomorphic-fetch": "2.2.1",
    "jsep": "0.3.0",
    "moment": "2.17.1",
    "react": "15.4.1",
    "react-addons-transition-group": "15.4.2",
    "react-autosuggest": "7.0.1",
    "react-dom": "15.4.1",
    "react-helmet": "3.3.0",
    "react-intl": "2.2.2",
    "react-onclickoutside": "5.8.3",
    "react-redux": "5.0.1",
    "react-router": "3.0.0",
    "react-text-mask": "0.19.0",
    "redux": "3.6.0",
    "redux-logger": "2.7.4",
    "redux-promise": "0.5.3",
    "redux-responsive": "4.1.1",
    "redux-saga": "0.13.0",
    "redux-thunk": "2.1.0",
    "reselect": "2.5.4",
    "sanitize-html": "1.13.0",
    "scroll-into-view": "1.7.1",
    "tether": "1.4.0",
    "validator": "6.2.0",
    "webpack": "2.2.1",
    "winston": "2.3.0",
    "winston-daily-rotate-file": "1.4.0"
  },
  "devDependencies": {
    "babel-core": "6.21.0",
    "babel-eslint": "7.1.1",
    "babel-loader": "6.2.10",
    "babel-plugin-css-modules-transform": "1.1.0",
    "babel-plugin-transform-class-properties": "^6.22.0",
    "babel-plugin-transform-decorators-legacy": "1.3.4",
    "babel-plugin-transform-es2015-modules-commonjs-optimized": "6.16.0",
    "babel-plugin-transform-react-remove-prop-types": "0.2.11",
    "babel-plugin-transform-react-router-optimize": "1.0.1",
    "babel-polyfill": "6.20.0",
    "babel-preset-es2015": "6.18.0",
    "babel-preset-react": "6.16.0",
    "babel-preset-react-optimize": "1.0.1",
    "babel-preset-stage-0": "6.16.0",
    "babel-react-render-defender": "1.1.1",
    "babel-register": "6.18.0",
    "chai": "3.5.0",
    "chunk-manifest-webpack-plugin": "1.0.0",
    "copy-webpack-plugin": "4.0.1",
    "css-loader": "0.26.1",
    "enzyme": "2.7.0",
    "esdoc": "0.5.2",
    "eslint": "3.12.2",
    "eslint-config-airbnb": "13.0.0",
    "eslint-plugin-import": "2.2.0",
    "eslint-plugin-jsx-a11y": "2.2.3",
    "eslint-plugin-react": "6.8.0",
    "file-loader": "0.9.0",
    "happypack": "3.0.2",
    "ignore-styles": "5.0.1",
    "jsdom": "9.9.1",
    "json-loader": "0.5.4",
    "lerna": "2.0.0-beta.31",
    "mocha": "3.2.0",
    "mock-css-modules": "1.0.0",
    "nyc": "10.0.0",
    "postcss-cssnext": "2.9.0",
    "postcss-discard-empty": "2.1.0",
    "postcss-focus": "1.0.0",
    "postcss-loader": "1.2.1",
    "postcss-modules": "0.5.2",
    "postcss-modules-extract-imports": "1.0.1",
    "postcss-modules-values": "1.2.2",
    "postcss-reporter": "3.0.0",
    "postcss-theme": "1.1.1",
    "react-addons-test-utils": "15.4.1",
    "react-hot-loader": "3.0.0-beta.6",
    "redux-devtools": "3.3.1",
    "redux-devtools-dock-monitor": "1.1.1",
    "redux-devtools-log-monitor": "1.1.1",
    "rimraf": "2.5.4",
    "style-loader": "0.13.1",
    "stylelint": "7.7.1",
    "stylelint-config-standard": "15.0.1",
    "supertest": "2.0.1",
    "uglify-js": "2.7.5",
    "url-loader": "0.5.7",
    "webpack-dev-middleware": "1.9.0",
    "webpack-hot-middleware": "2.14.0",
    "webpack-manifest-plugin": "1.1.0",
    "webpack2-externals-plugin": "1.0.0"
  },

Oh, so I think the README may not have made this clear enough but HappyPack would only work for loaders that can be configured by a set of JSON-serializable options. Passing function instances (like the return value of plugin.bind()) to the options is not serializable, and that's why you're seeing the error.
The only way to make this work is to figure out a way to configure the loader by means of a config file. That way, you pass only a string with the path to your config file in your webpack config, then the loader has to require that file and evaluate its exports when it needs to (which can then export any non-serializable stuff, like what you've shown.)
Hope that clears it up.

Seeing that this is babel, I wonder if you can just point it to a babelrc.js file that would then instantiate the cssnext plugin. You can try that.


413 error - entity too large - for relatively small datasets

One of our publishers is getting the following message when trying to create a new dataset “413 error - entity too large” but the datasets aren’t very big (one is less than 10 MB)
https://www.gbif.org/installation/240e89bc-4985-4468-aa2f-aaba905a8be4

Thanks Marie. Can they provide their logs please?
I can't also access the IPT https://ipt.pnwherbaria.org/rss.do

I asked them for their logs, I will put them here when they send them.

Any updates here?

They wrote back that they are trying to put their server online and that they will be in touch with us. I haven't heard since. I will follow up them in January if there is still no news.

We had the same problem in our IPT and it gave us the same message. Turns out it was a configuration problem in our server, we actualize the NginX parameters to increase the file size that you can upload.
Not sure if is the same problem, but maybe it can be helpful.

I wonder if this error message may also appear for url-based data sources.
 Eventually, it could grow beyond some limit.
Has the IPT any way to notify about errors raised by unattended publications?
Or we should just permanently look into IPT logs?
Thanks
@abubelinha

@abubelinha Thanks for you question
No, unfortunately, we don't have such functionality yet

Closing this. Feel free to reopen if the issue occurs again


Allow swarm join with --availability=drain

This fix (together with https://github.com/docker/docker/pull/24993) tries to address the issue raised in
https://github.com/docker/docker/issues/24596
where it was not possible to join as manager only (--availability=drain).
This fix adds NodeSpec to IssueNodeCertificateRequest so that it is
possible to specify the availability of the node when it joins the swarm.
This fix is related to
https://github.com/docker/docker/issues/24596
https://github.com/docker/docker/pull/24993
Signed-off-by: Yong Tang<EMAIL_ADDRESS>

Current coverage is 54.78% (diff: 70.00%)

Merging #1271 into master will increase coverage by 0.05%

@@             master      #1271   diff @@
==========================================
  Files            78         78
  Lines         12426      12424     -2
  Methods           0          0
  Messages          0          0
  Branches          0          0
==========================================
+ Hits           6801       6807     +6
+ Misses         4680       4670    -10
- Partials        945        947     +2

Powered by Codecov. Last update 36c1539...7d4316f

Thanks @aaronlehmann the pull request has been updated. Please let me know if there are any issues.

Thanks @aaronlehmann for the review. The comments for both agent/node.go and api/ca.proto|ca.pb.go have been updated. Please let me know if there are any issues.

LGTM
ping @diogomonica @stevvooe

Feature: 👍 Implementation: 😐

Agreed with @stevvooe, we should honor the role now, instead of trying all the tokens.

Agreed with @stevvooe we should honor the role, if we have one.

Thanks @aaronlehmann @diogomonica @stevvooe for the review.
I updated the pull request so that Spec has been renamed to InitialSpec.
The func GetRemoteSignedCertificate has also been consolidate to take
api.IssueNodeCertificateResponse as the argument, which should help make it cleaner.
Please take a look and let me know if there are any issues.

@diogomonica @stevvooe WDYT?

I'm 👍 with this, but pinged @aluzzardi about it since he seemed to have some reservations.

I'm still against the use of spec here, especially if we are completely ignoring role. Don't we get extra validation if we know the intent from the spec and the match of the token?
Imagine if we called these APIs from a node being used to provision others, that then distributes certs and key material.

I think doing role validation is way outside the scope of this PR, and there are some practical problems with it (for one, it would require readding a --manager flag to swarm join, which we consciously decided to remove during a design review).
If the use of the node spec is too controversial, maybe we should go back to just having an Availability field, and adding other fields as necessary.

@aaronlehmann I guess the question we need to answer is whether or not there is a way to validate the conditions provided in the NodeSpec. In this case, we only know we can trust the token. The availability is completely unverified. What happens with other parts of the NodeSpec, if we add them? Do we trust them, as well?

LGTM

ping @stevvooe @diogomonica PTAL

Not sure what to do here. We're now paying for the decision of not having a Certificates top-level object, and of abusing node to hold it. But still LGTM.

What's the status on this one?

Created a new issue #1731 to keep track of the args consolidation in LoadOrCreateSecurityConfig
(for https://github.com/docker/swarmkit/pull/1271#discussion_r85432528)

@thaJeztah No need for the extra type here. There is a bug in gogo that needs to be fixed to generate pointer types that will solve this. If we add the structure here, we can never remove it.

I guess we should remove that extra type for now.

@stevvooe @aaronlehmann @diogomonica The PR has been updated. Please take a look and let me know if there are any issues.

LGTM

@stevvooe @diogomonica A separate PR has been created #1751 to address the arg consolidation issue of LoadOrCreateSecurityConfig. Please take a look.

@aaronlehmann @stevvooe @diogomonica This PR has been updated with the args list refactor from #1757. Please take a look and let me know if there are any issues.

LGTM

Glad to see this finally merged. Thanks @yongtang for sticking with it.


Pass version to docker build

Hi,
I have no idea why these issues failed, non of them are related to my change...
Thanks

Hello, what are you trying to achieve?
IIRC the docker build command is only used to pass the binaries compiled with promu, which should have the correct version.

@roidelapluie I want to achieve certification by RedHat, they require version label in docker file. I know that promu set the correct version but it is accessible from outside so if you download it to air gap by 'docker save'/'docker load', this information is hard to find (this is why RedHat requires it in first place).

Is there any documentation of the requirements we would need?

@robbiet480 I hope that you have access to Technical Prerequisites

@robbiet480 PDF of above location Technical Prerequisites.pdf. The only additional problematic is release

Quoting:

The Base image must be (or must be based on) a supported Red Hat image, such as Red Hat Enterprise Linux or Red Hat Universal Base Image. Any third party or community supported images such as Ubuntu, Debian, Alpine, CentOS etc are not supported by Red Hat and cannot be certified.

So adding the release won't cut it.
Also:

The following labels must exist: name, maintainer, vendor, version, release, summary & description.

I don't think we add labels
Then:

You need to include any relevant licenses within the licenses/directory

We do not do that either.
I would recommend that you look at images provided by red hat, as all of this would be too vendor specific, and if tomorrow Suse or Ubuntu comes with their own conflicting specifications, we could not meet all of them. Additionally, we aim at starting with a really small base image, probably distroless, which would end up also not being supported by red hat.

Why are you tagging me @AssafKatz3

Why are you tagging me @AssafKatz3

I guess your nickname starts with ro like mine ;)

@roidelapluie I can solve the other problems for push gateway (other repositories are already supplied by RedHat or other partners) without impact this repository:
FROM registry.access.redhat.com/ubi8/ubi-minimal:8.3
ARG VERSION
LABEL maintainer="The Prometheus Authors<EMAIL_ADDRESS>LABEL name="Prometheus Pushgateway"
LABEL vendor="The Prometheus Authors"
LABEL version=$VERSION
LABEL release="N/A"
LABEL summary="Push acceptor for ephemeral and batch jobs"
LABEL description="The Prometheus Pushgateway exists to allow ephemeral and batch jobs to expose their metrics to Prometheus. Since these kinds of jobs may not exist long enough to be scraped, they can instead push their metrics to a Pushgateway. The Pushgateway then exposes these metrics to Prometheus."

ARG ARCH="amd64"
ARG OS="linux"
COPY --chown=nobody:root .build/${OS}-${ARCH}/pushgateway /bin/pushgateway

EXPOSE 9091
RUN mkdir -p /pushgateway && chown nobody:root /pushgateway
COPY --chown=nobody:root LICENSE /licenses/
WORKDIR /pushgateway

ENTRYPOINT [ "/bin/pushgateway" ]

My problem is therefore very focused - I need to supply VERSION build argument.

Message from the bug scrub: @roidelapluie and I are confused about the requested feature here. @bwplotka perhaps you are familiar enough with both Prometheus Dockerfiles and the RH-certified images? Or @discordianfish , you are my usual reference point for Docker questions…

No idea what this is about..
@AssafKatz3 Maybe you can just create your own Dockerfile with FROM prom/prometheus ... and add whatever labels are needed there?

@discordianfish I will add the labels to Dockerfile so it will be more clear why we need to pass the value of version label

@AssafKatz3 Yeah I don't think there is any standard for this kind of metadata which we could just adopt here. If there are more people in need for this, I would reconsider. But either way, it would probably make sense to ask on the mailinglist of other people are interested in that and discuss how we'd do that in a consistent way across all our repos.

@AssafKatz3 Yeah I don't think there is any standard for this kind of metadata which we could just adopt here. If there are more people in need for this, I would reconsider. But either way, it would probably make sense to ask on the mailinglist of other people are interested in that and discuss how we'd do that in a consistent way across all our repos.

@discordianfish There is no standard, but version is common as you can see in documentation. I opened thread in mailing list but got no answers.

The lack of response probably means that people rarely run into this kind of issue.
Personally, I don't feel strongly about those labels. So if nothing speaks against it, why not add them?
But I would mostly trust @discordianfish 's judgement here.

The lack of response probably means that people rarely run into this kind of issue.
Personally, I don't feel strongly about those labels. So if nothing speaks against it, why not add them?

For consistency. I don't think we have anyone willing to maintain this over all the projects we have? Can we automate that?

But I would mostly trust @discordianfish 's judgement here.

Note that the pull request as is has issues, e.g. it is doing chown /licenses but we don't have it.

@roidelapluie I added copy the to /licences in order to fix the chown issue. The other issues aren't related to this change at all.

That still does not make the image red-hat certified so I don't really see the point here :(

@roidelapluie The point here that I want to make push gateway RedHat certified, this requires changing Makefile.common here. Otherwise, this file will be overridden in push gatway from this repository (or more probably, my pull request to there will be rejected). If know better way, it will be very welcome :-)

Since we do not base ourselves on a red hat ceritifed base image, it will not be certified. If it is only the pushgateway, how did you make a prometheus/etc certified image?

All up for red hat certified images but that's something that should happen externally of the prometheus org and probably be done by redhat. So I'd close this issue.

We have looked at this during our bug scrub and decided to follow @discordianfish 's opinion. If there is a clear standard that would bring benefits we would follow it, but in this case, we don't see the added value.
Thanks for your contribution.


Trimming JSON-fields for whitespaces

Current Implementation
Decoded JSON fields might contain whitespaces.
E.g. GR/EL QR Vaccination Codes in Field "mp"
Suggested Enhancement
During the CBOR Deserialization, all fields must be trimmed for whitespaces.

@Hendrik-Schmidt-Schierhorn-TSI is this issue still existing?


let medibot express itself better

About the PR
adds the AM hate monologue from I Have No Mouth And I Must Scream to the list of medibot's rare quotes
Why / Balance
now medibots can express their real feelings
Media
untested
Requirements

[x] I have read and am following the Pull Request and Changelog Guidelines.
[ ] I have added media to this PR or it does not require an ingame showcase.

Changelog
no

I don't think this really fits...

I don't think this really fits...

it's rare and is meant to be funny via surprising you

update: tested, added media

Hey, I brought this up for maintainer discussion and the consensus was to close this for the following reasons:

The text is too long and causes visual spam.
We don't want to include something that mentions specific hate to one species ("human").
It does not really fit the medibot and not everyone will get the reference.

Thank you for your contribution.


Hygieia JIRA Collector - ALM JIRA - SSO Issue

Hi
I am getting below issue in JIRA Collector.
SSO Failure - Unable to process request. Either the requested URL was not specified in terms of a fully-qualified host name or OHS single sign-on is incorrectly configured. Please notify your administrator.
Please find attached Issue Details.
BNPPJIRAIssue.txt
I was able to successfully integrate Hygieia with Trail Version of JIRA for normal user. I am now facing to connect with Real Time PRD Version of ALM JIRA for SSO User.
Please suggest.
Thanks & Regards
Venkat

@venkiradkris  install ALM JIRA cert into cacerts using keytool and restart collector

Hi
[1] I have downloaded "-usoraclecomUAT.crt" file via browser from my UAT ALM JIRA Application.
[2] Have installed into cacerts using below command. Verified the certificate availability in cacerts.
keytool -import -file /home/veradhak/jirauatcert/-usoraclecomUAT.crt -alias jirauat -keystore "/usr/java/jdk1.8.0_161/jre/lib/security/cacerts"
[3] Started JIRA Collector and getting same issue.

@venkiradkris  make you you are using the correct JDk to install certs.

@venkiradkris  if that did not work, you can use enterprise proxy to connect to Jira using -Dhttps.proxyHost in jar command to startup collector


Webpack Fails on ag-grid-community, GraphQL with 5.45.0 Release

Bug report

What is the current behavior?
Webpack builds have started failing in our automated deployments as of the 5.45.0 release a few hours ago
If the current behavior is a bug, please provide the steps to reproduce.
In our project. ag-grid-community and GraphQL dependencies fail to compile (stack trace and webpack config file below).
Our project is closed source unfortunately but here is what I suspect are repro steps:

Initialize an NPM project.  Install ag-grid-community and/or GraphQL
Add an index.js which depends on these files in a way that requires them to be packed
execute webpack

What is the expected behavior?
Webpack should pack the files without error similar to the behavior of 5.44.0

Other relevant information:
webpack version:  5.45.0
Node.js version: 12.18.1
Operating System: MACOS/Linux/Windows all reproduce
Additional tools: Terser Plugin, Preamble Plugin

Webpack Config: https://github.com/pdemro/webpack-config-5.45.0-suspected-bug/blob/main/webpack.config.js

Stack Trace:

Entrypoint index 3.99 MiB = runtime-bundle.js 7.57 KiB vendor-bundle.js 2.14 MiB index-bundle.js 1.84 MiB
runtime modules 3.63 KiB 8 modules
modules by path ./src/ 1.43 MiB
  javascript modules 1.31 MiB 319 modules
  json modules 55.4 KiB 13 modules
  asset modules 64.5 KiB
    modules by path ./src/images/*.png 16.6 KiB 8 modules
    3 modules
modules by path ./node_modules/ 4.17 MiB
  modules by path ./node_modules/graphql/ 41.3 KiB 9 modules
  modules by path ./node_modules/highcharts/ 346 KiB 7 modules
  modules by path ./node_modules/graphql-request/ 16.6 KiB 4 modules
  modules by path ./node_modules/extract-files/public/*.js 2.06 KiB 4 modules
  modules by path ./node_modules/css-loader/dist/runtime/*.js 2.38 KiB
    ./node_modules/css-loader/dist/runtime/api.js 1.57 KiB [built] [code generated]
    ./node_modules/css-loader/dist/runtime/getUrl.js 830 bytes [built] [code generated]

ERROR in ./node_modules/ag-grid-community/dist/ag-grid-community.cjs.js 1530:19
Module parse failed: Unexpected token (1530:19)
You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file. See https://webpack.js.org/concepts#loaders
|         modules.forEach(function (module) { return ModuleRegistry.register(module, moduleBased); });
|     };
>     ModuleRegistry.assertRegistered = function (moduleName, reason) {
|         if (this.isRegistered(moduleName)) {
|             return true;
 @ ./src/components/agGridPopout/ag-grid-popout.ts 16:28-56
 @ ./src/ sync (^.?|\.[^d]|[^.]d|[^.][^d])\.(j|t)sx?$ ./components/agGridPopout/ag-grid-popout.ts
 @ ./src/index.ts 5:12-81

ERROR in ./node_modules/graphql/index.mjs 38:2
Module parse failed: Unexpected token (38:2)
You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file. See https://webpack.js.org/concepts#loaders
| , SchemaMetaFieldDef, TypeMetaFieldDef, TypeNameMetaFieldDef // Predicates
| , isSchema, isDirective, isType, isScalarType, isObjectType, isInterfaceType, isUnionType, isEnumType, isInputObjectType, isListType, isNonNullType, isInputType, isOutputType, isLeafType, isCompositeType, isAbstractType, isWrappingType, isNullableType, isNamedType, isRequiredArgument, isRequiredInputField, isSpecifiedScalarType, isIntrospectionType, isSpecifiedDirective // Assertions
> , assertSchema, assertDirective, assertType, assertScalarType, assertObjectType, assertInterfaceType, assertUnionType, assertEnumType, assertInputObjectType, assertListType, assertNonNullType, assertInputType, assertOutputType, assertLeafType, assertCompositeType, assertAbstractType, assertWrappingType, assertNullableType, assertNamedType // Un-modifiers
| , getNullableType, getNamedType // Validate GraphQL schema.
| , validateSchema, assertValidSchema } from "./type/index.mjs";
 @ ./node_modules/graphql-tag/lib/index.js 2:0-32 65:21-26
 @ ./src/generated/graphql.ts 7:38-60
 @ ./src/ sync (^.?|\.[^d]|[^.]d|[^.][^d])\.(j|t)sx?$ ./generated/graphql.ts
 @ ./src/index.ts 5:12-81

webpack 5.45.0 compiled with 2 errors in 31329 ms

Duplicate https://github.com/webpack/webpack/issues/13814

Already merged, patch release will be soon

You guys are on top of it.  Cheers 🍻


backported soft shadow fix from three

Fixes #942
Fixes #947
Fixes #263
This is a backport of https://github.com/mrdoob/three.js/pull/18328
Some more testing would be great, but this seems to have fixed both issues for me on my Pixel 3.

LGTM 👍 thanks for the quick turn-around on this. I would be interested to hear feedback from upstream, but it doesn't need to be a blocker for this PR.

I can confirm that this fixes the yellow/blue issues on Xiami Mi 9 with my test cases. Thanks!

Things are fine here in model-viewer but the object in Scene Viewer is texture corrupted, showing some purple-ish colours. Is it possible that the Scene Viewer also may need a similar fix? BTW, what's the best channel to report bugs on it? Thank you!

@tiagovignatti Is the purple coming through in AR? I'll let them know.

@tiagovignatti can you share a screenshot of the Scene Viewer issue?

This is the model in model-viewer displaying right colours:

This is the model in Scene Viewer (3D) with wrong colours:

@elalish the model looks wrong both in AR and 3D.

@tiagovignatti thanks for the update! Would you also be able to drop the model in here: https://vr.google.com/scene-viewer-preview
And report back if you see any errors or warnings?

@tiagovignatti So, neither of those screenshots is actually AR (no camera image). If your phone does not support AR, then SceneViewer actually uses an older version of model-viewer to display the 3D scene. If that's the case, then we've already fixed this. If you can see these errors on top of a camera image, then it's a SceneViewer bug.

@elalish I think their phone supports AR (it is showing the AR button and @tiagovignatti reported that it looks wrong in AR).

Correct, my phone is a Samsung S8. I just didn't send the AR screenshot here before. Anyhow, as I said the model looks wrong both in AR and 3D when using Scene Viewer (when in WebXR, things look good though).
Here's scene-viewer-preview screenshot, which besides the warnings the texture looks good:


Link telephone numbers for click to call

While many modern mobile browsers automatically detect phone numbers and convert them to links, it’s a good idea to do this directly in your code. By manually tagging each phone number, you can ensure that phone numbers are always enabled for click to call and that they will be styled to match your site.
To mark a phone number as a link, use the tel: scheme. The syntax is simple:
example:
+1<PHONE_NUMBER>
thanks,

Good to have feature. In case anyone wants to pick this up.

Hey, @bee-rickey , I'd like to take this up

@KrishnaPravin your PR is working fine on phone. Thanks 😊

Hi Team,
I can pick this issue, Please assign on my name.


build(deps): bump github.com/vektra/mockery/v2 in /tools

Bumps github.com/vektra/mockery/v2 from 2.38.0 to 2.40.3

Release notes
Changelog
Commits

updated-dependencies:

dependency-name: github.com/vektra/mockery/v2 dependency-type: direct:production update-type: version-update:semver-minor ...

This change bump mockery to v2.40.3 and regenerate the mocks using the new version.
mockery v2.40.3 is the last one that depends on
go1.21; when we bump to go1.22 we could bump mockery to higher version.
/close https://github.com/podengo-project/idmsvc-backend/pull/219

Closed and open a new PR with a different source branch name, because the pattern used could be confusing in the future. New source branch rhin-1706-bump-mockery-2.40.3


[API Endpoint Request] Post Requests

Currently there is no documentation specifying how to post to the API. We are students and need these endpoints in order to automate the creation of classrooms and assignments as a quality of life improvement for our teacher.
We could help you adding them to the GitHub Classroom CLI as well.
Is there a reason why the post endpoints are not public yet?

@lorenzholzbauer 👋 thank you for opening this issue and showing enthusiasm to help build new features on top of our current implementation! We have yet to prioritize work on publicizing other endpoints but we're in the process of planning so stay tuned!


Expose selector dependencies for testing purposes

This PR allows us to test which dependencies are passed into a selector.
Original conversation: https://github.com/reactjs/reselect/issues/76#issuecomment-299194186
It matters which dependencies a selector uses.  For example, if we have two selectors that use the same resultFunc...
const lookUp = (hash, ids) => ids.map((id) => hash[id]);

const getMyValues = createSelector(
  getMyHash,
  getMyIds,
  lookUp,
);

const getYourValues = createSelector(
  getYourHash,
  getYourIds,
  lookUp,
);

...testing the resultFunc isn't enough.  We actually need to test that the first two functions are what we expect them to be.  We can do this with an integration test (passing in the entire state into the whole selector) but, it forces us to essentially re-test getMyHash, getMyIds.
Desired API
We've been running into this problem.  The direction I'm leaning is to expose the dependencies so that we can test them with a simple equality check.
it('uses the dependencies we expect', () => {
  // We can expose `dependencies`
  const dependencies = getMyValues.dependencies;

  expect(dependencies).to.deep.equal([
    getMyHash,
    getMyIds,
  ]);
});

Coverage remained the same at 100.0% when pulling fd3238cf5a9c350e02ee118ee1cd3ba7282774ad on jimbol:expose-dependencies into 0c9150e6e2102e3d6d1eb1675401bb0cc81dc769 on reactjs:master.

Codecov Report

Merging #251 into master will not change coverage.
The diff coverage is n/a.

@@          Coverage Diff          @@
##           master   #251   +/-   ##
=====================================
  Coverage     100%   100%
=====================================
  Files           1      1
  Lines          15     15
=====================================
  Hits           15     15

Impacted Files
Coverage Δ

src/index.js
100% <ø> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0c9150e...fd3238c. Read the comment docs.

Any updates on this PR?  Are the maintainers absent?

Would love to have that feature too.

What does everyone think of this?

merge! 🚢

Just ran into the same issue myself, so this would be super helpful. Merge!

Merge! 👍

Wooohoooo! I can't believe I didn't see this. This will allow me to remove move a step in installing my reselect developer tools

Did this get merged?

@ellbee @alex3165 @markerikson can y'all take a look at this to see if we can merge it?

@ellbee could you take a look at this? It would be really handy to have in

Ok, this is not something that has been a pain point for me but seems to be for others so lets merge it in!

Has this change been published? I see it in master but not in my node_modules.

I do not see that there is a necessity to mock the dependencies of the selector and test the only resultFunction.
The selector is a pure function that receives several params. From the unit testing perspective, we have to test the entire function and test behavior, not implementation.
This function should be considered as a black box. There are input and output that's all. In that proposal, dependencies are considered everything inside the selector. But let's take a look at the following example?
If I have selector ```
export const getProjectAssignmentsTimeRange = createSelector(
(store, projectId) => getProjectAssignmentsByProjectId(store, projected),
store => store[ns.calendarRangeExtension].extraCount,
getPlanningAssignmentTabZoomLevel,
(projectGantt, extraCount, zoomLevel) => {
const yearAddition = extraCount || 0;
......
}
);
Are `getPlanningAssignmentTabZoomLevel` and `getProjectAssignmentsByProjectId` real dependencies for the main selector? I do not think so, since it's a pure function and does not have side effects, basically, you can copy the code of this function and put inside the selector.

However, if you mock getPlanningAssignmentTabZoomLevel you rely on the code inside of selector,
which will require the test update if the code/implementation of this pseudo-dependencies changes (store change, param change) even though the "business" requirement of the selector is still the same.

I do not think that it's a good approach to test the technical implementation rather than test what the selector should do.

As an example of the correct test approach, please find my test. I used the BDD (Behavior-Driven Development) approach.

describe('calendar range for calendar on Assignments Tab for Week view', () => {
beforeAll(() => {
store = {
[ns.calendarRangeExtension]: {
extraCount: 0,
},
[ns.projectGantts]: {
byProjectId: {
      },
    },
    [ns.planningSettings]: {
      projectAssignmentsZoomLevel: ZoomLevel.WEEK,
    },
  };
  spyToday = jest.spyOn(global, 'Date').mockImplementation(() => new Date(2020, 8, 24));
});

test('has to calculate range when there is no project allocation', () => {
  spyToday.mockRestore();
  expect(getProjectAssignmentsTimeRange(store, 1)).toEqual({
    startDate: new Date(getFakedUTCDate('2019-01-01')),
    endDate: new Date(getFakedUTCDate('2021-12-31')),
  });
});

Instead of mock function `getPlanningAssignmentTabZoomLevel` I mock the param which is store.


How to cite pandas_profiling?

Hello, this library is amazing and I would like to cite pandas_profiling in my report. Do you have an official way (or BibTeX) for that?
Thank you very much.

Thanks!
I'm currently working on a publication. Until then, you could use the following BibTeX reference:
@misc{pandasprofiling2019,
  author = {Simon Brugman},
  title = {{pandas-profiling: Exploratory Data Analysis for Python}},
  year = {2019},
  howpublished = {\url{https://github.com/pandas-profiling/pandas-profiling}},
  note = "Version: 2.X, Accessed: INSERT_DATE_HERE"
}

Perfect. Thank you very much.

Would it be better if cite it as ydata-profiling now?
For example,
@misc{pandasprofiling2019,
  author = {Simon Brugman},
  title = {{ydata-profiling: Exploratory Data Analysis for Python}},
  year = {2019},
  howpublished = {\url{https://github.com/pandas-profiling/pandas-profiling}},
  note = "Version: 2.X, Accessed: INSERT_DATE_HERE"
}


DirectivesMethod.java:37-40: Refactor DirectivesMethod...

The puzzle 534-d00ee4c1 from #534 has to be resolved:
https://github.com/objectionary/jeo-maven-plugin/blob/d3a897bf222a5b3f7eaa5a3ea66f3692cd92945a/src/main/java/org/eolang/jeo/representation/directives/DirectivesMethod.java#L37-L40
The puzzle was created by volodya-lombrozo on 02-Apr-24.
Estimate: 90 minutes,  role: DEV.
If you have any technical questions, don't ask me, submit new tickets instead. The task will be "done" when the problem is fixed and the text of the puzzle is removed from the source code. Here is more about PDD and about me.

@0pdd 3 puzzles #699, #700, #701 are still not solved.

@0pdd the puzzle #699 is still not solved; solved: #700, #701.

@0pdd all 3 puzzles are solved here: #699, #700, #701.


When using not minimized javascript, Which .js do i include?

When using not minimized javascript, Which .js do i include?

Check the documentation please http://www.roxyfileman.com/FAQ

Are you sure the answer is in FAQ?

You can find the answer into the fileman.html, check the scripts used! 👍

I mean when not using main.min.js, which *.js do i include?

Ohh check the scripts folder => https://github.com/jdnichollsc/Roxy-Fileman-for-Node.js/tree/master/public/js

There's no scripts folder on there, and scripts in package.json, there's nothing about that.

Could you add a minimize script on package.json.

Like a Node.js module?


model: convert checked exceptions to runtime exceptions

DuplicatePersonException and PersonNotFoundException are checked
exceptions. They are thrown by methods to signify that their relevant
preconditions were violated.

Using checked exceptions for such a purpose is slightly useful in that
it will force callers to handle the case where the above preconditions
are not met, such as when the methods are called with invalid user
input.

However, it imposes a HUGE cost on callers where the preconditions are
already known to be met (e.g. in test code, or when the user input has
already been validated before hand).  It is also
impossible to test the catch blocks as well since correct code will
ensure that the precondition holds and thus the exception will never be
thrown, leading to reduced code coverage.

Checked exceptions also don't work very well with the Java Streams API,
since the API doesn't accept lambdas which could throw checked
exceptions.

In AB-4, the amount of code which benefits from DuplicatePersonException
and PersonNotFoundException being checked exceptions is much smaller
than the amount of code which is negatively impacted.

As such, let's make the tradeoff in the other direction, by making
DuplicatePersonException and PersonNotFoundException runtime exceptions.
New callers _could_ forget to check that the preconditions hold before
calling the methods in question (although test cases should catch that),
but this is balanced out by the huge benefit of having more concise and
testable code.

@yamidark Sorry, please do concentrate on https://github.com/se-edu/addressbook-level2/pull/388 , I think @damithc would prefer that. Sorry for requesting your review and taking your time, this PR is not so important.

@eugenepeh

[10/10] Since UniquePersonList is in the package seedu.address.model.person;, UniquePersonListTest should be correspondingly the same.

fyi, this is not a new file, but I'll do it anyway in this PR since it's easy enough to do.

@Zhiyuan-Amos

Since we are on Java 9, should we use List.of(...) instead of Arrays.asList(...)?

I considered that, but we don't use List.of(...) at all in our code base, and this PR may not be a good time to "break new ground" in that sense. Putting aside immutability, Arrays.asList(...) is the same as List.of(...) and not that many more characters to type.
I'll wait for #871

[3/11] Also added some requireNonNull()s, missed that out in v1.


519 - use SmtSolverError for all Smt-related errors in booster

As in the legacy backend, the SMT errors are caught at the top level in a handler.
An SMTError is thrown generally when there is a problem with solver operations.
Undecided predicates during rewriting and simplifications are reported back to the client as SMTSolverError (code 5) and include the predicates in play in the data field. Other errors are reported as runtime errors.
As in the legacy backend, we lose all rewriting/simplification work done prior to the SMT error.
Fixes #519

Summary by CodeRabbit

New Features

Introduced enhanced error handling for SMT operations, including specific error messages and responses.
Added a new SetTimeout feature to control solver operation timeout settings.

Bug Fixes

Improved SMT error handling with the addition of handleSmtError function, ensuring more robust and informative error responses.

Refactor

Updated various modules to improve error handling logic and to support new timeout control feature.

Tests

Added a new test case to validate the handling of unknown states in SMT model generation.

Should be discussed whether or not to amend this to return the undecided predicate (and make other SMT errors general RuntimeErrors as they were previously. See https://github.com/runtimeverification/kontrol/issues/396

KEVM tests - no significant change

Test
519-use-SmtSolverError-in-booster time
master-618360c time
(519-use-SmtSolverError-in-booster/master-618360c) time

kontrol/test-emitcontracttest-testexpectemit-0-spec.k
45.03
57.37
0.7849050026146069

kontrol/test-expectreverttest-testfail_expectrevert_bytes4-0-spec.k
51.25
61.67
0.8310361602075563

kontrol/test-storetest-teststoreload-0-spec.k
96.21
103.19
0.9323577866072293

kontrol/test-owneruponlytest-testincrementasowner-0-spec.k
74.18
76.99
0.9635017534744773

kontrol/test-countertest-testincrement-0-spec.k
100.33
89.64
1.1192547969656403

kontrol/test-storetest-testaccesses-0-spec.k
59.41
51.82
1.1464685449633345

kontrol/test-expectreverttest-test_expectrevert_returnvalue-0-spec.k
64.31
55.11
1.1669388495735802

TOTAL
490.72
495.79
0.98977389

I modified the code to only use SmtSolverError when the solver returned Unknown  from a predicate check (matches the legacy backend behaviour), and fill the data field with the predicate in question (both premises and actual predicate to check.
Also implemented the --smt-timeout option which was previously parsed but ignored.

Added the get-model test from haskell-backend PR https://github.com/runtimeverification/haskell-backend/pull/3737 .
A simplify test does not produce an SmtUnknownError because booster does not currently use the SMT solver during simplification/evaluation.

55-62: The introduction of the SMTError data type with specific error variants (GeneralSMTError, SMTTranslationError, SMTSolverUnknown) is a significant improvement in error handling.

200-204: The addition of Booster.handleSmtError to the list of error handlers in the jsonRpcServer function call is a critical change that aligns with the PR's objective to standardize SMT solver error handling.

Yay, code rabbit, you got it right  :rabbit:  ! :stuck_out_tongue:


Enhance the way adapters are defined

Details
The problem with the adapters at the moment as defined in #3 is that really the annotated examples allow sort of override listeners more than actually implementing anything which is really how it should be. To really implement some adapter I think that DatabaseAdapter should be implemented.
Well the issue with that is that during the event stream the adapter is swapped out and the other annotated listeners are called. I think that the base adapter should be swapped out on setup right up front. That would mean that someone could completely implement an adapter via the annotation out of nowhere and it would get plugged in, e.g.
DBAdapter('megadb')
class MySuperMegaDBAdaper extends DatabaseAdapter { }

Then it would valid for them to have a:
@Model('megadb')
class User {}

This was changed with #14 to @Adapter


Is a premium version of open Ai needed for this to work?

Not the premium/plus version of open ai. What you need to run chatGPT as a model is access to they API which you do need to pay for after your inital free credit expires.

@FantomWolf182 thanks, that's what I was thinking. Inside the code the error message gave me that it did not recognize the open Ai API key. So I assume it was not active


ContactHands dataset

Hello, where can I download ContactHands?

You can find it here: http://vision.cs.stonybrook.edu/~supreeth/ContactHands_data_website/


Running with Mesen2 on a Mac

I can get Alchemy65 to build NES roms from C files on a M1 MacBook, and to run Mesen2 (as Mesen-X doesn't do MacOS), but I get this error in the Lua script when trying to create a host for VSCode to hook into when running:
Loading script...
adapter.lua:3: attempt to call a nil value (global 'require')
for the following code:
local socket = require("socket.core")
The ROM file runs fine, but VSCode eventually returns the following error as it cannot attach:
Connection error: unable to connect to Alchemy 65 debug host
presumably because the socket wasn't created
Any ideas?

Also, possibly related (but not sure) is several "Property XXX is not allowed" errors in the launch.json file for the following:
       "romPath": "${workspaceFolder}/full_game.nes",
        "dbgPath": "${workspaceFolder}/full_game.dbg",
        "program": "/Users/mreddy01/Desktop/NESDev/Mesen2/Mesen.app/Contents/MacOS/Mesen",
        "sourcePath": "${workspaceFolder}/",
        "stopOnEntry": true,
        "resetOnEntry": true


Write code to do networking

Gonna use reqwest and try really hard to be nice to the API (they say they'll ratelimit clients if they go above a request every 30 seconds)

NexTrip departure information updates every 30 seconds. Help conserve our bandwidth and server resources by writing your applications responsibly. Third party applications should not update departure information more frequently than every 30 seconds. Applications making excessive calls and updating more frequently than 30 seconds will be subject to restriction.

Sourced: https://svc.metrotransit.org/nextrip

Some structs for the response schemas:
struct Route {
    description: String,
    provider_id: String,
    route: String,
}

struct Departure {
    actual: bool;
    block_number: u64;
    departure_text: String;
    departure_time: String;
    description: String;
    gate: String;
    route: String;
    route_direction: String;
    terminal: String;
    vehicle_heading: u64;
    vehicle_latitude: f64;
    vehicle_longitude: f64;
}

struct Pair {
    text: String;
    value: String;
}

struct Locations {
    bearing: f64;
	block_number: u64;
	direction: u64;
	location_time: String;
	odometer: u64;
	route: String;
	speed: f64;
	terminal:String;
	vehicle_latitude: f64;
	vehicle_longitude: f64;

}

Networking, as well as Structs are handled by the gtfs_structures crate, which has been merged into development branch as of feature/8 pr


TP2000-763 Add error message to commodity and duties form

TP2000-763 Add error message to commodity and duties form
Why
The commodity and duties form in the create measure journey is lacking an error message to say that a selection is required if a user tries to continue without selecting a commodity.
What

Overrides non_form_errors() to provide a more meaningful error message
Adds unit test

Codecov Report
Base: 91.86% // Head: 91.86% // Increases project coverage by +0.00% :tada:

Coverage data is based on head (1ba4f50) compared to base (8d307ef).
Patch coverage: 94.11% of modified lines in pull request are covered.

Additional details and impacted files
@@           Coverage Diff           @@
##           master     #830   +/-   ##
=======================================
  Coverage   91.86%   91.86%
=======================================
  Files         399      399
  Lines       26233    26249   +16
  Branches     1988     1990    +2
=======================================
+ Hits        24099    24114   +15
  Misses       1750     1750
- Partials      384      385    +1

Impacted Files
Coverage Δ

measures/forms.py
97.67% <87.50%> (-0.20%)
:arrow_down:

measures/tests/test_forms.py
100.00% <100.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


feat: adds pipeline-dir flag for melange build

@imjasonh I just added the same input in melange-build. I missed this one.

@hectorj2f @imjasonh @k4leung4 kindly pinging for this PR.


Remove constants that are deprecated in HA 2025.1

HA asks to remove SOURCE_TYPE_GPS and instead use SourceType.GPS.
Log
SOURCE_TYPE_GPS was used from torque_logger, this is a deprecated constant which will be removed in HA Core 2025.1. Use SourceType.GPS instead, please create a bug report at https://github.com/junalmeida/homeassistant-torque/issues

@junalmeida Could you please look in to this


MODELS[major]: Replacing compiled models with raw XML models

TODO:

[ ] add info in the README for submodule installation
[ ] change pypi to be compatible with submodules


ng build optimization:true no longer get routing events

🐞 Bug report
Command (mark with an x)

[ ] new
[x] build
[ ] serve
[ ] test
[ ] e2e
[ ] generate
[ ] add
[ ] update
[ ] lint
[ ] xi18n
[ ] run
[ ] config
[ ] help
[ ] version
[ ] doc

Is this a regression?
Not sure
Description
In my app component I load this code (from an injected service)
this.router.events.pipe(
filter(e => e.constructor.name === 'NavigationEnd'),
pairwise()
).subscribe((e: any[]) => {
//do things with events
});
With optimization:false I receive events as I navigate through my app's routes.  With optimization:true I do not receive any events.
🔬 Minimal Reproduction
ng build using angular.json with
architect: {build: {options: optimization:true}}
🌍 Your Environment

Angular CLI: 9.0.7
Node: 13.11.0
OS: darwin x64

Angular: 9.1.3
... animations, common, compiler, compiler-cli, core, forms
... language-service, platform-browser, platform-browser-dynamic
... router
Ivy Workspace: Yes

Package                            Version
------------------------------------------------------------
@angular-devkit/architect          0.900.7
@angular-devkit/build-angular      0.901.3
@angular-devkit/build-ng-packagr   0.901.3
@angular-devkit/build-optimizer    0.901.3
@angular-devkit/build-webpack      0.901.3
@angular-devkit/core               9.0.7
@angular-devkit/schematics         9.0.7
@angular/cli                       9.0.7
@ngtools/webpack                   9.1.3
@schematics/angular                9.0.7
@schematics/update                 0.900.7
ng-packagr                         9.1.1
rxjs                               6.5.5
typescript                         3.7.5
webpack                            4.39.2

Anything else relevant?
Not browser specific

constructor.name is the problem in this case.  It should be avoided whenever possible as it will not work when minifying code which is one of the main actions of the optimization option.  It also does not conclusively determine whether the class is actually as expected.  Any class could have that name but might not have the expected functionality.
Something similar to the following should work in this case:
    router.events.pipe(
       filter(e => e instanceof NavigationEnd),
    ).subscribe((e: NavigationEnd) => {

    });

As @clydin mentioned, constructor.name will not work when using optimization. This is because one of the optimizations done to reduce the bundle size even further is to mangle class names.
Closing as this is working as intended and using instanceof is a better alternative.

Thanks for the clarification.  Perhaps a useful build feature would be emitting a build warning when the use of constructor.name is detected (or similarly a codelyzer lint rule)


Invalid redirects are handled wrongly

Reproduction
Steps to reproduce the behavior:
Fetch a URL which returns an invalid 'Location' header. e.g. https://unequivocal.eu/invalid-redirect which responds with:
HTTP/1.1 302 Moved Temporarily
Location: https://example.com:moo/

Expected behavior
Chrome throws an error TypeError: Failed to fetch. The Fetch spec looks like it says a "network error" should be returned (which appears to mean "resolve with a fake response with a status of zero" rather than throw an actual error). What node-fetch does is silently alter the returned Location header to say https://example.com/:moo/ and then carry on with that invented URL. No matter what it should be doing, it shouldn't be doing that.
Your Environment

software
version

node-fetch
2.6.1

node
16.1.0

In fact I've just realised this also happens simply with fetch('https://example.com:moo/'), which attempts to fetch https://example.com/:moo/ instead of doing something less random.

Looks like this was secretly fixed at some point


build: use --features testing with cargo neon alias

Most often I compile with cargo build_testing. Doing a cargo neon after it will require linking.

Uhhuh, sounds like a missing wait.

test_timeline_status[True]: debug

The security implications of this might be that if people use cargo neon locally in their deployments, then testing APIs will be enabled, which allow you to do dangerous things without JWT checks.


Move platform definitions to separate repositories

To simplify maintenance of the LUNA project, we'd like platform definitions to be kept in separate repositories going forward. The aim is to avoid having to update the LUNA package for every new platform, or platform change.
Platform definitions for Cynthion hardware have been added to the Cynthion repository, and @mndza is preparing a luna-boards repository to contain other board definitions for LUNA which are not currently maintained elsewhere.
Moving platform definitions out of the LUNA repository means that auto-detection of connected hardware will need to work differently to how it does currently. The approach implemented in this PR is that:

The get_appropriate_platform function will look for the apollo_fpga module, and if it is present, check for an Apollo-based device.
If an Apollo-based device is found, it will check for an APOLLO_PLATFORMS table in known modules, currently cynthion and luna_boards. If a corresponding platform is found, it will be used.
In any other case, LUNA will require the LUNA_PLATFORM environment variable to be set to select a platform.

This approach is intended to be backwards-compatible with existing usage. It may be extended to support other autodetection methods, or other sources of platform definitions.

Marking this as ready for review since the corresponding PRs have been merged to the Cynthion and luna-boards repos.


Training tests?

I was curious if any of these were training tests, or just inference
cc @avik-pal

There are training tests here using the direct enzyme api. Some more in the lux repo using the trainstate api that internally uses enzyme


GUI based Password Generator

I would like to contribute GUI based Password Generator made using Tkinter

That would be great addition to this project


Update CONTRIBUITING.md doc

📝 Summary
Fix deprecated dependency versions in documentation. See #572
⛱ Motivation and Context

📚 References

✅ I have run these commands

[x] make lint
[ ] make test-race
[ ] go mod tidy

Codecov Report

Merging #573 (92004d8) into develop (83a69d8) will increase coverage by 0.27%.
The diff coverage is n/a.

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
@@             Coverage Diff             @@
##           develop     #573      +/-   ##
===========================================
+ Coverage    69.69%   69.96%   +0.27%
===========================================
  Files            7        7
  Lines         1102     1102
===========================================
+ Hits           768      771       +3
+ Misses         288      287       -1
+ Partials        46       44       -2

Flag
Coverage Δ

unittests
69.96% <ø> (+0.27%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.
see 1 file with indirect coverage changes


DX in dev tools

It's not real fun to explore the dev tools and find all the shadow roots collapsed by default, with only the <react-shadow-scope> tag names visible. It makes scanning the DOM tree very difficult.

It would be fantastic to have some way to identify which component each scope contains. That could mean customizing the tagname from a prop, (even though that may feel redundant to the author... there's not much else I can do.)
Proposed interface:
<Scope tag="my-element">

outputs -> <my-element>
Additionally, to be safe:
<Scope tag="element">

outputs -> <x-element>

Added tag prop and documentation along with it.


convert to yaml format

i've converted the cloudformation template to yaml as its more readable and will also help with moving to the new SAM model as per
https://aws.amazon.com/about-aws/whats-new/2016/11/introducing-the-aws-serverless-application-model/

@neillturner Thanks again for your efforts. As I answered to your other suggestion in my Continuous Deployment for AWS Lambda behind API Gateway repo I do not intend to update this repository, since it is part of the blog post. Instead, I suggest that you to fork this repository.

you need to update the cloud formation template to "Runtime": "nodejs8.10", it errors when running on cloud formation else.

thanks


Log nullable columns on creation

This is a simple PR that adds logs indicating whether columns are nullable or not when online segments are created or when offline segments are being created and/or updated.
Currently whether an index is used or not is indirectly logged in most indexes because we log whenever we allocate offheap memory. But given null vectors use roaring bitmap, that is not logged.

Codecov Report
Attention: Patch coverage is 0% with 9 lines in your changes missing coverage. Please review.

Project coverage is 0.00%. Comparing base (59551e4) to head (b9445b7).
Report is 921 commits behind head on master.

Files
Patch %
Lines

...local/indexsegment/mutable/MutableSegmentImpl.java
0.00%
5 Missing :warning:

...ment/creator/impl/SegmentColumnarIndexCreator.java
0.00%
4 Missing :warning:

:exclamation:  There is a different number of reports uploaded between BASE (59551e4) and HEAD (b9445b7). Click for more details.
HEAD has 53 uploads less than BASE

Flag
BASE (59551e4)
HEAD (b9445b7)

integration
7
1

integration2
3
1

temurin
12
1

java-21
7
1

skip-bytebuffers-true
3
1

skip-bytebuffers-false
7
0

unittests
5
0

unittests1
2
0

java-11
5
0

unittests2
3
0

integration1
2
0

custom-integration1
2
0

Additional details and impacted files
@@              Coverage Diff              @@
##             master   #13868       +/-   ##
=============================================
- Coverage     61.75%    0.00%   -61.76%
=============================================
  Files          2436     2502       +66
  Lines        133233   138582     +5349
  Branches      20636    21541      +905
=============================================
- Hits          82274        0    -82274
- Misses        44911   138582    +93671
+ Partials       6048        0     -6048

Flag
Coverage Δ

custom-integration1
?

integration
0.00% <0.00%> (-0.01%)
:arrow_down:

integration1
?

integration2
0.00% <0.00%> (ø)

java-11
?

java-21
0.00% <0.00%> (-61.63%)
:arrow_down:

skip-bytebuffers-false
?

skip-bytebuffers-true
0.00% <0.00%> (-27.73%)
:arrow_down:

temurin
0.00% <0.00%> (-61.76%)
:arrow_down:

unittests
?

unittests1
?

unittests2
?

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Not sure if this can potentially flood log for wide tables. Should we consider logging all nullable columns as one line?

I don't think so, as explained in the description we already log at least one line per column when we allocate buffers for forward indexes. There are more lines if we use other indices that allocate buffers (ie dictionary). So the number of lines that will be logged when segments are created or sealed is going to increase linearly (with a constant in (0, 1] range).

Please address the comments before merge


Support multiple attempts on only a single rule, not all rules

I want to have a single rule retried but all other rules fail after one retry.  This is currently not possible for --restart-times I believe.
For the single rule I retry, I also want to retry it based on some lambda.  Of course, I could I set --restart-times to a large number, and then throw an exception when the rule has attempted enough times.
The use case would be to implement a memory-doubling strategy for a rule.  So the definition of a rule may say it starts at 2GB, doubles each time up to 128GB.   Those values may be variable depending on the user config.
Another use case would be to have some custom logic in the "do-retry" lambda that examines the previous output etc.  I could of course have that stuffed into the start of a "run" section.

See also #367.

@nh13: What do you think about the new rule-specific retries directive added in #1649?


Restructured and updated docs

Reviewer: @amiraliemami
Estimate: 30 mins

Restructured and updated docs
Ticket
Fixes: AAQ-146
Description, Motivation and Context

Added intro
Fleshed out contributing
Added architecture diagrams

Goal
Better docs!
How Has This Been Tested?
Locally ran mkdocs
To-do before merge

[x] DONE - Ensure PR #35 is merged

Checklist:
This checklist is a useful reminder of small things that can easily be forgotten.
Put an x in all the items that apply and remove any items that are not relevant to this PR.

[x] My code follows the style guidelines of this project
[x] I have reviewed my own code to ensure good quality
[x] I have tested the functionality of my code to ensure it works as intended
[x] I have resolved merge conflicts
[x] I have updated the automated tests (if applicable)
[x] I have written good commit messages
[x] I have updated the README file (if applicable)
[x] I have updated affected documentation (if applicable)

Extra note - the admin app screenshot's contents are a bit odd, should we have something more representative of an FAQ db there? Or just "FAQ#1", "FAQ#2" as placeholder text.

Yeah. You're right. I'll update that in another PR


Recursive relations

I can not understand how to implement recursive type definitions, like:
type User {
  friends: [User]!
}

I trying to use a runtime lazy compilation (building fields while accessing them using ArrayAccess) but I encountered an error:
Argument 1 passed to GraphQL\\Type\\Definition\\FieldDefinition::createMap() must be of the type array, object given, called in .../vendor/webonyx/graphql-php/src/Type/Definition/ObjectType.php on line 115
Is this solution a real?

'fields' can be callable. This resolves my solution.


External trigger element

Can I have external triggered element, something like this?
<button>Click me</button>
<v-popover>
  <MyAwesomeComponent slot="popover"/>
</v-popover>

I am loading the VPopover component synchronously so that the component is loaded only when I need it. This use case works something like this, but since the trigger element is inside the component it's also not visible.
<v-popover v-if="popoverVisible">
  <button>Click me</button>
  <MyAwesomeComponent slot="popover"/>
</v-popover>

How difficult would it be to implement an external trigger element?

Use a boolean data and the manual mode (see the docs).


Implicit import check during target validation

Fixes #127
When performing target validation (i.e., whether a component graph represents a component that matches the given world), we were just iterating the explicit imports. Now, we iterate all imports both implicit and explicit.
This PR also contains a slight refactoring of  encode_imports that isn't necessary but came from an early exploration and which I think makes the code more readable. Let me know if I should remove this.
When an implicit import is the cause of a validation error, the error message isn't wonderful since we don't have a span to point to like we do in the explicit import case. I think we can land this though before we figure out exactly how to improve the error message.

Thanks for digging into this! I haven't extensively studied the code just yet but I did try out the changes with the following example:
I have a component called spin:kv that targets:
world {
  import<EMAIL_ADDRESS>  import<EMAIL_ADDRESS>  import<EMAIL_ADDRESS>  import<EMAIL_ADDRESS>  import<EMAIL_ADDRESS>  import<EMAIL_ADDRESS>  import<EMAIL_ADDRESS>  import<EMAIL_ADDRESS>  import<EMAIL_ADDRESS>  import<EMAIL_ADDRESS>  import<EMAIL_ADDRESS>  import<EMAIL_ADDRESS>  import<EMAIL_ADDRESS>  import<EMAIL_ADDRESS>
  export<EMAIL_ADDRESS>}

And this wac script to do a targets check:
package test:test targets<EMAIL_ADDRESS>
export new spin:kv{...}...;

When running wac resolve script.wac I am getting the following error which I can't quite wrap my head around:
error: failed to resolve document

  × import<EMAIL_ADDRESS>has a mismatched type for target world<EMAIL_ADDRESS>  ╰─▶ instance has unexpected instance export<EMAIL_ADDRESS>
Thanks again for tackling this; could you add a test or two when you get a chance?

@fibonacci1729 thanks for finding the issue! It turns out we were always assuming a 1 to 1 correspondence between an import node (i.e., a node in the wac graph) and the item we wanted to type check. However, implicit imports don't have nodes in the wac graph (that's what makes them implicit after all!). The latest change more directly provides the validate_target function with the item definitions it needs to check instead of it having to look up items by node id from the graph.
I think this should fix the issue you're seeing - in my testing I got errors that made much more sense and correctly identified where the component and the target world didn't line up.
I'll look into how we can add tests for this.

I looked into adding a test, and discovered a new issue I'd like to address first. The issue is that just like wac, wit sort of has a similar concept of implicit imports. If you use an item inside of an imported interface from another interface, that other interface will be implicitly imported. I'll need to do the work of looking up interfaces through used items as well as direct imports.

I've addressed the issue of interfaces implicitly imported through uses and the test I've added now passes. I believe this is ready for final review and merge.

Awesome work @rylev! I'll give this a review and test on monday.


We now require urllib3 >= 1.26

We now require urllib3 >= 1.26

Is this due to the API regeneration?

Yes it is.


How to do Inference on the trained weight of GPT 2 model after finishing the training on CPU using train_gpt2.py and train_gpt2 ?

Hi Thank you very much for making everything so understandable  for even a noob like me. Sorry for such silly question though
I followed the instructions in the repository's README to train a language model on a Bengali text dataset with around 35,169 tokens, using your laptop's CPU (without a GPU). I modified the train_gpt2.py script to set my own starting words in Bengali instead of the default "<|endoftext|>".
Now, I want to know how to check if the trained model weights (the result of the training process) have improved the model's capabilities compared to before the training. I would like to compare the model's predictions with the actual text in the test dataset.
thank you for your time .

there is a file in the fodler /llm.c/dev/eval/export_hf.py which covnerts to safetensors. for example.
python /home/myles/llm.c/dev/eval/export_hf.py --input /home/myles/llm.c/log124M/model_00019560.bin --output converted_model
And then you can do something like this in python:
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
def generate_text(prompt, max_length=1000):
tokenizer = AutoTokenizer.from_pretrained("/home/myles/llm.c/converted_model")
model = AutoModelForCausalLM.from_pretrained("/home/myles/llm.c/converted_model", torch_dtype=torch.bfloat16 if torch.cuda.is_available() else torch.float32)
model.to('cuda' if torch.cuda.is_available() else 'cpu')
inputs = tokenizer(prompt, return_tensors="pt").to('cuda' if torch.cuda.is_available() else 'cpu')
attention_mask = inputs["attention_mask"]
outputs = model.generate(
    inputs["input_ids"],
    attention_mask=attention_mask,
    max_length=max_length,
    do_sample=True,
    top_p=0.95,
    top_k=50,
    pad_token_id=tokenizer.eos_token_id
)
return tokenizer.decode(outputs[0], skip_special_tokens=True)

if name == "main":
print("Chatbot: Hi! How can I help you today?")
while True:
user_input = input("You: ")
if user_input.lower() in ["exit", "quit", "stop"]:
print("Chatbot: Goodbye!")
break
prompt = f"User: {user_input}\nChatbot:"
response = generate_text(prompt)
print(f"Chatbot: {response}")


Smaller images

remove R .deb/.rpm files
remove apt cache from layers

I'd like to remove the commands to rm -rf /var/lib/apt/lists/* since it can cause problems with later installations or commands that users might add (and it appears to be breaking the CI build). Users can always customize these images and add these optimizations, so I'd like to keep these images as a simple starting point as much as possible.
We can definitely leave the commands to clean up the RPMs for R, thanks for catching that.

Oops @koverholt I hadn't removed the rm -rf /var/lib/apt/lists/* yet. Do you want me to fix that?

One thing worth considering for discussion about whether removing the apt cache is valuable. Even if it is included, we get this after time elapses:
E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/v/vim/xxd_8.0.1453-1ubuntu1.1_amd64.deb  404  Not Found [IP: <IP_ADDRESS> 80]
E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/v/vim/vim-common_8.0.1453-1ubuntu1.1_all.deb  404  Not Found [IP: <IP_ADDRESS> 80]
E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/v/vim/vim-runtime_8.0.1453-1ubuntu1.1_all.deb  404  Not Found [IP: <IP_ADDRESS> 80]
E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/v/vim/vim_8.0.1453-1ubuntu1.1_amd64.deb  404  Not Found [IP: <IP_ADDRESS> 80]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

And an apt-get update is still necessary. It would be better in my opinion if we remove the cache and just document that apt-get update is necessary any time you want to use apt / gdebi. A bad cache in the layer above a layer you are trying to change can be problematic too - another benefit for being explicit about the caching.

Ah apologies, I thought that latest commit was your fix since CI was passing. Feel free to continue in another PR!

@koverholt thoughts on my message above? Leaving the cache around gives you intermittent failures like the above. In practice, apt-get update is always required to make apt/gdebi functional in a docker image.
Should I follow up with a PR to remove the cache removal? My inclination is just to explain that apt-get update is always required to make things functional for extending.

So at the moment we apt-get update before every call to apt, and we remove cache afterwards? If so, this is consistent with what we do in the CentOS image and we can leave it as is... assuming my understanding is correct.

Yes, exactly. The only (maybe noteworthy) difference is that you also have to apt-get update before and remove cache after any call to gdebi as well. There has been some dev discussion around moving from gdebi to apt in the future because apt can now resolve dependencies for local files. So maybe we'll get there someday soon 😄

I think it is expected behavior for any Debain/Ubuntu derived image to not include the cache, making apt update necessary.


CountDown timer not working on the background state of App

I have a workout app that plays different exercises and some beep and tts onChange condition. So whenever the app goes on background timer stop working even not restarted in the background. Kindly help me to solve this problem. Thank you

same problem. Need to notify the user when my timer completes, even if in the background. Please clarify.

same problem.

Mine is also a workout app. I have an issue where it says it cannot call setState after dispose when I try to display this widget after it was hidden (another one was shown using a conditional).