[2019-10-05T20:07Z] api.go:811: WARNING: using op 'self' when with Number='1' for 'AF_POPMAX' from '/RED/RESOURCES/bcbio/genomes/Hsapiens/hg38/variation/gnomad_exome.vcf.gz' can result in out-of-order values when the query is multi-allelic
[2019-10-05T20:07Z] api.go:812:        : this is not an issue if the query has been decomposed.
[2019-10-05T20:07Z] api.go:811: WARNING: using op 'self' when with Number='1' for 'AC_ACR' from '/RED/RESOURCES/bcbio/genomes/Hsapiens/hg38/variation/gnomad_exome.vcf.gz' can result in out-of-order values when the query is multi-allelic
[2019-10-05T20:07Z] api.go:812:        : this is not an issue if the query has been decomposed.
[2019-10-05T20:07Z] api.go:811: WARNING: using op 'self' when with Number='1' for 'AC_AMR' from '/RED/RESOURCES/bcbio/genomes/Hsapiens/hg38/variation/gnomad_exome.vcf.gz' can result in out-of-order values when the query is multi-allelic
[2019-10-05T20:07Z] api.go:812:        : this is not an issue if the query has been decomposed.
[2019-10-05T20:07Z] api.go:811: WARNING: using op 'self' when with Number='1' for 'AC_ASJ' from '/RED/RESOURCES/bcbio/genomes/Hsapiens/hg38/variation/gnomad_exome.vcf.gz' can result in out-of-order values when the query is multi-allelic
[2019-10-05T20:07Z] api.go:812:        : this is not an issue if the query has been decomposed.
[2019-10-05T20:07Z] api.go:811: WARNING: using op 'self' when with Number='1' for 'AC_EAS' from '/RED/RESOURCES/bcbio/genomes/Hsapiens/hg38/variation/gnomad_exome.vcf.gz' can result in out-of-order values when the query is multi-allelic
[2019-10-05T20:07Z] api.go:812:        : this is not an issue if the query has been decomposed.
[2019-10-05T20:07Z] api.go:811: WARNING: using op 'self' when with Number='1' for 'AC_FIN' from '/RED/RESOURCES/bcbio/genomes/Hsapiens/hg38/variation/gnomad_exome.vcf.gz' can result in out-of-order values when the query is multi-allelic
[2019-10-05T20:07Z] api.go:812:        : this is not an issue if the query has been decomposed.
[2019-10-05T20:07Z] api.go:811: WARNING: using op 'self' when with Number='1' for 'AC_NFE' from '/RED/RESOURCES/bcbio/genomes/Hsapiens/hg38/variation/gnomad_exome.vcf.gz' can result in out-of-order values when the query is multi-allelic
[2019-10-05T20:07Z] api.go:812:        : this is not an issue if the query has been decomposed.
[2019-10-05T20:07Z] api.go:811: WARNING: using op 'self' when with Number='1' for 'AC_OTH' from '/RED/RESOURCES/bcbio/genomes/Hsapiens/hg38/variation/gnomad_exome.vcf.gz' can result in out-of-order values when the query is multi-allelic
[2019-10-05T20:07Z] api.go:812:        : this is not an issue if the query has been decomposed.
[2019-10-05T20:07Z] api.go:811: WARNING: using op 'self' when with Number='1' for 'AC_SAS' from '/RED/RESOURCES/bcbio/genomes/Hsapiens/hg38/variation/gnomad_exome.vcf.gz' can result in out-of-order values when the query is multi-allelic
[2019-10-05T20:07Z] api.go:812:        : this is not an issue if the query has been decomposed.
[2019-10-05T20:07Z] api.go:811: WARNING: using op 'self' when with Number='1' for 'AC_POPMAX' from '/RED/RESOURCES/bcbio/genomes/Hsapiens/hg38/variation/gnomad_exome.vcf.gz' can result in out-of-order values when the query is multi-allelic
[2019-10-05T20:07Z] api.go:812:        : this is not an issue if the query has been decomposed.
[2019-10-05T20:07Z] api.go:811: WARNING: using op 'self' when with Number='1' for 'AN' from '/RED/RESOURCES/bcbio/genomes/Hsapiens/hg38/variation/gnomad_exome.vcf.gz' can result in out-of-order values when the query is multi-allelic
[2019-10-05T20:07Z] api.go:812:        : this is not an issue if the query has been decomposed.
[2019-10-05T20:07Z] api.go:811: WARNING: using op 'self' when with Number='1' for 'AN_ANR' from '/RED/RESOURCES/bcbio/genomes/Hsapiens/hg38/variation/gnomad_exome.vcf.gz' can result in out-of-order values when the query is multi-allelic
[2019-10-05T20:07Z] api.go:812:        : this is not an issue if the query has been decomposed.
[2019-10-05T20:07Z] api.go:811: WARNING: using op 'self' when with Number='1' for 'AN_AMR' from '/RED/RESOURCES/bcbio/genomes/Hsapiens/hg38/variation/gnomad_exome.vcf.gz' can result in out-of-order values when the query is multi-allelic
[2019-10-05T20:07Z] api.go:812:        : this is not an issue if the query has been decomposed.
[2019-10-05T20:07Z] api.go:811: WARNING: using op 'self' when with Number='1' for 'AN_ASJ' from '/RED/RESOURCES/bcbio/genomes/Hsapiens/hg38/variation/gnomad_exome.vcf.gz' can result in out-of-order values when the query is multi-allelic
[2019-10-05T20:07Z] api.go:812:        : this is not an issue if the query has been decomposed.
[2019-10-05T20:07Z] api.go:811: WARNING: using op 'self' when with Number='1' for 'AN_EAS' from '/RED/RESOURCES/bcbio/genomes/Hsapiens/hg38/variation/gnomad_exome.vcf.gz' can result in out-of-order values when the query is multi-allelic
[2019-10-05T20:07Z] api.go:812:        : this is not an issue if the query has been decomposed.
[2019-10-05T20:07Z] api.go:811: WARNING: using op 'self' when with Number='1' for 'AN_FIN' from '/RED/RESOURCES/bcbio/genomes/Hsapiens/hg38/variation/gnomad_exome.vcf.gz' can result in out-of-order values when the query is multi-allelic
[2019-10-05T20:07Z] api.go:812:        : this is not an issue if the query has been decomposed.
[2019-10-05T20:07Z] api.go:811: WARNING: using op 'self' when with Number='1' for 'AN_NFE' from '/RED/RESOURCES/bcbio/genomes/Hsapiens/hg38/variation/gnomad_exome.vcf.gz' can result in out-of-order values when the query is multi-allelic
[2019-10-05T20:07Z] api.go:812:        : this is not an issue if the query has been decomposed.
[2019-10-05T20:07Z] api.go:811: WARNING: using op 'self' when with Number='1' for 'AN_OTH' from '/RED/RESOURCES/bcbio/genomes/Hsapiens/hg38/variation/gnomad_exome.vcf.gz' can result in out-of-order values when the query is multi-allelic
[2019-10-05T20:07Z] api.go:812:        : this is not an issue if the query has been decomposed.
[2019-10-05T20:07Z] api.go:811: WARNING: using op 'self' when with Number='1' for 'AN_SAS' from '/RED/RESOURCES/bcbio/genomes/Hsapiens/hg38/variation/gnomad_exome.vcf.gz' can result in out-of-order values when the query is multi-allelic
[2019-10-05T20:07Z] api.go:812:        : this is not an issue if the query has been decomposed.
[2019-10-05T20:07Z] api.go:811: WARNING: using op 'self' when with Number='1' for 'AN_POPMAX' from '/RED/RESOURCES/bcbio/genomes/Hsapiens/hg38/variation/gnomad_exome.vcf.gz' can result in out-of-order values when the query is multi-allelic
[2019-10-05T20:07Z] api.go:812:        : this is not an issue if the query has been decomposed.
[2019-10-05T20:07Z] api.go:811: WARNING: using op 'self' when with Number='1' for 'POPMAX' from '/RED/RESOURCES/bcbio/genomes/Hsapiens/hg38/variation/gnomad_exome.vcf.gz' can result in out-of-order values when the query is multi-allelic
[2019-10-05T20:07Z] api.go:812:        : this is not an issue if the query has been decomposed.
[2019-10-05T20:07Z] api.go:811: WARNING: using op 'self' when with Number='1' for 'Hom' from '/RED/RESOURCES/bcbio/genomes/Hsapiens/hg38/variation/gnomad_exome.vcf.gz' can result in out-of-order values when the query is multi-allelic
[2019-10-05T20:07Z] api.go:812:        : this is not an issue if the query has been decomposed.
[2019-10-05T20:07Z] api.go:811: WARNING: using op 'self' when with Number='1' for 'Hom_Male' from '/RED/RESOURCES/bcbio/genomes/Hsapiens/hg38/variation/gnomad_exome.vcf.gz' can result in out-of-order values when the query is multi-allelic
[2019-10-05T20:07Z] api.go:812:        : this is not an issue if the query has been decomposed.
[2019-10-05T20:07Z] api.go:811: WARNING: using op 'self' when with Number='1' for 'Hom_Female' from '/RED/RESOURCES/bcbio/genomes/Hsapiens/hg38/variation/gnomad_exome.vcf.gz' can result in out-of-order values when the query is multi-allelic
[2019-10-05T20:07Z] api.go:812:        : this is not an issue if the query has been decomposed.
[2019-10-05T20:07Z] vcfanno.go:194: Info Error: max_aaf_all not found in INFO >> this error/warning may occur many times. reporting once here...
[2019-10-05T20:07Z] vcfanno.go:194: Info Error: Hom_Female not found in header >> this error/warning may occur many times. reporting once here...
[2019-10-05T20:07Z] vcfanno.go:194: Info Error: clinvar_sig not found in INFO >> this error/warning may occur many times. reporting once here...
[2019-10-05T20:07Z] vcfanno.go:194: Info Error: af_adj_exac_sas not found in INFO >> this error/warning may occur many times. reporting once here...
[2019-10-05T20:07Z] vcfanno.go:194: Info Error: af_esp_all not found in INFO >> this error/warning may occur many times. reporting once here...
[2019-10-05T20:07Z] vcfanno.go:194: Info Error: CLNDN not found in INFO >> this error/warning may occur many times. reporting once here...
[2019-10-05T20:11Z] bix.go:251: chromosome chrM not found in /RED/RESOURCES/bcbio/genomes/Hsapiens/hg38/variation/exac.vcf.gz
[2019-10-05T20:11Z] bix.go:251: chromosome chrM not found in /RED/RESOURCES/bcbio/genomes/Hsapiens/hg38/variation/esp.vcf.gz
[2019-10-05T20:11Z] vcfanno.go:248: annotated 148115 variants in 275.74 seconds (537.2 / second)
[2019-10-05T20:11Z] tabix index _mnt_36642bae-9ec9-4100-88a2-ac173a20ea16_WORKDIR_TRIO_BRAZILIANS_F178-gatk-haplotype-nomultiallelic-annotated-gemini.vcf.gz
[2019-10-05T20:11Z] GEMINI: create database with vcf2db
[2019-10-05T20:11Z] skipping 'MMQ' because it has Number=R
[2019-10-05T20:11Z] /RED/RESOURCES/bcbio/anaconda/envs/python2/lib/python2.7/site-packages/sqlalchemy/sql/sqltypes.py:269: SAWarning: Unicode type received non-unicode bind param value '_mnt_36642bae-9ec9-4100-8...'. (this warning may be suppressed after 10 occurrences)
[2019-10-05T20:11Z]   (util.ellipses_string(value),),
[2019-10-05T20:11Z] /RED/RESOURCES/bcbio/anaconda/envs/python2/lib/python2.7/site-packages/sqlalchemy/sql/sqltypes.py:269: SAWarning: Unicode type received non-unicode bind param value '-9'. (this warning may be suppressed after 10 occurrences)
[2019-10-05T20:11Z]   (util.ellipses_string(value),),
[2019-10-05T20:11Z] /RED/RESOURCES/bcbio/anaconda/envs/python2/lib/python2.7/site-packages/sqlalchemy/sql/sqltypes.py:269: SAWarning: Unicode type received non-unicode bind param value '2'. (this warning may be suppressed after 10 occurrences)
[2019-10-05T20:11Z]   (util.ellipses_string(value),),
[2019-10-05T20:11Z] bad record:
[2019-10-05T20:11Z] AC 2
[2019-10-05T20:11Z] AF 0.333000004292
[2019-10-05T20:11Z] AN 6
[2019-10-05T20:11Z] BaseQRankSum -1.42299997807
[2019-10-05T20:11Z] ClippingRankSum 0.736999988556
[2019-10-05T20:11Z] DB None
[2019-10-05T20:11Z] DECOMPOSED None
[2019-10-05T20:11Z] DP 49
[2019-10-05T20:11Z] ExcessHet 3.97939991951
[2019-10-05T20:11Z] FS 0.0
[2019-10-05T20:11Z] LEN None
[2019-10-05T20:11Z] MLEAC 2
[2019-10-05T20:11Z] MLEAF 0.333000004292
[2019-10-05T20:11Z] MMQ (40, 40)
[2019-10-05T20:11Z] MQ 37.5800018311
[2019-10-05T20:11Z] MQ0 0
[2019-10-05T20:11Z] MQRankSum -2.37800002098
[2019-10-05T20:11Z] OLD_MULTIALLELIC None
[2019-10-05T20:11Z] OLD_VARIANT None
[2019-10-05T20:11Z] QD 7.92000007629
[2019-10-05T20:11Z] ReadPosRankSum -1.04200005531
[2019-10-05T20:11Z] SOR 0.675000011921
[2019-10-05T20:11Z] TYPE None
[2019-10-05T20:11Z] aa_change None
[2019-10-05T20:11Z] aa_length None
[2019-10-05T20:11Z] aaf 0.333333333333
[2019-10-05T20:11Z] ac 2
[2019-10-05T20:11Z] ac_adj_exac_afr 0.0
[2019-10-05T20:11Z] ac_adj_exac_amr 2.0
[2019-10-05T20:11Z] ac_adj_exac_eas 0.0
[2019-10-05T20:11Z] ac_adj_exac_fin 0.0
[2019-10-05T20:11Z] ac_adj_exac_nfe 0.0
[2019-10-05T20:11Z] ac_adj_exac_oth 0.0
[2019-10-05T20:11Z] ac_adj_exac_sas 0.0
[2019-10-05T20:11Z] ac_exac_all 2.0
[2019-10-05T20:11Z] af 0.333000004292
[2019-10-05T20:11Z] af_adj_exac_afr 0.0
[2019-10-05T20:11Z] af_adj_exac_amr 0.000295420002658
[2019-10-05T20:11Z] af_adj_exac_eas 0.0
[2019-10-05T20:11Z] af_adj_exac_fin 0.0
[2019-10-05T20:11Z] af_adj_exac_nfe 0.0
[2019-10-05T20:11Z] af_adj_exac_oth 0.0
[2019-10-05T20:11Z] af_adj_exac_sas 0.0
[2019-10-05T20:11Z] af_esp_aa -1.0
[2019-10-05T20:11Z] af_esp_all -1.0
[2019-10-05T20:11Z] af_esp_ea -1.0
[2019-10-05T20:11Z] af_exac_all 5.88199982303e-05
[2019-10-05T20:11Z] alt A
[2019-10-05T20:11Z] an 6
[2019-10-05T20:11Z] an_adj_exac_afr 1262.0
[2019-10-05T20:11Z] an_adj_exac_amr 6770.0
[2019-10-05T20:11Z] an_adj_exac_eas 2486.0
[2019-10-05T20:11Z] an_adj_exac_fin 1828.0
[2019-10-05T20:11Z] an_adj_exac_nfe 16896.0
[2019-10-05T20:11Z] an_adj_exac_oth 240.0
[2019-10-05T20:11Z] an_adj_exac_sas 4520.0
[2019-10-05T20:11Z] an_exac_all 34002.0
[2019-10-05T20:11Z] baseqranksum -1.42299997807
[2019-10-05T20:11Z] biotype None
[2019-10-05T20:11Z] call_rate 1.0
[2019-10-05T20:11Z] chrom chr1
[2019-10-05T20:11Z] clinvar_disease_name None
[2019-10-05T20:11Z] clinvar_sig None
[2019-10-05T20:11Z] clippingranksum 0.736999988556
[2019-10-05T20:11Z] codon_change None
[2019-10-05T20:11Z] common_pathogenic False
[2019-10-05T20:11Z] db False
[2019-10-05T20:11Z] decomposed False
[2019-10-05T20:11Z] dp 49
[2019-10-05T20:11Z] ds False
[2019-10-05T20:11Z] effect_severity None
[2019-10-05T20:11Z] end 146984765
[2019-10-05T20:11Z] ensembl_gene_id None
[2019-10-05T20:11Z] excesshet 3.97939991951
[2019-10-05T20:11Z] exon None
[2019-10-05T20:11Z] filter None
[2019-10-05T20:11Z] fs 0.0
[2019-10-05T20:11Z] gene None
[2019-10-05T20:11Z] gnomAD_AC 8
[2019-10-05T20:11Z] gnomAD_AF 0.000136870003189
[2019-10-05T20:11Z] gnomAD_AN (79506, 58450)
[2019-10-05T20:11Z] gnomad_ac 8
[2019-10-05T20:11Z] gnomad_af 0.000136870003189
[2019-10-05T20:11Z] gnomad_af_afr -1.0
[2019-10-05T20:11Z] gnomad_af_amr -1.0
[2019-10-05T20:11Z] gnomad_af_asj -1.0
[2019-10-05T20:11Z] gnomad_af_eas -1.0
[2019-10-05T20:11Z] gnomad_af_fin -1.0
[2019-10-05T20:11Z] gnomad_af_nfe -1.0
[2019-10-05T20:11Z] gnomad_af_oth -1.0
[2019-10-05T20:11Z] gnomad_af_popmax -1.0
[2019-10-05T20:11Z] gnomad_af_sas -1.0
[2019-10-05T20:11Z] gnomad_an (79506, 58450)
[2019-10-05T20:11Z] gt_alt_depths i
                                   ,

[2019-10-05T20:11Z] gt_alt_freqs dUD�?�q�q�?
[2019-10-05T20:11Z] gt_depths i
                               ,$
[2019-10-05T20:11Z] gt_phases ?
[2019-10-05T20:11Z] gt_quals f
                              ,�B@A0A
[2019-10-05T20:11Z] gt_ref_depths i
                                   ,
[2019-10-05T20:11Z] gt_types i
                              ,
[2019-10-05T20:11Z] gts S
                         (G/AG/GG/A
[2019-10-05T20:11Z] impact None
[2019-10-05T20:11Z] impact_severity None
[2019-10-05T20:11Z] impact_so None
[2019-10-05T20:11Z] is_canonical False
[2019-10-05T20:11Z] is_coding False
[2019-10-05T20:11Z] is_exonic False
[2019-10-05T20:11Z] is_lof False
[2019-10-05T20:11Z] is_splicing False
[2019-10-05T20:11Z] len None
[2019-10-05T20:11Z] max_aaf_all 0.000295420002658
[2019-10-05T20:11Z] mleac 2
[2019-10-05T20:11Z] mleaf 0.333000004292
[2019-10-05T20:11Z] mmq (40, 40)
[2019-10-05T20:11Z] mq 37.5800018311
[2019-10-05T20:11Z] mq0 0
[2019-10-05T20:11Z] mqranksum -2.37800002098
[2019-10-05T20:11Z] num_exac_Het 2.0
[2019-10-05T20:11Z] num_exac_Hom 0.0
[2019-10-05T20:11Z] num_exac_het 2.0
[2019-10-05T20:11Z] num_exac_hom 0.0
[2019-10-05T20:11Z] num_het 2
[2019-10-05T20:11Z] num_hom_alt 0
[2019-10-05T20:11Z] num_hom_ref 1
[2019-10-05T20:11Z] num_unknown 0
[2019-10-05T20:11Z] old_multiallelic None
[2019-10-05T20:11Z] old_variant None
[2019-10-05T20:11Z] polyphen_pred None
[2019-10-05T20:11Z] polyphen_score None
[2019-10-05T20:11Z] qd 7.92000007629
[2019-10-05T20:11Z] qual<PHONE_NUMBER>07
[2019-10-05T20:11Z] readposranksum -1.04200005531
[2019-10-05T20:11Z] ref G
[2019-10-05T20:11Z] rs_ids rs1437329596
[2019-10-05T20:11Z] sift_pred None
[2019-10-05T20:11Z] sift_score None
[2019-10-05T20:11Z] so None
[2019-10-05T20:11Z] sor 0.675000011921
[2019-10-05T20:11Z] start 146984764
[2019-10-05T20:11Z] sub_type ts
[2019-10-05T20:11Z] top_consequence None
[2019-10-05T20:11Z] transcript None
[2019-10-05T20:11Z] type snp
[2019-10-05T20:11Z] variant_id 7671
[2019-10-05T20:11Z] vcf_id rs1437329596
[2019-10-05T20:11Z] Traceback (most recent call last):
[2019-10-05T20:11Z]   File "/RED/TOOLS/bin/vcf2db.py", line 924, in <module>
[2019-10-05T20:11Z]     impacts_extras=a.impacts_field, aok=a.a_ok)
[2019-10-05T20:11Z]   File "/RED/TOOLS/bin/vcf2db.py", line 234, in __init__
[2019-10-05T20:11Z]     self.load()
[2019-10-05T20:11Z]   File "/RED/TOOLS/bin/vcf2db.py", line 319, in load
[2019-10-05T20:11Z]     i = self._load(self.cache, create=True, start=1)
[2019-10-05T20:11Z]   File "/RED/TOOLS/bin/vcf2db.py", line 312, in _load
[2019-10-05T20:11Z]     self.insert(variants, expanded, keys, i, create=create)
[2019-10-05T20:11Z]   File "/RED/TOOLS/bin/vcf2db.py", line 374, in insert
[2019-10-05T20:11Z]     vilengths, variant_impacts)
[2019-10-05T20:11Z]   File "/RED/TOOLS/bin/vcf2db.py", line 402, in _insert
[2019-10-05T20:11Z]     self.__insert(v_objs, self.metadata.tables['variants'].insert())
[2019-10-05T20:11Z]   File "/RED/TOOLS/bin/vcf2db.py", line 436, in __insert
[2019-10-05T20:11Z]     raise e
[2019-10-05T20:11Z] sqlalchemy.exc.InterfaceError: (sqlite3.InterfaceError) Error binding parameter 100 - probably unsupported type.
[2019-10-05T20:11Z] [SQL: INSERT INTO variants (variant_id, chrom, start, "end", vcf_id, ref, alt, qual, filter, type, sub_type, call_rate, num_hom_ref, num_het, num_hom_alt, num_unknown, aaf, gene, ensembl_gene_id, transcript, is_exonic, is_coding, is_lof, is_splicing, is_canonical, exon, codon_change, aa_change, aa_length, biotype, impact, impact_so, impact_severity, polyphen_pred, polyphen_score, sift_pred, sift_score, ac, af, an, baseqranksum, clippingranksum, db, decomposed, dp, ds, excesshet, fs, len, mleac, mleaf, mq, mq0, mqranksum, old_multiallelic, old_variant, qd, readposranksum, sor, ac_adj_exac_afr, ac_adj_exac_amr, ac_adj_exac_eas, ac_adj_exac_fin, ac_adj_exac_nfe, ac_adj_exac_oth, ac_adj_exac_sas, ac_exac_all, af_adj_exac_afr, af_adj_exac_amr, af_adj_exac_eas, af_adj_exac_fin, af_adj_exac_nfe, af_adj_exac_oth, af_adj_exac_sas, af_esp_aa, af_esp_all, af_esp_ea, af_exac_all, an_adj_exac_afr, an_adj_exac_amr, an_adj_exac_eas, an_adj_exac_fin, an_adj_exac_nfe, an_adj_exac_oth, an_adj_exac_sas, an_exac_all, clinvar_disease_name, clinvar_sig, common_pathogenic, gnomad_ac, gnomad_af, gnomad_af_afr, gnomad_af_amr, gnomad_af_asj, gnomad_af_eas, gnomad_af_fin, gnomad_af_nfe, gnomad_af_oth, gnomad_af_popmax, gnomad_af_sas, gnomad_an, max_aaf_all, num_exac_het, num_exac_hom, rs_ids, gts, gt_types, gt_phases, gt_depths, gt_ref_depths, gt_alt_depths, gt_quals, gt_alt_freqs) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)]
[2019-10-05T20:11Z] [parameters: (7671, u'chr1', 146984764, 146984765, u'rs1437329596', u'G', u'A',<PHONE_NUMBER>0703125, None, 'snp', 'ts', 1.0, 1, 2, 0, 0, 0.3333333333333333, None, None, None, 0, 0, 0, 0, 0, None, None, None, None, None, None, None, None, None, None, None, None, 2, 0.3330000042915344, 6, -1.4229999780654907, 0.7369999885559082, 0, 0, 49, 0, 3.9793999195098877, 0.0, None, 2, 0.3330000042915344, 37.58000183105469, 0, -2.378000020980835, None, 'None', 7.920000076293945, -1.0420000553131104, 0.675000011920929, 0.0, 2.0, 0.0, 0.0, 0.0, 0.0, 0.0, 2.0, 0.0, 0.0002954200026579201, 0.0, 0.0, 0.0, 0.0, 0.0, -1.0, -1.0, -1.0, 5.881999823031947e-05, 1262.0, 6770.0, 2486.0, 1828.0, 16896.0, 240.0, 4520.0, 34002.0, None, None, 0, 8, 0.00013687000318896025, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, (79506, 58450), 0.0002954200026579201, 2.0, 0.0, u'rs1437329596', <read-only buffer for 0x7f452b0f9148, size -1, offset 0 at 0x7f452a506270>, <read-only buffer for 0x7f452b0f9110, size -1, offset 0 at 0x7f452a506230>, <read-only buffer for 0x7f452b0f7ea0, size -1, offset 0 at 0x7f452a5061f0>, <read-only buffer for 0x7f452b0f9180, size -1, offset 0 at 0x7f452a5061b0>, <read-only buffer for 0x7f452b0f91b8, size -1, offset 0 at 0x7f452a506170>, <read-only buffer for 0x7f452b0f91f0, size -1, offset 0 at 0x7f452a506130>, <read-only buffer for 0x7f452b0f9228, size -1, offset 0 at 0x7f452a5060f0>, <read-only buffer for 0x7f452b0f84b0, size -1, offset 0 at 0x7f452a5060b0>)]
[2019-10-05T20:11Z] (Background on this error at: http://sqlalche.me/e/rvf5)
[2019-10-05T20:11Z] Uncaught exception occurred
Traceback (most recent call last):
  File "/RED/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/bcbio/provenance/do.py", line 26, in run
    _do_run(cmd, checks, log_stdout, env=env)
  File "/RED/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/bcbio/provenance/do.py", line 106, in _do_run
    raise subprocess.CalledProcessError(exitcode, error_msg)
subprocess.CalledProcessError: Command '/RED/TOOLS/bin/vcf2db.py /mnt/36642bae-9ec9-4100-88a2-ac173a20ea16/WORKDIR/TRIO_BRAZILIANS/F178/work/gemini/_mnt_36642bae-9ec9-4100-88a2-ac173a20ea16_WORKDIR_TRIO_BRAZILIANS_F178-gatk-haplotype-nomultiallelic-annotated-gemini.vcf.gz /mnt/36642bae-9ec9-4100-88a2-ac173a20ea16/WORKDIR/TRIO_BRAZILIANS/F178/work/gemini/_mnt_36642bae-9ec9-4100-88a2-ac173a20ea16_WORKDIR_TRIO_BRAZILIANS_F178-gatk-haplotype-nomultiallelic.ped /mnt/36642bae-9ec9-4100-88a2-ac173a20ea16/WORKDIR/TRIO_BRAZILIANS/F178/work/bcbiotx/tmpna8xq85x/_mnt_36642bae-9ec9-4100-88a2-ac173a20ea16_WORKDIR_TRIO_BRAZILIANS_F178-gatk-haplotype.db
codon_change None
common_pathogenic False
db False
decomposed False
dp 49
ds False
effect_severity None
end 146984765
ensembl_gene_id None
excesshet 3.97939991951
exon None
filter None
fs 0.0
gene None
gnomAD_AC 8
gnomAD_AF 0.000136870003189
gnomAD_AN (79506, 58450)
gnomad_ac 8
gnomad_af 0.000136870003189
gnomad_af_afr -1.0
gnomad_af_amr -1.0
gnomad_af_asj -1.0
gnomad_af_eas -1.0
gnomad_af_fin -1.0
gnomad_af_nfe -1.0
gnomad_af_oth -1.0
gnomad_af_popmax -1.0
gnomad_af_sas -1.0
gnomad_an (79506, 58450)
gt_alt_depths i
               ,

gt_alt_freqs dUD�?�q�q�?
gt_depths i
           ,$
gt_phases ?
gt_quals f
          ,�B@A0A
gt_ref_depths i
               ,
gt_types i
          ,
gts S
     (G/AG/GG/A
impact None
impact_severity None
impact_so None
is_canonical False
is_coding False
is_exonic False
is_lof False
is_splicing False
len None
max_aaf_all 0.000295420002658
mleac 2
mleaf 0.333000004292
mmq (40, 40)
mq 37.5800018311
mq0 0
mqranksum -2.37800002098
num_exac_Het 2.0
num_exac_Hom 0.0
num_exac_het 2.0
num_exac_hom 0.0
num_het 2
num_hom_alt 0
num_hom_ref 1
num_unknown 0
old_multiallelic None
old_variant None
polyphen_pred None
polyphen_score None
qd 7.92000007629
qual<PHONE_NUMBER>07
readposranksum -1.04200005531
ref G
rs_ids rs1437329596
sift_pred None
sift_score None
so None
sor 0.675000011921
start 146984764
sub_type ts
top_consequence None
transcript None
type snp
variant_id 7671
vcf_id rs1437329596
Traceback (most recent call last):
  File "/RED/TOOLS/bin/vcf2db.py", line 924, in <module>
    impacts_extras=a.impacts_field, aok=a.a_ok)
  File "/RED/TOOLS/bin/vcf2db.py", line 234, in __init__
    self.load()
  File "/RED/TOOLS/bin/vcf2db.py", line 319, in load
    i = self._load(self.cache, create=True, start=1)
  File "/RED/TOOLS/bin/vcf2db.py", line 312, in _load
    self.insert(variants, expanded, keys, i, create=create)
  File "/RED/TOOLS/bin/vcf2db.py", line 374, in insert
    vilengths, variant_impacts)
  File "/RED/TOOLS/bin/vcf2db.py", line 402, in _insert
    self.__insert(v_objs, self.metadata.tables['variants'].insert())
  File "/RED/TOOLS/bin/vcf2db.py", line 436, in __insert
    raise e
sqlalchemy.exc.InterfaceError: (sqlite3.InterfaceError) Error binding parameter 100 - probably unsupported type.

I am very frustrated. Neither the PED nor the VCFANNO file cause the problem. Do you want me to upload the fastq files so you can have a look?

I have uploaded he whole project folder so you can have a look!
https://drive.google.com/drive/folders/1BStwCZFMcFL7-6AruM723mbpSWH0XozS?usp=sharing

Hi @kokyriakidis  and everyone!
I missed a huge chunk of this discussion :)
Also, maybe using batch: 178F is safer than the huge string you are using.
SN

Hi Sergey!
I would like to ask you about the vcf annotation file in cre and the warnings it throws when I run it with it. Did something change in the annotation fields?

Sure, let us just finish first with the failure.
The last command which fails is
vcf2db.py \
_mnt_36642bae-9ec9-4100-88a2-ac173a20ea16_WORKDIR_TRIO_BRAZILIANS_F178-gatk-haplotype-nomultiallelic-annotated-gemini.vcf.gz \
_mnt_36642bae-9ec9-4100-88a2-ac173a20ea16_WORKDIR_TRIO_BRAZILIANS_F178-gatk-haplotype-nomultiallelic.ped \
_mnt_36642bae-9ec9-4100-88a2-ac173a20ea16_WORKDIR_TRIO_BRAZILIANS_F178-gatk-haplotype.db

I fails for me as well, complaining about field 100 in SQL query for variant rs1437329596.
It is indeed a bad one: gnomAD_AN=79506,58450
Should be just one value. Now let us figure out where it came from. In your config you don't have vcfanno, so it comes from VEP? But you don't have CSQ field, so it is not VEP.  Probably, you mixed several bcbio runs with different yaml settings.
Next I'll try to check this variant in my copy of gnomad exome and rerun your trio from scratch.
Sergey

@naumenko-sa

deleted

Rerun without vcfanno file and still get these errors:

[2019-10-06T10:22Z] tabix index _mnt_36642bae-9ec9-4100-88a2-ac173a20ea16_WORKDIR_TRIO_BRAZILIANS_F178-gatk-haplotype-decompose.vcf.gz
[2019-10-06T10:22Z] Annotating /mnt/36642bae-9ec9-4100-88a2-ac173a20ea16/WORKDIR/TRIO_BRAZILIANS/F178/work/gemini/_mnt_36642bae-9ec9-4100-88a2-ac173a20ea16_WORKDIR_TRIO_BRAZILIANS_F178-gatk-haplotype-nomultiallelic.vcf.gz with vcfanno, using /mnt/36642bae-9ec9-4100-88a2-ac173a20ea16/WORKDIR/TRIO_BRAZILIANS/F178/work/gemini/_mnt_36642bae-9ec9-4100-88a2-ac173a20ea16_WORKDIR_TRIO_BRAZILIANS_F178-gatk-haplotype-nomultiallelic-annotated-gemini-combine.conf
[2019-10-06T10:22Z] =============================================
[2019-10-06T10:22Z] vcfanno version 0.3.2 [built with go1.12.1]
[2019-10-06T10:22Z] see: https://github.com/brentp/vcfanno
[2019-10-06T10:22Z] =============================================
[2019-10-06T10:22Z] vcfanno.go:115: found 61 sources from 5 files
[2019-10-06T10:22Z] vcfanno.go:145: using 2 worker threads to decompress bgzip file
[2019-10-06T10:22Z] api.go:811: WARNING: using op 'self' when with Number='1' for 'AF_AFR' from '/RED/RESOURCES/bcbio/genomes/Hsapiens/hg38/variation/gnomad_exome.vcf.gz' can result in out-of-order values when the query is multi-allelic
[2019-10-06T10:22Z] api.go:812:        : this is not an issue if the query has been decomposed.
[2019-10-06T10:22Z] api.go:811: WARNING: using op 'self' when with Number='1' for 'AF_AMR' from '/RED/RESOURCES/bcbio/genomes/Hsapiens/hg38/variation/gnomad_exome.vcf.gz' can result in out-of-order values when the query is multi-allelic
[2019-10-06T10:22Z] api.go:812:        : this is not an issue if the query has been decomposed.
[2019-10-06T10:22Z] api.go:811: WARNING: using op 'self' when with Number='1' for 'AF_ASJ' from '/RED/RESOURCES/bcbio/genomes/Hsapiens/hg38/variation/gnomad_exome.vcf.gz' can result in out-of-order values when the query is multi-allelic
[2019-10-06T10:22Z] api.go:812:        : this is not an issue if the query has been decomposed.
[2019-10-06T10:22Z] api.go:811: WARNING: using op 'self' when with Number='1' for 'AF_EAS' from '/RED/RESOURCES/bcbio/genomes/Hsapiens/hg38/variation/gnomad_exome.vcf.gz' can result in out-of-order values when the query is multi-allelic
[2019-10-06T10:22Z] api.go:812:        : this is not an issue if the query has been decomposed.
[2019-10-06T10:22Z] api.go:811: WARNING: using op 'self' when with Number='1' for 'AF_FIN' from '/RED/RESOURCES/bcbio/genomes/Hsapiens/hg38/variation/gnomad_exome.vcf.gz' can result in out-of-order values when the query is multi-allelic
[2019-10-06T10:22Z] api.go:812:        : this is not an issue if the query has been decomposed.
[2019-10-06T10:22Z] api.go:811: WARNING: using op 'self' when with Number='1' for 'AF_NFE' from '/RED/RESOURCES/bcbio/genomes/Hsapiens/hg38/variation/gnomad_exome.vcf.gz' can result in out-of-order values when the query is multi-allelic
[2019-10-06T10:22Z] api.go:812:        : this is not an issue if the query has been decomposed.
[2019-10-06T10:22Z] api.go:811: WARNING: using op 'self' when with Number='1' for 'AF_OTH' from '/RED/RESOURCES/bcbio/genomes/Hsapiens/hg38/variation/gnomad_exome.vcf.gz' can result in out-of-order values when the query is multi-allelic
[2019-10-06T10:22Z] api.go:812:        : this is not an issue if the query has been decomposed.
[2019-10-06T10:22Z] api.go:811: WARNING: using op 'self' when with Number='1' for 'AF_SAS' from '/RED/RESOURCES/bcbio/genomes/Hsapiens/hg38/variation/gnomad_exome.vcf.gz' can result in out-of-order values when the query is multi-allelic
[2019-10-06T10:22Z] api.go:812:        : this is not an issue if the query has been decomposed.
[2019-10-06T10:22Z] api.go:811: WARNING: using op 'self' when with Number='1' for 'AF_POPMAX' from '/RED/RESOURCES/bcbio/genomes/Hsapiens/hg38/variation/gnomad_exome.vcf.gz' can result in out-of-order values when the query is multi-allelic
[2019-10-06T10:22Z] api.go:812:        : this is not an issue if the query has been decomposed.
[2019-10-06T10:22Z] api.go:811: WARNING: using op 'self' when with Number='1' for 'AC_ACR' from '/RED/RESOURCES/bcbio/genomes/Hsapiens/hg38/variation/gnomad_exome.vcf.gz' can result in out-of-order values when the query is multi-allelic
[2019-10-06T10:22Z] api.go:812:        : this is not an issue if the query has been decomposed.
[2019-10-06T10:22Z] api.go:811: WARNING: using op 'self' when with Number='1' for 'AC_AMR' from '/RED/RESOURCES/bcbio/genomes/Hsapiens/hg38/variation/gnomad_exome.vcf.gz' can result in out-of-order values when the query is multi-allelic
[2019-10-06T10:22Z] api.go:812:        : this is not an issue if the query has been decomposed.
[2019-10-06T10:22Z] api.go:811: WARNING: using op 'self' when with Number='1' for 'AC_ASJ' from '/RED/RESOURCES/bcbio/genomes/Hsapiens/hg38/variation/gnomad_exome.vcf.gz' can result in out-of-order values when the query is multi-allelic
[2019-10-06T10:22Z] api.go:812:        : this is not an issue if the query has been decomposed.
[2019-10-06T10:22Z] api.go:811: WARNING: using op 'self' when with Number='1' for 'AC_EAS' from '/RED/RESOURCES/bcbio/genomes/Hsapiens/hg38/variation/gnomad_exome.vcf.gz' can result in out-of-order values when the query is multi-allelic
[2019-10-06T10:22Z] api.go:812:        : this is not an issue if the query has been decomposed.
[2019-10-06T10:22Z] api.go:811: WARNING: using op 'self' when with Number='1' for 'AC_FIN' from '/RED/RESOURCES/bcbio/genomes/Hsapiens/hg38/variation/gnomad_exome.vcf.gz' can result in out-of-order values when the query is multi-allelic
[2019-10-06T10:22Z] api.go:812:        : this is not an issue if the query has been decomposed.
[2019-10-06T10:22Z] api.go:811: WARNING: using op 'self' when with Number='1' for 'AC_NFE' from '/RED/RESOURCES/bcbio/genomes/Hsapiens/hg38/variation/gnomad_exome.vcf.gz' can result in out-of-order values when the query is multi-allelic
[2019-10-06T10:22Z] api.go:812:        : this is not an issue if the query has been decomposed.
[2019-10-06T10:22Z] api.go:811: WARNING: using op 'self' when with Number='1' for 'AC_OTH' from '/RED/RESOURCES/bcbio/genomes/Hsapiens/hg38/variation/gnomad_exome.vcf.gz' can result in out-of-order values when the query is multi-allelic
[2019-10-06T10:22Z] api.go:812:        : this is not an issue if the query has been decomposed.
[2019-10-06T10:22Z] api.go:811: WARNING: using op 'self' when with Number='1' for 'AC_SAS' from '/RED/RESOURCES/bcbio/genomes/Hsapiens/hg38/variation/gnomad_exome.vcf.gz' can result in out-of-order values when the query is multi-allelic
[2019-10-06T10:22Z] api.go:812:        : this is not an issue if the query has been decomposed.
[2019-10-06T10:22Z] api.go:811: WARNING: using op 'self' when with Number='1' for 'AC_POPMAX' from '/RED/RESOURCES/bcbio/genomes/Hsapiens/hg38/variation/gnomad_exome.vcf.gz' can result in out-of-order values when the query is multi-allelic
[2019-10-06T10:22Z] api.go:812:        : this is not an issue if the query has been decomposed.
[2019-10-06T10:22Z] api.go:811: WARNING: using op 'self' when with Number='1' for 'AN' from '/RED/RESOURCES/bcbio/genomes/Hsapiens/hg38/variation/gnomad_exome.vcf.gz' can result in out-of-order values when the query is multi-allelic
[2019-10-06T10:22Z] api.go:812:        : this is not an issue if the query has been decomposed.
[2019-10-06T10:22Z] api.go:811: WARNING: using op 'self' when with Number='1' for 'AN_ANR' from '/RED/RESOURCES/bcbio/genomes/Hsapiens/hg38/variation/gnomad_exome.vcf.gz' can result in out-of-order values when the query is multi-allelic
[2019-10-06T10:22Z] api.go:812:        : this is not an issue if the query has been decomposed.
[2019-10-06T10:22Z] api.go:811: WARNING: using op 'self' when with Number='1' for 'AN_AMR' from '/RED/RESOURCES/bcbio/genomes/Hsapiens/hg38/variation/gnomad_exome.vcf.gz' can result in out-of-order values when the query is multi-allelic
[2019-10-06T10:22Z] api.go:812:        : this is not an issue if the query has been decomposed.
[2019-10-06T10:22Z] api.go:811: WARNING: using op 'self' when with Number='1' for 'AN_ASJ' from '/RED/RESOURCES/bcbio/genomes/Hsapiens/hg38/variation/gnomad_exome.vcf.gz' can result in out-of-order values when the query is multi-allelic
[2019-10-06T10:22Z] api.go:812:        : this is not an issue if the query has been decomposed.
[2019-10-06T10:22Z] api.go:811: WARNING: using op 'self' when with Number='1' for 'AN_EAS' from '/RED/RESOURCES/bcbio/genomes/Hsapiens/hg38/variation/gnomad_exome.vcf.gz' can result in out-of-order values when the query is multi-allelic
[2019-10-06T10:22Z] api.go:812:        : this is not an issue if the query has been decomposed.
[2019-10-06T10:22Z] api.go:811: WARNING: using op 'self' when with Number='1' for 'AN_FIN' from '/RED/RESOURCES/bcbio/genomes/Hsapiens/hg38/variation/gnomad_exome.vcf.gz' can result in out-of-order values when the query is multi-allelic
[2019-10-06T10:22Z] api.go:812:        : this is not an issue if the query has been decomposed.
[2019-10-06T10:22Z] api.go:811: WARNING: using op 'self' when with Number='1' for 'AN_NFE' from '/RED/RESOURCES/bcbio/genomes/Hsapiens/hg38/variation/gnomad_exome.vcf.gz' can result in out-of-order values when the query is multi-allelic
[2019-10-06T10:22Z] api.go:812:        : this is not an issue if the query has been decomposed.
[2019-10-06T10:22Z] api.go:811: WARNING: using op 'self' when with Number='1' for 'AN_OTH' from '/RED/RESOURCES/bcbio/genomes/Hsapiens/hg38/variation/gnomad_exome.vcf.gz' can result in out-of-order values when the query is multi-allelic
[2019-10-06T10:22Z] api.go:812:        : this is not an issue if the query has been decomposed.
[2019-10-06T10:22Z] api.go:811: WARNING: using op 'self' when with Number='1' for 'AN_SAS' from '/RED/RESOURCES/bcbio/genomes/Hsapiens/hg38/variation/gnomad_exome.vcf.gz' can result in out-of-order values when the query is multi-allelic
[2019-10-06T10:22Z] api.go:812:        : this is not an issue if the query has been decomposed.
[2019-10-06T10:22Z] api.go:811: WARNING: using op 'self' when with Number='1' for 'AN_POPMAX' from '/RED/RESOURCES/bcbio/genomes/Hsapiens/hg38/variation/gnomad_exome.vcf.gz' can result in out-of-order values when the query is multi-allelic
[2019-10-06T10:22Z] api.go:812:        : this is not an issue if the query has been decomposed.
[2019-10-06T10:22Z] api.go:811: WARNING: using op 'self' when with Number='1' for 'POPMAX' from '/RED/RESOURCES/bcbio/genomes/Hsapiens/hg38/variation/gnomad_exome.vcf.gz' can result in out-of-order values when the query is multi-allelic
[2019-10-06T10:22Z] api.go:812:        : this is not an issue if the query has been decomposed.
[2019-10-06T10:22Z] api.go:811: WARNING: using op 'self' when with Number='1' for 'Hom' from '/RED/RESOURCES/bcbio/genomes/Hsapiens/hg38/variation/gnomad_exome.vcf.gz' can result in out-of-order values when the query is multi-allelic
[2019-10-06T10:22Z] api.go:812:        : this is not an issue if the query has been decomposed.
[2019-10-06T10:22Z] api.go:811: WARNING: using op 'self' when with Number='1' for 'Hom_Male' from '/RED/RESOURCES/bcbio/genomes/Hsapiens/hg38/variation/gnomad_exome.vcf.gz' can result in out-of-order values when the query is multi-allelic
[2019-10-06T10:22Z] api.go:812:        : this is not an issue if the query has been decomposed.
[2019-10-06T10:22Z] api.go:811: WARNING: using op 'self' when with Number='1' for 'Hom_Female' from '/RED/RESOURCES/bcbio/genomes/Hsapiens/hg38/variation/gnomad_exome.vcf.gz' can result in out-of-order values when the query is multi-allelic
[2019-10-06T10:22Z] api.go:812:        : this is not an issue if the query has been decomposed.
[2019-10-06T10:22Z] vcfanno.go:194: Info Error: max_aaf_all not found in INFO >> this error/warning may occur many times. reporting once here...
[2019-10-06T10:22Z] vcfanno.go:194: Info Error: Hom_Female not found in header >> this error/warning may occur many times. reporting once here...
[2019-10-06T10:22Z] vcfanno.go:194: Info Error: clinvar_sig not found in INFO >> this error/warning may occur many times. reporting once here...
[2019-10-06T10:22Z] vcfanno.go:194: Info Error: af_adj_exac_sas not found in INFO >> this error/warning may occur many times. reporting once here...
[2019-10-06T10:22Z] vcfanno.go:194: Info Error: af_esp_all not found in INFO >> this error/warning may occur many times. reporting once here...
[2019-10-06T10:22Z] vcfanno.go:194: Info Error: CLNDN not found in INFO >> this error/warning may occur many times. reporting once here...
[2019-10-06T10:26Z] bix.go:251: chromosome chrM not found in /RED/RESOURCES/bcbio/genomes/Hsapiens/hg38/variation/exac.vcf.gz
[2019-10-06T10:26Z] bix.go:251: chromosome chrM not found in /RED/RESOURCES/bcbio/genomes/Hsapiens/hg38/variation/esp.vcf.gz
[2019-10-06T10:26Z] vcfanno.go:248: annotated 148115 variants in 275.46 seconds (537.7 / second)
[2019-10-06T10:27Z] tabix index _mnt_36642bae-9ec9-4100-88a2-ac173a20ea16_WORKDIR_TRIO_BRAZILIANS_F178-gatk-haplotype-nomultiallelic-annotated-gemini.vcf.gz
[2019-10-06T10:27Z] GEMINI: create database with vcf2db
[2019-10-06T10:27Z] skipping 'MMQ' because it has Number=R
[2019-10-06T10:27Z] Ethnicity None
[2019-10-06T10:27Z] Traceback (most recent call last):
[2019-10-06T10:27Z]   File "/RED/TOOLS/bin/vcf2db.py", line 924, in <module>
[2019-10-06T10:27Z]     impacts_extras=a.impacts_field, aok=a.a_ok)
[2019-10-06T10:27Z]   File "/RED/TOOLS/bin/vcf2db.py", line 228, in __init__
[2019-10-06T10:27Z]     self.samples = self.create_samples()
[2019-10-06T10:27Z]   File "/RED/TOOLS/bin/vcf2db.py", line 598, in create_samples
[2019-10-06T10:27Z]     vals = [r[i] for r in rows]
[2019-10-06T10:27Z] IndexError: list index out of range
[2019-10-06T10:27Z] Uncaught exception occurred
Traceback (most recent call last):
  File "/RED/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/bcbio/provenance/do.py", line 26, in run
    _do_run(cmd, checks, log_stdout, env=env)
  File "/RED/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/bcbio/provenance/do.py", line 106, in _do_run
    raise subprocess.CalledProcessError(exitcode, error_msg)
subprocess.CalledProcessError: Command '/RED/TOOLS/bin/vcf2db.py /mnt/36642bae-9ec9-4100-88a2-ac173a20ea16/WORKDIR/TRIO_BRAZILIANS/F178/work/gemini/_mnt_36642bae-9ec9-4100-88a2-ac173a20ea16_WORKDIR_TRIO_BRAZILIANS_F178-gatk-haplotype-nomultiallelic-annotated-gemini.vcf.gz /mnt/36642bae-9ec9-4100-88a2-ac173a20ea16/WORKDIR/TRIO_BRAZILIANS/F178/work/gemini/_mnt_36642bae-9ec9-4100-88a2-ac173a20ea16_WORKDIR_TRIO_BRAZILIANS_F178-gatk-haplotype-nomultiallelic.ped /mnt/36642bae-9ec9-4100-88a2-ac173a20ea16/WORKDIR/TRIO_BRAZILIANS/F178/work/bcbiotx/tmp71dgss81/_mnt_36642bae-9ec9-4100-88a2-ac173a20ea16_WORKDIR_TRIO_BRAZILIANS_F178-gatk-haplotype.db
skipping 'MMQ' because it has Number=R
Ethnicity None
Traceback (most recent call last):
  File "/RED/TOOLS/bin/vcf2db.py", line 924, in <module>
    impacts_extras=a.impacts_field, aok=a.a_ok)
  File "/RED/TOOLS/bin/vcf2db.py", line 228, in __init__
    self.samples = self.create_samples()
  File "/RED/TOOLS/bin/vcf2db.py", line 598, in create_samples
    vals = [r[i] for r in rows]
IndexError: list index out of range
' returned non-zero exit status 1.
Traceback (most recent call last):
  File "/RED/TOOLS/bin/bcbio_nextgen.py", line 245, in <module>
    main(**kwargs)
  File "/RED/TOOLS/bin/bcbio_nextgen.py", line 46, in main
    run_main(**kwargs)
  File "/RED/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/bcbio/pipeline/main.py", line 50, in run_main
    fc_dir, run_info_yaml)
  File "/RED/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/bcbio/pipeline/main.py", line 91, in _run_toplevel
    for xs in pipeline(config, run_info_yaml, parallel, dirs, samples):
  File "/RED/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/bcbio/pipeline/main.py", line 188, in variant2pipeline
    samples = population.prep_db_parallel(samples, run_parallel)
  File "/RED/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/bcbio/variation/population.py", line 403, in prep_db_parallel
    output = parallel_fn("prep_gemini_db", to_process)
  File "/RED/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/bcbio/distributed/multi.py", line 28, in run_parallel
    return run_multicore(fn, items, config, parallel=parallel)
  File "/RED/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/bcbio/distributed/multi.py", line 86, in run_multicore
    for data in joblib.Parallel(parallel["num_jobs"], batch_size=1, backend="multiprocessing")(joblib.delayed(fn)(*x) for x in items):
  File "/RED/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/joblib/parallel.py", line 921, in __call__
    if self.dispatch_one_batch(iterator):
  File "/RED/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/joblib/parallel.py", line 759, in dispatch_one_batch
    self._dispatch(tasks)
  File "/RED/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/joblib/parallel.py", line 716, in _dispatch
    job = self._backend.apply_async(batch, callback=cb)
  File "/RED/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/joblib/_parallel_backends.py", line 182, in apply_async
    result = ImmediateResult(func)
  File "/RED/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/joblib/_parallel_backends.py", line 549, in __init__
    self.results = batch()
  File "/RED/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/joblib/parallel.py", line 225, in __call__
    for func, args, kwargs in self.items]
  File "/RED/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/joblib/parallel.py", line 225, in <listcomp>
    for func, args, kwargs in self.items]
  File "/RED/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/bcbio/utils.py", line 55, in wrapper
    return f(*args, **kwargs)
  File "/RED/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/bcbio/distributed/multitasks.py", line 383, in prep_gemini_db
    return population.prep_gemini_db(*args)
  File "/RED/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/bcbio/variation/population.py", line 50, in prep_gemini_db
    gemini_db = create_gemini_db(ann_vcf, data, gemini_db, ped_file)
  File "/RED/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/bcbio/variation/population.py", line 142, in create_gemini_db
    do.run(cmd, "GEMINI: create database with vcf2db")
  File "/RED/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/bcbio/provenance/do.py", line 26, in run
    _do_run(cmd, checks, log_stdout, env=env)
  File "/RED/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/bcbio/provenance/do.py", line 106, in _do_run
    raise subprocess.CalledProcessError(exitcode, error_msg)
subprocess.CalledProcessError: Command '/RED/TOOLS/bin/vcf2db.py /mnt/36642bae-9ec9-4100-88a2-ac173a20ea16/WORKDIR/TRIO_BRAZILIANS/F178/work/gemini/_mnt_36642bae-9ec9-4100-88a2-ac173a20ea16_WORKDIR_TRIO_BRAZILIANS_F178-gatk-haplotype-nomultiallelic-annotated-gemini.vcf.gz /mnt/36642bae-9ec9-4100-88a2-ac173a20ea16/WORKDIR/TRIO_BRAZILIANS/F178/work/gemini/_mnt_36642bae-9ec9-4100-88a2-ac173a20ea16_WORKDIR_TRIO_BRAZILIANS_F178-gatk-haplotype-nomultiallelic.ped /mnt/36642bae-9ec9-4100-88a2-ac173a20ea16/WORKDIR/TRIO_BRAZILIANS/F178/work/bcbiotx/tmp71dgss81/_mnt_36642bae-9ec9-4100-88a2-ac173a20ea16_WORKDIR_TRIO_BRAZILIANS_F178-gatk-haplotype.db
skipping 'MMQ' because it has Number=R
Ethnicity None
Traceback (most recent call last):
  File "/RED/TOOLS/bin/vcf2db.py", line 924, in <module>
    impacts_extras=a.impacts_field, aok=a.a_ok)
  File "/RED/TOOLS/bin/vcf2db.py", line 228, in __init__
    self.samples = self.create_samples()
  File "/RED/TOOLS/bin/vcf2db.py", line 598, in create_samples
    vals = [r[i] for r in rows]
IndexError: list index out of range
' returned non-zero exit status 1.

I will norm now and I will update

@naumenko-sa Do we have any news?

Hi @kokyriakidis !

You have removed vcfanno from the config, but the error is still there. I think it is because you have tools_on: gemini. That causes vcfanno to annotate with genomes/Hsapiens/hg38/config/vcfanno/gemini.conf, which includes the same variation/gnomad_exome.vcf.gz file. Could you please try to remove tools_off: gemini to confirm that failing step is indeed vcfanno?

I've removed duplicates from variation/gnomad_exome.vcf.gz:
Before removing duplicates there were 3 variants at chr1:146984765

tabix gnomad_exome.vcf.gz chr1:146984765-146984765
chr1	146984765	3	G	T	7794.38
chr1	146984765	3	G	A	7794.38
chr1	146984765	3	G	A	8470.69

after removing duplicates with
bcftools norm -dall -Oz gnomad_exome.vcf.gz > gnomad_exome.no_duplicates.vcf.gz
tabix gnomad_exome.no_duplicates.vcf.gz

the duplicate record is gone, but the allele A is gone as well
#CHROM	POS	ID	REF	ALT	QUAL	FILTER
chr1	146984765	rs781933389	G	T	7794.38	PASS

I've submitted a bug to
https://github.com/samtools/bcftools/issues/1089

I used a rival vt tool to remove duplicates:
https://github.com/naumenko-sa/bioscripts/blob/master/variation/vcf.remove_duplicates.sh

this time the allele A is kept:
chr1	146984765	3	G	A	8470.69
chr1	146984765	3	G	T	7794.38

I've replaced gnomad_exome.vcf.gz with gnomad_exome.no_duplicates.vcf.gz in the bcbio installation:

cd genomes/Hsapiens/hg38/variation
mv gnomad_exome.vcf.gz gnomad_exome.with_duplicates.vcf.gz
ln -s gnomad_exome.no_duplicates.vcf.gz gnomad_exome.vcf.gz

I am running your WES trio project as a test.

I've  submitted fixed gnomad_exome recipe to remove duplicates in cloudbiolinux:
https://github.com/chapmanb/cloudbiolinux/pull/325

SN

Thanks so much for all your help Sergey!
So in order to get it work I just update bcbio after cloudbiolinux PR is merged?

Yes, you'd need to bcbio-nextgen.py upgrade -u skip --data , and let me know if you have any issues during installation of with your trio. I have not finished a trio run on my side (it is running), so there still might be issues.
SN

Ok! I am waiting for your confirmation! Thanks again for helping and testing!

It seems that I get this error now when running with GEMINI:
[2019-10-12T08:32Z] tabix index _mnt_36642bae-9ec9-4100-88a2-ac173a20ea16_WORKDIR_TRIO_BRAZILIANS_F178-gatk-haplotype-nomultiallelic-annotated-gemini.vcf.gz
[2019-10-12T08:32Z] GEMINI: create database with vcf2db
[2019-10-12T08:32Z] skipping 'MMQ' because it has Number=R
[2019-10-12T08:32Z] Ethnicity None
[2019-10-12T08:32Z] Traceback (most recent call last):
[2019-10-12T08:32Z]   File "/RED/TOOLS/bin/vcf2db.py", line 924, in <module>
[2019-10-12T08:32Z]     impacts_extras=a.impacts_field, aok=a.a_ok)
[2019-10-12T08:32Z]   File "/RED/TOOLS/bin/vcf2db.py", line 228, in __init__
[2019-10-12T08:32Z]     self.samples = self.create_samples()
[2019-10-12T08:32Z]   File "/RED/TOOLS/bin/vcf2db.py", line 598, in create_samples
[2019-10-12T08:32Z]     vals = [r[i] for r in rows]
[2019-10-12T08:32Z] IndexError: list index out of range
[2019-10-12T08:32Z] Uncaught exception occurred
Traceback (most recent call last):
  File "/RED/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/bcbio/provenance/do.py", line 26, in run
    _do_run(cmd, checks, log_stdout, env=env)
  File "/RED/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/bcbio/provenance/do.py", line 106, in _do_run
    raise subprocess.CalledProcessError(exitcode, error_msg)
subprocess.CalledProcessError: Command '/RED/TOOLS/bin/vcf2db.py /mnt/36642bae-9ec9-4100-88a2-ac173a20ea16/WORKDIR/TRIO_BRAZILIANS/F178/work/gemini/_mnt_36642bae-9ec9-4100-88a2-ac173a20ea16_WORKDIR_TRIO_BRAZILIANS_F178-gatk-haplotype-nomultiallelic-annotated-gemini.vcf.gz /mnt/36642bae-9ec9-4100-88a2-ac173a20ea16/WORKDIR/TRIO_BRAZILIANS/F178/work/gemini/_mnt_36642bae-9ec9-4100-88a2-ac173a20ea16_WORKDIR_TRIO_BRAZILIANS_F178-gatk-haplotype-nomultiallelic.ped /mnt/36642bae-9ec9-4100-88a2-ac173a20ea16/WORKDIR/TRIO_BRAZILIANS/F178/work/bcbiotx/tmp6wl69_05/_mnt_36642bae-9ec9-4100-88a2-ac173a20ea16_WORKDIR_TRIO_BRAZILIANS_F178-gatk-haplotype.db
skipping 'MMQ' because it has Number=R
Ethnicity None
Traceback (most recent call last):
  File "/RED/TOOLS/bin/vcf2db.py", line 924, in <module>
    impacts_extras=a.impacts_field, aok=a.a_ok)
  File "/RED/TOOLS/bin/vcf2db.py", line 228, in __init__
    self.samples = self.create_samples()
  File "/RED/TOOLS/bin/vcf2db.py", line 598, in create_samples
    vals = [r[i] for r in rows]
IndexError: list index out of range
' returned non-zero exit status 1.
Traceback (most recent call last):
  File "/RED/TOOLS/bin/bcbio_nextgen.py", line 245, in <module>
    main(**kwargs)
  File "/RED/TOOLS/bin/bcbio_nextgen.py", line 46, in main
    run_main(**kwargs)
  File "/RED/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/bcbio/pipeline/main.py", line 50, in run_main
    fc_dir, run_info_yaml)
  File "/RED/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/bcbio/pipeline/main.py", line 91, in _run_toplevel
    for xs in pipeline(config, run_info_yaml, parallel, dirs, samples):
  File "/RED/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/bcbio/pipeline/main.py", line 188, in variant2pipeline
    samples = population.prep_db_parallel(samples, run_parallel)
  File "/RED/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/bcbio/variation/population.py", line 403, in prep_db_parallel
    output = parallel_fn("prep_gemini_db", to_process)
  File "/RED/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/bcbio/distributed/multi.py", line 28, in run_parallel
    return run_multicore(fn, items, config, parallel=parallel)
  File "/RED/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/bcbio/distributed/multi.py", line 86, in run_multicore
    for data in joblib.Parallel(parallel["num_jobs"], batch_size=1, backend="multiprocessing")(joblib.delayed(fn)(*x) for x in items):
  File "/RED/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/joblib/parallel.py", line 921, in __call__
    if self.dispatch_one_batch(iterator):
  File "/RED/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/joblib/parallel.py", line 759, in dispatch_one_batch
    self._dispatch(tasks)
  File "/RED/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/joblib/parallel.py", line 716, in _dispatch
    job = self._backend.apply_async(batch, callback=cb)
  File "/RED/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/joblib/_parallel_backends.py", line 182, in apply_async
    result = ImmediateResult(func)
  File "/RED/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/joblib/_parallel_backends.py", line 549, in __init__
    self.results = batch()
  File "/RED/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/joblib/parallel.py", line 225, in __call__
    for func, args, kwargs in self.items]
  File "/RED/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/joblib/parallel.py", line 225, in <listcomp>
    for func, args, kwargs in self.items]
  File "/RED/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/bcbio/utils.py", line 55, in wrapper
    return f(*args, **kwargs)
  File "/RED/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/bcbio/distributed/multitasks.py", line 383, in prep_gemini_db
    return population.prep_gemini_db(*args)
  File "/RED/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/bcbio/variation/population.py", line 50, in prep_gemini_db
    gemini_db = create_gemini_db(ann_vcf, data, gemini_db, ped_file)
  File "/RED/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/bcbio/variation/population.py", line 142, in create_gemini_db
    do.run(cmd, "GEMINI: create database with vcf2db")
  File "/RED/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/bcbio/provenance/do.py", line 26, in run
    _do_run(cmd, checks, log_stdout, env=env)
  File "/RED/RESOURCES/bcbio/anaconda/lib/python3.6/site-packages/bcbio/provenance/do.py", line 106, in _do_run
    raise subprocess.CalledProcessError(exitcode, error_msg)
subprocess.CalledProcessError: Command '/RED/TOOLS/bin/vcf2db.py /mnt/36642bae-9ec9-4100-88a2-ac173a20ea16/WORKDIR/TRIO_BRAZILIANS/F178/work/gemini/_mnt_36642bae-9ec9-4100-88a2-ac173a20ea16_WORKDIR_TRIO_BRAZILIANS_F178-gatk-haplotype-nomultiallelic-annotated-gemini.vcf.gz /mnt/36642bae-9ec9-4100-88a2-ac173a20ea16/WORKDIR/TRIO_BRAZILIANS/F178/work/gemini/_mnt_36642bae-9ec9-4100-88a2-ac173a20ea16_WORKDIR_TRIO_BRAZILIANS_F178-gatk-haplotype-nomultiallelic.ped /mnt/36642bae-9ec9-4100-88a2-ac173a20ea16/WORKDIR/TRIO_BRAZILIANS/F178/work/bcbiotx/tmp6wl69_05/_mnt_36642bae-9ec9-4100-88a2-ac173a20ea16_WORKDIR_TRIO_BRAZILIANS_F178-gatk-haplotype.db
skipping 'MMQ' because it has Number=R
Ethnicity None
Traceback (most recent call last):
  File "/RED/TOOLS/bin/vcf2db.py", line 924, in <module>
    impacts_extras=a.impacts_field, aok=a.a_ok)
  File "/RED/TOOLS/bin/vcf2db.py", line 228, in __init__
    self.samples = self.create_samples()
  File "/RED/TOOLS/bin/vcf2db.py", line 598, in create_samples
    vals = [r[i] for r in rows]
IndexError: list index out of range
' returned non-zero exit status 1.

Without GEMINI everything worked fine

Hi Sergey and the rest!
Will sticking to gnomad 2.1 help with this issue?

This is still due to a PED file problem-- could you send along the PED file you are using?

178F_PED.zip

Thanks, could you also pass along the YAML file you are using?

F178.zip

Thanks, how about this file? /mnt/36642bae-9ec9-4100-88a2-ac173a20ea16/WORKDIR/TRIO_BRAZILIANS/F178/work/gemini/_mnt_36642bae-9ec9-4100-88a2-ac173a20ea16_WORKDIR_TRIO_BRAZILIANS_F178-gatk-haplotype-nomultiallelic.ped

_mnt_36642bae-9ec9-4100-88a2-ac173a20ea16_WORKDIR_TRIO_BRAZILIANS_F178-gatk-haplotype-nomultiallelic.zip

@roryk The PED file is fine right?

Hi @kokyriakidis,
Yes, it looks fine to me. Could you also send along the VCF file or a snippet from it so I can test locally?

The whole project is here:
https://drive.google.com/drive/folders/1BStwCZFMcFL7-6AruM723mbpSWH0XozS?usp=sharing

Thanks, the PED file and the VCF file don't have the same names. The PED file has the names as F178CHILD but the VCF file has them named F178_CHILD.

@roryk Sorry this was from another run. Please check again the link above. I have uploaded the updated run (which also fails)

Thanks, it looks like the PED file has the ethnicity column but no entry for it. If you add an ethnicity column to the PED file and populate it with -9 it should fix this problem.

hmm thanks! Maybe update the readme, so others know that

Thanks, I think we shouldn't be adding that column unless it is there, so it is a bug.

running the command again caused the same error when changes PED. should I delete work folder and run it again?

Heya, it will still be using the broken PED file if it is on the disk, I think if you just delete the PED file in the GEMINI directory it will populate a new one.

I'll update the documentation to mention we require the ethnicity as well, sorry about that.

It finally run successfully! Thanks a lot all of you for the help!

Great! Sorry for the problems!


Optimize loading of KoFiles

The current implementation has two issues, which can cause issues on larger projects:

Currently getKoFileDeclarations() will parse all Kotlin files in the project, including build files. They then later filtered out by KoScopeCreatorCore.getFiles, but this is very inefficient. We've seen lots of OOMs lately in our Konsist tests, and it seems to be because we also have a lot of generated Kotlin files.
KoFileDeclarationProvider will try and launch getKoFileDeclarations from object init (which is basically when the class is loaded). I assume that this was done to preload the classes, but I think that is too aggressive. These should be lazily loaded when they're first needed, which will usually be the first test.

This PR changes this so that getKoFileDeclarations can be provided with a filter block, which is provided by KoScopeCreatorCore.getFiles.
Also optimized all of the Regex creation. We don't want to create these on every file in the tree-walk (Regex instances are quite heavy).

Very nice
Thx for the contribution 🙏


[BUG] - UseWebLogin dosen't work if your SSO account is a guest on in the tenant you're trying to access

Reporting an Issue or Missing Feature
I know we're not supposed to use the -UseWebLogin but in some scenarios it's the only option, currently working with a client who's conditional access prevents me from using -Interactive when not on their network.
Using -UseWebLogin worked fine for a while, now however since my primary (windows) account has become a guest, it continues to try and use that, but my guest account does not have access to the root site collection, so I end up stuck on this screen

Essentially stating that external sharing has been turned off for https://.sharepoint.com/_layouts/15/settings.aspx.
Expected behavior
I would've hoped to have an option to run the loginprompt in incognito/inprivate so I could somehow avoid using my windows credentials.
Actual behavior
I end up being stuck on a "That didn't work" error message due to -UseWebLogin attempting to access the root site collection, and no option to "switch account"
Steps to reproduce behavior
On a tenant that's not the same as your windows accounts tenant:

Create a random site collection
Invite your windows connected account as a guest to this site
Turn off external sharing on the root site collection
Try and run the following command
Connect-PnPOnline "https://NonPrimaryTenant.sharepoint.com/sites/SiteThatYourGuestAccountIsInvitedTo/" -UseWebLogin
Seemingly that's all it takes

What is the version of the Cmdlet module you are running?
PnP.PowerShell 2.1.1
Which operating system/environment are you running PnP PowerShell on?

[X] Windows
[ ] Linux
[ ] MacOS
[ ] Azure Cloud Shell
[ ] Azure Functions
[ ] Other : please specify

hi @Tanddant , @koskila fixed this issue. You will need to supply additional parameter -RelativeUrl for this.
It is now available in the latest nightly build. Do check it out, thanks !
As mentioned, you should ideally use the interactive way , the weblogin method is not so safe and doesn't work with Graph related cmdlets.

@gautamdsheth, I'll keep an eye out, I finally managed to convince that specific admin to grant me the delegated permissions 😊
Always aiming for interactive, but sometimes admins don't understand the difference between delegated and app permissions, and no amount of explanation will ever be able convince them that it's not dangerous

Happy to hear you got around it @Tanddant 😅
I have to also clarify that I changed Connect-PnPOnline to actually use the supplied site instead of root site, and -RelativeUrl should only be used when the _layouts/15/settings.aspx (default) is not accessible in the site Connect-PnPOnline is used against.


About virtualization operating system

Hey, brother, unfortunately to see you say goodbye to ESXi. Which platform do you move to? I have used the PROXMOX operating system for so long. I think it is not a good virtualization system. ESXi is still the first choice. You can be regarded as a hero in virtualization.

Hi
I've retired from real world software development after 35 years of everything from IBM mainframe systems programming to being the CTO of a company. So my wife says it's time to do other things! Can't argue with her now we're both retired.
I've moved off Intel and I am using Apple Silicon kit at home, plus some other ARM stuff such as Raspberry PI & PICO. I was lucky enough to actually program on the original ARM1 dev kit back in 1986 and so looking forward to getting familiar with current ARM systems. So VMware really isn't much use to me now.
However I am still very interested in virtualisation of macOS and am looking at:
QEMU - https://qemu.org
UTM - https://utm.app
tart - https://tart.run
VirtualBuddy - https://github.com/insidegui/VirtualBuddy
I have a good friend who is using PROMOX on INtel and I think he's had macOS running on it.
So I'm still hacking but on different platform.
All the best
Dave

@DrDonk I also successfully run macOS on proxmox, but my windows is having issues, really frustrating, happy ya


[BUG]

Hello
How can I edit the format of prefix In chat

As you see in the picture
the team name written in "&b" color and there is a space in front of it
is there any possibility to edit this ?

Currently, the format for the team prefix is not in messages.yml this will be changed in version 3.0.8
Additionally, I recommend against using /teama reload except to reload the messages.yml file until 3.0.8 as parts of the config file are known to not work with /teama reload (this will be fixed for the next update)

Thank you

This will be released in version 3.0.8.


Assign space indices to native-fullscreen spaces

Discussion / Feature request
Factored out from https://github.com/koekeishiya/yabai/issues/116#issuecomment-510824950.
@koekeishiya

Say there are 4 spaces, numbered 1-4 accordingly. After the named Desktop 2, there is a fullscreen space. Would it not make sense to have the fullscreen space be mission-control index 3? I know this breaks with the names given by Apple, which appear to be a simple enumeration of user-spaces - there is no internal name given to spaces (except for the uuid string), which is why I suspect this.

@dominiklohmann

If people wanted the old way back, they could then still do this, assuming there'd also be a "native-fullscreen" attribute for spaces (added it to the list in the first post):

# this only works if the query output is guaranteed to be ordered by appearance
old_index=4
yabai -m space --focus \
    "$(yabai -m query --spaces
        | jq -re "[.[] | select(.\"native-fullscreen\" == 0)][${old_index}].index")"

Implemented on master. The literal mission control index is now used to refer to a space - not the index in the desktop name provided by Apple.

Bug: Sending a window to a native-fullscreen space in any ways needs to be a no-op.

Right makes sense - fixed on master.

Sorry, it's one of those days... what's inconsistent now is that there's no non-zero exit code.

Fixed it in a different way.

I could only test moving a fullscreen space around on the display it was created on. I fear that it will break when moved to another display and that this might require doing some more reverse engineering in the Dock.app. Could you test this when you have a moment?

Works fine.


chore: bump workspace-optimizer to v0.12.13

https://github.com/neutron-org/neutron-tests/actions/runs/5288059290

https://github.com/neutron-org/neutron-tests/actions/runs/5293650830


How to modify the default description

How to modify the default description？Please see the attached screenshot.
I want to modify the default description, but I can't find a corresponding method. Is it possible to provide an entry to modify the content of SPPermissionsText.description(for:), such as reading values from plist.

Sorry, I find SPPermissionsDataSource can do it.


UI no longer has datepickers when using a "Custom" lookback

With the introduction of the "lookback" dropdown, I no longer see datepickers, nor any sort of formatting help/advice for the from/to date/time fields.

The datepickers were helpful! Please bring them back. A "placeholder" attribute on the date and time textboxes indicating format would also be very much appreciated. It would be even better if the boxes could be prepopulated with the values from the most recent search or Lookback selection. :)

Also, the date/time fields don't repopulate after performing a search with a custom lookback.

@igorwwwwwwwwwwwwwwwwwwww Can you take a look at this if you have time?


ServiceKey and ServiceProvider

Hi Maksim, long time not seen :)! I hope you're fine.
I use dryioc combined with the serviceprovider.
Lets say I have a service in the serviceprovider IService and this IService is needed by other services in the serviceprovider.
Now I need a special version of IService for services in DryIoc, so I register IService with a service-key "special".
Now I have another service in DryIoc that needs the "default" IService (no servicekey) from serviceprovider but DryIoc resolver tells me that there is only the "special" registration and no other registration was found.
On the other hand when I register another IService with no servicekey in DryIoc the resolve works again - but this would mean alot of work and duplicate registrations and wrong settings.
I prepared two simple tests which demonstrate that the easy case is working (resolve IService direct with and without servicekey) but a dependent service that needs IService doesn't.
Is there a way to tell DryIoc to have a look into the serviceprovider registrations if the found registrations doesn't fit or is this a bug?
I hope you can help me!
Source:
Needed packages:
        <PackageReference Include="Microsoft.Extensions.DependencyInjection" Version="2.2.0" />
        <PackageReference Include="Microsoft.Extensions.DependencyInjection.Abstractions" Version="2.2.0" />

        <ProjectReference Include="..\..\src\DryIoc.Microsoft.DependencyInjection.AspNetCore2_1\DryIoc.Microsoft.DependencyInjection.AspNetCore2_1.csproj" />

using DryIoc.Microsoft.DependencyInjection;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Options;
using NUnit.Framework;

namespace DryIoc.IssuesTests
{
    [TestFixture]
    public class GenericServiceMix
    {
        [Test]
        public void ResolveDirect()
        {
            var services = new ServiceCollection();
            services.AddSingleton<MsDiServiceThatNeedsOptionsOption>(new MsDiServiceThatNeedsOptionsOption { Info = "A" });

            var container = new Container();
            container.Register<MsDiServiceThatNeedsOptionsOption>(
                serviceKey: "special",
                made: Made.Of(() => new MsDiServiceThatNeedsOptionsOption { Info = "B" }),
                reuse: Reuse.Singleton);

            var serviceContainer = container.WithDependencyInjectionAdapter(services);

            var resolvedMsDiService = serviceContainer.GetRequiredService<MsDiServiceThatNeedsOptionsOption>();

            Assert.NotNull(resolvedMsDiService);
            Assert.AreEqual("A", resolvedMsDiService.Info);

            var serviceFromDryIoc = serviceContainer.Resolve<MsDiServiceThatNeedsOptionsOption>(serviceKey: "special");

            Assert.NotNull(serviceFromDryIoc);
            Assert.AreEqual("B", serviceFromDryIoc.Info);

            var serviceFromMsDi = serviceContainer.Resolve<MsDiServiceThatNeedsOptionsOption>(serviceKey: null);

            Assert.NotNull(serviceFromMsDi);
            Assert.AreEqual("A", serviceFromMsDi.Info);
        }

        [Test]
        public void ResolveViaOptions()
        {
            var services = new ServiceCollection();
            services.AddOptions();

            services.AddSingleton<IMsDiServiceThatNeedsOptions, MsDiServiceThatNeedsOptions>();
            services.Configure<MsDiServiceThatNeedsOptionsOption>(c => c.Info = "Bob Builder");

            var container = new Container();

            container.Register<IDryIocServiceThatNeedsOptions, DryIocServiceThatNeedsOptions>(Reuse.Transient);

            // comment to avoid the "special" registation that just hides the default implementation
            container.Register<IOptions<MsDiServiceThatNeedsOptionsOption>>(
                serviceKey: "special",
                made: Made.Of(() => Options.Create(Arg.Index<MsDiServiceThatNeedsOptionsOption>(0)),
                    r => new MsDiServiceThatNeedsOptionsOption { Info = "Alice Anger" }),
                reuse: Reuse.Singleton);

            var serviceContainer = container.WithDependencyInjectionAdapter(services);

            var resolvedMsDiService = serviceContainer.GetRequiredService<IMsDiServiceThatNeedsOptions>();

            Assert.NotNull(resolvedMsDiService);
            Assert.AreEqual("Bob Builder", resolvedMsDiService.GiveInfo());

            var resolvedDryIocService = serviceContainer.GetRequiredService<IDryIocServiceThatNeedsOptions>();

            Assert.NotNull(resolvedDryIocService);
            Assert.AreEqual("Bob Builder", resolvedDryIocService.GiveInfo());
        }

        public class DryIocServiceThatNeedsOptions : IDryIocServiceThatNeedsOptions
        {
            private readonly IMsDiServiceThatNeedsOptions msService;

            public DryIocServiceThatNeedsOptions(IMsDiServiceThatNeedsOptions msService)
            {
                this.msService = msService;
            }

            public string GiveInfo() => msService.GiveInfo();
        }

        public interface IDryIocServiceThatNeedsOptions
        {
            string GiveInfo();
        }

        public class MsDiServiceThatNeedsOptions : IMsDiServiceThatNeedsOptions
        {
            private readonly string info;

            public MsDiServiceThatNeedsOptions(IOptions<MsDiServiceThatNeedsOptionsOption> options)
            {
                info = options.Value.Info;
            }

            public string GiveInfo() => info;
        }

        public interface IMsDiServiceThatNeedsOptions
        {
            string GiveInfo();
        }

        public class MsDiServiceThatNeedsOptionsOption
        {
            public string Info { get; set; }
        }
    }
}

Hi @SeriousM ,
I see that you are using AspNetCore2_1 package, is it the most?  Also, what is the referenced DryIoc version?
From the get-go I don't see a reason why keyed service is hiding the default one. I will check when I have time.

The dryioc version is the most recent one I've modified your test project,
that's why I needed 2.1 otherwise I would get version incompatibilities.
Thank you!
On Fri, 17 Jul 2020, 21:05 Maksim Volkau<EMAIL_ADDRESS>wrote:

Hi @SeriousM https://github.com/SeriousM ,
I see that you are using AspNetCore2_1 package, is it the most? Also, what
is the referenced DryIoc version?
From the get-go I don't see a reason why keyed service is hiding the
default one. I will check when I have time.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dadhi/DryIoc/issues/293#issuecomment-660288155, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AADQZPL4TGCVVT4KF2XWII3R4COHBANCNFSM4O6P6SWQ
.

Hi @SeriousM
I have changed the registrations so that parameter with "special" service key is injected,
but I fail to get the expected output for "Bob Builder" for MsDI. I am not knowledgeable with Options pattern, so I propose you to look. The full dotnet-script code will be in the next comment. See FIX and PROBLEM comments...

#r "nuget: Microsoft.Extensions.DependencyInjection, 2.2.0"
#r "nuget: Microsoft.Extensions.Options, 1.0.0"
#r "nuget: DryIoc.Microsoft.DependencyInjection, 4.0.0"
#r "nuget: NUnit, 3.12.0"
using DryIoc;
using DryIoc.Microsoft.DependencyInjection;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Options;
using NUnit.Framework;
ResolveViaOptions();
WriteLine($"{nameof(ResolveViaOptions)} passed");
ResolveDirect();
WriteLine($"{nameof(ResolveDirect)} passed");
public void ResolveViaOptions()
{
var container = new Container();
var services = new ServiceCollection();
services.AddOptions();
services.Configure<MsDiServiceThatNeedsOptionsOption>(c => c.Info = "Bob Builder");

// 1st FIX
// services.AddSingleton<IMsDiServiceThatNeedsOptions, MsDiServiceThatNeedsOptions>();
container.Register<IMsDiServiceThatNeedsOptions, MsDiServiceThatNeedsOptions>(
    Reuse.Singleton,
    made: Parameters.Of.Type<IOptions<MsDiServiceThatNeedsOptionsOption>>(serviceKey: "special"));

container.Register<IDryIocServiceThatNeedsOptions, DryIocServiceThatNeedsOptions>(Reuse.Transient);

// comment to avoid the "special" registation that just hides the default implementation
container.Register<IOptions<MsDiServiceThatNeedsOptionsOption>>(
    serviceKey: "special",
    made: Made.Of(() => Options.Create(Arg.Index<MsDiServiceThatNeedsOptionsOption>(0)),
        r => new MsDiServiceThatNeedsOptionsOption { Info = "Alice Anger" }),
    reuse: Reuse.Singleton);

var serviceContainer = container.WithDependencyInjectionAdapter(services);

var resolvedMsDiService = serviceContainer.GetRequiredService<IMsDiServiceThatNeedsOptions>();

// PROBLEM is it should be "Bob Builder"
Assert.NotNull(resolvedMsDiService);
Assert.AreEqual("Alice Anger", resolvedMsDiService.GiveInfo(), "MsDI");

var resolvedDryIocService = serviceContainer.GetRequiredService<IDryIocServiceThatNeedsOptions>();

Assert.NotNull(resolvedDryIocService);
Assert.AreEqual("Alice Anger", resolvedDryIocService.GiveInfo(), "DiDI");

}
public void ResolveDirect()
{
var services = new ServiceCollection();
services.AddSingleton(new MsDiServiceThatNeedsOptionsOption { Info = "A" });
var container = new Container();
container.Register<MsDiServiceThatNeedsOptionsOption>(
    serviceKey: "special",
    made: Made.Of(() => new MsDiServiceThatNeedsOptionsOption { Info = "B" }),
    reuse: Reuse.Singleton);

var serviceContainer = container.WithDependencyInjectionAdapter(services);

var resolvedMsDiService = serviceContainer.GetRequiredService<MsDiServiceThatNeedsOptionsOption>();

Assert.NotNull(resolvedMsDiService);
Assert.AreEqual("A", resolvedMsDiService.Info);

var serviceFromDryIoc = serviceContainer.Resolve<MsDiServiceThatNeedsOptionsOption>(serviceKey: "special");

Assert.NotNull(serviceFromDryIoc);
Assert.AreEqual("B", serviceFromDryIoc.Info);

var serviceFromMsDi = serviceContainer.Resolve<MsDiServiceThatNeedsOptionsOption>(serviceKey: null);

Assert.NotNull(serviceFromMsDi);
Assert.AreEqual("A", serviceFromMsDi.Info);

}
public class DryIocServiceThatNeedsOptions : IDryIocServiceThatNeedsOptions
{
private readonly IMsDiServiceThatNeedsOptions msService;
public DryIocServiceThatNeedsOptions(IMsDiServiceThatNeedsOptions msService)
{
    this.msService = msService;
}

public string GiveInfo() => msService.GiveInfo();

}
public interface IDryIocServiceThatNeedsOptions
{
string GiveInfo();
}
public class MsDiServiceThatNeedsOptions : IMsDiServiceThatNeedsOptions
{
private readonly string info;
public MsDiServiceThatNeedsOptions(IOptions<MsDiServiceThatNeedsOptionsOption> options)
{
    info = options.Value.Info;
}

public string GiveInfo() => info;

}
public interface IMsDiServiceThatNeedsOptions
{
string GiveInfo();
}
public class MsDiServiceThatNeedsOptionsOption
{
public string Info { get; set; }
}


Update Documentation to reflect behavior

The accessor property is only necessary when using the useSortBy hook, correct?

I was mistaken


GSE146165 - Microglia Require CD4 T Cells to Complete the Fetal-to-Adult Transition

This issue is related to https://github.com/ebi-ait/hca-ebi-wrangler-central/issues/268.
The purpose of this ticket is to track the status of the human subset of the data that has not been exported, because of living European donors.
Primary Wrangler:
Ray
Secondary Wrangler:
Wei

project short name: CD4_T_cells_in_brain

Ingest project: https://contribute.data.humancellatlas.org/projects/detail?uuid=e5d45579-1f5b-48c3-b568-320d93e7ca72

Associated files:

Google Drive: https://drive.google.com/drive/folders/1MYi5OHuARUQIIewc4WUWRISbQwsrktfD

Published study links

Paper:  Pasciuto E, Burton OT, Roca CP, et al. Microglia Require CD4 T Cells to Complete the Fetal-to-Adult Transition. Cell. 2020 Aug;182(3):625-640.e24. DOI: 10.1016/j.cell.2020.06.026.
Pubmed ID: 32702313
Accessioned data:

GSE146165: Homo sapiens
GSE144038: Mus musculus

Key Events

[x] Collect any matrix and cell-type annotation files.
[ ] Upload spreadsheet that contains the human subset of the metadata to validate.
[ ] Check linking using ingest graph validator.
[ ] Transfer raw human data files to ingest to validate data files.
[ ] Submit the update to Production.
[ ] Export the human subset of the data as an update

Only the mouse data seems to have been wrangled. I'll check the paper to try identifying why, and add all the human metadata and data. I'm adding too the analysis files for both mouse and human samples.

I think the reason for the human data is because it was from living europeans! Should be able to add analysis files and openly available human metadata though but definitely not fastqs.

Original spreadsheet: https://docs.google.com/spreadsheets/d/125l_1HEO059TmbWA3EnZu-s1KscU0eSR/edit?usp=sharing&ouid=112854478400213800085&rtpof=true&sd=true
Update spreadsheet: https://docs.google.com/spreadsheets/d/127W9zA8ExnUhLzaS1MRo6QyCiFo1B1PY/edit?usp=sharing&ouid=112854478400213800085&rtpof=true&sd=true

Changes performed:

Added all the metadata and analysis files for human samples
Added analysis files for the already wrangled mouse samples

Secondary Reviewer assigned: @idazucchi

Hello!
I've finished reviewing, overall it's good there are just a few small things to fix
Specimen from organism

storage method: the field is enum, so cold on wet ice is not allowed. Maybe the best available option is fresh? in the sense that the specimen wasn’t frozen
known disease: you could consider adding epilepsy as disease, as it affects the brain sample

Enrichment protocol

markers: the formatting rules are unclear —> either with spaces or ;

Analysis file

missing analysis protocol id

Asked for data delivery to s3://hca-ncbi-cloud-data-iperez for the human FASTQ files

Graph validator does not work OK with two submissions, waiting on a decision on how to proceed, or if we are gonna apply a workaround.

Unlocked and exporting!

No longer blocked


Fix SSL_state_string() and SSL_state_string_long()

These functions needed updates for the various state machine states that
have been added for TLSv1.3.
Fixes #4795

Pushed. Thanks.


Metamask notification does not appear

I'm currently writing E2E tests for our application and strange behavior occurs. Running the application manually everything works correctly, but if you run the same steps via synpress an error occurs. Specifically: clicking a button triggers an invocation of a contract via ethers.js and the metamask notification (the window where you accept the transaction) does not appear but is intercepted as shown in image below.

If the notification is not shown, the test fails.
Contract invocation
The contract is invoked via ether.js as follows:
const contract: Contract = new ethers.Contract(contractAddress, abi, signer);
contract.createToken(metadata, selectedAddresses, { from: address, gasLimit: process.env.GAS_LIMIT })

Tests execution
Tests are executed like below:
cy.findByTestId("trigger_transaction_button").click();
cy.confirmMetamaskTransaction().then((txData)=>{
    expect(txData).not.null;
    expect(txData.confirmed).to.be.true;
})

Result
Test fails as follow:

Enviroment
OS: windows 10
Node: 16.14.0
Synpress: 3.0.5
ethers: 5.7.2
next: 13.0.2
Any suggestions?

Hey @FrancescoPG, I can see that you're using Windows. Have you tried to run the same setup on Linux or macOS?

also, do you have any website where I can reproduce this issue?
Thanks,
Jakub.

Hi @drptbl thanks for your suggestion. I tested the same setup on linux, Ubuntu 18 and there is exactly the same problem. Further detail, even if I don't think it's influential, in the linux environment the version of node is 16.
Unfortunately I don't have any websites to provide to replicate the problem, they are local developments.

Hey @FrancescoPG,
Sorry for inconvenience.
Can you try again with<EMAIL_ADDRESS>and report back?
This issue should be fixed now.
Also, if you're using docker-e2e, I suggest to update to one of these:
- synthetixio/docker-e2e:14.21-debian
- synthetixio/docker-e2e:16.19-debian
- synthetixio/docker-e2e:18.13-debian

or..
- synthetixio/docker-e2e:14.21-ubuntu
- synthetixio/docker-e2e:16.19-ubuntu
- synthetixio/docker-e2e:18.13-ubuntu

Thanks,
Jakub.

Hey, this should not be a case anymore.
Please use provided examples inside CI tips & tricks section.
Also, make sure you're on latest<EMAIL_ADDRESS>In details your setup is probably missing xvfb and window manager dependencies.
If you're still going to encounter this issue, please re-open.
Thanks,
Jakub.


Use [skip-ci] tag for gh-pages

Right now I am copying the circle.yml manually to the gh-pages when it is being deployed.  But I can clean up the config file, by just changing the commit message to include [skip-ci]

:tada: This issue has been resolved in version 3.3.1 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


migration issue: table created without a foreign key  after giving foreign key error

Laravel Version
10.12.0
PHP Version
8.1.12
Database Driver & Version
Server version: 10.4.27-MariaDB
Description
Issue: SQLSTATE[HY000]: General error: 1005 Can't create table the_service.order_items (errno: 150 "Foreign key constraint is incorrectly formed") (SQL: alter table order_items add constraint order_items_order_id_foreign foreign key (order_id) references orders (id))
Description:
During the execution of a migration in Laravel, an error occurs that prevents the creation of a foreign key constraint between the order_items and orders tables. Instead of throwing an error, the migration continues and creates the order_items table without the expected foreign key relationship. The specific error message received is SQLSTATE[HY000]: General error: 1005 Can't create table the_service.order_items(errno: 150 "Foreign key constraint is incorrectly formed") (SQL: alter tableorder_itemsadd constraintorder_items_order_id_foreign foreign key (order_id) references orders (id)).
Expected Result:
The migration process should fail and halt execution when encountering a foreign key constraint error. The error message should be displayed, indicating the issue with the foreign key constraint, and the order_items table should not be created.
Actual Result:
The migration process does not stop despite encountering a foreign key constraint error. Instead, it continues execution and creates the order_items table without the expected foreign key relationship to the orders table. The error message received is SQLSTATE[HY000]: General error: 1005 Can't create table the_service.order_items(errno: 150 "Foreign key constraint is incorrectly formed") (SQL: alter tableorder_itemsadd constraintorder_items_order_id_foreign foreign key (order_id) references orders (id)).
Additional Information:
This issue typically occurs when the data type of the foreign key column (order_id) in the order_items table does not match the data type of the referenced column (id) in the orders table. In such cases, MySQL fails to create the foreign key constraint.
Steps To Reproduce
Steps to Reproduce:

Set up a Laravel project with the necessary configurations.
Create migration files for the order_items and orders tables**(order table id data type and order_items table foreign key must be different).**
Define the schema for both tables in their respective migration files.
Include a foreign key constraint in the order_items migration file that references the id column of the orders table.
Run the migration command (php artisan migrate) to execute the migrations.

Heya, thanks for reporting.
We'll need more info and/or code to debug this further. Can you please create a repository with the command below, commit the code that reproduces the issue as one separate commit on the main/master branch and share the repository here? Please make sure that you have the latest version of the Laravel installer in order to run this command. Please also make sure you have both Git & the GitHub CLI tool properly set up.
laravel new bug-report --github="--public"

Please do not amend and create a separate commit with your custom changes. After you've posted the repository, we'll try to reproduce the issue.
Thanks!

https://github.com/rj-eww/bug-report.git
check this

        Schema::create('subcategories', function (Blueprint $table) {
            $table->id();
            $table->string('name');
            $table->unsignedInteger('category_id');
            $table->foreign('category_id')->references('id')->on('categories');
            $table->timestamps();
        });

You're trying to set a foreign key on a column that doesn't exists yet. Try splitting this up in a separate create and table function call.


Feature request for solving connection drop problems.

Is your feature request related to a problem? Please describe.
Since this tool is usually is being used in restricted networks, connection drops are common. For example, in some networks in Iran, long TCP connection is sometimes dropped by the censorship systems. This causes the wstunnel to be broken with different errors. But this error is common after a few minutes of connection:
“ERROR :: Cannot do tls handshake with the server”
Describe the solution you'd like
I think two solutions are suitable for this issue:

Adding a re-connect feature or retry feature after a connection error.
The second solution is more generalized: specifying a script in command line options to be run on connection errors.

Describe alternatives you've considered
Currently I am running another python script that checks the connection status and restarts the connection for every hour. But this solution is not good enough since the connection drops are not predictable.

Hello,
wstunnel should reconnect, if a new connection to the client is made. But this is a hard question, as it will depend on the kind of traffic you are encapsulating.
If it is TCP, force/silently reconnecting will do no good, because it is most likely that your censorship system just dropped the packet of data in the middle. Reconnecting while the app is not knowing it, will just leave the connection in a broken state as some piece of data is missing.
If it is UDP, it might work, but will depend on again of if your application is resilient or not to such scenario.
So in the end, this feature is of limited use, because it can only work for a subset of UDP applications.
Can you let me know what kind of traffic you are forwarding, and share your python script, so I can understand it better ?

Thank you, I am going to try to look at it when i go back from vacation ~2 weeks. Will see if i can do something with those errors

Thank you for your assistance. If I can do any help please let me know. Enjoy your vacation!

Hi back,
Can you run wstunnel client in verbose mode (-v) and let me know the full error messages you get ?

I had the opportunity to check your issue, sadly I haven't managed to reproduce the exact same errors.
I made some change in this version https://github.com/erebe/wstunnel/suites/14662115643/artifacts/832088597 would you mind trying it and let me know if it solves your issue ?

Sorry for my late response. I will test it and post the results soon here.

It's about four days that wstunnel is running and it seems there were no connection issues and the problem is solved. thanks.

Wonderful, thank you :)
I letting this ticket open until I make a new release of wstunnel

https://github.com/erebe/wstunnel/releases/tag/v6.0

Hello @uuonda ,
This is not fixable at wstunnel level, when the connection with the server drop, the wstunnel client has no way to know which data has been really been transmitted to the server, or if some has been lost in the middle.
The only solution is to forward the close of the connection to the client and let it handle the the retry/re-connect.
We can imagine a world where wstunnel client would not forward the close to the client and silently handle the de-connection/re-connection, but it will mean in most casesm that the data transmitted will be corrupted.
I.e: sending AAAAABBBBBCCCCDDD the remote endpoint  will only see AAAAABBBBBDDD, if wstunnel client handle the reconnection silently.
The only way is for the client to handle the reconnection


Describe how to set up a dev environment for Pixel Perfect

As far as I saw there is no description so far on how to configure a development environment for Pixel Perfect. There are only vague hints by the links provided within the Further Resources section of the  README.md.
Like for Firebug.next there should be the detailed steps described how to configure your system to be able to hack on Pixel Perfect.
Sebastian

Good point, readme file updated.
Honza


authentication_classes = () in password validate/confirm endpoints?

I found that if the user has an (invalid) local Bearer: <hex> cookie that gets sent as a header, the three views can fail authentication and return a 401. Is this intentional? Shouldn't these views have authentication_classes = () so they work even if there's a leftover token in the browser?
There might be a security-related reason for it to be this way but I'm not sure I can think what it is.

yeah. it seems authentication needs to be changed on these views to allow unauthenticated access to the views.

I ended up just inheriting from all the views and adding my own throttling and authentication settings via the authentication_classes and throttling_classes settings

After completing this i suggest the documentation just mention how to inherit from the existing view classes.  There are many permission possibilities and throttling possibilities.

Hey @guzzijones , could you show how to inherit from the existing view classes? I have tried to override them in this way:
from django_rest_passwordreset.views import (
    ResetPasswordRequestToken,
    ResetPasswordConfirm,
    ResetPasswordValidateToken,
)

class CustomResetPasswordRequestToken(ResetPasswordRequestToken):
    """
    Allow unauthenticated users to request a reset password token by using the email parameter.
    """

    permission_classes = [
        AllowAny,
    ]
    authentication_classes = []

class CustomResetPasswordConfirm(ResetPasswordConfirm):
    """
    Using a valid token, the unauthenticated users password is set to the provided password.
    """

    permission_classes = [
        AllowAny,
    ]
    authentication_classes = []

class CustomResetPasswordValidateToken(ResetPasswordValidateToken):
    """
    Will return a 200 if a given token is valid.
    """

    permission_classes = [
        AllowAny,
    ]
    authentication_classes = []

And adding these views into urls.py:
...
    path(
        "password_reset/",
        CustomResetPasswordRequestToken.as_view(),
        name="password_reset",
    ),
    path(
        "password_reset/confirm/",
        CustomResetPasswordConfirm.as_view(),
        name="password_reset_confirm",
    ),
    path(
        "password_reset/validate_token/",
        CustomResetPasswordValidateToken.as_view(),
        name="password_reset_validate",
    ),
 ...

But I get:
django.urls.exceptions.NoReverseMatch: 'password_reset' is not a registered namespace
I agree with you saying that this should be inserted in the documentation.

Hi everyone, I'm facing the same problem. I've created a pull request to solve this issue https://github.com/anexia-it/django-rest-passwordreset/pull/148


1.8

This question requires that you evaluate the underlined text to determine if it is correct.
Azure Germany can be used by legal residents of Germany only.
Instructions: Review the underlined text. If it makes the statement correct, select "No change is
needed". If the statement is incorrect, select the answer choice that makes the statement correct.
A. no change is needed
B. only enterprises that are registered in Germany
C. only enterprises that purchase their azure licenses from a partner based in Germany
D. any user or enterprise that requires its data to reside in Germany

A

D*


Bump cluster chart dependency to v0.32.0

Towards: https://github.com/giantswarm/giantswarm/issues/30596
Related to: https://github.com/giantswarm/cluster/pull/178
Please check if PR meets these requirements

[ ] Results of the diffs have been examined and no unintended changes are being introduced.

Helper

to disable the GH Action generating manifests diffs for installations, between target and source branches, comment the /no_diffs_printing on the PR.

Trigger e2e tests

/run cluster-test-suites

/run cluster-test-suites


added cli analytics

A couple of comments from my security POV:

Out of respect for user privacy -- and to comply with Safe Harbor -- this needs to be something users can opt out of. Ideally, this would be opt-in: ask users the first time this runs if they're OK sharing this data, and only send it if they consent. Opt-out might be acceptable if there's a reason why opt-in won't work, but as a general principle asking before "phoning home" is a good privacy practice, and we should follow it if we can.

As @markpundsack mentions, this should be a heroku.com domain, to make it clear it's first-party.

Agreed on being good policy to ask up-front, but technically I thought we only needed to provide out-out for third-party analytics. If this is going directly to us, I was under the impression we didn’t absolutely have to provide an opt-out. Now my knowledge might be out of date, and certainly predates us being Safe Harbor, but when we implemented this in Dashboard, we made the opt-out explicitly for third-party. I don’t think we actually have any first-party analytics in Dashboard, so I don’t think we’re in violation either way. But it would be good to get clarity.

thanks for looking @jacobian! We can get opt-in functionality in, it's a little tricky since I need to make it some kind of auto-canceling opt-in (in case an automated script is running the CLI), but it's doable.

The prompting + timeout is now in. Just waiting this before we can release https://github.com/heroku/foundation/issues/826


Fix the alignment of the excuses

Just use the rect that is passed as argument instead of the activeScreenRect which was always the frame from the last screen

Awesome, thx!


Update README.md

Here is a new revision! Sorry for the mistakes I had made in last request!

As I said on your other pull request (PR), all you have to do to submit changes is make another commit on your branch. Making a completely new PR is not necessary and causes confusion, so please close your previous PR #45.


STUN issues on outbound call - 401 Unauthorized

Describe the bug
Hi,
I'm having an issue with outbound calls form my WebRTC client (SIP.JS via Web Socket). Results with no audio at all.
The issue is only happening for 1 call out of 3-5, and for some reason only on Google Chrome, on FireFox I am not able to reproduce it.
When the issue reproduced, what I'm seeing on Wireshark capture is that RTPEngine is sending me a STUN bind request, and I am sending back a response with error of 401 unauthorized (you can see it in the Wireshark capture).
This is my deployment:

SIP.js 0.17.1 client.
Kamailio 5.4.2
RTPEgine <IP_ADDRESS>
FreeSWITCH 1.10.5
Google Chrome 87.0.4280.66

No private TURN server in this deployment, the client is working on a local LAN and is using the default STUN server of SIP.js (Google's).
This is the call flow:
SIP.js client ---(SIP via WSS)--> Kamailio (+RTPEngine) ---(SIP)--> FreeSWITCH ---(SIP)----> PSTN
Logs
SIP messages (INVITE + 183 + 200):
SIP traffic from SIP.js client console
Also, Kamailio + RTPEngine logs if it helps:
logs
Because in those logs we can see the following error that is related to the ICE messages:
[1606730670.596288] NOTICE: [ji6h1tveo2ota7klijqn port 23020]: Received invalid STUN packet from <IP_ADDRESS>:59040: MESSAGE_INTEGRITY attribute missing
And here are the Wireshark captures of my PC that is running the SIP.js client and Kamailio server that is getting and responding the ICE messages.
Wireshark captures
To Reproduce (if possible)
Reproducible only on Google Chrome and probably only on my environment.
Observed behavior
When there's an issue with a call, we can see that RTPEngine is sending a STUN bind request and the SIP.js client (Chrome WebRTC on my PC) is responding with 401 Unauthorized. Happens only to a one call out of 3-5 more or less.
On chrome://webrtc-internals I can see that the relevant candidate couldn't be gathered and connected (because of the 401 error) and the ICE candidate pair couldn't be established.
Environment Information

SIP.js 0.17.1 client.
Kamailio 5.4.2
RTPEgine <IP_ADDRESS>
FreeSWITCH 1.10.5
Google Chrome 87.0.4280.66

Not really sure where the issue can be, it happens only on Google Chrome, so I also suspect the WebRTC SDK itself but I am not sure yet. Hope you can see something that I am missing here.
Please tell me if you need me to provide any other logs / info.
Thanks!

OK, so it's not a SIP.js issue it was Kamailio + RTPEngine issue (more like misconfiguration on my Kamailio config side).
I'll just share some small and brief explanation if someone else will experience this problem and encounter the same behavior:
When SIP.js client made an outbound INVITE, FreeSWITCH rejected it with 407 as there's call authentication enabled.
Only then, SIP.js client is sending the final INVITE with the autentication. So far so good, but not on Kamailio's side.
Kamailio got the initial INVITE (without authentication) and treated it as a call (which is OK), but then got a 407 and it immediately tore down the call and it's state. Then it gets the second INVITE (with the authentication part) and it is creating a new call on Kamailio (and RTPEngine) for this call.
So seems like sometimes (one call out of 3-5) had STUN issues as a result of the Kamailio is tearing the initial call state very fast so the hash authentication couldn't be done correctly (this is my guess).
So on my Kamailio config I added a condition that when there's a branch failure, first check if the failure is as a result of a 407 error that related to an INVITE. If not INVITE + 407, it'll act like it should: which is to tear down the call. If it's INVITE + 407, do nothing.
Also before solving the root cause as explained above, I found that using delete-delay=3 (3 seconds for example) on RTPEngine configurations is also solving it, but it is more like workaround instead of solution of the root issue.