WIP: normalize tool output to BED format

each tool currently outputs its own format. This PR is to make them all output BED. If users request --columns chrom, start, end, then those are ignored since chrom, start, end are always prefixed to be the first columns reported.
See changes in test-de-novo.sh . @arq5x does this seem the best way to go?
Todo

[ ] adjust non inheritance tools (compound_het, etc)
[ ] update tests (only de_novo updated so far).
[ ] re-open this PR on dev branch once it is up-to-date.

Great idea!  @brentp thanks for unifying the output!

@bgruening , glad it will be useful. Other than outputting BED (prefixing the current output with chrom, start, end), is there any other standardization that will make downstream use simpler?

In general the more similar the output of all those tools is to the standard gemini query format the better.

@brentp sticking to standards is all what I can imagine/wish (in general, actually gemini do a good job in this already.)
Does the switch to turn off the header will stay?

@bgruening this PR doesn't affect the main query program, just tools like de_novo and autosomal_recessive which don't have a switch to turn off the header. Is that what you mean?
@dgaston , does the output in the tests (https://github.com/arq5x/gemini/pull/436/files#diff-6467e2e50520e0dee563c76935e74e05R15) as changed seem reasonable. This is different from the main query format in that certain output columns must be reported.

@brentp in Galaxy such a switch would be handy, otherwise I can simple drop the first line, both works for me.

@brentp So for instance, why not just have the genotype columns reported as they are currently in the query format (each individual has their own column). The additional column of the family member details is ok.

@dgaston, that makes sense to me. So only required extra columns of:

family_id
family_members (comma delimited in single column)
family_genotypes (comma delimited)

and other columns can be requested with gt.* ? Those 3 columns will always appear at the end of the output. Then users can request --columns "chrom, start, end, ..." to get BED formatted output (this would remove the automatic bed output in commit above).
I'll wait for others to weigh in.

Summing up, we are proposing:

If --columns is not used, all columns will be reported, followed by family_id, family_members, and family_genotypes.
If --columns is  used, those columns will be reported, followed by family_id, family_members, and family_genotypes.
If BED format is wanted, users should start the --columns as : --columns "chrom,,start,end"

The compound heterozygotes tool is a special case because it has an extra comp_het_id column. In its case, the comp_het_id will follow (or precede?) the above four columns.
Also, the forthcomiung Mendelian violations tool will adhere to these format guidlines.
@jxchong - Given your feedback, I am wondering if you have any comments about these proposed changes?

I think these changes sound good.  I think for situation #1, the default should be all columns, with the columns beginning with chrom, start, end.
I think the comp_het tool should follow the BED format columns and have comp_het_id follow after (could just be column 4/5 or the last column?)
On April 13, 2015 at 12:16:07 PM, Aaron Quinlan<EMAIL_ADDRESS>wrote:
Summing up, we are proposing:
If --columns is not used, all columns will be reported, followed by family_id, family_members, and family_genotypes.
If --columns is used, those columns will be reported, followed by family_id, family_members, and family_genotypes.
If BED format is wanted, users should start the --columns as : --columns "chrom,,start,end"
The compound heterozygotes tool is a special case because it has an extra comp_het_id column. In its case, the comp_het_id will follow (or precede?) the above four columns.
Also, the forthcomiung Mendelian violations tool will adhere to these format guidlines.
@jxchong - Given your feedback, I am wondering if you have any comments about these proposed changes?
—
Reply to this email directly or view it on GitHub.

I have made the update to the change so that it is as discussed. If they don't request anything, then we just ask for "*" which will output chrom, start, end first since those are the first 3 columns in the database.
For the coming mendelian error tool, it will output requested columns along with a "violation" column indicating the type of violation (LOH/disomy, de novo, etc.). I will likely add a probability column that uses the PL's if available for mendelian errors too.
Adding tests and working on comp_het tool.

@brentp and I discussed this a bit more offline.

The sample column. There is an additional complexity for the comp_hets tool in that, unlike the others, it also adds a sample column reflecting the sample in which the CH was observed. I actually think this is a useful default column for the built in disease model tools, as simply listing family ids and genotypes makes it sometimes difficult to see which samples (particularly in a large family with multiple offspring) had the variant/event.  For example, if there are two or more affected individuals in a family, the report a de novo mutation would be slightly difficult to see by eye (or inspect programiatically) if when just one of the two kids has the de novo.  A sample column would state which of the two had the event.  For other disease models, the sample column could be used to list the samples that had the event.
Therefore, I propose that all tools report a samples column after the aforementioned family_id, family_members, and family_genotypes columns.

A common variant_id column. The comp_hets tool also uses a comp_het_id to track which lines represent a given comp het candidate (i.e., two rows per event, inherently). In the interest of being able to concatenate the results of multiple tools (e.g., running both autosomal_recessive and comp_hets in search of recessive variants), I propose that all tools report a unified variant_id column reflecting the variant_id column the variants table.  It should come before the family_id, family_members, family_genotypes, and samples columns. For all tools other than comp_hets, it should just be the variant_id in the table. For comp_hets, it should be a composite: variant_id + _ + comp_het_id.  For example:  if the first comp_het candidate was made up variant 7 and 9, the variant_ids for the two rows would be 1_7 and 1_9.

In summary, all tools should have the following fixed output columns that will always follow any columns specified by --columns:

variant_id
family_id
family_members (comma delimited in single column)
family_genotypes (comma delimited)
samples

So, if the user specified --columns "chrom,start,end", the output would be:

chrom
start
end
variant_id
family_id
family_members (comma delimited in single column)
family_genotypes (comma delimited)
samples

We are interested in your further thoughts, as we would like to wrap this up for the next release.

I think these additions are a GREAT idea and will make it considerably easier to combine analyses across multiple tools.
How would you represent variants with multiple families (and family members) with the same mutation? You’d have multiple family_ids (easy) but then possibly multiple family_members, family_genotypes, and samples per family_id,
On April 13, 2015 at 5:22:44 PM, Aaron Quinlan<EMAIL_ADDRESS>wrote:
@brentp and I discussed this a bit more offline.
The sample column. There is an additional complexity for the comp_hets tool in that, unlike the others, it also adds a sample column reflecting the sample in which the CH was observed. I actually think this is a useful default column for the built in disease model tools, as simply listing family ids and genotypes makes it sometimes difficult to see which samples (particularly in a large family with multiple offspring) had the variant/event. For example, if there are two or more affected individuals in a family, the report a de novo mutation would be slightly difficult to see by eye (or inspect programiatically) if when just one of the two kids has the de novo. A sample column would state which of the two had the event. For other disease models, the sample column could be used to list the samples that had the event.
Therefore, I propose that all tools report a samples column after the aforementioned family_id, family_members, and family_genotypes columns.
A common variant_id column. The comp_hets tool also uses a comp_het_id to track which lines represent a given comp het candidate (i.e., two rows per event, inherently). In the interest of being able to concatenate the results of multiple tools (e.g., running both autosomal_recessive and comp_hets in search of recessive variants), I propose that all tools report a unified variant_id column reflecting the variant_id column the variants table. It should come before the family_id, family_members, family_genotypes, and samples columns. For all tools other than comp_hets, it should just be the variant_id in the table. For comp_hets, it should be a composite: variant_id + _ + comp_het_id. For example: if the first comp_het candidate was made up variant 7 a nd 9, th e variant_ids for the two rows would be 1_7 and 1_9.
In summary, all tools should have the following fixed output columns that will always follow any columns specified by --columns:
variant_id
family_id
family_members (comma delimited in single column)
family_genotypes (comma delimited)
samples
So, if the user specified --columns "chrom,start,end", the output would be:
chrom
start
end
variant_id
family_id
family_members (comma delimited in single column)
family_genotypes (comma delimited)
samples
We are interested in your further thoughts, as we would like to wrap this up for the next release.
—
Reply to this email directly or view it on GitHub.

Good question, @jxchong.
I see two ways of handling that.

Use a different delimiter (e.g., "|" or ";") to separate the comma-separated family_id and family_genotype lists.
Report each family on a separate line and use the common variant_id to track the fact that the same variant was observed in multiple families.

From a output cleanliness perspective, I prefer #2. But I suspect #1 will be more useful from a data interpretation perspective.

I like 2. It's easier to parse and avoids packing stuff into an already overridden column.

I prefer 2 as well.  The case where it gets a bit harder is comp_hets where the variant_id is a composite of variant_id and comp_het_id.  But I agree that it is cleaner for sure.
One thing that could be done to further facilitate parsing and downstream filtering is to add one more column to the default set of columns called family_count, which would reflect the count of families that were observed to have a given variant id.  This would trigger the researcher to know that even though the given line reflects one family, there were N-1 other families with the same variant.

I like adding family_count as a column to make it easier to visually parse.
On April 13, 2015 at 6:01 PM, Aaron Quinlan<EMAIL_ADDRESS>wrote:
I prefer 2 as well. The case where it gets a bit harder is comp_hets where the variant_id is a composite of variant_id and comp_het_id. But I agree that it is cleaner for sure.
One thing that could be done to further facilitate parsing and downstream filtering is to add one more column to the default set of columns called family_count, which would reflect the count of families that were observed to have a given variant id. This would trigger the researcher to know that even though the given line reflects one family, there were N-1 other families with the same variant.
—
Reply to this email directly or view it on GitHub.

One potential source of confusion is that the --min-kindreds argument operates by gene. This will now  introducing a new, but similar count that is by variant.

Hmm, good point. However, the total family_count across variants will always be >= the --min-kindred setting. That said, I agree it might be a bit confusing.

Just throwing out another idea: just report the list of family_ids without reporting the family members with the variant?  In theory since this is (only?) an issue when using the built-in analysis tools, you should already be able to infer the genotypes of the family members for that variant since you already know the variant is in that family?
On April 14, 2015 at 10:33:07 AM, Aaron Quinlan<EMAIL_ADDRESS>wrote:
Hmm, good point. However, the total family_count across variants will always be >= the --min-kindred setting. That said, I agree it might be a bit confusing.
—
Reply to this email directly or view it on GitHub.

Here is the output from a few tools. I believe it matches what was proposed. Please let me know any changes:
mendel
brentp@x1:/usr/local/src/gemini/test$ gemini mendel_errors  --columns "chrom, start, end, ref, alt, impact, impact_severity" test.auto_dom.no_parents.db | cols
WARNING: Unable to find parents for family (2). GEMINI is currently only able to identify candidates from two generational families.
chrom  start     end       ref  alt  impact          impact_severity  violation  variant_id         family_id  family_members                                                     family_genotypes  affected_samples  family_count
chr10  1142207   1142208   T    C    stop_loss       HIGH             1          plausible de novo  1          1_dad(dad;unaffected),1_mom(mom;unaffected),1_kid(child;affected)  T/T,T/T,T/C       1_kid             1
chr10  48003991  48003992  C    T    non_syn_coding  MED              3          plausible de novo  1          1_dad(dad;unaffected),1_mom(mom;unaffected),1_kid(child;affected)  C/C,C/C,C/T       1_kid             1
chr10  48004991  48004992  C    T    non_syn_coding  MED              4          plausible de novo  1          1_dad(dad;unaffected),1_mom(mom;unaffected),1_kid(child;affected)  C/C,C/C,C/T       1_kid             1

autosomal dominant
brentp@x1:/usr/local/src/gemini/test$ gemini autosomal_dominant  --columns "chrom, start, end, ref, alt, impact_severity" test.auto_dom.db | cols
chrom  start      end        ref  alt  impact_severity  gene    variant_id  family_id  family_members                                                   family_genotypes  affected_samples  family_count
chr10  48003991   48003992   C    T    MED              ASAH2C  3           3          3_dad(dad;affected),3_mom(mom;unknown),3_kid(child;affected)     C/T,C/C,C/T       3_dad,3_kid       2
chr10  48003991   48003992   C    T    MED              ASAH2C  3           2          2_dad(dad;unaffected),2_mom(mom;affected),2_kid(child;affected)  C/C,C/T,C/T       2_mom,2_kid       2
chr10  48004991   48004992   C    T    MED              ASAH2C  4           3          3_dad(dad;affected),3_mom(mom;unknown),3_kid(child;affected)     C/T,C/C,C/T       3_dad,3_kid       2
chr10  48004991   48004992   C    T    MED              ASAH2C  4           2          2_dad(dad;unaffected),2_mom(mom;affected),2_kid(child;affected)  C/C,C/T,C/T       2_mom,2_kid       2
chr10  135336655  135336656  G    A    LOW              SPRN    5           3          3_dad(dad;affected),3_mom(mom;unknown),3_kid(child;affected)     G/A,G/G,G/A       3_dad,3_kid       1
chr10  1142207    1142208    T    C    HIGH             WDR37   1           3          3_dad(dad;affected),3_mom(mom;unknown),3_kid(child;affected)     T/C,T/T,T/C       3_dad,3_kid       2
chr10  1142207    1142208    T    C    HIGH             WDR37   1           2          2_dad(dad;unaffected),2_mom(mom;affected),2_kid(child;affected)  T/T,T/C,T/C       2_mom,2_kid       2

comp_het
brentp@x1:/usr/local/src/gemini/test$ gemini comp_hets     --ignore-phasing     --columns "chrom, start, end"     --families 3     test.comp_het_default.2.db | cols
chrom  start  end    gene    alt  variant_id  comp_het_id  family_id  family_members                                                       family_genotypes         affected_samples  family_count
chr1   17362  17366  WASH7P  T    1_1         1            3          dad_3(dad;unaffected),mom_3(mom;unaffected),child_3(child;affected)  TTCT|T,TTCT|TTCT,TTCT|T  child_3           2
chr1   17362  17366  WASH7P  T    1_1         1            3          dad_3(dad;unaffected),mom_3(mom;unaffected),child_3(child;affected)  TTCT|T,TTCT|TTCT,TTCT|T  child_3           2
chr1   17729  17730  WASH7P  A    2_1         1            3          dad_3(dad;unaffected),mom_3(mom;unaffected),child_3(child;affected)  C|A,C|A,A|C              child_3           2
chr1   17729  17730  WASH7P  A    2_1         1            3          dad_3(dad;unaffected),mom_3(mom;unaffected),child_3(child;affected)  C|A,C|A,A|C              child_3           2

I think this looks pretty good.
On April 14, 2015 at 12:47:19 PM, Brent Pedersen - Bioinformatics<EMAIL_ADDRESS>wrote:
Here is the output from a few tools. I believe it matches what was proposed. Please let me know any changes:
mendel
brentp@x1:/usr/local/src/gemini/test$ gemini mendel_errors  --columns "chrom, start, end, ref, alt, impact, impact_severity" test.auto_dom.no_parents.db | cols
WARNING: Unable to find parents for family (2). GEMINI is currently only able to identify candidates from two generational families.
chrom  start     end       ref  alt  impact          impact_severity  violation  variant_id         family_id  family_members                                                     family_genotypes  affected_samples  family_count
chr10  1142207   1142208   T    C    stop_loss       HIGH             1          plausible de novo  1          1_dad(dad;unaffected),1_mom(mom;unaffected),1_kid(child;affected)  T/T,T/T,T/C       1_kid             1
chr10  48003991  48003992  C    T    non_syn_coding  MED              3          plausible de novo  1          1_dad(dad;unaffected),1_mom(mom;unaffected),1_kid(child;affected)  C/C,C/C,C/T       1_kid             1
chr10  48004991  48004992  C    T    non_syn_coding  MED              4          plausible de novo  1          1_dad(dad;unaffected),1_mom(mom;unaffected),1_kid(child;affected)  C/C,C/C,C/T       1_kid             1
autosomal dominant
brentp@x1:/usr/local/src/gemini/test$ gemini autosomal_dominant  --columns "chrom, start, end, ref, alt, impact_severity" test.auto_dom.db | cols
chrom  start      end        ref  alt  impact_severity  gene    variant_id  family_id  family_members                                                   family_genotypes  affected_samples  family_count
chr10  48003991   48003992   C    T    MED              ASAH2C  3           3          3_dad(dad;affected),3_mom(mom;unknown),3_kid(child;affected)     C/T,C/C,C/T       3_dad,3_kid       2
chr10  48003991   48003992   C    T    MED              ASAH2C  3           2          2_dad(dad;unaffected),2_mom(mom;affected),2_kid(child;affected)  C/C,C/T,C/T       2_mom,2_kid       2
chr10  48004991   48004992   C    T    MED              ASAH2C  4           3          3_dad(dad;affected),3_mom(mom;unknown),3_kid(child;affected)     C/T,C/C,C/T       3_dad,3_kid       2
chr10  48004991   48004992   C    T    MED              ASAH2C  4           2          2_dad(dad;unaffected),2_mom(mom;affected),2_kid(child;affected)  C/C,C/T,C/T       2_mom,2_kid       2
chr10  135336655  135336656  G    A    LOW              SPRN    5           3          3_dad(dad;affected),3_mom(mom;unknown),3_kid(child;affected)     G/A,G/G,G/A       3_dad,3_kid       1
chr10  1142207    1142208    T    C    HIGH             WDR37   1           3          3_dad(dad;affected),3_mom(mom;unknown),3_kid(child;affected)     T/C,T/T,T/C       3_dad,3_kid       2
chr10  1142207    1142208    T    C    HIGH             WDR37   1           2          2_dad(dad;unaffected),2_mom(mom;affected),2_kid(child;affected)  T/T,T/C,T/C       2_mom,2_kid       2
comp_het
brentp@x1:/usr/local/src/gemini/test$ gemini comp_hets     --ignore-phasing     --columns "chrom, start, end"     --families 3     test.comp_het_default.2.db | cols
chrom  start  end    gene    alt  variant_id  comp_het_id  family_id  family_members                                                       family_genotypes         affected_samples  family_count
chr1   17362  17366  WASH7P  T    1_1         1            3          dad_3(dad;unaffected),mom_3(mom;unaffected),child_3(child;affected)  TTCT|T,TTCT|TTCT,TTCT|T  child_3           2
chr1   17362  17366  WASH7P  T    1_1         1            3          dad_3(dad;unaffected),mom_3(mom;unaffected),child_3(child;affected)  TTCT|T,TTCT|TTCT,TTCT|T  child_3           2
chr1   17729  17730  WASH7P  A    2_1         1            3          dad_3(dad;unaffected),mom_3(mom;unaffected),child_3(child;affected)  C|A,C|A,A|C              child_3           2
chr1   17729  17730  WASH7P  A    2_1         1            3          dad_3(dad;unaffected),mom_3(mom;unaffected),child_3(child;affected)  C|A,C|A,A|C              child_3           2
—
Reply to this email directly or view it on GitHub.

@arq5x and I just chatted. We'll have the standard columns as before and any tool that has additional columns like comp_het and mendel_error will append the the column at the end. This will simplify using the output from mutliple tools unchanged.

I agree.  @brentp, as we just discussed, I think the only thing to fix is that comp_hets has an extra comp_het_id column that disrupts the symmetry of the output across the tools.  Let's instead move that column to the end of the output so as to no disrupt the symmetry. Secondly, since comp_het_id is its own column, let's just make variant_id be the comp_het_id from the database, rather than a composite of variant_id and comp_het_id.

oops, you beat me to it.


Decide on how to clean up Open Tree of Life induced subtree

Open Tree of Life induced subtrees (as added by #40) include a large number of internal nodes that only have a single child, which can make visualizing the phylogeny complicated. However, we can't just delete all such internal nodes when importing a phylogeny from the Open Tree of Life as they might be useful -- for example, one of these nodes might be a specifier. As far as I know, there's no setting on the Open Tree /tree_of_life/induced_subtree API call to control this.
We could add a "Remove internal nodes with only a single child" button to the phylogeny view, which would ignore internal nodes that are used as specifiers. We could also go further and add a "Simplify phylogeny" button that removes all phylogeny nodes except specifiers and their parents. Alternatively, we could decide that this is too much complexity for Klados and leave it up to the user to use an external tool to clean up their phylogenies in any way they choose.

It seems reasonable and also useful to a Simplify phylogeny option that removes internal nodes that only have one child and whose label does not match any of the specifiers.
I think that's neither of the alternatives you present but an amalgamation of them. Does this make sense, though?


New post

Description of changes:
AWS Amplify + React Native / Authentication 🔐 FULL SETUP
https://itnext.io/aws-amplify-react-native-authentication-full-setup-7764b452a138
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Thanks for the post! merging


Windows antivirus reports that it contains a trojan

it gets blocked after downloading zip or clone repository
Trojan:Script/Wacatac.B!ml

Most likely a false positive from Defender. A Google search will tell you more.


Issue with Conv1D when groups > 1 and using our own TensorFlow builds

System information

OS Platform and Distribution: Linux Red Hat Enterprise 8.1
TensorFlow installed from: source
TensorFlow version: v2.5.0-0-ga4dfb8d1a71 2.5.0 (more generally any version starting from 2.3.1)
Python version: 3.7.10
Bazel version: 3.7.2
GCC/Compiler version: 8.3.1
CUDA/cuDNN version: 11.2 / 8.0
GPU model and memory: NVIDIA V100

Describe the current behavior
When using our own TensorFlow builds (starting at version 2.3.1 which added the  groups parameter), the following code snippet fails for groups greater than 1:
import tensorflow as tf
import traceback

for g in (1,2,4):
    try:
        print(f"groups={g}")
        c = tf.keras.layers.Conv1D(4,4,groups=g)
        print(c(tf.ones((2,16,4))))
    except Exception as e:
        traceback.print_exc()
    finally:
        print()

Output is as follow:
groups=1
tf.Tensor(
[[[ 0.28654966  0.9454404  -1.1466699  -0.91166556]
  [ 0.28654966  0.9454404  -1.1466699  -0.91166556]
  [ 0.28654966  0.9454404  -1.1466699  -0.91166556]
  [ 0.28654966  0.9454404  -1.1466699  -0.91166556]
  [ 0.28654966  0.9454404  -1.1466699  -0.91166556]
  [ 0.28654966  0.9454404  -1.1466699  -0.91166556]
  [ 0.28654966  0.9454404  -1.1466699  -0.91166556]
  [ 0.28654966  0.9454404  -1.1466699  -0.91166556]
  [ 0.28654966  0.9454404  -1.1466699  -0.91166556]
  [ 0.28654966  0.9454404  -1.1466699  -0.91166556]
  [ 0.28654966  0.9454404  -1.1466699  -0.91166556]
  [ 0.28654966  0.9454404  -1.1466699  -0.91166556]
  [ 0.28654966  0.9454404  -1.1466699  -0.91166556]]

 [[ 0.28654966  0.9454404  -1.1466699  -0.91166556]
  [ 0.28654966  0.9454404  -1.1466699  -0.91166556]
  [ 0.28654966  0.9454404  -1.1466699  -0.91166556]
  [ 0.28654966  0.9454404  -1.1466699  -0.91166556]
  [ 0.28654966  0.9454404  -1.1466699  -0.91166556]
  [ 0.28654966  0.9454404  -1.1466699  -0.91166556]
  [ 0.28654966  0.9454404  -1.1466699  -0.91166556]
  [ 0.28654966  0.9454404  -1.1466699  -0.91166556]
  [ 0.28654966  0.9454404  -1.1466699  -0.91166556]
  [ 0.28654966  0.9454404  -1.1466699  -0.91166556]
  [ 0.28654966  0.9454404  -1.1466699  -0.91166556]
  [ 0.28654966  0.9454404  -1.1466699  -0.91166556]
  [ 0.28654966  0.9454404  -1.1466699  -0.91166556]]], shape=(2, 13, 4), dtype=float32)

groups=2
Traceback (most recent call last):
  File "groups.py", line 8, in <module>
    print(c(tf.ones((2,16,4))))
  File "/.../pub/anaconda-py3/2020.11/envs/tensorflow-gpu-2.5.0/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 1030, in __call__
    outputs = call_fn(inputs, *args, **kwargs)
  File "/.../pub/anaconda-py3/2020.11/envs/tensorflow-gpu-2.5.0/lib/python3.7/site-packages/tensorflow/python/keras/layers/convolutional.py", line 249, in call
    outputs = self._convolution_op(inputs, self.kernel)
  File "/.../pub/anaconda-py3/2020.11/envs/tensorflow-gpu-2.5.0/lib/python3.7/site-packages/tensorflow/python/util/dispatch.py", line 206, in wrapper
    return target(*args, **kwargs)
  File "/.../pub/anaconda-py3/2020.11/envs/tensorflow-gpu-2.5.0/lib/python3.7/site-packages/tensorflow/python/ops/nn_ops.py", line 1019, in convolution_v2
    name=name)
  File "/.../pub/anaconda-py3/2020.11/envs/tensorflow-gpu-2.5.0/lib/python3.7/site-packages/tensorflow/python/ops/nn_ops.py", line 1149, in convolution_internal
    name=name)
  File "/.../pub/anaconda-py3/2020.11/envs/tensorflow-gpu-2.5.0/lib/python3.7/site-packages/tensorflow/python/util/dispatch.py", line 206, in wrapper
    return target(*args, **kwargs)
  File "/.../pub/anaconda-py3/2020.11/envs/tensorflow-gpu-2.5.0/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py", line 602, in new_func
    return func(*args, **kwargs)
  File "/.../pub/anaconda-py3/2020.11/envs/tensorflow-gpu-2.5.0/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py", line 602, in new_func
    return func(*args, **kwargs)
  File "/.../pub/anaconda-py3/2020.11/envs/tensorflow-gpu-2.5.0/lib/python3.7/site-packages/tensorflow/python/ops/nn_ops.py", line 1892, in conv1d
    name=name)
  File "/.../pub/anaconda-py3/2020.11/envs/tensorflow-gpu-2.5.0/lib/python3.7/site-packages/tensorflow/python/ops/gen_nn_ops.py", line 932, in conv2d
    _ops.raise_from_not_ok_status(e, name)
  File "/.../pub/anaconda-py3/2020.11/envs/tensorflow-gpu-2.5.0/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 6897, in raise_from_not_ok_status
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.InvalidArgumentError: input and filter must have the same depth: 4 vs 2 [Op:Conv2D]

groups=4
Traceback (most recent call last):
  File "groups.py", line 8, in <module>
    print(c(tf.ones((2,16,4))))
  File "/.../pub/anaconda-py3/2020.11/envs/tensorflow-gpu-2.5.0/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 1030, in __call__
    outputs = call_fn(inputs, *args, **kwargs)
  File "/.../pub/anaconda-py3/2020.11/envs/tensorflow-gpu-2.5.0/lib/python3.7/site-packages/tensorflow/python/keras/layers/convolutional.py", line 249, in call
    outputs = self._convolution_op(inputs, self.kernel)
  File "/.../pub/anaconda-py3/2020.11/envs/tensorflow-gpu-2.5.0/lib/python3.7/site-packages/tensorflow/python/util/dispatch.py", line 206, in wrapper
    return target(*args, **kwargs)
  File "/.../pub/anaconda-py3/2020.11/envs/tensorflow-gpu-2.5.0/lib/python3.7/site-packages/tensorflow/python/ops/nn_ops.py", line 1019, in convolution_v2
    name=name)
  File "/.../pub/anaconda-py3/2020.11/envs/tensorflow-gpu-2.5.0/lib/python3.7/site-packages/tensorflow/python/ops/nn_ops.py", line 1149, in convolution_internal
    name=name)
  File "/.../pub/anaconda-py3/2020.11/envs/tensorflow-gpu-2.5.0/lib/python3.7/site-packages/tensorflow/python/util/dispatch.py", line 206, in wrapper
    return target(*args, **kwargs)
  File "/.../pub/anaconda-py3/2020.11/envs/tensorflow-gpu-2.5.0/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py", line 602, in new_func
    return func(*args, **kwargs)
  File "/.../pub/anaconda-py3/2020.11/envs/tensorflow-gpu-2.5.0/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py", line 602, in new_func
    return func(*args, **kwargs)
  File "/.../pub/anaconda-py3/2020.11/envs/tensorflow-gpu-2.5.0/lib/python3.7/site-packages/tensorflow/python/ops/nn_ops.py", line 1892, in conv1d
    name=name)
  File "/.../pub/anaconda-py3/2020.11/envs/tensorflow-gpu-2.5.0/lib/python3.7/site-packages/tensorflow/python/ops/gen_nn_ops.py", line 932, in conv2d
    _ops.raise_from_not_ok_status(e, name)
  File "/.../pub/anaconda-py3/2020.11/envs/tensorflow-gpu-2.5.0/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 6897, in raise_from_not_ok_status
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.InvalidArgumentError: input and filter must have the same depth: 4 vs 1 [Op:Conv2D]

Describe the expected behavior
If I run the same code snippet using a TensorFlow build from pip, I do not get any errors but I have a hard time understanding how this can build dependent:
groups=1
tf.Tensor(
[[[-0.05965281 -1.473797   -0.5487337  -0.34858704]
  [-0.05965281 -1.473797   -0.5487337  -0.34858704]
  [-0.05965281 -1.473797   -0.5487337  -0.34858704]
  [-0.05965281 -1.473797   -0.5487337  -0.34858704]
  [-0.05965281 -1.473797   -0.5487337  -0.34858704]
  [-0.05965281 -1.473797   -0.5487337  -0.34858704]
  [-0.05965281 -1.473797   -0.5487337  -0.34858704]
  [-0.05965281 -1.473797   -0.5487337  -0.34858704]
  [-0.05965281 -1.473797   -0.5487337  -0.34858704]
  [-0.05965281 -1.473797   -0.5487337  -0.34858704]
  [-0.05965281 -1.473797   -0.5487337  -0.34858704]
  [-0.05965281 -1.473797   -0.5487337  -0.34858704]
  [-0.05965281 -1.473797   -0.5487337  -0.34858704]]

 [[-0.05965281 -1.473797   -0.5487337  -0.34858704]
  [-0.05965281 -1.473797   -0.5487337  -0.34858704]
  [-0.05965281 -1.473797   -0.5487337  -0.34858704]
  [-0.05965281 -1.473797   -0.5487337  -0.34858704]
  [-0.05965281 -1.473797   -0.5487337  -0.34858704]
  [-0.05965281 -1.473797   -0.5487337  -0.34858704]
  [-0.05965281 -1.473797   -0.5487337  -0.34858704]
  [-0.05965281 -1.473797   -0.5487337  -0.34858704]
  [-0.05965281 -1.473797   -0.5487337  -0.34858704]
  [-0.05965281 -1.473797   -0.5487337  -0.34858704]
  [-0.05965281 -1.473797   -0.5487337  -0.34858704]
  [-0.05965281 -1.473797   -0.5487337  -0.34858704]
  [-0.05965281 -1.473797   -0.5487337  -0.34858704]]], shape=(2, 13, 4), dtype=float32)

groups=2
tf.Tensor(
[[[ 0.555179   -0.11330175  0.22858751 -1.1606797 ]
  [ 0.555179   -0.11330175  0.22858751 -1.1606797 ]
  [ 0.555179   -0.11330175  0.22858751 -1.1606797 ]
  [ 0.555179   -0.11330175  0.22858751 -1.1606797 ]
  [ 0.555179   -0.11330175  0.22858751 -1.1606797 ]
  [ 0.555179   -0.11330175  0.22858751 -1.1606797 ]
  [ 0.555179   -0.11330175  0.22858751 -1.1606797 ]
  [ 0.555179   -0.11330175  0.22858751 -1.1606797 ]
  [ 0.555179   -0.11330175  0.22858751 -1.1606797 ]
  [ 0.555179   -0.11330175  0.22858751 -1.1606797 ]
  [ 0.555179   -0.11330175  0.22858751 -1.1606797 ]
  [ 0.555179   -0.11330175  0.22858751 -1.1606797 ]
  [ 0.555179   -0.11330175  0.22858751 -1.1606797 ]]

 [[ 0.555179   -0.11330175  0.22858751 -1.1606797 ]
  [ 0.555179   -0.11330175  0.22858751 -1.1606797 ]
  [ 0.555179   -0.11330175  0.22858751 -1.1606797 ]
  [ 0.555179   -0.11330175  0.22858751 -1.1606797 ]
  [ 0.555179   -0.11330175  0.22858751 -1.1606797 ]
  [ 0.555179   -0.11330175  0.22858751 -1.1606797 ]
  [ 0.555179   -0.11330175  0.22858751 -1.1606797 ]
  [ 0.555179   -0.11330175  0.22858751 -1.1606797 ]
  [ 0.555179   -0.11330175  0.22858751 -1.1606797 ]
  [ 0.555179   -0.11330175  0.22858751 -1.1606797 ]
  [ 0.555179   -0.11330175  0.22858751 -1.1606797 ]
  [ 0.555179   -0.11330175  0.22858751 -1.1606797 ]
  [ 0.555179   -0.11330175  0.22858751 -1.1606797 ]]], shape=(2, 13, 4), dtype=float32)

groups=4
tf.Tensor(
[[[-0.9354149   0.7340534  -0.01698902  0.4960574 ]
  [-0.9354149   0.7340534  -0.01698902  0.4960574 ]
  [-0.9354149   0.7340534  -0.01698902  0.4960574 ]
  [-0.9354149   0.7340534  -0.01698902  0.4960574 ]
  [-0.9354149   0.7340534  -0.01698902  0.4960574 ]
  [-0.9354149   0.7340534  -0.01698902  0.4960574 ]
  [-0.9354149   0.7340534  -0.01698902  0.4960574 ]
  [-0.9354149   0.7340534  -0.01698902  0.4960574 ]
  [-0.9354149   0.7340534  -0.01698902  0.4960574 ]
  [-0.9354149   0.7340534  -0.01698902  0.4960574 ]
  [-0.9354149   0.7340534  -0.01698902  0.4960574 ]
  [-0.9354149   0.7340534  -0.01698902  0.4960574 ]
  [-0.9354149   0.7340534  -0.01698902  0.4960574 ]]

 [[-0.9354149   0.7340534  -0.01698902  0.4960574 ]
  [-0.9354149   0.7340534  -0.01698902  0.4960574 ]
  [-0.9354149   0.7340534  -0.01698902  0.4960574 ]
  [-0.9354149   0.7340534  -0.01698902  0.4960574 ]
  [-0.9354149   0.7340534  -0.01698902  0.4960574 ]
  [-0.9354149   0.7340534  -0.01698902  0.4960574 ]
  [-0.9354149   0.7340534  -0.01698902  0.4960574 ]
  [-0.9354149   0.7340534  -0.01698902  0.4960574 ]
  [-0.9354149   0.7340534  -0.01698902  0.4960574 ]
  [-0.9354149   0.7340534  -0.01698902  0.4960574 ]
  [-0.9354149   0.7340534  -0.01698902  0.4960574 ]
  [-0.9354149   0.7340534  -0.01698902  0.4960574 ]
  [-0.9354149   0.7340534  -0.01698902  0.4960574 ]]], shape=(2, 13, 4), dtype=float32)

Any ideas would be greatly appreciated!

@RemiLacroix-IDRIS
I was able to reproduce the code shared in tf2.5 with no errors.Please find the gist here.
similar issue tracking here follow the thread aswell and also refer this comment.Thanks!

@UsharaniPagadala : As mentioned, that code works fine with the TensorFlow builds from pip but crashes with the builds we compile from source but that doesn't seem to make any sense.
I don't see how compilation can have in impact on this.

I had a closer look and the call path seems to be the same for both our compiled build and the build from pip.
The last call I could track is in function conv2d in tensorflow/python/ops/gen_nn_ops.py:
  _ctx = _context._context or _context.context()
  tld = _ctx._thread_local_data
  if tld.is_eager:
    try:
      _result = pywrap_tfe.TFE_Py_FastPathExecute(
        _ctx, "Conv2D", name, input, filter, "strides", strides,
        "use_cudnn_on_gpu", use_cudnn_on_gpu, "padding", padding,
        "explicit_paddings", explicit_paddings, "data_format", data_format,
        "dilations", dilations)
      return _result
    except _core._NotOkStatusException as e:
      _ops.raise_from_not_ok_status(e, name)
    except _core._FallbackException:
      pass

The call to pywrap_tfe.TFE_Py_FastPathExecute returns successfully in the PIP build but fails with ours. Not sure how to get more information on what happens in that call.
Note that the issue occurs both in a CPU only-context and a GPU context.

Not sure if it can help troubleshooting but it seems that non-eager mode works as expected on both our builds and the PIP builds:
import tensorflow as tf

class test(object):
   def __init__(self):
     self.cg1=tf.keras.layers.Conv1D(4,4,groups=1)
     self.cg2=tf.keras.layers.Conv1D(4,4,groups=2)
     self.cg4=tf.keras.layers.Conv1D(4,4,groups=4)

   @tf.function
   def call(self, inputs):
      tf.print("cgn", self.cg(inputs))
      tf.print("cg1", self.cg1(inputs))
      tf.print("cg2", self.cg2(inputs))
      tf.print("cg4", self.cg4(inputs))

tt=test()
tt.call(tf.ones((2,16,4)))

Not sure if it can help troubleshooting the issue but it seems that non-eager mode works as expected on both our builds and the PIP builds when using a GPU device:
import tensorflow as tf

class test(object):
   def __init__(self):
     self.cg1=tf.keras.layers.Conv1D(4,4,groups=1)
     self.cg2=tf.keras.layers.Conv1D(4,4,groups=2)
     self.cg4=tf.keras.layers.Conv1D(4,4,groups=4)

   @tf.function
   def call(self, inputs):
      tf.print("cg1", self.cg1(inputs))
      tf.print("cg2", self.cg2(inputs))
      tf.print("cg4", self.cg4(inputs))

tt=test()
tt.call(tf.ones((2,16,4)))

but it still crashes when using a CPU device and the error is different depending on the build used:

our own build:

Traceback (most recent call last):
  File "groups_non_eager.py", line 19, in <module>
    tt.call(tf.ones((2,16,4)))
  File "/.../pub/anaconda-py3/2020.11/envs/tensorflow-gpu-2.5.0/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 889, in __call__
    result = self._call(*args, **kwds)
  File "/.../pub/anaconda-py3/2020.11/envs/tensorflow-gpu-2.5.0/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 950, in _call
    return self._stateless_fn(*args, **kwds)
  File "/.../pub/anaconda-py3/2020.11/envs/tensorflow-gpu-2.5.0/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3024, in __call__
    filtered_flat_args, captured_inputs=graph_function.captured_inputs)  # pylint: disable=protected-access
  File "/.../pub/anaconda-py3/2020.11/envs/tensorflow-gpu-2.5.0/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1961, in _call_flat
    ctx, args, cancellation_manager=cancellation_manager))
  File "/.../pub/anaconda-py3/2020.11/envs/tensorflow-gpu-2.5.0/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 596, in call
    ctx=ctx)
  File "/.../pub/anaconda-py3/2020.11/envs/tensorflow-gpu-2.5.0/lib/python3.7/site-packages/tensorflow/python/eager/execute.py", line 60, in quick_execute
    inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.InvalidArgumentError:  input and filter must have the same depth: 4 vs 2
         [[node conv1d_3/conv1d (defined at groups_non_eager.py:16) ]] [Op:__inference_call_162]

Function call stack:
call

PIP build:

Traceback (most recent call last):
  File "groups_non_eager.py", line 19, in <module>
    tt.call(tf.ones((2,16,4)))
  File "~/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 889, in __call__
    result = self._call(*args, **kwds)
  File "~/.local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 950, in _call
    return self._stateless_fn(*args, **kwds)
  File "~/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3024, in __call__
    filtered_flat_args, captured_inputs=graph_function.captured_inputs)  # pylint: disable=protected-access
  File "~/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1961, in _call_flat
    ctx, args, cancellation_manager=cancellation_manager))
  File "~/.local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 596, in call
    ctx=ctx)
  File "~/.local/lib/python3.7/site-packages/tensorflow/python/eager/execute.py", line 60, in quick_execute
    inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.UnimplementedError:  Fused conv implementation does not support grouped convolutions for now.
         [[node conv1d_3/conv1d (defined at groups_non_eager.py:16) ]] [Op:__inference_call_162]

Function call stack:
call

Hello,
I just realized that our builds have XLA enabled (XLA_FLAGS=--xla_hlo_profile), I guess it could be relevant for this issue.

@angerson : Hi. Would it be possible to have some feedback on this issue?

Hi. Any news regarding this issue?

Hi @RemiLacroix-IDRIS ! I was not getting any error in 2.8 though. Can you please let us know from your end?

Hi @RemiLacroix-IDRIS ! I was not getting any error in 2.8 though. Can you please let us know from your end?

We currently don't have our own build of 2.8 since I couldn't try yet. It matters because the version from pip always was unaffected, my guess would be because it does not have XLA_FLAGS=--xla_hlo_profile.

The problem does not seem to appear in version 2.8.0 compiled from source.


check input error.

I've set up the linter with the following settings:
- name: Fail on malformatted files
        uses<EMAIL_ADDRESS>        with:
          check: true
          write: false

I am getting the error attached in the screenshot.
Despite I set up 'write:false'. Looks like it's ignoring it, then I get the error:
=> Searching for .hcl files
Error, -check and -write=true cannot be used together

Hello, thank you for using my action!
I'll have a look at it.
Maybe already I'll replace using those scripts with simply using Terragrunt, as it finally can format all .hcl and .tf files.

I've released v0.3.4 which should resolve the problem.

I also checked terragrunt hclfmt, but it just fixes the files without any options that terraform fmt has. But it's much faster, so maybe I'll consider it in the future as an option if somebody has a lot of files.

Thanks @ChristophShyper for your prompt response and action. I have a question:
I need it fails in case it find an error in the linting process. I thought using check input it fails but not.
Do you know how can achieve that goal?
See screenshot

Hello @jevalenciap .
You're right I recently removed set -e from the dependent script it stopped failing on the first type of files.
But now I fixed it in origin and released v0.3.6 version where it's fixed, as I just tested.

Thank you so much! It's working fine now

Hello @ChristophShyper,
Sorry, but now the github action is always failing despite there are not linting errors:

I think it needs to fail once a linting error is found but not when there are not errors found.
May you please apply a fix?
Thanks in advance!

hi @jevalenciap I've tested it on all 4 possible scenarios and all work well for me.
Are you sure all your files are formatted properly?

Thanks @ChristophShyper, I don't know why it was failing but after removing a couple of files that were not longer needed, it's working fine.
Apologies for the inconveniences.

@jevalenciap No need to apologize. Thanks to you, I noticed an error in the script being used here.


chore: Rename files relating to what were "instances"

Follow-up to https://github.com/AztecProtocol/aztec-packages/pull/8362 where I rename files according to the new class names therein. Instances had been with sumcheck, but they don't really belong there, so I move them to ultra_honk. I also rename the recursive versions of thins and the instance inspector in place.

Benchmark results
No metrics with a significant change found.

Detailed results
All benchmarks are run on txs on the Benchmarking contract on the repository. Each tx consists of a batch call  to create_note and increment_balance, which guarantees that each tx has a private call, a nested private call, a public call, and a nested public call, as well as an emitted private note, an unencrypted log, and public storage read and write.
This benchmark source data is available in JSON format on S3 here.
Proof generation
Each column represents the number of threads used in proof generation.

Metric
1 threads
4 threads
16 threads
32 threads
64 threads

proof_construction_time_sha256_ms
5,769
1,563
718 (+1%)
758 (+2%)
777 (+1%)

proof_construction_time_sha256_30_ms
11,602 (+1%)
3,089
1,389
1,441 (+1%)
1,459 (-1%)

proof_construction_time_sha256_100_ms
44,122
11,850
5,454
5,421
5,732 (-2%)

proof_construction_time_poseidon_hash_ms
78.0
34.0
34.0
59.0 (+2%)
87.0 (-1%)

proof_construction_time_poseidon_hash_30_ms
1,532
423
204
222 (-2%)
269

proof_construction_time_poseidon_hash_100_ms
5,667
1,513
677
737 (+1%)
748

L2 block published to L1
Each column represents the number of txs on an L2 block published to L1.

Metric
4 txs
8 txs
16 txs

l1_rollup_calldata_size_in_bytes
4,356
7,876
14,884

l1_rollup_calldata_gas
50,256
92,872
178,180

l1_rollup_execution_gas
843,097
1,576,609
3,361,973

l2_block_processing_time_in_ms
249 (-3%)
459 (+3%)
817

l2_block_building_time_in_ms
11,244
22,214 (+1%)
43,874

l2_block_rollup_simulation_time_in_ms
11,244
22,214 (+1%)
43,874

l2_block_public_tx_process_time_in_ms
9,592
20,534 (+1%)
42,178

L2 chain processing
Each column represents the number of blocks on the L2 chain where each block has 8 txs.

Metric
3 blocks
5 blocks

node_history_sync_time_in_ms
3,006 (+1%)
3,993 (+2%)

node_database_size_in_bytes
12,648,528
16,715,856

pxe_database_size_in_bytes
16,254
26,813

Circuits stats
Stats on running time and I/O sizes collected for every kernel circuit run across all benchmarks.

Circuit
simulation_time_in_ms
witness_generation_time_in_ms
input_size_in_bytes
output_size_in_bytes
proving_time_in_ms

private-kernel-init
74.8 (+1%)
380
21,391
44,898
N/A

private-kernel-inner
159 (-1%)
676
72,372
45,020
N/A

private-kernel-reset-tiny
301 (+2%)
689 (-1%)
65,097
44,886
N/A

private-kernel-tail
160 (+1%)
131 (-2%)
50,608
52,921
N/A

base-parity
5.52
N/A
160
96.0
N/A

root-parity
35.1
N/A
73,948
96.0
N/A

base-rollup
3,120
N/A
189,193
664
N/A

block-root-rollup
41.0 (-1%)
N/A
58,205
2,448
N/A

public-kernel-setup
82.4 (+1%)
N/A
105,085
71,222
N/A

public-kernel-app-logic
94.5 (+1%)
N/A
104,899
71,222
N/A

public-kernel-tail
844 (+1%)
N/A
390,582
16,414
N/A

private-kernel-reset-small
294
N/A
66,341
45,629
N/A

private-kernel-tail-to-public
761 (-1%)
589 (-1%)
453,751
1,825
N/A

public-kernel-teardown
80.8
N/A
105,349
71,222
N/A

merge-rollup
19.9 (-1%)
N/A
38,174
664
N/A

undefined
N/A
N/A
N/A
N/A
82,844 (-1%)

Stats on running time collected for app circuits

Function
input_size_in_bytes
output_size_in_bytes
witness_generation_time_in_ms

ContractClassRegisterer:register
1,344
11,731
343

ContractInstanceDeployer:deploy
1,408
11,731
18.0 (-1%)

MultiCallEntrypoint:entrypoint
1,920
11,731
401

FeeJuice:deploy
1,376
11,731
384 (-1%)

SchnorrAccount:constructor
1,312
11,731
62.5

SchnorrAccount:entrypoint
2,336
11,731
370

FeeJuice:claim
1,344
11,731
34.5 (-4%)

Token:privately_mint_private_note
1,280
11,731
72.6 (-5%)

FPC:fee_entrypoint_public
1,344
11,731
24.4 (+1%)

Token:transfer
1,312
11,731
211 (-1%)

Benchmarking:create_note
1,344
11,731
75.2 (+1%)

SchnorrAccount:verify_private_authwit
1,280
11,731
26.5

Token:unshield
1,376
11,731
500

FPC:fee_entrypoint_private
1,376
11,731
661 (-5%)

AVM Simulation
Time to simulate various public functions in the AVM.

Function
time_ms
bytecode_size_in_bytes

FeeJuice:_increase_public_balance
45.7 (+1%)
2,431

FeeJuice:set_portal
12.9
2,930

Token:constructor
81.8 (+10%)
15,813

FPC:constructor
47.5 (-4%)
9,720

FeeJuice:check_balance
36.5
2,357

Token:mint_public
48.3 (-1%)
5,011

Token:assert_minter_and_mint
470
3,155

AuthRegistry:set_authorized
37.3 (-20%)
924

FPC:prepare_fee
70.5 (-7%)
4,245

Token:transfer_public
17.5 (-17%)
10,812

FPC:pay_refund
42.6 (-2%)
5,195

Benchmarking:increment_balance
1,208 (+1%)
3,496

Token:_increase_public_balance
37.1 (-7%)
2,408

FPC:pay_refund_with_shielded_rebate
54.0 (+7%)
5,195

Public DB Access
Time to access various public DBs.

Function
time_ms

get-nullifier-index
0.149 (-3%)

Tree insertion stats
The duration to insert a fixed batch of leaves into each tree type.

Metric
1 leaves
16 leaves
64 leaves
128 leaves
256 leaves
512 leaves
1024 leaves

batch_insert_into_append_only_tree_16_depth_ms
2.15 (-1%)
3.97 (+3%)
N/A
N/A
N/A
N/A
N/A

batch_insert_into_append_only_tree_16_depth_hash_count
16.8
31.7
N/A
N/A
N/A
N/A
N/A

batch_insert_into_append_only_tree_16_depth_hash_ms
0.111 (-1%)
0.111 (+3%)
N/A
N/A
N/A
N/A
N/A

batch_insert_into_append_only_tree_32_depth_ms
N/A
N/A
11.1
17.8
30.7 (-1%)
60.2 (+2%)
112 (-2%)

batch_insert_into_append_only_tree_32_depth_hash_count
N/A
N/A
95.9
159
287
543
1,055

batch_insert_into_append_only_tree_32_depth_hash_ms
N/A
N/A
0.106 (+1%)
0.103
0.100 (-1%)
0.103 (+2%)
0.100 (-2%)

batch_insert_into_indexed_tree_20_depth_ms
N/A
N/A
14.3
25.6 (-1%)
43.5 (-2%)
83.9 (+2%)
159 (-4%)

batch_insert_into_indexed_tree_20_depth_hash_count
N/A
N/A
108
207
355
691
1,363

batch_insert_into_indexed_tree_20_depth_hash_ms
N/A
N/A
0.110
0.103 (-1%)
0.105 (-2%)
0.104 (+2%)
0.101 (-4%)

batch_insert_into_indexed_tree_40_depth_ms
N/A
N/A
16.7 (+1%)
N/A
N/A
N/A
N/A

batch_insert_into_indexed_tree_40_depth_hash_count
N/A
N/A
132
N/A
N/A
N/A
N/A

batch_insert_into_indexed_tree_40_depth_hash_ms
N/A
N/A
0.107 (+1%)
N/A
N/A
N/A
N/A

Miscellaneous
Transaction sizes based on how many contract classes are registered in the tx.

Metric
0 registered classes
1 registered classes

tx_size_in_bytes
64,838
668,997

Transaction size based on fee payment method
| Metric |  |
| - |  |


188 using prefix to separate latent random variables used in different contexts

This PR fixes the problem mentioned in #188 : That is that different components of a full epi model could both call a latent model leading to naming collision.
The solution is that all latent processes generated to be used in an epi model (for example, $\log R_t$) get sampled with naming prefix form latent.varname, whereas latent processes sampled as part of an overall observation model get prefix form obs.varname. This is unit tested with a time varying log-ascertainment scale here.
For me, the only open question is whether we want to custom generate prefixes, leave the version in this PR or plan a future issue about custom prefixes (e.g. as part of model struct).
Closes #188

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 92.11%. Comparing base (12ba5c1) to head (5aa10e5).

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #190   +/-   ##
=======================================
  Coverage   92.11%   92.11%
=======================================
  Files          38       38
  Lines         368      368
=======================================
  Hits          339      339
  Misses         29       29

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Something to flag is we also have this problem when we have stacked ascertainment models potentially (if they share a latent model for example) or multiple observations processes.

Potentially, changing every @submodel to have prefix=true could work here... but I'm not sure how readable the varnames would be.

(merge was automated but agree it was good to merge to fix the issue and then we can iterate in another feature for something more magical with a bit less urgency)


Task selector

Created task scheduler with 3 example tasks and example task request client

Move interfaces to interfaces directory that is immediate child of src
Rename id in TaskRequest.srv to task_id
Add pubsub communication from scheduler to GUI

Add a launch file to start the scheduler & some more documentation


[BUG]

Is there an existing issue for this?

[X] I have searched the existing issues

Current Behavior
Whenever I do anything in the app it checks EVERY mod again.  Never turned it on, actually.  Yet it refreshes on every action.  If I copy a mod to another folder, move a mod, etc...
Expected Behavior
Check ALL the mods once and then never again unless refresh all is chosen.  Only check mods in the folder that a mod was dropped into. But even then, why verify the mod when it's been verified and checked? Why not just update the file system?
Steps To Reproduce
Just copy any mod from one folder to another.
Environment
- Mod Assistant Version: 2.2.3
- OS: Windows 11 Pro (newest version always)
- Virtualized: NO

Anything else?
Since I have moved my 40 favorites to every map folder I am well over 1400 mods now.  And I have not even put NON favorite mods in the map folders yet.  As I have averaged what I use per map I am going to be over 13,000 mods that it will check EVERY time I do anything in the app.  For example, every time I copy a mod from the ALL MODS folder to a map folder it will scan NOT just the map folder but every mod in every folder.  Just before posting this I was waiting 3 minutes for it to complete a scan and I still have to move 23 map files into their own folders. That's going to be over an hour of waiting just because I need to copy one file into each map folder. UGH!
I have cleaned the cache and Cleared the cache to see if that would fix it. It did not.

Did you try it with the sync mode turned on?  Mod Assitant looks at the file name, location, size,  and last modified time to decide if the mod is "new" or not - any drive sync program can mess with the last modified time, so the "one drive mode" tells mod assistant to use the creation time of the file instead - by doing this however, it is possible (although unlikely) that an update will be missed.

I will try it with that...but as I read the instructions, like you stated, it will most likely MISS updates with this turned on.  Sort of a bummer to lose that great feature.
On a side note, I do very much wish the decision to NOT use Hard LInks instead of direct copying will be re-looked at.  At least the option so users can decide to use them or not.  I am currently wasting literally 240+ gigs of space because of all the duplicate files.  UGH!  :)
Greg


Unable to load css after deploying to gh-pages

repo: https://github.com/sharath29/portfolio
I have made few changes on top of your repo but after i ran the pre deploy and the scripts to deploy using gh-pages, I am not able to see any css or image being loaded to my web page.
webpage: https://sharath29.github.io/portfolio/

I'm not sure what CSS you are talking about. Can you explain more? As I can see it renders CSS and image.

Hey! I had to manually correct this part to make it work. After the npm run predeploy it generates index.html which has src="/_next/...". I had to change this to src="_next/...". I am not sure if any setting is there to make sure the paths don't start with "/"
You can find more on this issue here:
https://stackoverflow.com/questions/52003005/css-not-working-on-github-pages

@sharath29 I am having the same issue with my page. I removed the slash but it still doesn't work. Is there anything else you did in a addition to removing '/'? Thanks! This is my page btw: http://hdianacardenas.com

@hildidi that was the only change that I had to do, also please make sure you have the .nojekyle file as mentioned in this link:
https://stackoverflow.com/questions/61450307/js-and-css-not-loading-when-hosting-next-application-on-github-pages

Hmm, I think you are trying to publish manually. Well I have added this touch ./out/.nojekyll in my github workflow. I'll look into this.

The headshot path is broken on github action run for forks, has a duplicate '/portfolio' in image path (e.g. src="/portfolio/portfolio/_next/static/media/profile.34397709.png"). I was able to get images loading at least by removing this duplicate manually


chr: remove SonarCloud fallback

@dblankevia Is it still necessary? :)

@ds-ext-dskrzypczak yes, there is some cleanup remaining. Please review the updated PR.


Implement google analytics

Make sure to track bfcache restores as new pageviews.

react-ga4
flowchart TD

A[Do we need more than the basic gtag.js?]
B[gtag.js supports automatic collection of page views]
C[gtag.js measures page_views on browser history change if enabled on GA dashboard]
D[What do packages like react-ga4 provide?]
E[injects gtag.js script]
F[This is trivial to do by appending script tag to document]
G[Provides methods send page views, custom events]
H["ReactGA internally calls gtag() in its methods. These methods just provide a slightly more convenient API for certain operations"]
I[Using gtag.js API directly seems simple, if we need anything more sophisticated than what is collected automatically]

A --> B
B --> C
C --> D
D --> E
E --> F
D --> G
G --> H
H --> I

click B "https://support.google.com/analytics/answer/9234069?sjid=13764265460356742502-NC"
click C "https://developers.google.com/analytics/devguides/collection/ga4/single-page-applications?implementation=browser-history"
click I "https://developers.google.com/tag-platform/gtagjs/reference"


scala.deprecated annotation is not available in field annotations

@adamw Turns out that @deprecated annotation is on the case field and not on the constructor argument (like every other field annotation). I didn't remove the println yet, since the annotation has a weird @random_uuid suffix. I wasn't sure whether to leave it or to try adding a hack to strip it. WDYT?

I think the suffix comes from the fact that @deprecated is a class, not a case class, so there's no toString - I don't think we can do anything about that.
I wonder though, if it's possible for the same annotation to be both on the field, and on the constructor argument? Maybe we should perform some deduplication? But then, there's no test which would reproduce such scenario, so probably not, merging :)


docs: Fix "time duration" link

Fixes the "time duration" link found in the following page: https://prometheus.io/docs/prometheus/latest/querying/basics/#range-vector-selectors

Thanks!

Thanks for reviewing this! 🙏


request: return ListenableFutures in PubSub async methods rather than bare Futures

The com.google.cloud.pubsub.PubSub interface defines async operations which return java.util.concurrent.Future values.
For anyone who uses the ListenableFuture concept from Guava (or CompletableFutures in Java 8), these Futures have to be adapted into ListenableFutures by spawning a thread that blocks on the Future.get() return.
It would be much more convenient if the Futures returned from PubSub were actually ListenableFuture instances (even if the interface did not define the return value to be ListenableFuture) - then it would not be necessary for users to block one thread per future they wish to adapt into a ListenableFuture.
For instance, internally PubSubImpl.pullAsync(final String subscription, int maxMessages) is implemented by using the listenable-like com.google.cloud.pubsub.spi.PubSubRpc.PullFuture and attaching callbacks to it, but the returned Future is an anonymous subclass of Future created by Futures.lazyTransform(..).
Is it a conscious decision to avoid using ListenableFutures in the interfaces defined in gcloud-java-pubsub and other clients?
It seems like the com.google.cloud.pubsub.spi layer has access to ListenableFuture instances via the generated grpc client code (like subscriberApi.acknowledgeCallable().futureCall(request)) but then these are discarded in the PubSubImpl layer.

Is it a conscious decision to avoid using ListenableFutures in the interfaces defined in gcloud-java-pubsub and other clients?

Yes, it's a conscious decision. We don't want to expose Guava-related classes in our interfaces and we still want to support Java 7 (for GAE and Android, mainly).
Could MessageConsumer pullAsync(String subscription, MessageProcessor callback, PullOption... options) help? If not, what is it missing that you need (I would be happy to add more features)?

Thanks @mziccard for the fast reply.
MessageConsumer is great - it perfectly fits my needs for consuming messages. It is awesome to not have to implement something that pulls messages in a loop and processes them in our own code.
The (minor) pain point comes more in code that deals with methods like Future<String> publishAsync(String topic, Message msg), when I would like to be able to transform the returned future to some other type or combine it with another future. We are heavy users of ListenableFuture and find it natural to transform and otherwise chain ListenableFutures together, so it is a small annoyance to have to use JdkFutureAdapters.listenInPoolThread and create a thread that blocks on Future.get() to be able to adapt this response to a ListenableFuture.
I understand not wanting to return Guava types in the PubSub interface, but wouldn't it be possible to return ListenableFuture implementations (SettableFuture, etc) in the PubSub implementation? This way adapters like JdkFutureAdapters would avoid having to spawn a blocked thread since the returned future would pass a if (future instanceof ListenableFuture) ... check.

I understand not wanting to return Guava types in the PubSub interface, but wouldn't it be possible to return ListenableFuture implementations (SettableFuture, etc) in the PubSub implementation?

I will look into this, I'll keep you posted!


[Idea] Use the Minecraft DataFixerUpper to update nbt to components

At the moment, both OptiFine and CITResewn are appear to hard code the component updating, specifically for nbt.display.Name -> components.minecraft\:custom_name and nbt.display.Lore -> components.minecraft\:lore.
I'm not sure how feasible this is, but since Minecraft has to have the ability to update NBT to components included in the DataFixerUpper to update older worlds, perhaps it's possible to hook into that, and request any .properties files that have NBT properties but no component properties to update those NBT paths?
This would allow backwards compatibility for all resource packs below 1.20.5.

As cool as that would be, DFU only works on actual data(the properties dont have item data, only checks for item data) and so it is not really repurposable like that.


Outer leaves / fur blocks make coral fan sounds instead of leaf sounds

Tenanea Outer Leaves, Glowing Pillar Leaves, Mossy Glowshroom Fur and Blue Vine Fur are all either leaf blocks or similar (I do not know what 'fur' means for the fur blocks in this case), but they make coral fan sounds when placed or broken instead of leaf block sounds. This is especially noticeable if one sets out to harvest tenanea leaves for saplings, and is constantly hearing the two differing breaking sounds while doing so.

I will replace sounds to more "plant like"

Added in 0.9.2


Fix fabric8 client dependency (related to issue 900)

This PR should go after this one and then changes will make a lot more sense. I did not want to muddy the waters in that PR, that fixes an actual problem... When that one is merged, I'll comment on this one more if needed.
But basically the idea is the same as there, we track kubernetes-client dependency in our spring-cloud-kubernetes-dependencies, while it is provided by bom. And we do that for <scope>test</scope> and <type>test-jar</type>, which makes it even more compelling to remove it from there.

Seems like some of these changes duplicate whats in https://github.com/spring-cloud/spring-cloud-kubernetes/pull/904/

right, I branched from that one - because otherwise the build would have failed. That is why I said that if we agree on that one and merge it, and then look at this one, it will make more sense (the overlapping changes would not be present)

@ryanjbaxter it should make a lot more sense now


Update pylint version required from 2.6.0 to 2.7.0

Superseeds https://github.com/dmwm/WMCore/pull/10892
Status
ready
Description
This new version of pylint is required to fix two security vulnerabilities, as listed by this automatic Snyk PR:
https://github.com/dmwm/WMCore/pull/10892
I am not adding the urllib3 dependency because none of the WMCore services has a direct dependency on that library, besides, we are already using a version that contains that fix (1.26.6).
Is it backward compatible (if not, which system it affects?)
YES
Related PRs
None
External dependencies / deployment changes
cmsdist spec: https://github.com/cms-sw/cmsdist/pull/7494

Failures are unrelated to this change. Still, they have to be investigated in the coming days.

Spec has been merged in the cmsdist repository.


New Connector Request: Ethereum Classic Network, ETCswap V2

Description
1.) Chain Connector: Ethereum Classic, Chain 61. https://chainlist.org/chain/61. Add Chain connector guide
2.) AMM-V2 connector: https://v2.etcswap.org A vanilla fork of Uniswap V2. Add V2 connector guide
The primary goal is connect the ETC/USC markets on ETCswap V2 to the ETC/USD CEX markets via Hummingbot XEMM strategy. This is the foundational DeFi pair for Ethereum Classic.
The secondary goal is to have clear guides that allow novice developers in the Ethereum Classic development community open resources for one of the few decentralized networks in blockchain. This is a perfect open source community that deserves Hummingbot support.
Contact<EMAIL_ADDRESS>Protocol Contract List: https://github.com/etcswap
Bounty

Sponsor: EthereumClassic.com
Bounty amount: 3750 USDC (includes 20% Handling Fee to Hummingbot Foundation)
Developer portion: 3000 USDC

Sounds interesting, after I completed my current V2 docs bounty I would like to take it.

note for the developers. if an RPC endpoint is needed: https://etc.rivet.link is the funded public endpoint. This should be the most stable endpoint for ETC.

I can work on this and get it in asap

I will like to be assigned this bounty. i can get it done as soon as possible

@nikspz I can create a pr for this over this weekend as I have snippets for this v2 as a result of my work on the v3.

I can work on this @nikspz if still open.

NCP Proposal is submitted:
https://snapshot.org/#/hbot-ncp.eth/proposal/0x5face44f80c3c0d3d290da1a6973ec0c030a483023a4647ff91e4339af085770

Thanks for your participation, the bounty has been sent to @vic-en. The connector has been deployed on development branch and hummingbot version 2.1.0


oneAPI Base Toolkit image no longer has cmake installed

Firstly, thanks for providing the images, they have been useful :)
I am not sure if this was done intentionally or not, if it was done intentionally feel free to close the issue, it is just that most oneMKL examples are built with cmake. However, the base toolkit image no longer has cmake installed, it was removed in the last commit

Latest images include cmake


Programmatically zoom to Area

Currently trying to figure out a way to Zoom to an Area programmatically and running into some issues. It seems that Unovis is dealing with all the different layers, TopoJSON / GeoJSON and d3 in the background, but I cannot figure out a way to do this.
Is there a chance we can add this feature in the TopoMap component?
An example would be to click on a country/area and zoom + center the map to this. Is this currently possible?

@Rkallenkoot Zooming to a specific area is not yet supported by TopoJSONMap, although I agree that it would be a nice feature to have. At the same time, I'm not sure how useful it will be, since TopoJSON maps are schematic and typically don't provide detailed views. Can you tell us more about the problem you're trying to solve with this feature?

Thanks for the response!
My current use-case is to display a statistic per Area. Some countries / islands (example: Caribbean islands) are so small that you can't see them unless zoomed in quite a bit.
I'm displaying the exact stats in a table next to the Map, the user can click on the "Country" and I'd like to center the map around this Country with some additional Zoom. It would also be nice to be able to programmatically reset the center and zoom.


Research Security for App

Bcrypt is an ecryption method like sha1 or md5.
This is a side note of gcp security when we use app engine: https://cloud.google.com/appengine/docs/standard/nodejs/application-security

4 steps for login:
register, login, getting user, logging out
Register:
salting and hashing user's new password
adding user details to database like email, username, etc
Login:
search if username exists
if no user return 404
if user yes but password compare no then 400
if all good then store details as jwt
use cookie to authenticate
Getting User:
searching for jwt inside cookies
check if claim equal and if not then 401
if works grabs user details from database
Logging Out:
change max age for cookie for 0 creates instant expiration

Passwords are not stored in the database, but since the hash is stored that will be checked against whatever the user puts in to try to authenticate. The answer to how they match is that the SALT IS STORED WITH THE HASH.


CFFL

CFFL

Linter check found the following problems:
The following problems have been found:
ERROR: /tmp/abs_f1ezrsh16i/clone/recipe/meta.yaml:18: pip_install_args: pip install should be run with --no-deps.
ERROR: /tmp/abs_f1ezrsh16i/clone/recipe/meta.yaml:23: missing_wheel: For pypi packages, wheel should be present in the host section
ERROR: /tmp/abs_f1ezrsh16i/clone/recipe/meta.yaml:31: missing_pip_check: For pypi packages, pip check should be present in the test commands
ERROR: /tmp/abs_f1ezrsh16i/clone/recipe/meta.yaml:37: missing_license_family: The recipe is missing the about/license_family key.
WARNING: /tmp/abs_f1ezrsh16i/clone/recipe/meta.yaml:37: missing_description: The recipe is missing a description
ERROR: /tmp/abs_f1ezrsh16i/clone/recipe/meta.yaml:37: http_url: http://github.com/six8/pytailer is not https
ERROR: /tmp/abs_f1ezrsh16i/clone/recipe/meta.yaml:37: missing_dev_url: The recipe is missing a dev_url
ERROR: /tmp/abs_f1ezrsh16i/clone/recipe/meta.yaml:37: missing_documentation: The recipe is missing a doc_url or doc_source_url
Errors were found


[logs] truncate heavy log file

From this issue https://github.com/Inria-Asclepios/medInria-public/issues/166
Will be rebased with https://github.com/Inria-Asclepios/medInria-public/pull/178
Add a medLogger::truncateLogFileIfHeavy() function in medLogger::medLogger() which allow to check the size of the log file at launching of the app, and truncate it if needed (old lines are discarded).
:m:

This PR has been rebased

The log file reduction can be quite slow if we already have a big log file (mine was 40Mo) since the algo removes 4Mo at every launch (if size>5Mo).
(I don't know how often manips close the app)
Maybe we could erase 80% of the log file instead, so that it takes into account this scenario.

I don't know, nobody would have 40Mo file, since the app tests the size at launch ^^

H and the manips have logs enabled for few weeks now, so they may have a big log file.

I agree @LoicCadour, it's just that this won't be the normal behavior :)

OK, but does it mean we should ignore this scenario which might happen to our only users ?
The 80% reduction fits both your normal scenario and my apocalyptic scenario :)

I think it should be done the other way around: you define a minimum size and if it is over that size you truncate it by how much it exceeds that size.

I am ok with that as well

Let's summarize:

Math: (file > 5Mo) -> (file loose 4Mo)
Loic: (file > 5Mo) -> (file loose -80%)
Flo: (file > 5Mo) -> (file cut to 5Mo)

The first and second method are quite similar in a routine. The third one in routine would means that the shrinking of the file is done at each launch of the application as soon as we reach the limit.
After thinking about these methods and fuse their advantages, i've got an other one with a minimum and maximum limit:

(file > 5Mo) -> (file cut to 1Mo)

It would be ok for you?

ok

Why did you close the PR?

Updated @fcollot :)

@fcollot updated!


Add CODEOWNERS file

What does this PR do?
This pull request adds a CODEOWNERS file to the repository
Checklist

[ ] Pull request contains a clear definition of changes
[ ] Tests (either unit, integration, or acceptance) written and passing
[ ] Relevant documentation produced and/or updated

Pull Request Test Coverage Report for Build 69

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 100.0%

Totals

Change from base Build 66:
0.0%

Covered Lines:
152

Relevant Lines:
152

💛  - Coveralls


feat(chart2proto): chart to proto transformations for helm grpc client

Preliminary chart to protobuf transformations for helm grpc client. This code is responsible for constructing the in memory protobuf representation of charts.
Also included is the updated "helm install" command and grpc wrapper APIs that leverage the leverage the above.

needs a gofmt

I'm kinda sorta blocked waiting for this PR, since I need to do some server-side work that takes advantage of the improvements Brian makes here.

walk this way


Documentation and Cleanup

Hey @aaronparker
I just removed the DEV prefix and updated the documentation. If this PR is closed I think I can remove my fork.

I've merged and have made some changes in the development branch, but will take a look at these updates to see the best way forward.


how to test on new images

I have a person's image wearing a top. This is my reference image. Also, I have the segmentation map of the reference image. I take one more image which
is my target image. And I have created 18 points key points from the open pose of the target image. Now I have an in-shop clothe(top), whose binary mask I have calculated using OpenCV. I need to wrap this clothe on the target image(whose key points I have generated). Haven't read the full paper yet

What all things I need to generate to test on my own data?
Do I need to resize all the images to 256x192 before feeding it to the network
in the demo_txt file I am using the below format
reference image.jpg    target_pose_keypoints.json     clothe_item.jpg

and I am getting below error while running demo.sh. I ran the demo.sh with the example settings and it generated images. So the code is working. The only thing is I am not able to test for my own images.
Thanks in advance
Traceback (most recent call last):
File "demo.py", line 185, in
forward(opt, paths, 4, opt.forward_save_path)
File "demo.py", line 109, in forward
warped_cloth = warped_image(gmm, result)
File "/media/trinanjan/BACKUP/Trinanjan/current_project/virtual_try_on/Down-to-the-Last-Detail-Virtual-Try-on-with-Detail-Carving/utils/warp_image.py", line 28, in warped_image
agnostic = torch.cat([source_parse_shape, im_h, target_pose_embedding], 1)
RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 1. Got 256 and 1100 in dimension 2 at /pytorch/aten/src/THC/generic/THCTensorMath.cu:87

This issue is solved. I had to resize the images(ref,target,product) to (h=256,w=192) and calculate.The binary mask, segmentation map, pose key points should be created for the resized images.

I have resized the images but while loading the dataset i am getting the below error.  Image at entry 0 is taken from official dataset. image at entry 4 is my own image. so facing an issue with my images they are not same shape datasets img shape.
Traceback (most recent call last):
File "demo.py", line 192, in
forward(opt, paths, 4, forward_save_path)
File "demo.py", line 108, in forward
for i, result in enumerate(val_dataloader):
File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 345, in next
data = self._next_data()
File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 856, in _next_data
return self._process_data(data)
File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 881, in _process_data
data.reraise()
File "/usr/local/lib/python3.6/dist-packages/torch/_utils.py", line 395, in reraise
raise self.exc_type(msg)
RuntimeError: Caught RuntimeError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/worker.py", line 178, in _worker_loop
data = fetcher.fetch(index)
File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch
return self.collate_fn(data)
File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/collate.py", line 74, in default_collate
return {key: default_collate([d[key] for d in batch]) for key in elem}
File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/collate.py", line 74, in
return {key: default_collate([d[key] for d in batch]) for key in elem}
File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/collate.py", line 55, in default_collate
return torch.stack(batch, 0, out=out)
RuntimeError: stack expects each tensor to be equal size, but got [1, 256, 192] at entry 0 and [1, 256, 192, 3] at entry 4

Looks like your data loader has an issue as it is failing in the getitem. try creating a Jupyter notebook only for the dataloder to see if you can confirm the below shapes(visualization also might help)
Not sure if this will help or not if you just run the dataloader and try to check the data coming it will look something like this for batch of 1
This is train dataloder
source_parse (20, 256, 192)
target_parse (20, 256, 192)
source_parse_vis torch.Size([3, 256, 192])
target_parse_vis torch.Size([3, 256, 192])
source_pose_embedding (18, 256, 192)
target_pose_embedding (18, 256, 192)
target_pose_loc 18
source_image torch.Size([3, 256, 192])
target_image torch.Size([3, 256, 192])
cloth_image torch.Size([3, 256, 192])
cloth_parse torch.Size([1, 256, 192])
source_parse_shape torch.Size([1, 256, 192])
im_h torch.Size([3, 256, 192])
im_c torch.Size([3, 256, 192])
source_image_name 5
target_image_name 5
cloth_image_name 5
warped_cloth_image torch.Size([3, 256, 192])
warped_cloth_name 5
warped_cloth_path 25
source_img_path 27
target_img_path 27
target_pose_path 45
target_parse_path 31
source_parse_vis_path 35
target_parse_vis_path 35
target_pose_img (256, 192, 3)
warped_cloth_parse torch.Size([1, 256, 192])
target_parse_cloth (256, 192)

@trinanjan12 : Thanks for the quick reply.

Clothes mask are binary images (0/1).something like this I think

prod_images = glob.glob(resize_img_path + '/cloth_image/{}*.png'.format('product'))
for index,img_path in tqdm.tqdm(enumerate(prod_images), total = len(prod_images)):
image = plt.imread(img_path)
image_copy = image.copy()
white_pixels_mask = np.all(image == [255, 255, 255], axis=-1)
non_white_pixels_mask = np.any(image != [255, 255, 255], axis=-1)
image_copy[white_pixels_mask] = [0,0,0]
image_copy[non_white_pixels_mask] = [1,1,1]
cv2.imwrite(resize_img_path +'/cloth_mask/product{}_mask.png'.format(index+1), non_white_pixels_mask*255)

Thank you so much. this is so simple. I have used grabcut and many other things to get the binary mask and still have issue with the same. but your is working fine but the mask is not so clear

I haven't found any models yet which works only on product images. grabcut/segmentation methods are not working well on the product images. I haven't found any deep learning model that was trained only with product images for segmentation. Let me know if you have a luck with that. My function is a brute force way of doing it if the background is white.

Yes. Even i am facing same issue. Even My code is also not working for production images.
I will let you know if i found something useful.

@j-sieger @trinanjan12 Can anyone tell what is the issue here. I have resized all the images to (256, 192). Currently I am thinking there are two possible regions of error: Openpose json file, which used BODY25 model with face and hands model, or there is some error internal error.
`/content/drive/My Drive/Eroom-models-working/Down-to-the-Last-Detail-Virtual-Try-on-with-Detail-Carving
Running forward

CUDA_VISIBLE_DEVICES=0 python demo.py --batch_size_v 80 --num_workers 4 --forward_save_path demo/forward
Namespace(G_GAN=1, G_VGG=1, G_nn=1, batch_size_t=128, batch_size_v=80, beta1=0.5, dataroot=False, dataset='MPV', dataset_mode='regular', decay_iters=10, epoch=200, face_L1=10, face_gan=3, face_img_L1=1, face_residual=False, face_vgg=1, fine_height=256, fine_width=192, forward='normal', forward_save_path='demo/forward', gan_mode='lsgan', gpu_ids=[0, 1, 2, 3], grid_size=5, init_gain=0.02, init_type='normal', input_nc_D_app=6, input_nc_D_face=6, input_nc_D_parsing=56, input_nc_G_app=26, input_nc_G_face=6, input_nc_G_parsing=36, isdemo=False, isval=False, joint=False, joint_G_parsing=1, joint_all=False, joint_parse_loss=False, lambda_L1=1, lr=0.0002, mask=1, mask_tvloss=False, momentum=0.9, n_layers_D=3, ndf=64, netD_app='resnet_blocks', netD_face='resnet_blocks', netD_parsing='basic', netG_app='treeresnet', netG_face='treeresnet', netG_parsing='unet_256', ngf=64, no_dropout=False, norm='instance', num_workers=4, output_nc_app=4, output_nc_face=3, output_nc_parsing=20, pool_size=100, print_freq=10, resume_D_app='', resume_D_face='', resume_D_parse='', resume_G_app='pretrained_checkpoint/app.tar', resume_G_face='pretrained_checkpoint/face.tar', resume_G_parse='pretrained_checkpoint/parsing.tar', resume_gmm='pretrained_checkpoint/step_009000.pth', save_epoch_freq=1, save_time=False, size=(256, 192), start_epoch=0, suffix='', train_mode='parsing', use_gmm=False, val_freq=200, warp_cloth=False, weight_decay=0.0001)
initialization method [normal]
initialization method [normal]
initialize network with normal
initialize network with normal
initialize network with normal
====================
====================
====================
====================
==>loaded model
Traceback (most recent call last):
File "demo.py", line 184, in
forward(opt, paths, 4, opt.forward_save_path)
File "demo.py", line 106, in forward
for i, result in enumerate(val_dataloader):
File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 363, in next
data = self._next_data()
File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 989, in _next_data
return self._process_data(data)
File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 1014, in _process_data
data.reraise()
File "/usr/local/lib/python3.6/dist-packages/torch/_utils.py", line 395, in reraise
raise self.exc_type(msg)
RuntimeError: Caught RuntimeError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/worker.py", line 185, in _worker_loop
data = fetcher.fetch(index)
File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch
return self.collate_fn(data)
File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/collate.py", line 74, in default_collate
return {key: default_collate([d[key] for d in batch]) for key in elem}
File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/collate.py", line 74, in
return {key: default_collate([d[key] for d in batch]) for key in elem}
File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/collate.py", line 64, in default_collate
return default_collate([torch.as_tensor(b) for b in batch])
File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/collate.py", line 55, in default_collate
return torch.stack(batch, 0, out=out)
RuntimeError: stack expects each tensor to be equal size, but got [18, 256, 192] at entry 0 and [25, 256, 192] at entry 4`

@bhavesh2799 Haven't worked on it for some time. I am attaching a file that I had used for data preparation. Just running the file may not work. I am attaching for you reference purpose. It has a script to convert 25 keypoints to 18 keypoints.
https://colab.research.google.com/drive/1IM84ldwD36XTEPN90PVenzj2R_Q1lUad

@bhavesh2799 Haven't worked on it for some time. I am attaching a file that I had used for data preparation. Just running the file may not work. I am attaching for you reference purpose. It has a script to convert 25 keypoints to 18 keypoints.
https://colab.research.google.com/drive/1IM84ldwD36XTEPN90PVenzj2R_Q1lUad

@trinanjan12 This works for me. But came to another error in running demo.sh
/content/drive/My Drive/Eroom-models-working/Down-to-the-Last-Detail-Virtual-Try-on-with-Detail-Carving
Running forward

CUDA_VISIBLE_DEVICES=0 python demo.py --batch_size_v 80 --num_workers 4 --forward_save_path demo/forward
Namespace(G_GAN=1, G_VGG=1, G_nn=1, batch_size_t=128, batch_size_v=80, beta1=0.5, dataroot=False, dataset='MPV', dataset_mode='regular', decay_iters=10, epoch=200, face_L1=10, face_gan=3, face_img_L1=1, face_residual=False, face_vgg=1, fine_height=256, fine_width=192, forward='normal', forward_save_path='demo/forward', gan_mode='lsgan', gpu_ids=[0, 1, 2, 3], grid_size=5, init_gain=0.02, init_type='normal', input_nc_D_app=6, input_nc_D_face=6, input_nc_D_parsing=56, input_nc_G_app=26, input_nc_G_face=6, input_nc_G_parsing=36, isdemo=False, isval=False, joint=False, joint_G_parsing=1, joint_all=False, joint_parse_loss=False, lambda_L1=1, lr=0.0002, mask=1, mask_tvloss=False, momentum=0.9, n_layers_D=3, ndf=64, netD_app='resnet_blocks', netD_face='resnet_blocks', netD_parsing='basic', netG_app='treeresnet', netG_face='treeresnet', netG_parsing='unet_256', ngf=64, no_dropout=False, norm='instance', num_workers=4, output_nc_app=4, output_nc_face=3, output_nc_parsing=20, pool_size=100, print_freq=10, resume_D_app='', resume_D_face='', resume_D_parse='', resume_G_app='pretrained_checkpoint/app.tar', resume_G_face='pretrained_checkpoint/face.tar', resume_G_parse='pretrained_checkpoint/parsing.tar', resume_gmm='pretrained_checkpoint/step_009000.pth', save_epoch_freq=1, save_time=False, size=(256, 192), start_epoch=0, suffix='', train_mode='parsing', use_gmm=False, val_freq=200, warp_cloth=False, weight_decay=0.0001)
initialization method [normal]
initialization method [normal]
initialize network with normal
initialize network with normal
initialize network with normal
====================
====================
====================
====================
==>loaded model
Traceback (most recent call last):
File "demo.py", line 184, in
forward(opt, paths, 4, opt.forward_save_path)
File "demo.py", line 106, in forward
for i, result in enumerate(val_dataloader):
File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 363, in next
data = self._next_data()
File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 989, in _next_data
return self._process_data(data)
File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 1014, in _process_data
data.reraise()
File "/usr/local/lib/python3.6/dist-packages/torch/_utils.py", line 395, in reraise
raise self.exc_type(msg)
TypeError: Caught TypeError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/worker.py", line 185, in _worker_loop
data = fetcher.fetch(index)
File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py", line 44, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/content/drive/My Drive/Eroom-models-working/Down-to-the-Last-Detail-Virtual-Try-on-with-Detail-Carving/data/demo_dataset.py", line 101, in getitem
source_pose = a['people'][0]['pose_keypoints_2d']
TypeError: list indices must be integers or slices, not str

Hard to say. A good idea is to go through line by line the data loader file. This kind of problem happens if you don't prepare the data exactly the way the repo had suggested. For example the keypoint files should have exact Json structure others it will show error, as it is being index with exact string key.

Hard to say. A good idea is to go through line by line the data loader file. This kind of problem happens if you don't prepare the data exactly the way the repo had suggested. For example the keypoint files should have exact Json structure others it will show error, as it is being index with exact string key.

Yeah, I checked it. Earlier I tried converting the openpose json output to coco format through the openpose model, and the format was quite different from desired.  So, now I am using the script mentioned in https://colab.research.google.com/drive/1IM84ldwD36XTEPN90PVenzj2R_Q1lUad by you. But it's showing index error here.
`# convert 25 keypoints to 18 keypoints
import numpy as np
import json
key_18_path = '/content/drive/My Drive/coco-json'
export_path = '/content/drive/My Drive/Eroom-models-working/my-input-source-images/'
for index,file_name in enumerate(key_25_files):
key_25,key_18 = [],[]
x = {"version":1.3,"people":[{"person_id":[-1],"pose_keypoints_2d":[],"face_keypoints_2d":[],"hand_left_keypoints_2d":[],"hand_right_keypoints_2d":[],"pose_keypoints_3d":[],"face_keypoints_3d":[],"hand_left_keypoints_3d":[],"hand_right_keypoints_3d":[]}]}
with open(file_name,'r') as f:
key_25 = np.array(json.load(f))
key_18 = key_25[0:8]
key_18 = np.append(key_18,key_25[9:19])
x_axis = key_18[::3]
y_axis = key_18[1::3]
z_axis = key_18[2::3]

extension = '.jpg'
img_file_name =  export_path +os.path.basename(file_name).split('_keypoint')[0]+extension
img_orig = Image.open(img_file_name)

x_ratio = (192/img_orig.size[0])
y_ratio = (256/img_orig.size[1])

small_x_axis = x_axis * x_ratio
small_y_axis = y_axis * y_ratio
small_key_18 = list(zip(small_x_axis,small_y_axis,z_axis))

small_key_18 = np.ravel(np.array(small_key_18))
x['people'][0]['pose_keypoints_2d'] = small_key_18.tolist()
with open( key_18_path + '/{}'.format(os.path.basename(file_name)),'w') as f:
    json.dump(x,f)`

IndexError                                Traceback (most recent call last)
 in ()
9     with open(file_name,'r') as f:
10         key_25 = np.array(json.load(f))
---> 11     key_18 = key_25[0:8]
12     key_18 = np.append(key_18,key_25[9:19])
13
IndexError: too many indices for array

@bhavesh2799 could be a problem with the openpose keypoints. Again the colab was for your reference. I have worked on it long back, so not sure if both of our scenarios same. Checkout openpose keypoints mapping https://github.com/CMU-Perceptual-Computing-Lab/openpose/blob/master/doc/output.md. Also I don't think you have to use openpose only. I think you can use detectron2 also for keypoints extraction. Detectron2 is much easier.

@trinanjan12 , could you please tell where could be the error. I updated the json to coco format. So I am pretty sure, that there isn't any error wrt json file. All images are of size (256,192). I don't understand why it is showing size as [1,256,192,3].
First 4 examples in demo.txt are the inference samples, and the last is my custom sample.
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>test
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>test
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>test
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>test
10resized.jpg 10resized_keypoints.json s-1resized.jpg test

/content/drive/My Drive/Eroom-models-working/Down-to-the-Last-Detail-Virtual-Try-on-with-Detail-Carving
Running forward
+ CUDA_VISIBLE_DEVICES=0 python demo.py --batch_size_v 80 --num_workers 4 --forward_save_path demo/forward
Namespace(G_GAN=1, G_VGG=1, G_nn=1, batch_size_t=128, batch_size_v=80, beta1=0.5, dataroot=False, dataset='MPV', dataset_mode='regular', decay_iters=10, epoch=200, face_L1=10, face_gan=3, face_img_L1=1, face_residual=False, face_vgg=1, fine_height=256, fine_width=192, forward='normal', forward_save_path='demo/forward', gan_mode='lsgan', gpu_ids=[0, 1, 2, 3], grid_size=5, init_gain=0.02, init_type='normal', input_nc_D_app=6, input_nc_D_face=6, input_nc_D_parsing=56, input_nc_G_app=26, input_nc_G_face=6, input_nc_G_parsing=36, isdemo=False, isval=False, joint=False, joint_G_parsing=1, joint_all=False, joint_parse_loss=False, lambda_L1=1, lr=0.0002, mask=1, mask_tvloss=False, momentum=0.9, n_layers_D=3, ndf=64, netD_app='resnet_blocks', netD_face='resnet_blocks', netD_parsing='basic', netG_app='treeresnet', netG_face='treeresnet', netG_parsing='unet_256', ngf=64, no_dropout=False, norm='instance', num_workers=4, output_nc_app=4, output_nc_face=3, output_nc_parsing=20, pool_size=100, print_freq=10, resume_D_app='', resume_D_face='', resume_D_parse='', resume_G_app='pretrained_checkpoint/app.tar', resume_G_face='pretrained_checkpoint/face.tar', resume_G_parse='pretrained_checkpoint/parsing.tar', resume_gmm='pretrained_checkpoint/step_009000.pth', save_epoch_freq=1, save_time=False, size=(256, 192), start_epoch=0, suffix='', train_mode='parsing', use_gmm=False, val_freq=200, warp_cloth=False, weight_decay=0.0001)
initialization method [normal]
initialization method [normal]
initialize network with normal
initialize network with normal
initialize network with normal
====================
====================
====================
====================
==>loaded model
Traceback (most recent call last):
  File "demo.py", line 184, in <module>
    forward(opt, paths, 4, opt.forward_save_path)
  File "demo.py", line 106, in forward
    for i, result in enumerate(val_dataloader):
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 435, in __next__
    data = self._next_data()
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 1085, in _next_data
    return self._process_data(data)
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 1111, in _process_data
    data.reraise()
  File "/usr/local/lib/python3.6/dist-packages/torch/_utils.py", line 428, in reraise
    raise self.exc_type(msg)
RuntimeError: Caught RuntimeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/worker.py", line 198, in _worker_loop
    data = fetcher.fetch(index)
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch
    return self.collate_fn(data)
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/collate.py", line 73, in default_collate
    return {key: default_collate([d[key] for d in batch]) for key in elem}
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/collate.py", line 73, in <dictcomp>
    return {key: default_collate([d[key] for d in batch]) for key in elem}
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/collate.py", line 55, in default_collate
    return torch.stack(batch, 0, out=out)
RuntimeError: stack expects each tensor to be equal size, but got [1, 256, 192] at entry 0 and [1, 256, 192, 3] at entry 4

I don't understand why it is showing it this why with my custom image.

@bhavesh2799 the error says it is expecting a image of batch_size,h,w and it is getting a image of batch_size,h,w,num_channel. I think the error is because the image expected to be a binary map of the cloth image but instead it is getting a rgb image. Try following the exact procedure that they have mentioned with naming and folder convention as the folder names and the id's of images are indexed directly in the files.

@trinanjan12 I tried doing that. It is returning the same error. Is there a chance of internal code error. I checked the sample dataset images, none of them are binary.

@bhavesh2799 aren't the mask images binary. https://github.com/JDAI-CV/Down-to-the-Last-Detail-Virtual-Try-on-with-Detail-Carving/blob/master/dataset/cloth_mask/1VJ21D02B-Q11%4010.1%3Dcloth_front_mask.png

nope, they have three channels

@trinanjan12

@bhavesh2799 if you check the parse cloth function in demo dataset it is actually filtering out the mask. So if you have set paths correctly it should be taken care of. This could be some other problem.

@trinanjan12 It's really strange. I have checked the dataset multiple times. It doesn't seem to work with new images, but works fine with their own inference images. There maybe some issue with the Dataloader, I don't know actually...
If you have any working colab implementation, or know anyone who has tested successfully on custom dataset, please tell

@bhavesh2799 I had made it work with their existing code. So I don't think it's a problem with the code of this repo. Mostly dataloader. I don't have a working copy of the same code, I had modified a lot of things for my own work, so that may not work for you!
If you need help regarding other virtual tryon repo checkout papers with code.
you could follow these people https://github.com/Yagami360, https://github.com/minar09
Good luck!

@trinanjan12 Finally solved it. It's working now. Problem was with binary masks itself. But it generates really strange output, and why there is a woman's face on all images.

@bhavesh2799 Congrats!
1.you can't use it for bigger resolution, most of the vton system available works for small resolution.
2. The face part blends 2 faces from the source and target image using a face network, check out the paper
3.the better the output of predicted segmentation map the better the gan output
4.Qulality of the output depends on the data trained and the difficulty of poses.

@bhavesh2799 Congrats!
1.you can't use it for bigger resolution, most of the vton system available works for small resolution.
2. The face part blends 2 faces from the source and target image using a face network, check out the paper
3.the better the output of predicted segmentation map the better the gan output
4.Qulality of the output depends on the data trained and the difficulty of poses.

@trinanjan12 Can we somehow disable the generation network, if I am using the same image as source and target, so at least it works optimally  in warping cloth on target image?

@trinanjan12 Can we somehow disable the generation network, if I am using the same image as source and target, so at least it works optimally in warping cloth on target image?

congratulations @bhavesh2799 ,

the strange cloth shape is because of the keypoint modification which results in strange wrapping cloth, so take care of joints/points while making any changes in them.

not sure much about face , but, there are other methods which helps preserve the face  and body , you can checkout https://github.com/switchablenorms/DeepFashion_Try_On (for reference)

could you please share the modified code here or to my mail<EMAIL_ADDRESS>as i am also facing a lot of issues while testing the repo on custom images ( in colab,its tough, man)

Can any of you , both , guide me clear steps on how to test it up on custom images @trinanjan12 @bhavesh2799

Thanks in advance,
satyam.