strawman provenance support, using a new SequencerInput object

-1 -- this is not ready to commit; it's meant to drive discussion.
Following up on #390, here's a sketch of the schema changes needed to add formal provenance support. It does three things:

defines a new SequencerInput object, whose function is to collapse all the 'wet' provenance into a single "here's where all the bits came from" object. (I think it's roughly the same as the 'informatics sample' concept that @lh3 discusses, but I'm not sure.) The name is terrible but explicit -- the idea is that all of the work of experiment design, subject selection, tissue collection, and sample prep results in one blob of DNA that gets run through a sequencer, generating one blob of reads that we want to process as a unit.
adds explicit derived-from pointers to the ReadGroupSet object
adds explicit derived-from pointers to the CallSet object, and does related small cleanups to that object

The resulting formal provenance chain looks like:
SequencerInput ==> raw ReadGroupSet ==> aligned ReadGroupSet ==> CallSet

Those are all potentially one-to-many arrows. For example, while each CallSet comes from exactly one aligned ReadGroupSet and exactly one SequencerInput, there could be many ReadGroupSet and CallSet objects that all come from the same SequencerInput.
Things we'd have to resolve before committing something like this:

what's the right name and description for a SequencerInput object?
think through how users will create and manage SequencerInput objects. Different server implementors could come up with different approaches, but we need to be sure there's at least one practical answer. Options include:

derive the formal hierarchy from the ad hoc information found in BAM and VCF files. The Google implementation attempts to do this today by extracting sample names from files, and storing them in the sampleId fields, but there's no guarantee of completeness or uniqueness in the raw data
add support for importing manifest files of some sort, which explicitly define the set of tissue sources and how they map to the BAM and VCF files
build a console that lets users view/edit the provenance of their data

add support for filtering by provenance (e.g. add optional sourceSequencerInputId and sourceReadGroupSetid fields to the SearchReadGroupSet and SearchCallSet methods). These should be easy to define, and relatively easy to implement, but they do add work for all  server implementors. I think they're necessary for this formal provenance to be worth doing, though.
add methods to manage these new objects (e.g. GetSequencerInput and SearchSequencerInput)
MAYBE LATER: add first-class metadata fields to the SequencerInput object, including ways to join to external registries
MAYBE LATER: add more upstream provenance objects (e.g. Subject, Experiment)

Your SequenceInput is the exact opposite of NamedSample in #383 (I renamed Sample to NamedSample for now). NamedSample is not defined from experimental processes, but yours is.
Come back to the 1000g pilot 1 example. 1000g sequenced NA12878 on Illumina, 454 and SOLiD, but it called the variants from the three types of data jointly. In my model, there is one NamedSample, which is NA12878. There is one VariantSet. There is one or many ReadGroupSets (it doesn't matter either way). There are many ReadGroups with experimental information (e.g. library, run and sequencing technology) being their attributes. Everything works fine -- this is how 1000g models their data. In your model, though, these are definitely three separate SequencerInputs at least (more if you think a SequencerInput is a Run). You can't create one VariantSet for all the reads.
I will make a bold assertion here: any attempts to model the experimental process with primary objects will not succeed. The right way of thinking is to define primary objects conceptually and then assign experimental information to these objects as attributes.

There is something to think about in Heng's bold assertion.
The experimental process will change.  We need to make sure we
can represent the changes.  Also, the experimental process is
best represented as a graph.  Nodes could be polymorphic or
stored a set of attributes, the results are the same.
The important thing is that the structure of the graph not be
fixed and steps can be added or removed.
Heng Li<EMAIL_ADDRESS>writes:

I will make a bold assertion here: any attempts to model the experimental
process with primary objects will not succeed. The right way of thinking is to
define primary objects conceptually and then assign experimental information to
these objects as attributes.

I have sympathy with @lh3's assertion too --- this is all very complicated, and attempting to fully model all the possible messiness will lead to something that is difficult to work with (and, probably too rigid for future changes). I think we should aim for something which is sufficiently flexible to make it possible to record the provenance of everything if we want, but also to make it possible to deliberately discard some of the complexity if we want to.
Suppose we have a record called Sample which maps to the idea of a SequencerInput in this PR. It is a low-level, provenance-preserving object which can be related back to the wet lab. We have another object called a SampleSet, which is the aggregation of a number of Samples. The SampleSet corresponds to NA12878 in 1000G: in reality, it corresponds to a bunch of different biological samples, library preps, etc, but at higher levels of abstraction, we don't care. The rest of the schema then looks like @lh3's PR #383, except we use a SampleSet instead of a NamedSample in the VariantSet and ReadGroup.
The idea of the SampleSet is to provide a deliberate narrow waist, to optionally discard some of the complexity of the wet lab. We can group together a bunch of samples, so we ultimately know where everything comes from, but we're not required to carry all this complexity through our analyses. If we do want to carry the identifiers through the full analysis, we can simple make a SampleSet containing exactly one Sample.
The names here are probably badly chosen, and will likely be confusing. Does the basic idea make sense though? I.e., we make an object that records precise biological provenance, and then group a set of these together as the basis for our analyses.

If we prefer the "Set" based naming, perhaps a better name is "BioSpecimenSet". We are so used to the lousy meaning of "sample" that "SampleSet" would sound like a collection of different individuals.

See also the wiki pages on Biological specimen and Lab specimen.

@jeromekelleher -- thanks for the "narrow waist" metaphor; I shamelessly appropriate it below.
I think there's consensus that the full experimental process is messy and hard to formally model. All of the proposals I've heard collapse that process through some sort of "narrow waist":

our current schema does completely informal provenance, by providing a single under-defined sampleId text field in ReadGroupSet and CallSet objects
#383 and #391 replace that field with a new object that marks the start of the 'dry' provenance chain, and reference it in roughly the same places

The suggested proposals have similar models. The one proposed here is:
SequencerInput ==> raw ReadGroupSet ==> aligned ReadGroupSet ==> CallSet

I think the model proposed in #383 is:
NamedSample ==> {ReadGroupSet, column-of-calls}

And @jeromekelleher suggests:
Sample ==> SampleSet  ==> {ReadGroupSet, column-of-calls}

If that's right, the proposals have a lot in common. Differences I see are:

what do we call the "narrow waist", and how do we document it? (dG: I don't love any of the proposals yet; let's keep trying.)
do we want zero (as today), one (as in #383 and #391), or two (as in @jeromekelleher's suggestion) objects in that narrow waist? (dG: I lean towards one, if we can define it well.)
do we want to model the derived-from relationships between reads and calls, or just say "all of these come from the same object-formerly-known-as-sample"?
a mostly-unrelated discussion on whether to have a formal object to represent a column-of-calls

Once we agree on naming and model, we'll also need to get more specific about how we expect users to create and manage these narrow-waist objects. I have some thoughts there, but let's first get aligned on these basics.
===============
P.S. For easy reference, the header comment for NamedSample is:
A sample uniquely named in a Dataset. The sample name typically appears in a
@RG line in a BAM file and in the sample header line of a VCF file. Unrelated
samples in different Datasets may have the same sample name.

And the header comment for SequencerInput is:
A `SequencerInput` represents prepared tissue intended to be sequenced and analyzed
together, eventually generating one or more `ReadGroupSet` and/or `CallSet` objects.

I agree @dglazer, I think there is a lot of consensus on what we want. I think we'd probably figure this out in an hour if there was a handful of us standing around a whiteboard with some coffee...
One thing I consistently dislike about your characterisation though is the idea of a "SequencerInput generating a CallSet". Surely this should be "is used to derive a VariantSet"? The way its phrased now suggests that CallSets exist independently of VariantSets, and that somehow Variants exist independently of the data. (Which I guess they do, sometimes, but this is a philosphical question.) VariantSets are also ultimately derived from SequencerInputs, ReadGroupSets, and other VariantSets. Limiting the scope to the 'column-of-calls' idea is a severe over-simplification I think.
Perhaps this is another issue though?

What @jeromekelleher said reflects the wrong assumption underlying CallSet from the very beginning. The current schema assume that we generate one callset and then attach them to an existing VariantSet. I have said in #383 that this is not the past and the current practice, and will never be the future practice unless we switch to a binary/unary allele model, which demands more substantial changes.
In #390, @jjfarrell said a word "callset". That is what everyone in this field means by a "callset". A callset is the set of calls we make from a single calling process across one or multiple samples. It is in effect a VCF or a VariantSet in GA4GH. We never use a phrase "variant set" in our field.

The current schema assume that we generate one callset and then attach them to an existing VariantSet. I have said in #383 that this is not the past and the current practice, and will never be the future practice unless we switch to a binary/unary allele model, which demands more substantial changes.

First - @lh3 is absolutely right
However: I know numerous very good scientists -- perhaps a little less familiar with the depths of the VCF data model -- who either ignore or consciously accept the fallacies in merging called VCFs, and do so in a way that might be described as naive by those who have descended into the seventh circle of joint genotyping. The claims used to answer my protests are that the problematic cases are very rare, and/or that downstream analysts only end up looking at biallelic SNVs anyway. (In slight amelioration, they usually have some auxiliary coverage BED, or the VCFs are EMIT_ALL_CONFIDENT_SITES.)
This raises a very interesting question of whether we should take a position as a group on whether this practice is wrong in the current data model, and make it un-representable, or whether to accommodate it as something people do even though sub-optimal.

These are all representable, I believe. No matter how VCFs are generated, VCFs are VCFs and each VariantSet represents a VCF. Even with naive VCF merge, command lines used to generate individual calls are similar and can be embedded in the header of merged VCF and hence VariantSetMetadata. We are all doing this nowadays anyway.

Yes- "unrepresentable" was an exaggeration by me, sorry, I had in mind the generic principle to "make illegal states unrepresentable". My observation (riffing on yours) is that the current schema's factorization of callsets can be viewed as a normalization (in the RDBMS) sense of all these single-sample VCFs, i.e.  reducing repetition. This would be elegant if indeed you could merge a new callset into an existing database in a simple way, only adding new stuff to the database. Some choose to operate in that way even though there are problems. Currently the schema goes out of its way to make that more elegant for them, and I can agree that need not be the case especially if it confuses others.

I agree with the points that modeling the experimental process is too much
to tackle, and that we need something very simple and very
abstract/extensible for this.
On Tue, Aug 25, 2015 at 4:00 AM, Jerome Kelleher<EMAIL_ADDRESS>wrote:

I have sympathy with @lh3 https://github.com/lh3's assertion too ---
this is all very complicated, and attempting to fully model all the
possible messiness will lead to something that is difficult to work with
(and, probably too rigid for future changes). I think we should aim for
something which is sufficiently flexible to make it possible to record
the provenance of everything if we want, but also to make it possible to
deliberately discard some of the complexity if we want to.
Suppose we have a record called Sample which maps to the idea of a
SequencerInput in this PR. It is a low-level, provenance-preserving
object which can be related back to the wet lab. We have another object
called a SampleSet, which is the aggregation of a number of Samples. The
SampleSet corresponds to NA12878 in 1000G: in reality, it corresponds to
a bunch of different biological samples, library preps, etc, but at higher
levels of abstraction, we don't care. The rest of the schema then looks
like @lh3 https://github.com/lh3's PR #383
https://github.com/ga4gh/schemas/pull/383, except we use a SampleSet
instead of a NamedSample in the VariantSet and ReadGroup.
The idea of the SampleSet is to provide a deliberate narrow waist, to
optionally discard some of the complexity of the wet lab. We can group
together a bunch of samples, so we ultimately know where everything comes
from, but we're not required to carry all this complexity through our
analyses. If we do want to carry the identifiers through the full analysis,
we can simple make a SampleSet containing exactly one Sample.
The names here are probably badly chosen, and will likely be confusing.
Does the basic idea make sense though? I.e., we make an object that records
precise biological provenance, and then group a set of these together as
the basis for our analyses.
—
Reply to this email directly or view it on GitHub
https://github.com/ga4gh/schemas/pull/391#issuecomment-134555050.

@haussler -- glad we all agree about using a "narrow waist" to avoid modeling the full experimental process.
@jeromekelleher -- yes, whiteboard+coffee would be more efficient -- feel like dropping by the Bay Area tomorrow? :)
Re how callsets relate to variantsets, as discussed by @jeromekelleher, @lh3, and @mlin -- I hear a few topics mixed together, including object naming, how genomics data is represented in today's file formats, how today's API works, and how we might want tomorrow's API to work. As a baseline, here's the understanding that I've had for a while of the current API, as documented in the diagram on ga4gh.org, as implemented by Google, and as tested by the current/old compliance test. The original intent of this approach was to allow full-fidelity import of legacy data from today's tools / file formats, but not be limited to those legacy tools and formats.

colloquial mental model:

start with a glob of goop
use a sequencer to generate a collection of reads, that you want to analyze together, for that goop
use a variant caller to generate a collection of calls from those reads

more formally:

the glob of goop doesn't have a formal model today; that's what started this discussion
a ReadGroupSet is the "collection of reads that you want to analyze together". There's lots of potential complexity in how you went from goop to lanes / runs / readgroups / BAM files / etc., but a ReadGroupSet hides all that complexity. (You can still find the messy details if you dig into the metadata, but you don't have to.)
a CallSet is a collection of calls generated from one of those "collections of reads that you want to analyze together". Again, there's potential complexity in how you went from reads to calls / gVCF files / VCF files / etc., including whether you processed one set of reads at a time or did some sort of joint calling, but a CallSet hides all that complexity.
a VariantSet contains several CallSets that make sense to analyze together. (Where 'make sense' is up to the data owner, using heuristics like those @mlin describes.)

how that maps to legacy file formats:

reads: a common case is that one ReadGroupSet is created by importing all the data in one BAM file. Different server implementors can choose to add more import flexibility -- e.g. see Google's current implementation, which includes parsing RG and SM tags, and optionally merging several files into one output.
variants: a common case is that one VariantSet is created by importing all the data in one VCF file. Different server implementors can choose to add more import flexibility -- e.g. see Google's current implementation, which includes support for merging multiple VCF files into one VariantSet, and for including variants produced by Complete Genomics.

Most of variant-centric projects I was involved have three components: 1) a spreadsheet describing sample phenotypes such as gender and population, 2) BAMs as raw data, and 3) typically one annotated VCF made in whatever way as the primary derived data. The only thing that is common across the three components is sample names. This view of variant-centric projects is simple yet very effective and most relevant to users' needs. We just need to mirror it in GA4GH. That is #383 - NamedSample=1, ReadGroupSet=2 and VariantSet=3. These are the three primary objects to a project. I feel frustrated when we, me included, keep diving into the unnecessary complexity.

In addition, I have already showed that the proposed schema with SequencerInput won't work. Should we close this PR such that we can avoid spreading discussions across too many threads (e.g. #390 and #383)?

The only problem that I can see continues to be terminology that
doesn't accurately described the concepts and conflicts with
other, well established, uses.
GA4GH APIs are intended to go way beyond replace BAMs, VCFs, and
spreed sheets.  Defining a API is different than hacking out a C
program, the terminology needs to be logical in the larger
context of the API.
NamedSample is really a grouping of sample derived data.
SampleDerivedData is a clear if this is meant to be able to
group other data from sample, such as expression data.
Also things like SampleAnalysisGroup, SampleDataAnalysisGroup,
SampleVariantData, etc.
An exact definition of what NameSample can lead you an what
context it will be used would be helpful.
Heng Li<EMAIL_ADDRESS>writes:

Most of variant-centric projects I was involved have three components: 1) a
spreadsheet describing sample phenotypes such as gender and population, 2) BAMs
as raw data, and 3) typically one annotated VCF made in whatever way as the
primary derived data. The only thing that is common across the three components
is sample names. This view of variant-centric projects is simple yet very
effective and most relevant to users' needs. We just need to mirror it in
GA4GH. That is #383 - NamedSample=1, ReadGroupSet=2 and VariantSet=3. These are
the three primary objects to a project. I feel frustrated when we, me included,
keep diving into the unnecessary complexity.
—
Reply to this email directly or view it on GitHub.*

Provenance is an extremely important problem for bioinformatics.
We need to not ignored it.  This ticket is a good part of
the discussion.
Heng Li<EMAIL_ADDRESS>writes:

In addition, I have already showed that the proposed schema with SequencerInput
won't work. Should we close this PR such that we can avoid spreading
discussions across too many threads (e.g. #390 and #383)?
—
Reply to this email directly or view it on GitHub.*

The model I hear @lh3 describing is exactly what I describe above (https://github.com/ga4gh/schemas/pull/391#issuecomment-134821662) as a common case, which today's API supports well -- one BAM is imported as one ReadGroupSet, one VCF is imported as one VariantSet, and the sample name is imported as the sampleId field. Glad we're in sync there.
And we seem to all agree it's a good idea to add a formal GoopGlob object, which would represent the-thing-bioinformaticians-often-call-a-sample, and would let us capture all the data from the phenotype spreadsheet. We haven't settled on a name for that object we like, or a formal description of it, but afaict NamedSample and SequencerInput are filling exactly the same role in the data model.
We also seem to all agree that we should use goopGlobIds to capture the provenance of reads and variants, replacing today's informal sampleId field.
The remaining questions, which seem tractable, are:

do we also want to track the intra-object provenance (e.g. which reads came from which reads, which variant calls came from which reads), or are we okay continuing to only track "all of these reads and variant calls came from the same GoopGlob"?
mostly independent of adding provenance, do we want to revisit the way we represent columns-of-calls within a VariantSet ?

Yes, the consensus is we need a data type above ReadGroupSet and VariantSet. Do you agree that this object should be decoupled from the experimental process? If yes, SequencerInput certainly does not fulfill the role. If your answer is no, how could you represent the 1000g pilot 1 example?
#390 has already covered the provence issue while the SequencerInput idea in this PR doesn't work. I found we started to discuss the same thing across threads and repeated ourselves, so thought it would be clearer to focus on fewer threads.

Currently, my NamedSample and @diekhans' SampleDerivedData, SampleAnalysisGroup etc are generic experiement-oblivious names. @jeromekelleher proposed two objects. He at least thinks a generic SampleSet is necessary. @haussler wants to avoid going too deep into experiments. @mlin does not object to having this global generic object. @dglazer, if you are also on board, we will have a consensus on the role of this global "sample"-like object.

On naming, the best so far is @diekhans' SampleAnalysisGroup in my view. Thanks for the suggestion.

Since this is a data sharing API, we need to allow missing data.
Ideal ReadGroupSet and VariantSet sent should not be decoupled,
but the support just might not exist, especially for variants
(e.g. clinvar).
The higher level grouping should just point to ReadGroupSet,
VariantSet, and the missing AlignmentSet.
The reason to keep this PR open is to continue to discuss
provenance, not to block other progress.
Heng Li<EMAIL_ADDRESS>writes:

Yes, the consensus is we need a data type above ReadGroupSet and VariantSet. Do
you agree that this object should be decoupled from the experimental process?
If yes, SequencerInput certainly does not fulfill the role. If your answer is
no, how could you represent the 1000g pilot 1 example?
#390 has already covered the provence issue while the SequencerInput idea in
this PR doesn't work. I found we started to discuss the same thing across
threads and repeated ourselves, so thought it would be clearer to focus on
fewer threads.
—
Reply to this email directly or view it on GitHub.*

@diekhans you are right about missing data. In #383, we can simply set VariantSet::sampleNames to an empty array for your clinvar example. This dataset will only have one VariantSet, but no ReadGroupSets or SampleAnalysisGroups.

Including a description of what it means for ReadGroupSets to be
empty in the schema doc would be useful.
Heng Li<EMAIL_ADDRESS>writes:

@diekhans you are right about missing data. In #383, we can simply set
VariantSet::sampleNames to an empty array for your clinvar example. This
dataset will only have one VariantSet, but no ReadGroupSets or
SampleAnalysisGroups.
—
Reply to this email directly or view it on GitHub.*

@dglazer wrote:

mostly independent of adding provenance, do we want to revisit the way we represent columns-of-calls within a VariantSet

I'd withdrawn my claim in #383 that these can be separated, but since there is now some traction, I think we should reorganize the tickets that way, maybe refactor the diffs and open new PRs with relatively neutral summaries of the discussion so far. I'd volunteer to write the summary for CallSet since I think I see both sides of it.

@lh3, when you say "SequencerInput certainly does not fulfill the role" of an object "decoupled from the experimental process", are you referring to the object name or to the object definition?

if it's the name, I agree -- I don't like any of the names I've heard yet, which is why I've started using GoopGlob as a placeholder until we come up with a good one
if it's the description in the header comment, I'm wide open to improvement -- do you have a description you like? (The current description in #383 doesn't really say what the object is; it just says where it comes from in BAM and VCF files.)
if it's the definition, I'm confused -- I think NamedSample in #383 and SequencerInput here are almost identical objects, consisting of an id, name, and generic metadata.

@diekhans, re intra-object provenance -- I agree our model has to work whether or not we know all the backlinks. I think that's easy to support; this PR does it by simply allowing the "what was I derived from" fields to be null:

ReadGroupSet contains an optional goopGlobId and an optional readGroupSetId. If you know what goop it came from, store that. And/or, if you know what unaligned rgset it came from, store that.
CallSet contains an optional goopGlobId and an optional readGroupSetId. If you know what goop it came from, store that. And/or, if you know what aligned rgset it came from, store that.

In both cases, null values in those id fields simply mean "not specified" -- the reads and calls clearly came from some goop, and the calls likely came from some reads, but if the data owner doesn't know the details, they shouldn't have to make something up.

@min, re splitting out the discussion of "how we represent columns-of-calls within a VariantSet" -- feel free to open a new PR focused on that question if you think it will help the discussion. (I slightly lean towards yes, it will help, but only slightly.)  Two notes if you do so:

I think this PR already shows how we can add the provenance we've been discussing without changing how we represent a column-of-calls, so we don't need a new PR to cover that.
I'll be curious to see how you propose representing "column 1 was called using toolchain T1 on date D1" and "column 2 was called using toolchain T2 on date D2"; that's where I get stuck when I think about any big changes to CallSet.

The name and description of SequencerInput are wrong. More importantly, the intention is wrong in that you are trying to explain this global "sample" with experiments. I will state this: the global "sample"-equivalent object should not be mapped to any experimental material (e.g. individual, tissue, sequencing technology, run, library, barcode, etc). If you agree with the statement, we have a consensus.
I will talk about CallSet in detail once we have a specific thread. Just briefly, the CallSet in your mind is an attribute, having a similar role to Program in reads.avdl. It is not part of the provence chain. The primary objects in this chain should be:
record Dataset {} // the largest container
record SampleAnalysisGroup { string datasetId; string sampleName; }
record ReadGroupSet { string datasetId; }
record ReadGroup { string readGroupSetId; string sampleId; }
record ReadAlignment { string readGroupId; }
record VariantSet { string datasetId; array<string> sampleIds; }
record Call {} // embedded in Variant only
record Variant { string variantSetId; array<Call> calls; }

@dglazer sure. I cannot resist shooting from the hip to your second question: the debate will ultimately come down to "Should we actually be solving for [the situation you describe]?", because performing that composition is problematic -- not just due to the file format, but fundamentally due to difficulties in our abstract model of genetic variation. This has been mentioned tangentially before, e.g. in #49 and #168, but a relatively focused discussion could be really valuable (if you ever wondered what all that "monoallelic" or "unary" business was about -- this). I think I see both sides of the issue as I said, there are both theoretical and pragmatic arguments, and I will try to write something as clear as possible, perhaps over the weekend.

@lh3, I like your model. It's simple, and covers the situations we are likely to encounter with today's VCF based variant data. As @mlin points out, VCF isn't just a file format, it represents a specific way of defining variants (I'm looking forward to your summary here @mlin!) .
A couple of questions for @lh3:

When you have a sampleId in ReadGroup and VariantSet, is this a reference back to the SampleAnalysisGroup? sampleAnalysisGroupId is perhaps a bit clumsy, but consistency is a good thing.

Do you think there is value in @dglazer's GoopGlob being modelled, and being contained in the SampleAnalysisGroup? That is, SampleAnalysis group contains an array of GoopGlobs, which can be empty.

@lh3 @jeromekelleher I would support something like sampleAnalysisGroupId. We have currently Sample as record type, representing a biological extraction & corresponding to the "biosample" concept. I'll bring up (probably today) in metadata if we should change this to BioSample; but irrespective of the outcome, "sample" is very polymorphic & probably should best be used with an additional qualifier. sampleId will lead to misunderstandings.

@jeromekelleher On 1, yes, it should be sampleAnalysisGroupId. On 2, I believe GoopGlob suggested by @dglazer is just SampleAnalysisGroup.
@mbaudis yes, we also need BioSample from you. My intention is to add array<string> bioSampleIds to SampleAnalysisGroup at some point. Most of time, SampleAnalysisGroup corresponds to one BioSample, but occasionally, it will corresponds to multiple BioSamples. BioSample sits above Datasets to identify biological samples used in multiple datasets.

@lh3, when you say, "The name and description of SequencerInput are wrong.", I think that means that the object definition is right. If so we've been in agreement on the "code" all along, which is great, and wasn't clear to me from the discussion. Do you have an object description to propose? Once we agree on the description, it should be easier to agree on a short accurate name.
@jeromekelleher -- @lh3 is right that I think a GoopGlob is the same as whatever we end up calling this object -- we only need one of them.
@mlin -- looking forward to your PR. I understand at least some of the challenges with composition, and also know there are benefits -- we can discuss the tradeoffs there.

@lh3 O.k.; I'll bring this up. I'm generally in support of this approach (array<string> bioSampleIds) which is in my way of thinking about references/object relations.
The point not addressed here is then the type of ids to use (we'll probably need id (i.e. locally unique, readable ...), guid (collision free uuid, robust for referencing, even across collections) and possibly some version id/hash. I'll have to write this up as a general issue one day (id @dglazer won't do it first ...).

I didn't answer your question directly because I didn't know what "definition" means exactly in this context. Now I see that by "definition", you mean the "code". No, we have a fundamental difference: mine has a datasetId field and forces the uniqueness of sampleNames within a Dataset, but yours doesn't. In its current form, your SequencerInput is a loose definition of BioSample. It is not what we need most right now.

Sorry for the long email that follows.
I want to jump in with a concrete example in order to see if I understand
the approach we are converging on with these primary objects:
record Dataset {} // the largest container
record SampleAnalysisGroup { string datasetId; string sampleName; }
record ReadGroupSet { string datasetId; }
record ReadGroup { string readGroupSetId; string sampleId; }
record ReadAlignment { string readGroupId; }
record VariantSet { string datasetId; array sampleIds; }
record Call {} // embedded in Variant only
record Variant { string variantSetId; array calls; }
Here is the example.  They want
the most complete genetic analysis of the child's cancer possible. In
particular, they want a completely phased version of their child's germline
genome to compare to the sequencing reads of the child's tumor biopsy so
they can best detect the specific genetic changes in the child's tumor. So
the parent's genome get their germline genomes sequenced along with the
child's germline genome is sequenced and phased using the parent's genomes.
There is a biopsy taken from the child's tumor and sequenced. The biopsy is
2/3 cancer cells and 1/3 normal germline cells. In summary:

Raw input:

There are 4 biological tissue samples: blood from mom, dad and child,
tumor from child.

There are 4 sequencing machine runs, one for each biological sample

Desired output for clinical analysis:

For every place where the tumor cells are carrying a somatic variant
allele that is not in the child's germline, the doctor would like to have
the following alleles: the mom's 2 alleles, the dad's 2 alleles, the
child's 2 germline alleles, and the alleles that were observed in the tumor
biopsy.

Some information on the frequencies that these alleles are observed.

Notes: In most cases all 6 germline alleles will be the same, because it is
unlikely to have a somatic mutation at a heterozygous position because
heterozygous positions are relatively rare, but this does occur sometimes.
When it happens one wants to check that there is not an error or artifact.
It is expected that the germline alleles will occur in the biopsy along
with the new somatic variant, even if the somatic variant is homozygous or
hemizygous in the tumor cells. This is because there are germline cells
mixed in with the tumor cells in the tumor biopsy. So there are really two
different allele frequency estimates one wants for the tumor cells: (1)
what is the observed frequency and (2) what is the frequency one would
estimate after subtracting off the number of reads that one would expect
came from the normal cells within the tumor sample.
So, starting with "Variant" and working backwards, it seems that there
should be (at minimum) a Variant  record for each somatic variant that is
is detected. This record will be an array of 4 calls, one each for mom,
dad, child, and tumor. Each call would be associated with some frequency
information (distinguishing the two types of frequencies above, and maybe
quality and confidence.
The identifiers for these 4 samples, "mom", "dad", "child", "tumor", are
the SampleIDs in VariantSet
There would be 4 corresponding read group sets, identified with
ReadGroupSetIDs, each paired with a SampleID in a ReadGroup record.
Altogether these data form a SampleAnalysisGroup, which is part of a
DataSet.
Do I have this right? It seems quite sensible to me.  Thanks. -D
On Thu, Aug 27, 2015 at 6:57 AM, Heng Li<EMAIL_ADDRESS>wrote:

I didn't answer your question directly because I didn't know what
"definition" means exactly in this context. Now I see that by "definition",
you mean the "code". No, we have a fundamental difference: mine has a
datasetId field and forces the uniqueness of sampleNames within a
Dataset, but yours doesn't. In its current form, your SequencerInput is a
loose definition of BioSample. It is not what we need most right now.
—
Reply to this email directly or view it on GitHub
https://github.com/ga4gh/schemas/pull/391#issuecomment-135442432.

@haussler Thanks for the very good/clear/example based description!
Just some notes from my POV (metadata centric):

I'm not sure about SampleAnalysisGroup;  why specifying "Sample", not just "AnalysisGroup" or "SequenceAnalysisGroup"? The "Sample" part is either unnecessary, and/or too much of a land grab (generally, we would like to use the data model for other types of molecular data).
We're about to change the Sample record in the current metadata branch to Biosample (or Biospecimen, alas). So this would be the main reference point; in your case 4 Biosample ids from 3 Individuals.

Ok, name SampleAnalysisGroup is still confusing, probably caused by word "Group". I will use AnalysisSample for now (this may be the 10-th or 11-th name of this object and more are likely to come; sorry!). With the name change, the primary objects in my mind are:
// in metadata.avdl
record BioSample {} // equivalent to a sample in the BioSample database
record Dataset {}
record AnalysisSample { string datasetId, name; array<string> bioSampleIds; }
// in reads.avdl
record ReadGroupSet { string datasetId; }
record ReadGroup { string readGroupSetId, analysisSampleId, bioSampleId; }
record ReadAlignment { string readGroupId; }
// in variants.avdl
record VariantSet { string datasetId; array<string> analysisSampleIds; }
record Call {} // embedded in Variant only
record Variant { string variantSetId; array<Call> calls; }

For your example, @haussler, there will be:

one Dataset for all the data you mentioned.
four AnalysisSamples, with their AnalysisSample::name being "dad", "mom", "child-normal" and "child-tumor", respectively. As @mbaudis said, the relationship between "child-normal" and "child-tumor" will be tracked by global objects such as BioSample and Individual.
four ReadGroupSets by convention. (It may be good to discuss whether to force this convention in our API by moving ReadGroup::analysisSampleId to ReadGroupSet, but that is another topic. Let's not discuss this here.)
one or multiple ReadGroups per ReadGroupSet, depending on the experiment procedure.
one VariantSet consisting of all four AnalysisSamples
multiple Variants in the VariantSet, with each keeping a potential somatic mutation associated with per-sample frequency, etc.

Note that analysisSampleIds are computer generated identifiers. AnalysisSample::name is the common name, which shall be unique within a Dataset.
@mbaudis I put a word "sample" in AnalysisSample because it is the lousy "sample" we say daily when we do analysis. I actually find the word "Group" is confusing. It is theoretically correct in that an AnalysisSample groups bio specimens. However, in practice, it sounds like it is a group of lousy "sample"s. Anyway, I agree that AnalysisSample is not quite right, either, and I am always open to better alternatives and more precise definition of this object.

Thanks for AnalysisSample @lh3 -- it's the best name I've heard yet; I agree it's not perfect and is likely to change again, but it's a better placeholder than GoopGlob -- I'll update this PR when I get a chance.
@haussler -- I'll respond on your detailed example later.

AnalysisGroup?
;-)

Name things clearly is hard, but worth taking the time for future
users.  My concern is for a term that conveys this is much more
than a biological sample.  here is another try: SampleDerivedData
Heng Li<EMAIL_ADDRESS>writes:

Ok, name SampleAnalysisGroup is still confusing, probably caused by word
"Group". I will use AnalysisSample for now (this may be the 10-th or 11-th name
of this object and more are likely to come; sorry!). With the name change, the
primary objects in my mind are:
// in metadata.avdl
record BioSample {} // equivalent to a sample in the BioSample database
record Dataset {}
record AnalysisSample { string datasetId, name; array bioSampleIds; }
// in reads.avdl
record ReadGroupSet { string datasetId; }
record ReadGroup { string readGroupSetId, analysisSampleId, bioSampleId; }
record ReadAlignment { string readGroupId; }
// in variants.avdl
record VariantSet { string datasetId; array analysisSampleIds; }
record Call {} // embedded in Variant only
record Variant { string variantSetId; array calls; }
For your example, @haussler, there will be:
• one Dataset for all the data you mentioned.
• four AnalysisSamples, with their AnalysisSample::name being "dad", "mom",
"child-normal" and "child-tumor", respectively. As @mbaudis said, the
relationship between "child-normal" and "child-tumor" will be tracked by
global objects such as BioSample and Individual.
• four ReadGroupSets by convention. (It may be good to discuss whether to
force this convention in our API by moving ReadGroup::analysisSampleId to
ReadGroupSet, but that is another topic. Let's not discuss this here.)
• one or multiple ReadGroups per ReadGroupSet, depending on the experiment
procedure.
• one VariantSet consisting of all four AnalysisSamples
• multiple Variants in the VariantSet, with each keeping a potential somatic
mutation associated with per-sample frequency, etc.
Note that analysisSampleIds are computer generated identifiers.
AnalysisSample::name is the common name, which shall be unique within a
Dataset.
@mbaudis I put a word "sample" in AnalysisSample because it is the lousy
"sample" we say daily when we do analysis. I actually find the word "Group" is
confusing. It is theoretically correct in that an AnalysisSample groups bio
specimens. However, in practice, it sounds like it is a group of lousy
"sample"s. Anyway, I agree that AnalysisSample is not quite right, either, and
I am always open to better alternatives and more precise definition of this
object.
—
Reply to this email directly or view it on GitHub.*

Sorry to add another opinion regarding the naming, but I see 2 general concepts which could be used:

SequenceAnalysisGroup, which would make it very descriptive but also  specific to the technology at hand
AnalysisGroup, with additional type descriptor, which would be more inclusive/forward looking

The first option would seem like a perfect fit for David H.'s example; the 2nd would work equally well, but should use a standard qualifier for type or the like.
This is more about the concepts (separate record types for different analysis schemes vs. a fixed one with additional descriptors) than about the specific names.

I think there is room for some exploration in an alpha release, or series
of them. We should be trying to use something like this in practice and
getting a sense of what does and doesn't work with an eye for making
breaking changes that improve things.
I'm just seeing a lot of conversation about the specific structure used to
represent this or that and in almost no instance is anyone saying "we
do/did this and this it had these consequences and this rationale". (@mlin
made a practical example but I can't remember any others.)
So basically, what color should we paint the bike shed?
These conversations seem innocuous but they are taking energy that could be
spent making things.
On Aug 28, 2015 8:05 AM, "Michael Baudis"<EMAIL_ADDRESS>wrote:

Sorry to add another opinion regarding the naming, but I see 2 general
concepts which could be used:

SequenceAnalysisGroup, which would make it very descriptive but also
specific to the technology at hand
AnalysisGroup, with additional type descriptor, which would be more
inclusive/forward looking

The first option would seem like a perfect fit for David H.'s example; the
2nd would work equally well, but should use a standard qualifier for type
or the like.
This is more about the concepts (separate record types for different
analysis schemes vs. a fixed one with additional descriptors) than about
the specific names.
—
Reply to this email directly or view it on GitHub
https://github.com/ga4gh/schemas/pull/391#issuecomment-135657531.

How about SampleAnalysisTag or SampleTag?  I think tag captures the concept
better than group.
From Wikipedia----
In information systems https://en.wikipedia.org/wiki/Information_system,
a tag is a non-hierarchical keyword or term
https://en.wikipedia.org/wiki/Index_term assigned to a piece of
information (such as an Internet bookmark
https://en.wikipedia.org/wiki/Bookmark_(World_Wide_Web), digital image,
or computer file https://en.wikipedia.org/wiki/Computer_file). This kind
of metadata https://en.wikipedia.org/wiki/Metadata helps describe an item
and allows it to be found again by browsing or searching. Tags are
generally chosen informally and personally by the item's creator or by its
viewer, depending on the system.
On Fri, Aug 28, 2015 at 3:05 AM, Michael Baudis<EMAIL_ADDRESS>wrote:

Sorry to add another opinion regarding the naming, but I see 2 general
concepts which could be used:

SequenceAnalysisGroup, which would make it very descriptive but also
specific to the technology at hand
AnalysisGroup, with additional type descriptor, which would be more
inclusive/forward looking

The first option would seem like a perfect fit for David H.'s example; the
2nd would work equally well, but should use a standard qualifier for type
or the like.
This is more about the concepts (separate record types for different
analysis schemes vs. a fixed one with additional descriptors) than about
the specific names.
—
Reply to this email directly or view it on GitHub
https://github.com/ga4gh/schemas/pull/391#issuecomment-135657531.

--
John Farrell, Ph.D.
IT Manager
Biomedical Genetics-Evans 218
Boston University Medical School
72 East Concord Street
Boston, MA
ph:<PHONE_NUMBER>

@jjfarrell I don't object to SampleTag. I also think Tag is better than Group. Thanks for the suggestion.
@ekg We have been challenging each other with real and hypothetical use cases and showed why a proposal is not working. We care about practical uses.

@haussler -- I like your example of four tissue sources. As a baseline, here's how the data you describe would be represented in today's 0.5.1 API if you imported it into Google's implementation:

you'd create one Dataset for all the data you mentioned (and possibly for more data from other sources; your choice)

there would be four ReadGroupSets in the Dataset

there would be one or more ReadGroups in each ReadGroupSet (depending on the experimental procedure). Each ReadGroup would have a sampleId text field saying whether it came from "mom", "dad", "child", or "tumor".

there would be one VariantSet in the Dataset

there would be four CallSets in the VariantSet. Each CallSet would have a sampleId text field saying whether it came from "mom", "dad", "child", or "tumor".
there would be multiple Variants in the VariantSet
there would be multiple Calls in the VariantSet, with each containing information about the likelihood that a particular Variant was present in a particular CallSet

This 0.5.1 representation works pretty well imo -- it does a good job of representing all the 'dry' objects, and a decent but unstructured way of representing how those objects are related. Here's how I picture that representation changing after this PR:

you'd create one Dataset for all the data you mentioned (and possibly for more data from other sources; your choice)

there would be four AnalysisSamples in the Dataset. Each AnalysisSample would have a name text field saying whether it came from "mom", "dad", "child", or "tumor", and probably would have other freeform metadata fields describing the tissue (e.g. tumor vs. normal).
there would be four ReadGroupSets in the Dataset. Each ReadGroupSet would have an analysisSampleId field pointing back to one of the four AnalysisSample objects.

there would be one or more ReadGroups in each ReadGroupSet (depending on the experimental procedure). Each ReadGroup would have an analysisSampleId field..

there would be one VariantSet in the Dataset

there would be four CallSets in the VariantSet. Each CallSet would have an analysisSampleId field, and a readGroupSetId field indicating which ReadGroupSet it was derived from.
there would be multiple Variants in the VariantSet
there would be multiple Calls in the VariantSet, with each containing information about the likelihood that a particular Variant was present in a particular CallSet

Note that very little changes -- the data model representing the "dry" objects is unchanged, The new part is that there would now be

a) a single new object type representing the "wet" source of those dry objects
b) a formal way of tracking the inter-object relationships.

(Note also that @lh3 and I gave very similar answers -- the differences I see are the discussion about columns-of-calls that we've already decided to fork off, and the relatively minor point about whether we should allow propagating analysis sample info from the readgroups up to the readgroupset.)

In addition to CallSet, we haven't reached an agreement on AnalysisSample: I argue that an AnalysisSample ought to be attached to a Dataset because at times a sample name is meaningless outside a Dataset. Tell me what is wrong with having a AnalysisSample::datasetId field.

Use cases to show why lacking AnalysisSample::datasetId is problematic. One of the first things we query a project is to get the list of sample names. With an AnalysisSample::datasetId field, this is trivial. Without it, you have to go into ReadGroups and your CallSets to get the list. What if you get two different lists? Take a union? In addition, where do we impose the sample name uniqueness? - trivial with my proposal - or do you think you don't need this naming uniqueness? Show me use cases about why having AnalysisSample::datasetId is wrong.

@lh3 -- I just added AnalysisSample::datasetId. I believe my mental model of its behavior is still a bit different from yours (more below), but I think that means we're now in complete agreement on the 'code' of the proposed new object.
My API-wide model is that name fields are assigned by humans for primarily human consumption, and can't be depended on to have rigid semantics. We introduce formal objects with machine-assigned id fields when we need formal structure and semantics. I agree that it's good practice for the sample name fields to be unique within a particular experiment, but I don't think it's appropriate for the API to enforce that.
One benefit of separating human-readable names from machine-assigned ids is that it adds data management flexibility. For example, it makes it easier to take data from several independent experiments and do cross-analysis. It's not a magic bullet -- you still need some way to tell which data came from which source -- but you can use whatever messy real-world combination of filenames, directories, manifests, etc. you want, as long as you make sure that, at data import time, the right things get mapped to the right objects.

Naming uniqueness within a dataset is a fundamental assumption almost everywhere. When you import data from BAM/VCF, you have to make this assumption - then the repo simply follows the assumption. When users say "give me the NA12878 reads", they assume they would not get data from two unrelated individuals. In IGV, we would be surprised to see an NA12878 in VCF is different from an NA12878 alignment at most variants. During VariantSet traversal, we would not know what to do if we see two columns having the same name but different IDs. You may argue we can replace NA12878 with IDs in my examples, but human brains work better with common names than meaningless IDs - this is why we give samples reasonable names in the first place. Yes, the world still functions without the uniqueness naming assumption, but then API users and developers will have many ways to blow their feet.

@lh3 Regarding guid (global) vs. id (local), the schema should include both types for must objects (with guid usage itself declared optional).
There are reasons for stripping guids from objects depending on the use case (privacy/identification etc.), and generally, as you said, guide are machine but not brain compatible. However, many use cases will benefit from the identification of identical objects across collections (though even that cannot be guaranteed without a central authority, which is not intended).
My point above about which id type to use for AnalysisGroup, Dataset etc. references could actually be left open; you just reference a list of object ids, and the API queries all id types for matches. However, this is probably nothing SQL developers would like :-/

@mbaudis striping GUIDs will cause huge amounts of
complexity. If striping GUIDs removes privacy problems, there is
most likely a privacy problem already even with it removed.
Michael Baudis<EMAIL_ADDRESS>writes:

@lh3 Regarding guid (global) vs. id (local), the schema should include both
types for must objects (with guid usage itself declared optional).
There are reasons for stripping guids from objects depending on the use case
(privacy/identification etc.), and generally, as you said, guide are machine
but not brain compatible. However, many use cases will benefit from the
identification of identical objects across collections (though even that cannot
be guaranteed without a central authority, which is not intended).
My point above about which id type to use for AnalysisGroup, Dataset etc.
references could actually be left open; you just reference a list of object
ids, and the API queries all id types for matches. However, this is probably
nothing SQL developers would like :-/
—
Reply to this email directly or view it on GitHub.*

Thanks so much for the deep analysis of the mom, dad, child, tumor biopsy
case! It all sounds great and very sensible, and I'm glad there is so much
convergence. I have only one additional point of discussion. It regards
these three parts

there would be four CallSets in the VariantSet. Each CallSet would
have an analysisSampleId field, and a readGroupSetId field indicating which
ReadGroupSet it was derived from.
there would be multiple Variants in the VariantSet
there would be multiple Calls in the VariantSet, with each containing
information about the likelihood that a particular Variant was present
in a particular CallSet

The tumor biopsy is a mixture of both tumor and normal cells. One way to
produce a call set for such a mixed sample is to call anything that exists
in any part of the mixture. But in this case, perhaps a better way to
produce a call set is to just call what exists (by inference) in the tumor.
You do the latter by using your knowledge of the normal genome (obtained
from the other 3 samples in this case) to essentially "subtract off" reads
from the normal cells in the tumor biopsy. People should be allowed to
produce either type of call set, but since it is vitally important for the
interpretation of the calls to know which type of call set it is, I would
hope that this information on how to interpret the call set is quite
apparent. Will it be made apparent?
There is in VCF 4.2 a formal notion of a sample containing a mixture of
distinct genomes in specified proportions. In VCF 4.2 a distinction can be
made between calls on the sample (i.e. calling anything that appears
anywhere in the mixture according to its overall frequency in the mixed
sample) and calls on each genome separately. The latter are specified in
two steps. First you name the genomes you think are mixed together in the
sample and say what proportion of the sample each occupies, and then you
make a separate call set for each genome. This "deconvolutes" the mixture.
I'm assuming this is seldom used. Probably people just define virtual
samples to deconvolute a mixture, as follows:mixed-tissue
In cancer we essentially always want to deconvolute the tumor calls from
the normal calls in a biopsy. In fact, the tumor may contain, say, 5
subclones, each with a different genome. In that case we would want to
deconvolute the tumor biopsy into 6 parts: calls from the normal cells and
call from each of the 5 different types of tumor cells. This perhaps gets
more difficult to convey naturally when there is just 1 sample for the
tumor biopsy. However, we can "virtually divide" the tumor biopsy sample
into 6 different samples, so we now have 9 samples in the data set: mom,
dad, child-germline-from-blood, child-germline-from-tumor-biopsy,
clone1-from-biopsy, ..., clone5-from-biopsy. (Or we could combine the two
child germile samples and get 8). From my reading of the spec, you are
allowed to define such "virtual samples". Each of these samples could have
a different call set. Then all you would need to know in addition to this
is what proportion of the cells in the tumor-biopsy were each of the 6
possible child genomes. (And of course somewhere you should be able to
determine that these were not actually separate biological samples but were
inferred.)
In the end, somebody might want to store an analysis of the same raw
dataset done both ways: once with 4 callsets: mom, dad, child, and
tumor-biopsy, and once with 8 or 9 callsets for "virtual samples" as above.
It sounds to me like all this is possible, right?   -d
On Fri, Aug 28, 2015 at 5:44 AM, David Glazer<EMAIL_ADDRESS>wrote:

@haussler https://github.com/haussler -- I like your example of four
tissue sources. As a baseline, here's how the data you describe would be
represented in today's 0.5.1 API if you imported it into Google's
implementation:

you'd create one Dataset for all the data you mentioned (and
possibly for more data from other sources; your choice)

there would be four ReadGroupSets in the Dataset

there would be one or more ReadGroups in each ReadGroupSet
(depending on the experimental procedure). Each ReadGroup would
have a sampleId text field saying whether it came from "mom",
"dad", "child", or "tumor".

there would be one VariantSet in the Dataset

there would be four CallSets in the VariantSet. Each CallSet
would have a sampleId text field saying whether it came from
"mom", "dad", "child", or "tumor".
there would be multiple Variants in the VariantSet
there would be multiple Calls in the VariantSet, with each
containing information about the likelihood that a particular
Variant was present in a particular CallSet

This 0.5.1 representation works pretty well imo -- it does a good job of
representing all the 'dry' objects, and a decent but unstructured way of
representing how those objects are related. Here's how I picture that
representation changing after this PR:

you'd create one Dataset for all the data you mentioned (and
possibly for more data from other sources; your choice)

there would be four AnalysisSamples in the Dataset. Each
AnalysisSample would have a name text field saying whether it came from
"mom", "dad", "child", or "tumor", and probably would have other freeform
metadata fields describing the tissue (e.g. tumor vs. normal).
there would be four ReadGroupSets in the Dataset. Each
ReadGroupSet would have an analysisSampleId field pointing back to one of
the four AnalysisSample objects.

there would be one or more ReadGroups in each ReadGroupSet
(depending on the experimental procedure). Each ReadGroup would
have an analysisSampleId field..

there would be one VariantSet in the Dataset

there would be four CallSets in the VariantSet. Each CallSet
would have an analysisSampleId field, and a readGroupSetId field indicating
which ReadGroupSet it was derived from.
there would be multiple Variants in the VariantSet
there would be multiple Calls in the VariantSet, with each
containing information about the likelihood that a particular
Variant was present in a particular CallSet

Note that very little changes -- the data model representing the "dry"
objects is unchanged, The new part is that there would now be

a) a single new object type representing the "wet" source of those
dry objects
b) a formal way of tracking the inter-object relationships.

(Note also that @lh3 https://github.com/lh3 and I gave very similar
answers -- the differences I see are the discussion about columns-of-calls
that we've already decided to fork off, and the relatively minor point
about whether we should allow propagating analysis sample info from the
readgroups up to the readgroupset.)
—
Reply to this email directly or view it on GitHub
https://github.com/ga4gh/schemas/pull/391#issuecomment-135765705.

replace "follows:mixed-tissue" with "follows:"
On Fri, Aug 28, 2015 at 1:07 PM, David Haussler<EMAIL_ADDRESS>wrote:

Thanks so much for the deep analysis of the mom, dad, child, tumor biopsy
case! It all sounds great and very sensible, and I'm glad there is so much
convergence. I have only one additional point of discussion. It regards
these three parts

there would be four CallSets in the VariantSet. Each CallSet would
have an analysisSampleId field, and a readGroupSetId field indicating which
ReadGroupSet it was derived from.
there would be multiple Variants in the VariantSet
there would be multiple Calls in the VariantSet, with each
containing information about the likelihood that a particular Variant
was present in a particular CallSet

The tumor biopsy is a mixture of both tumor and normal cells. One way to
produce a call set for such a mixed sample is to call anything that exists
in any part of the mixture. But in this case, perhaps a better way to
produce a call set is to just call what exists (by inference) in the tumor.
You do the latter by using your knowledge of the normal genome (obtained
from the other 3 samples in this case) to essentially "subtract off" reads
from the normal cells in the tumor biopsy. People should be allowed to
produce either type of call set, but since it is vitally important for the
interpretation of the calls to know which type of call set it is, I would
hope that this information on how to interpret the call set is quite
apparent. Will it be made apparent?
There is in VCF 4.2 a formal notion of a sample containing a mixture of
distinct genomes in specified proportions. In VCF 4.2 a distinction can be
made between calls on the sample (i.e. calling anything that appears
anywhere in the mixture according to its overall frequency in the mixed
sample) and calls on each genome separately. The latter are specified in
two steps. First you name the genomes you think are mixed together in the
sample and say what proportion of the sample each occupies, and then you
make a separate call set for each genome. This "deconvolutes" the mixture.
I'm assuming this is seldom used. Probably people just define virtual
samples to deconvolute a mixture, as follows:mixed-tissue
In cancer we essentially always want to deconvolute the tumor calls from
the normal calls in a biopsy. In fact, the tumor may contain, say, 5
subclones, each with a different genome. In that case we would want to
deconvolute the tumor biopsy into 6 parts: calls from the normal cells and
call from each of the 5 different types of tumor cells. This perhaps gets
more difficult to convey naturally when there is just 1 sample for the
tumor biopsy. However, we can "virtually divide" the tumor biopsy sample
into 6 different samples, so we now have 9 samples in the data set: mom,
dad, child-germline-from-blood, child-germline-from-tumor-biopsy,
clone1-from-biopsy, ..., clone5-from-biopsy. (Or we could combine the two
child germile samples and get 8). From my reading of the spec, you are
allowed to define such "virtual samples". Each of these samples could have
a different call set. Then all you would need to know in addition to this
is what proportion of the cells in the tumor-biopsy were each of the 6
possible child genomes. (And of course somewhere you should be able to
determine that these were not actually separate biological samples but were
inferred.)
In the end, somebody might want to store an analysis of the same raw
dataset done both ways: once with 4 callsets: mom, dad, child, and
tumor-biopsy, and once with 8 or 9 callsets for "virtual samples" as above.
It sounds to me like all this is possible, right?   -d
On Fri, Aug 28, 2015 at 5:44 AM, David Glazer<EMAIL_ADDRESS>wrote:

@haussler https://github.com/haussler -- I like your example of four
tissue sources. As a baseline, here's how the data you describe would be
represented in today's 0.5.1 API if you imported it into Google's
implementation:

you'd create one Dataset for all the data you mentioned (and
possibly for more data from other sources; your choice)

there would be four ReadGroupSets in the Dataset

there would be one or more ReadGroups in each ReadGroupSet
(depending on the experimental procedure). Each ReadGroup would
have a sampleId text field saying whether it came from "mom",
"dad", "child", or "tumor".

there would be one VariantSet in the Dataset

there would be four CallSets in the VariantSet. Each CallSet
would have a sampleId text field saying whether it came from
"mom", "dad", "child", or "tumor".
there would be multiple Variants in the VariantSet
there would be multiple Calls in the VariantSet, with each
containing information about the likelihood that a particular
Variant was present in a particular CallSet

This 0.5.1 representation works pretty well imo -- it does a good job of
representing all the 'dry' objects, and a decent but unstructured way of
representing how those objects are related. Here's how I picture that
representation changing after this PR:

you'd create one Dataset for all the data you mentioned (and
possibly for more data from other sources; your choice)

there would be four AnalysisSamples in the Dataset. Each
AnalysisSample would have a name text field saying whether it came from
"mom", "dad", "child", or "tumor", and probably would have other freeform
metadata fields describing the tissue (e.g. tumor vs. normal).
there would be four ReadGroupSets in the Dataset. Each
ReadGroupSet would have an analysisSampleId field pointing back to one of
the four AnalysisSample objects.

there would be one or more ReadGroups in each ReadGroupSet
(depending on the experimental procedure). Each ReadGroup would
have an analysisSampleId field..

there would be one VariantSet in the Dataset

there would be four CallSets in the VariantSet. Each CallSet
would have an analysisSampleId field, and a readGroupSetId field indicating
which ReadGroupSet it was derived from.
there would be multiple Variants in the VariantSet
there would be multiple Calls in the VariantSet, with each
containing information about the likelihood that a particular
Variant was present in a particular CallSet

Note that very little changes -- the data model representing the "dry"
objects is unchanged, The new part is that there would now be

a) a single new object type representing the "wet" source of those
dry objects
b) a formal way of tracking the inter-object relationships.

(Note also that @lh3 https://github.com/lh3 and I gave very similar
answers -- the differences I see are the discussion about columns-of-calls
that we've already decided to fork off, and the relatively minor point
about whether we should allow propagating analysis sample info from the
readgroups up to the readgroupset.)
—
Reply to this email directly or view it on GitHub
https://github.com/ga4gh/schemas/pull/391#issuecomment-135765705.

Are there 4 or 8+ AnalysisSamples? If it is 4 (dad, mom, child and tumor), I guess we can't create a "callset" for "child" plus the normal part of "tumor". If we have 8, what's the use of CallSet? Similarly, what if we have both blood and saliva samples and want to give two separate variant sets and also a blood-saliva mixed variant set? How many AnalysisSamples and CallSets are there?
I think the confusion is caused by the two "sample" related objects AnalysisSample and CallSet. We need to deal with the relationship between them, which is not always obvious. It would seem cleaner to just have 8 AnalysisSamples (dad, mom, child+tumorNormal, clone1, ..., clone5), labeling all the possible definitions of samples used in this dataset, with their relationship kept track by other attribute or meta fields -- we need these fields anyway for tumor-normal and pedigree information. VariantSets just reference these sample labels/tags.

@lh3  @haussler   Another approach is to tag each read in the ChildTumor ReadSet with the appropriate genome eg:  (read.info['genome']="clone1" .   CallSets would then need to be based on a query that are a subset of the ReadSet: something like  TumorReadSet.filter("info[genome]="clone1") .   Besides this cancer example, ReadSet filtering is needed based on MQ, RD, Genome Mask and other QC metrics.  The resulting CallSet would need to store the query or search request.   Using SQL for these type of queries would be much more flexible than the present API which is very limited.
When the CallSet is merged into a VariantSet, the AnalysisSample.name could include the CallSetName as a prefix (clone1.ChildTumor).
This example and the GATK best practice pipeline would be useful exercises to sketch out the steps to go from a fastq file to a final QCed VariantSet.

@haussler, here are my thoughts on how to represent the more sophisticated analysis you describe above:

yes, I think the best way to represent that analysis is to keep the 4 AnalysisSample and 4 ReadGroupSet objects, and to add a handful of new CallSet objects, each with metadata describing how it was derived

in 0.5.1 you'd probably set the sampleId text field to "tumor" for those new objects
in the world of this PR, you'd use the formal analysisSampleId field instead

as you surmise, you can represent the "vitally important ... information on how to interpret the call set" -- it's done using the info field on the CallSet object, which is intended to hold information on how that particular call set was derived
possible future enhancements:

we could make the new provenance fields arrays instead of singletons, to let you formally say "this call set was derived from these N analysis samples and/or read group sets"
we could add more structured call set metadata to formally represent these types of mixed-sample analyses

A problem with tagging individual reads is that most are ambiguous as to
which clone they come from, and many also could have come from either
normal or cancer cells. For example, there are some reads that could have
come from normal cells or any of the 5 clones. There are some that could
have come from clones 2, 3 and 5, but no other clones and not from normal
cells, etc. You could tag each one with a subset I suppose, but this gets
complex...
On Sun, Aug 30, 2015 at 7:50 AM, jjfarrell<EMAIL_ADDRESS>wrote:

@lh3 https://github.com/lh3 @haussler https://github.com/haussler
Another approach is to tag each read in the ChildTumor ReadSet with the
appropriate genome eg: (read.info['genome']="clone1" . CallSets would
then need to be based on a query that are a subset of the ReadSet:
something like TumorReadSet.filter("info[genome]="clone1") . Besides this
cancer example, ReadSet filtering is needed based on MQ, RD, Genome Mask
and other QC metrics. The resulting CallSet would need to store the query
or search request. Using SQL for these type of queries would be much more
flexible than the present API which is very limited.
When the CallSet is merged into a VariantSet, the AnalysisSample.name
could include the CallSetName as a prefix (clone1.ChildTumor).
This example and the GATK best practice pipeline would be useful exercises
to sketch out the steps to go from a fastq file to a final QCed VariantSet.
—
Reply to this email directly or view it on GitHub
https://github.com/ga4gh/schemas/pull/391#issuecomment-136151560.

I'm closing this PR and deleting the branch -- the conversation has moved on. Summarizing the relevant code changes for posterity:

add analysisSampleId to ReadGroup (in reads.avdl)
add analysisSampleId and readGroupSetId to ReadGroupSet (in reads.avdl)
add analysisSampleId and readGroupSetId to CallSet (in variansts.avdl)
define a new AnalysisSample object (in metadata.avdl):

/**
An `AnalysisSample` represents prepared tissue intended to be sequenced and analyzed together,
eventually generating one or more `ReadGroupSet` and/or `CallSet` objects.
TODO: agree on a formal metadata model to describe where the tissue came from, including
ways to represent the source organism, sample collection, and sample prep.
*/
record AnalysisSample {

  /** The analysis sample ID. */
  string id;

  /** The analysis sample. */
  union { null, string } name = null;

  /** The ID of the dataset this analysis sample belongs to. */
  string datasetId;

  /** When this analysis sample object was created in milliseconds from the epoch. */
  union { null, long } created = null;

  /** When this analysis sample object was last updated in milliseconds from the epoch. */
  union { null, long } updated = null;

  /** A map of additional analysis sample information. */
  map<array<string>> info = {};
}

@dglazer Just to document: These code changes (adding of AnalysisSample) were never implemented.

Thanks for clarifying @mbaudis -- correct; the opening comment said "this is not ready to commit; it's meant to drive discussion"  -- it did drive discussion; it was not committed.

We need to have a full metadata model integrated.  Right now, it
isn't even possible to query for variants associated with a
disease.
Michael Baudis<EMAIL_ADDRESS>writes:

@dglazer Just to document: These code changes (adding of AnalysisSample) were
never implemented.
—
Reply to this email directly or view it on GitHub.*

"metadata" covers a lot of topics, including:

provenance of sequence-based objects (as was discussed here)
tracking of pre-sequencing material (samples, subjects, batches, etc.)
knowledgebases of the attributes of particular genomic regions and/or variants

Those are all worthy, and can be tackled separately, ideally coupled to shovel-ready projects.

I am increasingly reluctant to even use the term metadata', as often it depends on what one is doing if data. Are clinical outcomes are data' or `metadata'?
All data objects, including `metadata', need provenance.
Changing a metadata attribute can change the interpretation of
the data.
Derived data, such as genomic annotations, is not generally
considered `metadata'; although it has metadata.
David Glazer<EMAIL_ADDRESS>writes:

"metadata" covers a lot of topics, including:
• provenance of sequence-based objects (as was discussed here)
• tracking of pre-sequencing material (samples, subjects, batches, etc.)
• knowledgebases of the attributes of particular genomic regions and/or
variants
Those are all worthy, and can be tackled separately, ideally coupled to
shovel-ready projects.
—
Reply to this email directly or view it on GitHub.*

Metadata as used by the GA4GH.DWG.MTT

A core set of both experimental and sample related attributes, with emphasis on both consistent (i.e. use of internationally recognised standards) as well as flexible (e.g. integration of ontologies) value definitions. The (developing) core metadata schema does not intend to model complex object relations. Particularly rich as well as use case dependent data structures (e.g. electronic health records) are not part of "GA4GH Metadata", but should be connected to using unambiguous reference structures.

This is no quote, but IMO / rough current consensus of active voices in metadata.
And yes, I prefer "metadata" for all the objects & attributes not directly reporting molecular read-out data defined the GA4GH schema. Happy for a sane alternative; no luck so far.
There seems to be an upcoming MTT note at the next DWG call.

@diekhans @dglazer ... and yes, so far we haven't considered genomic annotations as part of "metadata as a topic". The cut is basically at the "variantCall" level (whatever that describes; could be methylation etc., too). However, we are considering general attributes (e.g. time formats, ID varieties) as something being defined in metadata; see https://github.com/ga4gh/schemas/blob/metadata/doc/source/metadata_intro.rst .


Feature request: Removing slide by #id

Hi! Can you implement this feature?
I use slick with image manager on my project and I find it difficult to control indexes to remove slides

@kenwheeler could you help remove a slick by class name with up to date slick syntax please? tried this $('.js-banner').slick('slickRemove',$('.js-banner')[0].indexOf('.removeSlick')); but get an error of:
indexOf is not a function error


⏱ Uptime rewards and phase events

This PR add detection of phases event like when a new phase start or end. Also by detecting if a phase is finish, a new mongo query has been added to calculate the uptime point based on the task description.
📥 Events
I call this event but not event has been created, it's more like a detection of event on each new block. If a block is the first block of a phase, this trigger a function to do some action and the same for the last block of a phase.
No action has been configured on first phase block but a new action has been set on the end of a phase : calculate uptime.
⏱ Uptime
The calcul used for uptime calculation is here :
2501^0.01x - 1 with x = uptime percentage. That's give in mongo :
{
   "uptime":{
      "$subtract":[
         {
            "$pow":[
               2501,
               {
                  "$multiply":[
                     0.01,
                     {
                        "$subtract":[
                           100,
                           {
                              "$divide":[
                                 {
                                    "$multiply":[
                                       100,
                                       "$totalMissedBlock"
                                    ]
                                 },
                                 {
                                    "$subtract":[
                                       "$phase.blocks.to",
                                       "$phase.blocks.from"
                                    ]
                                 }
                              ]
                           }
                        ]
                     }
                  ]
               }
            ]
         },
         1
      ]
   }
}

The mongo aggregation will before calulate uptime, detecting all missedBlocks in the given phase and cut missedBlocks range if a missed block range is between two phases.
https://github.com/okp4/nemeton-leaderboard/blob/d87dac24d2c763ec630b289fd08a7aa610881707/app/nemeton/store.go#L479-L508

@amimart Since reward point is in Long format, I've fixed mongo aggregation to make an absolute round on points calculation when setting points on validator collection. I've decide to cut the decimal number since a validator with 2499,6 points cannot be rounded to 2500 points because he missed some block.
So I've used { "$toLong": "$$newPoints" } to convert 2499,6 to 2499 points. What do you think about that ? Or maybe do we need to made a correct rounding?

:tada: This PR is included in version 2.0.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


transformer is needed to create catalog from spread sheet

Issue description / feature objectives
Similar to the task xlsx-to-oscal-component-definition, a transformer is need to create a catalog from a spread sheet.
Caveats / Assumptions
Completion Criteria
New task xlsx-to-oscal-catalog is implemented.

See issue #1238.


Camunda 7.15.0 instructions

This no longer works on Camunda 7.15.0
$ git clone https://github.com/datakurre/camunda-cockpit-plugins.git
$ docker run --rm -p 8080:8080 -v $(pwd)/camunda-cockpit-plugins:/camunda/webapps/camunda/app/cockpit/scripts/:ro camunda/camunda-bpm-platform:latest

because it would override the default camunda-cockpit-ui.js located now at scripts.
I welcome the second best option to try out these plugins on 7.15.0.
Options:
a) Add Dockerfile and instruct building a custom image
b) Instruct to start the default image, copy camunda-cockpit-ui.js from it and then use the old instructions
c) Instruct to mount every file with separate mount to prevent overriding camunda-cockpit-ui.js

or copy camunda-cockpit-ui.js to this project as past of setup :)


[ie/afl] added support for Australian Football League (AFL) websites

This PR contains extractors for the following:

AFL Video
AFL Podcast
AFC Video
Brisbane Lions Video
Carlton FC Video
Carlton FC Podcast
OmnyFM Show: example

Note:

OmnyFM is added in this PR as AFL/Carlton FC podcast are OmnyFM show embeds.
AFC radioshow is simply a Listenr embed. It can be added separately in a different PR.

Partially Fixes #10909
AFC radioshow support has not been added
Template

Before submitting a pull request make sure you have:

[x] At least skimmed through contributing guidelines including yt-dlp coding conventions
[x] Searched the bugtracker for similar pull requests

In order to be accepted and merged into yt-dlp each piece of code must be in public domain or released under Unlicense. Check all of the following options that apply:

[x] I am the original author of this code and I am willing to release it under Unlicense
[ ] I am not the original author of this code but it is in public domain or released under Unlicense (provide reliable evidence)

What is the purpose of your pull request?

[ ] Fix or improvement to an extractor (Make sure to add/update tests)
[x] New extractor (Piracy websites will not be accepted)
[ ] Core bug fix/improvement
[ ] New feature (It is strongly recommended to open an issue first)

Your git config appears to be wrong. Open https://patch-diff.githubusercontent.com/raw/yt-dlp/yt-dlp/pull/11035.patch and you will see From: subrat-lima <subrat.k.lima<EMAIL_ADDRESS>. You may need to correct it for future use. See Setting your commit email address - GitHub Docs.
After correcting your email, please do NOT update those existing commits (i.e., force-push), otherwise we would lose all review suggestions.


some problems about global context and cls_token

Hi @ma-xu
[gmp_list.append(F.adaptive_max_pool1d(self.gmp_map_listi, 1))](https://github.com/ma-xu/pointMLP-pytorch/blob/b5dcf4d0ca2085d10fe1dc5e6f2972cddec8399d/part_segmentation/model/pointMLP.py#L437)
https://github.com/ma-xu/pointMLP-pytorch/blob/b5dcf4d0ca2085d10fe1dc5e6f2972cddec8399d/part_segmentation/model/pointMLP.py#L441
The features of each layer of the encoder are concat. After max pooling,the feature is concat with the last layer of the decoder.
Why not concat with each layer of the decoder?
Thanks.

@mmiku1 It could be, but unnecessary.  Empirically, this setting can achieve promising performance.

Thank you for your answer. Wish you a happy life！


Changes V3.1 - split preamble, notice, Mappings (Value-Only, Metadata, Path)

index and preamble split + editorial changes + add notice chapter etc.
ValueOnly and other changes

@j-blume : may you please review this (small) change? Thank you

@mjacoby may you please review with respect to idShortPath and ValueOnly

Adapt title of PR, because Value Only, PathType and MetaData Mappings are discussed

Value-Only will be discussed in further PR https://github.com/admin-shell-io/aas-specs/pull/375

findings were resolved, branch will be merged so that also the other can review it in html.

Adapt title of PR, because Value Only, PathType and MetaData Mappings are discussed

done


Updating all the themes' Window.Code and Console.Code

Yall lasy.
Special thanks to:
Aero for having an updated Window.Code
Windows XP theme for having an updated Window.Code

sussy unity, are you sure you didn't break it? i modified window.code in unity to makke thee title button and other stuff work, PLZ don't make sussy commit

sussy unity, are you sure you didn't break it? i modified window.code in unity to makke thee title button and other stuff work, PLZ don't make sussy commit

I went ahead and manually added in the pieces of code from all the previous commits making sure that those lines weren't modified. I didn't do a degenerate move and copy-paste the entire window.Code like a madman.


[stable/falco] Enable eBPF support for Sysdig Falco helm chart

What this PR does / why we need it:
This PR enables eBPF support for Sysdig Falco helm chart.
Sysdig now supports the ability to run the fully featured system event capture engine using eBPF as instrumentation backend as opposed to the traditional kernel module.
Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #
Special notes for your reviewer:

/assign @bencer

/lgtm
/approve


Automate deploy to pypi

Set up travis to deploy the package to pypi. See the OKI examples for how.

The issue here we have is that I own dpm on PyPI but i've lost access to pypi! Since they are unresponsive to the support requests I've opened I think only option will be to publish the package as dpmpy for now or similar.


Implement bass clef transposition

Transpose two (maybe 3?) of the voices to use bass clef instead of treble clef ("C" = MIDI note 60).
Transposing the percussion voice (1), will make it easier to write drum parts without having to add a bunch of , octave lowering suffixes to everything.
Transposing one or two of the other voices will make it easier to write parts for bass instruments. I'm not sure if it makes sense to have two dedicated bass voices. Using two would allow for one voice to have long-running drones or pads while the second does chords or whatever on top. The difficulty is that, currently, chord notation expects all the notes of the chord to start and end at the same time.

done


WIP: version is now stored on a single file, VERSION.txt

Overview
This PR makes pycsw store its version on a single location, the file VERSION.txt.
This change is more profound than it seems at first sight. In order to be able to have a single source of truth for the version, I had to replace distutils with setuptools - This was necessary because retrieving the installed version at runtime is done using setuptools' pkg_resources. Taking advantage of setuptools, I've refactored the setup.py file, removing code that was duplicating the job of setuptools.
However, I do not foresee a big problem with this change (lets hope the reviewers agree), as setuptools has become the standard way to install python packages and is even included by default with python 2.7.9 and python 3.4+
I'm leaving this as a WIP until PR #517 is merged because the README has changed extension and I'll need to update the setup.py code then.
Related Issue / Discussion
See #516
Additional Information
Contributions and Licensing
(as per https://github.com/geopython/pycsw/blob/master/CONTRIBUTING.rst#contributions-and-licensing)

[x] I'd like to contribute [feature X|bugfix Y|docs|something else] to pycsw. I confirm that my contributions to pycsw will be compatible with the pycsw license guidelines at the time of contribution.
[x] I have already previously agreed to the pycsw Contributions and Licensing Guidelines

It still is WIP because I'm wiating to merge #517, but please feel free to review

@tomkralidis I've update the code with your proposed changes.

Thanks @ricardogsilva!

@tomkralidis this wasn't intended for merging yet (hence the WIP) ;)
I was hoping to have #517 merged in first because the extension of the README has changed in that PR. I'd have to adjust it here afterwards.

Oops sorry @ricardogsilva. I was too excited with these changes! Note to self to read PR titles next time :)


Make the discovered Drupal extensions accessible via a service

We need to make the discovered extensions available from a service, just like ServiceMap.
This would help https://github.com/mglaman/phpstan-drupal/issues/247 and also improve the existing load include return type extensions.

Should we enhance the existing ExtensionDiscovery and  Extension or should we create another service and/or another value object using them? 🤔

Fixed via https://github.com/mglaman/phpstan-drupal/pull/264


PR #290

it's been 3 months since we submitted our PR #290... please update ASAP :)
Add Punk TSO

submitting on both Flare and Songbird - update ASAP please :D


add docker setup for 5.4 and head/main

motivation: CI for 5.4 and latest
changes: add docker setup files for 5.4 and main nightly images

@swift-server-bot test this please


Update Pangolin and Freyja

Closes https://github.com/UPHL-BioNGS/Cecret/issues/357

Need to reduce size of images. By a lot


Added documentation to example and a refocus button

Cleaned up version of https://github.com/villor/bevy_lookup_curve/pull/12.

Perfect! Good job on the commit cleanup.
Merged.


WIP: remove numpy as depedency

WIP to remove numpy as a dep.
The blocker is that we use it in our default embedding model https://github.com/chroma-core/chroma/blob/main/chromadb/utils/embedding_functions.py#L248. We like making chroma "batteries-included" - so we don't want to force an install if you try to use it. It should JustWork:tm:
We could replace it with pure python... not sure what the speed tradeoff is yet (havent profiled)

leaving it in

only land the typefixes (just abstract class), not removing numpy


Add support for basic auth

When trying to use this check to monitor a small Ganeti cluster based on Debian Jessie, I noticed that the RAPI defaults to require basic authentication. This pull request adds additional, optional parameters to the check script which allow to pass username, password and an optional realm for basic authentication in the RAPI calls.

LGTM :+1:
@waldiTM @zobelhelas as I don't have merge perms anymore, please review and merge :)


Port to rcm?

rcm is a dotfile management system (https://github.com/thoughtbot/rcm) by Thoughbot that was recently open sourced. A lot of dotfiles projects (janus, oh-my-zsh, etc.) use their own installation scripts and it makes it harder to have unified a way to install/manage dotfiles. I'm currently writing my own rcm based dotfiles folder and I can't figure out a clean way to incorporate oh-my-zsh into my set of dotiles.
Anyways, just some food for thought!

I just found out about antigen, which solves my problem more or less.


[catloop] Always Bound Read from Control Duplex Pipe

This PR removes the ability to ever ready from the control duplex pipe on the accept with anything other than the size of MAGIC_CONNECT. Because each MAGIC_CONNECT represents a separate connection request, the code will crash if we ever read more than one at a time. Thus, this PR closes #777 by simply removing the ability to read anything from the control duplex pipe that is not a single MAGIC_CONNECT value.

We did. I just made it not optional.


Fixed how class names are identified with css modules to support themes properly

Summary

Fixed how class names are identified with css modules to support themes properly
Fixes #3095

Lets get the tests updated and it would also be nice if the documentation example page supported the lowlight theme


Add unsubscribe link

Currently, only Node.js library supports unsubscribe link in the email: https://resend.com/docs/knowledge-base/how-do-i-add-an-unsubscribe-link-to-my-emails.
Do you plan on adding such functionality to the python library?

Oh, I found it on another page that you can use headers in python code:

Would be neat to have this in python example.

good catch @ovyan , ill introduce that into the project examples 👍


Please add Disk Browser Focus to API?

Is it true that the Renoise API 5 does not afford us control over whether Disk Browser is focused or not?
I see renoise.app().window.disk_browser_is_visible, _observable but, while having this is good, I would like to make the disk browser visible and focus the Keyboard to the Disk Browser.
P.S. Would be good to be able to enter Extended Disk Browser mode, too.
@connerbw @snabeljoel @emuell

would still be pretty cool to have :)

hi, still here, still a good feature to have @bjorn-nesby


Fixed TDSParser stuck on TDS_COLMETADATA issue

Actually read COLMETADATA.

hi @mmimica
Thank you for your contribution. Can you please add test code to repro and test the issue that this is fixing?

How does the bug manifest?
This WARN log message confirms the fix works. The TDS_COLMETADATA is read and discarded.


Use custom templates

Hi all,
It's possible to use custom templates without modifying the included with the packages? Like pointing to a path with a setting or similar.
If it's not possible, it would be appreciated create a setting to do this.
Thank you!

Since this project is based on Django, in principle it should be possible to replace all of the built-in templates with your own custom templates. You just need to drop your templates into their own folder in your Django project directory, and tweak your settings to search your custom files before the built-in ones. You can read https://docs.djangoproject.com/en/1.11/topics/templates/ for more details.
You would need to make sure your templates include all of the same template tags and such as the originals, or they may not render correctly. But otherwise, you should be able to customize as much as you want!

Thanks!! Changing the Django configuration properly. I've managed to use my custom templates, and it did the trick!


Primitives Tab & "New" button functionality

The New Matrix Eater includes the ViewportActivity interface, allowing new model-space manipulations to be added with ease. This has been about a year in development, but now that it's nearly finished, it should be fairly trivial to add a tab to the Matrix Eater for drawing primitive shapes, as long as I can work out the basic high school algebra of how to draw the primitives.
Once this is implemented, there should be a way to make a "new" model, and to add primitives, and begin the process of authoring a model.

This functionality is now present in Retera Model Studio. Let's open a new issue if we want more features developed for this.


what are you doing

i can't understanded

come form bilibili

come from bilibili too

LOL
Most extraordinary job I've seen this year


ADVZ constructor take erasure_code_rate instead of payload_chunk_size

Blocking #393 ?
Current constructor:
https://github.com/EspressoSystems/jellyfish/blob/8d071e2df74e5093d2b3c35c32773b09370c2452/primitives/src/vid/advz.rs#L98-L102
payload_chunk_size = degree of polynomials. One could argue this is an implementation detail that should not be exposed to the user. Indeed, in the future we might wish to keep control over this argument for ourselves. Example: the optimization of #393 seems to require us to control this argument.
Proposal
Replace payload_chunk_size with erasure_code_rate. How best to specify it? Options:

usize: set to n means the rate is 1/n. (So if n is 3 then num_storage_nodes is 3x polynomial degree.) This captures most common choices but it does limit use to rates of the form 1/n
arbitrary rational number: captures all plausible choices but adds complexity. eg. How to specify the rational? Do we take 2 args (numerator & denominator)? Use an external crate such as num_rational - Rust? (ick.) This is probably overkill. I suspect restricting to 1/n rates is fine.

Let us sync with consensus before we move forward with this task.

Change of plans.
tldr;
We will not take a erasure_code_rate arg. Instead we'll make only a minor tweak.
New proposal
Offer two constructors:

new simple/generic: any VID scheme should take num_storage_nodes and num_shares_needed_for_recovery (I suggest we call it recovery_threshold). We choose multiplicity for you.
with_multiplicity complex/implementation-specific: allow caller to specify multiplicity.

Side request from @mrain : remove the existing limit on multiplicity to better facilitate GPU.
Explanation
Currently payload_chunk_size must be a power of 2 (until we get KZG in eval form). The original proposal is to compute payload_chunk_size = num_storage_nodes / erasure_code_rate, which won't always give a power of 2. The easiest solution is to round that quotient to a power of 2 until this restriction is lifted in the future. But there's a problem: the user won't (easily) know how many shares are needed for recovery!
This problem highlights the difficulty with a constructor based on erasure_code_rate. Starting to think VID scheme should take num_storage_nodes, num_shares_needed_for_recovery (aka payload_chunk_size, recovery_threshold). No objections in zulip discussion.


BD-1196 Add credit in Bundlegen

BD-1196 - Add credit in Bundlegen

Blackduck scan failure details
Summary: 0 violations, 0 files pending approval, 1 file pending identification.

Protex Server Path: /home/blackduck/github/BundleGen/93/rdkcentral/BundleGen

Commit: 5a7b2fcc26054da47035c93091427a472343618b

Report detail: gist

WARNING: A Blackduck scan failure has been waived
A prior failure has been upvoted

Upvote reason: Thanks Tom

Commit: 5a7b2fcc26054da47035c93091427a472343618b


refactor: Update signup validator to use a more robust one from community

This PR updates the sign up validators to be more robust like PR #86. It removes the old custom validator that uses simple regex.

@nikiburggraf This is like the other PR but for sign up page. 😄

@nikiburggraf I've applied your suggested changes. There's a change that I've punted to a different PR, and your comment on styling, I've created a new issue to tackle that 😄 Thanks!