Decide API for equilibrium expectation datasets

We need to settle how to handle non-ThermoML datasets within the PropertyCalculator framework, most immediately for calculation of equilibrium expectation values of other properties.
As noted here ( https://github.com/open-forcefield-group/open-forcefield-tools/pull/3#issuecomment-227601790 ) the current draft API (https://github.com/jchodera/open-forcefield-tools/blob/master/README.md) being brought in via #3 envisions PhysicalPropertyMeasurement objects compiled into a PhysicalPropertyDataset. One specific example is given for ThermoMLDataset as a derivative of a PhysicalPropertyDataset, and that's certainly one of the key datasets we need in the short term.
One of the other things we plan in the short term is to run equilibrium MD simulations of the AlkEthOH set in the gas phase using parm@frosst parameters and compute equilibrium expectations - energies (total as well as possibly divided into components from individual bonds, angles, and torsions) and bond, angle, and torsional distributions. We would then use this (or subsets thereof) as a dataset for parameterization to see if we can recover the parameter set. So, we need to decide how this would be framed in terms of a dataset - either a PhysicalPropertyDataset or something else.
It seems like the description of the physical system we're interested in could fit within the existing framework of a PhysicalPropertyMeasurement, i.e.:

A Mixture specifying the substance that the measurement was performed on: Here, this would be a single component mixture
A ThermodynamicState specifying the thermodynamic conditions under which the measurement was performed: Here this would specify gas phase and the particular temperature.
A PhysicalProperty is the physical property that was measured: Here we would have multiple such measurements and the only significant change would arise here. Examples below for my case.
A MeasurementMethod specifying the kind of measurement that was performed: Here this would be EquilibriumMDSimulation as the source of our equilibrium expectations.

As noted elsewhere, each PhysicalPropertyMeasurement has several properties:

.substance - the Mixture for which the measurement was made
.thermodynamic_state - the ThermodynamicState
.value - the unit-bearing measurement value (it is not explicitly stated in the API docs, but I assume this would be, for example, a PhysicalProperty such asExcessMolarEnthalpy`, @jchodera ?)
.uncertainty - the standard uncertainty of the measurement
.reference - the literature reference (if present) for the measurement
.DOI - the literature reference DOI (if available) for the measurement

If we were to extend the API to allow equilibrium expectations from the envisioned gas phase MD simulations, our crucial need would be for suitable PhysicalProperty types. Probably we would need something like:

GasPhaseSingleMoleculeAverageEnergy
GasPhaseSingleMoleculeBondDistance
GasPhaseSingleMoleculeBondAngle
GasPhaseSingleMoleculeTorsionAngle

These would look for a definition of which molecule to be examined in the Mixture (which must, for these, be a single-component since the data will be for that single component in the gas phase). I think that for GasPhaseSingleMoleculeAverageEnergy, no additional info is needed beyond what's already provided by PhysicalPropertyMeasurement. However, for the bonded properties we would need some additional info:

.smarts: SMARTS pattern uniquely identifying (within the specified molecule) which bond, angle, or torsion is being examined

Any other needed properties?
A usage example would be:
dataset = GasPhaseSingleMoleculeAverageEnergy( '/Users/dmobley/phenol_gasphase.cdf4')
dataset += GasPhaseSingleMoleculeBondDistance('/Users/dmobley/phenol_gasphase.cdf4', '[#5]~[#8]') #Analyze C-O bond length
# Compute physical properties for these measurements
estimator = PropertyEstimator(nworkers=1)
computed_properties = estimator.computeProperties(dataset, parameters)
# Write out statistics about errors in computed properties
for (computed, measured) in (computed_properties, dataset):
   property_unit = measured.value.unit
   print('%24s : parm@frosst value %8.3f (%.3f) | calculated new value %8.3f (%.3f) %s' % (measured.value / property_unit, measured.uncertainty / property_unit, computed.value / property_unit, computed.uncertainty / property_unit, str(property_unit))

I'm ambivalent at this point whether GasPhaseSingleMoleculeAverageEnergy and relatives would actually analyze stored trajectory data (as I have it currently written above) or whether these would read results of analysis done separately via another tool. Input, @jchodera or others?
In all likelihood I've missed some key aspects, but does that look feasible?

@mrshirts , do you want to weigh in on this one at all?

A PhysicalProperty is the physical property that was measured: Here we would have multiple such measurements

Couldn't this hold for datasets as well?  There could be multiple measurements at the same conditions.

Couldn't this hold for datasets as well? There could be multiple measurements at the same conditions.

There is already the infrastructure in place in the API for i.e. thermoML to specify multiple measurements, which I think is what you're asking about.

Can't we just have:
ThermodynamicState = gasPhase
PhysicalProperty = SingleMoleculeAverageEnergy
= MoleculeBondDistance
MeasurementMethod = EquilibriumMD
Do we need to have more complicated PhysicalProperty objects like:
GasPhaseSingleMoleculeAverageEnergy

@mrshirts - that may work. I was having trouble convincing myself that "gas phase" really was something that belonged in the ThermodynamicState; of course it does at some level, but for condensed phase properties we'll want to specify a meaningful pressure and temperature there. I guess we could make P=0 trigger "special case" gas phase behavior or something. Is that what you're thinking? Certainly, compared to most properties we'll be interested in, gas phase is a special case, so a key question is how to best recognize that...

I think that "phase" can be an additional information beyond just T and P.  For example, in crystal polymorphs, you can have different polymorphs that are metastable at the same T and P. So it's not weird to make the phase a different quantity.   Or for polymers, you can have different phases (gyroid, lamellar, etc).
Ideal gas could be a separate phase -- that's a imaginary phase that is approached by real systems when T->inf and P->0, but it might make sense to call it something separate.  QM calculations are pretty good in ideal gas for heat capacities. . .

OK, @mrshirts . Can you then propose (ideally via a pull request to modify the README.md) what needs to be in the API draft in the README.md here: https://github.com/open-forcefield-group/open-forcefield-tools to handle ThermodynamicState info of this sort? i.e., how specifically would we store this type of info in ThermodynamicState? John is planning to implement tomorrow and this weekend, so if we need it in there we have to codify how exactly.
Right now his example of ThermodynamicState is "298 Kelvin, 1 atmosphere", so we're saying ThermodynamicState contains a temperature and pressure. If we want to generalize it to the cases we mentioned (to specify gas phase, or potentially to specify a particular polymorph down the road) what additional information does it need to have and how exactly should it be stored?
I think what you're proposing right now is that ThermodynamicState just have an additional property aside from .temperature and .pressure which would be .phase which would be a ____ (?? text string?) describing the phase of the system. Though, to accommodate the gas phase MD simulation data we'd be using we'd also need to specify that if I say I want a .phase='gas' simulation of methane, I don't mean methane GAS, I mean a single molecule of methane in the gas phase. How does that fit in to your idea of ThermodynamicState? How would we say we want only one molecule?
The reason I proposed the SingleMoleculeGasPhase... stuff was that I thought it involves two special cases at once:

we want to simulate only a single molecule
we want to simulate in the gas phase
and neither of those fits neatly into our definition of what we "normally" want to simulate here (typically a liquid mixture), so the combination of the two is sort of a "very special" case which I thought meant it needed to be carefully broken out for special treatment. Treating them as a special type of PhysicalPropertyMeasurement allows us to avoid having to introduce a lot of new concepts into the definition of a ThermodynamicState and just pick these up because we run into a special case type of measurement.

To put it another way - I'm all for making ThermodynamicState more general when it's going to help us cover more ground and use cases down the line, but I'm not that enthusiastic about loading up more information there only to help us handle weird special cases which are not representative of general use cases. So I'm trying to sort out which territory we're in here - will putting more info in ThermodynamicState for THIS problem help us later, or is it just loading up info we aren't going to be using?

Also, on this part, @mrshirts :

Ideal gas could be a separate phase -- that's a imaginary phase that is approached by real systems when T->inf and P->0, but it might make sense to call it something separate. QM calculations are pretty good in ideal gas for heat capacities. . .

Just to clarify again - we're not here talking about gas phase, we're talking about a simulation of a single molecule of methane, or cyclohexane, or whatever, in the gas phase. I agree that if I were talking about gas phase simulations, it might make sense to include this info in the ThermodynamicState (and indeed, we should expand ThermdynamicState to include those when/if we get to where we want to simulate such systems) but right now I'm talking about simulating single molecules in the gas phase, which is a weird corner case in my view.

Let's not change ThermodynamicState to include the concept of gas phase.
One has really no bearing on the other.
We want to replace Mixture (a container for specifying a liquid mixture
system) with something that has a concept of a single molecule in gas
phase. Can you come up with something like that?

Maybe IsolatedMolecule, since even IdealGas implies some finite (but
noninteracting) concentration ?

@jchodera

We want to replace Mixture (a container for specifying a liquid mixture
system) with something that has a concept of a single molecule in gas
phase. Can you come up with something like that?

Ah, that would work well. So a MeasuredPhysicalProperty would take a Composition' which could be a Mixturespecifying a liquid (pure or mixed) or other types to be determined later, such as anIdealGas(later) or anIsolatedMolecule` (now). Seems perfect.

(note I don't think we want to use IdealGas anywhere, since that's a theoretical construction. I updated my comment above to reflect this.)

So, then, we'd be able to do as I suggested above but generalize so that we'd have something like:

AverageEnergy
BondDistance
BondAngle
TorsionAngle

as properties we would want to calculate for single molecules (and probably later, for molecules in mixtures, etc., so now we kill two birds with one stone).
These would look for a definition of which molecule to be examined in the Composition or Substance. For the bonded properties we would need some additional info, even for the IsolatedMolecule case, specifically:

.smarts: SMARTS pattern uniquely identifying (within the specified molecule) which bond, angle, or torsion is being examined

More generally for a Mixture we would probably want the property calculator just to return a measurement for EACH molecule matching the specified smirks, separately (i.e. if I specify a C-C bond distance and I'm looking at a mixture of cyclohexane and ethanol, I want to be able to look at the two distributions separately).
@jchodera - if this sounds reasonable to you I can put together a PR to update the README.md accordingly. This is a much better solution than what I'd come up with.

Let's not change ThermodynamicState to include the concept of gas phase. One has really no bearing on the other.

Well, I'd argue the thermodynamics state includes the phase (see polymorphs comment above)  Whether the ThermodynamicState does may be a different question.

since even IdealGas implies some finite (but noninteracting) concentration ?

The per-molecule residual properties would be identically the same (i.e. properties minus the ideal gas term), so if T and P are specified, the properties could be mapped one-to-one with no error.  So 1 molecule of an ideal gas could be identified with an isolated molecule; though it's not.  It is a theoretical construction, but a pretty well-defined one.
Would be relatively easy to fix up later if needed, and operationally, wouldn't really be difference either way.  So I don't really care that much on that in the end :)
A gas can be a mixture just as easily well as a liquid, so maybe something like just  instead of .  You can tell if something is pure if it's has a single component.
So we could have , , , and potentially .
Though again, solids (both metals and polymers) do include polymorphs that have different properties, and it would probably be useful to think about how that might fit in somewhere (even if that is down the road).

Well, I'd argue the thermodynamics state includes the phase (see
polymorphs comment above) Whether the ThermodynamicState does may be a
different question.

The probability that a given phase or mixture of phases exists at
equilibrium is certainly controlled by the thermodynamic state, but
@davidlmobley is looking for a way to encode the information "this
measurement was made for an isolated molecule in vacuum". That should be
the job of the container object specifying the system composition.
We had previously defined a Mixture (to which we presumably intend to
attach utility functions that generated initial conditions for a mixture in
the liquid state), so it seems appropriate to define a different container
that means "this measurement was made for an isolated molecule in vacuum".
Alternatively, we can separate the container object from an object that
prepares initial conditions if you want to use the Mixture container along
with a LiquidInitialConditions and IsolatedMoleculeInitialConditions, but
this can get weird because not all Mixture objects would be supported by
all initial conditions.

The per-molecule residual properties would be identically the same (i.e.
properties minus the ideal gas term), so if T and P are specified, the
properties could be mapped one-to-one with no error. So 1 molecule of an
ideal gas could be identified with an isolated molecule; though it's not.
It is a theoretical construction, but a pretty well-defined one.

Good point, but I am not sure we will be dealing with ideal gas mixture
measurements, and I am not sure we gain anything by building in the stat
mech of ideal gases when our current use case doesn't deal with that.

Would be relatively easy to fix up later if needed, and operationally,
wouldn't really be difference either way. So I don't really care that much
on that in the end :)

OK

Though again, solids (both metals and polymers) do include polymorphs
that have different properties, and it would probably be useful to think
about how that might fit in somewhere (even if that is down the road).

Are you mostly worried about solid crystals or amorphous solids?
In the current scheme, we would either keep these as Mixture objects
(though for crystals we would need precise stoichiometries, or maybe
chemical potentials? if simulating the crystal growth process) so would not
want to use mole fractions; or we could define a different
AmorphousSolidMixture or Crystal or something. This does conflate phase
with composition, but given how hard it is to a sample across phase
boundaries, that may not be a big issue.
The more important consideration is this: Given that we wanted to make it
as easy as possible to use ThermoML data as input, it is most convenient if
there is a one-to-one correspondence between ThermoML information and
Python objects. @davidlmobley is already trying to break this by using data
sources not in ThermoML, and we are trying to accommodate that without
destroying the whole design. Is there a better way to do this that will
keep the code very simple conceptually and technically?
For example, could @davidlmobley generate synthetic ThermoML files where we
simply extend the set of measurement types in a consistent way? That would
have the least impact on the design.

One additional thing I thought of; above a critical point, there not really a dividing line between Gas and Liquid.  This is generally handled by calling it a fluid -- mostly, there is only one experimentally accessible system at a given T and P that is a fluid (superheated or supercooled fluids are probably not going to be parameterized).
At this point, I'm just thinking through concepts to see what makes logical sense or not. This leads me back to the add an attribute .phase, which could be isolated molecule, liquid, gas, lamellar, etc. But there are drawbacks as well.

In the current scheme, we would either keep these as Mixture objects
(though for crystals we would need precise stoichiometries, or maybe
chemical potentials? if simulating the crystal growth process) so would not
want to use mole fractions;

Stochiometries won't work, since there can be multiple phases with the same composition (especially for pure, but can happen for mixtures).

Stochiometries won't work, since there can be multiple phases with the same composition (especially for pure, but can happen for mixtures).

I just meant for a crystal, we really do need to simulate the correct number of molecules in the unit cell; we can't be off by a molecule due to a rounding error or else we won't get correct crystal packing.

@jchodera :

The more important consideration is this: Given that we wanted to make it
as easy as possible to use ThermoML data as input, it is most convenient if
there is a one-to-one correspondence between ThermoML information and
Python objects. @davidlmobley is already trying to break this by using data
sources not in ThermoML, and we are trying to accommodate that without
destroying the whole design. Is there a better way to do this that will
keep the code very simple conceptually and technically?

Presumably down the line we will be interested in lots of other data not in ThermoML, such as log D data, other measurements we might make directly ourselves, and Gilson will probably want to try including host-guest binding data (presumably we wouldn't have that as part of our main effort but I'm imagining he's going to want to try it in his group anyway to see what happens). So, I'm all for a clear correspondence where each type of ThermoML information corresponds to a specific Python object, but I think we have to be clear at this point that we're going to have Python objects which DON'T correspond to ThermoML information. Otherwise we're effectively putting ourselves in a world where we say, "well, you can parameterize using any type of data you want... but it has to be data in ThermoML", which is not going to fly well if we are trying to convince others (such as the pharma industry) that this is a framework they can pick up and use however they want, potentially even using their own internal data in new parameterization efforts.
So, to reiterate: each type of data in ThermoML should correspond to a Python object, but not all of our data types will correspond to types of data in ThermoML. This has to be a design consideration here or we won't have nearly enough flexibility.

For example, could @davidlmobley generate synthetic ThermoML files where we
simply extend the set of measurement types in a consistent way? That would
have the least impact on the design.

Maybe I'm missing something here, but naively this approach seems to me like an incredible hack -- are you saying that, to avoid having to accommodate non-ThermoML data in our framework, we should basically generate "fake" ThermoML datasets/data types for single molecule gas phase measurements? I'm missing what's beneficial about this. Already our property estimator is going to have to be able to recognize (at least when we need it to!) each type of data available in ThermoML when we want to calculate the relevant property -- presumably, this will be some kind of conditional leading to code for computing that property. I am missing why it's better to have the conditional recognize a ThermoML dataset type corresponding to single molecule gas phase data rather than to have the conditional recognize a non-ThermoML dataset type corresponding to single molecule gas phase data...

@davidlmobley is looking for a way to encode the information "this
measurement was made for an isolated molecule in vacuum". That should be
the job of the container object specifying the system composition.

I very much agree with this. In fact I THOUGHT this resolved the whole issue as I explained in this comment: https://github.com/open-forcefield-group/open-forcefield-tools/issues/10#issuecomment-229210542
Should I re-write the Issue that started this off using the idea from that last comment so you can see what I'm imagining this code would look like?

Just to comment on the ThermoML aspect:
ThermoML is a really nice format for storing data because it IS
standardized. There may be advantages in trying to standardize the
community around this format.
If current datasets are not in ThermoML format, it is almost as easy to
write a converter to cram the data into ThermoML format as it is to cram it
into some internal Python container representation. So why not persuade
people to put it in ThermoML format?
If there is data that is currently not supported by the ThermoML format, we
get to help drive the extension of the format to accommodate this data.
If one of our main goals is to help define and encourage the adoption of
community standards for data sharing that make it easier to collect
machine-readable datasets and parameterize forcefields from them, I see a
lot of advantages to using this as a carrot: These tools will work
automatically on data, as long as it is in the ThermoML standard. It need
not be in the Archive, but getting it into the ThemoML format is 90% of the
way to sharing it.

Light bulb comes on, @jchodera :

ThermoML is a really nice format for storing data because it IS
standardized. There may be advantages in trying to standardize the
community around this format.

I see. So you are NOT talking about wanting all the datasets to be in ThermoML or ThermoML datasets, but in ThermoML FORMAT.

If current datasets are not in ThermoML format, it is almost as easy to
write a converter to cram the data into ThermoML format as it is to cram it
into some internal Python container representation. So why not persuade
people to put it in ThermoML format?

OK, so I need to look at the format spec and see if I can think of anything I see us wanting to do that wouldn't fit. Do you have a link to the spec handy? I'll google momentarily.
In any case, now I finally get why you are attached to ThermoMLDatasets - it's not because you want ThermoML datasets, it's because you want ThermoML format datasets. Very different. :)

Side note - I need to create a PR to modify the API to clarify that ThermoMLDataset is NOT necessarily a ThermoML dataset but a ThermoML format dataset. :)

Format for ThermoML is here: http://trc.nist.gov/ThermoMLRecommendations.pdf

Gaah, this format document is complex enough I'm not going to be able to quickly reach any useful conclusion about whether things we want would all fit. Superficially it looks like this basically can be framed to hold any type of average property we want via the right type of extensions.
The one thing that I could see possibly being a problem is that Christopher is very interested in being able to (ultimately) work with distributions rather than just averages. To give a very simple example, imagine we have collected data from gas phase simulations of, say, some alkanes and we observe a particular torsional distribution has two peaks. If we are going to fit this torsion, we presumably need to end up with not just the average value of the torsion, but info on the distribution of torsion angles. So I think his view is that ultimately we'll need to be able to work with and fit to distribution data. But, maybe this still fits - i.e. equilibrium populations in particular torsional bins?

In any case, now I finally get why you are attached to ThermoMLDataset - it's not because you want ThermoML datasets, it's because you want ThermoML format datasets. Very different. :)

Yes! For convenience, we will make it very easy to grab these datasets from the NIST ThermoML Archive, but you will also be able to specify a directory or URL with ThermoML format files in it instead.

Format for ThermoML is here: http://trc.nist.gov/ThermoMLRecommendations.pdf . 72 pages.

There have been a few revisions to the format---see this page for the latest pointers to the format updates. Kenneth Kroenlein is in the middle of a new set of revisions, which makes it all the more timely for @bmanubay and @mrshirts to work with them to correct deficiencies or enhance the format if needed.

Gaah, this format document is complex enough I'm not going to be able to quickly reach any useful conclusion about whether things we want would all fit.

The best thing to do is to look through some sample XML files. You can use thermopyl to grab a local copy of some files.

OK, so @jchodera - I am totally OK with everything being a ThermoML-format dataset as long as you think it can hold everything we need. Can you think of any obvious types of things it CAN'T hold?

@jchodera - I've poked a little bit at XML files and am starting to get the picture. So as long as it's something we can frame as a named property which has a single value and associated uncertainty (i.e in ePropName for the name of the property) we should be OK. Is that right?
The issue above about populations/histogram type info is probably relevant, but maybe this can be fit into the ThermoML dataset framework as long as we're allowed to be liberal in naming things, such as properties of particular torsional bins if we got to that point.
(Aside from my silly example of why we might look at torsional populations, we'll also at some point have to do torsion drives in this framework presumably, so we have to think about how that might fit into this format...)
Offline for a couple of hours beginning shortly.

An additional API question:
If we're being asked to compute multiple physical properties for multiple parmeter sets, how do we want to return the data? My original API example just illustrated retrieving a list of computed properties that correspond to measurements in a PhysicalPropertyDataset, but for a single input ParameterSet:
# Attach to my compute and storage resources
estimator = PropertyEstimator(...)
# Estimate some properties
computed_properties = estimator.computeProperties(dataset, parameters)
# Get statistics about simulation data that went into each property
for property in computed_properties:
   # Get statistics about simulation data that was reweighted to estimate this property
   value = property.value
   ...

In the case of multiple parameter sets, do we want to return a list of lists? Or do we want to add another container type, and return a list of ComputedPhysicalPropertyDataset objects?

@jchodera : I actually thought of this when extending it to multiple parameter sets, but realized that your existing framework seems to handle this case reasonably well at least from my perspective. Particularly:

PropertyEstimator.computeProperties(...) returns a list of ComputedPhysicalProperty objects that provide access to several pieces of information:

property.value  - the computed property value, with appropriate units
...
property.parameters - a reference to the parameter set used to compute this property
...
"""

So, as long as we get back a list of properties, and each has a particular parameter set tied to it, there's no ambiguity, in that if we have any questions about which entry in the list goes with which parameter set, that information is directly tied to the entries themselves.
I don't have any problem with a list of lists if you prefer that, but even a list of the results would be unambiguous. And the docs could clarify whether this list is ordered by properties and then parameter set, or parameter set and then property.
It seems unnecessary to have another container.
Your call though.

@jchodera : As you get to working more on the property calculation framework this weekend, I just wanted to reawaken the "equilibrium expectation" issue on your radar, as the very first property we'll be trying to parameterize to (really, as soon as we have the framework in place to do so!) is equilibrium expectations extracted from simulations with parm@frosst.
The one big concern I have at this point is that I want to make sure we know how we'll fit data on distributions into the ThermoML format (or at least know that we can without too much pain). Particularly, we'll need to be able to simulate with parm@frosst and extract and store (presumably to an extended ThermoML format) information on the distribution of torsions sampled, not just the average value for a particular torsion. Otherwise, if we simulate a molecule which has a torsion with, say, two equally populated peaks at +/-90 degrees, we'll be able to fit it equally well by (a) a torsion with a single peak at zero degrees, or (b) a torsion with two peaks at +/-N degrees, or (c) a torsion with three peaks at zero, +/-N, or (d)... (e) ... (f)...
The other observables we have in mind (bond, angle, energy, etc.) this should not be as much of an issue for, as we expect these often to basically be unimodal so the mean captures much of the key information (though the standard deviation might perhaps also be useful). But since torsions are periodic, the mean is not a particularly useful quantity and we'll need to be able to work with distribution info, probably for particular torsional bins we would define.

On page 592-600ish there's a discussion of representing ThermoML properties
as equations using MathML  I haven't fully investigated it yet, but it
could be possible represent distributions as Fourier transforms to some
appropriate periodicity.
On Fri, Jul 8, 2016 at 5:55 PM, David L. Mobley<EMAIL_ADDRESS>wrote:

@jchodera https://github.com/jchodera : As you get to working more on
the property calculation framework this weekend, I just wanted to reawaken
the "equilibrium expectation" issue on your radar, as the very first
property we'll be trying to parameterize to (really, as soon as we have the
framework in place to do so!) is equilibrium expectations extracted from
simulations with parm@frosst.
The one big concern I have at this point is that I want to make sure we
know how we'll fit data on distributions into the ThermoML format (or
at least know that we can without too much pain). Particularly, we'll need
to be able to simulate with parm@frosst and extract and store (presumably
to an extended ThermoML format) information on the distribution of torsions
sampled, not just the average value for a particular torsion. Otherwise, if
we simulate a molecule which has a torsion with, say, two equally populated
peaks at +/-90 degrees, we'll be able to fit it equally well by (a) a
torsion with a single peak at zero degrees, or (b) a torsion with two peaks
at +/-N degrees, or (c) a torsion with three peaks at zero, +/-N, or (d)...
(e) ... (f)...
The other observables we have in mind (bond, angle, energy, etc.) this
should not be as much of an issue for, as we expect these often to
basically be unimodal so the mean captures much of the key information
(though the standard deviation might perhaps also be useful). But since
torsions are periodic, the mean is not a particularly useful quantity and
we'll need to be able to work with distribution info, probably for
particular torsional bins we would define.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/open-forcefield-group/open-forcefield-tools/issues/10#issuecomment-231499118,
or mute the thread
https://github.com/notifications/unsubscribe/AEE31DGOde-v3rcdc_Jg2S1_XF9RXqv2ks5qTuNrgaJpZM4I8MWR
.

I'm with @mrshirts that we want to avoid histograms. I think these will only cause us headaches.
What about expectations of spherical harmonics or some other periodic function?

Or circular moments?

I like the idea of something like a cosine expansion; I was not familiar with the terminology "circular moments" but those would also do nicely. The key point is that we can't get by with only a simple mean of a torsion, but we can get by with a simple mean of something which is periodic.
I wonder if there are related issues that come up in any other observables we would be interested in? Torsions was the only thing where this was obvious to me. I suppose it could be an issue for bonds if we ever went to a potential more like a Morse potential or something which is decidedly anharmonic. But, again, this should fit well into a framework which would allow circular moments, fourier transforms or similar.

Morse potential or something which is decidedly anharmonic. But, again, this should fit well into a framework which would allow circular moments, fourier transforms or similar.

It seems you can represent observations as any object in MathML (again, we'll have to look into this a bit more closely)
If we are collecting a finite amount of data, the Fourier series representation with a decent number of terms (10-ish) should represent most functions with a pretty good amount of fidelity.

@jchodera asked for a reminder of what IsolatedMolecule data we would want to deal with (i.e. for our first test of parameterizing , so here's a reminder:
Initially (simplest test), averages of:

energy
bond distance for specified bonds
angle for specified angle
torsion for specified torsion

Very soon:

Standard deviations of the above (since we can't fit a force constant without the width of a distribution)

Soon:

Fourier series representation for torsions instead of average/standard deviation (since averages and standard deviations only provide enough info for things which are unimodal and symmetric)

Later:

More complicated specifications for the above, like pull out all of the bond/angle/torsional information for BATs for all molecules /cases matching the specified SMIRKS in the specified system

I've got to the point where I need this API to be nailed down, since you indicated you need this first.
Going back to @davidlmobley's proposed API:
dataset = GasPhaseSingleMoleculeAverageEnergy( '/Users/dmobley/phenol_gasphase.cdf4')
dataset += GasPhaseSingleMoleculeBondDistance('/Users/dmobley/phenol_gasphase.cdf4', '[#5]~[#8]') #Analyze C-O bond length
# Compute physical properties for these measurements
estimator = PropertyEstimator(nworkers=1)
computed_properties = estimator.computeProperties(dataset, parameters)
# Write out statistics about errors in computed properties
for (computed, measured) in (computed_properties, dataset):
   property_unit = measured.value.unit
   print('%24s : parm@frosst value %8.3f (%.3f) | calculated new value %8.3f (%.3f) %s' % (measured.value / property_unit, measured.uncertainty / property_unit, computed.value / property_unit, computed.uncertainty / property_unit, str(property_unit))

Some questions:

Do you really want to use GasPhase here, or are we calling it IsolatedMolecule now?
Do you really want to have a ton of measured properties, one for each valence term and moment order, which would create dozens and dozens of things like

GasPhaseSingleMoleculeBondDistance
GasPhaseSingleMoleculeBondDistanceStandardDeviation
GasPhaseSingleMoleculeBondDistanceThirdMoment
GasPhaseSingleMoleculeAverageAngle
etc.
or would it make more sense to have a single function IsolatedMoleculeValenceMoment that takes a type='bond' argument and an order=2 for the order n of the moment E[x^n] that should be computed?

Do we want moments E[x^n] or central moments E[(x-E[x])^n]?
Do we really want to use SMARTS strings to match? What happens if multiple valence terms match?

@jchodera:
Yes, we're calling it IsolatedMolecule, not GasPhase. And, I think we actually said that it might be helpful to have IsolatedMolecule tied to the composition/substance/mixture rather than to the name of the property, since then one doesn't have to separately implement ValenceMoment for isolated molecules versus for molecules in solution when these are really the same thing.
And yes, I now agree with you that we don't want things like GasPhaseSingleMoleculeBondDistance. Basically we want just a framework that can handle averages of properties. So yes, ValenceMoment would work well (or if for some reason it doesn't make sense to have IsolatedMolecule tied to the composition, we could do IsolatedMoleculeValenceMoment.

Do we want moments E[x^n] or central moments E[(x-E[x])^n]?

Seems like central moments, actually. We'll want the mean, and then the standard deviation.
Perhaps it is not yet time to consider it, but for torsions we'll also want (eventually) a fourier series representation, as noted elsewhere.

Do we really want to use SMARTS strings to match? What happens if multiple valence terms match?

Maybe there would be other ways to achieve the same effect other than SMARTS/SMIRKS. But my thinking here is that we're going to be using this to do things like "pull out all the carbon-carbon bond distances and standard deviations for this molecule" (since we have one set of parameters for those) so we can fit to them. So we're going to want to get them all in one shot, and it would be nice to do it by pulling the same SMIRKS descriptors from the FFXML file - i.e. we loop over parameters from the FFXML file, and for each one we use the SMIRKS string to pull all of the relevant data from our simulations.
If multiple terms match, my thought was that we would (a) get the data aggregated across all matching terms, and (b) optionally get back the data organized by specific match, i.e. as a list of [(atomnumbers, average), (atomnumbers, average), ...]. I'm OK to adjustments to this though - the main point though is that in this case we're going to be looking at aggregated data across our entire set that provides info on C-C bond lengths (for example), and it seems like it would be most convenient just to be able to pull that by specifying the relevant SMIRKS. I'm ambivalent about whether the aggregation occurs before or after pulling the data though.

I'll note that the changes required for this are really making everything much more complicated, since they are such a departure from the original planned API.

I agree this is different than most of the data we're talking about using, but it IS the data it makes most sense for us to fit to first. If you can think of a better way to handle it I'm certainly open to suggestions.

Do you really want to have a ton of measured properties, one for each valence term and moment order . . . or would it make more sense to have a single function IsolatedMoleculeValenceMoment that takes a type='bond' argument and an order=2 for the order n of the moment E[x^n] that should be computed?

It depends on how we want these stored. If they are stored as ThermoML entries, then we can't have extra functions -- it's not necessarily the end of the world if we have a bunch of properties.
If we have ThermoML-parallel data structures, then of course we can have extra functions, and the data can be stored in them however we want.  We'll need to be a little careful about how to define the 'bond' argument -- will it just be a pair of atom integers?  I'm not exactly what other pattern we could use.  We'll need to link it somehow to the molecular structure (in whichever format is appropriate -- it starts out in AMBER .prmtop formats, but we can convert it and use some other format.

Do we want moments E[x^n] or central moments E[(x-E[x])^n]?

We need the first moment, since we need the mean.  That's not any of the central moments.  We need the 2nd central moment (the variance, sigma^2), though sigma is fine.  For torsions, we want a Fourier expansion -- moments won't work well with multimodal functions.

Do we really want to use SMARTS strings to match? What happens if multiple valence terms match?

I agree with David's comments here (if I understand them correctly).   We want multiple valence terms to match so we can decide if they should be lumped together or not.

I'll note that the changes required for this are really making everything much more complicated, since they are such a departure from the original planned API.

I'd advocate for trying to keep the API simpler, and us handling the departures with separate interfacing functions. One example, just giving up and having a bunch of weird property names IsolatedMoleculeBondMeanMolecule734Atom1Atom10), since we won't do something like this again with the main API -- if there's a few hundred lines of special use code, I would see that as an OK tradeoff for not having to have too complicated an API.

I fear this is turning into a really vaguely-specified feature request that we're trying to shoehorn into an API design for something else entirely. We'll try a bit more to iterate the design to something concrete, but this is going to significantly slow us down by not having something specific I can implement.

e. as a list of [(atomnumbers, average), (atomnumbers, average), ...].

The atom numbers will be entirely meaningless unless we make the only way of specifying the IsolatedMolecule system be to provide an OEMol object.  That is, instead of the analog of
mixture = Mixture()
mixture.addComponent('methanol', mole_fraction=0.5)
mixture.addComponent('water')

which is
ethane = IsolatedMolecule(iupac='ethane')
methanol = IsolatedMolecule(smiles='CO')

we would require you provide
molecule = IsolatedMolecule(oemol)

Be sure to think through the ramifications of all that. Do you really want to have to construct these from OEMol objects that you store serialized alongside the computed valence property averages in order to ensure deterministic atom ordering?

@jchodera - that's a good point relating to OEMols and atom numbering. I don't think it makes sense to have the only way that these be initialized be an OEMol, which leads me back to my preference for SMARTS/SMIRKS for the relevant properties.
Are there any other options you think are reasonable? I just don't see any, because we really want something that allows us to say "pull the data on that valence parameter" without having to worry about atom ordering, etc.
Allowing specifications of SMARTS/SMIRKS and returning properties by those will give us all the flexibility I think we'll need - i.e. if we're looking at CURRENT parameters, then we just query via our current SMARTS/SMIRKS, but if we want to distinguish (later) whether a new environment might be warranted, we can query based on SMIRKS which are derivatives of our current ones (i.e. if we want to see if the data would warrant two C-C bond types, we can query based on what we're proposing those to be).

@jchodera - would it help if I write down an updated usage example here, and then also do a PR into the README.md with what I am proposing?

Here's my current thinking for API:
For testing against synthetic data, we also make use of isolatedMolecule objects (also subclasses of Substance) that represent a single molecule:
phenol = IsolatedMolecule(iupac='phenol')
ethane = IsolatedMolecule(smiles='CC')

The various properties are all subclasses of MeasuredPhysicalProperty and generally follow the <ePropName/> ThermoML tag names.
Some examples:

MassDensity - mass density
ExcessMolarEnthalpy - excess partial apparent molar enthalpy
HeatCapacity - molar heat capacity at constant pressure
There are also some examples of measured physical properties we use exclusively for testing against synthetic data:
MeanPotentialEnergy - mean potential energy of a system
BondMoment - the nth moment (mean E[x] if n==1, or central moment E[(x-E[x])^n] if n >= 2) of a specified bond of an isolated molecule
AngleMoment - the nth moment (mean E[x] if n==1, or central moment E[(x-E[x])^n] if n >= 2) of a specified angle of an isolated molecule
TorsionMoment - the nth circular moment (E[e^{i*n*phi}]) of a specified torsion of an isolated molecule

molecule = IsolatedMolecule(smiles=smiles)
thermodynamic_state = ThermodynamicState(temperature=300*unit.kelvin)
mean_potential = MeanPotentialEnergy(molecule, thermodynamic_state, value=124.4*unit.kilojoules_per_mole, uncertainty=14.5*unit.kilojoules_per_mole)
bond_average = BondMoment(molecule, thermodynamic_state, value=1.12*unit.angstroms, uncertainty=0.02*unit.angstroms, moment=1, smirks='[#6:1]-[#6:2]')
bond_variance = BondMoment(molecule, thermodynamic_state, value=0.05*unit.angstroms**2, uncertainty=0.02*unit.angstroms**2, moment=2, smirks='[#6:1]-[#6:2]')
angle_average = AngleMoment(molecule, thermodynamic_state, value=20*unit.degrees, uncertainty=0.05*unit.degrees, moment=1, smirks='[#6:1]-[#6:2]-[#6:3]')

QUESTIONS

Is it useful to average over any bonds that match the SMARTS strings? How would you even construct those SMARTS strings algorithmically for the molecules in the set, to avoid matching anything else?
Note that we need the first noncentral moment (average), so I had an exception where we don't compute the central moments for moment == 1
Do we really want the variance (second central moment) instead of the standard deviation?

@jchodera :
That looks good to me.

How would you even construct those SMARTS strings algorithmically for the molecules in the set, to avoid matching anything else?

The main use case I see is this: in our initial testing we will take a forcefield in SMIRFF format (with initial, "bad" parameters but with fixed "atom types" (SMIRKS strings)) and fit parameters to reproduce parm@frosst. In this case, we already have the SMIRKS strings (they are in the SMIRFF file already, i.e. for AlkEthOH) and we are just trying to learn the parameters which should be associated with the SMIRKS strings. In this use case, they are going to be "handed to us" effectively. Though, more generally, once we have Smarty working for bonds, angles, and torsions, we will learn the SMIRKS patterns we are going to use for a specified set, then we would be feeding them into this.

Is it useful to average over any bonds that match the SMARTS strings?

I think we will ultimately be averaging across all matches, but we may want to get the un-averaged data back as well. Otherwise, I think there's a bit of a weighting problem, in that if molecule A has five matches for a particular SMIRKS and molecule B has only one, and I later want to combine, I don't know that molecule A's results should be weighted five times as much in the average.

Note that we need the first noncentral moment (average), so I had an exception where we don't compute the central moments for moment == 1

Perfect.

Do we really want the variance (second central moment) instead of the standard deviation?

Well, we should be able to get by with either, presumably. Seems like for API simplicity/consistency we can get by with the variance.

OK, sounds like were making progress here. Can you flesh out a use case using the proposed API just a bit more, to make sure we're on the same page?

@jchodera : Yes.
So for example, starting from what you have above, if I want to SPECIFY certain values:
molecule = IsolatedMolecule(iupac='ethanol')
thermodynamic_state = ThermodynamicState(temperature=300*unit.kelvin)

mean_potential = MeanPotentialEnergy(molecule, thermodynamic_state, value=124.4*unit.kilojoules_per_mole, uncertainty=14.5*unit.kilojoules_per_mole)
bond_average = BondMoment(molecule, thermodynamic_state, value=1.52*unit.angstroms, uncertainty=0.02*unit.angstroms, moment=1, smirks='[#6:1]-[#6:2]')
bond_variance = BondMoment(molecule, thermodynamic_state, value=0.05*unit.angstroms**2, uncertainty=0.02*unit.angstroms**2, moment=2, smirks='[#6:1]-[#6:2]')
angle_average = AngleMoment(molecule, thermodynamic_state, value=109.5*unit.degrees, uncertainty=0.05*unit.degrees, moment=1, smirks='[#6:1]-[#6:2]-[#6:3]')

We'd also have
torsion_moment_3 = TorsionMoment(molecule, thermodynamic_state, moment=3, value=30, uncertainty = 0.05, smirks='[#6X4:1]-[#6X4:2]-[#8X2:3]-[#1:4]')

for torsions, for example.
But we also want to tell the property estimator we want to calculate these values. We haven't totally made explicit what the dataset structure would be for this (I'm OK with it being ThermoMLDatasets if that's what's easiest for you/makes the most sense, but it could also just a simple collection of the above objects - i.e. `dataset = PhysicalPropertyDataset([bond_average, bond_variance, angle_average]).
Let's assume the latter for now. So then I'd be jumping off of the code above to do something like:
parameter_set = [ SMIRFFParameterSet('smarty-initial.xml') ]
dataset = PhysicalPropertyDataset( [bond_average, bond_variance, angle_average, torsion_moment_3])
# Compute physical properties for these measurements
estimator = PropertyEstimator(nworkers=10) # NOTE: multiple backends will be supported in the future
computed_properties = estimator.computeProperties(dataset, parameter_set)
# Write out statistics about errors in computed properties
for (computed, measured) in (computed_properties, dataset):
   property_unit = measured.value.unit
   print('%24s : parm@frosst value %8.3f (%.3f) | newly calculated %8.3f (%.3f) %s' % (measured.value / property_unit, measured.uncertainty / property_unit, computed.value / property_unit, computed.uncertainty / property_unit, str(property_unit))

Is that what you're after? Key point - it's just like we would be doing with ThermoML datasets except that the datasets themselves are rather different.

Ultimately of course we'd be running through this same type of queries for all of the parameters in our dataset. i.e. we'd pull all the SMIRKS from the SMIRFF XML for AlkEthOH (https://github.com/open-forcefield-group/smarty/blob/master/smarty/data/forcefield/Frosst_AlkEtOH.ffxml) and then, with some other (non-parm@frosst) set of parameters:

Loop over all molecules in AlkEthOH
Get the average potential energy for each
For each molecule, loop over all smirks which match any vdw type/bond/angle/torsion
For each match, get the mean and variance (for bonds and angles) or a range of circular moments (for torsions)
Compare to reference values for these calculated with parm@frosst parameters

The atom numbers will be entirely meaningless unless we make the only way of specifying the IsolatedMolecule system be to provide an OEMol object

Well, we could just stick with the ordering given by the amber .prmtops.  If SMIRKS matching was much easier/faster using OEMol objects, of course, that would be the way to go.
I think there are possibly some super-elegant ways to solve this problem, but I'm not sure they are the right way to go, as we want to focus super-elegant things on the main usage cases.
I agree with David about the workflow.

There will be no amber prmtops anymore, and long term we'll want to have
the tool chain be independent of any particular file format/molecular
representation, so John is absolutely right that in the user-facing tool we
have to ditch atom numbers in those. Smirks will be the way to go
-functionally they will work well, and this is also rather in keeping with
the whole FF philosophy so far where we use smirks for chemical environment
selection.
On Mon, Jul 18, 2016, 6:35 PM Michael Shirts<EMAIL_ADDRESS>wrote:

The atom numbers will be entirely meaningless unless we make the only way
of specifying the IsolatedMolecule system be to provide an OEMol object
Well, we could just stick with the ordering given by the amber .prmtops.
If SMIRKS matching was much easier/faster using OEMol objects, of course,
that would be the way to go.
I think there are possibly some super-elegant ways to solve this problem,
but I'm not sure they are the right way to go, as we want to focus
super-elegant things on the main usage cases.
I agree with David about the workflow.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/open-forcefield-group/open-forcefield-tools/issues/10#issuecomment-233507184,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AGzUYbBTC1T-2pozyY3GlHq1mqxcmq7lks5qXCn-gaJpZM4I8MWR
.

There will be no amber prmtops anymore,

OK, I'm not quite seeing the workflow anymore for generating the data for single molecules from the simulations. The .nc (or .dcd) generated by OpenMM are processed, and how are we storing each of the individual properties computed?  What sort of dictionary will we use to map between this data (that is recorded by atom number) and the SMIRKS representation and when will it be generated? It's not totally clear to me the sequence.  Is SMIRKS used from the beginning to analyze the .nc files by querying the atom order from the internal OpenMM representation?
I suppose one way to do it (that I think its implied above) is to pull together and process the .nc data (by atom number), then store it in a data type that was an OEChem molecule of that type created from the .prmtop, then assigned property value to each bond (assuming the atom numbers are preserved from the .prmtop). Presumably any creation of the molecule would preserve bonds, since OpenMM is preserving the bond numbers when it creates the .nc file.

@mrshirts - probably this discussion should go into a separate issue, as I think what we're talking about right now is how reference isolated molecule data that we have already generated or might be generating outside the property estimator framework and get it in to our framework for use in fitting, which is a separate issue. Maybe you want to start a new issue for this, and you and I can sort it out (John doesn't need to be involved)?
To clarify: there are two types of isolated molecule data here:

Data we generate as reference data for fitting to, which might or might not come through the property estimator framework
Data we will generate with our property estimator framework to use as part of fitting to the reference data in item 1.

The API being discussed here concerns issue 2, and I think you're asking about issue 1.
My group currently has some "one-time" reference data along the lines of item 1, where we went ahead and ran isolated molecule gas phase simulations of AlkEthOH molecules with parm@frosst parameters beginning from AMBER .prmtop and .crd files. We could extract expectations from this and easily put it into this format:
mean_potential = MeanPotentialEnergy(molecule, thermodynamic_state, value=124.4*unit.kilojoules_per_mole, uncertainty=14.5*unit.kilojoules_per_mole)
bond_average = BondMoment(molecule, thermodynamic_state, value=1.52*unit.angstroms, uncertainty=0.02*unit.angstroms, moment=1, smirks='[#6:1]-[#6:2]')

I know you've been working on processing that data by atom number; I could easily generate the SMIRKS patterns which would allow us to link the atom numbers (in this one case) to SMIRKS strings.  Alternatively, we could just re-run the gas phase calculations using the property estimator framework when it is online, using parm@frosst parameters as input.
To get back to this issue:

There will be no amber prmtops anymore

Aside from the reference data from Item 1 above, we'll be parameterizing molecules henceforth using the SMIRFF XML format and going directly into OpenMM, so we're no longer going to be using AMBER file formats.

OK, I'm not quite seeing the workflow anymore for generating the data for single molecules from the simulations. The .nc (or .dcd) generated by OpenMM are processed, and how are we storing each of the individual properties computed? What sort of dictionary will we use to map between this data (that is recorded by atom number) and the SMIRKS representation and when will it be generated? It's not totally clear to me the sequence. Is SMIRKS used from the beginning to analyze the .nc files by querying the atom order from the internal OpenMM representation?

Hopefully the above helped on this point. Basically right now we have this "one off" gas phase data that we generated from AMBER prmtop and crd files, and I can easily generate a dictionary which will map SMIRKS patterns to atom numbers or vise versa, just for the purposes of pulling the data we want from those simulations.

I suppose one way to do it (that I think its implied above) is to pull together and process the .nc data (by atom number), then store it in a data type that was an OEChem molecule of that type created from the .prmtop, then assigned property value to each bond (assuming the atom numbers are preserved from the .prmtop). Presumably any creation of the molecule would preserve bonds, since OpenMM is preserving the bond numbers when it creates the .nc file.

This may work, but it may also be more trouble than it's worth.
So, my suggestion is to create a separate issue where you explain exactly what you've achieved so far in terms of extracting the data, and then I can sort out the issue of the associated SMIRKS patterns there.

John, what specific things do you need more information on to write the API?  My current feeling is that the IsolateMolecule stuff can be ignored for now -- we want to be focused on liquid phase.  I think that if we need to go back and do some additional isolated molecule work, we can probably write some relatively limited case tools.  I think we've decided that the applications for isolated molecules are somewhat limited and going in different directions than the liquid ones, so making a unified API isn't important.


about model fuse？

I train a model for detection, i want to fuse model, but i can not find example for fusing model??

finish


Implement tagged unions for sealed traits

This PR implements tagged unions for sealed traits.
For example:
sealed trait Foo
case class Bar(s: String) extends Foo
case class Baz(s: String) extends Foo

translates to:
export interface IBar {
  s: string
  type: "Bar"
}
export interface IBaz {
  s: string
  type: "Baz"
}
export type Foo = (IBar | IBaz)

The name of the discriminator field can be changed, and the feature can be disabled for compatibility purposes.

Thanks for the PR, tagged unions are insanely common and thus useful to support. Fixes #50
I took a look at your PR and have a bit of difficulty with modifying TSUnion for this to add the boolean, while a type being DU has nothing to do with the union itself:
type SchrodingersDiscriminatedUnion= A | B
// Current representation: TSAlias("SchrodingersDiscriminatedUnion", Seq(ref("A"),ref("B")))

// With this definition the union is not a DU
type A = string
type B = number

// With this definition it is
interface A {
  type: 'A'
}
interface B {
  type: 'B'
}

This means that ideally a TSUnion is totally oblivious to whether it's a DU or not, since it's an emergent property and not something just with the union itself (which would merit adding a field).
With your PR, when looking at SchrodingersDiscriminatedUnion, the representation should have tagged = false in the one case and tagged = true in the other. Suddenly you can't just look at a snippet of Typescript code and create the corresponding Scala representation. That can lead to difficulties and bugs.
So how could this be done instead? The right place for this is the macro. fromSealed should add some meta-information, which the serializer can pickup and treat like an extra field for interfaces.
Luckily such a meta-wrapper already exist in TSTypeReference, where this can be added:
    * @param discriminator the discrimininator value for the type if this type is part of a discriminated union
  case class TSTypeReference(name: String, impl: Option[TypescriptType] = None, discriminator: Option[TypescriptType] = None) extends TypescriptNamedType with TypescriptAggregateType {

this can then be used by the serializer to add the right field, and by the macro to add the information:
val operands = children map { symbol =>
       q"TypescriptType.nameOrType(${getTSType(symbol.asType.toType)}.get, discriminator = symbol.name.toString)"
}

If you modify the PR with this or solve the changes in another way* I'll review again, otherwise I'll try this myself when I get around to it™
* The ideal castle-in-a-cloud solution would be the macro should add the underlying TSInterface representation's to add a $kind: 'XXX' field, where $kind is the configurable SBT property, but I don't think that's doable.

I updated the code according to your request, fixed a few things and improved the tests, and I rebased onto master.


Support for ic_launcher_round

RN 0.57.0 adds a new icon to the default android project, ic_launcher_round.

It would be great if rn-toolbox:assets was updated to support generating this image, too. It might need a separate source that the normal icon, given how much a circle cuts off.

Any update?

#216  Just submit a pull request

Maybe the readme in generator-rn-toolbox/generators/assets/README.md  should also be updated as it currently suggests generating the android launcher icon through android studio

@schumannd  good point , though i found that icons generated by the android studio way , has better resolution , better picture quality on my test device .

Hello, we just revamped the generator 😁
For up to date icon and splashscreen generation please use @bam.tech/react-native-make
The new version generates Android adaptive icons, so it should solve your issue as long as the input image follows adaptive icons guidelines😉


Display Date  Label as group in axes - chart js

Need to display labels date in the x axes in the interval . Data should remain same only date interval shoould be displayed as per the snapshot.

For simple numbers on y axes able to achieve this using "ticks"
        max: 5000,
        min: 0,
        stepSize:4000

But in case of date not sure the property which is to be used. Attached the code snippet stackblitz url (https://stackblitz.com/edit/ng2-chart-example-5xxqc9?file=app/app.component.ts

)

Please ask questions on StackOverflow: https://stackoverflow.com/questions/tagged/chart.js


Clean up / optimize makefile, sandbox builds better.

Removed a bunch of cruft from overly-generic makefile copypasta
(many of it doesn't do anything here, and makes it harder to read).
Corrected a number of dependency chains.  Let me know if I missed
something!  I've tried to be thorough, but it's hard to be sure.
in any case, now it shouldn't re-do work unless it's necessary, and
it can even init in parallel (make -j4) from a fresh clone without
any problems I've seen.
I was unable to trick Glide into including the necessary packages / versions
to reliably build things out of vendor... so now there's a versioned_go_build.sh
script to do that.  It'll also warn you if there's a newer version of tool X
available.

Dep can do this pretty easily, add the target command to require... though
I'm not sure what that implies for users-of-this-lib.  Can investigate later.

Last but not least: go test ./... is equivalent to (or better than)
testing each package separately, and is often much faster because it
doesn't spend time rebuilding everything.  And now that we're on Go 1.10, it can
do coverage across all packages too!

Before all changes:
$ make clean
rm -rf cadence-client
rm -Rf ./build
rm -f dummy

$ time make
rm -rf .gen
go get './vendor/go.uber.org/thriftrw'
go get './vendor/go.uber.org/yarpc/encoding/thrift/thriftrw-plugin-yarpc'
thriftrw --plugin=yarpc --pkg-prefix=go.uber.org/cadence/.gen/go/ --out=.gen/go idl/github.com/uber/cadence/cadence.thrift
thriftrw --plugin=yarpc --pkg-prefix=go.uber.org/cadence/.gen/go/ --out=.gen/go idl/github.com/uber/cadence/shared.thrift
go run ./internal/cmd/tools/copyright/licensegen.go --verifyOnly
go build -i -o dummy internal/cmd/dummy/dummy.go
ok  	go.uber.org/cadence/internal	4.175s	coverage: 66.6% of statements
ok  	go.uber.org/cadence/internal/common/backoff	1.046s	coverage: 93.2% of statements
ok  	go.uber.org/cadence/internal/common/cache	1.462s	coverage: 73.1% of statements
ok  	go.uber.org/cadence/internal/common/metrics	1.055s	coverage: 85.0% of statements
ok  	go.uber.org/cadence/internal/common/util	1.024s	coverage: 5.2% of statements
ok  	go.uber.org/cadence/mocks	1.037s	coverage: 6.7% of statements

real	0m20.177s
user	0m11.994s
sys	0m5.099s

$ time make
rm -rf .gen
go get './vendor/go.uber.org/thriftrw'
go get './vendor/go.uber.org/yarpc/encoding/thrift/thriftrw-plugin-yarpc'
thriftrw --plugin=yarpc --pkg-prefix=go.uber.org/cadence/.gen/go/ --out=.gen/go idl/github.com/uber/cadence/cadence.thrift
thriftrw --plugin=yarpc --pkg-prefix=go.uber.org/cadence/.gen/go/ --out=.gen/go idl/github.com/uber/cadence/shared.thrift
go run ./internal/cmd/tools/copyright/licensegen.go --verifyOnly
ok  	go.uber.org/cadence/internal	4.177s	coverage: 66.8% of statements
ok  	go.uber.org/cadence/internal/common/backoff	1.046s	coverage: 93.2% of statements
ok  	go.uber.org/cadence/internal/common/cache	1.460s	coverage: 73.1% of statements
ok  	go.uber.org/cadence/internal/common/metrics	1.051s	coverage: 85.0% of statements
ok  	go.uber.org/cadence/internal/common/util	1.027s	coverage: 5.2% of statements
ok  	go.uber.org/cadence/mocks	1.038s	coverage: 6.7% of statements

real	0m18.736s
user	0m10.713s
sys	0m4.605s

After all changes:
$ make clean
rm -Rf .build
rm -Rf .gen

$ time make
./versioned_go_build.sh go.uber.org/thriftrw v1.11.0 .build/thriftrw
./versioned_go_build.sh go.uber.org/yarpc v1.29.1 encoding/thrift/thriftrw-plugin-yarpc .build/thriftrw-plugin-yarpc
	go.uber.org/yarpc has a newer tag: v1.30.0
thriftrw: idl/github.com/uber/cadence/cadence.thrift
go run ./internal/cmd/tools/copyright/licensegen.go --verifyOnly
touch .build/copyright
go build -i -o .build/dummy internal/cmd/dummy/dummy.go
go test ./... -coverprofile=.build/cover.out -race
?   	go.uber.org/cadence	[no test files]
?   	go.uber.org/cadence/activity	[no test files]
?   	go.uber.org/cadence/client	[no test files]
?   	go.uber.org/cadence/encoded	[no test files]
ok  	go.uber.org/cadence/internal	4.188s	coverage: 66.8% of statements
?   	go.uber.org/cadence/internal/cmd/dummy	[no test files]
?   	go.uber.org/cadence/internal/cmd/tools/copyright	[no test files]
?   	go.uber.org/cadence/internal/common	[no test files]
ok  	go.uber.org/cadence/internal/common/backoff	1.060s	coverage: 93.2% of statements
ok  	go.uber.org/cadence/internal/common/cache	1.490s	coverage: 73.1% of statements
ok  	go.uber.org/cadence/internal/common/metrics	1.065s	coverage: 85.0% of statements
ok  	go.uber.org/cadence/internal/common/util	1.037s	coverage: 5.2% of statements
ok  	go.uber.org/cadence/mocks	1.048s	coverage: 6.7% of statements
?   	go.uber.org/cadence/testsuite	[no test files]
?   	go.uber.org/cadence/worker	[no test files]
?   	go.uber.org/cadence/workflow	[no test files]

real	0m34.329s
user	0m25.309s
sys	0m18.131s

$ time make
go test ./... -coverprofile=.build/cover.out -race
?   	go.uber.org/cadence	[no test files]
?   	go.uber.org/cadence/activity	[no test files]
?   	go.uber.org/cadence/client	[no test files]
?   	go.uber.org/cadence/encoded	[no test files]
ok  	go.uber.org/cadence/internal	4.175s	coverage: 66.8% of statements
?   	go.uber.org/cadence/internal/cmd/dummy	[no test files]
?   	go.uber.org/cadence/internal/cmd/tools/copyright	[no test files]
?   	go.uber.org/cadence/internal/common	[no test files]
ok  	go.uber.org/cadence/internal/common/backoff	1.062s	coverage: 93.2% of statements
ok  	go.uber.org/cadence/internal/common/cache	1.500s	coverage: 73.1% of statements
ok  	go.uber.org/cadence/internal/common/metrics	1.060s	coverage: 85.0% of statements
ok  	go.uber.org/cadence/internal/common/util	1.034s	coverage: 5.2% of statements
ok  	go.uber.org/cadence/mocks	1.042s	coverage: 6.7% of statements
?   	go.uber.org/cadence/testsuite	[no test files]
?   	go.uber.org/cadence/worker	[no test files]
?   	go.uber.org/cadence/workflow	[no test files]

real	0m6.635s
user	0m8.372s
sys	0m2.819s

So the net effect is a slightly slower initial build, and a much faster second+ build.
Seems worth it :)

... huh.  ./versioned_go_build.sh: line 104: DEBUG: unbound variable
bash version difference maybe?  can't reproduce.  fixed anyway.
and I wonder if grep is old or something on travis, it seems to be interpreting the regex wrong (version tags are being detected as non-versioned).  trying a fix and rerunning from what appear to be unrelated test failures.

I'm thinking of making a bins folder, actually.  something that's not cleaned, so versioned-bins don't need to be redownloaded just because someone wanted to rebuild.


clean base controller code

Each pull request should address a single issue, and have a meaningful title.
Description
Explain what you have changed, and why.
Checklist:

[ ] Securely signed commits
[ ] Component(s) with PHPdocs
[ ] Unit testing, with >80% coverage
[ ] User guide updated
[ ] Conforms to style guide

---------Remove from here down in your description----------
Notes

Pull requests must be in English
If the PR solves an issue, reference it with a suitable verb and the issue number
(e.g. fixes 12345
Unsolicited PRs will be considered, but there is no guarantee of acceptance
Pull requests should be from a feature branch in the contributor's fork of the repository
to the develop branch of the project repository

Please use GPG signed requests next time. How-to in the contributor docs.


Bug Fixes for Mock PE (Pin Xian)

This pull request will close the following bugs detected from mock practical exam.
Close #227 Close #247 : Set minimum width and height for UI display. Increase size of initial UI display.
Close #245: Improve clarity of user guide, involving the case where user repeats parameters.
Close #244: Prevent visits from the future from being added to VirusTracker.
Close #243: Create a limit that IDs should be at least 5 characters long.
Close #242 Close #255 : Update user guide to set user expectations of the valid formats for parameters.

Codecov Report

Merging #261 into master will decrease coverage by 0.00%.
The diff coverage is 33.33%.

@@             Coverage Diff              @@
##             master     #261      +/-   ##
============================================
- Coverage     84.06%   84.05%   -0.01%
- Complexity     1108     1110       +2
============================================
  Files           125      125
  Lines          3000     3005       +5
  Branches        399      399
============================================
+ Hits           2522     2526       +4
- Misses          397      398       +1
  Partials         81       81

Impacted Files
Coverage Δ
Complexity Δ

...n/java/seedu/address/commons/core/GuiSettings.java
73.07% <ø> (ø)
7.00 <0.00> (ø)

...java/seedu/address/logic/commands/ListCommand.java
100.00% <ø> (ø)
14.00 <0.00> (+1.00)

.../java/seedu/address/model/util/SampleDataUtil.java
9.09% <0.00%> (+0.26%)
1.00 <0.00> (ø)

src/main/java/seedu/address/ui/MainWindow.java
0.00% <0.00%> (ø)
0.00 <0.00> (ø)

.../address/logic/commands/visit/AddVisitCommand.java
91.42% <100.00%> (+0.25%)
23.00 <0.00> (+1.00)

...rc/main/java/seedu/address/model/attribute/Id.java
100.00% <100.00%> (ø)
8.00 <1.00> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3725f09...87fe6fe. Read the comment docs.


is not support react v16?

hello
this library have npm warning react v16

Yeah. Seems to require a peer dependency of react ^15.0.0. Any updates on this?

i want to use on react v16

@Shane0531 me too...


Provide an integration example with Agents.jl

See: https://github.com/JuliaDynamics/Agents.jl/issues/659

available at 0.7.26+docs1


Make seat chart responsive and scalable

The seating chart looks fine when rendered on mobile, but it does not auto-scale to fit the size of the screen.  This makes it very hard to use on a smartphone.  It would be nice if the chart would auto-scale to fit its parent container and then also have the ability to zoom in/out to find the specific seats the user wants.

Thank you, this is definitely a good candidate for future development.

For what it's worth, we have implemented this by scaling the seatmap-div to the size of the viewport:
scale = Math.min($( window ).width() / ($('#seatmap').width()+30)); //add 30 for margin
$('#seatmap').css('transform', 'scale('+scale+')');
$('#seatmap').css('transform-origin', 'top left');

Did anyone implement responsive seat chart map?


send buff array

https://github.com/psygame/UnityWebSocket/blob/52bc60a44003d8ccf16c551c12106a9cadb9910b/Plugins/WebGL/WebSocket.jslib#L304
should change to
instance.ws.send(HEAPU8.slice(bufferPtr, bufferPtr + length));

fixed in 2.6.0


Hyperlinks from Roles page and each role description to their list of permissions in the Permissions reference

Current State
This page describes each role, but doesn't offer a link to the specific set of its permissions
Desired State
Please add links from https://docs.layer5.io/cloud/security/roles/ to here: https://docs.layer5.io/cloud/reference/default-permissions/

Contributor Guides and Resources

📚 Instructions for contributing to documentation

Meshery documentation site and source

🛠 Meshery Build & Release Strategy
🎨 Wireframes and designs for Layer5 site in Figma (open invite)
🙋🏾🙋🏼 Questions: Layer5 Discussion Forum and Layer5 Community Slack

@dragon-slayer875

Hi,
I would like to work on the issue .
Could you please assign this task to me?
Thank you!

hey @leecalcote and @jameshorton2337  is this the desired functionality
also what links should i add to the team and workspace managers?
https://github.com/user-attachments/assets/10fd1a7e-9432-4551-94c1-b6843aaa20f3

@iushdoescode, any progress on this?

i made a pull request for this issue , is there any feedback or any changes i should do ?

@iushdoescode, Mention the issue in PR, so that it may link.


Add a Gitter chat badge to README.md

Construktion/Construktion now has a Chat Room on Gitter
@JoeGannon has just created a chat room. You can visit it here: https://gitter.im/Construktion_/Lobby.
This pull-request adds this badge to your README.md:

If my aim is a little off, please let me know.
Happy chatting.
PS: Click here if you would prefer not to receive automatic pull-requests from Gitter in future.

https://github.com/gitterHQ/readme-badger/issues/38#issuecomment-322808496


Intital Commit of ONT-ID Website Interaction Standards

This is the initial commit of the ONT-ID / Website Interaction standards that were demonstrated to Ontology.  More information and the implementation can be found at https://bridge.abine.com

I was unable to get the mediawiki syntax to allow me to set image sizes, so they are very large.


[🐛 Bug]: Export Path for ./bin Missing in package.json

next-on-pages environment related information
System:
	Platform: darwin
	Arch: arm64
	Version: Darwin Kernel Version 23.0.0: Fri Sep 15 14:41:43 PDT 2023; root:xnu-10002.1.13~1/RELEASE_ARM64_T6000
	CPU: (10) arm64 Apple M1 Pro
	Memory: 32 GB
	Shell: /bin/zsh
Package Manager Used: npm (8.19.2)

Relevant Packages:
	@cloudflare/next-on-pages: 1.7.3
	vercel: N/A
	next: N/A

Description
I'm filing this as a bug, but maybe you feel it is a feature request. it's a one line fix.
When using @cloudflare/next-on-pages with @naxodev/nx-cloudflare, an error occurs due to a missing export path in the package.json of @cloudflare/next-on-pages. The error is related to the package subpath './bin' not being defined in the "exports" field of package.json.
Error Message
The specific error message encountered is:
Error [ERR_PACKAGE_PATH_NOT_EXPORTED]: Package subpath './bin' is not defined by "exports" in<EMAIL_ADDRESS>
Expected Behavior
The package should export the ./bin subpath to allow other tools or packages to correctly access the binaries required for operations.
Actual Behavior
The bin directory is not exported in the package.json, causing the above error when trying to access ./bin/index.js.
Temporary Solution
A temporary solution that resolved the issue was to modify the package.json of @cloudflare/next-on-pages to include the ./bin subpath in the exports field as follows:
"exports": {
  "./__experimental__next-dev": "./dist/next-dev/index.cjs",
  "./bin": "./bin/index.js"
}

Suggested Fix
I suggest that the package.json of @cloudflare/next-on-pages be updated to officially include the ./bin subpath in the exports field. This would allow other packages to have peer dependencies on next-on-pages.
Reproduction

Set up a project with @naxodev/nx-cloudflare (or manually try to reference .bin/index.js
Run a build command programmatically
Observe the error related to the package subpath './bin'.

Pages Deployment Method
None
Pages Deployment ID
No response
Additional Information
No response
Would you like to help?

[x] Would you like to help fixing this bug?

Hi @khalil-omer, thanks for the issue 🙂
The next-on-pages package is not supposed to be used programmatically but as a CLI tool, so that's why bin is not being exported and I would prefer it if we could not export the binary (not for any specific reason but just because it feels semantically wrong to me 😅)
It seems that the issue is that nx-cloudflare tries to get the path of next-on-pages in the following way:
https://github.com/naxodev/oss/blob/783726bd1c0a13b5876a57c8274b17745da69b0b/packages/plugins/nx-cloudflare/src/executors/next-build/build.impl.ts#L147
Could you instead try to open an issue there and/or contribute there and see if maybe we can replace the path detection with the following (that relies on the next-dev submodule to discern the next-on-pages location):
-      require.resolve('@cloudflare/next-on-pages/bin'),
+      path.resolve(require.resolve('@cloudflare/next-on-pages/next-dev'), '../../..'),

? 🙏
What do you think?
If the above is not ok (for example the author refuses the change, etc...) I think we can add the bin entry in the exports here as you mentioned, but I'd personally prefer it if we could avoid that 🙂 (again, as a personal preference)

@khalil-omer I'm closing this issue as I believe it has been fixed in the nx-cloudflare package 🙂
I hope it's fine, if you think that the issue should not have been closed and/or that there is still something to do here please feel free to reopen it 🙂


Firestore doesn't work

I have problem when doing with Firebase Database. I have been figuring out what was going on with it, by analyzing this addon, for why doesn't it work.
The Auth here works, but The Firestore no longer works. I even tried the demo if I made wrong, but infact The demo also no longer works for updloading firebase
I have tried different method to do with Firebase, by trying the simpler script found in GDQuest tutorial here. and its finished file right here. Here I created another script to become the Singleton under different node name, let's say FirebaseQuest.
and GDQuest version works right away (both Auth and Firestore), no problem
here's example of push from GDQuest
func save_document(path: String, fields: Dictionary, http: HTTPRequest) -> void:
	var document := { "fields": fields }
	var body := to_json(document)
	var url := FIRESTORE_URL + path
	http.request(url, _get_request_headers(), false, HTTPClient.METHOD_POST, body)

and yours from FirebaseReference.gd
func push(data):
	var to_push = JSON.print(data)
	if pusher.get_http_client_status() == HTTPClient.STATUS_DISCONNECTED:
		pusher.request(_get_list_url() + db_path + _get_remaining_path(), PoolStringArray(), true, HTTPClient.METHOD_POST, to_push)
	else:
		push_queue.append(data)

Let me now comment what best I could understand. correct me if wrong, it's quite complicated for me to learn back this addon, I know.:

GDquest version do concern about request headers in the custom header every HTTP request. Maybe now it is required to have custom header when uploading HTTP, since yours lack the custom header, I think Google rejects it from now on.
Note that I had Firestore security rule designed that only allow read and write if user logged in. And again, attempting writing to and reading from there, only GDquest version that works
If the HTTP has finished doing its business (HTTPClient.STATUS_DISCONNECTED), well that mean it should be able to write so, since the http is not connected right now.
GDQuest version place every data to be pushed or patched with into a dictionary of {"fields":fields} in which the "fields" contains all the data you'd like to push or update with. I think the upload structure might have changed, so it's no longer compatible with older technique.
The structure of the data then should looks like

var tobePush = {
		"fields":{
			"Auza":{"stringValue":"aaaaaaaaaaaw"}
		},
	}

notice the stringValue there. the datatype has to be defined first. Haven't seen again if your demo had one.
So basically, you just do

var data = { "aaaa" : {"stringValue" : "sample" } }
FirebaseQuest.push(data)

and that's it. it works (assuming you abide by your Firestore rule e.g. logged in).
what's going on?

You're actually looking in the wrong place. Firestore is not implemented via references. References are used only for the realtime database from Firebase. Look at FirebaseFirestore.gd, FirebaseFirestoreDocument.gd, and FirebaseFirestoreCollection.gd and you might get a better understanding of how to use it.
Also, GDQuest's version does not properly handle streamed updates, from what I remember. In the future, it would make sense not to look at their codebase and expect mine to behave the same way; I'm a professional Firebase developer, they are not.

I have issued #37 for continuation here as I mistakenly thought it didn't work while actually it works, but it uses Realtime Database instead


error on production build with 1.0.13

if i build app with NODE_ENV production then i have error.
//bundle.js:43779 Uncaught TypeError: Cannot convert undefined or null to object
var allowedProps = typeof wrapper === 'function' ? this.props : (0, _omit2.default)(this.props, Object.keys(Errors.propTypes));

i reproduce the problem in this repository
https://github.com/Maxim-Chugaev/react-redux-form_problem/tree/prop-types-problem

Fix published as 1.0.14


Remove full_text_extracts field from Preferences

Slack doesn't seem to be returning this field for us anymore, which
leads to crashes at startup unless we remove the field.

Thanks


Enter insert mode when editing in org-capture

Hey! I'm currently on the "learning org-mode step" of my Emacs spiritual journey. I've been slowly reassembling your org-mode implementation, and one thing I noticed that might be an improvement is to automatically enter insert mode after selecting a capture template to edit.
I achieved this by adding the following hook (in the org|init-capture):
    (add-hook 'org-capture-mode-hook 'evil-insert-state)

Your org-mode implementation is extremely well thought out, and I really appreciate it more and more as I dig into org. Anyways, just thought I'd share this tidbit if you are interested.

Thanks! org has been (and still is) the toughest part of this whole config, and isn't even close to finished yet.
In any case, I liked the suggestion so I pushed it upstream. Also, sorry for the commit spam, I was a bit overeager with that force push! Won't happen again.

That totally makes sense ... I feel like the hardest part to using org-mode is deciding how you want to use it. There's so many different possibilities and workflows, I keep finding myself with decision paralysis trying to move forward fleshing out the config. I've read / watched four or five different blog posts / screencasts and everyone uses it an entirely different way.
No worries on the commits, small price to pay for getting so much awesome stuff for free :)


Complete IETE Club Page with the details.

Mention here if you want to be assigned with this issue.

assign me


[FLINK-2104] [ml] Fixes problem with type inference for fallback implicits

The solution seems to be to fix the output type to Any. This has the consequence that all following operations will also be fallback operations if they don't support Any inputs. But this should be fine since only the first fallback operation will throw an exception stating why the operator could not find a suitable operation.

If there are no more objections, then I would merge it in the next hours.


session flashs, form styles, basic dmca form

NOTE TO SELF: install php7.0-bcmath before merging

:clap: not only does it fix the core problem, several auxiliary things cleaned up or added as well! Great work. It did not actually send the email for me, but I assume that is related to your personal note.


[fix] Upgrade jose4j to 0.9.4

Motivation
Pin Jose4j library lib to latest version. This is a dependency of K8S Java client API.
org.bitbucket.b_c:jose4j:jar:0.9.4:compile which has a fix for:
┌──────────────────────────────────────┬────────────────┬──────────┬────────┬───────────────────┬───────────────┬─────────────────────────────────────────────────────────────┐
│               Library                │ Vulnerability  │ Severity │ Status │ Installed Version │ Fixed Version │                            Title                            │
├──────────────────────────────────────┼────────────────┼──────────┼────────┼───────────────────┼───────────────┼─────────────────────────────────────────────────────────────┤
│ org.bitbucket.b_c:jose4j             │ CVE-2023-51775 │ MEDIUM   │ fixed  │ 0.9.3             │ 0.9.4         │ jose4j: denial of service (CPU consumption) via a large p2c │
│ (org.bitbucket.b_c-jose4j-0.9.3.jar) │                │          │        │                   │               │ (aka PBES2...                                               │
│                                      │                │          │        │                   │               │ https://avd.aquasec.com/nvd/cve-2023-51775                  │
└──────────────────────────────────────┴────────────────┴──────────┴────────┴───────────────────┴───────────────┴─────────────────────────────────────────────────────────────┘

Modifications

Verifying this change

[ ] Make sure that the change passes the CI checks.

(Please pick either of the following options)
This change is a trivial rework / code cleanup without any test coverage.
(or)
This change is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:
(example:)

Added integration tests for end-to-end deployment with large payloads (10MB)
Extended integration test for recovery after broker failure

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

[ ] Dependencies (add or upgrade a dependency)
[ ] The public API
[ ] The schema
[ ] The default values of configurations
[ ] The threading model
[ ] The binary protocol
[ ] The REST endpoints
[ ] The admin CLI options
[ ] The metrics
[ ] Anything that affects deployment

Documentation

[ ] doc
[ ] doc-required
[X] doc-not-needed
[ ] doc-complete

Matching PR in forked repository
PR in forked repository:

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 73.62%. Comparing base (bbc6224) to head (8ee7983).
Report is 54 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #22273      +/-   ##
============================================
+ Coverage     73.57%   73.62%   +0.04%
- Complexity    32624    32722      +98
============================================
  Files          1877     1887      +10
  Lines        139502   139391     -111
  Branches      15299    15287      -12
============================================
- Hits         102638   102620      -18
+ Misses        28908    28828      -80
+ Partials       7956     7943      -13

Flag
Coverage Δ

inttests
26.97% <ø> (+2.39%)
:arrow_up:

systests
24.40% <ø> (+0.08%)
:arrow_up:

unittests
72.88% <ø> (+0.04%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.
see 104 files with indirect coverage changes


Update bakery to work with separated git resources

@tomjw64 - I believe we addressed your comment on the fetch_map_resources.py script in that all images should be covered by looking for <image>. Were there any other comments on this PR to address?

Just a note on this PR: I'm holding off merging until this PR is approved / ready to merge to avoid breaking the pipeline for folks still pointing to master / latest when testing the git pipeline.


Check for valid LiveViewMode values

This addresses #3 and makes sure the value of LiveViewMode is a valid value. Looking at the WWDC session and Apple's own playgrounds, the only possible values for this key are VisibleByDefault and  HiddenByDefault.

Fantastic, thanks!


Move html2rss-web to the html2rss orga

[x] move repo
[x] change mergify
[x] change urls in all repos (website etc)

done.


[Agardeo to Choreo Hub] azure-websubhub production migration

Description:

$subject

Complete the production migration for azure-websubhub revamp.

Completed following tasks:

Update the migration script.

https://github.com/wso2-enterprise/azure-websubhub/pull/84
https://github.com/wso2-enterprise/azure-websubhub/pull/85

Work with SRE to team to run the migration script.
Post migration verification.
Fix data inconsistencies after migration.


Arithmetic

This should add math functions as requested in #208. It makes a best guess at the number format.

I'm slightly torn over whether these should be separate operations or not. I think they probably should, especially as it will make them easier to search for. At the very least, we should include the names of each of the arithmetic ops in the description so that they turn up in the search results.
What do people think about these ops being separate or accessible through one all-encompassing "Arithmetic" op? It feels to me that if we go down this route, all the logical ops (NOT, OR, AND etc.) should be encompassed in a similar manner. The same could probably be said for all the various types of encryption and encoding ops.

I was kinda torn over it too but I got to thinking that they all have the same args and doing similar functions (i.e. Math). With the logical ops they are doing similar functions and a lot of them have the same args but not all of them. I always forget about being able to search the operation though.

Awesome work again!
I've made the following small tweaks:

Moved the operation configs further up the OperationConfig.js file, as this has a bearing on where they appear in search results.
Modified the description for the "Sum" operation to include the phrase "add together" so it can be searched for using that term.
Changed the return types for each of the functions to "number" instead of "string". Each operation now returns NaN if it receives no valid input.
Re-ordered the DELIM_OPTIONS list so that "Line feed" is the first option. I personally think this will be the most common format required.

I like your use of the reduce function. It's nice, clean code.


ipv6: Allow IPv6 and HTTPS async connections

I noticed IPv6-only hosts cannot use the netlib for https:// URIs, tracked it down to this small bug. The endpoint protocol should be based on the results of the resolver, which takes the source host's stack + resolver's own priority into account.

LGTM
Good catch @theopolis -- thanks for this!


Add option to enable/disable followSymlinks of watchpack

Context:
You have project A in babel 6 and project B transpiled via babel 7. You run a webpack watch in package B directory to create bundle while you code.
If you npm link package B in package A,  and run webpack in package A, it will throw an error because he doesn't find dependencies.
A webpack's issue is open here: https://github.com/webpack/webpack/issues/1866
Solution
In webpack, it's possible to disable symlinks via symlinks: false. It works and babel (of package A) won't try to transpile package B. Sadly, if you do this, webpack won't react to change.
It's possible to change this behavior with followSymlinks in watchpack/chokidar. A PR has been added there to add this option: https://github.com/webpack/watchpack/pull/72
Now, we need the last part: add an exposed option in webpack configuration to enable/disable this watchpack new option. As stated by @TheLarkInn, it could be planned for webpack v5.

Why is this planned for webpack 5? Can't this land in webpack 4 as a feature request? I don't see how any of this would be a breaking change, this is primarily a non-breaking feature (or even a bugfix).

@SEAPUNK because it is breaking changes in webpack/watchpack and can't breaking some build for some developers.

Would also like to see this included in webpack 4 as it is a bug.

Agree with the sentiment here, this is either a backwards compatible feature or a bug-fix so there should be no issue shipping it with with v4 and it seems quite a few people would benefit from it.

@birdy-  will be happy to get an instructions or a link to where should i start fixing it
(new to submitting PR's to such a library)
thanks

Well, since MR on watchpack didn't land yet, we can't do a lot... Comments and upvote MR linked in inital post :(

AND FINALLY in Webpack 5

greatttt!!!!

@birdy @evilebottnawi - I saw this issue was marked as "D0 first commit" so I decided to pursue it :P
In researching this issue, I think I understand the problem, but I am unable to reproduce the issue with webpack not picking up on changes in symlinked module. I created a sample repo to demonstrate how I am unable to reproduce this issue. Is this still an issue? Or has it been fixed?
Thanks!

@immortal-tofu it looks like something you can close

I close issue for now. I think it's ok now.

For information, watchpack finally add the option for followSymlinks.
https://github.com/webpack/watchpack/pull/114
Now we need this option exposed in webpack.
Maybe webpack solved the issue with something else. I didn't try again.

@wittibs Basically the issue is simple:

Create a project A with a babel 6 and some plugin in .babelrc
Create a project B with babel 7 and some plugin in .babelrc. This project is transpiled "main": "dist/index.js"
Link project B in project A.
Start webpack with babel-loader in project A.
Project A's Webpack will try to transpile project B with babel 6 and even use their .babelrc file. Since you have 2 versions of babel, you have issue like "this module doesn't exist" because it exists in Project B but not in Project A...
A mess.

First solution:
So far, you can disable symlink with symlink option. It works, but you loose watch.  But at least, it won't try to transpile or read your linked module's babelrc.
Second solution:
You want the first solution BUT with the watch ability. Tadam, now you can since watchpack exposed followSymlinks option. Sadly Webpack doesn't allow you (yet) to use this param on watchOptions.
This is my summary. Maybe the problem described doesn't exist anymore, I switched to babel 7 everywhere so I encounter far less issue.
I keep close this issue, but tell me if it occurs again. Since watchpack added the option, if the problem still occurs, it can be solved by webpack team.

I believe this issue still exists regarding files in symlinked folders not being watched when symlink is set to false. It looks like followSymlinks was never exposed, so this should be re-opened, correct?

@EmpireJones fixed in webpack@5

Is this really fixed in v5?

I don't know if it's fixed in v5. I opened this issue 2 years ago and so far, I didn't see an answer "Problem solved see commit 35D43F53" ;)

Solved for webpack@5, you can the option here https://github.com/webpack/watchpack#api

Solved for webpack@5, you can the option here https://github.com/webpack/watchpack#api

That's on Watchpack (i.e. it now allows passing the flag). But what would an appropriate weback.config.js look like though to use the flag? I imagined something like this:
module.exports = {
  //...
  watchOptions: {
    followSymlinks: true
  }
};

Or to allow full flexibility on what is passed to the Watchpack constructor:
module.exports = {
  watchOptions: {
  //...
    watchpackConfig: {  // this probably overrides Webpack's default config options
      followSymlinks: true,
      // ...
    }
  }
};

But none of these are reflected in the WebpackOptions schema currently.

Oh, yes, we need update schema, WIP

Thanks, I just modified the source and changed followSysmlinks to true for now.

It'll be solved as part of webpack v5?
Someone has any recommendation how to implement nicely for webpack v4?

Not a particularly "nice" way, but since Webpack v5 is not yet released, a solution is to use the patch-package library to patch watchpack. Here's the patch diff, which also sets the depth parameter to 1 (needed for MacOS):

diff --git a/node_modules/watchpack/lib/DirectoryWatcher.js b/node_modules/watchpack/lib/DirectoryWatcher.js
index 940f616..210a1b7 100644
--- a/node_modules/watchpack/lib/DirectoryWatcher.js
+++ b/node_modules/watchpack/lib/DirectoryWatcher.js
@@ -53,8 +53,8 @@ function DirectoryWatcher(directoryPath, options) {
 	this.watcher = chokidar.watch(directoryPath, {
 		ignoreInitial: true,
 		persistent: true,
-		followSymlinks: false,
-		depth: 0,
+		followSymlinks: true,
+		depth: 1,
 		atomic: false,
 		alwaysStat: true,
 		ignorePermissionErrors: true,

@slint It works !

Implemented followSymlinks https://github.com/webpack/watchpack

Need to send a PR to watcherOptions https://github.com/webpack/webpack/blob/master/schemas/WebpackOptions.json#L3596

Is this working with webpack 5? I have enabled followSymlinks  but no luck.

devServer doesn't have this option

devServer doesn't have this option

Are you sure about this? I am a little confused based on docs which actually point to watchOptions inside devServer conf. Also, when I put { followSymlinks: true } in webpack conf, nothing happens but adding it to devServer actually makes my app to recompile (though it does not hot reloads).


add  to vx-shape

:rocket: Enhancements
I have added a new simple <Polygon /> component that takes sides and length props and does the rest.  I have added it to vx-shape
// hexagon
<Polygon sides={6} size={25} {...additionalProps} />

:memo: Documentation
I have added a demo to vx-demo and updated the vx-shape README

Thanks for the PR! Don’t want to leave you in suspense, I’m just traveling and away from computer until Monday. Will take a look then.

Hi @hshoff any thoughts on this?