Impact on the model change

This imporant issue comes in continuation of  MANGO Annotation Scope.
It continues the discussion whose content is recalled here:

have to be mapped. The rest can (must) be ignored.  The mapping
block represents a subset of the model.  If the model changes keep
the backward compatibility, the 'old' annotations remain consistant
and the interoperability between dataset mapped with different DM
versions is preserved.

Yes -- that's a minor version.  These aren't a (large) problem, and
indeed I'm claiming that our system needs to be built in a way that
clients don't even notice minor versions unless they really want to
(which, I think, so far is true for all proposals).

If you are saying that clients must be updated to take advantage of
new model features, you are right, whatever the annotation scheme
is, this is just because. new model class => new role => new  processing.

No, that is not my point.  My point is what happens in a major
version change.  When DM includes Coord and Coord includes Meas and
you now need to change Meas incompatibly ("major version), going to
Meas2 with entangled DMs will require new Coord2 and a DM2 models,
even it nothing changes in them, simply to update the types of the
references -- which are breaking changes.
With the simple, stand-alone models, you just add a Meas2 annotation,
and Coord and DM remain as they are.  In an ideal world, once all
clients are updated, we phase out the legacy Meas annotation.  The
reality is of course going to be uglier, but still feasible, in
contrast to having to re-do all DM standards when we need to re-do
Meas).

I very concerned by the question of the model changes. But I cannot figure out how a class describing a measure could be upgraded in a way that breaks the backward compatiblity.
Do you have some example?

On Wed, Mar 24, 2021 at 08:16:55AM -0700, Laurent MICHEL wrote:

I very concerned by the question of the model changes. But I cannot
figure out how a class describing a measure could be upgraded in a
way that breaks the backward compatiblity.
Do you have some example?

Right now, Meas depends on on Coords, and given that that hasn't seen
a lot of real-world usage, I'm rather sure we will want to fix
things because they're just too clumsy.  For instance, I suspect
everyone will want time0 to just be a JD, and nobody will get it
right if the offset and the value are in different frames (but that's
really just an example, don't get hung up on this particular example
too much).
If we change time0 to a float, that's a breaking change, and that
would then take Meas with it because it references Coords.
I give you in that particular instance, we could fudge it by just
adding a time0float attribute and tell people to not use time0, and
we'd get away with it, but that's one more wart I'll have to
apologise for when trying to get people to take up our standards.
Planning on accumulating hacks like that is something I'd really try
to avoid as long as there's an alternative that, for all I can tell,
will work at least as well for the plausible use cases.
More generally: The assumption that we've gotten something right
that has by and large not been consumed in practice seems dangerous
to me.  I'm saying that as someone who's been trying to fix things
that turned out wrong in Registry (which, mind you, has been a
surprisingly robust design).  I'm mentioning rights in VOResource
and, in particular, the whole caproles mess
(https://ivoa.net/documents/caproles/).

I cannot imagine a int -> real cast breaking anything. All clients are able do deal with this.
I remember an old discusssion about  data typing in models and sometime I regret the absence of a number type in VODML.
I tried to figure out different situations where model changes would be be letal.

Inapropriate downcasting (e.g. string -> numeric)
Splitting classes (stupid e.g. use one object from RA and another for DEC instead on one for [RA,DEC])
Merging classes (Coords attributes moved to Measure class)
Removing things (no more link between CoordSys and CoordFrame)
Renaming things (long -> longitude)

In the real life none is a real threat. The analogy with caprole must be carefully handle because our models are talking about physical quantities observed in the sky and caprole are talking about computer protocols that are completely   abstract and flexible things.
Considering the worst, all of these cases would break the interoperability which represents a seriouser issue than encompassing model versionning:

Data providers have to revise their annotation procedures
Client code have to deal with the new version and even worse, manage the cohabitation of both versions.

I would even say that having encompassing model is safer, since the upgrade process garanties that all components still can work together both structuraly and semanticaly.

On Fri, Mar 26, 2021 at 04:14:12AM -0700, Laurent MICHEL wrote:

I cannot imagine a int -> real cast breaking anything. All
clients are able do deal with this.

The time0-thing would be a change from a complex time instance to
just a real, so that would hurt.

Inapropriate downcasting (e.g. string -> numeric)
Splitting classes (stupid e.g. use one object from RA and another for DEC instead on one for [RA,DEC])
Merging classes (Coords attributes moved to Measure class)
Removing things (no more link between CoordSys and CoordFrame)
Renaming things (long -> longitude)

In the real life none is a real threat. The analogy with caprole
must be carefully handle because our models are talking about
physical quantities observed in the sky and caprole are talking
about computer protocols that are completely   abstract and
flexible things.

Well, our models are just as flexible, and indeed going from STC1
to STC2 to MCT we've already seen two breaking changes (and,
actually, many more in between).  So, frankly, I don't buy the
reasoning.

Considering the worst, all of these cases would break the
interoperability which represents a seriouser issue than
encompassing model versionning:

Data providers have to revise their annotation procedures
Client code have to deal with the new version and even worse,
manage the cohabitation of both versions.

That's the flag day that I claim non-entangled models will prevent:
Data providers have years or decades to migrate -- and most of their
annotations won't even have to change, as they are unaffected from
other model's version changes.
Clients can easily pick the most appropriate annotation they can deal
with; this why the use case
https://github.com/msdemlei/astropy#choosing-the-most-expressive-annotation
is something I'd really like to see canonised...

going from STC1 to STC2

Good example, I guess you agree on that the major concern about moving from STC1 to MCT is not the update of the host models (that do not exist any way). This is my point.
Let's imagine that I've e.g. a model named lm_dm_1 that embeds stc1.13,  Char_1 and dataset_1.5
Now stc1.13 has been moved to stc2:
In consequence I've to update my model to lm_dm_2 (stc2,  Char_1 , dataset_1.5).
My points are

upgrading lm_dm is not big deal
using lm_dm_2 might be safer than using individual (stc2,  Char_1 , dataset_1.5) instances since this guaranties that all three models are compliant each to other (e,g. vocabulary mismatches).
last but mot least: lm_dm (1 or 2) keeps giving a complete description of the modeled objects (e.g. Cubes) out of the scope of any particular data container. I know you deny this requirement, but I insist to claim that this the interoperability key.

On Wed, Mar 31, 2021 at 07:05:13AM -0700, Laurent MICHEL wrote:

going from STC1 to STC2

Good example, I guess you agree on that the major concern about
moving from STC1 to MCT is not the update of the host models (that
do not exist any way). This is my point.

Not sure what "host model" means, but certainly no STC1 client will
understand any STC2, even if we had an annotation scheme covering
both (and, frankly, I think a really good STC model would deviate
even further from STC1 than current Coords, but that's another
matter).

Let's imagine that I've e.g. a model named lm_dm_1 that embeds
stc1.13,  Char_1 and dataset_1.5
Now stc1.13 has been moved to stc2:
In consequence I've to update my model to lm_dm_2 (stc2,
Char_1 , dataset_1.5).
My points are

upgrading lm_dm is not big deal

Not for you as the model author, of course.  But for all of your
clients.  Which break, and cannot read/find Char_1 and dataset_1.5
metadata any more, although they'd be perfectly capable of
understanding what's in there if it weren't for the changed
container.
True, perhaps updating them would again be no big deal.  Except that
the client code still needs to be maintained and released and, most
difficult of all, distributed.  All of which are big deals in
practice in a huge and diverse system like the VO.
This kind of, if you will, wanton destruction of existing and
perfectly maintainable functionality is what I think most of this
discussion is (or ought to be) about.

using lm_dm_2 might be safer than using individual (stc2,
Char_1 , dataset_1.5) instances since this guaranties that all
three models are compliant each to other (e,g. vocabulary
mismatches).

Umm -- could you be a bit more specific here?  What annotation in
a non-entangled Char_1 could possibly break when you change stc?
And since you're mentioning vocablaries, given we're in RFC for that
I'd be particularly interested in your concerns about their
interaction with DMs and their mutual compatibility.

last but mot least: lm_dm (1 or 2) keeps giving a complete
description of the modeled objects (e.g. Cubes) out of the scope of
any particular data container. I know you deny this requirement,
but I insist to claim that this the interoperability key.

Perhaps this is a point we'll have to discuss interactively, because
I can't help feeling it's obvious that it's the other way round: If
you kill all DM annotations for all existing clients if you just
change a single DM incompatibly, that's then end of interoperability
as soon as evolution sets in.
Sure, you can hope that models won't evolve, or you can hope that by
that time all you have to replace are a few wep apps on AstroBook
(TM) and hence you don't need iteroperability in the first place --
but both of these I'd consider rather undesirable.

On this topic, I'm finding myself agreeing more with Markus. (ack!)
To a certain level anyway, as I agree with Laurent's assertion 3 in this comment
Models are going to change, I think that's pretty much a given.

as we fold in new products (Mango, TimeSeries, SED)
broaden usage domains. eg: Radio or Planetary needs different Target type than we have spec'd in Dataset
or just forgot something: "Spectrum doesn't support Echelle Spectra".

VODML compliant models import specific versions of their dependencies

to make new features available under the parent models will require an update to the model (VODML/XML).

This is true, whether a major version or minor version change.
What is the impact?

At the rate we are cranking out models in the VO, I don't think this is going to be a burden
At the rate we SHOULD be cranking out models in the VO, this could be annoying.  This annoyance may be mostly on our side, updating the VODML/XML for the model hierarchy just to change the dependency.
Operationally a model namespace is constant for minor version changes, so users should only be impacted when a major version change is made.

Impact on clients/providers should be marginal.  IF they want to produce/use the newer version content, they make the appropriate changes (if any, other than recognizing the new namespace) and are done.  This is needed even to handle new content in minor version changes to the models.
Most changes would be contained to the parser/writer level, not to the classes/objects the implementations convert them to.  eg: the parser package I'm using (rama) will produce an AstroPy SkyCoord for any version of meas:Point.  No change to the application using it is required.
If they are integrating a major version change, then they should expect to be updating their implementation.

I don't think decoupling the models makes this go away:

If a client reads Cube/Mango instances, and there is a change to Meas model, and they want to use/produce the new Meas content, is it any easier to test/verify

Mango-vA with (Meas-vA or Meas-vB) content instead of
(Mango-v1 with Meas-v1) or (Mango-vB with Meas-vB)

Where it DOES have a big impact is on the annotation.  This is probably a good case to mock-up (since we don't have multiple versions of any models) and annotate.
With the current model/annotation relations, I believe the client would need to:

Annotate 2 versions of the parent model objects (Source/SparseCube) to accommodate the 2 flavors of Measurement leaves.

looking at Standard_Properties and TimeSeries examples (Mapping syntax and ModelInstanceInVot syntax)

~100:150 lines of annotation for Source instance with 5 Properties; includes 13:25 per Measure
~85:140 lines of annotation for SparseCube with tiny Dataset + 3 Observables; includes 7:10 per Measure (no errors)

If decoupled

Can annotate 1 version of parent model, and EITHER version of Measure at the leaves.

adding no additional annotation lines

Or in some way allow annotation of BOTH versions of Measure at leaves.

adding 7:25 additional lines of annotation per Measure that has multiple versions.

If we are considering the case I'm currently working, Master(Source) with associated Detections(Source) and associated LightCurve(SparseCube), this could add up to serious real estate.
Up to now, I've considered all this "The cost of doing business.", and am comfortable with that position.
But, after seeing the ModelInstance in Mango, maybe this needs more serious consideration.  I had an idea this morning, inspired by this, which may be a good compromise.  It could allow looser coupling of models, but still have definitive links between them for verification/interoperability.  (ie: no ivoa:anyType stuff).  Once I've thought that through a bit, I'll post it in a new thread.

[@msdemlei] And since you're mentioning vocablaries, given we're in RFC for that
I'd be particularly interested in your concerns about their
interaction with DMs and their mutual compatibility.

Message understood....
...  but for the immediate time, the discussion is rather focused on  DM concept itself.

Perhaps this is a point we'll have to discuss interactively..

Sure .I'm unable to connect the existence of Cube with the disaster that you announce

I'm in line with the @mcdittmar's summary.
Yay!  I always like hearing that!

My expectation is that the introduction of new players (e.g. radio) won't break existing stuff but introduce new patterns.

e.g. Radio dish FoVs seem a bit more complex that simple cones.

I think the most likely breaks will come from us having concrete objects defined in a model which we later find needs to be abstract in order to support branching by different domains.  It is the main reason I have elements like the abstract Uncertainty type in the Measurements model.. to help guard against major version change.  I grudgingly removed the abstract Point from Coords on the last iteration, and with this Mango work, we're finding an interest in restoring the space-centric LonLatPoint/SphericalPoint. This would be a major version update in Coords.

Data provider will have to annotate with both versions until all clients supports them (will likely never occur)
Clients will have to support both versions as soon as a one data provider use the new version (will likely occur).
I was thinking about this, and it seems more likely (no evidence) that the clients would prefer to output as V1 OR V2 at the user's request, rather than annotating to both in the same output.

I agree that the condition for the risk (as pointed by @msdemlei) of breaking models with new features to be very low is that models have abstract classes. i.e. things that can be extended without altering existing stuff.
MANGO showed up (too much apparently) the ability of extending MCT without breaking anything.

On Wed, Apr 07, 2021 at 05:59:07AM -0700, Mark Cresitello-Dittmar wrote:

I was thinking about this, and it seems more likely (no evidence)
that the clients would prefer to output as V1 OR V2 at the user's
request, rather than annotating to both in the same output.

Hm... First, it'll generally be the servers that output annotated
data, with clients consuming them.
That has the consequence that annotated data may sit around for
decades without being touched.  And while it is true that on that
time frame, it's clear that certain annotations won't be understood
any more by new clients, with small, independent data models there's
a good chance that most annotations will still work (Modern client:
"Ah, this is a position in ICRS; but this meas:NaiveMeasurement thing
that's in there I've forgotten about ages ago") whereas with the big
God model it's virtually certain the new client will not recognise
anything ("what's this ivoa-timeseries1:Root thing again?").
Conversely, clients have a way to hang around for decades in
specialised workflows.  Again, with small, independent models, they
will keep understanding the majority of the annotations even if
confonted with data produced long after they were written, whereas
they'll be entirely broken on the first major change with the God
model.
In the VO, you just can't "roll out version 2" -- you'll always have
a wild mixture of modern and legacy services and modern and legacy
clients, even 20 years from now.  That's why it's so useful to limit
the damage radius of breaking changes.

On Wed, Mar 31, 2021 at 02:05:18AM -0700, Bonnarel wrote: My 2 cents on this. What kind of situation can we imagine for a model change ? Where does it has impact when we are considering data transport using VOTable ? Why is one (or several) of the models changing ? I can imagine two reasons:
Well, the most common reason is: we simply did it wrong. As someone who did it wrong several times already, I think I'm entitled do say that. To mention my worst goof: Using ParamHTTP interfaces in TAPRegExt, which blew up badly more than half a decade later ("caproles"). It happened because I hadn't understood the full problem, didn't see the long-term consequences, and generally shunned the work of defining an extra type for TAP-like interfaces. You could say other people are less lazy, think deeper, and see farther, and they probably do. But who knows, perhaps one day I'll make a DM, and then it'd be reassuring to know that if I get it as wrong as the interfaces in TAPRegExt, the VO can shed by mistake without taking the whole DM system with it.

Well I think "do it wrong" is close to "does not allow an optimal interpretation". Of course this can always happen with everybody. This doesn't imply we have to let the client manage alone with the relationships between our break and pieces

The key thing is the independance of the annotation and of the data.
Could you explain this a bit more? Do you mean lexical independence (e.g., annotation sits in an element of its own rather than, say, in FIELD's utype attributes)? Or semantic independence (in which case you'd have to explain how that should work)? Or yet something else? To me, I'd say the annotation depends strongly, and ideally "injectively" on the data's structure (i.e., different structures will have different annotations) and not at all on the data (which rules out follies like having pieces of photometry metadata in column values). Conversely, data and data structure do not depend at all on the annotation (which is less obvious than it may sound, but it in partiuclar means that you can attach as many different annotations to data structures as you like).

I clearly meant lexical independence. We can clearly imagine two strategies : either youy really map your data structure onto your model (and this requires a new schema each time you change the model - the things we did with xml schema distinct for each DM 15 years ago) or you add  an (evoluating) mapping layer on top of more stable  (VO)Tables.

whereas they'll be entirely broken on the first major change with the God model. In the VO, you just can't "roll out version 2" -- you'll always have a wild mixture of modern and legacy services and modern and legacy clients, even 20 years from now. That's why it's so useful to limit the damage radius of breaking changes.
But we are not dealing with God models when speaking of TimeSeries or sparse Cubes or Source model with Parameters We have real situations which are meaningful cross projects and cross wavelengths (or even cross messengers) and want to interoperate them. Rapidly the way the things organize become complex. Very often we find tables with several different positions, times, magnitudes. Is there an independant time only  and the other depend of ot like fluxes or whatever ? several (see ZTF and Beta Lyrae in Vizier examples ) ? are all the parameters independant (event list)?  all but one (eg flux in a regularly sampled cube) ? Can we use the relationships between these parameters or axes to transform data from one data type to another one ?  Providers may want to help users and clients do such things to compare or combine data. I imagine that with separate Cube-with-one-independant-axis-only  and Cordinates annotation it will rapidly be  a mess for the client to find its way.

On Wed, Apr 07, 2021 at 05:59:07AM -0700, Mark Cresitello-Dittmar wrote: I was thinking about this, and it seems more likely (no evidence) that the clients would prefer to output as V1 OR V2 at the user's request, rather than annotating to both in the same output.
Hm... First, it'll generally be the servers that output annotated data, with clients consuming them. That has the consequence that annotated data may sit around for decades without being touched. And while it is true that on that time frame, it's clear that certain annotations won't be understood any more by new clients, with small, independent data models there's a good chance that most annotations will still work (Modern client: "Ah, this is a position in ICRS; but this meas:NaiveMeasurement thing that's in there I've forgotten about ages ago") whereas with the big God model it's virtually certain the new client will not recognise anything ("what's this ivoa-timeseries1:Root thing again?"). Conversely, clients have a way to hang around for decades in specialised workflows. Again, with small, independent models, they will keep understanding the majority of the annotations even if confonted with data produced long after they were written, whereas they'll be entirely broken on the first major change with the God model. In the VO, you just can't "roll out version 2" -- you'll always have a wild mixture of modern and legacy services and modern and legacy clients, even 20 years from now. That's why it's so useful to limit the damage radius of breaking changes.

But we are not dealing with God models when speaking of TimeSeries or sparse Cubes or Source model with Parameters We have real situations which are meaningful cross projects and cross wavelengths (or even cross messengers) and want to interoperate them. Rapidly the way the things organize become complex. Very often we find tables with several different positions, times, magnitudes. Is there an independant time only and the other depend of ot like fluxes or whatever ? several (see ZTF and Beta Lyrae in Vizier examples ) ? are all the parameters independant (event list)? all but one (eg flux in a regularly sampled cube) ? Can we use the relationships between these parameters or axes to transform data from one data type to another one ? Providers may want to help users and clients do such things to compare or combine data. I imagine that with separate Cube-with-one-independant-axis-only and Cordinates annotation it will rapidly be a mess for the client to find its way.

On Thu, Apr 08, 2021 at 09:41:16AM -0700, Bonnarel wrote:

But we are not dealing with God models when speaking of TimeSeries
or sparse Cubes or Source model with Parameters We have real

Well, as far as I can work out the idea is that there is one root
node and everything else is then relative to it; it is this "there's
one big class describing the whole of a document" is what I call God
model.
My skepticism to them is not only aesthetic: Having them means that if
you don't understand this root node, you can't use any annotation,
and that a client that knows how to find, say, a value/error in a
time series will have to be taught anew how to do it in an object
catalogue (and trouble with versioning, and much more; there's plenty
of good reasons why the God object is considered an antipattern).
Not to mention, of course, that few programmers will appreciate that
you're trying to impose your data structures on them.

situations which are meaningful cross projects and cross
wavelengths (or even cross messengers) and want to interoperate
them. Rapidly the way the things organize become complex. Very
often we find tables with several different positions, times,
magnitudes. Is there an independant time only and the other depend
of ot like fluxes or whatever ? several (see ZTF and Beta Lyrae in
Vizier examples ) ? are all the parameters independant (event
list)? all but one (eg flux in a regularly sampled cube) ? Can we
use the relationships between these parameters or axes to transform
data from one data type to another one ? Providers may want to help
users and clients do such things to compare or combine data. I
imagine that with separate Cube-with-one-independant-axis-only and
Cordinates annotation it will rapidly be a mess for the client to
find its way.

I like the concrete examples and questions, because with them you can
test whether stuff works.  And I contend all of the questions are
rather straightforwardly answerable by the simple scheme I'm
proposing over in https://github.com/msdemlei/astropy.
If you disagree, what sort of workflow do you think won't be covered
by it?

On Thu, Apr 08, 2021 at 09:41:16AM -0700, Bonnarel wrote: But we are not dealing with God models when speaking of TimeSeries or sparse Cubes or Source model with Parameters We have real
Well, as far as I can work out the idea is that there is one root node and everything else is then relative to it; it is this "there's one big class describing the whole of a document" is what I call God model. My skepticism to them is not only aesthetic: Having them means that if you don't understand this root node, you can't use any annotation, and that a client that knows how to find, say, a value/error in a time series will have to be taught anew how to do it in an object catalogue (and trouble with versioning, and much more; there's plenty of good reasons why the God object is considered an antipattern). Not to mention, of course, that few programmers will appreciate that you're trying to impose your data structures on them.
Well, there is an old consensus in IVOA that we are dealing with "datasets" or "dataproducts" and that dataproduct_type makes sense. A top level model is just some common description of the formal internal relationships between various parts of these data products consistent with the definition of the dataproduct type. DataProvider should succeed in agreeing on what is required and what is optional there. The constraint on application programmers will not come artificially from datamodelers but from DataProviders interoperability   requirements

situations which are meaningful cross projects and cross wavelengths (or even cross messengers) and want to interoperate them. Rapidly the way the things organize become complex. Very often we find tables with several different positions, times, magnitudes. Is there an independent time only and the other depend of it like fluxes or whatever ? several (see ZTF and Beta Lyrae in Vizier examples ) ? are all the parameters independent (event list)? all but one (eg flux in a regularly sampled cube) ? Can we use the relationships between these parameters or axes to transform data from one data type to another one ? Providers may want to help users and clients do such things to compare or combine data. I imagine that with separate Cube-with-one-independant-axis-only and Cordinates annotation it will rapidly be a mess for the client to find its way.
I like the concrete examples and questions, because with them you can test whether stuff works. And I contend all of the questions are rather straightforwardly answerable by the simple scheme I'm proposing over in https://github.com/msdemlei/astropy. If you disagree, what sort of workflow do you think won't be covered by it?

Well as far as I understood this works because the raw data are rather simple. But would happen with a catlog like this :
Shenavrin et al, Astronomicheskii Zhurnal, 2011, Vol. 88, No. 1, pp. 34–85. available in Vizier.
Here obviously there is one single independant time and the others parameters, including the other times depend of it. In addition there are several instances of TimeSeries in the same catalog (because there are several sources). Why shoud we discover all the times and then discover which one is the independent in another annotation ?
In the following catalog http://vizier.u-strasbg.fr/viz-bin/VizieR?-source=J/ApJ/790/L21&-to=3 . All parameters have the same importance. It's an event list. Why should we not know that from the top ?

On Wed, Apr 21, 2021 at 12:43:32AM -0700, Bonnarel wrote:

Well, there is an old consensus in IVOA that we are dealing with
"datasets" or "dataproducts" and that dataproduct_type makes sense.

...which still makes it desirable that a "Position", say, works the
same way regardless of dataproduct type.  The little inconsitencies
between how, say, SIAP and SSAP deal with positions have been driving
our developers (including me) crazy for a long time.  Let's not do
more of that.

Well as far as I understood this works because the raw data are
rather simple. But would happen with a catlog like this :
Shenavrin et al, Astronomicheskii Zhurnal, 2011, Vol. 88, No. 1,
pp. 34–85. available in Vizier.

That's
https://vizier.u-strasbg.fr/viz-bin/VizieR-3?-source=J%2FAZh%2F88%2F34&-out.max=50&-out.form=HTML Table&-out.add=_r&-out.add=_RAJ,_DEJ&-sort=_r&-oc.form=sexa
, right?

Here obviously there is one single independant time and the others
parameters, including the other times depend of it. In addition
there are several instances of TimeSeries in the same catalog
(because there are several sources). Why shoud we discover all the
times and then discover which one is the independent in another
annotation ?

This is a catalogue, not a time series, unless I'm badly mistaken.
What's in there is something like the photometry point we (I think)
once had in SDM2 and that might make it to a PhotPoint class in
PhotDM2.  In which case this would look like this (to avoid
shocking tag soup, I'm using SIL annotation
(http://docs.g-vo.org/DaCHS/ref.html#annotation-using-sil), but it'll
be the same in XML):
(phot2:PhotPoint) {
  value: @Jmag
  epoch: @JDJ
}
(phot2:PhotCal) {
  value: @Jmag
  bandName: J
  spectralLocation: 1.248e-6
}
(meas:Measurement) {
  value: @Jmag
  naiveError: @e_Jmag
  flag: @u_Jmag
}

(where I'm not yet convinced that flag in measurement is a good idea,
but see parallel discussion with Gilles).
And so on for the other bands.
Or do I mis-understand your intention here?

In the following catalog
http://vizier.u-strasbg.fr/viz-bin/VizieR?-source=J/ApJ/790/L21&-to=3
. All parameters have the same importance. It's an event list. Why
should we not know that from the top ?

So, that would be
(ds:Dataset) {
  productType: event
}

(ndcube:Cube) {
  independentAxes: [@Arrival]
  dependentAxes: [@Theta, @E, @RAJ2000, @DEJ2000]
}

(stc2:Position) {
  ... (annotation for REJ2000, DEJ2000
}

What would you be missing?

Hi Markus
Le 22/04/2021 à 10:30, msdemlei a écrit :

On Wed, Apr 21, 2021 at 12:43:32AM -0700, Bonnarel wrote:

Well, there is an old consensus in IVOA that we are dealing with
"datasets" or "dataproducts" and that dataproduct_type makes sense.

...which still makes it desirable that a "Position", say, works the
same way regardless of dataproduct type. The little inconsitencies
between how, say, SIAP and SSAP deal with positions have been driving
our developers (including me) crazy for a long time. Let's not do
more of that.
OK. But are we not working on something which will make coordinates more
stable ?

Well as far as I understood this works because the raw data are
rather simple. But would happen with a catlog like this :
Shenavrin et al, Astronomicheskii Zhurnal, 2011, Vol. 88, No. 1,
pp. 34–85. available in Vizier.

That's
https://vizier.u-strasbg.fr/viz-bin/VizieR-3?-source=J%2FAZh%2F88%2F34&-out.max=50&-out.form=HTML Table&-out.add=_r&-out.add=_RAJ,_DEJ&-sort=_r&-oc.form=sexa
, right?

Yes but only table 3 !

Here obviously there is one single independant time and the others
parameters, including the other times depend of it. In addition
there are several instances of TimeSeries in the same catalog
(because there are several sources). Why shoud we discover all the
times and then discover which one is the independent in another
annotation ?

This is a catalogue, not a time series, unless I'm badly mistaken.
Table 3 is indeed a collection of TimeSeries See screenshot below
What's in there is something like the photometry point we (I think)
once had in SDM2 and that might make it to a PhotPoint class in
PhotDM2. In which case this would look like this (to avoid
shocking tag soup, I'm using SIL annotation
(http://docs.g-vo.org/DaCHS/ref.html#annotation-using-sil), but it'll
be the same in XML):
(phot2:PhotPoint) {
value: @Jmag
epoch: @JDJ
}
(phot2:PhotCal) {
value: @Jmag
bandName: J
spectralLocation: 1.248e-6
}
(meas:Measurement) {
value: @Jmag
naiveError: @e_Jmag
flag: @u_Jmag
}

I think we were not dealing with the same table. Table 3 has several
times and plenty of raws for the same source. But the time are not
independant between them. (because at a given time you observe at one
band and then you rotate rapidlly the filter wheel and this gives you
little time shifts with respect to the "main" time)

(where I'm not yet convinced that flag in measurement is a good idea,
but see parallel discussion with Gilles).
And so on for the other bands.
Or do I mis-understand your intention here?

In the following catalog
http://vizier.u-strasbg.fr/viz-bin/VizieR?-source=J/ApJ/790/L21&-to=3
. All parameters have the same importance. It's an event list. Why
should we not know that from the top ?

So, that would be
(ds:Dataset) {
productType: event
}

(ndcube:Cube) {
independentAxes: [@Arrival]
dependentAxes: [@Theta, @E, @RAJ2000, @DEJ2000]
}

(stc2:Position) {
... (annotation for REJ2000, DEJ2000
}

I don't think there are dependant axes. It is a list of "VoEvents"-like
things which occurs at some position, some time, some energy, etc..;
All these variables are independant with respect to each other

What would you be missing?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/ivoa/dm-usecases/issues/24#issuecomment-824648857,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AMP5LTEF2B2H4NKRH4JLLQLTJ7NATANCNFSM4ZXO3FWQ.

On Thu, Apr 22, 2021 at 08:39:58AM -0700, Bonnarel wrote:

Le 22/04/2021 à 10:30, msdemlei a écrit :

That's
https://vizier.u-strasbg.fr/viz-bin/VizieR-3?-source=J%2FAZh%2F88%2F34&-out.max=50&-out.form=HTML Table&-out.add=_r&-out.add=_RAJ,_DEJ&-sort=_r&-oc.form=sexa
, right?

Yes but only table 3 !

Ah, ok.
The first part is that here, there would be several time series, so
you'd say
(ndcube:Cube) {
independentAxes: [@JDJ]
dependentAxes: [@Jmag]
}

(ndcube:Cube) {
independentAxes: [@JDH]
dependentAxes: [@Hmag]
}

(ndcube:Cube) {
independentAxes: [@KDH]
dependentAxes: [@Kmag]
}

(this part is as for a normalised Gaia time series).
The "would be" above is because again this is highly de-normalised in
that the table is (logically) a union of different (sets of) time
series.  You immediately notice that something is wrong with this
table because sorting by Jmag, say, is physically meaningless.
I'm still claiming that if our annotation tries to include relational
algebra, we still won't be able to fix the urgent use cases five
years from now (don't laugh -- people have felt over-confident in the
DM business fifteen and ten years ago, and we still have nothing).
Let's first get the annotation of actual, normalised tables right and
then figure out if there's a point for doing structural modifications
(normalisation) later.
Excursion: That later spec, I submit, should again refrain from
re-inventing relational algebra and just embed a suitable subset of
ADQL, perhaps like that:
(table:virtualtable) {
  id: timeseries-betLyr
  definition: "select * from table3 where name='beta Lyr'"
}

(table:virtualtable) {
  id: timeseries-iotaBoo
  definition: "select * from table3 where name='iota Boo'"
}

-- and these things would then need to receive annotations as if they
were normal VOTable TABLEs.
But, really, really, let's postpone this.  This is complicated, and
we'll need years to get it right.  And it'll make the standard so
complex that we'll have a hard time getting people on board.  Let's
not forget the STC-1 lesson.
And until then, we can still annotate table3 with photometry, time
metadata, and everything.  It's just that users will have to manually
pick out time series for the individual objects.

In the following catalog
http://vizier.u-strasbg.fr/viz-bin/VizieR?-source=J/ApJ/790/L21&-to=3
. All parameters have the same importance. It's an event list. Why
should we not know that from the top ?

So, that would be
(ds:Dataset) {
productType: event
}

(ndcube:Cube) {
independentAxes: [@Arrival]
dependentAxes: [@Theta, @E, @RAJ2000, @DEJ2000]
}

(stc2:Position) {
... (annotation for RAJ2000, DEJ2000
}

I don't think there are dependant axes. It is a list of "VoEvents"-like
things which occurs at some position, some time, some energy, etc..;
All these variables are independant with respect to each other

Well, of course -- each dependent axis just depends on the
independentAxes, not on each other.  That's the definition.
So, I'm not sure what your objection aims at -- perhaps if you stated
what the annotation you are missing (?) would be enabling I could be a
bit more specific?
In the way of an example, the use case for the present annotation is:
"A plotting component wants to present the user with a sensible
choice of what to plot against what" -- in this case, you'd always
have Arrival on the abscissa, and any of Theta, E, or the coordinates
on the ordinate.  A particularly smart plotting programme would
notice that RA and Dec make up a position and would offer to bring up
a plot type that draws positions of time when the user wants to plot
any of RA or Dec.

Just some thoughs on the impact on the model changes.
The @msdemlei  client README, makes the assumption that the VOTable is delivered with distinct annotations for 2 different versions of the same model (Coords). I do not thing that this case is the most likely because the annotation process is a very though job (ask @gilleslandais) and I doubt that data curators will duplicate their efforts to support multiple variants of the same model.
The most likely situation is to have a client trying to put together (e.g. xmatch ) data sets annotated with different versions on the same model.

How to cross match data annotated with CoordsV3 against data annotated with CoordsV4.?
How to cross match data annotated with CubeV3 against data annotated with CubeV4.?

This a critical point that cannot be worked around just by using un-entangled models.

Hi Markus,
Le 23/04/2021 à 08:50, msdemlei a écrit :

On Thu, Apr 22, 2021 at 08:39:58AM -0700, Bonnarel wrote:

Le 22/04/2021 à 10:30, msdemlei a écrit :

That's

https://vizier.u-strasbg.fr/viz-bin/VizieR-3?-source=J%2FAZh%2F88%2F34&-out.max=50&-out.form=HTML Table&-out.add=_r&-out.add=_RAJ,_DEJ&-sort=_r&-oc.form=sexa

, right?

Yes but only table 3 !

Ah, ok.
The first part is that here, there would be several time series, so
you'd say
(ndcube:Cube) {
independentAxes: [@JDJ]
dependentAxes: [@Jmag]
}

(ndcube:Cube) {
independentAxes: [@JDH]
dependentAxes: [@Hmag]
}

(ndcube:Cube) {
independentAxes: [@KDH]
dependentAxes: [@Kmag]
}

(this part is as for a normalised Gaia time series).

If you read the paper, I don't think we can consider this as several
monocolor time series. Of course we have as much TimeSeries as we have
astronomical sources.
It is a complex TimeSeries with one independant Time (the one you want)
and several complex "Parameters" made of Time and magnitude
Just because all the times on a row are very close to each other, they
are part of  the same "observation" where you rotate the filter wheel.
If you look at this as a Time Series of a set of 5 such Paremeters there
is a degenerescence on the time chosen as independant : it is mapped
once as independant time and once as it's color dependant time.

The "would be" above is because again this is highly de-normalised in
that the table is (logically) a union of different (sets of) time
series. You immediately notice that something is wrong with this
table because sorting by Jmag, say, is physically meaningless.

We cannot say they are wrong, I think this is one of the prerequisite of
the workshop.
If you read the paper what they have done makes sense.
Take the tables as they are and try to annotate them  with a
sufficiently smart model

I'm still claiming that if our annotation tries to include relational
algebra, we still won't be able to fix the urgent use cases five
years from now (don't laugh -- people have felt over-confident in the
DM business fifteen and ten years ago, and we still have nothing).

I was not about relational algebra (apart from selecetin the source) on
this one but trying to have a TS model able to manage such use case
Beside the independant TimeSeries we need structured dependant
Properties. I think it's possible to do that using Mango Parameters.

Let's first get the annotation of actual, normalised tables right and
then figure out if there's a point for doing structural modifications
(normalisation) later.
Excursion: That later spec, I submit, should again refrain from
re-inventing relational algebra and just embed a suitable subset of
ADQL, perhaps like that:
(table:virtualtable) {
id: timeseries-betLyr
definition: "select * from table3 where name='beta Lyr'"
}

(table:virtualtable) {
id: timeseries-iotaBoo
definition: "select * from table3 where name='iota Boo'"
}

-- and these things would then need to receive annotations as if they
were normal VOTable TABLEs.

Well the proposal in ModelInstanceIntoVOT is to do that using "GROUP_BY".
This GROUP_BY helps to define Instances  (which maybe instances of
complex classes) below.

But, really, really, let's postpone this. This is complicated, and
we'll need years to get it right. And it'll make the standard so
complex that we'll have a hard time getting people on board. Let's
not forget the STC-1 lesson.
And until then, we can still annotate table3 with photometry, time
metadata, and everything. It's just that users will have to manually
pick out time series for the individual objects.

I think Laurent's code allows to extract the TimeSeries of one source
just by using this "GROUP_BY" feature
If we don't try to map these things will we force authors and providers
to shape their tables as WE want them to do ?

In the following catalog

http://vizier.u-strasbg.fr/viz-bin/VizieR?-source=J/ApJ/790/L21&-to=3

. All parameters have the same importance. It's an event list. Why
should we not know that from the top ?

So, that would be
(ds:Dataset) {
productType: event
}

(ndcube:Cube) {
independentAxes: [@Arrival]
dependentAxes: [@Theta, @E, @RAJ2000, @DEJ2000]
}

(stc2:Position) {
... (annotation for RAJ2000, DEJ2000
}

I don't think there are dependant axes. It is a list of "VoEvents"-like
things which occurs at some position, some time, some energy, etc..;
All these variables are independant with respect to each other

Well, of course -- each dependent axis just depends on the
independentAxes, not on each other. That's the definition.
So, I'm not sure what your objection aims at -- perhaps if you stated
what the annotation you are missing (?) would be enabling I could be a
bit more specific?
In the way of an example, the use case for the present annotation is:
"A plotting component wants to present the user with a sensible
choice of what to plot against what" -- in this case, you'd always
have Arrival on the abscissa, and any of Theta, E, or the coordinates
on the ordinate. A particularly smart plotting programme would
notice that RA and Dec make up a position and would offer to bring up
a plot type that draws positions of time when the user wants to plot
any of RA or Dec.

I don't think Time has something special in this example. All parameters
are INDEPENDANT,
You could also decide to see this as a  multidimensional Sky Cube where
values are time, energy and teta

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/ivoa/dm-usecases/issues/24#issuecomment-825432999,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AMP5LTHF2WJ67DBJ3JD4ZC3TKEKC7ANCNFSM4ZXO3FWQ.

On Mon, Apr 26, 2021 at 01:55:28AM -0700, Bonnarel wrote:

Le 23/04/2021 à 08:50, msdemlei a écrit :

(ndcube:Cube) {
independentAxes: [@JDJ]
dependentAxes: [@Jmag]
}

(ndcube:Cube) {
independentAxes: [@JDH]
dependentAxes: [@Hmag]
}

(ndcube:Cube) {
independentAxes: [@KDH]
dependentAxes: [@Kmag]
}

(this part is as for a normalised Gaia time series).

If you read the paper, I don't think we can consider this as several
monocolor time series. Of course we have as much TimeSeries as we have

Why not?  And what else would it be instead?  How would you expect a
client to deal with that thing?

It is a complex TimeSeries with one independant Time (the one you want)
and several complex "Parameters" made of Time and magnitude

I'm afraid I still can't see what kind of code would consume an
annotation saying about that much.  Because you see, if there's
no code to consume it, we shouldn't bother marking it up
machine-readably (but that's just an excursion: I think there is a
clear, machine-readable interpretation of this data; it's just that
the table structure chosen is such that you need relational
operations to pull it out, and thus my plea is to postpone this until
we can at least reliably annotate actual ("normalized") tables).

Just because all the times on a row are very close to each other, they
are part of  the same "observation" where you rotate the filter wheel.

The data providers could choose to have one time and then magnitude
and delta-t as dependent axes, true.  Both of the choices is
rational and perfectly annotatable without relational algebra (and in
particular my proposal).
What makes this table denormalised is that it is not a time series,
it is several timeseries mixed together.  I don't think it's
surprising that such a thing cannot be annotated as a time series.

The "would be" above is because again this is highly de-normalised in
that the table is (logically) a union of different (sets of) time
series. You immediately notice that something is wrong with this
table because sorting by Jmag, say, is physically meaningless.

We cannot say they are wrong, I think this is one of the prerequisite of
the workshop.
If you read the paper what they have done makes sense.

That's not the point.  The point is that their table is not a time
series, and trying to somehow pretend it is is going to blow up our
standard, both in time (which we don't have, 20 years into the VO and
still unable to say "this is ra, dec in ICRS") and complexity (which
comes at a high price, as evidenced by the failure of STC1).

I'm still claiming that if our annotation tries to include relational
algebra, we still won't be able to fix the urgent use cases five
years from now (don't laugh -- people have felt over-confident in the
DM business fifteen and ten years ago, and we still have nothing).

I was not about relational algebra (apart from selecetin the source) on
this one but trying to have a TS model able to manage such use case

Oh, but the operations you need to re-structure the tables are
relational algebra, and once you start re-structuring tables in your
annotation, you will re-discover all of Codd (1970): selection,
projection, grouping, inner joins, left and right outer joins,
complex conditions: You'll need it all, and we'll be busy for the
next couple of years.
And given the experience with ADQL 1 (which was about the same thing:
expressions of relational algebra written in XML) I expect in the end
we will hate it and do what we did in ADQL 2: just use some subset of
SQL as everyone else uses it.

If we don't try to map these things will we force authors and providers
to shape their tables as WE want them to do ?

It's the other way round: Data providers are asking us: "How should
we write our data such that topcat, astropy, and whatever else can
optimally read them".  And we should give them sound advice.  Which
is: Normalise your data, do not produce things with metadata that
changes per row.  That will make it a lot easier on everyone, them,
their consumers, our implementors, and us as well.
If data providers see that machines will then understand their data a
lot better, they won't feel "forced", they will feel well-advised.
And for a good reason, because "avoid per-row metadata" is about
the soundest advice you could give anyone writing tables.
Eventually being able to annotate legacy, de-normalsed data so it can
be processed with modern, VO-compliant clients is then a nice
thing, but it's not nearly as urgent.  Also, that effort becomes a lot
more attractive when those DM-aware clients exist.  Which they don't
right now, and which they won't until we put out a nice, easily
implementable standard that lets them do interesting things.

http://vizier.u-strasbg.fr/viz-bin/VizieR?-source=J/ApJ/790/L21&-to=3

. All parameters have the same importance. It's an event list. Why
should we not know that from the top ?

(ds:Dataset) {
(ndcube:Cube) {
independentAxes: [@Arrival]
dependentAxes: [@Theta, @E, @RAJ2000, @DEJ2000]
}

I don't think Time has something special in this example. All parameters
are INDEPENDANT,
You could also decide to see this as a  multidimensional Sky Cube where
values are time, energy and teta

Ok -- given the context I had expected this to be a time series.
You're right, physically, there's (almost) no point presenting it as
one.  It's an event list, and, frankly, I'd just annotate a position
in there.  What other annotation should a client use?  What
functionality would be enabled/triggered by that annotation?

Le 26/04/2021 à 15:48, msdemlei a écrit :

On Mon, Apr 26, 2021 at 01:55:28AM -0700, Bonnarel wrote:

Le 23/04/2021 à 08:50, msdemlei a écrit :

(ndcube:Cube) {
independentAxes: [@JDJ]
dependentAxes: [@Jmag]
}

(ndcube:Cube) {
independentAxes: [@JDH]
dependentAxes: [@Hmag]
}

(ndcube:Cube) {
independentAxes: [@KDH]
dependentAxes: [@Kmag]
}

(this part is as for a normalised Gaia time series).

If you read the paper, I don't think we can consider this as several
monocolor time series. Of course we have as much TimeSeries as we have

Why not? And what else would it be instead? How would you expect a
client to deal with that thing?

It is a complex TimeSeries with one independant Time (the one you want)
and several complex "Parameters" made of Time and magnitude

I'm afraid I still can't see what kind of code would consume an
annotation saying about that much. Because you see, if there's
no code to consume it, we shouldn't bother marking it up
machine-readably (but that's just an excursion: I think there is a
clear, machine-readable interpretation of this data; it's just that
the table structure chosen is such that you need relational
operations to pull it out, and thus my plea is to postpone this until
we can at least reliably annotate actual ("normalized") tables).
But MapIntoVOT interpretor by Laurent can create Json instances with
that at least.

Just because all the times on a row are very close to each other, they
are part of  the same "observation" where you rotate the filter wheel.

The data providers could choose to have one time and then magnitude
and delta-t as dependent axes, true. Both of the choices is
rational and perfectly annotatable without relational algebra (and in
particular my proposal).
What makes this table denormalised is that it is not a time series,
it is several timeseries mixed together. I don't think it's
surprising that such a thing cannot be annotated as a time series.

ah, this is a matter of definition : I thought that TimeSeries were
things were "something" (whatever it is) was varying with time.
If it is not the case and we are only dealing with "single-scalar"
curves then the problem is really simplified.
And the two views : entanggled or not, are not that different in the
simple case
But real life shows more complex things and a small extension of Mango
allowed to encompass such Complex Time Series as this table 3 example

The "would be" above is because again this is highly de-normalised in
that the table is (logically) a union of different (sets of) time
series. You immediately notice that something is wrong with this
table because sorting by Jmag, say, is physically meaningless.

We cannot say they are wrong, I think this is one of the
prerequisite of
the workshop.
If you read the paper what they have done makes sense.

That's not the point. The point is that their table is not a time
series, and trying to somehow pretend it is is going to blow up our
standard, both in time (which we don't have, 20 years into the VO and
still unable to say "this is ra, dec in ICRS") and complexity (which
comes at a high price, as evidenced by the failure of STC1).

I'm still claiming that if our annotation tries to include relational
algebra, we still won't be able to fix the urgent use cases five
years from now (don't laugh -- people have felt over-confident in the
DM business fifteen and ten years ago, and we still have nothing).

I was not about relational algebra (apart from selecetin the source) on
this one but trying to have a TS model able to manage such use case

Oh, but the operations you need to re-structure the tables are
relational algebra, and once you start re-structuring tables in your
annotation, you will re-discover all of Codd (1970): selection,
projection, grouping, inner joins, left and right outer joins,
complex conditions: You'll need it all, and we'll be busy for the
next couple of years.
And given the experience with ADQL 1 (which was about the same thing:
expressions of relational algebra written in XML) I expect in the end
we will hate it and do what we did in ADQL 2: just use some subset of
SQL as everyone else uses it.

If we don't try to map these things will we force authors and providers
to shape their tables as WE want them to do ?

It's the other way round: Data providers are asking us: "How should
we write our data such that topcat, astropy, and whatever else can
optimally read them". And we should give them sound advice. Which
is: Normalise your data, do not produce things with metadata that
changes per row. That will make it a lot easier on everyone, them,
their consumers, our implementors, and us as well.
If data providers see that machines will then understand their data a
lot better, they won't feel "forced", they will feel well-advised.
And for a good reason, because "avoid per-row metadata" is about
the soundest advice you could give anyone writing tables.
Eventually being able to annotate legacy, de-normalsed data so it can
be processed with modern, VO-compliant clients is then a nice
thing, but it's not nearly as urgent. Also, that effort becomes a lot
more attractive when those DM-aware clients exist. Which they don't
right now, and which they won't until we put out a nice, easily
implementable standard that lets them do interesting things.

http://vizier.u-strasbg.fr/viz-bin/VizieR?-source=J/ApJ/790/L21&-to=3

. All parameters have the same importance. It's an event
list. Why
should we not know that from the top ?

(ds:Dataset) {
(ndcube:Cube) {
independentAxes: [@Arrival]
dependentAxes: [@Theta, @E, @RAJ2000, @DEJ2000]
}

I don't think Time has something special in this example. All
parameters
are INDEPENDANT,
You could also decide to see this as a  multidimensional Sky Cube where
values are time, energy and teta

Ok -- given the context I had expected this to be a time series.
You're right, physically, there's (almost) no point presenting it as
one. It's an event list, and, frankly, I'd just annotate a position
in there. What other annotation should a client use? What
functionality would be enabled/triggered by that annotation?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/ivoa/dm-usecases/issues/24#issuecomment-826849646,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AMP5LTCMHMWPXOOJWS6CCIDTKVVJZANCNFSM4ZXO3FWQ.

On Fri, Apr 23, 2021 at 08:37:26AM -0700, Laurent MICHEL wrote:

The @msdemlei  client
README,
makes the assumption that the VOTable is delivered with distinct
annotations for 2 different versions of the same model (Coords). I
do not thing that this case is the most likely because the
annotation process is a very though job (ask @gilleslandais) and I
doubt that data curators will duplicate their efforts to support
multiple variants of the same model.

Perhaps not, but for instance in cases of "oh, our serialisation
library supports both anyway" they may.  And they will, in
particular, if it gives them a tanglible benefit ("Oh, we just have
to add this instance declaration and our data still works with old
TOPCATs").
But that, really, is a side show.  The main point, and I'll not tire
of stressing this, is that DMs can evolve independently.  I'll try
again: suppose we have phot2.  Since timeseries1 contains references
into phot, and that we don't have any more, it will have to become
timeseries2 even if nothing changes anywhere else.  The thing that
I'm worrying most about is that in such a world, a timeseries1 client
can't even find an entirely unchanged coords annotation any more.
In contrast, with isolated DMs, even if there is only phot2 (and
not phot), a client that knows coords will still be able to work out
which coordiates belong together and that they're in ICRS.
Given the entanglement comes at such a high cost, it would have to
give us a great benefit.  But, frankly, I'm not seeing even a minor
one at this point.

The most likely situation is to have a client trying to put
together (e.g. xmatch ) data sets annotated with different
versions on the same model.

How to cross match data annotated with CoordsV3 against data
annotated with CoordsV4.?
How to cross match data annotated with CubeV3 against data
annotated with CubeV4.?

This a critical point that cannot be worked around just by using
un-entangled models.

That assumes that the programmes just de-serialise your models and
then work with that data structure.  I really, really don't think
that will be the case.  I'm saying this as someone who really has
tried keeping the STC1 data structure in his code.
You see, data structures are at the root of any non-trivial
programme.  It's hubris if we think we can make these choices for
people.
What will in general happen is that people map our annotation into
their internal data structures.  That's what we ought to make easy.
And then the question just goes away: A programme "knows" a DM if it
can map its annotation into its data structures.  So, a programme
that knows both coordsv3 and coordsv4 can fill its internal data
structures from both annotations (and will, unless we mess it up,
prefer coordsv4 if it can get it).  And then it will work in its
preferred way.
I'm not sure what you mean by cross matching cubev3 and cubev4; what
functionality do you have in mind for that?

I understand you point of view and I do not underestimate your arguments in favor of using individual model elements.
You ask the right question: what is the advantage of using integrated model (sorry but  entangled is a bit pejorative)?
Let me recap my points, already exposed sometime ago:

A model integrating components from other models will assure that all of those components are consistent each to others (vocabulary, roles at least) and that there is no risk of confusion (RFC validation).
Integrated models can describe complex datasets. I would be very sorry to have to say to some data provider: sorry but your dataset is too complex, I cannot annotate it. examples:

detections attached to sources
Same properties but with different frames
Multi-objects data tables
3- Instances of integrated data models can be shared among different peers, e.g sending by SAMP Mango serializations of individual catalogue rows.
4- Integrating components from other models does not mean dissolve them in the host model. They remain usable as such since they keep both integrity and  dmtypes even within an integrated model. You client strategy can be applied on integrated data models as well (matter of XQuery). I've somewhere in my code a search_element_by_type(dmtype) method able to retrieve your coords2 class in any arbitrary nested model.

I'm not sure what you mean by cross matching cubev3 and cubev4; what
functionality do you have in mind for that?

Cross processing (doing anything with both datasets together) would be more appropriated

On Wed, Apr 28, 2021 at 01:15:21AM -0700, Laurent MICHEL wrote:

I understand you point of view and I do not underestimate your
arguments in favor of using individual model elements.  You ask the
right question: what is the advantage of using integrated model
(sorry but  entangled is a bit pejorative)?

This is probably a good place to summarise the main arguments in this
question in the run-up to the workshop, so let me point out why I
think these benefits do not weigh up the high cost of having
dependencies between the models:

A model integrating components from other models will assure that
all of those components are consistent each to others (vocabulary,
roles at least) and that there is no risk of confusion (RFC
validation).

Isolated models of course should have no dependencies between them,
and thus there should be no question of them being consistent with each
other.  This being the real world, I admit we perhaps won't always
get away with that, but I maintain it is prudent to at least try
hard.
And sure, there are unavoidable dependencies that we simply will have
to model as such and bit the bullet of dependencies.  But the design
goal should be to minimise those, because no amount of RFC validation
will cope with the interdependencies between 10 models when one
starts to move.
I'd be happy to discuss this futher on a concrete example on which
you see a danger of inter-model inconsistencies.

Integrated models can describe complex datasets. I would be very
sorry to have to say to some data provider: sorry but your dataset
is too complex, I cannot annotate it. examples:

I think the reverse is true: If you have isolated models, you can
annotate positions, photometry, provenance, whatever, in arbitrarily
complex data that you simply cannot model interoperably (and it
wouldn't make any sense either, since they're one-of-a-kind).
Still, your annotation will just work: Clients will be able to make
out positions or photometry and do the right thing with them.
If, on the other hand, you just have a few "integrated" models (cube,
source), and something doesn't quite fit, you're altogether out of
luck, even if a, say, time in there would of course profit from an
annotation as to what its time scale and reference position is.
That, incidentally, is the what my minimal use cases should have
expressed.

detections attached to sources
Same properties but with different frames
Multi-objects data tables

I still have not seen a single use case on a normalized table that
isolated data models couldn't cope with (nb. the question of having
relational transformations as part of the annotation is distinct
from the question at hand here).  So, after my last few paragaphs,
I'd say the chances that you can add meaningful annotation to complex
datasets is higher with isolated data models.

3- Instances of integrated data models can be shared among
different peers, e.g sending by SAMP Mango serializations of
individual catalogue rows.

For one, you don't need integrated models for that -- just transmit
whatever annotations you deem necessary.  But then, I'm rather
skeptical of this use case altogether, as it is perliously close to
what the existing SAMP Mtype table.highlight.row does, and I think
it's rarely a good idea to have two distinct mechanisms when the use
case is hardly different (if at all).

4- Integrating components from other models does not mean dissolve
them in the host model. They remain usable as such since they keep
both integrity and  dmtypes even within an integrated model. You
client strategy can be applied on integrated data models as well
(matter of XQuery). I've somewhere in my code a
search_element_by_type(dmtype) method able to retrieve your
coords2 class in any arbitrary nested model.

As long as that is true, I am almost happy.  I am not entirely happy
because we would still have the time bomb of the integrated models
ticking in our system, waiting to blow the whole thing up when we do
have to have a major version change in one of our data models.
Hence, for the workshop participants' benefit, let me briefly recap a
few of the advantages of having small-ish, isolated DMs:
(1) Lesson from STC-1: data providers can adopt DMs one at a time
(say, Coords first, then Meas, then Dataset, then Cube) -- and they
get a bit more functionality in some client for each such annotation.
People will panic if they see DMs with 70 boxes, in particular if
there's no clear benefit in diggin in.  They won't panic when there
are just 5 boxes, and doing something with them already gives them
some nice goodie in TOPCAT.
(2) Lesson from TAP validation: Look at slide 23 of the last euro-vo
registry weather report: https://wiki.ivoa.net/internal/IVOA/InterOpNov2020Ops/20201118-Euro-VOResourcesValidationStatus.pdf
-- essentially all TAP services in the VO are marked invalid (the
yellow and red areas).  Mind you: Almost all of these services work just
fine when you go there with TOPCAT or pyVO.  There's often just one
little interoperability problem that nobody will stumble into in
years that makes them fail.
An "integrated" DM has about the same complexity as TAP.  There's no
reason to believe it'll behave differently from TAP, and so you'll
have a huge lake of invalid annotations where nobody can say if it's
just a small detail that's wrong of if it's utter breakage.
So, let's plan instead such that valid instances are (a) humanly
possible (i.e., in particular: small), (b) halfway stable, and (c)
there's a small damage radius, i.e., a problem in one annotation
doesn't make the whole thing invalid.  Instead, a validator would
report: "valid Coords, invalid Photometry, valid NDCube, invalid
Dataset".  And people would have a good idea what sort of
functionality they can rely on and what's a little more shaky.
(3) Separate evolvability (that's discussed in some depth elsewhere,
and we ignore it at our successors' peril).
(4) Flexibility and Consistency: A client that knows how to deal with
Coords annotation can do that whether these Coords are in a time
series, a catalogue, or a SIAP response -- always in the same way.
And you don't need anything else at all: If it's a position, you can
slap Coords annotation on it, and it'll just work.  I'm not saying
this isn't possible at all with your integrated models, it's just
that I've not been able yet to actually write code using them that
does this (help is gratefully appreciated).
(5) Small, isolated models, I claim, are what most programmers what:
Let's give them Lego bricks rather than pre-assembled models.
The Legos are going to fit much better into their programmes.

This is probably a good pl

sure

interdependencies between 10 models

Little exaggeration?
Our positions are not converging at all, there is no need to run a new discussion loop.
I would just like to repeat what I  wrote 2 weeks ago.
Technically, the actual annotation scheme works for any model granularity, hence your proposal is actually to ask the VO not to RECommand integrated models. This is what I do not agree with.

Hence, for the workshop participants' benefit, let me briefly recap a
few of the advantages of having small-ish, isolated DMs:

My answers

(1) Lesson from STC-1...

Not really applicable here.

As DM chair I've always said that both Meas and Coords must be adopted together.
I've nothing against using them as independant components as long as the annotation remains faith to the model structure.
I'm against the  integrated models diabolization

(2) Lesson from TAP validation:

The comparison with TAP is unfair. TAP is a complete database infrastructure encompassing all VO fields:   Nothing to do with a simple measure container as MANGO is.

(3) Separate evolvability

That's discussed in some depth elsewhere

(4) Flexibility and Consistency:

I agree with you, our 2 approaches do work.... for the simplest cases.
Have you tried to figure out (on paper at least) whether your code design could be applied to a MANGO mapping?

(5) Small, isolated models

As the impact of model changes has already been discussed many time here, I prefer to have a little fun with your Lego metaphor: When I was young, I spent a lot time playing with Lego bricks. At that time Lego was mostly sold as brick boxes, but year after year the company has marketed more and more complex (entangled) objects (Star War ships, robots..) with a growing success. Just to warn you against this sort of comparison :-)

Let's continue most of this at the workshop(s), but just one thing:
On Wed, May 05, 2021 at 07:48:39AM -0700, Laurent MICHEL wrote:

(4) Flexibility and Consistency:

I agree with you, our 2 approaches do work.... for the simplest
cases.  Have you tried to figure out (on paper at least) whether
your code design could be applied to a MANGO mapping?

I believe most of MANGO isn't so far from what I'm proposing; as far
as I can see, you only need to drop the top-level grouping, and
you'll have your isolated annotations.
But it's much more productive to talk about use cases: So, in the
run-up of the workshop it would be good to have one or two of these
complex use cases that might not work wir isolated models.  Any
chance you could give a sketch of them in the form of "A client wants
to do X, and it needs Y and Z for that"?

When I was young, I spent a lot time playing with Lego bricks. At
that time Lego was mostly sold as brick boxes, but year after year
the company has marketed more and more complex (entangled) objects
(Star War ships, robots..) with a growing success. Just to warn
you against this sort of comparison :-)

Well, you got me there on the marketing.  But then we don't have the
advertising budget of Lego, and hence I still believe if we want
folks to buy our stuff, we'll have to go for their recipes when
they didn't have those budgets either...

I believe most of MANGO isn't so far from

This is true as long as you do not have associated data.
This is also why I'm repeating that our difference on this topic is a matter of Xpath
For the workshop I'll insist on the complex data especially. I'm even more motivated for this since Yesterday when we had a long meeting with exoplanet people who are asking for mapping highly connected data and even for JSON serializations.

Well, you got me there on the marketing ..

:thumb up:


Move Tasks from Core to Engine

Tasks are currently defined in Core, but they belong to Engine

I'm not sure about that. You could want a task queue elsewhere than in the main loop (e.g. in a "precomputing step" in a plugin).

Closing as won't fix for that reason.


[FLINK-17944][sql-client] Wrong output in sql client's table mode

What is the purpose of the change
This PR fix the issue of wrong output in sql client's table mode of stream output.
Brief change log
This PR add a new method fieldsEquals in class Row, and use this method to compare rows in sql-client.
Verifying this change
Verify it manully, the output is this sql statement is now correct.
SELECT name, COUNT(*) AS cnt FROM (VALUES ('Bob'), ('Alice'), ('Greg'), ('Bob')) AS NameTable(name) GROUP BY name;

Output
Alice                         1
Greg                         1
Bob                          2

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (no)
The public API, i.e., is any changed class annotated with @Public(Evolving): ( no)
The serializers: ( no )
The runtime per-record code paths (performance sensitive): (no )
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: ( no)
The S3 file system connector: (no)

Documentation

Does this pull request introduce a new feature? ( no)
If yes, how is the feature documented? (not applicable )

@twalthr Could you help review it ? Thanks

@flinkbot run azure

Thanks @zjffdu. I will add a test and merge this.


Add linux deriving host.docker.internal, fixes #843

The Problem/Issue/Bug:
Although the hostname "host.docker.internal" is defined and supported for Docker for Windows and Docker for Mac, it's not provided in the container on linux docker at this time. That breaks our approach to xdebug in v0.18.0, which worked great on the other situations.
How this PR Solves The Problem:
On linux, use ip address show dev docker0 to get the IP address, then add the address to extra hosts.
Note that this is a bit awkward, and a throwaway address is used on non-linux. This is because golang templating by default escapes the required quotes, breaking yaml :(  There are workarounds for it, but it's a bit of work (You have to treat things as HTML attributes instead of as strings in the golang templating.)
Manual Testing Instructions:
xdebug instructions should work for Linux, Mac, and Windows as shown in our docs
Automated Testing Overview:
I haven't added a test for this.
Related Issue Link(s):
The problem was introduced in https://github.com/drud/ddev/pull/785
Release/Deployment notes:

[ ] Update Stack Overflow article with Linux fix on release.
[ ] When Docker for Linux is fixed, remove this code.

I'd actually be curious if @andrewfrench could take the first pass of a manual test on this in addition to code/test coverage review. Given that this is a OS specific regression, if he can confirm it's working then that would be sufficient for me to skip this review.

Agreed, working towards that right now @rickmanelius

It's actually a great project for that: Experience with PHP debugging, getting linux review going, all kinds of good stuff.


vulkan test coverage for 1d, 3d, 3dlist images

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

@pkaur19 thanks for this contribution. I am not talkin in behalf of OpenCL work group but as someone who recently went across missing coverage of vulkan interoperability. As result I've created this issue. That being said, I am not sure if OpenCL-CTS needs to test list of 3D images (VulkanImage3DList class), it sounds to me a bit like time consuming operation meanwhile many vendors already complain about CTS execution time. Would that make any sense to replace list of image3d_t test with image1d_array_t / image_2d_array_t tests ?
BTW. I went briefly through your code and I can't spot any location of construction of VulkanImage3DList object, am I missing something? Is this work in progress?
Please sign CLA license to move forward with your PR, thanks.

@shajder
The purpose of the test is to add the test coverage you requested as part of #1834, not for addition to generic image test coverage. This is the work-in-progress, but I have left a comment to point you to part you are referring to. Also, I would prefer to get some basic functional changes in and follow up with another PR as we keep adding more coverage.

@pkaur19 thanks for this contribution. I am not talkin in behalf of OpenCL work group but as someone who recently went across missing coverage of vulkan interoperability. As result I've created this issue. That being said, I am not sure if OpenCL-CTS needs to test list of 3D images (VulkanImage3DList class), it sounds to me a bit like time consuming operation meanwhile many vendors already complain about CTS execution time. Would that make any sense to replace list of image3d_t test with image1d_array_t / image_2d_array_t tests ?
BTW. I went briefly through your code and I can't spot any location of construction of VulkanImage3DList object, am I missing something? Is this work in progress?
Please sign CLA license to move forward with your PR, thanks.

Sure if 3DList is not needed I will replace 3DList with 1DList. AlsoI see VulkanImage2DList is already present we don't need to implement that.
VulkanImage3DList is created here.

I cannot build this change.  Can you please ensure is builds with CI?

Closing this in favor of https://github.com/KhronosGroup/OpenCL-CTS/pull/1887. Please, review https://github.com/KhronosGroup/OpenCL-CTS/pull/1887 instead.


Do packages really rebuild after failed init? Should they?

The documentation says:

On the other hand, straight-declare-init-finished is for
correctness: it allows straight.el to optimize the saving and
loading of the build cache, since otherwise it could not guarantee
that the build cache was saved after a failed init. (The practical
consequence of this would be that straight.el would forget about all
the packages it rebuilt on the failed init, and rebuild them again
next time -- making for very slow debugging indeed!) If you don't call
straight-declare-init-finished, then your initial Emacs startup is
still fast, since straight.el can use after-init-hook to perform
its build-cache optimization correctly. But reloading your init-file
will be very slow.

This seems a little fishy to me. I think that after-init-hook is always called, even on error, so there should be no concern about packages being rebuilt after a failed init. Maybe a failed re-init, but that is less of an issue.

This is no longer relevant with the removal of straight-declare-init-finished, the introduction of straight--build-cache, and the corresponding restructuring of the build-cache.el format.


Refactoring variational methods

The PR refactors variational methods in zhusuan based on the whitepaper draft, to enable more unified variational API that are easy-to-compose and easy-to-doc.
Variational methods are categorized by objectives.

Exclusive KL

zs.variational.elbo(), return EvidenceLowerBoundObjective

Monte-Carlo  objectives

zs.variational.importance_weighted_objective(), return ImportanceWeightedObjective

Inclusive KL

zs.variational.klpq(), return InclusiveKLObjective

Different gradient estimators are provided for each objective class.

EvidenceLowerBoundObjective

sgvb
reinforce

ImportanceWeightedObjective

sgvb (IWAE under this situation)
vimco

InclusiveKLObjective

rws

Also a first step to enhance doc strings for the variational APIs.
I'll add tests and migrate examples in the following days.

@ssydasheng @cjf00000 Can you comment on the new API?

it seems that whitepaper has the api for importance weighted as
zs.iw_bound
but in this pull request, the api for importance weighted objective is in monte_carlo.py, which doesn't seems very appropriate. And the api is still importance_weighted_objective (different from iw_bound and kind of long)

@ssydasheng Yeah I may adapt the name in the paper a little later. Now it's a bit long but I think iw is not very clear and it's okay to stick to the longer name currently. Or any good name you can come up with?
btw, I mean if you can comment on whether the design is appropriate and any suggestions on that?

I think attributing them by objectives is quite reasonable.


WIP: Aggregated NCDA - further adjustments

#1040 (per https://github.com/TIP-Global-Health/eheza-app/issues/1040#issuecomment-1982102009)

Demo for https://github.com/TIP-Global-Health/eheza-app/issues/1040#issuecomment-1982102009:


Prevent changes to a configuration after it has been resolved that would affect the resolved graph

Expected Behavior
Once a configuration has been resolved, any change to that configuration that would affect the resolved graph should be disallowed.
Current Behavior
Some changes (only adding/removing dependencies) from a configuration after it has been resolved fail with the message:

Cannot change dependencies of configuration ':implementation' after it has been resolved.

But there are many other ways to impact which dependencies may be selected.
Context
The current behavior prevents a mistake like this from going unnoticed:
dependencies {
   implementation("org:dependency:1.0")
}
println configurations.implementation.files
dependencies {
   implementation("org:other:1.0")
}
// where's the "other" dependency?
println configurations.implementation.files

This isn't a new problem.
Steps to Reproduce (for bugs)
apply plugin: 'base'

repositories {
    jcenter()
}

configurations {
    implementation
}

dependencies {
    implementation("log4j:log4j") {
        version {
            prefer "1.2.17"
        }
    }
}

task resolve {
    dependsOn configurations.implementation
    doLast {
        println configurations.implementation.files
        // Fails with "Cannot change dependencies of configuration ':implementation' after it has been resolved."
        // project.dependencies.add("implementation", "com.google.guava:guava:17.0")

        // This does not fail, though it would change the versions in the graph:
        configurations.implementation {
            dependencies.all {
                version {
                    prefer "1.2.16"
                }
            }
        }
        println configurations.implementation.files
    }
}

Run with gradle resolve.  Notice if you move the configuration outside of the task action, it does change the resolved graph.
Your Environment
This always happens.

@big-guy What happens if you do this:
dependencies {
   implementation("org:dependency:1.0")
}
println configurations.implementation.files
dependencies {
   implementation("org:other:1.0")
}
// where's the "other" dependency?
println configurations.implementation.resolvedConfiguration.files

Does the above issue still happen?

@JLLeitschuh Yes, the build still fails as it should.


[Kubernetes] Provide way for in_tail plugin relay on container state

Environment information:

Kubernetes on EKS
in_tail > kubernetes metadata > cloud watch logs

We are collecting logs from the container using the path /var/log/containers/*my-app*.log
When a pod is deleted or a log rotation happens the in_tail plugin detects as expected
The problem is, if the pod is restarted, failed or completed(job) the logs still there but it won't have no more data. So in_tail will open an stream and leave it there forever
I was going to try the option limit_recently_modified expecting that a file that is stopped for a while probably is done for some of the states
But even with it, it will just mark the file to be closed on the shutdown of fluentd

expand_paths won't return the file when it gets old
marked as unwatched
but the stop is not immediate, leaving the file open detach_watcher_after_rotate_wait until the pod is removed in that case

At some point because of that and much other files, it will start to show Too many open file

I was thinking
It could work in stages and validating the kubernetes pod status, those stages could be saved in disk

to_consume (files that were never started to be consumed)
if the file was never consumed, even if the container is in the state {failed, completed, restarted}. Send it to in_tail marking as consuming.
if the file consume was started, checks the state of the container, {failed, completed, restarted} should finish the current stream and close it marking as consumed (similar to open_on_every_update)

It could work in stages and validating the kubernetes pod status, those stages could be saved in disk

So your suggestion is in_tail access to k8s to check log file status?

No, in_tail is plugin and it works fine for its purpose. I don't think it should have access to k8s
Maybe a way to extend it or a new plugin made for k8s that integrates with it
<source>
      @type kubernetes
      selectors ["app=my-app"]
...

Internally, k8s access, k8s metadata and pushes to in_tail which files it should be read


Unhandled Exception:  System.Exception:  occurred

var tabs = new FreshMvvm.FreshTabbedNavigationContainer();
tabs.AddTab<MainTabPageModel>("Main", "icon_home.png", null);
tabs.AddTab<AboutTabPageModel>("About", "icon_browse.png", null);
MainPage = tabs;
I am using the convention of ###PageModel  && ###Page
Still new to FreshMvvm but the error doesn't provide too much information :/

@orassr : The error is pretty generic and have very little information. If you dont have any logs, you might need to create a sample project and upload, so one of us can give it a try and see what exactly happens.

@orassr I recently wrote a blog post that might help you solve your issue.
I don't suspect this is a issue with FreshMvvm but more to do with your code. I'm going to close this off but reopen if you think it's FreshMvvm.
https://michaelridland.com/xamarin/the-visual-studio-debugging-trick-that-every-xamarin-net-developer-should-know/


Compiler warnings in wayland code

wl_window.c: In function ‘xdgToplevelHandleConfigure’:
wl_window.c:299:14: error: variable ‘resizing’ set but not used [-Werror=unused-but-set-variable]
     GLFWbool resizing = GLFW_FALSE;

And the issue I tried to fix in #1143 is still not fixed
wl_init.c: In function ‘pointerHandleAxis’:
wl_init.c:167:5: error: ‘y’ may be used uninitialized in this function [-Werror=maybe-uninitialized]
     _glfwInputScroll(window, x, y);
     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
wl_init.c:167:5: error: ‘x’ may be used uninitialized in this function [-Werror=maybe-uninitialized]

Is initializing x and y to zero, really such a big deal? The asserts dont make the warning go away when compiled as:
gcc -MMD -Wno-missing-field-initializers -Wall -std=c99 -D_XOPEN_SOURCE=700 -Werror -g3 -Og -DDEBUG -fwrapv -fstack-protector-strong -pipe -fvisibility=hidden -fPIC -pthread -fpic -D_GLFW_WAYLAND -D_GLFW_BUILD_DLL ...

@linkmauve True. What about 53b193a16173ab989ce49006dc75c4c9f7f88946? It keeps the flow control and the explicit assert.


No more auto-correct for frozen_string_literal: true

Recently (don't know exactly with which update), Rubocop has stopped auto-adding the # frozen_string_literal: true line at the top of my files when linting / auto-correct.
Tried to add the following rule to my rubocop.yml, no luck:
Style/FrozenStringLiteralComment:
  EnforcedStyle: always_true

Expected behavior
It should auto-add the line.
Actual behavior
It doesn't anymore (it properly did a few weeks ago).
Steps to reproduce the problem
I use vscode latest version.
RuboCop version
Include the output of rubocop -V or bundle exec rubocop -V if using Bundler. Here's an example:
0.88.0 (using Parser <IP_ADDRESS>, rubocop-ast 0.2.0, running on ruby 2.7.1 x86_64-linux)

The could also be possibly related to my changes, since it seems related to editor integration. How can this problem be reproduced?

Style/FrozenStringLiteralComment is an unsafe cop. The auto-correct (rubocop -a) option behaviour has been changed from RuboCop 0.87. It only auto-corrects safe cops.
https://github.com/rubocop-hq/rubocop/blob/master/CHANGELOG.md#changes-2
The option to auto-correct unsafe cops has been changed to new rubocop -A ( rubocop --auto-correct-all) option .
https://docs.rubocop.org/rubocop/0.88/usage/auto_correct.html
I think it is necessary to integrate VSCode that supports the new RuboCop options.

Ah ok, makes sense, thanks!
I'll take a look at why it's an unsafe cop.
I think vscode integration is behaving as it should, I wouldn't want unsafe cops to be auto-run on my code :)

I know this is a pretty old issue, but I tripped across this today and I believe the "right" way to allow this is to add it to your config file:
Style/FrozenStringLiteralComment:
  Enabled: true
  SafeAutoCorrect: true

I consider the frozen string "safe" in my project, so this was an easy way to enable it on save with the extension in VSCode.

adding SafeAutoCorrect: true working for me


review ticket 9

CBPT grading
for some reason the link was broken so I only have a link to the file on github
API hacks
DATABASE HACK

.85 CPT
.9 API
.9 DB


Move new/recent column from Welcome to Explorer view

📯 This is an early idea and we'd love to hear your thoughts, ideas and concerns to further explore it.
Context

The Explorer is the natural UI surface of files and folders.
Explorer already duplicates items that are also offered on the Welcome page, like opening a folder, checking out a repo, or creating a new file.
Explorer also offers extension contributions that are already used by Java to add their project wizard.
If empty, the Explorer has a lot of unused vertical space.

Proposed solution
Move the new and recent shortcuts from the left side of the Welcome page to the Explorer.
Also revisit current content for the welcome.
Open questions

Should this also open Explorer by default for new windows to make sure the options are visible?

Circulated this with a few people and got little excitement.
Given that VS Code starts with Explorer closed, this requires a bit more impact. If we have additional content for the Get Started page and need the space, we can revisit.


Kansas - UP Receiver Migration Pre-Work

User Story:
As a developer, I want to compare the messages generated from the Covid and Universal pipelines, so that I can successfully onboard Kansas to the UP
Description/Use Case
Kansas is already receiving data from the covid pipeline they will be onboarded to the UP where they will be receiving data from SimpleReport including Covid, Flu and RSV. We need to make sure the message we generate from the UP matches the Covid pipeline message.
Risks/Impacts/Considerations
Dev Notes:

[x] Fetch [STLT] organization settings from production and load them locally

[x] Use the attached SimpleReport covid postman collection and make sure the message gets routed to Kansas locally. Modify the message to meet [STLT] filter if needed Simple Report Covid.postman_collection

[x] Make a copy of the [STLT] organization settings to onboard them to the UP. See How to Migrate an existing receiver to the UP documentation for more details: https://github.com/CDCgov/prime-reportstream/blob/master/prime-router/docs/onboarding-users/migrating-receivers.md

[x] Use this Postman collection to send a FHIR bundle the UP and make sure the message gets routed to the new UP Kansas receiver. You may need to update the Simple Report sender to use the simple-report-sender-transform.yml if it's not using it. Simple Report UP.postman_collection

To migrate the Covid translation settings start by looking at their current translation settings. If the receiver uses  any of the following settings you will need to create a receiver schema:

receivingApplicationName
receivingApplicationOID
receivingFacilityName
receivingFacilityOID
messageProfileId
replaceValue
replaceValueAwithB
reportingFacilityName
reportingFacilityId
reportingFacilityIdType
suppressQstForAoe: true
suppressHl7Fields
suppressAoe: true
defaultAoeToUnknown
replaceUnicodeWithAscii
useBlankInsteadOfUnknown
usePid14ForPatientEmail: true
suppressNonNPI
replaceDiiWithOid
useOrderingFacilityName not STANDARD
receivingOrganization
stripInvalidCharsRegex

More documentation on how to set-up these transforms in the UP will be provided, but for now you can look for examples on how to set this up in either the NY-receiver-transforms or CA-receiver-transforms

If the receiver uses any of those transforms you will need to create a receiver transform under metadata/hl7_mapping/receivers/STLTs/ and update the receiver settings to point to this schema.

After migrating the receiver setting to the UP. Send another test message using the postman collection or feel free to use the PrimeCLI to test the sender and receiver transforms using this command:
./prime fhirdata --input-file {PATH TO INPUT FILE} -s metadata/fhir_transforms/senders/SimpleReport/simple-report-sender-transform.yml --output-format HL7 -r metadata/hl7_mapping/receivers/STLTs/{RECEIVER SCHEMA} --output-file {PATH TO OUTPUT FILE}

Compare the two messages again and document if there are any differences between the two messages. If there are review them with the team. We will probably need to ask the sender to add missing data or add a sender transform.

[ ] Use a diff tool of your choice and compare the two messages generated between the two pipelines and document the differences. You can find an example here RI/SR UP vs Covid

If there are no major differences we can move on to sending test messages to the STLTs staging environment.

Acceptance Criteria

[ ] Created and sent data to Kansas through the covid pipeline locally
[ ] Created and sent data to Kansas through the universal pipeline locally
[ ] Migrated  Covid receiver translation settings to the UP receiver settings
[ ] Successfully generated a message with migrated UP receiver settings
[ ] Review transforms settings with the team
[ ] Compared messages from the covid and universal pipelines and documented differences and review with team

KS-DOH is satisfied with their UP messages.  They are ready for production, now.


Unable to remove cluster members of an imported k3s cluster via cluster config page

Update - Only affects k3s types, not all imported clusters
Update 2 - Only affects the cluster config page. Removing user roles is still possible via the Cluster Explorer / RBAC / Cluster Members list

Create an Imported Cluster (Generic) and add a member other than the currently signed in one
Run registration cmd against cluster to import
Wait for it to come up (important!)
Go to Edit Config of cluster and expand Member Roles --> no - arrow at the end

Doesn't seem to be a problem with RKE1 clusters using the same embedded edit cluster experience

It looks like this is specifically disabled way back in 25 Feb 2020 for types that are k3s (https://github.com/rancher/ui/commit/4eb82ace5b26fee7507203eb33d3720f6eef528d#diff-2c6a01bdf096d36761158688bf7d1f51ad8b6eeeb312fddee623ba309eb7b5edR68)
I'll update title to reflect this.
Current line is...
@editing={{and (and notView (not (or isK3sCluster isRke2Cluster))) (not isEksClusterPending)}}

Talked with Neil, bumping this out to 2.6.x given how long the behaviours been there

The behaviour has slightly changed. Users can use the Add Member button but they cannot see any members

Additionally they cannot remove rows


Enable spending limit for Polygon

Overview
Enable spending limit for Polygon

We have disabled Spending limits for Polygon in the first iteration in order to keep things simple.
We have gotten user requests about this feature.

Todo

[ ] Deploy the module on this network (safe-modules repo, read the readme there)
[ ] Enable on the CGW
[ ] Check that It works

N.B. ask Ivan for MATIC
Requirements

Deploy module contract
Allow users to set, change and remove spending limits in the UI like on mainnet.
Update features on Django to include SPENDING_LIMIT

Franco is taking care of this task on the web side. Will keep an eye and help with testing in needed :)

I can see that all is green on the web side so closing this ticket. The avalanche one will remain in QA as testing will be performed once it is deployed to prod.
Cheers!


default folder for image update

Follow discussion with @hunvreus
Currently while you open the upload image modal, it displays the root level file structure of the repo, and your image will be uploaded to the root folder. You have to choose the destination folder manually.
Maybe we should set a default image upload folder for a collection.
For example, for posts we default the upload folder to images/posts, for article we default to images/article

Actually I'd like to be able to set it in the schema.

This was done in PR #72
You can specify the default upload folder("ui:options": {"file_dir": "<folder_path_here>"}) in schema file, for example
...
"uiSchema": {
 "body": {
   "ui:widget": "customCodeMirror",
   "ui:help": "Description of the product",
   "ui:options": {
     "file_dir": "images/products"
   }
 },


Google maps progress

are these accidental PRs where people forget to remove the .git?

@mikedevita yeap.

lovely. 🥇


LayerNorm param_shape inconsistent with tensorflow/paper

LayerNorm's uses the final dimension of inputs to define param_shape. This is inconsistent with both tensorflow's implementation, and my understanding of the original paper. haiku's implementation looks closer to sonnet's.
If this different is intentional, clear documentation would be appreciated.

Confirming this is (silently!) wrong. @tomhennigan probably worth a fix.

Thanks for bumping this @trevorcai and for reporting this @jackd. I am not sure why I missed this when originally reported.
This is indeed something we have carried into Haiku from Sonnet 1 (via Sonnet 2). The Sonnet 1 impl from 2018 is here https://github.com/deepmind/sonnet/blob/cd5b5fa48e15e4d020f744968f5209949ebe750f/sonnet/python/modules/layer_norm.py#L142-L144
I'll look into what we should do here and report back.


[backport] docs: user_guide/basic.rst: various improvements

Backport of #5356 by @maxim-belkin

[automatic post] Jenkins, test this please.

Jenkins CI test (for commit 2a6878f511690d4e6b7db20892c4696b5e1384e9, target branch v9) succeeded!


Regexes near perfection ;)

Add 'googleadservices' to regex list

Now it should be all ok

Wasn't all ok...

Sorry for the extremely long wait, I´ll merge it.


Typo in README.MD

In the LANGUAGES selection, https://github.com/sdkcarlos/artyom.js#languages
Japanese / ja-JP
it writes "日本人" which means "Japan people"
it should be "日本語" which means "Japan language"
because both of them in English are "Japanese"

A cursory google translate search confirms this. I will fix this.