I will start drafting the domain variable issue now .... it would be useful if someone could supply a sentence or two describing your use case - thanks.

@davidhassell here's a few words on the use case:
Short form

We propose an approach to reduce the volume of coordinate data by using parametrised interpolation on tie-points. For some remote sensing data products, this can lead to a 50x data volume reduction factor. Several such approaches exist, but they are only minimally standardised, so that specialised software is needed in order to use data whose coordinates has been thus reduced. Our proposal

combines the strengths of existing approaches into a single algorithm, and
can be encoded in a way that is compatible with the CF Conventions.

Long form

Increasing data volumes present a challenge for all types of Earth science data, whether it is derived from models, in-situ, or remote sensing observations. The sheer volume of the data required in order to study the Earth and its systems is one of the most difficult obstacles to overcome in order to produce scientific insights and policy recommendations.
Data reduction is one of the lowest-hanging fruits in the toolkit of big data processing approaches. The gains that can be made in this area while preserving the integrity of the data in question are potentiated each time the data transmitted, stored, and processed. The number of use cases in which this is the case grows as data products become more accessible through the opening of innovative new data services and the availability of novel processing techniques, such as the application of artificial intelligence, which make it possible to exploit data products in new disciplines.
Although these savings are relevant for all types of data, they are of particular interest in the area of remote sensing. This is because in remote sensing data products are often produced such that a multitude of coordinate metadata is required to interpret the observations in question. In contrast to e.g. model data, the position of an observing platform is generally not static, and coordinates are unique to a given observation. Thus they and cannot be re-used across different data granules of the same type. Therefore the full set of coordinate data needed to interpret any given data point often includes

latitude
longitude
satellite zenith angle
satellite azimuth angle
solar zenith angle
satellite azimuth angle

In some cases moon zenith and azimuth angles are also needed for the data to be utilised. It is for this reason that coordinate data in Earth observation data products can comprise up to 90% of a given data product, meaning that90% of the product’s data volume is devoted to providing metadata that is needed to interpret the "essential" 10% of observational data.
Thus it is unsurprising that several methods already exist to reduce the volume of coordinate data in remote sensing data products. For some data products, this can lead to a 50x data volume reduction factor. Despite the existence of several useful approaches, however, these approaches remain only minimally standardised, so that specialised software is often needed in order to make use of the data. The Earth sciences community and other scientific communities that make use of Earth observation data would benefit greatly from a common approach to reducing coordinate data volumes while preserving the integrity and interpretability of the data products.
We propose a standardised approach that

combines the strengths of existing approaches into a single algorithm, and
can be encoded in a way that is compatible with the CF Conventions.

Are @ajelenak's super-sampled coordinates still a use case? You could argue that it is still a form of compression, as, assuming we need the high resolution coordinates anyway, it still allows us to not store the target domain coordinates.
As far as a I can tell, what we have for subsampled coordinates will work OK for super-sampled ones, but a difference in the interpretation of the tie point indices: If the tie point coordinate dimension is larger than its target  dimension, then the tie point indices relate to the tie point dimension (rather than the target dimension, which is the subsampled case).
I have tried this out with some CDL, but it sounds plausible ....

I don't see a reason to exclude this use case.

Good, me neither. It just needs a to be addressed in the detail, that's all (I should have said "I have not yet tried this out with some CDL", and that is still True!)

It think that would work and would be a nice feature.
We then have to be careful with the definition of the offsets. I would suggest that they should be measured in unit of cells of the finer resolution, if the the two are different.

I've made a start at CF-ising some of our ideas:
[[compression-by-interpolation, Section 8.3, "Compression by Coordinate Interpolation"]]
=== Compression by Coordinate Interpolation

For some applications the coordinates of a data variable can require
considerably more storage than the data itself. Space may be saved in
the netCDF file by the storing coordinates at a lower resolution than
the data which they describe. The uncompressed coordinate and
auxiliary coordinate variables can be reconstituted by interpolation
from the lower resolution coordinate values, to the domain of the
data. This process will likely result in losses in accuracy (as
opposed to precision) in the uncompressed variables, due to rounding
errors in the interpolation calculations, but it assumed that these
will be small enough to not be of concern to user of the uncompressed
dataset.

The lower resolution coordinates are stored in a __tie point
variable__. This terminology is chosen to ackowledge that, whilst the
values of a tie point variable may be a subset of the uncompressed
coordinate values, they can also be different to some or all of them.

The method used to uncompress the tie point variables is described by
an __interpolation variable__ that acts as a container for the
attributes that define the interpolation technique that should be
used. The variable should be a scalar (i.e. it has no dimensions) of
arbitrary type, and the value of its single element is immaterial.

This obviously just a work in progress, and I realise that I will rapidily run out of language when the text moves from the general overview (started above) to the specifics ... but it's a start ...

Hi - I was wondering about how we can tell which interpolation zone dimensions correspond to which tie point dimensions.
This matters because we need to know the dimension order of the coefficicent variables relative to the tie point variables.
In https://github.com/cf-convention/discuss/issues/37#issuecomment-693319829 we have:
// Tie points and interpolation zones (shared between VIIRS M-Band and I-Band)
  tp_track = 96 ;  // 48 VIIRS scans
  tp_scan = 205 ;
  track_interpolation_zone = 48 ;
  scan_interpolation_zone = 200 ;

But if tp_track and tp_scan have the same size, and track_interpolation_zone and scan_interpolation_zone also have the size, then are we in trouble? E.g.
  tp_track = 96 ;
  tp_scan = 96 ;
  track_interpolation_zone = 48 ;
  scan_interpolation_zone = 48 ;

Hi @davidhassell

Hi - I was wondering about how we can tell which interpolation zone dimensions correspond to which tie point dimensions.

Would a solution be to change:
  // Interpolation variable
  char tp_interpolation ;
    tp_interpolation:interpolation_name = "bi_quadratic_1" ;
    tp_interpolation:interpolation_coefficients = "expansion_coefficient_track alignment_coefficient_track expansion_coefficient_scan alignment_coefficient_scan" ;
    tp_interpolation:interpolation_flags = "interpolation_zone_flags" ;

  // Interpolation variable
  char tp_interpolation ;
    tp_interpolation:interpolation_name = "bi_quadratic_1" ;
    tp_interpolation:interpolation_coefficients = "(tp_track, tp_scan): expansion_coefficient_track alignment_coefficient_track expansion_coefficient_scan alignment_coefficient_scan" ;
    tp_interpolation:interpolation_flags = "(tp_track, tp_scan): interpolation_zone_flags" ;

using a notation similar to the one for the tie_point_indices and tie_point_offset attributes?

Hi @AndersMS
Good idea. I'm inclined (a minor change) to drop the parentheses, though and re-use the "colon" notation that is already standard, in this proposal and elsewhere in CF:
// Interpolation variable
  char tp_interpolation ;
    tp_interpolation:interpolation_name = "bi_quadratic_1" ;
    tp_interpolation:interpolation_coefficients = "tp_track: tp_scan: expansion_coefficient_track alignment_coefficient_track expansion_coefficient_scan alignment_coefficient_scan" ;
    tp_interpolation:interpolation_flags = "tp_track: tp_scan: interpolation_zone_flags" ;

We'd need to state in the conventions that the order of dimensions of each coefficient variable must be a) the same and b) match the order given (i.e. "tp_track: tp_scan:" forces t the variables to have dimensions (tp_track, tp_scan)).
I'm happy insisting on a dimension order for these variable, as other "related" variables in CF already have the same restriction (e.g. coordinate bounds variables must have te same dimension order as their parent coordinates).

Hi @davidhassell
Good. The "colon" notation is fine with me, in particular if this is in line with standard notation in CF.
The sentence

(i.e. "tp_track: tp_scan:" forces t the variables to have dimensions (tp_track, tp_scan)).

is probably a bit too strong.
In our example we have:
  // Interpolation coefficients and flags
  short expansion_coefficient_track(track_interpolation_zone, tp_scan) ;
  short alignment_coefficient_track(track_interpolation_zone, tp_scan) ;
  short expansion_coefficient_scan(tp_track, scan_interpolation_zone) ;
  short alignment_coefficient_scan(tp_track, scan_interpolation_zone) ;

So the notation either associates dimensions (track_interpolation_zone with tp_track and scan_interpolation_zone with tp_scan) or state that they are identical.
Would you agree?

It is then equivalent to the way we associate tie point dimensions with target dimensions, in this example m_track with tp_track:
m_radiance: tie_point_indices = "m_track: m_track_indices  m_scan: m_scan_indices  time_scan: m_time_scan_indices" ;

int m_track_indices(tp_track)

Hello, I was reading slide 14 of the presentation that describes interpolation zones and interpolation groups. On the right hand side of the slide it says "Interpolation within each interpolation zone using the interpolation method", but do you not interpolate zones within a group simultaneously? E.g.  in this example, zones A and B would be interpolated simultaneously using the six leftmost tie points:
.     .     ..     .
   A     B      C
.     .     ..     .

This is not an encoding question, rather on interpretation of the final product.
Thanks.

Another question - what about periodic coordinates, such as longitude, that wrap-around the whole Earth. This is never an issue for remote sensing, I imagine. But it could come up in general. The issue is for the interpolation implementation to know that it has to account for tie points either side of the the "edges" of the earth (seen as a rectangular array).
I think that it may be enough to say in the text that if the first and last tie points (or their appropriate bounds) along the tie point dimension are same modulo 360, then wrap-around interpolation should be used.
Thanks,
David

Hi @davidhassell,
You wrote:

Hello, I was reading slide 14 of the presentation that describes interpolation zones and interpolation groups. On the right hand side of the slide it says "Interpolation within each interpolation zone using the interpolation method", but do you not interpolate zones within a group simultaneously?

There are two levels to this:

Within a group, interpolation zones will share common tie points as well as interpolation coefficients and flags belonging to common vertices or edges. None of this sharing takes place between interpolation zones belonging to different groups.
However, beyond the above sharing of static data, the actual interpolation computations can be done fully independent for each interpolation zone, making them an excellent candidate for parallel computation.

The main purpose of the groups is to safeguard against geometric discontinuities in the grid that may be present even if the data is stored in a single two dimensional array.
Did that answer the question?

Hi @AndersMS, re. interpolation zones:
Thanks - I think I get it now. It might be better, in that case, if it was stated that each zone's interpolation must be carried independently of the others, just to remove any ambiguities that could arise from perhaps, negative offsets (I'm wholly sure that this is a problem, but it seems like it could at least cause confusion).

Hi @davidhassell,
You wrote:

what about periodic coordinates, such as longitude, that wrap-around the whole Earth.

I does actually come up in remote sensing also. There are several ways to deal with it.
In the VIIRS example  we set the interpolation_zone_flags to "location_use_cartesian" for all interpolation zones that are spanning the date line (180 deg longitude). The cartesian coordinates doesn't have the wrap-around issue. This approach would be consistent with the style of  putting complexity and decision making on the data provider side and keeping the user side algorithms as simple as possible.
Another method that could be applied by a method is to check if differences in tie point longitudes are larger than 180 degrees, and if so, subtract/add 360 degrees from those longitudes before interpolating.
Finally, we could also consider a notation to mark a dimension that have wrap-around behaviour. See also the 3rd bullet on this slide  from the mesh discussion at the 2020 CF meeting.

Hi @davidhassell, regarding interpolation zones:

It might be better, in that case, if it was stated that each zone's interpolation must be carried independently of the others

Thank you for the proposal, I agree and I will include that in the text.
Yes, a negative offset could be confusing, but it would change the above principle of independent interpolation.

Yes, a negative offset could be confusing, but it would change the above principle of independent interpolation.

Just to be sure, are you saying that if the offset is negative then the zone interpolations are not independent?

No, it was a typo, I have corrected it now:
Yes, a negative offset could be confusing, but it would not change the above principle of independent interpolation.

The offsets are geometric offsets and they don't change the zone and group logic.

Thanks  the wrap-around information, @AndersMS. I need to think a bit more to try to understand  these situations better -  I'm mainly thinking of the case when the first and last tie point longitudes (i.e. at either ends of the tie point dimension) are in fact adjacent in physical space. But I'm finishing for the day, now.....

Hello,
In the general case when spherically regridding 2-D lats and lons to another set of 2-D lats and lons, it is necessary to know which axis is "X" and which is "Y". Their order in the arrays doesn't matter, as long as their identities are known.
I presume that this extends to, say scan and track dimensions - you need to know which is which. Is that right?

Hi,
In the example CDL the lat and lon tie point variables are two dimensional (X and  Y). Could they also have, say, a time dimension? In that case how would we know which of the three dimensions are the two dimensions that need to horizontally interpolated?
Also, if time interpolation is in play, we'd need to know which is the time dimension, and would have to  add lat: lon: time: time_interpolation, I presume (?).
I'm sure that this is possible, but I can't quite see it this morning.
Thnaks,
David

Hi @davidhassell
You are asking very good questions!

In the general case when spherically regridding 2-D lats and lons to another set of 2-D lats and lons, it is necessary to know which axis is "X" and which is "Y". Their order in the arrays doesn't matter, as long as their identities are known.
I presume that this extends to, say scan and track dimensions - you need to know which is which. Is that right?

In our original method for VIIRS data (https://doi.org/10.1016/j.acags.2020.100025) we do indeed need to know what is scan and track. The reason is that we model the scanning geometry of the instrument.
In the improved method I am proposing for our present work, we do not need to know what is track and scan, as we are only looking at the lat/lon locations, without assumptions on how they were generated from a scanning instrument.
In response to your question I have looked a bit into general purpose bi-linear and bi-quadratic (and in principle higher order) interpolation methods. They are all invariant to the order of dimensions, so they yield the same interpolation results if you apply them to a (X, Y) grid and a (Y, X) grid, apart from the results being swapped accordingly. So you wouldn't need to know.
I have to check if the bi-quadratic method that I explained recently (slide 11-14 here), actually has this property too. I will come back to you on that.
So if we limit ourselves  to general purpose interpolation methods, we could do without having to associate the dimensions of the grid with particular dimension of the interpolation method.
However, if we want to remain open for interpolation methods that rely on dimension like scan/track, then we would need a CDL notation to cover that case. Probably an attribute in the interpolation variable would do.
Possibly my preference would be that, for now, we limit ourselves to general purpose interpolation methods, not needing this mapping.
What do you think?

Hi @davidhassell

In the example CDL the lat and lon tie point variables are two dimensional (X and Y). Could they also have, say, a time dimension? In that case how would we know which of the three dimensions are the two dimensions that need to horizontally interpolated?

In the example you reference we already have time as an auxiliary coordinate variable. I guess you are now referring to having time as a coordinate variable. So you would have a lat/lon grid per time value, each of which would need separate interpolation? I guess that would work, but all the values stored in the interpolation variable would need that time dimension as well.

Also, if time interpolation is in play, we'd need to know which is the time dimension, and would have to add lat: lon: time: time_interpolation, I presume (?).

Can we actually have time twice, both as a coordinate variable and an auxiliary coordinate variable? I am not sure I fully understand the case.
I guess that if we require our coordinates to have standard name (as we said a method could be specified to require), then that would let us tell what is time and what is spatial coordinates.

@AndersMS @davidhassell I agree with Anders' comment that

Possibly my preference would be that, for now, we limit ourselves to general purpose interpolation methods, not needing this mapping.

As long as we don't have a concrete use case in mind I would prefer to focus on what we know we need to do. Forward compatibility is good, but it looks like this will be solvable if and when we need to solve it, and like we don't need to solve special-case interpolation methods right now. What do you think?

@erget @davidhassell

but it looks like this will be solvable if and when we need to solve it, and like we don't need to solve special-case interpolation methods right now

I would agree that it could be added later and I would support to let it rest for now.
However, it would probably be good to discuss the topic briefly now, in order to demonstrate that we would have an elegant way to add this later.
Correcting what I wrote above:

Probably an attribute in the interpolation variable would do.

I think that attribute would have to be on the data variable, as the association of dimensions would be different for different resolution, say VIIRS I and M band. For the VIIRS example, it could be something like:
 m_radiance:tie_point_dimensions = "m_track: track  m_scan: scan" ;
....
 i_radiance:tie_point_dimensions = "i_track: track  i_scan: scan" ;

where track and scan are dimension names known by the interpolation method "bi_quadratic_1".

Hi @davidhassell

For unstructured grids (such as tripolar which, deep down, I was fixating on) you do need to know, but you can't subsample and interpolate those grids anyway, so they are not a valid use case. So, I think we're OK on that front, with no changes.

I would find it interesting to see how ell we could interpolate a tripolar grid. But it is not a high priority to ty that out right now and you my well be right that it wouldn't work.
Just to keep the matter semi-open, did you like the notation proposed above?

Hi @davidhassell, @erget

However, I think this example does not relate to track/scan case, as for that we have lat(track, scan), lon(track, scan), time(track, scan), i.e. no 1-D time axis. Is that right?

Yes, that is right.
As I currently understand it, time can be appear in our interpolation in two different ways, as an auxiliary coordinate variable or as a coordinate variable. But it can't appear as both at the same time I think (?).
The track/scan case is an example of time as an auxiliary coordinate variable, and I think we have covered that case well already.
In the 3-D lats and lons: (T, Y, X) case, time is a coordinate variable. An example could be a sequence of 2D images taken from a drone at different times t1, t2, ... t10 as the drone is flying along its trajectory. As the individual images are taken instantaneously, all pixels of image 1 has the time stamp t1 and all pixels of image 2 has the time stamp t2. The locations of the pixels are different between image 1 and image 2, since the drone is moving along its trajectory.
If we store time at full resolution and have no interpolation in time, then both the tie point lat/lons and the target resolution lat/lons would have the time dimension at target resolution. Time would work just as an index, and the lat/lons would be interpolated independently for each time t1, t2, ... as in this example:
Example A - No time interpolation
dimensions :
  time = 10 ;
  image_x = 1000;
  image_y = 1000;

  tp_image_x = 100;
  tp_image_y = 100;

variables:
  float radiance(time, image_x , image_y) ;
    radiance:tie_points = "lat: lon:  tp_interpolation" ;
    radiance:tie_point_indices = "image_x : x_indices  image_y: y_indices" ;

  // Time as coordinate variable at target resolution
  double time(time) ;
    t : long_name = "time" ;
    t : units = "days since 1990-1-1 0:0:0" ;

  // Tie point indices
  int x_indices(tp_image_x ) ;
  int y_indices(tp_image_y ) ;

  // Tie points
  float lat(time, tp_image_x , tp_image_y) ;
    lat : standard_name = "latitude" ;
    lat : units = "degrees_north" ;
  float lon(time, tp_image_x , tp_image_y, time) ; ;
    lon : standard_name = "longitude" ;
    lon : units = "degrees_east" ;

  // Interpolation variable
  char tp_interpolation ;
    tp_interpolation:interpolation_name = "bi_linear" ;

If the drone motion is sufficiently regular, we could consider storing the tie point lat/lons only at a subset of times t1, t2..., say every fifth time stamp. In this case we would have the tie point lat/lons at a lower resolution and would need to interpolate to get the tie point lat/lons at the full time resolution. But again both the tie point lat/lons and the target resolution lat/lons would both have the time dimension as in this example:
Example B - Time interpolation
dimensions :
  time= 10 ;
  image_x = 1000;
  image_y= 1000;

  tp_time = 2
  tp_image_x = 100;
  tp_image_y = 100;

variables:
  float radiance(image_x , image_y , time) ;
    radiance:tie_points = "lat: lon: time  tp_interpolation" ;
    radiance:tie_point_indices = "image_x : x_indices  image_y: y_indices" ;

  // Time as coordinate variable at tie point resolution
  double time(time) ;
    t : long_name = "time" ;
    t : units = "days since 1990-1-1 0:0:0" ;

  // Tie point indices
  int x_indices(tp_image_x ) ;
  int y_indices(tp_image_y ) ;
  int time_indices(tp_time ) ;

  // Tie points
  double time(tp_time) ;
    t : long_name = "time" ;
    t : units = "days since 1990-1-1 0:0:0" ;
  float lat(tp_time, tp_image_x , tp_image_y) ;
    lat : standard_name = "latitude" ;
    lat : units = "degrees_north" ;
  float lon(tp_time, tp_image_x , tp_image_y) ;
    lon : standard_name = "longitude" ;
    lon : units = "degrees_east" ;

  // Interpolation variable
  char tp_interpolation ;
    tp_interpolation:interpolation_name = "bi_linear_with_linear_time" ;

So I think the proposal

Can we solve all this by saying that the tie_point variables must only span the dimensions that are being interpolated, with no others?

would not work as demonstrated by the above examples, which include time as dimension of the tie points, even if no interpolation in time is required.
If we are only talking about time in addition to one or more spatial dimensions, one solution would be that a method may require standard_name to be given for the coordinates, in the same way as we agreed to do to distinguish lat/lon and azi/zen. In this case the standard name would also reveal the time dimension as being time, since it is a coordinate variable. And the method would know to treat time differently from the spatial coordinates.
The dimensions that actually require interpolation are clear from the tie_point_indices attribute and the related indices variables.
If we have more dimensions than the spatial and time dimensions, then the best would probably be to introduce the new tie_point_dimensions  attribute on the data variable, as proposed here .
For this case, we could consider all dimensions that are not listed in the tie_point_dimensions  as non-interpolation dimensions, for which no interpolation is performed. Rather, the interpolation would be repeated independently for each combination of the non-interpolation dimensions. The non-interpolation dimensions would be present in both tie point resolution variables and target resolution variables. (probably a better attribute name than tie_point_dimensions  would be interpolation_dimensions ).
I believe the above would be in line with

that the interpolation method cannot be "compound", i.e. cannot be the union of two independent interpolations. Non-compound interpolations are good, because we avoid trouble relating to dimension identification, order of application, etc.

as both examples have a single interpolation method, with the method in example B handling both time and spatial coordinates.
Let me know what you think.

Hi @AndersMS,
Re. the tripolar example, the link you mentioned in this post did not resolve - could you re-send it?
On time interpolation., I agree with your CDL examples above with the exception in the second (with time interpolation) case that shouldn't time also be in the tie point indices:
  radiance:tie_point_indices = "image_x : x_indices  image_y: y_indices time: time_indices" ;

I think we concur that
a) tie point variable must only contain interpolated dimensions
b) each tie point must only be associated with one interpolation container
In that case, I'm not sure what you mean with the interpolation_dimensions. The issue is not with the data variable having non-interpolated dimensions, only the tie point coordinates, and by insisting that all dimension of the tie point coordinates are to be interpolated, all ambiguity is removed. The non-interpolation dimensions of the data variable are then easily determined as those for which there are no tie point indices. In this example, the data variable two non-interpolation dimensions:
dimensions :
  time= 10 ;
  image_x = 1000;
  image_y= 1000;
  dim1 =<PHONE_NUMBER>000 ;
  dim2 = 3 ;

  tp_time = 2
  tp_image_x = 100;
  tp_image_y = 100;

variables:
  float radiance(dim1, image_x , image_y , dim2, time) ;  // 2 extra dimensions
    radiance:tie_points = "lat: lon: time  tp_interpolation" ;
    radiance:tie_point_indices = "image_x : x_indices  image_y: y_indices time: time_indices" ;

  // Time as coordinate variable at tie point resolution
  double time(time) ;
    t : long_name = "time" ;
    t : units = "days since 1990-1-1 0:0:0" ;

  // Tie point indices
  int x_indices(tp_image_x ) ;
  int y_indices(tp_image_y ) ;
  int time_indices(tp_time ) ;

  // Tie points
  double time(tp_time) ;
    t : long_name = "time" ;
    t : units = "days since 1990-1-1 0:0:0" ;
  float lat(tp_time, tp_image_x , tp_image_y) ;
    lat : standard_name = "latitude" ;
    lat : units = "degrees_north" ;
  float lon(tp_time, tp_image_x , tp_image_y) ;
    lon : standard_name = "longitude" ;
    lon : units = "degrees_east" ;

  // Interpolation variable
  char tp_interpolation ;
    tp_interpolation:interpolation_name = "bi_linear_with_linear_time" ;

All the best,
David

Hi @davidhassell
I have fixed the unresolved link in this post .
You wrote:

On time interpolation., I agree with your CDL examples above with the exception in the second (with time interpolation) case that shouldn't time also be in the tie point indices:

You are right, I have corrected the CDL example to include time: time_indices
You wrote:

I think we concur that
a) tie point variable must only contain interpolated dimensions

Here I am not sure we concur, but there may well be a misunderstanding on my side. In the Example A above there is no time interpolation, but still, we have tie point variables with time as dimension:
  // Tie points
  float lat(time, tp_image_x , tp_image_y) ;
    lat : standard_name = "latitude" ;
    lat : units = "degrees_north" ;

Do you see what I am trying to get at? Or am I mistaken?

dimensions:
y = 228;
x = 306;
time = 41;
// Tie point dimensions
tp_y = 58;
tp_x = 52;
variables:
int lambert_conformal ;
lambert_conformal:grid_mapping_name = "lambert_conformal_conic" ;
lambert_conformal:standard_parallel = 25.0 ;
lambert_conformal:longitude_of_central_meridian = 265.0 ;
lambert_conformal:latitude_of_projection_origin = 25.0 ;
double time(time) ;
time:standard_name = "time" ;
time:units = "days since 2021-03-01" ;
// Interpolation variables
char spherical_bilinear ;
bilinear:interpolation_name = "spherical_bilinear" ;
char linear ;
linear:interpolation_name = "linear" ;
// Tie point variables
double y(tp_y) ;
y:units = "km" ;
y:standard_name = "projection_y_coordinate" ;
double x(tp_x) ;
x:units = "km" ;
x:standard_name = "projection_x_coordinate" ;
double lat(tp_y, tp_x) ;
lat:units = "degrees_north" ;
lat:standard_name = "latitude" ;
double lon(tp_y, tp_x) ;
lon:units = "degrees_east" ;
lon:standard_name = "longitude" ;
// Tie point index variables
int y_indices(tp_y) ;
y.long_name	= "Mapping of y dimension to its ",
"corresponding tie point dimension" ;
int x_indices(tp_x) ;
x.long_name = "Mapping of x dimension to its ",
"corresponding tie point dimension" ;
// Data variable
float Temperature(time, y, x) ;
Temperature:standard_name = "air_temperature" ;
Temperature:units = "K" ;
Temperature:grid_mapping = "lambert_conformal" ;
Temperature:tie_points = "lat: lon: spherical_bilinear y: x: linear" ;
Temperature:tie_point_indices = "y: y_indices x: x_indices" ;

Three grid mapping examples, of increasing complexity. Hope they make sense!
A No 2-D latitudes or longitudes:
dimensions:
  y = 228;
  x = 306;
  time = 41;

  // Tie point dimensions
  tp_y = 58;
  tp_x = 52;

variables:
  int lambert_conformal ;
    lambert_conformal:grid_mapping_name = "lambert_conformal_conic" ;
    lambert_conformal:standard_parallel = 25.0 ;
    lambert_conformal:longitude_of_central_meridian = 265.0 ;
    lambert_conformal:latitude_of_projection_origin = 25.0 ;
  double time(time) ;
    time:standard_name = "time" ;
    time:units = "days since 2021-03-01" ;

  // Interpolation variables
  char linear ;
    linear:interpolation_name = "linear" ;

  // Tie point variables
  double y(tp_y) ;
    y:units = "km" ;
    y:standard_name = "projection_y_coordinate" ;
  double x(tp_x) ;
    x:units = "km" ;
    x:standard_name = "projection_x_coordinate" ;

  // Tie point index variables
  int y_indices(tp_y) ;
    y.long_name	= "Mapping of y dimension to its ",
                  "corresponding tie point dimension" ;
  int x_indices(tp_x) ;
    x.long_name = "Mapping of x dimension to its ",
                  "corresponding tie point dimension" ;

  // Data variable
  float Temperature(time, y, x) ;
    Temperature:standard_name = "air_temperature" ;
    Temperature:units = "K" ;
    Temperature:grid_mapping = "lambert_conformal" ;
    Temperature:tie_points = y: x: linear" ;
    Temperature:tie_point_indices = "y: y_indices x: x_indices" ;

B With 2-D latitudes and longitudes
dimensions:
  y = 228;
  x = 306;
  time = 41;

  // Tie point dimensions
  tp_y = 58;
  tp_x = 52;

variables:
  int lambert_conformal ;
    lambert_conformal:grid_mapping_name = "lambert_conformal_conic" ;
    lambert_conformal:standard_parallel = 25.0 ;
    lambert_conformal:longitude_of_central_meridian = 265.0 ;
    lambert_conformal:latitude_of_projection_origin = 25.0 ;
  double time(time) ;
    time:standard_name = "time" ;
    time:units = "days since 2021-03-01" ;

  // Interpolation variables
  char spherical_bilinear ;
    bilinear:interpolation_name = "spherical_bilinear" ;
  char linear ;
    linear:interpolation_name = "linear" ;

  // Tie point variables
  double y(tp_y) ;
    y:units = "km" ;
    y:standard_name = "projection_y_coordinate" ;
  double x(tp_x) ;
    x:units = "km" ;
    x:standard_name = "projection_x_coordinate" ;
  double lat(tp_y, tp_x) ;
    lat:units = "degrees_north" ;
    lat:standard_name = "latitude" ;
  double lon(tp_y, tp_x) ;
    lon:units = "degrees_east" ;
    lon:standard_name = "longitude" ;

  // Tie point index variables
  int y_indices(tp_y) ;
    y.long_name	= "Mapping of y dimension to its ",
                  "corresponding tie point dimension" ;
  int x_indices(tp_x) ;
    x.long_name = "Mapping of x dimension to its ",
                  "corresponding tie point dimension" ;

  // Data variable
  float Temperature(time, y, x) ;
    Temperature:standard_name = "air_temperature" ;
    Temperature:units = "K" ;
    Temperature:grid_mapping = "lambert_conformal" ;
    Temperature:tie_points = "lat: lon: spherical_bilinear y: x: linear" ;
    Temperature:tie_point_indices = "y: y_indices x: x_indices" ;

C Also with time interpolation. In this case the 3-D interpolation variable has not standardised name, rather some explicit instructions on what to do, and in what order.

variables:
  int lambert_conformal ;
    lambert_conformal:grid_mapping_name = "lambert_conformal_conic" ;
    lambert_conformal:standard_parallel = 25.0 ;
    lambert_conformal:longitude_of_central_meridian = 265.0 ;
    lambert_conformal:latitude_of_projection_origin = 25.0 ;

  // Interpolation variables
  char time_horizontal ;
    bilinear:description = "1. Identify the time dimension from the 1-D time tie point variable"
                           "2. Do spherical bilinear interpolation on the other two interpolated dimensions",
                           "3. Do linear interpolation on the time dimension" ;
  char bilinear ;
    linear:interpolation_name = "bilinear" ;
  char linear ;
    linear:interpolation_name = "linear" ;

  // Tie point variables
  double time(tp_time) ;
    time:standard_name = "time" ;
    time:units = "days since 2021-03-01" ;
  double y(tp_time, tp_y) ;
    y:units = "km" ;
    y:standard_name = "projection_y_coordinate" ;
  double x(tp_time, tp_x) ;
    x:units = "km" ;
    x:standard_name = "projection_x_coordinate" ;
  double lat(tp_time, tp_y, tp_x) ;
    lat:units = "degrees_north" ;
    lat:standard_name = "latitude" ;
  double lon(tp_time, tp_y, tp_x) ;
    lon:units = "degrees_east" ;
    lon:standard_name = "longitude" ;

  // Tie point index variables
  int time_indices(tp_time) ;
    x.long_name = "Mapping of time dimension to its ",
                  "corresponding tie point dimension" ;
  int y_indices(tp_y) ;
    y.long_name	= "Mapping of y dimension to its ",
                  "corresponding tie point dimension" ;
  int x_indices(tp_x) ;
    x.long_name = "Mapping of x dimension to its ",
                  "corresponding tie point dimension" ;

  // Data variable
  float Temperature(time, y, x) ;
    Temperature:standard_name = "air_temperature" ;
    Temperature:units = "K" ;
    Temperature:grid_mapping = "lambert_conformal" ;
    Temperature:tie_points = "lat: lon: time_horizontal y: x: bilinear time: linear" ;
    Temperature:tie_point_indices = "y: y_indices x: x_indices "time: time_indices" ;

After a very interesting discussion with Anders, I have updated the grid mapping examples, including a new example, some explanatory notes, improved text, and fixing some typos.
Thanks

hi @davidhassell
Would this be a way to link the interpolation zone dimension with the other dimensions?
dimensions :
  // VIIRS M-Band (750 m resolution imaging)
  m_track = 768 ;
  m_scan = 3200 ;
  m_channel = 16 ;

  // VIIRS I-Band (375 m resolution imaging)
  i_track = 1536 ;
  i_scan = 6400 ;
  i_channel = 5 ;

  // Tie points and interpolation zones (shared between VIIRS M-Band and I-Band)
  tp_track = 96 ;  // 48 VIIRS scans
  tp_scan = 205 ;
  zone_track = 48 ;   // track interpolation zone
  zone_scan= 200 ;  // scan interpolation zone

  // Time, stored at scan-start and scan-end of each scan
  time_scan = 2;

variables:
  // VIIRS M-Band
  float m_radiance(m_track, m_scan, m_channel) ;
    m_radiance:tie_points = "m_lat: m_lon: m_sen_azi_ang: m_sen_zen_ang: m_sol_azi_ang: m_sol_zen_ang: tp_interpolation  t: time_interpolation" ;
    m_radiance:tie_point_indices = "m_track: zone_track : m_track_indices  m_scan: zone_scan: scan_iz: m_scan_indices  time_scan: m_time_scan_indices" ;
    m_radiance:tie_point_offsets = "m_track: offset  m_scan: offset" ;

  // VIIRS I-Band
  float i_radiance(i_track, i_scan, i_channel) ;
    i_radiance:tie_points = "lat: lon: sen_azi_ang: sen_zen_ang: sol_azi_ang: sol_zen_ang: tp_interpolation  t: time_interpolation" ;
    i_radiance:tie_point_indices = "i_track: zone_track: i_track_indices  i_scan: zone_scan: i_scan_indices  time_scan: i_time_scan_indices" ;
    i_radiance:tie_point_offsets = "i_track: offset  i_scan: offset" ;

  double offset;   // = -0.5

  // Tie point index variables
  int m_track_indices(tp_track) ;   // shared by tp_interpolation and time_interpolation
  int m_scan_indices(tp_scan) ;
  int m_time_scan_indices(time_scan)
  int i_track_indices(tp_track) ;   // shared by tp_interpolation and time_interpolation
  int i_scan_indices(tp_scan) ;
  int i_time_scan_indices(time_scan)

  // Tie points
  float m_lat(tp_track, tp_scan) ;
    m_lat : standard_name = "latitude" ;
    m_lat : units = "degrees_north" ;
  float m_lon(tp_track, tp_scan) ;
    m_lon : standard_name = "longitude" ;
    m_lon : units = "degrees_east" ;
  float sen_azi_ang(tp_track, tp_scan) ;
    m_sen_azi_ang : standard_name = "sensor_azimuth_angle" ;
    m_sen_azi_ang : units = "degrees" ;
  float sen_zen_ang(tp_track, tp_scan) ;
    m_sen_zen_ang : standard_name = "sensor_zenith_angle" ;
    m_sen_zen_ang : units = "degrees" ;
  float sol_azi_ang(tp_track, tp_scan) ;
    m_sol_azi_ang : standard_name = "solar_azimuth_angle" ;
    m_sol_azi_ang : units = "degrees" ;
  float sol_zen_ang(tp_track, tp_scan) ;
    m_sol_zen_ang : standard_name = "solar_zenith_angle" ;
    m_sol_zen_ang : units = "degrees" ;

.......

  float lat(tp_track, tp_scan) ;
    lat : standard_name = "latitude" ;
    lat : units = "degrees_north" ;
  float lon(tp_track, tp_scan) ;
    lon : standard_name = "longitude" ;
    lon : units = "degrees_east" ;
  float sen_azi_ang(tp_track, tp_scan) ;
    sen_azi_ang : standard_name = "sensor_azimuth_angle" ;
    sen_azi_ang : units = "degrees" ;
  float sen_zen_ang(tp_track, tp_scan) ;
    sen_zen_ang : standard_name = "sensor_zenith_angle" ;
    sen_zen_ang : units = "degrees" ;
  float sol_azi_ang(tp_track, tp_scan) ;
    sol_azi_ang : standard_name = "solar_azimuth_angle" ;
    sol_azi_ang : units = "degrees" ;
  float sol_zen_ang(tp_track, tp_scan) ;
    sol_zen_ang : standard_name = "solar_zenith_angle" ;
    sol_zen_ang : units = "degrees" ;

  // Interpolation variable
  char tp_interpolation ;
    tp_interpolation:interpolation_name = "bi_quadratic_1" ;
    tp_interpolation:interpolation_coefficients = "exp1: expansion1  align1: alignment1  exp2: expansion2  align2:alignment2" ;
    tp_interpolation:interpolation_configuration = "flags: interpolation_zone_flags" ;

  // Interpolation coefficient and configuration variables
  short expansion1(zone_track , tp_scan) ;
  short alignment1(zone_track , tp_scan) ;
  short expansion2(tp_track, zone_scan) ;
  short alignment2(tp_track, zone_scan) ;
  byte interpolation_zone_flags(zone_track , zone_scan) ;
    interpolation_zone_flags : valid_range = "1b, 7b" ;
    interpolation_zone_flags : flag_masks = "1b, 2b, 4b" ;
    interpolation_zone_flags : flag_meanings = "location_use_cartesian  sensor_direction_use_cartesian  solar_direction_use_cartesian" ;

  // Time tie points
  double t(tp_track, time_scan) ;
    t : long_name = "time" ;
    t : units = "days since 1990-1-1 0:0:0" ;

  // Time interploation variable
  char time_interpolation ;
    time_interpolation : interpolation_name = "bi_linear" ;

hi @davidhassell
Would this be a way to link the interpolation zone dimension with the other dimensions?
dimensions :
  // VIIRS M-Band (750 m resolution imaging)
  m_track = 768 ;
  m_scan = 3200 ;
  m_channel = 16 ;

  // VIIRS I-Band (375 m resolution imaging)
  i_track = 1536 ;
  i_scan = 6400 ;
  i_channel = 5 ;

  // Tie points and interpolation zones (shared between VIIRS M-Band and I-Band)
  tp_track = 96 ;  // 48 VIIRS scans
  tp_scan = 205 ;
  zone_track = 48 ;   // track interpolation zone
  zone_scan= 200 ;  // scan interpolation zone

  // Time, stored at scan-start and scan-end of each scan
  time_scan = 2;

variables:
  // VIIRS M-Band
  float m_radiance(m_track, m_scan, m_channel) ;
    m_radiance:tie_points = "m_lat: m_lon: m_sen_azi_ang: m_sen_zen_ang: m_sol_azi_ang: m_sol_zen_ang: tp_interpolation  t: time_interpolation" ;
    m_radiance:tie_point_indices = "m_track: zone_track : m_track_indices  m_scan: zone_scan: scan_iz: m_scan_indices  time_scan: m_time_scan_indices" ;
    m_radiance:tie_point_offsets = "m_track: offset  m_scan: offset" ;

  // VIIRS I-Band
  float i_radiance(i_track, i_scan, i_channel) ;
    i_radiance:tie_points = "i_lat: i_lon: i_sen_azi_ang: i_sen_zen_ang: i_sol_azi_ang: i_sol_zen_ang: i_tp_interpolation  t: time_interpolation" ;
    i_radiance:tie_point_indices = "i_track: zone_track: i_track_indices  i_scan: zone_scan: i_scan_indices  time_scan: i_time_scan_indices" ;
    i_radiance:tie_point_offsets = "i_track: offset  i_scan: offset" ;

  double offset;   // = -0.5

  // Tie point index variables
  int m_track_indices(tp_track) ;   // shared by tp_interpolation and time_interpolation
  int m_scan_indices(tp_scan) ;
  int m_time_scan_indices(time_scan)
  int i_track_indices(tp_track) ;   // shared by tp_interpolation and time_interpolation
  int i_scan_indices(tp_scan) ;
  int i_time_scan_indices(time_scan)

  // Tie points
  float m_lat(tp_track, tp_scan) ;
    m_lat : standard_name = "latitude" ;
    m_lat : units = "degrees_north" ;
  float m_lon(tp_track, tp_scan) ;
    m_lon : standard_name = "longitude" ;
    m_lon : units = "degrees_east" ;
  float sen_azi_ang(tp_track, tp_scan) ;
    m_sen_azi_ang : standard_name = "sensor_azimuth_angle" ;
    m_sen_azi_ang : units = "degrees" ;
  float sen_zen_ang(tp_track, tp_scan) ;
    m_sen_zen_ang : standard_name = "sensor_zenith_angle" ;
    m_sen_zen_ang : units = "degrees" ;
  float sol_azi_ang(tp_track, tp_scan) ;
    m_sol_azi_ang : standard_name = "solar_azimuth_angle" ;
    m_sol_azi_ang : units = "degrees" ;
  float sol_zen_ang(tp_track, tp_scan) ;
    m_sol_zen_ang : standard_name = "solar_zenith_angle" ;
    m_sol_zen_ang : units = "degrees" ;

  float i_lat(tp_track, tp_scan) ;
    i_lat : standard_name = "latitude" ;
    i_lat : units = "degrees_north" ;

.............

  // Interpolation variable
  char tp_interpolation ;
    tp_interpolation:interpolation_name = "bi_quadratic_1" ;
    tp_interpolation:interpolation_coefficients = "exp1: expansion1  align1: alignment1  exp2: expansion2  align2:alignment2" ;
    tp_interpolation:interpolation_configuration = "flags: interpolation_zone_flags" ;

  // Interpolation coefficient and configuration variables
  short expansion1(zone_track , tp_scan) ;
  short alignment1(zone_track , tp_scan) ;
  short expansion2(tp_track, zone_scan) ;
  short alignment2(tp_track, zone_scan) ;
  byte interpolation_zone_flags(zone_track , zone_scan) ;
    interpolation_zone_flags : valid_range = "1b, 7b" ;
    interpolation_zone_flags : flag_masks = "1b, 2b, 4b" ;
    interpolation_zone_flags : flag_meanings = "location_use_cartesian  sensor_direction_use_cartesian  solar_direction_use_cartesian" ;

  // Time tie points
  double t(tp_track, time_scan) ;
    t : long_name = "time" ;
    t : units = "days since 1990-1-1 0:0:0" ;

  // Time interploation variable
  char time_interpolation ;
    time_interpolation : interpolation_name = "bi_linear" ;

@davidhassell  and all,
For the figures in the CF document:

Would the orientation of the x and y axes shown below be he conventional/acceptable way?
Which of the two coordinate orderings shown would be  the conventional/acceptable way?

Anders

https://github.com/erget/subsampled-coordinates/issues/10#issuecomment-736384702 - I'd say (x_dim, y_dim).

Hi @erget
That would also be my natural choice, however, in the CF Conventions document most CDL variables are lon(yc,xc)
Try searching on "(y".
So I was wondering what would be the best fit for the CF Conventions document.

We also tend to prefer (y, x) because many languages (C, Python, Java, etc...) provide a view of 2D matrices as arrays of rows.
So you would use matrix[4][2] to access the third column of the fifth row, assuming 0-indexed arrays.
There may be some languages that go column-first but I don't use them.

I'm with @oceandatalab here. I always thought that  (t, z, y, x) was the generally the preferred order in the conventions, because netCDF stores data in a row-major fashion (i.e. arrays are stored by rows first, then by columns), so that the right-most dimension is varying the fastest. It is therefore more performant to put the dimensions that you expect to vary fastest in the commonest access patterns furthest towards the right. And, I guess, slices for a given time are considered the more compelling use case than timeseries at a given location; and there tend to be (in GCMs, at least, the original CF use case) more longitudinal points than latitudinal ones.

Thank you for the feedback! :-)
So in we use:
x-axis is horizontal, positive to the right
y-axis is vertical, positive downwards
and
(y, x) when referencing an array value

Here is a figure I am working on for the CF Conventions text. Comments welcome.

Updated figures for your review. Still one to come for the interpolation coefficient/configuration variables.
Process for generating Interpolation Zones for grids without and with discontinuities:

Overview of the different dimensions:

These pictures are fantastic - thanks, @AndersMS

Next figure....
Through combination of dimensions, interpolation coefficients and configuration variables may provide values for each interpolation zone, for couples of neighboring interpolation zones or for multiple interpolation zones sharing a boundary.:

Hi All,
I have a NetCDF CF question.
A data variable can have angular coordinate pairs with standard names such as:
(latitude, longitude)
(sensor_azimuth_angle, sensor_zenith_angle)
(solar_azimuth_angle, solar_zenith_angle)
Is it permitted for a data variable to have multiple coordinates pairs with the same standard names, like (lat1, lon1) and (lat2, lon2), both with standard name  (latitude, longitude)?
If it is permitted, is it also common?
I am asking because of the way we use the interpolation_configuration attribute to set flags for performing coordinate conversions before and after coordinate interpolation, for example near the polar areas.
If we can assume only one pair of each, then it makes the notation simpler as we can just refer to the standard name.

Hi All,
I have a NetCDF CF question.
A data variable can have angular coordinate pairs with standard names such as:
(latitude, longitude)
(sensor_azimuth_angle, sensor_zenith_angle)
(solar_azimuth_angle, solar_zenith_angle)
Is it permitted for a data variable to have multiple coordinates pairs with the same standard names, like (lat1, lon1) and (lat2, lon2), both with standard name  (latitude, longitude)?
If it is permitted, is it also common?
I am asking because of the way we use the interpolation_configuration attribute to set flags for performing coordinate conversions before and after coordinate interpolation, for example near the polar areas.
If we can assume only one pair of each, then it makes the notation simpler as we can just refer to the standard name.

Hi @AndersMS,
It is indeed permitted, I'm afraid. I suspect that it is "rare but locally abundant", like some butterfly species. If you use it, you may well use it a lot, but I suspect that many people will never to do this.
When NWP forecasting ensembles have multiple time axes, they can have more than one dimension coordinate a standard name of time.
However, it may not be much of a problem, as other parts of CF that need to identify items by standard name do not worry about this. E.g. the following needs a radiation_wavelength coordinate:
downwelling_photon_flux_per_unit_wavelength_in_sea_water

downwelling_photon_flux_per_unit_wavelength_in_sea_water alias: downwelling_spectral_photon_flux_in_sea_water
Downwelling  radiation is radiation from above. It does not mean "net downward". The  sign convention is that "upwelling" is
positive upwards and  "downwelling" is positive downwards. A photon flux is specified in terms  of numbers of photons expressed
in moles. In accordance with common  usage in geophysical disciplines, "flux" implies per unit area, called  "flux density" in
physics. A coordinate variable for radiation  wavelength should be given the standard name radiation_wavelength.
... without worrying about the possibility of there being more than one.
Thanks,
David

Hi @AndersMS,
It is indeed permitted, I'm afraid. I suspect that it is "rare but locally abundant", like some butterfly species. If you use it, you may well use it a lot, but I suspect that many people will never to do this.
When NWP forecasting ensembles have multiple time axes, they can have more than one dimension coordinate a standard name of time.
However, it may not be much of a problem, as other parts of CF that need to identify items by standard name do not worry about this. E.g. the following needs a radiation_wavelength coordinate:
downwelling_photon_flux_per_unit_wavelength_in_sea_water

downwelling_photon_flux_per_unit_wavelength_in_sea_water alias: downwelling_spectral_photon_flux_in_sea_water
Downwelling  radiation is radiation from above. It does not mean "net downward". The  sign convention is that "upwelling" is
positive upwards and  "downwelling" is positive downwards. A photon flux is specified in terms  of numbers of photons expressed
in moles. In accordance with common  usage in geophysical disciplines, "flux" implies per unit area, called  "flux density" in
physics. A coordinate variable for radiation  wavelength should be given the standard name radiation_wavelength.
... without worrying about the possibility of there being more than one.
Thanks,
David

@davidhassell
Thank you for the answer.
Then I would propose that for now we require that an interpolation variable will only be called with one (latitude, longitude) and (sensor_azimuth_angle, sensor_zenith_angle) etc. at a time.
In the rare case of multiple, say (lat1, lon1) and (lat2, lon2), the following data variable attribute should ensure that there is only one pair of (latitude, longitude) per call of tp_interpolation:
tie_points = "lat1: lon1: tp_interpolation  lat2: lon2: tp_interpolation"
Cheers
Anders

@davidhassell
Thank you for the answer.
Then I would propose that for now we require that an interpolation variable will only be called with one (latitude, longitude) and (sensor_azimuth_angle, sensor_zenith_angle) etc. at a time.
In the rare case of multiple, say (lat1, lon1) and (lat2, lon2), the following data variable attribute should ensure that there is only one pair of (latitude, longitude) per call of tp_interpolation:
tie_points = "lat1: lon1: tp_interpolation  lat2: lon2: tp_interpolation"
Cheers
Anders

Dear All,
Finally, with some delay, the Appendix J is now ready for your review.
Available for review here Review PR: Proposed text for subsampled coordinates by erget · Pull Request #1 · erget/cf-conventions (github.com).
Cheers
Anders

Dear All,
I have found a problem with our notation for the compression by subsampling. The problem came to my attention when looking at Example 8.6, VIIRS M- and I-Band in chapter 8.
In the example we are correctly reusing the dimensions (tp_track, tp_scan) for the M-Band and I-band tie points. However, the actual lat/lon values for the M-Band and I-band tie points are now slightly different. In the past, when we used the offsets and the cell corners as tie points, the lat/lon were identical and we had a shared set of lat/lon variables.
The problem is now that in the interpolation container variable, which is referenced by both the M- and I-band radiance data variables, is defining a shared set of interpolation parameters. This is an inconsistency as the calculation of these interpolation parameters depends on the tie point locations (see my note from the last meeting) – and they are now slightly different for the two bands.
Here are two options for solving the problem:

Move the interpolation_parameters attribute from the interpolation variable to the data variable and have different set of parameters for each band. However, then only the interpolation_name is left for the interpolation variable. In option the interpolation variable is rendered pretty useless, as you could just name the interpolation methods directly:
tie_points = "i_lat: i_lon: i_sen_azi_ang: i_sen_zen_ang: i_sol_azi_ang: i_sol_zen_ang: bilinear t: linear" ;
and it loads even more non-physically meaningful metadata onto the parent variable
Reuse between different spatial resolutions (like M- and I-Band) is not possible. The attributes tie_point_dimensions and tie_point_indices are moved to the interpolation variable to ensure consistency and increased sharing. Actual reuse would then be between channels on the same grid. Some people prefer a data variable per band, ch1_radiance(i_track, i_scan), rather than a single data variable, i_radiance(i_track, i_scan, i_channel). This also helps when some channels have different units (radiance, reflectance, brightness temperature), or even when a channel is provided twice with different units (the case for VIIRS). Finally you may have multiple data variables like pressure and temperature on the same grid. All these data variables would then only need a single attribute for interpolation. That would even make it a better more similar to the grid_mapping, where most information is in the container variable. In this option, non-physically meaningful metadata is completely removed from the data variable and provides a name-space for the all of the interpolation components  - i.e. the interpolation variable.

After an exchange with David, we propose option 2) as the solution.
Let me know what you think and let’s discuss the problem in our meeting on Wednesday 7 April.
Anders

Hi @AndersMS from my perspective you make a convincing case for (2). I agree that this sounds like a good solution.

Hi everybody, I (Sylvain) am also in favor of (2), making the interpolation mechanism less intrusive and removing interpolation details from the data variables is perfectly fine with me :)

Dear All,
I am planning to update Example 8.6 in chapter 8 to reflect the discussion above.
VIIRS M- and I-Band sensor data in the level-1 product is give as

Radiance, unit  W/(m2 sr μm)

for all channels and additionally as either

Reflectance, unitless

or

Brightness Temperature, unit K

depending on the channel wave length.
What would be the correct CF standard_name and unit for the VIIRS Radiance, Reflectance and Brightness Temperature?
Anders

Hi @AndersMS I'm a bit puzzled myself - the best I could find is

toa_outgoing_radiance_per_unit_wavelength (W m-2 sr-1 m-1)
toa_bidirectional_reflectance (unitless)
brightness_temperature

Maybe I'm overlooking some equivalent measures in an obvious way - I always use the search function here when looking for standard names.
Note that CF doesn't require the use of a specific unit, as long as the units can be converted by udunits - this normally means as long as the units are scientifically "reasonable" ;)
We're also not required to use standard_name to be CF compliant, but if we discover that there's a geophysical quantity that's relevant in CF data, my preferred route would be to get it adopted into the standard names table ASAP alongside our proposal.

Quick fingers, @erget... 😄
The "bidirectional" is the key part in "toa_bidirectional_reflectance" for me. Is that what the VIIRS radiance data are? Or is the reflectance calculated assuming a Lambertian TOA?

@ajelenak
In the VIIRS documentation I only find reference to Calibrated TOA Reflectance. No mention of Lambertian TOA.
Two references I looked in:
VIIRS Sensor Data Records Data Format
Visible Infrared Imaging Radiometer Suite (VIIRS) Sensor Data Record (SDR) User’s Guide

We're also not required to use standard_name to be CF compliant, but if we discover that there's a geophysical quantity that's relevant in CF data, my preferred route would be to get it adopted into the standard names table ASAP alongside our proposal.

I agree, particulary if we are going to include special instructions for interpolating such variables - we need a guaranteed way of identifying them, and  that has to come in this case from the standard name, I reckon. It's only the "special" spatio-temporal (X, Y, Z, T) variables that have other ways of letting us know what they are (such as by units of degrees_east, etc.).


Schedule page

Current coverage is 42.48% (diff: 11.76%)

Merging #694 into development will decrease coverage by 1.55%

@@           development       #694   diff @@
=============================================
  Files                6          6
  Lines              654        685    +31
  Methods             70         71     +1
  Messages             0          0
  Branches           106        111     +5
=============================================
+ Hits               288        291     +3
- Misses             366        394    +28
  Partials             0          0

Powered by Codecov. Last update ffe1499...ff7fe6e

not yet complete.
this is for https://github.com/fossasia/open-event-webapp/issues/581

complete now
@aayusharora please check and merge

Thank you @championswimmer I am merging this
But it contains design issues
like 1. No HTML rendering in popboxes
2. Showing Speakers in popboxes when they are not present.
3. Some design fixes.
I will make an issue and will resolve them


added a new button

Added a flip button

Thank you for your contribution. 🤩
Star the repo if you like it. 🌟


⚠ Bofrost.lu has degraded performance

In 8b522f8, Bofrost.lu (https://www.bofrost.lu) experienced degraded performance:

HTTP code: 200
Response time: 1809 ms

Resolved: Bofrost.lu performance has improved in d67ff42.


short type is missing

Hello!
Is it possible to add short type?

Yes definitely. Should be there some time next week.

@nzapolski by the way are you interested in contributing? I'm happy for you to do this fix.

I will be happy to do this!

@nzapolski thanks, assigned to you )


Water and energy consumption unavailable

The problem
Hello, the water and energy consumption of a Miele washing machine are not available in this integration, although works fine in the old / other integration. But as this integration correctly reports a lot of info that the other doesn't (washing cycle, rpm, etc) therefore is better, any ideas on how to get those water/energy values?
thanks

Version of miele integration having the issue?
miele-0.0.20
Version of Home Assistant Core having the issue?
core-2022.5.4
Anything in the logs that might be useful for us?
No response
Additional information
No response

I see on your screen dump that the sensor might be disabled. Dive into Advanced settings and enable the sensor, wait for 30 sec and refresh the screen.

You're right, that was it. Totally my fault, sorry and thanks!


(maint) Bump EZBake version

This comit bumps ther version of EZBake
due to a new release

:+1:

@ScottGarman @McdonaldSeanp is there going to be an ANN email about this ezbake release?  It includes a fix for a CI issue that I'd like to mention, in a reply on the ANN thread or something.

@cprice404 Yes, I will be doing so shortly


Jenkins longest job stage analysis script

Signed-off-by: Andrew Leonard<EMAIL_ADDRESS>

Test run: https://ci.adoptopenjdk.net/view/work-in-progress/job/andrew-stage-analysis/26/console


CXL Blog Post Ideas and Keyword Research

Keyword Research:

ubuntu mainline

mainline kernel

ubuntu kernel mainline

ubuntu mainline kernel

mainline kernel install

ubuntu kernel ppa

linux damon

cxl

compute express link

cxl ai (artificial intelligence)

cxl ml (machine learning)

cxl memory

what is cxl

cxl consortium

cxl protocol

cxl meaning

cxl vs pcie

cxl memory pooling

cxl 3.0

cxl 2.0

cxl institute

intel cxl

amd cxl

arm cxl

cxl specification

cxl switch

micron cxl (and other vendors)

cxl memory expander

cxl pcie (cxl vs pcie)

cxl technologies

cxl technology

cxl memory sharing

cxl courses (or training)

cxl conference

cxl interface

cxl bandwidth

cxl latency

fpga cxl

fabric manager cxl

gfam cxl

cxl hdm

cxl hdm decoder

cxl hardware

cxl io

cxl interleaving

cxl ide

linux cxl

cxl nvme

cxl nvlink (cxl vs nvlink)

nvidia cxl

cxl over ethernet

cxl persistent memory

cxl qemu

qemu cxl device

qemu cxl 3.0

qemu cxl

cxl rdma

cxl ras (reliability, availability, serviceability)

cxl types

cxl type 1 device

cxl type 2 device

cxl type 3 device

cxl uio (unordered IO)

cxl ucie

[ ] Demystifying CXL: A Comprehensive Guide

[ ] Understanding Compute Express Link (CXL)

[ ] CXL vs. PCIe: Unraveling the Differences

[ ] Exploring CXL 1.0: Features and Advancements

[ ] Exploring CXL 1.1: Features and Advancements

[ ] Exploring CXL 2.0: Features and Advancements

[ ] Exploring CXL 3.0: Features and Advancements

[ ] Exploring CXL 3.1: Features and Advancements

[ ] CXL Solutions: Memory Expansion and Beyond

[ ] CXL Switches: Managing High-Speed Connections

[ ] CXL Bandwidth and Latency Considerations

[ ] FPGA Integration with CXL: Opportunities and Challenges

[ ] CXL HDM (Host Device Manager): Key Concepts

[ ] CXL Hardware Components: CPUs, GPUs, and More

[ ] Linux Support for CXL: Drivers and Kernel Modules

[ ] CXL Over Ethernet: Extending Reach and Scalability

[ ] CXL Persistent Memory: Unlocking New Possibilities

[ ] QEMU and CXL: Emulating CXL Devices

[ ] CXL RDMA: Enabling High-Speed Networking

[ ] Reliability with CXL: Ensuring System Availability

[ ] Types of CXL Devices: Type 1, 2, and 3 Explained

[ ] CXL UIO (Unordered IO): Managing Asynchronous Data

[ ] CXL UCIE (Unified Communication Interface Extension)

[ ] CXL Security Considerations: Protecting Data Flow (IDE and TEE)

[ ] CXL Interleaving: Enhancing Memory Performance

[ ] CXL Conferences: Networking and Knowledge Sharing

[ ] CXL in AI Workloads: Accelerating Deep Learning

[ ] CXL in AI Workloads: Accelerating Inferencing

[ ] CXL and AI Accelerators: Boosting Training Speeds

[ ] CXL and Machine Learning: Enhancing Training Pipelines

[ ] CXL and Cloud Computing: Scalability and Efficiency

[ ] CXL Performance Metrics: Measuring Throughput

[ ] CXL Fabric Architecture: Building Scalable Networks

[ ] CXL and In-Memory Databases: Accelerating Queries

[ ] CXL and GPU Clusters: Parallel Computing

[ ] CXL and HPC Applications: Scientific Simulations

[ ] CXL and Servers: Data Center Backbone

[ ] Harnessing CXL Type 3 Devices for Scalable AI Infrastructure

[ ] CXL 3.0 Fabrics: The Backbone of Next-Gen Machine Learning Platforms

[ ] Accelerating LLMs with CXL Type 3: A Paradigm Shift in AI Computing

[ ] Optimizing AI Workloads with CXL 2.0 Memory Expansion Capabilities

[ ] CXL: Accelerating AI Inference at the Edge

[ ] Large Language Models and CXL Type 3: Revolutionizing Natural Language Processing

[ ] CXL Fabrics: Enabling Faster Data Transfer for Machine Learning

[ ] CXL: The Key to Unlocking Massive Machine Learning Datasets

[ ] Leveraging CXL Type 3 for High-Speed AI Model Deployment

[ ] Type 3 CXL Devices: A Game Changer for Large Language Model Training

These topics are designed to rank within the top 1-10 results on Google and Bing, leveraging current trends and user interests to drive traffic to your site.

How to Run Llama 2 Locally on Ubuntu 22.04 for a Private ChatGPT Experience

Description: A comprehensive guide on setting up Meta's Llama 2 model locally, providing users with a private and efficient AI assistant.
SEO Potential: High search volume due to Llama 2's popularity; low competition for detailed setup guides.

Setting Up Stable Diffusion on Ubuntu 22.04: Your Guide to Local AI Art Generation

Description: Step-by-step instructions for installing and using Stable Diffusion to generate AI art locally.
SEO Potential: Growing interest in AI art; users seek reliable tutorials for local installations.

Fine-Tuning Llama 2 with Personal Data on Ubuntu 22.04

Description: A tutorial on customizing Llama 2 models using your own datasets for specialized applications.
SEO Potential: Niche topic with increasing demand among developers and researchers.

Top 5 Open-Source ChatGPT Alternatives You Can Run Locally on Ubuntu

Description: A curated list of the best open-source AI models compatible with Ubuntu for local deployment.
SEO Potential: Listicles attract clicks; keywords like "open-source," "ChatGPT alternatives," and "Ubuntu" are highly searched.

Optimizing Local AI Models on Ubuntu 22.04 for Faster Performance

Description: Tips and techniques to enhance the speed and efficiency of AI models running on Ubuntu.
SEO Potential: Addresses common pain points; practical solutions are highly sought after.

Building a Private AI Assistant Using Open-Source Tools on Ubuntu

Description: Guide to creating a personalized AI assistant without relying on cloud services, ensuring data privacy.
SEO Potential: Increasing concerns over data privacy make this a timely and relevant topic.

Deploying OpenAI's Whisper Locally on Ubuntu 22.04 for Speech Recognition

Description: Instructions on setting up Whisper for transcribing speech to text locally.
SEO Potential: High interest in speech recognition technologies; limited competition for local deployment guides.

Comparing Open WebUI and Ollama: Which Is Best for Your Local AI Needs?

Description: An in-depth comparison to help users decide between Open WebUI and Ollama for their AI projects.
SEO Potential: Comparison articles perform well; users appreciate detailed analyses.

Running AI Models on Low-End Hardware: A Guide for Ubuntu Users

Description: Strategies for deploying efficient AI models on less powerful machines without compromising performance.
SEO Potential: Appeals to a broader audience; addresses hardware limitations that many users face.

Securing Your AI Projects: Best Practices for Local Deployment on Ubuntu

Description: Tips on safeguarding your AI models and data when running them locally.
SEO Potential: Security is a perennial concern; practical advice is highly valued.

Why These Topics Will Rank Well:

Relevance to Current Trends: AI and machine learning are rapidly evolving fields. Topics like Llama 2, Stable Diffusion, and OpenAI's Whisper are highly relevant and have been trending in searches.
User Intent Alignment: These articles address common user needs, such as setting up AI models locally, optimizing performance, and ensuring data privacy.
Low Competition Keywords: By focusing on specific tools and Ubuntu 22.04, you target long-tail keywords with less competition but high intent.
Educational Value: In-depth tutorials and comparisons provide value, increasing the likelihood of shares and backlinks, which boost SEO.

Tips to Maximize SEO Impact:

Keyword Optimization: Include relevant keywords in your titles, headers, and content, but avoid keyword stuffing.
High-Quality Content: Ensure your articles are well-researched, accurate, and provide step-by-step guidance.
Engaging Media: Use images, code snippets, and videos to enhance user engagement.
Internal Linking: Link to your existing articles to improve site navigation and SEO.
Mobile Optimization: Ensure your blog is mobile-friendly, as search engines prioritize mobile-optimized sites.


Fix inverted logic for checking if DISPLAY set

I think -n and -z were mixed up here. The previous code overwrote the DISPLAY variable if it's already set. This update only sets a default value if DISPLAY is NOT already set.
From man test:
       -n STRING
              the length of STRING is nonzero
       -z STRING
              the length of STRING is zero

Hi, thanks for finding this. Do you use a different DISPLAY than the ":1.0" on your system? I may have used the wrong test since the default option was already in use by my OS.
I added some echos and an exit to check this behavior, closely.
Using the proposed -z flag it works this way.

The -z works when the DISPLAY is already set, and it does not work when the DISPLAY is empty. I saw the DISPLAY empty mostly on WLS on Windows.
Using -n it works this way:

But, using the -n, the already set DISPLAY is overridden as you showed me. However, it works nicely when DISPLAY is not set.
So, I am seeing both solutions do not work completely. We have to use something else.
After some tests, I could find a way to catch both scenarios.

Do you want to update your PR?
Test with this test if it works for you, and if it works, I will merge it.
if [ -z "" -a -z ${DISPLAY+x} ]; then

If you don't mind I will fix it myself.

On all of my systems, DISPLAY=:0.
When I replaced echo -e "Current DISPLAY = ${DISPLAY:=[EMPTY]}" with echo -e "Current DISPLAY = ${DISPLAY}", everything worked perfectly for me with -z.
My test script:
#!/bin/bash

echo -e "\n==========================================="
echo -e "Current DISPLAY = ${DISPLAY}"
if [[ -z ${DISPLAY} ]]; then
	export DISPLAY=:1.0
	echo -e "Setting DISPLAY = ${DISPLAY}"
fi
echo -e "===========================================\n"

My results:
~ via  v3.12.0a5
❯ /tmp/test_display.sh

===========================================
Current DISPLAY = :0
===========================================

~ via  v3.12.0a5
❯ DISPLAY= /tmp/test_display.sh

===========================================
Current DISPLAY =
Setting DISPLAY = :1.0
===========================================

~ via  v3.12.0a5
❯ unset DISPLAY

~ via  v3.12.0a5
❯ /tmp/test_display.sh

===========================================
Current DISPLAY =
Setting DISPLAY = :1.0
===========================================

Is [EMPTY] really a thing in bash? After the line echo -e "Current DISPLAY = ${DISPLAY:=[EMPTY]}", I tried doing echo ${#DISPLAY} which printed 7. I think this line is setting DISPLAY to "[EMPTY]". Since this is not an empty string, the code isn't going into the -z branch.


Technical Documentation Project - Test Error

I've been streaming going through the freeCodeCamp curriculum, and came across a bug today.
https://learn.freecodecamp.org/responsive-web-design/responsive-web-design-projects/build-a-technical-documentation-page
During this project, the user story for User Story #4: Each section element with the class of main-section should also have an id that corresponds with the text of each header contained within it. Any spaces should be replaced with underscores (e.g. The section that contains the header "Javascript and Java" should have a corresponding id="Javascript_and_Java").
When adding in content for my sections, sometimes there was punctuation in the title. When the tests run, they fail if I've taken the punctuation out of the slugs I've created.

It would probably be helpful to add some content to the user story letting people know not to use punctuation in class and id names, and to have the test parse out the punctuation when users are creating the url slugs for their links.
Tell us about your browser and operating system

Browser Name: Chrome
Browser Version: Version 68.0.3440.106 (Official Build) (64-bit)
Operating System: Windows 10 Home

Similar, I had to rename my sections.. I had a & tag in one of my titles.

Same thing here. Had to remove ** ? ** on my titles.
https://codepen.io/akmalist/full/vzzdxp/

@tbushman Are you aware of such issues?

@RandellDawson @rachelnicole You should not need to omit special characters from your ids.
Having tested:

Introduction!@#$%^&*()?&

with
     Introduction!@#$%^&*()?&

@tbushman So maybe the challenge instructions and/or the project tests should give show this disclaimer?

@RandellDawson OK

Thank you for reporting this issue.
This is a standard message notifying you that the issue has become stale, so we are going to close it.
If you think we're wrong in closing this issue, please request for it to be reopened. Thank you and happy coding.


search github for repos and clone

i.e.
$ gogh get gogh

It clones github.com/kyoh86/gogh.
introduce: https://github.com/motemen/ghq/issues/91

I should that gogh could solve it.

If we call gogh get foo, gogh get a repository from $GOGH_GITHUB_USER/gogh.
Don't search repository from GitHub.

hmm...
Is it good feature?
I think that I want to search repos with interactive fuzzy searching.
gogh repos shows list of remote repositories.
So we should use it as source, and search a repository by fuzzy-search (i.e. fzf) and clone it.
e.g.
function search-repos-and-get() {
    target=$(gogh repos | fzf)
    if [[ -n "$target" ]]; then
        gogh bulk-get
    fi
}

I don't want to implement interactive fuzzy-search interface...


Greenscreen background image

Please check if the PR fulfills these requirements

[x] You've followed the contributing guidelines
[x] You've adheared the code of conduct

What kind of change does this PR introduce? (Bug fix, feature, docs update, ...)

[ ] Bug Fix
[x] New Feature
[ ] Documentation Update
[ ] Other

If bug fixes or new features

What is the current behavior? (You can also link to an open issue here)

You can only publish a raw video source, closes #5.

What is the new behavior (if this is a feature change)?

You can now publish a video source with colors chroma keyed and replaced with a background image.
The user can upload a image of their choice, and once uploaded, customize the X/Y position, width, and height of the image.
To assist with positioning, a Overlay option is provided to draw the image on top of the video.
Additionally, the user may select a low and high colors as their chroma key color if they wish to use a non-green color.

Although it's not the most user friendly due to using the native color picker, which uses various color representations - HSV, RGB, etc - while the actual color representation used for calculations is HSL.

Does this PR introduce a breaking change? (What changes might users need to make in their application due to this PR?)

No

Other information:

In order to have better performance, the image manipulation is done in a Worker.
I attempted to make it reusable by exposing a getUserStream method, so the user can be given another chance to provide a video source by calling this method again.

I couldn't come up with a simple way to directly return the native stream when no background image is in use, but have it replaceable with the canvas one when the background image is in use.
Therefore even when there is no background image, the video is still being passed through a canvas.

I avoided actual styling of the form aside from making it collapsible with the Open/Close button.

While requestAnimationFrame() is used, it's limited to the reported frame rate of the user input video.

I had intended to pass the ImageData by reference to the Worker, but I ran out of time - doing this I imagine would further increase performance, with only modifications to sending/receiving portions of the code being required.

My attempt to use the implementation described here did not have positive results in what testing I did - the inability to use a Worker resulted in decreased performance, and this was before I was even able to implement all the existing customization features - you can see it here if you're interested.

Thank you so very much for this @RascalTwo! This is a great contribution, and it's good to see you free up the main thread with a worker. However, when testing, I wasn't able to get the output from the canvas into the publisher - instead it looked like this:

Any idea why this might be the case?

I'm using Chromium 86 in Brave 1.15 😄

After testing, I'm quite confident it was due to the fact that I had disabled alpha in the canvas contexts, so hopefully this has resolved it for you also.

Only in hindsight I realized that since the fake device is all green, the greenscreen effect isn't actually visible above - so the below gif actually displays the effect working.

Additionally, I've brought it up to date with the latest main commit - 685da9b -  and added some basic styles that should match with the rest of the page.

Unrelated, but turns out my errors in chromium browsers with opentok were due to the low quality of my USB webcam, hence I used a fake device/video to demonstrate it in Brave.

Hey! I tried this but still couldn't get it to work. This time, I can't see anything except the overlay image, and my chroma min and max are sensible. This is still an excellent piece of work, and I'll mark it as valid for the sake of Hacktoberfest.

If you want to jump on a call to try and debug this let me know. I'll close this for now unless you want to do any further work 💖

I'm glad the initial error was truly solved, wasn't able to be there live but I did watch your VOD.

The issue you had with pulling my PR was likely due to the fact that I rebased my changes to match the updated styles at the time, so this time I didn't do that so you shouldn't encounter the same issue again.

As for the chroma-keying not working, it appears that's due to the rigidity/inaccuracy of the algorithm I choose, so I've swapped it out for a different one that I can get working on just about any greenscreen video - even yours, even at the low resolution I was able to scrape from the VOD.
After seeing how much performance really drops in the VOD, I also decided to make performance a priority, so along with various micro optimizations, it's now much faster - probably as fast as I personally can make it without getting WebGL involved.
Just to be safe though, in case performance is still lackluster, I've added options to allow for customization of the output - turning down the output resolution allows for higher frame rates.

Now I'm no Vue expert, so while I've got it all working, I's likely I've broken some Vue rules or missed some things that Vue offers to make certain things much easier, but hopefully not too many though.

I personally follow these steps to get each background working:

Upload image
Enable Preview Overlay
Adjust background Width/Height as needed
Disable Preview Overlay
Pick the Darkest and Lightest chromas

Pausing if needed

If needed, adjust Tolerance

Decrease if the person is being replaced
Increase if there is a significant chroma border around the person, or not all chroma is being detected

I was only able capture your webcam from the VOD at 320x180, hence the low quality, but as you see with the second example - with a resolution of 1920x1080 - it does work when the quality is higher.

Lost some quality to get the .gif under 10 MB for the upload limit, here's the higher quality video if you're interested.

UI Clarification
I've also attempted to make the UI clearer, so hopefully the options are more obvious, but just in case here's a quick rundown:
Background
The Pause/Play button pauses or plays the input video - useful to fine tune the settings without having to deal with new frames or that one pixel you're trying to pick jumping around.

Preview Overlay

Places the background image on top no matter the chroma
I thought it'd be useful to see the position without guessing,

Especially since the default chroma values will probably make the image invisible to most

X/Y

Image Offsets from the top-left corner
Higher X is right
Higher Y is down

Width/Height

The dimensions of the image
New images are automatically shrunk (aspect ratio preserved) when larger then the input video

Chrome Dark/Chroma Light

The darkest/lightest color to replace with background image
Pick button

Makes it so whatever pixel you click on in the canvas will be used for this chroma

Tolerance

How close outside of the chrome colors the color can be get replaced
Higher means pixels further outside the dark->light range are replaced
Lower means only pixels within the dark->light range are replaced
I recommend only adjusting this after picking chroma values, although the default 5% seems to work for most greenscreens

Output

Width/Height

The dimensions of the output canvas/video
If there is high frame loss, decreasing these values should decrease the frame loss

FPS

The target Frames Per Second to render at

With all the inputs being instantly reactive thanks to Vue, changes are now visible instantly instead of having to wait for focus to leave the input (the change event), so it's now easier and quicker to fine tune all these settings.

I appreciate the offer to jump into a call, and the time you've taken for this single issue, perhaps if these latest changes don't work as expected I'll take you up on that offer.


[SPARK-27771][SQL][DOCS] Add documentation for grouping functions (cube, rollup, grouping and grouping_id)

What changes were proposed in this pull request?
Both look added as of 2.0 (see SPARK-12541 and SPARK-12706). I referred existing docs and examples in other API docs.
How was this patch tested?
Manually built the documentation and, by running examples, by running DESCRIBE FUNCTION EXTENDED.

Test build #105531 has started for PR 24642 at commit 9126895.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10787/
Test PASSed.

Test build #105531 has finished for PR 24642 at commit 9126895.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105531/
Test PASSed.

Thanks!


Refactor ClientUI using CardLayout

Fixes #3199 (completely, and other bugs of the sort)
The current way ClientUI switches tabs on the side panel is by adding
and removing them from a container panel. This is slow on panels with a
large number of components.
To fix this, I decided to use a cardlayout, as suggested by abex.
So now, the container panel has a CardLayout and all the plugin panels
are added with their names as keys, and when a NavigationButton is
clicked, the cardlayout will show a panel based on its key. (Ex: Xp
Tracker).
Before this commit, switching from the construction skill calcs to any other tab would result in a +/- 1 second delay, after this, even after every item is repeated 20 times, smooth transition with no delay, that's a giant improvement.
I would request a look from @Abextm and @deathbeam.

Updated to apply @deathbeam 's review requests.


Codeclimate Feature

This should allow CodeClimate to work.
The reason it wasn't working before was because it took a while to update itself, only finishing today...
Basically, I added the config file and coverage into the Travis config file.

Current coverage is 0.00% (diff: 100%)

Merging #23 into develop will not change coverage

@@           develop   #23   diff @@
====================================
  Files            2     2
  Lines           80    80
  Methods          0     0
  Messages         0     0
  Branches         5     5
====================================
  Hits             0     0
  Misses          80    80
  Partials         0     0

Powered by Codecov. Last update 009ad40...e80e3a0


README adjustments

:+1:

@motdotla change the README title as well as the package name in the require (it changed to html-preflight).


MunkiReport Client Runner for version 5.7.0 no longer works on Macs with 12.2.1 and below

Client runner is apparently using the third-party Python install and failed to detect or use the built-in MacOS Python install.

As highlighted in the release notes, MunkiReport now requires the installation of MunkiReport Python2 for all clients for 5.7.0 on all versions of macOS: https://github.com/munkireport/munkireport-php/releases/tag/v5.7.0
The wiki has been updated, but please let us know if any additional clarifying details could be added: https://github.com/munkireport/munkireport-php/wiki/Client-setup#python-2
I also wrote a blog post about the change that I hope is helpful: https://www.kevinmcox.com/2022/02/munkireport-and-python-2/

Thank you @kevinmcox.
It didn't say there though... Only mentioned that Apple removed Python from MacOS 12.3. :) Would have been easier if the autodetect script was present so one wouldn't have to install Python to hundred of Macs on existing setup after installing the update.
I can still use the built-in Python though after upgrading to 5.7 by creating a symbolic link to the python executable.
Thanks! :)


Security issue

While I was experimenting with this program, I tried using it with a game I didn't actually own on steam. When I gave my friend access, we discovered that whenever I tabbed out of the game that he could see my whole desktop. I gave him access, and he was able to access everything on my computer while there was nothing I could do on my end but fight for mouse control (lol). Idk if there's anything you guys could do on your end, but a hotkey to immediately revoke control privilege or something of that extent seems necessary.

Simple bat file with a taskkill command and a keyboard shortcut to that bat file to launch it in an emergency is simple enough solution, or just unplug your internet. Mind you, official Steam remote play works the same way. i can browse my pc with it if you know how to force minimize the game on host (actually sometimes handy).
Other than that its not really a security issue, you are willingly sharing your PC controls with another person, if you don't trust the person on another end, don't give him/her keyboard and mouse controls.

Thanks for the reply, I was unaware you could do the same thing with base remote play.


Can't find 'Enable Sentiment Analysis.'

After publishing my app, I can't find 'Enable Sentiment Analysis.' option in the 'Publish' page. Are there any settings I need to do?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 38408714-a973-877d-0773-8ddf5c8eea1f
Version Independent ID: a594e769-2de8-418a-f261-0cdf486d5208
Content: Tutorial to create a LUIS app that returns sentiment analysis - Azure
Content Source: articles/cognitive-services/LUIS/luis-quickstart-intent-and-sentiment-analysis.md
Service: cognitive-services
GitHub Login: @diberry
Microsoft Alias: diberry

@sodafatty Thank you for taking the time to ask this question. Can you take a screen shot, save it to OneDrive or another storage provider as public and add a link to it here? #in-progress

@sodafatty Are you able to take a screenshot and share it?

@sodafatty We will now proceed to close this thread. If there are further questions regarding this matter, please reopen it and we will gladly continue the discussion.

I can't find the sentiment analysis in this Luis App which language is zh_CN

@sodafatty Thank you for letting me know which culture you are using. Sentiment analysis is currently not supported in that culture. I will make sure the documentation for this feature links back to the supported languages. Sentiment analysis is provided by text analytics, which currently doesn't support zh-cn. I will let you know when the docs are updated for this issue.

@YutongTie-MSFT Please reopen this issue. If I can do it, I don't know how.

@sodafatty Thanks for the screenshot.
@diberry Hi, I have reopened it. The reason why I closed it is the customer has no response within 3 days. Thanks!

@sodafatty The supported languages table is modified to be more clear that sentiment is not included in Chinese. The other pages are in a longer rewrite cycle. It may be up to 2 weeks before those docs are available.

@sodafatty
We will now proceed to close this thread. If there are further questions regarding this matter, please reopen it and we will gladly continue the discussion.


Merge of management and training in a single main menu Faq entry

As discussed in Brest, faq management and faq training in a single main menu entry with tabs.

Thank you !


DynamoDB for Terraform OSS backend state lock is not encrypted with KMS CMK

Describe the outcome you'd like
The DynamoDB table (and its replica) used as a state lock table for Terraform OSS back ends should be encrypted using a customer-managed KMS key, presumably the same KMS CMK as is used for the Terraform state in S3. The DynamoDB tables storing other AFT data are already encrypted in this way.
Is your feature request related to a problem you are currently experiencing? If so, please describe.
The default server-side encryption used flags up in external security testing of our AWS Landing Zone. Many third-party sets of security guidelines and best practices include this requirement, and for good reason.
Additional context
This can be remediated, I think, by the following change around line 271 of modules/aft-backend/main.tf:
 replica {
    region_name = var.secondary_region
    kms_key_arn = aws_kms_key.encrypt-secondary-region.arn
  }

  server_side_encryption {
    enabled     = true
    kms_key_arn = aws_kms_key.encrypt-primary-region.arn
  }

IAM permissions are already granted for the key to the role using the Terraform OSS back end.

Hey @david-musgrove!
I went ahead and created an item in our backlog for this feature request.


[#108440520] Fix the dashboard so that it fills the whole screen.

This was set to 1000px high, so was leaving a blank space at the bottom
on a 1080p display. Setting this to 1064px allows it to fill the whole
screen (allowing for the 8px padding top and bottom).
The 8px margin was being added by the Safari user-agent stylesheet, so I
added it here to make it explicit, and also make it more obvious why the
height is set to 1064px.

LGTM - Merging


Using NAS to store the LN/Blockchain data

Hi
Would it be possible to store the HD data to an external NAS? It could be NFS mounted to the Raspi? What would be the drawbacks with this setup? What would happen in case of network failure etc?  Could the blockchain or LN get easily corrupted?

Hi
Would it be possible to store the HD data to an external NAS? It could be NFS mounted to the Raspi? What would be the drawbacks with this setup? What would happen in case of network failure etc? Could the blockchain or LN get easily corrupted?

I wondered about this too.  I don't see why not but I've not looked into it much.  I read that the ethernet port is actually faster than the USB 2 ports so could it even improve performance? dunno.
Try it! or search the Pi and Raspbian forums.

I have once lost my ln bitcoins due to power outage and I don't have UPS on my NAS. What happens if NAS shuts down unexpected? On the other hand NAS has RAID5 compared to USB HDD. I could try it out anyway, but after LN gets backup support, then NAS seems more robust.

I think its worth a try but its a very manual special setup that would classify as a standard setup. To protect LND data better I think the research on this issue is more easy to setup and hopefully part of future releases: https://github.com/rootzoll/raspiblitz/issues/329

Why not spin this thing up... can you make a VM version of it like for ESXI oder Hyper-V? VMWare in the small version is free and you can do free backups with synology or qnap nas or even veeam. So is makes room for bigger private nodes with more security.

@DaReaLDeviL so the idea would be running the RaspiBlitz completly on a NAS system like synology or qnap? Interesting idea.

Qnap and Synology are the biggest platforms (as far as I know) for home and professional nas  with 3rd party app installer and automated update options. This should be huge on the user base (tech driven people) and ease of install with better security brought by the nas itself. From the concept it should be a no brainer to jump on it. From the tech side I can deliver my time and effort to test. I can’t code, but read some of it and whould help as far as I can.

@DaReaLDeviL at least some info on how to deliver packaged software to QNap/Synology would be interesting. As far as I remember they have quite different processor architecture, so I dont think porting RaspiBlitz is not an easy task. Also can everbody release packages that can be installed from outside the company stores?

Regarding  building a Bitcoin and Lightning node on a NAS (FreeNAS) this is an awesome guide: https://github.com/seth586/guides/blob/master/FreeNAS/README.md
Another option is - since I love to use the Odroid XU4 with RaspiBlitz-on-DietPi already - to test this: https://ameridroid.com/collections/cases/products/cloudshell2-nas-kit

I Would like something like this as well. I would like to be able to mount a folder on my NAS to the Pi, so the Pi stores all the data on my NAS. Did it before with an other installation. So it is possible. I just don’t know how to set this up with RaspiBlitz.

I have been too busy to try this out, but it should work by just copying /mnt/hdd to NAS share and making smb mount to NAS on boot?
For example put this to /etc/fstab:
//192.168.100.x/share/raspiblitz/ /mnt/hdd cifs user,username="user",password="xxxx",iocharset=utf8,file_mode=0777,dir_mode=0777 0 0
I don't know if this mount will happen early enough on boot so that everything is ready when bitoind and lnd starts. Another problem might be upgrade, where you need to make the fstab linking again. Is there a single place/way to tell the raspiblitz where it seeks the external hdd? This might be a useful config feature?

Is there a single place/way to tell the raspiblitz where it seeks the external hdd? This might be a useful config feature?

Not at the moment. Maybe check if the is a way for Raspbian to set something on the sd card before boot - like the way you can set WIFI before first boot.

Try adding:
rootdelay=5
in /boot/cmdline.txt
and
sudo mount -a
in /etc/rc.local
fstab is for OS dependant drives. Prob not the place to put it.

I just tried to do a setup with a folder on my NAS mounted on the pi as /mnt/hdd, but I get the message that there is not HDD attached during setup. How I can accomplish the storage of my data on my NAS?

I'm afk right now. But I would suggest you grep for sda in /home/admin.

I'm afk right now. But I would suggest you grep for sda in /home/admin.

Sorry, don't really understand what you mean here. :)

I am wondering if it would be possible to symlink the bitcoin and lnd on the external HDD to a folder on my NAS. It's not ideal, but then atleast the blockchain, etc would be stored on the NAS and not on the HDD.

I understand what you are trying to do. Currently this is not supported. In order to accomplish this you would need to remove some of the existing checks.
So if you run the following command you will get a list of locations where scripts currently assume that a physical external hard drive is connected to the Pi. You would need to "bypass" these.
grep sda /home/admin/*

I understand what you are trying to do. Currently this is not supported. In order to accomplish this you would need to remove some of the existing checks.
So if you run the following command you will get a list of locations where scripts currently assume that a physical external hard drive is connected to the Pi. You would need to "bypass" these.
grep sda /home/admin/*

Ah ok. Now I get it. :-) Thanks! I will look into it.

@openoms @rootzoll I was asked about this feature again today... having a RaspiBlitz with only the SD Card and no USB storage. And the Blockchain (and everything else) would then be stored on a remote device (e.g. via NFS or SMB). What do think of this?

It sounds like a very special edge case setup. Did anybody managed to setup something like this (just a bitcoind that stores blockchain data over NFS/SMB) manually and can share some performance data?


Login button gets stuck loading

@jmoreira-valory I tested this locally – my login button was hanging.
In my services directory, I had two services folders:

bafybei...1
bafybei...2

I prepended deleted_ to bafybei...1. I then retried logging in, and it worked. After starting the agent, I then saw two directories deleted_bafybei...1 and bafybei...1.


Problem with Persian and Arabic languages

Hi
I'm using laravel 5.4 and eloquent-sluggable 4.2.5 in one of my apps which is in 2 languages: English and Persian.
This works perfect with English but for Persian it generates strange slugs.
For example if i have "تست" then result is "tst" but i expect to get "تست" or
if i have this "این یک تست است" (it means 'this is a test') the result is "n-tst-st" but i expect to have this
"این-یک-تست-است"

By default, the slugging engine we use does a "transliteration" of the slug into ASCII characters.  The assumption was that people would use this library to generate URLs (which can use UTF characters, but tend to be ASCII-only).
If you want to keep your slugs in Persian, you could probably do something like @DJafari suggested, or just use the following in your config:
'method' => function ($string, $separator) {
    $slug = mb_strtolower(
        preg_replace('/([?]|\p{P}|\s)+/u', $separator, $string)
    );
    return trim($slug, $separator);
}

@cviebrock This works but i cannot cache my config file. Any solution?

You can't cache configs that use closures, correct.
Putting that logic into a helper function or class/method, and then referencing that in the config should work.  See the second example here.

@cviebrock Thank you so much, i searched a lot for this, don't know how i didn't notice it.