Support of groups in CF

As discussed in CF meeting on 20 June 2018 in Reading, UK, we'd like to add support for the use of groups in CF files. Charlie and I have drafted an appropriate pull request containing the suggestions we'd like to implement.
Basically, the idea is to allow elements in CF files to refer to others which are not in the same group. The proposal defines ways of locating variables in this situation and tries to capture current ways of doing so "in the wild".
I'll update this issue with further material on this (Pull Request reference, summary presentation discussed at the meeting last June, and the proposal as drafted up till that point) as soon as the PR has an URL.
We haven't updated the conformance rules and checker yet, as we want to agree on the content first.

Further resources:

The original proposal which was used to hash out ideas around these proposed changes
The presentation discussed at the meeting in Reading

This issue is implemented in https://github.com/cf-convention/cf-conventions/pull/145

Dear Daniel and Charlie
Thanks for this proposal. I think it's a sensible way to support groups, and I agree with it, except that I'm not convinced about the lateral search. You don't sound convinced about it either, in saying that it's allowed for backward-compatibility and may be deprecated in future. Why not deprecate it now (meaning that the CF-checker would give a warning)? Why allow it at all? Is there a more specific extra search rule you could provide instead of your general lateral algorithm to deal with the existing datasets you have in mind?
I think that at the meeting we talked about how you would flatten a file, if you didn't want groups. Could we include that aspect in the proposal? If it's reversible, that would be neat.
I feel that the text could be made simpler and easier to understand, and most of my suggestions below are along those lines. Although I think I get the spirit of your search rules, I have to say I don't properly follow them in detail. The text seems obscure to me - maybe you could say it more plainly, and diagrams would help, I expect.

In ch01 you propose a number of definitions. However, you don't propose to use these terms anywhere in the convention outside your new subsection of ch02. Hence I suggest you start your ch02 subsection with these definitions, instead of putting them in ch01, so they're in the place where the reader of ch02 will see them.

You haven't defined what you mean by ancestor group, sibling group, descendant group, identifier and element. Why not say "nearest" instead of "most proximal"? What does "nearest" mean in the description of "local apex group"?

For the first sentence of the new Groups subsection, I would suggest, "Groups provide a mechanism to structure data hierarchically within files." It doesn't seem especially "powerful" to me - it's just what you'd expect it to be! It's also not the only way, since hierarchy can equally well be done with directories.

Are "object" and "element" the same? If so, I'd use only one of these terms, and "element" seems better to me because it's not language-specific. Say what sort of thing can be an "element" - a dimension? a variable? of any role?

Say what an "identifier" is and in which contexts in CF it can occur. What is a "location"?

"Resolves to" is programmatic jargon - could you say this in more ordinary language?

Is the convention of paths to name groups and elements something that you are  defining here as part of CF, rather than built into netCDF? I guess that it is, and if so, I would state explicitly that you are doing so, because it reads as though you assume the reader knows what is meant by a path to a group. As part of this, please explain how to interpret a path.

In a path, is . allowed? Is // allowed?

Does the restriction on out-of-group refs ("The dataset producer etc.") apply only to the case of absolute paths, or in all cases? If the latter, it should be stated before or after the three cases, I suppose.

In "Search by absolute path", what does "nearest dimension" mean?

In "Search by absolute path" you use the phrase "out-of-group", before its definition, which appears further down.

In "Search by proximity", say what "with no path" means. What's a "direct ancestor" - I mean, could an ancestor be "indirect"? I would combine the Note with the paragraph which follows, because in the Note it's not obvious what strategy you're talking about i.e. the "special case".

I think that you shouldn't list the attributes concerned, because this is redundant within the convention. This information is in Appendix A. If it's repeated here, it'll probably become inconsistent. If you want, you could give just a single example from each list e.g. title, units.

Is the last paragraph saying that groups can have attributes, like global attributes? If so, please say that - "being present within" sounds vague. Does a group attribute override a global attribute? Are global attributes assumed (unless overridden) to apply to all variables in all groups? I suggest that this part should come in the same place as the text about global attributes, before the text about variable attributes. I think that (what is currently) the last sentence isn't necessary - that's implied by the previous sentence, isn't it?

I hope that helps. Best wishes
Jonathan

Dear Jonathan,
Thanks for reviewing this.
Inline responses
I'm responding to your comments inline and abridging as I determine relevant; if there's anything you're missing, let me know.

... I'm not convinced about the lateral search. You don't sound convinced about it either, in saying that it's allowed for backward-compatibility and may be deprecated in future. Why not deprecate it now (meaning that the CF-checker would give a warning)? Why allow it at all? Is there a more specific extra search rule you could provide instead of your general lateral algorithm to deal with the existing datasets you have in mind?

I have to agree with you on this, and this is in line with my understanding of the discussion at the meeting in Reading. @czender could you say a few words on this?

I think that at the meeting we talked about how you would flatten a file, if you didn't want groups. Could we include that aspect in the proposal?

@czender has some nice material on this in the original proposal which I omitted for brevity. Would you propose importing this section (possible in edited form) into the pull request? I struggled with not adding too much text to the Conventions themselves, but we could include it as an appendix chapter. I do worry about overloading people new to the game with text.

The text seems obscure to me - maybe you could say it more plainly, and diagrams would help, I expect.

Diagrams are always good. In the original proposal we had:

To my knowledge it would be the only diagram in the whole of the Conventions.  As a new contributor I didn't want to snub current practices, but I see no problem with including it, if the community is happy with that.

In ch01 you propose a number of definitions. However, you don't propose to use these terms anywhere in the convention outside your new subsection of ch02. Hence I suggest you start your ch02 subsection with these definitions, instead of putting them in ch01, so they're in the place where the reader of ch02 will see them.

I don't really agree with this - having multiple glossaries scattered throughout the document only makes the terms easy to find if everybody is reading them in electronic form, and in that case we could just as well define terms next to their first use. So I prefer to keep all definitions together, as has been done for all the other definitions thus far.

You haven't defined what you mean by ancestor group, sibling group, descendant group, identifier and element. Why not say "nearest" instead of "most proximal"? What does "nearest" mean in the description of "local apex group"?

I think it would be a good idea to add these to the glossary. @czender, do you have an opinion on "nearest" vs. "most proximal"?

Are "object" and "element" the same? If so, I'd use only one of these terms, and "element" seems better to me because it's not language-specific. Say what sort of thing can be an "element" - a dimension? a variable? of any role?

@czender I see no difference here and propose adopting "element". Please correct me if I'm overlooking something.

Is the convention of paths to name groups and elements something that you are defining here as part of CF, rather than built into netCDF? I guess that it is, and if so, I would state explicitly that you are doing so, because it reads as though you assume the reader knows what is meant by a path to a group. As part of this, please explain how to interpret a path.

I understand what you're saying here, but since we say that paths are treated as UNIX paths, I think it would be overredundant to describe how to follow a path according to the UNIX conventions. Do you think it would be valuable to link to the POSIX pathname resolution specification?  The same comment applies to your following bullet point, which I have omitted.

Does the restriction on out-of-group refs ("The dataset producer etc.") apply only to the case of absolute paths, or in all cases? If the latter, it should be stated before or after the three cases, I suppose.

You're right, this should be noted at the beginning of the chapter, since it applies to all out of group references. For our reference I note that it's in ch02.adoc, line 190.

In "Search by proximity", say what "with no path" means. What's a "direct ancestor" - I mean, could an ancestor be "indirect"? I would combine the Note with the paragraph which follows, because in the Note it's not obvious what strategy you're talking about i.e. the "special case".

Agreed, I think we can describe this more clearly in the next update.

I think that you shouldn't list the attributes concerned, because this is redundant within the convention. This information is in Appendix A. If it's repeated here, it'll probably become inconsistent. If you want, you could give just a single example from each list e.g. title, units.

I agree with that too.

Is the last paragraph saying that groups can have attributes, like global attributes? If so, please say that - "being present within" sounds vague.  Does a group attribute override a global attribute? Are global attributes assumed (unless overridden) to apply to all variables in all groups? I suggest that this part should come in the same place as the text about global attributes, before the text about variable attributes. I think that (what is currently) the last sentence isn't necessary - that's implied by the previous sentence, isn't it?

I agree that precedence should be clearly stated when group attributes conflict with global attributes.
Next steps
We'll update the PR to address the points as noted above and ping again when that's been done. In addition to the items noted explicitly, I propose placing the following new terms in the glossary:

Element (or object)
Identifier
Location
Resolves to - or change language
Nearest dimension
Ancestor, sibling, descendant group

In response to Jonathan's question on paths (or full names)

Is the convention of paths to name groups and elements something that you are defining here as part of CF, rather than built into netCDF?

I just wanted to mention that full names are supported (to varying degrees) by the netCDF C and Java libraries and the ncdump and nccopy utilities. Documentation, as far as I've found, is a bit sparse and scattered.
In the C library, you can find a group given its full name, and with a group (ncid) you can get its full name. But that doesn't work for variables or dimensions. The Java library provides similar functionality for groups, variables, and dimensions. The ncdump and nccopy utilities support full paths in the -v vars option. It also mentions relative paths but its really more partial path matching, e.g., g/v would match /g0/g/v and /g1/g/v and /g/v.

Given that, would that call for search by proximity as the only approach in section 5, Scope? Having all potential approaches complicates interpretation of the data and building services on top of them. Stating absolute or relative relations explicitly would be beneficial although I see the point of all being a C user.

Dear Daniel
Thanks for your responses and comments. Answers to a few questions:

Yes, I think it would be fine to include Charlie's material on flattening and dismembering, except for the final paragraph about implementations, which isn't essential to the convention.

Including a proper diagram would be unprecedented, I agree, but I don't know a reason not to. Actually there are some tables which are like diagrams in chapter 9 - they used to be in colour, and were prettier like that.

If you're sure you want to put your definitions in ch01, then I'd suggest keeping them together all there, rather than interleaving them with other definitions, and give them a heading in ch01 indicating that they are relevant to groups. You could provide a reference to the later chapter on groups, and a backwards reference from there to the definitions. One reason for my original comment is that the considerable number of these group-related definitions, which are used only in one place in the document, seems unbalanced to me with respect to the existing definitions in ch01, which are used in multiple places.

Yes, I think it's OK to say that paths are like Unix paths. I must have missed that. Not every CF-netCDF user is a user of Unix, though, so it may still be useful to spell it out a bit.

It'll be interesting to hear Charlie's view on the lateral search.

I look forward to the next version. Thanks for your work.
Cheers
Jonathan

Dear Charlie
All the GitHub postings are distributed to everyone on the CF email list, so
that it shouldn't be necessary to make individual arrangements for particular
issues.
Best wishes
Jonathan

It appears that these github hub messages are going to the cf-convention mail list rather than the cf-metadata mail list, so perhaps @czender is not on the cf-convetion mailing list?

Yes, I only learned there was a cf-convention mail list (distinct from cf-metadata) yesterday :)

FYI I'm finally making progress on this and expect to have a full response and update PR next week. Thank you for your patience.

All, the changes described below were just submitted as a PR to Daniel's tree, which hopefully means they will modify #145 once accepted. I'm non-expert at Github, so perhaps that is inefficient. The original is viewable in my tree at https://github.com/czender/cf-conventions/tree/groups

I'm responding to your comments inline and abridging as I determine relevant; if there's anything you're missing, let me know.

... I'm not convinced about the lateral search. You don't sound convinced about it either, in saying that it's allowed for backward-compatibility and may be deprecated in future. Why not deprecate it now (meaning that the CF-checker would give a warning)? Why allow it at all? Is there a more specific extra search rule you could provide instead of your general lateral algorithm to deal with the existing datasets you have in mind?

I have to agree with you on this, and this is in line with my understanding of the discussion at the meeting in Reading. @czender could you say a few words on this?

I must use many words. I wish I had been at the Reading conference to say this in person:
I may be the only defender of the lateral search feature so it falls on me to make the case for it. The one word that best summarizes why I think lateral search would be good for CF is "User-base". Geoscience researchers use an enormous amount of satellite-retrieved data stored with groups by providers notably including NASA and ESA. These agencies use HDF5EOS or netCDF4 to store data from dozens of platforms/instruments (Aura, MLS, OMI, S5P, TES, etc) and quite often data fields are in sibling (not ancestor) groups to the coordinates. For example, S5P has geophysical variables in /BAND1_RADIANCE/STANDARD_MODE/OBSERVATIONS and their coordinates (latitude and longitude) in the sibling group /BAND1_RADIANCE/STANDARD_MODE/GEODATA. MLS on Aura has geophysical variables in /HDFEOS/SWATHS/O3NadirSwath/Data\ Fields and latitude and longitude in /HDFEOS/SWATHS/O3NadirSwath/Geolocation\ Fields. The list of other examples exceeds my energy to type it.
This state of affairs developed over many years, and indicates that dataset producers like having coordinates in sibling groups of data fields. I doubt that NASA/ESA/etc. will start putting coordinates in ancestor groups rather than sibling groups just because CF recommends it, and even it they do, there will remain a decades-high mountain of data with the current organization. If/once CF adopts this groups proposal, one of three things is likely to happen:

If CF supports lateral search then these agencies can comply with CF and continue storing data with the same organization as before, and use lateral search to find out-of-group (OOG) coordinates, should they prefer lateral search over relative/absolute paths. Relative/absolute paths are, IMHO, fragile and susceptible to breakage in downstream processing so avoiding them is best for long-term dataset interoperability.
If CF DOES NOT support lateral search then these agencies can comply with CF and continue storing data with the same organization as before only by using relative/absolute paths to refer to OOG coordinates. This is fragile and IMHO a mistaken approach for long-term dataset interoperability.
NASA/ESA/etc will start to design future datasets so that OOG coordinates are not in sibling groups. Existing data will not be CF-compatible unless it already uses relative/absolute paths.
In my opinion it is best for users, data producers, and CF if some combination of (1) and (3) occurs, so that the sibling-oriented organization of existing datasets is "grandfathered in" to being CF-compliant by allowing lateral search, and producers start to deprecate the necessity of lateral search for future data products. Since many dataset producers have preferred lateral (rather than purely ancestral) dataset organization over the years and lateral search is the appropriate mechanism to resolve lateral associations, then CF should respect and support past dataset organization decisions and not impose relative/absolute paths as a requirement for earning CF-conformance for such datasets.

You haven't defined what you mean by ancestor group, sibling group, descendant group, identifier and element. Why not say "nearest" instead of "most proximal"? What does "nearest" mean in the description of "local apex group"?

I think it would be a good idea to add these to the glossary. @czender, do you have an opinion on "nearest" vs. "most proximal"?

To me "nearest" and "most proximal" mean the same thing. "Nearest" seems more vernacular, perhaps too vernacular? Nevertheless, I changed to "nearest" in the current PR and added to the glossary.

Are "object" and "element" the same? If so, I'd use only one of these terms, and "element" seems better to me because it's not language-specific. Say what sort of thing can be an "element" - a dimension? a variable? of any role?

See immediately below.

@czender I see no difference here and propose adopting "element". Please correct me if I'm overlooking something.

I think there is no difference in the way we use "element" and "object" in the text for Groups proposal. However, "element" is already frequently used in CF to refer to elements of an array, definitely not the meaning that Groups intends to convey. In practice "objects" as used in the Groups proposal can only be variables (including coordinate variables, of course), and dimensions. In theory a group could itself be an "object", however the proposal does not at this point need to do that. Off-hand, I can't think of an instance where an out-of-group (OOG) attribute needs to be explicitly referred to. Attributes of OOG variables can be important, yet they are always referred to via the OOG variable, and there is never a direct reference to an OOG attribute. It's the variable's scope that matters, an attribute is always locally attached to a variable or group. Are there any counter-examples to this?
If not, I think it is clearer and more precise to eliminate the use of both "element" and "object" in the Groups proposal, and replace those words with what they actually stand for, i.e., "variable" and/or "dimension". I modified the PR to do that. Similarly, I eliminated "identifier". We could instead have defined "identifier" to mean "a variable or dimension". That would be possible. Nevertheless, I think it's clearer to just say "variable or dimension" everywhere in the text. A group could also be considered an "identifier", thought this proposal does not need to do so.

Next steps
We'll update the PR to address the points as noted above and ping again when that's been done. In addition to the items noted explicitly, I propose placing the following new terms in the glossary:
Element (or object)
Identifier
Location
Resolves to
Nearest dimension
Ancestor, sibling, descendant group

The updated (by CZ) PR eliminates "element", "object", "identifier", and "Resolves to". The updated glossary now defines "Nearest dimension", "location", and Ancestor, sibling, and descendant groups.

Dear Charlie
If you're not an expert in GitHub, I'm an ignoramus, and I can't work out where to view the modifications you are proposing to the document. Perhaps you or someone else can lead me straight to the right place with a URL.
Your comments above are helpful. Thanks for making the changes. I can't remember the detail of the lateral search algorithm, which Daniel showed us at the meeting. It was hard to remember, I thought, suggesting it's not obvious. I appreciate your reasons for wanting to retain it, though. Is there a more general and easily memorable algorithm which would include the present lateral-search as a subset? For example, you could accept any variable of the right name, wherever it can be found in the entire file, if it's not found in the ancestry of the reference. That would imply that it should be an error to include more than one variable of that name in the file, and if there is more than one, there's no rule about which one to choose. The data-reader could implement any search algorithm they want to find an acceptable variable.
Best wishes
Jonathan

Jonathan,
Regarding your question about the lateral search algorithm, a succinct way of remembering it is "search upwards to the apex, from there search across going downwards". Your suggestion about using any variable with the right name if it can't be found works well for simple trees. I don't know of an existing counterexample off-hand. However, it's easy to imagine one. For example, it would perform poorly in any tree where there are multiple coordinate variables of the same name (e.g.,latitude) all stored in sibling groups as is often the NASA/ESA style. Only the in-scope latitudes (i.e., those located beneath the apex group) will have the right latitude dimension. If the desired latitude were in a sibling group, and there were other coordinate systems in the file, then the search would often return an incorrect latitude variable.
Here is another way to visualize the capabilities that would be lost by omitting lateral search from CF: Consider an exising NASA MLS, or ESA S5P or other dataset that places coordinates in sibling groups not ancestor groups. The dataset, in1.nc, would work well with your suggested simpler algorithm. However, combining that dataset with a second dataset in2.nc with
ncecat in1.nc in2.nc out.nc would perform poorly with the simpler algorithm, though it would work fine with the lateral search algorithm.

As an FYI, if you want to compare two branches or forks, you can view a pull request without actually creating it for others to see and comment on. So in this case, we can compare @czender's "groups" branch against the current cf-conventions master by going to "Pull Requests" in either repository and clicking "New Pull Request" you can select "compare across forks" and select the fork / branch(es) you want to compare. In this case you get a comparison page with a URL like this:
https://github.com/cf-convention/cf-conventions/compare/master...czender:groups

If you scroll down you can switch to "Rich Diff" with the button that looks like a page in the upper left of a given asciidoc document.
As long as you don't click "Create Pull Request" -- you are able to just look over the diffs.
Carry on.

Sorry about the long silence everyone, I've been traveling for the past several weeks and simply hadn't seen the notifications.
As Charlie noted, he's made several changes to address the changes we've been discussing and I've incorporated them into the Pull Request. You can see the full set of changes here and the changes relative to the original PR here.
As @dblodgett-usgs notes the rich diff function is really helpful for getting an overview of the changes.
I have the impression that we're nearing conversion, is this shared by the rest of the group?
I'd like to add my thoughts to what @czender has written concerning the lateral search: Although I do think that it is a bit confusing, especially for people who are used to object-oriented paradigms using inheritance and classes, there are a plethora of data in the wild which implement this and I think that it's good to describe how things should be found. I also agree that there are many data out there which reuse variable names when referencing coordinates, and the amount of such data is set to grow. It is likely that EUMETSAT is producing such data, as we will be disseminating products which contain observations from multiple instruments, all of which use different viewing geometries. Therefore they would have different lat and lon in the same file. Lateral search won't be needed in order to find the coordinates, but it's nonetheless a good example of multiple uses of the same names. So in my view the current PR, with the new edits from @czender , does a good job of finding the middle ground and ensuring readability across data from the past, present and the future.

Dear Daniel and Charlie
Thanks to you for the changes, and to Dave for reminding me where to find the "rich diff". I had forgotten which icon it is. Here are some comments on the latest version.

Thanks for the extra definitions in chapter 1. I feel that "element" should not be included here because it's used in other senses elsewhere (for instance, in "elements of an array" in the "CDL syntax" entry of sect 1.2). It would be fine to use it in this sense in the group discussion, if you define it specifically there with that meaning.

"Local apex group" as defined here refers to the common ancestor starting from two different places, but that's not quite the sense in which you use it in chapter 2, it seems. Where you discuss "lateral search", "local apex group" appears to mean the very top of the tree. You search upwards until you hit the top, and if you haven't found it, you start the lateral search, top-down.

"Nearest" is defined as being according to the rules for finding things (in chapter 2). But in those rules, you use "nearest" to say how to find things! This is circular. (A means "see B" and B means "see A".)

In chapter 2, I am puzzled by the last sentence of "search by absolute path", which says, "Thus a variable /g2/temperature with coordinates attribute containing /g1/lat is permissible so long as there is no other lat dimension defined 'in between' those locations, e.g., /lat." I thought absolute meant absolute, but you're making it conditional. That's not what a Unix absolute path means. Could you explain what you mean here, please? I don't understand the preceding sentence either.

Search by proximity depends on a definition of "nearest" (see comment above). I think by "nearest" you mean the one which is reached by the smallest number of upward steps. Because downward steps aren't allowed this isn't the usual meaning of "nearest", so it's worth being clear. Usually you would say that a daughter was a near relation of a mother, but here a daughter is entirely discounted.

Lateral search. There is no problem with having many variables in a file with the same name if they're in different groups. As Daniel says, by itself that doesn't require the lateral search algorithm. As currently stated, the lateral search algorithm doesn't look well-defined, because it says it searches "across" each level, but doesn't specify what collating order "across" means. No doubt that could be defined as part of the algorithm. I understand that you want to allow this algorithm because there are many existing datasets which depend on it, but (a) the existing datasets aren't ever going to be CF datasets, unless they're all rewritten to insert an appropriate Conventions attribute; (b) for new datasets, couldn't the data-writers use a better convention than they currently do? In many applications, CF compliance means doing things in a slightly different way to before. If they aren't willing or able to change to absolute/relative/nearest paths, maybe you could define lateral search as a subconvention which they could use but most CF data-writers would not?

I have a fairly large number of reservations about the proposed Appendix I. Partly this is because I don't understand some of it, partly because I feel that it goes beyond what CF normally does in being prescriptive. If you're keen to include this in the convention, I think that more discussions in detail will be needed.

I am keen on the approach in general and most of the details! I hope we will agree soon. Thanks for your efforts and patience.
Best wishes
Jonathan

Hi Jonathan,
Thanks for this input - I share your sentiment and also hope that we can reach a positive conclusion soon. I'll respond to your points inline below. All of these changes are reflected in the updated PR.

Thanks for the extra definitions in chapter 1. I feel that "element" should not be included here because it's used in other senses elsewhere (for instance, in "elements of an array" in the "CDL syntax" entry of sect 1.2). It would be fine to use it in this sense in the group discussion, if you define it specifically there with that meaning.

I see what you mean about the use of "element" in other parts of the Convention, particularly in reference to individual items within an array. I've reviewed our proposal and don't see a reason to define "element" specifically, and thus I've removed the definition from the glossary.

"Local apex group" as defined here refers to the common ancestor starting from two different places, but that's not quite the sense in which you use it in chapter 2, it seems. Where you discuss "lateral search", "local apex group" appears to mean the very top of the tree. You search upwards until you hit the top, and if you haven't found it, you start the lateral search, top-down.

This is true. I've also modified the wording in that section of the proposal; you can find it in the updated version. @czender please correct this if I have misunderstood how the lateral search works.

"Nearest" is defined as being according to the rules for finding things (in chapter 2). But in those rules, you use "nearest" to say how to find things! This is circular. (A means "see B" and B means "see A".)

I hope to have resolved this by changing the "Search by proximity" text as follows:
"A variable or dimension specified with no path is the variable or dimension of the same name which can be found via a search of direct ancestor groups with the shortest number of intermediate groups.
For example, a coordinates attribute of lat refers to the lat variable (if any) in the present group.
If the variable or dimension is not in the referring group then it is termed an out-of-group reference.
Out-of-group references are resolved by searching all direct ancestors, starting from the direct ancestor and proceeding toward the root group, until the specified variable or dimension is found."
Do you find this clearer?

In chapter 2, I am puzzled by the last sentence of "search by absolute path", which says, "Thus a variable /g2/temperature with coordinates attribute containing /g1/lat is permissible so long as there is no other lat dimension defined 'in between' those locations, e.g., /lat." I thought absolute meant absolute, but you're making it conditional. That's not what a Unix absolute path means. Could you explain what you mean here, please? I don't understand the preceding sentence either.

I see what you mean here, actually I think this is just a byproduct of text being moved around a lot. I've changed those three sentences to read:
"When referencing out-of-group variables, it is the responsibility of the dataset producer to ensure that the dimension(s) utilized by the out-of-group variable are the same as those used by the referring variable."
The intent here is to make sure that you don't refer to e.g. an auxiliary coordinate variable which uses different dimensions than the referring variable. If a data producer were to do this, it would not be possible to match the coordinates to the variable calling them up; the original text was designed to prevent this. However, the two sentences you referred to were in the wrong spot, and in my opinion they're not needed anyway, as the scoping rules clearly explain how that should work anyway, so I've deleted them and moved the whole block into the "Scope" section.

Search by proximity depends on a definition of "nearest" (see comment above). I think by "nearest" you mean the one which is reached by the smallest number of upward steps. Because downward steps aren't allowed this isn't the usual meaning of "nearest", so it's worth being clear. Usually you would say that a daughter was a near relation of a mother, but here a daughter is entirely discounted.

Right. I think that the newest version is clearer and reflects this unambiguously.

Lateral search. There is no problem with having many variables in a file with the same name if they're in different groups. As Daniel says, by itself that doesn't require the lateral search algorithm. As currently stated, the lateral search algorithm doesn't look well-defined, because it says it searches "across" each level, but doesn't specify what collating order "across" means. No doubt that could be defined as part of the algorithm. I understand that you want to allow this algorithm because there are many existing datasets which depend on it, but (a) the existing datasets aren't ever going to be CF datasets, unless they're all rewritten to insert an appropriate Conventions attribute; (b) for new datasets, couldn't the data-writers use a better convention than they currently do? In many applications, CF compliance means doing things in a slightly different way to before. If they aren't willing or able to change to absolute/relative/nearest paths, maybe you could define lateral search as a subconvention which they could use but most CF data-writers would not?

@czender I'll pass this to you, as the King of Lateral Search :)

I have a fairly large number of reservations about the proposed Appendix I. Partly this is because I don't understand some of it, partly because I feel that it goes beyond what CF normally does in being prescriptive. If you're keen to include this in the convention, I think that more discussions in detail will be needed.

What would you suggest for the appendix? Correct me please if I'm wrong, but I don't think that the appendix is prescriptive - the title is "Recommended Practices and Annotated Examples for Using Groups" so I see no danger that people would see this as a prescription. At the last meeting in Reading my feeling was that the community felt that having examples as guidance for different data types would be helpful; we've included them in the appendix with the intent of providing such examples without being prescriptive. I'm happy to discuss this in greater detail, and I don't want to regard your reservations, but I need more information about what they are in order to try to address them :)
Best regards,
Daniel

Dear Daniel
Thanks. This is much better. "Search by absolute path" should start with "A" not "An". In "Search by relative path" I would say "(i.e. containing a slash but not with a leading slash e.g. child/lat)".
I think "Search by proximity" can be stated more concisely and perhaps more clearly with less repetition, thus:
"A variable or dimension specified with no path (for example, lat) refers to the variable or dimension of that name, if there is one, in the referring group. If the variable or dimension is not in the referring group then it is termed an out-of-group reference. Out-of-group references are resolved by searching all direct ancestors, starting from the direct ancestor and proceeding toward the root group, until the specified variable or dimension is found.
"For coordinate variables that are not found in the referring group or its ancestors, a further strategy is provided, called lateral search. The lateral search proceeds downwards from the root group width-wise through each level of groups until the sought coordinate is found. This placement of coordinates and lateral search strategy to find them are discouraged. They are allowed mainly for backwards-compatibility with existing datasets, and may be deprecated in future versions of the standard."
I didn't understand the justification "Because coordinate variables must share dimensions with the variables that reference them", which I have omitted above. However, it made me realise that the lateral search algorithm may have problems with dimensions. It could find a coordinate variable with a dimension that has the right name e.g. lat(lat) but where lat isn't the same dimension as in the referring group (i.e. it has a different size)  if it's been defined differently somewhere between the root group and the group which contains the coordinate variable. This is dangerous. At least it requires another rule to state that the dimensions must be of the same size as in the referring group. Is the lateral search algorithm intended to work for auxiliary and scalar coordinate variables too?
The section on "Application of attributes" would be simpler and more compatible with the conventions if it referred to Appendix A. Maybe I've said that before? It seems to me that global atts mentioned in Appendix A should be allowed as group atts, with some exceptions e.g. Conventions, maybe external_variables, but why would you made an exception of title for example? Perhaps the simplest thing would be to modify Appendix A to show which G (global) atts are not permitted as group atts, and say that other G atts are allowed as group atts. I feel that you don't need to list the variable atts in this section.
Sorry to be vague about Appendix I. Yes, I understand and appreciate the intention of it. Please could we talk about that once we've agreed about the text of the main conventions?
Best wishes and thanks
Jonathan

Hi All,
Just a note to reassure everyone that I am receiving/following these recent comments questions and will compose a response next week after I unpack. In the meantime, please be aware that the ancestor search algorithm and the lateral search algorithm, as envisaged by the proposers, are not quite what Jonathan seems to think they are, so we need to do a better job of explaining them. Both searches should search no higher than the local apex group where the dimensions underlying the referring variable (and thus its coordinates) are defined. This is in general (and in practice) often lower down than the root group. In netCDF speak, the dimensions underlying the variable and the coordinate must be not only the same name, but the same ID (dimension IDs are unique in netCDF). Otherwise chaos ensues. The variable and the coordinate must both be in the scope (i.e., child groups) of the group where the dimension(s) are defined. That "feature" is due to the way that netCDF implements the Common Data Model (CDM). Hence ancestor search goes no higher than the group where the dimension(s) are defined, same as lateral search. This is the local apex group.

Dear Charlie
You're right, I had not understood that from the current wording, but indeed, this is something that has to be addressed, which I hadn't thought of. Also, does this algorithm apply to multidimensional auxiliary coordinate variables? It's possible that their dimensions might be defined in different levels of group, which is trickier. The upward search can proceed only as far as the group in which the dimension (or any of the dimensions) of the coordinate (or auxiliary coordinate) variable is defined.
Best wishes
Jonathan

Hi @JonathanGregory and community,
First of all happy New Year! Sorry about the long silence, it seems my data format work was on hibernation for a while, but I'm hoping that I can revive it ;)
I've tried to respond to https://github.com/cf-convention/cf-conventions/issues/144#issuecomment-429236366 point-for-point below (abridged):

"Search by absolute path" should start with "A" not "An".

Done.

In "Search by relative path" I would say "(i.e. containing a slash but not with a leading slash e.g. child/lat)".

OK.

I think "Search by proximity" can be stated more concisely and perhaps more clearly with less repetition, thus:
"A variable or dimension specified with no path (for example, lat) refers to the variable or dimension of that name, if there is one, in the referring group. If the variable or dimension is not in the referring group then it is termed an out-of-group reference. Out-of-group references are resolved by searching all direct ancestors, starting from the direct ancestor and proceeding toward the root group, until the specified variable or dimension is found.

This is fine for me, I've adopted the text as proposed.

"For coordinate variables that are not found in the referring group or its ancestors, a further strategy is provided, called lateral search. The lateral search proceeds downwards from the root group width-wise through each level of groups until the sought coordinate is found. This placement of coordinates and lateral search strategy to find them are discouraged. They are allowed mainly for backwards-compatibility with existing datasets, and may be deprecated in future versions of the standard."

I've adopted the text above with modifications to be somewhat clearer as per @czender's comment https://github.com/cf-convention/cf-conventions/issues/144#issuecomment-429464117. I think this is clearer and also clearly refers back to the local apex group which is defined in the proposed additions to Chapter 1 (https://github.com/cf-convention/cf-conventions/issues/144#issuecomment-429464117). Thus it now reads:
A special case exists for coordinate variables.
Because coordinate variables must share dimensions with the variables that reference them, the ancestor search is executed only until the local apex group is reached.
For coordinate variables that are not found in the referring group or its ancestors, a further strategy is provided, called lateral search.
The lateral search proceeds downwards from the local apex group width-wise through each level of groups until the sought coordinate is found.

[NOTE]
====
This use of the lateral search strategy to find them is discouraged.
They are allowed mainly for backwards-compatibility with existing datasets, and may be deprecated in future versions of the standard.
====

I didn't understand the justification "Because coordinate variables must share dimensions with the variables that reference them", which I have omitted above. However, it made me realise that the lateral search algorithm may have problems with dimensions. It could find a coordinate variable with a dimension that has the right name e.g. lat(lat) but where lat isn't the same dimension as in the referring group (i.e. it has a different size) if it's been defined differently somewhere between the root group and the group which contains the coordinate variable. This is dangerous. At least it requires another rule to state that the dimensions must be of the same size as in the referring group. Is the lateral search algorithm intended to work for auxiliary and scalar coordinate variables too?

I believe this is what Charlie meant (https://github.com/cf-convention/cf-conventions/issues/144#issuecomment-429464117) and indeed in my opinion when "local apex group" is so clearly specified (abridged from https://github.com/cf-convention/cf-conventions/issues/144#issuecomment-429464117: "The nearest (to a referring group) ancestor group in which a dimension underlying an out-of-group coordinate is defined.") I think it's pretty clear. It's also clear, I believe, in the revised text that coordinate variables must be the same size as the variables that refer to them, because "coordinate variables must share dimensions with the variables that reference them". I believe this is OK now but correct me if you don't agree.

The section on "Application of attributes" would be simpler and more compatible with the conventions if it referred to Appendix A. Maybe I've said that before? It seems to me that global atts mentioned in Appendix A should be allowed as group atts, with some exceptions e.g. Conventions, maybe external_variables, but why would you made an exception of title for example? Perhaps the simplest thing would be to modify Appendix A to show which G (global) atts are not permitted as group atts, and say that other G atts are allowed as group atts. I feel that you don't need to list the variable atts in this section.

I get this here. I've shortened the section and down and think that the result is much shorter and clearer.
In order to make the logic of attributes and how they apply to variables very explicit I have added the text "Furthermore, per-variable attributes must be attached to the variables to which they refer.
They may not be attached to a group, even if all variables within that group use the same attribute and value.". That should ensure that e.g. missing_value isn't attached to a group if all variables within that group use the same missing_value.
So now the section has been reduced to 3 paragraphs - I still think this makes sense because it's specifically about the logic of groups specifically - how some attributes can only be used in /, how not to use attributes which actually refer to variables, and how attributes attached to groups are inherited (or not inherited) by other groups. Initially I did move some stuff into Appendix A but after re-reading I decided to keep it all in Chapter 2, as it is right now. Let me know what you think about that.

Sorry to be vague about Appendix I. Yes, I understand and appreciate the intention of it. Please could we talk about that once we've agreed about the text of the main conventions?

Yes, let's discuss once we've got the changes to Chapter 2 nailed down (soon I hope!).
I think it's important to note that this proposal seems to work in practice at least, as shown by @czender's use of it in the NCOs and the implementation in GDAL (shown above this comment). This also highlights for me the importance of agreeing on the final wording soon, so that any software operating using this concept can have a stable baseline.
What do you think of the latest changes? Are we nearing concensus? If so, we could move on to addressing the issue with Appendix I, which I have left out of this response.
Best regards,
Daniel

Hello Jonathan, hello all,
Thanks for reviving discussion on this topic. A lot of it has taken place close to the proposal itself (which I personally prefer, as it makes it easier to pinpoint what and where exactly changes are needed). These contributions have been resolved in the pull request itself.
The quoted text below is from Jonathan's comment, which I am abridging and answering inline for the sake of clarity. Thanks very much for the productive input - I'm glad to see so much discussion around this important topic.
Responses to Jonathan's comment

Could we please omit Appendix I for the moment from the pull request? By the way, it is entitled Appendix A.

I don't really understand this request, to be honest, because I think that it would make it easier to interpret the additions, but I've done it for the moment (fcad2b7cddc264b2ccb7bf38876d0a3b4f5a5257). Also, the reason it's called "Appendix A" in the rendering is because the appendix numbering actually takes place during the compilation of the Conventions as a whole, so when you view the individual file, the [appendix] tag is converted into "Appendix A" due to the lack of context. If you compile the complete Conventions it's titled appropriately.
I don't think that the appendix is really a break from convention, especially in light of Appendix H, which is large and comprehensive, as well as extremely useful, and would prefer to keep it, but I'll defer to your judgement.

On the title page, please add yourself as an additional author, and also Charlie if you and he think you should both be named.

Done (782a15cb704620b2e99abfa14169a5ab95898704).

I think the definition of "local apex group" is clear, but below I argue that we should discard this notion.
Is it necessary to define "location" as a group-specific term? It sounds to me like a generally applicable idea, and I wouldn't be surprised if people use it in senses not related to groups. I suggest we don't need to define it. It's obvious what it means, in context.

I agree with you here and have removed it.

You have a definition for "nearest coordinate or variable", which starts "The variables that ...". It seems confused about whether you mean any kind of variable, or specifically a coordinate variable. Please could you make it consistently one or the other?

Agreed, I've updated this to read "nearest variable".

The definition of nearest dimension seems unnecessarily complicated to me. What about:

Nearest dimension. The nearest dimension (having a given name) is the one in the referring group, if that exists, or else in the nearest ancestor group of the referring group.

But do you really need to define nearest variable and dimension? These terms are never used, as far as I can see.

Here you're right again, I've removed the "nearest dimension" definition - that should cut back the bloat some.
All the changes to the glossary can be found in 8480c30666d0988447d1c177b3df74fb25fd85ef.

Turning to the new groups section in ch02, I would delete the "Instead" in the first para. I don't see why it's "Instead" - it makes better sense to me without that word!

This is fine by me.

I don't think it's quite clear what this means:

When referencing out-of-group variables, it is the responsibility of the dataset producer to ensure that the dimension(s) utilized by the out-of-group variable are the same as those used by the referring variable.

Charlie clarified that it doesn't just mean the same name, or the same size, which are both possible interpretations, but the same netCDF dimension. I think we need to be absolutely clear about this, because it's essential, as I now appreciate. I suggest we could replace this with a simpler statement:

If any dimension of an out-of-group variable has the same name as a dimension of the referring variable, the two must be identical dimensions (i.e. they must have the same netCDF dimension ID).

Agreed, but I've no gone a step further. The text now reads "If any dimension of an out-of-group variable has the same name as a dimension of the referring variable, the two must be the same dimension (i.e. they must have the same netCDF dimension ID)." (bold added to highlight change).

You point out that ".." is as in Unix. If I understand correctly, paths in general follow Unix conventions, so perhaps this should be stated. For instance,

A variable or dimension specified with an absolute path (i.e., with a leading slash "/") is at the location indicated relative to the root group, as in the UNIX convention.

and

Following the UNIX convention, a variable or dimension specified with a relative path (i.e., containing a slash but not with a leading slash, etc...

OK. I have implemented this mostly like that, but rather than using the term "UNIX convention" I write "as in a UNIX-style file convention." I'm not aware of a Unix convention and wouldn't want people erroneously searching for one.

Under "search by proximity", we don't need to define out-of-group, because it's already been used in the section, and it's in the terminology section. Also "direct" is repeated in the last sentence of this para. I realise I drafted the current text, but now it's in context I see it could be even shorter! I think the para could read

A variable or dimension specified with no path (for example, lat) refers to the variable or dimension of that name, if there is one, in the referring group. If not, we search the ancestors of the referring group for it, starting from the direct ancestor and proceeding toward the root group, until it is found.

Yessss, shorter is better. Agreed, this makes me happy.
The changes agreed above are implemented in f376a3b7ad688595aa9492d6fd0a98cd30098a77.

I've read again and thought about what you and Charlie wrote concerning the local apex group. Now I understand it, I think there's a problem. You explain that you have to stop looking for a coordinate variable at the group where its dimension is defined, because above that level it could not have the same dimension. You can apply this rule for a coordinate variable because it has the same name as its dimension, so you know the name of the dimension before you find the variable. However you could not apply it for auxiliary coordinate variables, cell measures, or any other kind of variable that might be referred to. Until you find the variable, you don't know its dimensions, but by the time you find it, you might already have gone past the definition of the dimensions. For example, in pseudo-CDL,
parent group:
x=10;
y=20;
float x(x);
float latitude(x,y);
float longitude(x,y);

child group:
x=30;
y=60;
float y(y);

grandchild group:
float temperature(x,y);
  temperature:coordinates="latitude longitude";

In this situation, you can find y(y) for temperature in the child group, but the search fails for x(x), since you find the dimension x before you reach the variable x, and therefore you know that it can't be the right x(x). However, you don't know the dimensions of latitude and longitude until you find them. They aren't the right ones either, but to discover that you'd have to search downwards again, checking for definitions of their dimensions. This seems rather complicated.
The problem would be avoided if proximal search was allowed only for coordinate variables (in the Unidata sense) and not for any of the other kinds of variable that CF defines. That would be a shame, I think, as it would restrict the CF application of groups. I think that a simpler solution would be to abandon the idea of local apex group. Instead, you carry out the search, and check the dimensions afterwards. Repeating what I put above, and continuing it.

A variable or dimension specified with no path (for example, lat) refers to the variable or dimension of that name, if there is one, in the referring group. If not, we search the ancestors of the referring group, starting from the direct ancestor and proceeding toward the root group, until the variable or dimension is found. The search fails if no variable or dimension of the given name is found. In the case of a variable, the search fails also if the first variable found of the given name has any dimension which has the same name as, but is not identical with, a dimension of the referring variable.

Without the notion of "local apex group", we don't know where to start the lateral search. Also, the apex group is not well-defined if more than one dimension is involved, as for multidimensional aux coord vars. Hence I suggest restricting the lateral search to coordinate variables only. Since this is a legacy feature, I assume that's the only application required anyway. Therefore I suggest replacing the paragraph starting "A special case" as follows:

For out-of-group coordinate variables (in sense of the NUG, not auxiliary coordinate variables), a further strategy is provided, called lateral search. To execute the lateral search, we first locate by proximal search the definition of the dimension of the coordinate variable we seek. (We already know that the coordinate variables itself is not in this group, because we did not find it by proximal search.) The lateral search looks for the coordinate variable in all the groups below the group defining the dimension, except the one which is an ancestor of the referring group (since we know the variable we seek is not there), then the groups below them, and thus continues downwards until a coordinate variable of the given name is found. We recommend not depending on the lateral search strategy. It is allowed for backwards-compatibility with existing datasets, and may be withdrawn in future versions of the standard.

As you see, I think we should deprecate it now. If I've understood it correctly, it is problematic even for coord vars. What if the dimension is redefined in one of the sibling groups that is searched? That group and any children of it would contain a wrong variable. Of course you could discard it because the dimension ID will not match what you want, but if that's part of the algorithm it should be stipulated. Another problem is that at any level there might be more than one coordinate variable, in sibling groups, with the required name and dimension. Since the order of the width-wise search is not defined, it's not predictable whic one we will choose. Is there a rule for that? I continue to think that lateral search is problematic, as well as complicated - a diagram might help.

I have to defer this to @czender as he understands the intricacies of the needs for lateral search better than I do. The solution sounds reasonable to me but I will hold off on changing the text until he's had a chance to respond.

I like your clarification of the handling of attributes, but I have some comments on them.

Why do you want to disallow title and history for groups?

@czender any input on this? I don't feel strongly about it, but I do think it's a good idea. It makes sense to track the history and title of a file as an atomic entity, in my mind, even if a file might be "recombobulated" at a later time with grousp which have been copied from elsewhere. Otherwise there's scope for multiple, confusing entries which come from different sources and only apply to certain subgroups. Sure, this fits with the idea of inheritance of attributes and would in theory be well-defined, but I predict that in practice it would be cumbersome, annoying, and confusing in almost any case where it was used, and the benefit in any case where it was used in a way that did not confuse or annoy would be negligible.

Conventions should have a capital C.

This was fixed in https://github.com/cf-convention/cf-conventions/pull/145/commits/9ca5ba76c35b7af5ed05fdbe7fa4c3f3135d0ef8.

I think it would make sense to disallow external_variables, which refers explicitly to files.

I don't really see the benefit of disallowing that, but am open to discussing it. Could you provide some more information on what you'd like to see here?

What about featureType? In Appendix A, it too refers to files, so it too should be disallowed for groups as it stands. If you want to allow it at a group level (which you do implicitly by not excluding it), the text in App A should be changed.

I agree, I've adapted the text in 5332e8c as follows: "Specifies the type of discrete sampling geometry to which the data in the scope of this attribute belongs, and implies that all data variables in the scope of this attribute contain collections of features of that type."
Conclusions
I believe that we're very close to consensus now and hope that you do too - in my mind the last outstanding item is some input concerning local apex groups. @czender if you could sound off on that it would be much appreciated. Thanks for the valuable collaboration again - I'm looking forward to the next iteration of comments on this proposal.
Best regards,
Daniel

Dear Daniel
Thanks for this. I agree, we've almost agreed!
Sorry to be unclear about Appendix I. I agree that it is fine to have such an Appendix. My request was that we should postpone it until we'd agreed on everything else, because it's quite long and will need some study. That is, could you reintroduce it as a new issue when we've finalised this one?
I may not have understood how to read the pull request. Do you think you need the definitions for location and nearest dimension? They appear to be present in the rich diff, but they're standing out against the margin, so perhaps they aren't there really. Also, I am not sure you need "nearest variable". Do you actually use this term?
Whether "local apex group" is needed depends on the definition of the lateral search algorithm, on which we need Charlie's advice, as you say.
Thanks for "UNIX-style file convention".
I think external_variables should not be allowed for groups because it specifically identifies and hence "tolerates" variables which are named by attributes but not present in the file; it does not suggest where to find them. This treatment is not necessary for variables not present in groups. The change we are discussing is specifically about what to do when a variable is not present in the group that refers to it.
Best wishes
Jonathan

Hi Jonathan,

Thanks for this. I agree, we've almost agreed!

That's great to hear! :)

Sorry to be unclear about Appendix I. I agree that it is fine to have such an Appendix. My request was that we should postpone it until we'd agreed on everything else, because it's quite long and will need some study. That is, could you reintroduce it as a new issue when we've finalised this one?

Now I understand better. I'm happy to do that.

I may not have understood how to read the pull request. Do you think you need the definitions for location and nearest dimension? They appear to be present in the rich diff, but they're standing out against the margin, so perhaps they aren't there really. Also, I am not sure you need "nearest variable". Do you actually use this term?

I also find the rich diff a bit hard to read, as I have deuteranomaly and red/green are the signal colours here. In fact, I had removed "locaiton" and "nearest variable" from the current draft and intended to remove "nearest dimension", but forgotten it. I've corredcted this error in 31a16e5.

Whether "local apex group" is needed depends on the definition of the lateral search algorithm, on which we need Charlie's advice, as you say.

Agreed.

Thanks for "UNIX-style file convention".

Happy to.

I think external_variables should not be allowed for groups because it specifically identifies and hence "tolerates" variables which are named by attributes but not present in the file; it does not suggest where to find them. This treatment is not necessary for variables not present in groups. The change we are discussing is specifically about what to do when a variable is not present in the group that refers to it.

I see your point here and agree with it. I've updated it in c3286fe.

Daniel, Jonathan, and others interested,
This is fresh enough in people's minds, and evolving quickly enough, that I do not need to respond point-by-point to Jonathan's substantive recommendations. By and large I agree with them and with Daniel's revisions. Any exceptions I note below, after providing additional context on the implementation of ancestor-search and lateral-search. To cut to the chase: I concur with restricting lateral search to NUG coordinate variables, and I think that the definition of "local apex group" should be retained for that purpose. The rest of this provides additional background in the previous discussions, and then, at the end, responses to the question about per-group "title" and "history". Apologies for the length. Viewer discretion is advised.
Jonathan's preceding comments show good understanding of the details necessary to implement a lateral or ancestor search algorithm. This is great for informed decision-making regarding the Groups extension to CF. He constructs cases that are difficult for the lateral (and possibly ancestor) search algorithm to navigate, and suggests these cases illustrate why lateral search should be restricted to NUG-style (traditional) coordinate variables rather than auxiliary coordinate variables. His examples are technically valid yet artificial, and may be of limited or no "real-world" importance. I look at well, many netCDF4 datasets in my capacities as NCO designer and (full-disclosure) sometimes NASA interoperability architect and cannot remember any like those in his example. All dataset designers strive to organize dimensions, coordinates, and variables in a transparent way. Even if placing multidimensional auxiliary coordinates in cousin groups is not one's cup of tea, it is clear and predictable not convoluted and indecipherable. It is a pattern that nearly two full generations of data scientists are familiar with and I think CF could learn from it before discounting it as something unsupportable.
I agree that there is no completely unambiguous search algorithm that will resolve all bizarre ways dataset producers could organize dimensions and auxiliary coordinate variables in netCDF4 files. Most potential ambiguities are handled by the lateral search algorithm I proposed if the algorithm also checks for exact agreement between the dimensions of the variable and its Out-of-Group (OOG) coordinates (both NUG and auxiliary). The algorithm keeps searching until it finds a variable of the right name that contains the identical dimensions (i.e., same dimension IDs) as the referencing variable. Even then one could have two such auxiliary coordinates with the same name and different numbers of (valid) dimensions in different cousin groups, so the notion of "nearest" becomes important and then the ordering ambiguity of the lateral groups at a given level comes into play. Contriving implausible situations is not too difficult in theory, and one must weigh their value with their probability and realism.
I would like for the Groups extension to be valid for any implementation of the extended Unidata Common Data Model (CDM). Unidata's netCDF4-C Library is one implementation of the Extended CDM. netCDF-Java is a closely related implementation. My understanding is that xarray has developed or is developing a third. The preceding paragraph references two behaviors that may be implementation-dependent: 1) That each defined dimension gets a unique ID valid in the global namespace of the file. 2) Group IDs are returned by nc_inq_grps() in the order of their creation. Behaviors (1) and (2) are not mandated by the CDM, and are documented poorly, if at all, by netCDF4-C, and indeed other implementations of the CDM might choose to implement different behaviors. Behavior (2) is highly relevant to lateral search because it determines the ordering of sibling groups. That ordering could change if/when the file is copied group-by-group but in a different creation order, meaning lateral search could return different results from copies of the same file.
These potentially implementation-dependent behaviors of the extended CDM are a weakness for the lateral search algorithm, and are partially why I left dimension ID's out of the original draft of lateral search, even though dimension ID's are so very helpful at resolving ambiguities. And the preceding explains the undocumented behavior that one can expect (at least from netCDF-C) for the ordering of sibling groups within lateral search. All of these potential uncertainties notwithstanding, I think 99.999999% of all "real world" cases for OOG searches would be solved as intended using the proposed lateral search mechanism simply because dataset producers are sensible, and are incentivized to organize datasets that work well with the widely supported CF Conventions.
Most CF supporters are not interested in understanding the details of which lateral searches are which and are not well-posed. I concede that the CF implementation of groups that I favor is more powerful yet vulnerable to abuse and misunderstanding and possibly breakage than the less powerful yet more robust implementation favored by most of the CF community who have weighed in. I appreciate the compromises that Jonathan and others have already made in considering lateral search in a deprecated and weakened form. I think the best way forward is to proceed with Jonathan's suggestion to allow lateral search for NUG-coordinates, and to disallow it for auxiliary coordinates. Some form of lateral search is better than none.
Conventions are living documents. Once people start to use CF Groups they may think of new ways to extend their power and propose extensions that we cannot yet anticipate. The limited and deprecated lateral search capabilities will give people a taste of what could be supported in the future with more robust and exhaustive test. Thanks to these discussions the potential pitfalls of more powerful rules for Groups are well known. NCO will continue to be an implementation of the original Groups lateral search features and people can experiment with that if they wish.
It is important to recognize what CF Groups loses and gains by not adopting the lateral search algorithm originally proposed. Losses include support for OOG multi-dimensional auxiliary coordinate variables. Much of NASA's Level 2 (swath-like) data has latitude and longitude as 2-dimensional auxiliary coordinate variables in cousin groups to the referencing data. Orphaning these data outside of CF-compliance by adopting a watered-down lateral search will estrange a significant amount of otherwise CF-compliant data. Yet CF will gain support for hierarchical datasets without introducing any known ambiguities. Such are the trade-offs of compromise. I hope you will support this compromise.
Regarding the global attributes like "history" and "title":
I think it is fine if datasets have per-group "history" and "title" attributes so long as CF better delineates their scope and priority. Per-group "history" and "title" can easily arise in practice through aggregation of flat (e.g., ncecat --gag) into hierarchical files. Should this be CF-compliant and, if so, what should those per-group attributes mean? I think "history" and "title" could be allowed in groups if CF also explicitly states that the "canonical" attribute to be read/used/modified by applications is the root-group attribute. The per-group attributes would then be relegated to providing additional provenance and description of the subsidiary data. Per-group "history" would possibly fall out-of-date relative to the root "history". Root-level attributes should be authoritative and up-to-date. For example, data processing should alter root-group "history" not all sub-group "history"s. "title" in the root group is THE name of the (collection of) dataset(s) in the file, while per group "titles" describe only the title of their respective data.
The solution for "history" and "title" will likely (as Daniel writes) be either messy and confusing if CF allows per-group versions, or difficult to implement and lossy if only root-group attributes are allowed. Difficult to implement because stripping-out those attributes during aggregation is burdensome to tools, and lossy because those attributes can contain valuable and irreplaceable information. NCO faces a similar problem when one appends data from a source to a destination file (i.e., ncks -A -v temperature source.nc destination.nc). NCO tries to preserve the provenance of both datasets. It updates the "history" of the destination file, and it copies (and appends) the "history" of the source file into a separate attribute named "history_of_appended_files" in the destination file.
I think the "Conventions" attribute strengthens the case for explicitly declaring the precedence of (formerly) global attributes that may be stored in groups due to aggregation. It is too crazy to expect CF readers and compliance checker to have to adhere to different CF conventions version. That said, there are definitely files "in the wild" with per-group "Conventions" attributes (simply because ncecat --gag does not strip "Conventions" from groups). Regardless of whether such files are CF-compliant, I think CF should explicitly and pro-actively state that the root-level "Conventions" takes precedence over any per-group "Conventions" attributes.
In summary, I think that CF should allow "Conventions" and "external_variables" only in the root-group. At the same time, CF should provide a well-defined precedence for handling the non-compliant files: that the root-group attribute, if any, takes precedence. And I think CF should allow per-group "title" and "history" since forbidding them would degrade dataset provenance. The canonical "title" and "history" of a dataset would be those at root-level.
Charlie

I would like to add my voice to this one bit where I am reasonably confident I understand the issues. I appreciate that per-group "history" and "title" attributes can feel overly noisy/messy/confusing, but Charlie lists important use cases, and I think there are other use cases for data that is aggregated directly by data systems (that is, not just through ncecat and related tools). Being able to capture this information on a more local level is extremely helpful in understanding the composition of the entire data set, and I recommend if the cost is not unusually high (and in fact, it seems negligibly different or even cheaper to leave those terms in).
Leaving Conventions in likewise is informative, and I support Charlie's recommendation for declaring the root value is the one that takes precedence. The others are informative but can not be drivers, as that would be expensive.
John
On Feb 7, 2019, at 11:35 AM, Charlie Zender <notifications<EMAIL_ADDRESS>wrote:
Regarding the global attributes like "history" and "title":
I think it is fine if datasets have per-group "history" and "title" attributes so long as CF better delineates their scope and priority. Per-group "history" and "title" can easily arise in practice through aggregation of flat (e.g., ncecat --gag) into hierarchical files. Should this be CF-compliant and, if so, what should those per-group attributes mean? I think "history" and "title" could be allowed in groups if CF also explicitly states that the "canonical" attribute to be read/used/modified by applications is the root-group attribute. The per-group attributes would then be relegated to providing additional provenance and description of the subsidiary data. Per-group "history" would possibly fall out-of-date relative to the root "history". Root-level attributes should be authoritative and up-to-date. For example, data processing should alter root-group "history" not all sub-group "history"s. "title" in the root group is THE name of the (collection of) dataset(s) in the file, while per group "titles" describe only the title of their respective data.
The solution for "history" and "title" will likely (as Daniel writes) be either messy and confusing if CF allows per-group versions, or difficult to implement and lossy if only root-group attributes are allowed. Difficult to implement because stripping-out those attributes during aggregation is burdensome to tools, and lossy because those attributes can contain valuable and irreplaceable information. NCO faces a similar problem when one appends data from a source to a destination file (i.e., ncks -A -v temperature source.nc destination.nc). NCO tries to preserve the provenance of both datasets. It updates the "history" of the destination file, and it copies (and appends) the "history" of the source file into a separate attribute named "history_of_appended_files" in the destination file.
I think the "Conventions" attribute strengthens the case for explicitly declaring the precedence of (formerly) global attributes that may be stored in groups due to aggregation. It is too crazy to expect CF readers and compliance checker to have to adhere to different CF conventions version. That said, there are definitely files "in the wild" with per-group "Conventions" attributes (simply because ncecat --gag does not strip "Conventions" from groups). Regardless of whether such files are CF-compliant, I think CF should explicitly and pro-actively state that the root-level "Conventions" takes precedence over any per-group "Conventions" attributes.
In summary, I think that CF should allow "Conventions" and "external_variables" only in the root-group. At the same time, CF should provide a well-defined precedence for handling the non-compliant files: that the root-group attribute, if any, takes precedence. And I think CF should allow per-group "title" and "history" since forbidding them would degrade dataset provenance. The canonical "title" and "history" of a dataset would be those at root-level.

Dear colleagues,
Thanks for these valuable inputs. I agree with Charlie's and John's ideas and have implemented them as follows:
"Nearest variable" definition
Charlie has done an excellent job of explaining the necessity of understanding what is the nearest dimension, group, variable and think I do think that this is crucial to being able to implement the search correctly. Originally I had replaced the definitions for nearest dimension, variable and coordinate in 8480c30666d0988447d1c177b3df74fb25fd85ef and had instead simplified them into "nearest variable" in 31a16e5c5783de59616e30ea16ff2f334cb3c5aa. I still think that this simplification is an improvement and have thus reverted 31a16e5. However, in order to encompass also groups, I've changed the wording. This introduces the "nearest item" entry and is thus referred to in the "local apex group" definition. The final wording can be found in bb9ce25.
Limitations to lateral search
Like many others in the CF community, I have my difficulties with the lateral search algorithm and have specified the compromise proposed by Charlie in 94ddcc3, namely that lateral search is allowed for NUG coordinates, and not for auxiliary coordinates.
That being said, I do believe in making standards for the real world and agree that it's a shame to leave such a large amount of data "outside" the convention if the producers would like them in, and their variable discovery mechanisms work in a practical, demonstrated fashion. The lack of documentation of how group IDs are returned by nc_inq_grps is a barrier here. Is it too much to dream of keeping this alive somehow and perhaps pursuing this in upstream standards, such as the netCDF or HDF standards? I'd like to revisit the thought outside the frame of this discussion and am willing to pursue an unambiguous solution at a later time. If we can guarantee that the group IDs will be returned in the same order across implementations, we could consider amending the Groups section at a later time in order to accomodate existing e.g. NASA datasets. I expect this would take quite some time but could very well be worthwhile.
Root-only attributes
As discussed exhaustively in the last two comments from @czender and @graybeal , I've provided what I find to be clearer guidance on the use of the attributes

title
history
Conventions
external_variables
in 03326d2.

Conclusions
I believe that all outstanding issues have been addressed (@JonathanGregory please correct me if I am wrong). Does anybody have any remaining concerns?
Best regards,
Daniel

FWIW these latest changes look good to me. I also second Daniel's suggestion to re-visit search issues after groups have been in CF long enough so that it is clearer what users want and whether the various netCDF implementations can adopt a common and unambiguous order for returning child groups, and whether unique, global dimension IDs can be made a part of the netCDF standard so they can be relied upon.

I support this proposal, with some minor caveats.
I have put some editorial comments into a review on #145
https://github.com/cf-convention/cf-conventions/pull/145#pullrequestreview-205691511
There are a couple of aspects that I think are important to address, in order for the PR to be self consistent, with regard to the inclusion of the appendix, and consistent with other use, in the history page.
I also have suggestions on text additions for clarification which may be worthy of consideration
thank you
mark

Hi @marqh thanks again for your valuable review. I believe I've addressed all the issues you raised in https://github.com/cf-convention/cf-conventions/pull/145#pullrequestreview-205691511 - please let me know if you agree on the resolution of the outstanding issues.
@JonathanGregory , I believe that @marqh , @czender and myself all believe that the proposal would benefit from the inclusion of Appendix I. Would you object to its re-inclusion? We've got some more detailed discussion on this in the link above.
Thanks all for your valuable input!
Daniel

Dear colleagues,
I'm cross-posting this on the CF metadata list too in order to ensure that this has the proper visibility, since I'm not too sure how far the migration to GitHub is and what's most appropriate. Sorry for the noise.
The discussion on the issue of regulating the use of netCDF-4 groups in CF has fallen into a lull now for three weeks, after substantial improvements through contributions from @marqh , @czender and @JonathanGregory . I believe this is due to the fact that we have reached consensus, or as near as we will come to it. @JonathanGregory as the moderator of the discussion, would you be willing to summarise the discussion so that we can move into the next phase before adoption?
Looking forward to hearing back from you all and have a nice weekend,
Daniel

Dear Daniel
Thanks for your email. In fact since we don't have the GitHub link to email
switched on I wasn't aware of the ongoing discussion. I'm watching it now.
I'm busy this week with a meeting and will look next week.
Best wishes
Jonathan
On Fri, Mar 29, 2019 at 01:51:00PM -0700, Daniel Lee wrote:

Date: Fri, 29 Mar 2019 13:51:00 -0700
From: Daniel Lee<EMAIL_ADDRESS>To: cf-convention/cf-conventions<EMAIL_ADDRESS>Cc: JonathanGregory<EMAIL_ADDRESS>Mention
<EMAIL_ADDRESS>Subject: Re: [cf-convention/cf-conventions] Support of groups in CF (#144)
Dear colleagues,
I'm cross-posting this on the CF metadata list too in order to ensure that this has the proper visibility, since I'm not too sure how far the migration to GitHub is and what's most appropriate. Sorry for the noise.
The discussion on the issue of regulating the use of netCDF-4 groups in CF has fallen into a lull now for three weeks, after substantial improvements through contributions from @marqh , @czender and @JonathanGregory . I believe this is due to the fact that we have reached consensus, or as near as we will come to it. @JonathanGregory as the moderator of the discussion, would you be willing to summarise the discussion so that we can move into the next phase before adoption?
Looking forward to hearing back from you all and have a nice weekend,
Daniel
--
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/cf-convention/cf-conventions/issues/144#issuecomment-478145546

Dear Daniel and others
Sorry to have to ask this GitHub novice question yet again, but please can you tell me how to see a version including all the changes currently proposed, after the above discussion, preferably as rich diff and as complete amended text.
Thanks very much
Jonathan

The way I do it is to go to the pull request  - https://github.com/cf-convention/cf-conventions/pull/145 in this case - and then click on the "Files changed" tab. This then shows all the files changed as raw diffs, and you can choose to view the Rich diff, or view the whole file.
If there's a different/better way - I'd like to know, too!

I usually do it the way David describes.   However, if that doesn't work, you could download the files and diff them using your favorite tool.

Dear Daniel, Charlie and others
Regarding my novice question, the point (for me) to remember is that the pull request number can't be discovered automatically from an issue. Hence it's convenient for readers to have it restated it in an issue whenever it's relevant, I suppose.
Thanks for your work on this, and for the clarifications and compromises. I am almost content with this proposal now! I realise I'm still contributing just now as a reviewer rather than a moderator, but don't be concerned. I can be objective and tell myself when the debate has been concluded satisfactorily. :-) Also let me emphasise that I think this proposal fits well into CF and will be a valuable addition.

The lateral search is well-defined now, and the problems I was concerned are avoided by defining the local apex group as being where the dimension is found, and not allowing this algorithm for aux coord vars. A minor point: in the definiton of local apex group you write, "in which a dimension underlying an out-of-group coordinate is defined", and I wonder, why "underlying"? It's not more special than "of" would mean, is it? I appreciate that you don't want to deprecate lateral search at present because of excluding some significant datasets. I also understand that it's regrettable to exclude aux coord vars. I think we could consider admitting them too provided a reliable algorithm is defined. I hope that with the datasets that use this method there is indeed a way to be sure you've got the right variable. We could return to that when this proposal is agreed.

I don't we should allow Conventions at group level, even if it is overridden at the file level. It seems potentially confusing to me that a single file could simultaneously use more than one version of CF, and possibly it might be inconsistent. For example, what if an out-of-scope time coordinate variable used a calendar which is permitted in its version of CF but is not included in the earlier version of CF applicable to the referring variable. Maybe there are more likely examples than that - I haven't spent long thinking about it, but it concerns me!

I think that Appendix A should be modified to show the new status for title and history being allowed for groups. I suppose we haven't thought about this before. Letter G is current used for "global", but you could amend that to "F" for file and use G for "group" instead.

I am sure we can adopt this proposal soon. As soon as we have, we should consider your appendix on best practice for groups as a separate addition, if you don't mind - only because otherwise I'm worried about delaying the main proposal.
Best wishes
Jonathan

Thank you for refreshing this proposal with your new questions and comments, Jonathan. Regarding "underlying": I agree that it is synonymous with and could be replaced by "of" in this context. Regarding Conventions: I share your opinion that Conventions should only be allowed at the file level. Anything else would be considered non-compliant. I will consider modifying NCO to automatically strip (or rename, more likely) group-level Conventions attributes when aggregating files, just to make it simpler to produce CF-compliant files using ncecat --gag, and harder to produce non-compliant files.

Hi @JonathanGregory and all, sorry for spamming with multiple force pushes but I was having trouble with table formatting and didn't have access to an offline renderer. I've implemented the comments as requested. Jonathan, do you still feel good about summarising the discussion or should we reach out to somebody else from the governance panel when the 3 weeks are up? I feel you're pretty objective but I don't want to put you in an uncomfortable spot ;)

Dear all
Thanks for the latest changes, Daniel. Personally I am happy with this now. I am willing to be moderator still as well! Speaking in that capacity, my summary of this discussion is that there are apparently no unanswered comments or questions. Does anyone have any further comment to make on the proposal as currently shown in https://github.com/cf-convention/cf-conventions/pull/145? Note that this pull request does not contain the new appendix on best practice for groups, which we'll consider separately when this one is agreed. Daniel, is there a corresponding pull-request for the conformance document?
Best wishes
Jonathan

Thanks @JonathanGregory !
Currently there is no corresponding pull request for the conformance document - I'm a bit swamped right now but I'll see if I can find a willing partner in crime so we can produce one ;)
Best regards,
Daniel

@erget and @JonathanGregory ,
I drafted a test netCDF file:
cf_grp.cdl
cf_grp.nc
I tried to make it succinct, yet able to test numerous Groups features, including global/group attributes, absolute/relative/ancestor searches for out-of-group coordinates in an ancestor group, absolute/relative/ancestor searches for out-of-group coordinates in a sibling group, distinct coordinates with the same name yet different dimensions in the same file, incorporating ensemble realization in both the group name and as metadata. Feedback welcome. Is this too much? too little? Feedback welcome. If it looks good I'll create a PR for it.
Charlie

Hi @JonathanGregory - we've passed the three week boundary again. Should we freeze the proposal as it stands? What are the next steps? My thoughts are that now that the content of the change is agreed we might move on to "implementing" it as rules in the conformance document.

Dear Daniel
I had noted this as due for attention tomorrow! If no-one has any more comments today (14 May), then this change has been agreed and should be included in the next release of CF, provided that the conformance document changes have also been prepared by then.
Thanks for the test file, Charlie. That's not something we usually have, but it's a good idea, often stated to be desirable! For the conformance document, we need a list of checks that can be made, and whether the failure of each of them should produce a warning (because it breaks a recommendation) or an error (because it breaks a requirement).
Cheers
Jonathan

@erget I suggest that this text be incorporated into a PR for the Conformance document. These points are relatively easy to test for conformance. I could not think of any other points that the Groups feature allows, forbids, or recommends that could be easily tested for. Do you think this is complete?
2.7 Groups:
Requirements:
Auxiliary coordinate variables must reside in the referring group or one of its direct ancestors.
The attributes Conventions and external_variables may be used in the root group only.
Per-variable attributes including units, _FillValue, scale_factor, add_offset, valid_min, valid_max, and valid_range must only be attached directly to variables, and not provided as group-level attributes.
Recommendations:
NUG-coordinate variables that are not in the referring group or one of its direct ancestors should be referenced by absoulute or relative paths rather by relying on the lateral search algorithm.

@czender @JonathanGregory forgot to update on this - I've proposed a corresponding update to the Conformance document in this PR. Next steps?

Daniel @erget, yes, I will look at the proposed update to the conformance document. I will do that next week. Ros has had a look at it and commented by email as well.

@JonathanGregory thanks very much!

Dear Daniel et al.
I'm reopening this because the conclusion of the conformance document is an essential part of making
the change to the conventions. Once we have them in the same repository, so they can be done will the same pull-request, that will seem more obvious! Ros, David and I discussed the conformance requirements this morning. Ros is going to post some comments on the draft requirements in https://github.com/cf-convention/Conformance/pull/8, and a small number of suggested further changes to the conventions about groups, which came up while considering conformance checking, for clarity and consistency. (These aren't large changes of substance to the proposal, as you will see.)
Best wishes
Jonathan

Dear All,
As Jonathan has said, as the result of looking at the conformance requirements for groups I have a couple of comments/clarifications.  (Sorry if these points have already been discussed and I've missed it.)

With the introduction of variables/dimensions being referenced as paths this obviously impacts many areas of the convention, but does the groups proposal expect that these all may contain out-of-group references? E.g. cell boundaries, cell measures, etc

If paths are allowed in all areas of the convention it would be good to have a general statement that where "variable" or "dimension" is used this can either be a single variable name or a path to a variable.  Perhaps we could also add path to the definitions in chapter 1?  So that we don't have to define the term specifically in the conformance document.

Although the CF standard doesn't standardize group names, which I understand is due to the abundance of NASA datasets that wouldn't meet this, I wonder whether it might be good to recommend that group names follow the same naming conventions as variables, dimensions and attributes (ie. begin with a letter and be composed of letters, digits and underscores).  This way we would be encouraging human readable names but still allowing them not to be if required by some organizations.

Best Regards,
Ros.

Although the CF standard doesn't standardize group names, which I understand is due to the abundance of NASA datasets that wouldn't meet this, I wonder whether it might be good to recommend that group names follow the same naming conventions as variables, dimensions and attributes (ie. begin with a letter and be composed of letters, digits and underscores).  This way we would be encouraging human readable names but still allowing them not to be if required by some organizations.

Yes. The NASA Dataset Interoperability Recommendations for Earth Science: Part 2 recommends exactly that.

With the introduction of variables/dimensions being referenced as paths this obviously impacts many areas of the convention, but does the groups proposal expect that these all may contain out-of-group references? E.g. cell boundaries, cell measures, etc

Yes. Cell boundaries, cell measures, ancillary_variables, climatology, formula_terms, etc. may reference out-of-group (OOG) variables with the same restrictions that apply to coordinates (which are the focus of the OOG portion of the Groups documentation). Since OOG coordinates are allowed, we must allow OOG bounds variables, for example, and the reasoning for the whole menagerie like anvillary_variables, climatology, formula_terms, etc. follows suit.

If paths are allowed in all areas of the convention it would be good to have a general statement that where "variable" or "dimension" is used this can either be a single variable name or a path to a variable. Perhaps we could also add path to the definitions in chapter 1? So that we don't have to define the term specifically in the conformance document.

Agreed.

@erget -- can we identify a moderator for this? I haven't been following this closely enough to play that roll unfortunately. Maybe @JonathanGregory?

Jonathan moderated (very helpfully and patiently!). I'm closing this as the PR has been merged.


Vendor

Add vendoring using govendor. Removes make updatedeps from Makefile, Travis config and Readme.
Hopefully prevents future issues by users having problems with updatedeps 🎈 Should take care of #46 .

Sorry for the slow merge here. I wanted to read up on govendor and how it worked. (Internally at my day job we have our own setup for vendoring and I haven't experienced the open source tooling. This one looks pretty slick though.)

Interesting. I'm guessing the custom tooling you use at Dropbox is the result adopting Go before open source vendor tooling became mature enough?

Yeah we adopted Go in our infrastructure way back in the 1.1/1.2 days. We realized we'd want static dependencies for production, so we created an internal vendor branch basically and we put dependencies there.