Reference UGRID conventions in CF

As discussed in Trac ticket 171 we would like to associate a specific version of UGRID with each version of CF.
We propose to simply add a section 1.5 to the Conventions Document called "Relationship to the UGRID Conventions" which would say:

UGRID is a convention for unstructured (e.g. triangular) grids that supplements the CF Conventions, including specification of grid topology and location of data on grid elements.   Each version of CF is associated with a particular version of UGRID through the Conventions attribute in 2.6.1.

Then in Section 2.6.1, modify the beginning to read:

We recommend that netCDF files that follow these conventions indicate this by setting the NUG defined global attribute Conventions to the string value "CF-1.8" which also implies "UGRID-1.0".

I think that the biggest issue concerning versioning is that every enhancement to CF needs to be checked for UGRID inconsistencies, and every enhancement to UGRID needs to be checked for CF inconsistencies.
How much a proposer of a change needs to know about this is up for discussion.
For the CF data model, I have suggested (cf-convention/cf-convention.github.io@a170505) that the proposer does not need to know about data model issues - the default someone will look at the proposal on their behalf and decide if any action needs to be taken. Action could mean changing the proposal or changing the data model. This approach could also work for how CF changes  might affect UGRID, and vice versa, but there might be alternative approaches that are better suited to the UGRID (and CF data model) case - I would welcome any thoughts on this.
For the most part, I imagine the no action would be required - all 28 tickets that contributed to CF-1.7 had no impact on the CF data model.
That said, there are two issues for CF-1.8 that will require changes to the data model - geometries and UGRID. These changes have been worked out and are backwards compatible (more on that another time), so all is well! Such structurally challenging changes are unusual, though.
Thanks, David

Hello,
I have been looking at the Finite Element based CF proposal for Unstructured Grid data model (https://publicwiki.deltares.nl/display/NETCDF/Finite+Element+based+CF+proposal+for+Unstructured+Grid+data+model) which was written up some time ago by Bert.  This proposes an encoding for the information required for a consistent spatial interpretation of the values.
Given that UGRID is going to be incorporated into CF [1], I was looking at this with to see if backward incompatible changes could occur if this new proposal (or something like it) become part of UGRID at a later date.
[1] it seems like this will indeed happen, once the management side is sorted out ...
My conclusion from a quick read of Bert's document was that adding the "Function Space" capability, would not impact on files encoded using UGRID 1.0 - which would be good. However, the proposal does suggest renaming certain special attributes (e.g.  face_node_connectivity becomes element_vertex_connectivity). This could be bad for CF backwards compatibility, but, I presume, is easily avoided with a little thought at this stage.
Does all this sound like a a reasonable assessment?
Thanks,
David

I think that this is essentially correct. The naming issues are also not insurmountable. Finite element has its own conventions, but they are just naming conventions. If UGRID has finite difference naming then this will merely be a bit confusing for finite element users.
In the specific cases above, there are two relevant differences to note:

Vertex vs node. Node means something quite different in finite element (a node is a basis function in the dual space to the finite element space), which is why finite element codes usually talk about vertices when discussing the mesh topology.

face vs element or cell. node, edge, face, volume are names for mesh entities of a given dimension. Finite element is often more concerned with codimension, which counts downwards from the mesh dimension. A cell (or element) is an entity of codimension 0, i.e. an entity of maximal dimension. On a 3D mesh the cells are volumes and on a 2D mesh they are faces (cell is also defined for 2D or 1D meshes). A facet is an entity of codimension 1. Facets form the boundaries between cells. On a 3D mesh, the facets are faces while on a 2D mesh they are edges, and on a 1D mesh they are nodes. Given that a UGRID mesh knows its dimension, it is possible for software to identify the cells or facets so the difference in naming convention is not so significant.

Hello,
Some colleagues were asking after the status of this proposal. As far as I'm aware, there are no outstanding objections other than the requirement to spell out some rules for the co-management of the two conventions: CF and UGRID.
The CF data model has now been accepted, and the rules for its management will be in CF-1.9. I think there are some similarities between the requirements rules for evolving the data model, and for evolving UGRID.
I am happy to draft some rules for UGRID, if that helps to get the ball rolling.
Thanks,
David

Yes, please. That would be most helpful. Jonathan

Here are my proposed additions to https://github.com/cf-convention/cf-convention.github.io/blob/master/rules.md. I probably haven't quite got it yet, but it's a start.
A key thing to note is the second line: "The assessment will be carried out by a member of the conventions committee or another suitably qualified person. If no-one volunteers, the chairman of the committee will ask someone to do it."
What this means is that every enhancement proposal must be "signed off" for any UGRID issues.  Almost always this will be a trivial task (e.g. the introduction a new grid mapping attribute parameter - no problem!), but it needs to be done.
For simple cases, as in the example just mentioned, someone who is familiar (rather than expert) with UGRID could take care of this, but for more complicated proposals, the opinion of an expert from the UGRID community must be available.
The first and last sentences covers how to increment the version of UGRID that is acceptable to a give version of CF.

Additional rules relating to the UGRID conventions
All new proposals will be assessed to see if the new features defined in the proposal are compatible with the named version of UGRID that is defined for the current version of the CF conventions.
The assessment will be carried out by a member of the conventions committee or another suitably qualified person. If no-one volunteers, the chairman of the committee will ask someone to do it.
If the proposal is deemed to be not compatible with UGRID in some way, then an attempt must be made to modify the proposal so that its new features are compatible with UGRID, and in such a way that the proposal's intent is not compromised.
If the proposal cannot be acceptably modified to conform to the UGRID conventions, then UGRID will need to be modified to accommodate the new features. If UGRID is extended or generalized in some way that allows the new features but does not affect its existing structure and functionality, the proposal is considered backwards compatible. This is the preferred solution.
Any such changes to UGRID must be defined in general terms, and preferably with a detailed description of the UGRID alterations. However, to facilitate the progress of a proposal that requires UGRID changes, it is sufficient for the general nature of the UGRID modifications to be identified, on the understanding that the UGRID conventions will be updated in detail at a later date, possibly after the proposal has been accepted in all other aspects. Final acceptance will always rely on the completion of changes to the UGRID conventions, which is at the discretion of the UGRID community.
The UGRID conventions exist independently from CF and have their own repository and governance. Therefore the acceptance of a new version of UGRID, whether it arises from a change to CF or from an independent change to UGRID itself, must be raised and discussed in its own enhancement proposal in the usual manner. It follows that a change to CF that requires a change to UGRID will be associated with two GitHub issues - one for the change to CF and one for accepting the new version of UGRID.

In general, I'm not a huge fan of tight coupling but I don't have objections to the thrust of these changes - they aren't very onerous and won't force us to increment if we have no reason to.
The specifics make me slightly more nervous. My understanding is that, if proposed changes to CF are not compatible with the currently referenced version of UGRID,

then UGRID will need to be modified to accommodate the new features

via a proposal to UGRID, but

to facilitate the progress of a proposal that requires UGRID changes, it is sufficient for the general nature of the UGRID modifications to be identified, on the understanding that the UGRID conventions will be updated in detail at a later date, possibly after the proposal has been accepted in all other aspects. Final acceptance will always rely on the completion of changes to the UGRID conventions, which is at the discretion of the UGRID community.

Emphasis is my own.
If this is the case, does that mean that we would accept the proposal formally from CF's side when we have verified that a proposal to UGRID has been made, and then simply wait to merge the CF proposal until the UGRID proposal has been accepted? The danger is that this might take a long time, and then the CF baseline might have drifted, so that the original CF proposal would need revisiting. In all cases I think that this would have to be checked again to ensure that we don't introduce inconsistencies into the Conventions.
Alternatively we could merge immediately in the hopes that the UGRID proposal would be accepted - a bit too optimistic for my taste though and the risk would be that the two standards diverge.

Hi @erget,
Thanks for highlighting these concerns. I agree with them. I guess I was thinking that the UGRID community is more integrated into the CF community than, say, the CRS-WKT community. In the latter example, we don't claim any influence on the "other" community, don't specify a CRS-WKT version, and have carefully instructed the user of a dataset on what to expect.
I checked back to Trac ticket 171, and there is not really an in depth discussion there, nor is there here other than on these governance rules. These concerns were discussed very carefully for the CRS-WKT case, and should be given more thought here, as well.
The CRS-WKT case is simpler, as it is more self contained. There are, however, many aspects of CF which, if changed, could affect UGRID. Perhaps the recent proposal for a domain variable could be one, for example.
It would be great to hear from some folks who work on UGRID.
I would like to see the two conventions evolve simultaneously, but also do not want to see the delay of new CF features that are needed by user communities that have no interest in UGRID.
It should be noted that if UGRID was formally moved into CF (e.g. as a new chapter 10), then all of this governance stuff goes away. I don't know if this has been discussed as an option elsewhere, but it should be stated in this issue why that's not desirable (if that is indeed the case).
Thanks,
David

It would be great to hear from some folks who work on UGRID.

I see that the UGRID conventions GitHub repository has not been updated for 2 years, and the version being recommended for CF is UGRID 1.0, which was released 4 1/2 years ago and is also the latest version. At this time CF was at CF-1.6. This is absolutely not a criticism (CF has gone for longer periods in the past with no readily available signs of advancement, although progress was always going on), but we do need to be sure that UGRID 1.0 is compatible with the draft of CF-1.9. Has anyone looked at this?

The original proposal suggests

We recommend that netCDF files that follow these conventions indicate this by setting the NUG defined global attribute Conventions to the string value "CF-1.8" which also implies "UGRID-1.0".

This says to me that the proposal is functionally equivalent to UGRID 1.0 being incorporated into CF in a new chapter that describes the UGRID that CF recognizes.
It follows, I think, that CF checkers will be expected to check the UGRID conventions (unlike CRS-WKT, which does not need to be checked). Therefore there is a need for conformance rules and recommendations that cover UGRID in relation to CF. Does UGRID already have it's own conformance rules?
For example, when  a data variable has mesh topology, it would need to be stated unambiguously whether or not the coordinates attribute is mandatory, and when it is present whether or not it should always contain the coordinate variables implied by the values of  the mesh, location, and location_index_set data variable attributes.
I do very much support UGRID being a part of CF, but think there are some important structural details that need to be worked out first.
At the current time, having thought about all this again, I wonder if the best two options are:

Incorporate UGRID into CF into as a new chapter.

If that is not acceptable, simply drop this issue, and if people want to use UGRID they should say so in the Convenions attribute "CF-1.9 UGRID-1.0". This would be the status quo approach, I think. In this case it would be up to the UGRID developers alone to ensure consistency with CF.

Thanks,
David

Dear @davidhassell
The proposal was made because the UGRID developers at that time decided that they wanted UGRID to become part of CF. Regarding UGRID as part of CF, even though not in the same document, means that new developments in CF are obliged to consider UGRID, as well as the reverse. For both conventions it has both costs and benefits. However it was proposed not to incorporate it as a new chapter in the CF document because that would require reworking and rehosting their document, which would take substantial effort. Could we frame the rules as though UGRID was in effect a chapter of CF? We could even insert a chapter in CF that says "Chapter N, UGRID. See UGRID document at URL".
I agree with you that the conformance document needs rules relating to UGRID. That is necessary to making it part of CF. I also agree that it would be very helpful to hear from the UGRID developers and users about what they now think is the best way to proceed.
Best wishes
Jonathan

Dear David, Jonathan,
Thank you for moving this forward. In the very beginning we were thinking to move the definiiton for unstructured grids into the CF conventions, later the discussion moved more in the direction of a separate but affiliated convention, and in the end the discussion seems to have settled on a separately documented but linked convention. Since documentation for the CF conventions has shifted to GitHub as well I don't know whether the effort of porting the documents over to the CF repository is still a major issue, but there is a bit of a difference in documentation style. Since UGRID has received quite some uptake already, it may be advantagious to keep the separate identy whereas being more integrated into CF may increase the uptake in the wider OGC and GIS domains ... although via MDAL support (closely linked to CF) we have already made a good step forward in that direction. By keeping the conventions separate it helps to lower the threshold for implementing either convention. The coin can still flip either way for me, but the concept of modularity introduces a slight preference for keeping the conventions somewhat separate. Many of the core developers of the UGRID conventions are well embedded in the CF conventions, so I think it would be quite acceptable to set out some formal rules as you describe to assure that the two conventions develop in a mutually compatible way.
The UGRID 1.0 convention has indeed remained quite stable, the main reason for this is that we already had quite some discussion and uptake before we released version 1.0 ... and ... conventions for specific topics can remain stable for much longer periods than the CF conventions that cover a wide range of features. I don't know of any recent changes in CF that would make UGRID less compatible with 1.9 than with 1.6. We use it ourselves in combination with 1.8 on a daily basis. The geometries are the only development closely related to UGRID and at some time it used a very similar definition format, but given the considered use cases it made more sense there to drop the concept of shared nodes in favour of simplicity and consistency with other storage formats for GIS feature sets. At Deltares we have a draft extension to allow the 1D discretisations within UGRID to be defined on a curved 1D space ... for this extension we build on those geometries introduced in CF 1.8 to describe the shape of the space (i.e. river) before descretisation. So, we try to adopt new CF features as much as possible.
There is one element in UGRID that is not fully compatible with CF in any version and that's the use of the attribute "cf_role" to identify the "mesh_topology" and various types of connectivity variables such as the "face_node_connectivity". The name of this attribute was chosen when it was anticipated that UGRID was to move into CF, but if remains a separate convention we may have to reconsider this attribute name ... or CF would have to formally permit this type of use.
The question about a UGRID checker has popped up twice or so in discussions I had regarding UGRID over the last year. We haven't implemented a formal UGRID checker, but it would indeed be useful to have.
Best regards,
Bert

Dear Bert,
Thank  you for describing all of the history that has occurred here - it really is very helpful, particularly the interactions you have had on the geometries front.
A summary of my position would be that I would support UGRID being moved into CF ("chapter 10"), but if this is not possible then I think that we would still be able to find a way to make things work satisfactorily.
Governance
If UGRID were incorporated into the CF text, there is no governance issue. It's an issue that only arises if it lives outside.
If UGRID lived outside, I genuinely do not expect any problems in the CF/UGRID working relationship, for the same reasons described by @hrajagers, but the part of the  point of the governance rules is to prevent problems arising, however unlikely. We would need to cover situations thorny situations, such those raised by @erget.
One of my own concerns with UGRID not being inside CF is its general invisibility to users. The proposed introductory text is very short and will surely not result in most users seeking out and reading the full UGRID conventions. For instance, what would happen if a user applied mesh and location attributes to a non-UGRID data variable in good faith, having checked in Appendix A that these attributes are not standardised? The UGRID-aware CF checker might tell them their dataset is broken. OK, they could then rename the attributes with this new-found knowledge, but they need to be able to ascertain this  before creating datasets, and I don't think that this is easy enough were UGRID to exist elsewhere.
cf_role
This comes down to variable identification, clearly. From a CF perspective, we know that a data variable employs UGRID    because  it has location_index_set, or both mesh and location, attributes. One of these attributes identifies the relevant mesh  container variable, that in turn identifies other required variables (such as edge node connectivity variables). I don't need cf_role to make any of these connections, so the use of cf_role comprises a redundancy.
So, if we were starting ab initio, I would argue for dropping the cf_role attributes altogether. But we are not! So I can see there is an argument for keeping it for backwards UGRID-compatibility, i.e. extending the use of cf_role to include UGRID (as well as DSGs). Or perhaps it could be dropped without problems?
What do others think?
Conformance
I presume from what you say that there are no conformance rules (rather than there are rules but no existing software)? Is that right. We only need the rules to get UGRID (in any form) into CF. If it were helpful, I would be happy to draft some CF-style conformance rules.

Dear all
@davidhassell makes a good point about the attributes. If UGRID is to be regarded as part of CF (whether within the document or as linked document with a consistent version) it would make sense for UGRID's attributes to be included in Appendix A, or listed in a separate Appendix (like the grid-mapping ones are) since they aren't general-purpose attributes. As David also said, its requirements should appear as a section of the CF conformance document. Also any important terms which it needs to define could be added to section 1.2, and throughout the CF document any relevant references to UGRID should be inserted. These things would naturally be done if the main UGRID description were included in the CF document, and would help with visibility and consistency to do them in any case. If UGRID isn't completely moved into the CF document as a new chapter, I think it would be worthwhile adding a subsection to describe it. That could be in section 1, like the subsection we have there about COARDS.
I agree with David that if the presence or function of UGRID variables can be identified by the presence of particular attributes, cf_role isn't needed. It's redundant and therefore could cause inconsistency. A possible approach would be to deprecate it, which means the CF checker (when made UGRID-aware) would emit a warning if cf_role was included in these roles. The checker should also give an error if cf_role is present and wrongly used - that would be a consistency rule that would appear in the conformance document.
Best wishes
Jonathan

Dear @davidhassell and @JonathanGregory,
I'm not afraid of UGRID getting too little traction as it has already received quite some uptake outside CF, but always formulated as closely linked to CF. However, it would definitely be good to discuss the different use cases for pure CF and CF extended with UGRID somewhere clearly in the CF document.
Governance
I agree on the need for governance documents when UGRID wouldn't be integrated. Regarding the comment made by @erget, I do understand the need for the statement "Final acceptance will always rely on the completion of changes to the UGRID conventions, which is at the discretion of the UGRID community." since one community can never decide what another community must do, but I also agree that your progress shouldn't be limited by another slower community. That's exactly the same reason -- but looking from the other side -- why some may prefer to keep UGRID as a lean and mean separate convention instead of moving it into the bigger CF community. The CF community can always decide at any time to take all the UGRID ideas and merge them into the CF docs (the reverse is extremely unlikely), but my expectation is that as long as the main contributers for the two conventions don't change significantly we'll be able to work out a shared path forward. We can ask developers from the other convention to evaluate proposed modifications if we expect that such changes may cause incompatibilities. If such incompatibilities are detected only in hindsight due to misjudgement then it's in our shared interest and collaboration to resolve them as soon as possible -- most likely by the community that caused the incompatibility in the first place, but the two communities may decide otherwise in collaboration. If desirable changes in one convention would require changes in the other convention, then we can propose such changes to the other community and discuss them together as part of the shared interests. The second community may accept those adjustments, or propose an alternative. If none of the alternatives is acceptable for both communities ... then it would be time to merge the last set of accepted UGRID conventions into CF. Someone else may be able to capture this approach in a more formal way.
cf_role
Regarding the usage of cf_role I agree that the face_node_connectivity role and similar connectivity roles could be dropped since their purpose and meaning should be clear from the corresponding attribute names pointing to them, and most UGRID implementations will probably not check them anyway. However, I'm hesitant about dropping the role mesh_topology since I expect that this will break most if not all current UGRID readers. Your reasoning that one could follow the mesh attributes on the data variables to identify the mesh(es) in the file works in many cases, but wouldn't work if there is only a mesh in the file and no data variable (except for the node, edge and face coordinates) which may or may not refer back to the mesh container variable depending on your philosophy (UGRID doesn't specify whether that is allowed, required or optional). In regular CF the mesh is also only implied by the auxiliary coordinates listed on the data variable (or the dimensions used), so that pathway would indeed be consitent but we do have use cases with only mesh and node coordinates on the file. If we must drop the use of cf_role for CF compatibility, I would be in favour of introducing a new attribute called mesh_type or grid_type to replace it ... the only allowed value would initially be ugrid, but ...
sgrid
... it could be extended with sgrid for staggered structured meshes (see decription of the SGRID conventions). The style of attribute was copied from the geometry_type attribute introduced for the geometry container variable.
conformance
Reading through the UGRID conventions document, I realize that we have sometimes used the word "should" when we actually intended "must", but overall we tried to be very explicit about required and optional attributes so writing down the conformance requirements in a form consistent with these CF pages should be fairly straightforward.
Best regards,
Bert

Dear Bert,
Keeping UGRID separate clearly has some advantages, as you describe. In addition (and I think that you implied this), CF could always stick to an older version of UGRID, if newer features are not to its liking. The "nuclear" option, of merging the last set of accepted UGRID conventions into CF if resolution can not be reached, is a good backstop.
Assuming that UGRID were to remain separate, I think that the governance framework that you describe would work well - thanks.
So with my CF hat on I would prefer incorporation, but I'm fine with UGRID being separate if we can work out how to deal with any structural items that have been mentioned, such as the issue of how to let the rest of CF know about attributes reserved for UGRID.
Another area of possible friction is highlighted by the use of datasets with meshes but no data. Right now, storing a mesh without a data variable is not allowed - there's no encoding for it, and (more pertinently) it is not allowed by the CF data model. However, help is at hand with the proposed  introduction of a new "domain" variable (#301), that will allow domains (meshes) to exist on their own. Would UGRID want to change to adopt the domain variable approach, or could we allow a special case for UGRID meshes?
Thanks,
David

Dear Bert, Jonathan, and all,
I would like to try to summarize the ideas that have been discussed in the form of some broad proposals that I hope could be acceptable to allow us conclude this issue. I welcome your feedback.
In no particular order:
(A) The governance is written up along the lines of @hrajagers ideas: https://github.com/cf-convention/cf-conventions/issues/153#issuecomment-703858946
(B) Comprehensive conformance rules are written up for UGRID. These should be maintained alongside UGRID in its repository, and referenced from (not copied into) the CF conformance document.
(C) Update the aforementioned CF Appendix A to include the relevant UGRID attributes, thereby making them visible to all users. Mention in the governance rules that this table needs maintaining.
(D) Based on @hrajagers previous comment, dropping the standardisation of cf_role on the "connectivity" variables, but retaining it on the mesh topology variable. This is related to my previous comments about the use of datasets with meshes but no data, which I now withdraw. A mesh topology variable can actually contain multiple domains in the CF-sense, one of which can "picked out" by a data variable. This makes it sufficiently different, I realise, to the proposed CF domain variable (#301) that we shouldn't to unify them at this time.
(E) Add some text to CF 5.8 (Domain Variables) (currently being proposed in #301) to explain the UGRID mesh topology variable and how it relates to a domain variable. It may the case that the occasional note relating to UGRID would be useful in other sections. I don't propose to review for these, but they could always be added as when it was felt to be useful.
Thanks,
David

A note on the CF data model:
I think that  UGRID need not affect the CF data model at this time.
This is because CF does not currently formalise connections between data variables, on the same or different domains. A mesh topology variable collates multiple domains (one for faces, one for edges, etc.), but a given data variable only refers to one of them (e.g.   data:location = "face" ;). How you relate a "face" data variable to an "edge" one is moot when you abstract out the netCDF encoding - you get to the same place if you do it by inspection of the coordinate values, or by inspection of the mesh topology and data variable attributes.
I realise that you could say that the point of UGRID is to make these relations explicit, but if that is to be the case then it should be propagated to other areas of CF (e.g. as SGRID proposes), and so should be considered in the round at a later stage.
Does sound reasonable?
Thanks,
David

Thanks to everyone who is working through this important issues.
I strongly support incorporating SGRID into this same framework.

Dear @davidhassell
Thanks for this summary. I agree with (A), (B) and (E) as given.
Re (C), I would suggest that only the UGRID attributes which can appear on data variables should be added to CF Appendix A. If I have read the document correctly, these are mesh, location and location_index_set. The other attributes belong to mesh variables. To prevent accidental collision with CF, it would nonetheless be useful to tabulate them, but I'd suggest we put them in a new CF appendix specifically about the UGRID mesh topology variable, consisting of the table with an introductory sentence or two. This would be like the treatment of the attributes of the grid mapping variable, which are in a table in Appendix F, not in Appendix A. I note that the geometry variable attributes appear in Appendix A, but there are only five of them, whereas there are 18 attributes of the mesh topology variable.
Re (D), if Bert @hrajagers and UGRID colleagues are OK with dropping cf_role for connectivity variables, that's good. They could continue to be allowed but deprecated, rather than disallowed. That's a decision to be made when the conformance rules (B) are written. It seems to me that cf_role is also redundant on the mesh topology variable, because it must also have a topology_dimension attribute, it seems. Couldn't that be used as the defining characteristic of a mesh topology variable? If so, this cf_role could also be deprecated; it can't be disallowed if current software depends on it, as Bert says.
(F) I proposed earlier that we could add a short subsection of CF section 1 to introduce UGRID and its purpose, to make clear its special synergy with CF, to remark on the appearance of attributes in CF appendices, and to say that it has its own conformance document which complements the CF conformance document. What do you think?
I agree with you that the relationship between domains of different data variables is not currently considered in the CF data model, but not inconsistent with the data model. If UGRID is not being included in CF, we don't have to consider it at the moment.
Regarding Ryan @rabernat's comment, I think it would be fine to consider SGRID as well, but let's do it as a separate issue, and perhaps after UGRID, because we may not have enough mental capacity to deal with both at once.
Best wishes
Jonathan

Dear @JonathanGregory,
Thanks for these comments
I agree with your updated (C).
I'm also fine in (D) with a mesh topology variable getting its canonical identity from the  topology_dimension

(sorry - pressed send too early - I'll try again!)
Dear @JonathanGregory,
Thanks for these comments
I agree with your updated (C).
I'm also fine in (D) with a mesh topology variable getting its canonical identity from the topology_dimension attribute (which is similar in principle to how a domain variable is to be identified). However, it would be good to here from @hrajagers if making this cf_role use optional is OK.
I agree with (F).

If UGRID is not being included in CF, we don't have to consider it at the moment.

I'm not sure what you mean here. Is it "Even though UGRID is being included in CF, we don't have to consider the relationship between domains of different data variables at the moment.". Apologies if I have misunderstood.
Also on the data model, I'd just like to highlight that the integer-valued  interconnectivity variables (e.g. "edge_node_connectivity") need not feature in the data model, for the same reasons that a list variable for compression by gathering does appear in the data model.
I agree that the issue of SGRID, and interconnected domain/data variables in general, is a conversation for another issue; and now that we are starting to  formalise the storage of domains as independent entities, this is a good moment to reignite this conversation.
All the best,
David

Dear @davidhassell

Is it "Even though UGRID is being included in CF, we don't have to consider the relationship between domains of different data variables at the moment."

Yes, that is what I meant. By "not being included in CF" I meant "in the CF document". Sorry to be unclear.
Thanks
Jonathan

Dear @JonathanGregory and @hrajagers,
Here is a synthesis current proposal. How does it sound?
(A) The governance is written up along the lines of @hrajagers ideas: #153 (comment)
(B) Comprehensive conformance rules are written up for UGRID. These should be maintained alongside UGRID in its repository, and referenced from (not copied into) the CF conformance document.
(C) Document the standardised UGRID attributes in the CF conventions, thereby making them visible to all users. Mention in the governance rules that they need maintaining. Only the UGRID attributes which can appear on data variables (mesh, location and location_index_sets should be added to CF Appendix A. All other attributes UGRID attributes (such as those on mesh variables) should go into a new CF appendix specifically about the UGRID mesh topology variable, consisting of the table with an introductory sentence or two. This would be like the treatment of the attributes of the grid mapping variable, which are in a table in Appendix F, not in Appendix A. Note that the geometry variable attributes appear in Appendix A, but there are only five of them, whereas there are 18 attributes of the mesh topology variable.
(D) Deprecate the  cf_role attribute for connectivity variables (but retaining it on the mesh topology variable). If and only if @hrajagers and UGRID colleagues are in agreement, also deprecate the  cf_role attribute mesh variables, deferring to the presence of the topology_dimension attribute for identification. This latter suggestion will make the mesh variable "more CF-like", but that is not worth at this time if the expense is a lot of broken existing software.
(E) Add some text to CF 5.8 (Domain Variables) (now accepted for CF-1.9) to explain the UGRID mesh topology variable and how it relates to a domain variable. It may the case that the occasional note relating to UGRID would be useful in other sections. I don't propose to review for these, but they could always be added as when it was felt to be useful.
(F) Add a short subsection of CF section 1 to introduce UGRID and its purpose, to make clear its special synergy with CF, to remark on the appearance of attributes in CF appendices, and to say that it has its own conformance document which complements the CF conformance document.
Thanks,
David

Thank you, @davidhassell. I think this is fine.

Thank you, @davidhassell . Looking good to me as well. As shown above on the discussion page, I have posted in the UGRID forum (as well as the SGRID forum following this comment) a request for feedback on this proposal. I'll also reach out to a number of people offline.

Hello,
Following on from some discussions that have been taking place on the UGRID issue tracker, a couple more items to the proposal are needed (G and H). Here is the new set:
(A) The governance is written up along the lines of @hrajagers ideas: https://github.com/cf-convention/cf-conventions/issues/153#issuecomment-703858946
(B) Comprehensive conformance rules are written up for UGRID. These should be maintained alongside UGRID in its repository, and referenced from (not copied into) the CF conformance document.
(C) Document the standardised UGRID attributes in the CF conventions, thereby making them visible to all users. Mention in the governance rules that they need maintaining. Only the UGRID attributes which can appear on data variables (mesh, location and location_index_sets should be added to CF Appendix A. All other attributes UGRID attributes (such as those on mesh variables) should go into a new CF appendix specifically about the UGRID mesh topology variable, consisting of the table with an introductory sentence or two. This would be like the treatment of the attributes of the grid mapping variable, which are in a table in Appendix F, not in Appendix A. Note that the geometry variable attributes appear in Appendix A, but there are only five of them, whereas there are 18 attributes of the mesh topology variable.
(D) Deprecate the cf_role attribute for connectivity variables (but retaining it on the mesh topology variable). If and only if @hrajagers and UGRID colleagues are in agreement, also deprecate the cf_role attribute mesh variables, deferring to the presence of the topology_dimension attribute for identification. This latter suggestion will make the mesh variable "more CF-like", but that is not worth at this time if the expense is a lot of broken existing software.
(E) Add some text to CF 5.8 (Domain Variables) (now accepted for CF-1.9) to explain the UGRID mesh topology variable and how it relates to a domain variable. It may the case that the occasional note relating to UGRID would be useful in other sections. I don't propose to review for these, but they could always be added as when it was felt to be useful.
(F) Add a short subsection of CF section 1 to introduce UGRID and its purpose, to make clear its special synergy with CF, to remark on the appearance of attributes in CF appendices, and to say that it has its own conformance document which complements the CF conformance document.
(G) Further to the discussion on the order in which the corner nodes of a volume are specified (ugrid-conventions/ugrid-conventions/issues/53), the UGRID conventions drops support for the fully 3D meshes (When someone actually needs it, it can reintroduced without the current ambiguities, which will be fine CF.)
(H)  Further to the discussion on implications on the CF data model (ugrid-conventions/ugrid-conventions/issues/52), the CF data model needs to be updated to allow the storage of topological connections  between cells ("cells" in the CF data model sense). It is not necessary at this stage for the connectivity between cell elements (such as the edge of a face) to be a part of the CF data model.
Thanks,
David

This is fine, thanks. If we agree, I suppose that we need to write propose some textual changes in the CF documents for some of these points.

Hello,
In starting to think about:

(H) Further to the discussion on implications on the CF data model (ugrid-conventions/ugrid-conventions/issues/52), the CF data model needs to be updated to allow the storage of topological connections between cells ("cells" in the CF data model sense). It is not necessary at this stage for the connectivity between cell elements (such as the edge of a face) to be a part of the CF data model.

For the one case that I thought was not possible in the current CF data model, i.e. a 1D network topology,

I now realise the current data model is OK. I was thinking that in this case the CF domain cells were the nodes, but really a cell is an edge  bounded by two nodes. In this view, the existing CF framework suffices. Remember that the CF data model is independent of the netCDF encoding, including the UGRID. It doesn't matter if the CF data model view of the world looks quite different to that of the UGRID view, as long as it is possible to unambiguously map between the two.
Thanks,
David

I agree that a cell which is an edge bounded by two nodes is fine in the CF data model, yes. (H) is correct that CF doesn't explicitly recognise connectivity, although you could infer it from coincidence - isn't that right? Jonathan

Dear Jonathan,
That's right. I should have made that clearer, so many thanks for pointing it out!
CF provides cell connectivity by inspection of coincident (or possibly overlapping) bounds.
UGRID provides an index based encoding for making the connectivity easier to find in many circumstances, but the CF data model does not need to copy this. Not including these connectivity indices in the CF data model in no way prevents a software application from managing, and utilising, them "alongside" the data model representation.

Note that not having explicit connectivity creates a (potentially) rather large computational overhead for the reconstruction of it. This will be exacerbated with more complicated grids in the future, think time-dependent unstructured grids, perhaps with regionally varying timestep.
I think it would be good to provide a standardized way of storing connectivity, though I agree it also makes sense to make it optional, at least for most commonly used classes of grids today.

Dear Klaus,
I hope to allay your concerns by noting that all of the UGRID machinery for storing connectivity will certainly be imported into CF unchanged - it is just that in the logical data model we don't need to make special mention of it. This is because it turns out the connectivity variables are just one (very convenient) way of encoding something in CF that can in principle already be done another way. We call the CF data model a "minimal" logical data model, meaning that we keep it simple by not overlapping the contents of one part of the model with another.
All the best,
David

Hi, sorry to be late to the table with this issue, but I have been listening in here a while, in hopes of understanding better and maybe contributing.
I think that I may have spotted a potential problem with the removal of 'cf_role' :
As stated (above), the role of a mesh-variable is identifiable by its having a 'topology_dimension' attribute, but the same is not true of a location-index-set :
So, if we include an index set in a "mesh only file" (aka "meshfile" or "gridfile" in some quarters), then we will be unable to distinguish it from a data-variable.
This in turn implies that when loading a file, we would need to "know" whether it was intended to be a "mesh file" or a "normal datafile" as you can't determine that by inspection.
( This problem is entirely analagous to an existing problem with auxiliary-coordinates in standard CF :  If the data-variable which references them is removed, then they are not distinguishable from data-variables -- so they "become" variables )
Our context : Here at the UK MetOffice, we are working to support unstructured data within Iris, using UGRID as the template for our internal data-model (as we already do for CF).
Locally, we have a particular interest in the use of location-index-sets, and we are also intending to use "mesh files" i.e. files with only the mesh structure and no data.
Also, our practical experience in tools development and support shows that files that do not fully comply with conventions are, sadly, just not that uncommon even in the respected international archives.
Thus, just as Iris has to somehow handle files with invalid standard-names and units, or mis-specified grid-mappings, so our trial UGRID files suffer from problems like missing optional connectivity links, the odd miss-spelling and so on.
What this tells us is, that the "robustness" of the format is also a important consideration.
From the point of view of a generic code library developer, the unambiguous identification of the 'role' of elements within a file will definitely make writing parsing code more straightforward -- and not least because dealing with incorrect input in a helpful way is an important usability factor (just as it is in compiler design).
So, I must confess that I personally  was preferring the way that UGRID labels each component unambiguously, instead of relying on links from other components to infer the role of a variable.
Solutioneering maybe, but ... could we instead allow the attribute to be named 'ugrid_role', and simultaneously deprecate the older 'cf_role' usage ?

Dear Patrick,
Thank you for bringing up "location index set" variables. I agree that in the absence of the cf_role attribute it is not always possible to distinguish one from data variable, so I would be happy with retaining it on these variables.
By extension, I think that we should drop the suggestion (it is just that at this stage) of removing cf_role from the mesh topology variable. This is because a location index set variable is logically identical to a mash variable, so having a common mechanism of identification would be nice.

( This problem is entirely analogous to an existing problem with auxiliary-coordinates in standard CF : If the data-variable which references them is removed, then they are not distinguishable from data-variables -- so they "become" data-variables )

I wouldn't call this a problem, rather a feature! When we read a dataset, we tend to not cast variables that have been identified as auxiliary coordinates (or other roles) as data variables as well, but that is only a default behaviour that is what most of want most of the time.
All the best,
David

Hello @ChrisBarker-NOAA and all,
After some very illuminating discussion over at https://github.com/ugrid-conventions/ugrid-conventions/issues/52, and some offline discussions, @JonathanGregory and I have come up with some suggested text to the data model in Appendix I that we hope will update the CF data model for the mythical connectivity.
The new text is hopefully self-explanatory and edits the "Domain construct" section of Appendix I (edits in italics) and provides a new section describing a new construct type - the "network topology construct". Here we are using "network topology" in this sense, but there may well be better names.
Remember that the CF data model is a logical data model that is independent of the encoding, so it not a problem that  the connectivity array  does not look like a UGRID netCDF connectivity variable.
(I took the liberty of rearranging some the network topology text after Jonathan last saw this - I take full responsibility for any negative consequences.)

Domain construct
The domain construct (figure 3) describes a domain comprising measurement locations and cell properties. The domain construct is the only metadata construct that may also exist independently of a field construct. The domain construct contains properties to describe the domain (in the same sense as for the field construct) and relates the following metadata constructs

Domain axis constructs.
Dimension coordinate and auxiliary coordinate constructs.
Coordinate reference constructs.
Domain ancillary constructs.
Cell measure constructs.
Network topology constructs.

All of the constructs contained by the domain construct are optional (as indicated by "0.." in figure 3).
In CF-netCDF, domain information is stored either implicitly via data variable attributes (such as coordinates), or explicitly in a domain variable, or a UGRID mesh topology variable. In the latter two cases, the domain exists without reference to a data array.
Network topology construct
A network topology describes the connectivity of cells of domain indexed by a subset of the domain axis constructs. When two cells are connected, operations on the data stored on them may be assumed to be continuous across their common boundary. A network topology construct describes logically and explicitly the network topology of cells indexed by a single domain axis construct. A network topology construct contains a connectivity array that spans a unique domain axis construct with the addition of an extra dimension of the same size, such that each dimension indexes the cells. The array is symmetrical, and each element indicates whether the pair cells to which its indices refer are connected.
A network topology that has no corresponding network topology construct, which includes those for multiple domain axis constructs, is nonetheless implicitly defined by a function of both the physical contiguousness of the cells, and the nature of the real world or simulated processes that produced the data. For example, in a field which contains both land and ocean cells, connections between land and ocean cells might be excluded for some physical processes.The description of such an implicit network topology may require metadata that is external to CF.
In CF-netCDF a network topology can only be provided for a domain defined by a UGRID mesh topology variable. In this case, the connectivity array is supplied by a UGRID connectivity variable (such as a "face_face_connectivity" variable).

@davidhassell:
Thanks! this looks good to me.
Am am curious why you chose "network topology" rather than "mesh topology" -- I"m not sure it matters much but at least that wikipedia article is pretty focused on the topology of communication networks which is pretty similar, but not quite.
On the other hand, "mesh topology" is used for computer networks (https://www.computerhope.com/jargon/m/mesh.htm)
I have no idea if there's a standard term for what we mean in this case, but modeling folks do use teh term "mesh" or "grid" in this context, and I've never heard anyone use the term "network".
Though the GIS folks use "Triangulated Irregular Network" (https://en.wikipedia.org/wiki/Triangulated_irregular_network) -- maybe that's where "network" came from?
The computer graphics folks use: "Triangle mesh" (https://en.wikipedia.org/wiki/Triangle_mesh)
So there's president for that, too.

Hi @ChrisBarker-NOAA,
Great - real progress!
I'll start work on the other aspects of the data model that need to be updated (like the table of construct types, the figures, and a high-level explanation of how UGRID fits in).
Thanks for some context on the choice of name. Whilst I can't remember where the network topology idea came from (presumably from me and/or Jonathan), I think that we wanted to avoid using the term "mesh topology" to differentiate our new construct from a UGRID mesh topology variable. The new network topology data model construct contains only the connectivity, but the mesh topology variable also contains the actual node coordinates, and other items, and map to one or more CF domain constructs.
Assuming that that's a valid argument, any sensible name other than "mesh topology construct" is fine by me.
On the other hand if my reasoning turns out to be specious, I'd be happy with "mesh topology construct"!
"grid topology" is tempting, but right from the start of the data model development many years ago we steered clear of the word "grid" due to its connotations with regularity and structuredness, and neither of those are in play here.
The choice of name is most certainly important, but we don't necessarily have to decide straight away - nothing will be held up if we want to change the name at a later stage.
Thanks,
David

Dear @ChrisBarker-NOAA  and @davidhassell
Would "domain topology" make sense?
I suggest that the penultimate paragraph should begin, "A domain that has no ?? topology construct, including domains with more than one domain axis construct"
Best wishes
Jonathan

Dear @JonathanGregory
I like "domain topology". (I'll carry on using ?? below so as not to normailise yet)

I suggest that the penultimate paragraph should begin, "A domain that has no ?? topology construct, including domains with more than one domain axis construct"

I don't think that's right. A domain has as many ?? topologies as there are domain axis subset combinations. For example, a domain with three domain axes has  six ?? topologies. The three that apply to two or more domain axes are always implicit, and each of the three that apply to only one domain axis is either implicit or else is explicity provided with a ?? topology construct.
A common case will be a domain with two domain axes - one for time and one for UGRID cells - but  there can still be a ?? topology for the UGRID axis.

Dear David
It's a good point that there may be domain axes which aren't involved in the topology. Thanks for the correction. We can supply a ?? topology construct provided it refers to a single domain axis (the UGRID case). I'm not sure there would be more than topology for a domain, although I agree it's conceivable, but I think that's in the realm of speculation, since we don't currently have a use-case for describing the situation with more than one domain axis involved in the topology.
Best wishes
Jonathan

Dear Jonathan,
Thanks. It sounds like we agree that the domain could have multiple topologies (as opposed to topology constructs). It's a good point that we have no use case for two or more topology constructs, each of which applies to a single unique domain axis, and in fact we have no way of encoding it, so that case should indeed be excluded.
I think that it is important to keep the general definition of a domain topology, that could span multiple dimensions, and to note that a domain could have more than one domain topology, even though we only allow one 1-D domain topology construct to be provided. In this way the land-sea example makes sense.
Here's a new attempt. New text in italics, and I'm trying out "domain topology " for size:
Domain topology construct
A domain topology describes the connectivity of domain cells indexed by a subset of the domain axis constructs. When two cells are connected, operations on the data stored on them may be assumed to be continuous across their common boundary. A domain topology construct describes logically and explicitly the domain topology of cells indexed by a single domain axis construct. A domain topology construct contains a connectivity array that spans a single domain axis construct with the addition of an extra dimension of the same size, such that each dimension indexes the cells. The array is symmetrical, and each element indicates whether the pair of cells to which its indices refer are connected. A domain construct may contain at most one domain topology construct.
For any subset of the domain axis constructs, excluding a domain axis construct for which there is a domain topology construct, there is an  implicit domain topology that is defined by a function of the physical contiguousness of the cells, and/or the nature of the real world or simulated processes that produced the data. For example, in a field which contains both land and ocean cells, connections between land and ocean cells might be excluded for some physical processes.The description of such an implicit network topology may require metadata that is external to CF.
In CF-netCDF a network topology can only be provided for a domain defined by a UGRID mesh topology variable. In this case, the connectivity array is supplied by a UGRID connectivity variable (such as a "face_face_connectivity" variable).

Dear @davidhassell
Thanks. That looks fine to me. I wonder what @ChrisBarker-NOAA thinks.
Best wishes
Jonathan

Honestly, I'm a bit lost :-(  so I"ll trust this is clear to sometone that has taken the time to study teh data model more than I have.
But to the extent I understand it, it looks good to me :-)
But two questions:
"In CF-netCDF a network topology can only be provided for a domain defined by a UGRID mesh topology variable"
Didn't you change the terminology so that that would be:
"In CF-netCDF a domain topology can only be provided for a domain defined by a UGRID mesh topology variable"
and:
"The array is symmetrical,..."
I'm not sure I follow -- in what sense is it symmetrical? Does that mean that if cell A connects to cell B then cell B connects to cell A? Which is the case, but I don't know that that means the array that holds the is symmetrical. Perhaps I'm getting it confused with a symmetric matrix, which is something else altogether.

in what sense is it symmetrical?

Just trying to get my head around this, it seems to me that what is meant is : we provide a connectivity array,  which is a square matrix describing the connectivity relation "from cells to cells",
i.e. connectitivy[i,j] == cell[i] adjoins cell[j] where i and j both range over the same cell dimension.
In which case, it is just a symmetric matrix,
meaning exactly that .. 'adjoins' is a symmetiric relation ; or cell[i] adoins cell[j] <--> cell[j] adoins cell[i]
(thus, the array == it's own transpose, which is a definition of symmetry).
I think there are also a few technical aspects that we might want consider pinning down a bit further...

presumably, connectivity values are always boolean (or equivalent).
Perhaps we should state that.
we might also state whether cells are considered to connect "to themselves", as appearing in values of "connectivity[i,i]".
at the least, I would say we don't want that aspect to be variable,
I.E. maybe we state that all "connectivity[i,i]" are always 1/true ; or false ; or state that they have 'no meaning regardless of value'.

in what sense is it symmetrical?

Just trying to get my head around this, it seems to me that what is meant is : we provide a connectivity array, which is a square matrix describing the connectivity relation "from cells to cells",
i.e. connectitivy[i,j] == cell[i] adjoins cell[j] where i and j both range over the same cell dimension.
In which case, it is just a symmetric matrix,
meaning exactly that .. 'adjoins' is a symmetric relation ; or cell[i] adjoins cell[j] <--> cell[j] adjoins cell[i]
(thus, the array == its own transpose, which is a definition of symmetry).

Well, conceptually maybe -- but that's not how anyone encodes the connectivity -- and it would be a huge, very sparse boolean array: number_of_cells X number_of_cells. The connectivity is encoded, in UGRID at least, as a length-number-of-cells array of (cell_number, cell_number, cell_number) triples (for a triangular mesh)
I just looked at the UGRID docs, and oddly, even though the examples are tiny, we didn't seem to put the actual example arrays in, which does make it less clear than it could be :-(

@ChrisBarker-NOAA Well, conceptually maybe -- but that's not how anyone encodes the connectivity

Oh I see!
I was really only thinking in abstract terms, about how why you would describe the validity constraint as a 'symmetry', and not the actual 'face-face-connectivity' arrays or similar.  So, I assumed that was the viewpoint of the more abstract description in the CF datamodel.
Looking at it again, from the viewpoint of actual UGRID connectivity data,
that does make the 'does X connect to X' question seem a rather abstruse and unimportant.
However, the basic symmetry rule is still a relevant requirement : and it is still not implicit in the form of the data.
E.G. ...
Suppose we have a "face-face-connectivity" array "ffc[i,j]",  where i ranges over faces and j over 1..nA, where nA is a maximum adjacency number (i.e. typically 3 or 4).
Then a "no-self-connections" rule is something like for all j in 1..nA ; ffc[i,j] != i
But a "symmetry rule" is rather more involved : something like ...
for all i1 in 1..nF ; for all j1 in 1..nA ;
  EITHER ffc[i1,j1] = missing
  OR exists j2 in 1..nA s.t. ffc[i2, j2] = ffc[i1, j1]

-- obviously, the 'missing' possibility complicates it
There are also some other possible rules you might want to state here, such as
for any i, all the ffc[i,j] should be different (except for missing points).
I think that this is maybe showing that the more abstract form of a "binary connections map" does provide a more natural and flexible route for describing the mathematical properties  I'm really no expert, but I think that basic graph theory is written in terms like these.

Looking at it again, from the viewpoint of actual UGRID connectivity data,
that does make the 'does X connect to X' question seem a rather abstruse and unimportant.

?? -- but does X connect to Y is the key concept we are trying to capture here.
From the data model perspective, there needs to be SOME way to define the connectivity. how it's done is a matter of the "encoding", yes?

that basic graph theory is written in terms like these.

I'll take your work for it :-) But in "lay" terms, I might say something like:
"""
The connectivity arrays capture how the individual component of the mesh relate to each other. For example, for the face-face connectivity, each face has N neighbors (where N is the order of the cells, e.g. triangles are N=3). So for each face, the indexes of the three connecting faces is provided, resulting in a num_faces X N array. Note that there is duplication where there is symmetry: that is, if face i connects to face j, face j connects to face i.
"""
For the most trivial example of a two-triangle mesh, you would have, for the face-face connectivity:
[[-1, -1, 1],  # face 0 connects only to face 1
 [-1, -1, 0],  # face 1 connects only to face 0
]

where -1 means "nothing" and the faces are zero indexed.
So that those two array entries have dublicative information, representing the symmetry
NOTE: I realised as I was writing that that the encoding does not have to be duplicative and symmetric. We tend to store it that way because it's easy to ask the question:"what are the neighbors of face i?" -- but we could only store the connections, which would make it harder to interpret, but easy to encode without duplication. The above two-face mesh would simply be:
[[1, 0]]
that is: face one connects to face zero. (and therefor face zero connects to face 1 as well)
In fact, in UGRID, the "edges" connectivity is described exactly like that.

From the data model perspective, there needs to be SOME way to define the connectivity. how it's done is a matter of the "encoding", yes?

Yes, that is what I was "trying" to say !

?? -- but does X connect to Y is the key concept we are trying to capture here.
in "lay" terms, I might say something like: ...

All agreed, absolutely
I was just trying to express that I think the two descriptions are compatible, and in fact basically mean the same thing.
I think the key point is that how UGRID actually organises things (what I termed "encoding") is not the same terminology in which the CF datamodel description is decribed.
So, we do need to be confident that they are equivalent, but that also means exploring the  rules and constraints which are part of the description (on either side of the fence).
If I read correctly what has been said here ...
The CF datamodel only states that the connectivity relation must be symmetrical : "If A connects to B, then B connects to A".
It also describes such relations only on one "domain axis".
My ouststanding concern is that UGRID does define additional information structures (and governing "rules") which the CF 'domain' model has no interest in.
(1) Firstly, a UGRID mesh with multiple locations relates to multiple domain-axes in CF
That can "work", because any data-variable can only reference one of them,  as described :

we have no use case for two or more topology constructs, each of which applies to a single unique domain axis, and in fact we have no way of encoding it, so that case should indeed be excluded.

So, the CF 'topology construct' can only be atttached to a single domain axis of a domain.
That means that UGRID data which maps to a different mesh location is modelled as belonging to a separate, independent "domain".
This seems okay for now, but it means of course that the CF decsription has no concept of the intercoupled nature of the different locations.
(2) likewise, the CF description has no place for the cross-location connectivities (like face-edge-connectivity)
So, I'm not sure where any of that might become practically relevant, but I'm still a little worried that they aren't really describing the same ideas with the same range of possibilities.

So, the CF 'topology construct' can only be atttached to a single domain axis of a domain.
That means that UGRID data which maps to a different mesh location is modelled as belonging to a separate, independent "domain".
This seems okay for now, but it means of course that the CF decsription has no concept of the intercoupled nature of the different locations.

(2) likewise, the CF description has no place for the cross-location connectivities (like face-edge-connectivity)

So, I'm not sure where any of that might become practically relevant, but I'm still a little worried that they aren't really describing the same ideas with the same range of possibilities.

I"m concerned as well. ONe thing that's become clear to me working with these botu unstructured and "staggered" grids in the last few years is that the entire "mesh" or whatever we want to call it, is one "thing". That is, you cannot treat the collection of nodes, the collection of faces, the collection of edges, each as a separate meshes. Sure, data may be associated specifically with only one of these, but, for example:
If values on the nodes ar treated as their own thing, you can plot them fine -- but if you need to interpolate the values, you need to know how the faces are defined.
Values on the edges, similarly, culd be plotted, but and edge is very much an interace between two faces -- it does not live on its own at all.
And boundaries are again, very much properties of the faces.
I'm still not clear what a "domain axis" means in the CF data model, but we really do need to capture the concept that the mesh is not unrelated pieces, it is one thing -- you can't really know what any of the data fully means without the full mesh description.

Hi @ChrisBarker-NOAA and @pp-mo,
I have been away and am just catching up with the conversation. Thank you for an interesting read!

From the data model perspective, there needs to be SOME way to define the connectivity. how it's done is a matter of the "encoding", yes?

Absolutely.
Patrick's description of symmetry (https://github.com/cf-convention/cf-conventions/issues/153#issuecomment-882630042) is right - it is symmetric in the square matrix sense. This is indeed not at all how UGRID actually encodes this information, but the CF data model is independent of the encoding, and the symmetric matrix is logically what is going on here. Whilst there is no expectation that anyone should encode it in this manner, it is tempting (to me!) because the square connectivity matrix can be easily updated in subspacing operations.
It's a good point about what to do on this matrix's diagonal - I think that the option of these values having 'no meaning regardless of value' is sufficient for the data model. Whether you use booleans, integers, strings, etc. to denote connected/not connected is entirely an encoding choice and has no impact on the data model.

we really do need to capture the concept that the mesh is not unrelated pieces, it is one thing -- you can't really know what any of the data fully means without the full mesh description.

(2) likewise, the CF description has no place for the cross-location connectivities (like face-edge-connectivity)

The CF model does connect (e.g.) faces with edges and nodes, but in  a different manner to UGRID. In CF, a "cell" is typically defined as the "space" enclosed by bounds, and the edges of the cell are the connections between adjacent cell bounds. This space may be 0-d, in which case it is just a "node"; or 1-d, in which case it is an "edge" connecting two "nodes"; or 2-d, in which case it is a "face" defined by "edges" and "nodes"; (etc, but we don't generalise to 3-d and beyond, yet). The nodes (i.e. bounds) and edges do not have an independent existence - they are elements of the CF cell definition. The new domain topology construct makes explicit the cell connectivities.

(1) Firstly, a UGRID mesh with multiple locations relates to multiple domain-axes in CF
That can "work", because any data-variable can only reference one of them, as described :

we have no use case for two or more topology constructs, each of which applies to a single unique domain axis, and in fact we have no way of encoding it, so that case should indeed be excluded.

So, the CF 'topology construct' can only be atttached to a single domain axis of a domain.
That means that UGRID data which maps to a different mesh location is modelled as belonging to a separate, independent "domain".
This seems okay for now, but it means of course that the CF decsription has no concept of the intercoupled nature of the different locations.

Yes. A CF field construct contains a domain that is limited to describe just the parent field's data. Although, be careful not to confuse "domain axis" and "domain". A "domain axis" is essentially a dimension of the domain. We restrict the new domain topology constructs to apply to a single domain axis simply because there is no current way of encoding a domain topology construct that applies to multiple domain axes. UGRID only describes a mesh with a 1-d discrete axis. If this ever changed, it would be described by a simple generalisation of the data model text. The CF data model mustn't provide capabilities that are not allowed by the CF conventions.
If there are no data variables in a file - i.e. just the mesh is stored in a dataset - then the entire mesh  definition is captured by a CF domain construct. However, if one or more a data variables are defined on the mesh, then their CF domains are only allowed to be those elements of the mesh that are in use. For example, if a temperature variable is stored on faces and a U-wind is stored at nodes, then the domain of the former will include the UGRID faces, edges and nodes, but the domain of  the latter will only know about the nodes. In both cases, a connectivity matrix will retain the required connectivities.
I just wrote "If there are no data variables in a file - i.e. just the mesh is stored in a dataset - then the entire mesh  definition is captured by a CF domain construct.", however I realise that that's not necessarily the case if there are edge coordinates as part of a mesh with faces. In this case, the edge coordinates can only be represented by the CF data model in a second domain that comprised 1-d cells defining each edge and how it's connected to others. I can't decide right now if this is a problem for the data model (i.e. should edge coordinates be a new feature of coordinate constructs?). The only issue I might have is that the "round trip" of reading a mesh variable into a CF domain construct and then writing it back to disk might not give an exactly comparable result to what you started with, but that isn't a promise of the data model, so perhaps not an issue? @JonathanGregory, it would be interesting to know your thoughts on this.
CF nor UGRID makes the promise that multiple data variables defined by the same mesh variable are guaranteed to be in some way combinable. That is a decision or assumption that has to be made by the user. Therefore, separate domain definitions for each data variable is an appropriate view for the CF data model. This becomes clear when you consider, say, multiple datasets from the same model simulation - each dataset contains the same mesh, but we only know it is the same by inspection or by the promise of non-standardised metadata.
When formal connections between data variables are possible in CF we'll need some carefully thought out extensions to the data model, but that's not something for this discussion (fortunately!).

If values on the nodes are treated as their own thing, you can plot them fine -- but if you need to interpolate the values, you need to know how the faces are defined.

If values are only defined at nodes, then the  cells of the domain are defined by single points with a connectivity array that implicitly  defines the edges and faces, so the data model is storing everything we need for interpolation.

"In CF-netCDF a domain topology can only be provided for a domain defined by a UGRID mesh topology variable"

Yes indeed - thanks for spotting my mistake.
All the best,
David

I think I'm (slowly) getting this, but a few thoughts on language:
"the CF data model is independent of the encoding, and the symmetric matrix is logically what is going on here."
That makes sense but the language I first noticed said a "symmetric array" -- I think we need to be careful about the abstraction of a symmetric matrix and the realization of an "array" in code, or a netcdf file, or ... And it needs to be clear that it's a abstract (sparse) boolean matrix, rather than an actual array (necessarily).
"The nodes (i.e. bounds) and edges do not have an independent existence - they are elements of the CF cell definition. The new domain topology construct makes explicit the cell connectivities."
That makes sense -- that is the whole point after all :-)
" A "domain axis" is essentially a dimension of the domain. We restrict the new domain topology constructs to apply to a single domain axis simply because there is no current way of encoding a domain topology construct that applies to multiple domain axes. UGRID only describes a mesh with a 1-d discrete axis."
Now I'm getting confused again -- I think this is (at least in my head) a result of a fundamental mismatch between orthogonal meshes and unstructured meshes. The whole concept of an "axis"  matches orthogonal coordinates of some sort. (that's kind the definition of orthogonal, yes?). Mapping a Ugrid to the real world, the real world is 2D (Or 3D, but let's not go there yet) -- and that 2D world has 2 orthogonal axis -- X, Y, Lat, Long, whatever. So a mesh represents a 2D space -- but I wouldn't say it has two axes, or a 1-d discrete axis either, ir is simply something else. Or "axis" means something different in this context than I think it does.
"separate domain definitions for each data variable is an appropriate view for the CF data model. "
That means there could be separate definitions, or data variables could share the same one, yes?  If, for instance, multiple variables in the same file are on the same mesh, they share a single domain definition, yes?
Thanks for taking the time to educate me!

Dear Chris and David
Thanks for this discussion.
In David's revised text

Unlike the domain topology construct's connectivity array, a
UGRID connectivity variable's data is not stored as a symmetric matrix that
indicates the connectivity between any two cells. Instead, the trailing
dimension of ...

I think that "unlike" implies it's somehow inconsistent. I would suggest
something like, "The information represented by the symmetrical connectivity
array of the domain topology construct in the CF data model is stored in a
different but equivalent way in UGRID. The trailing dimension of ..."
David wrote, "If there are edge coordinates as part of a mesh with faces ...
the edge coordinates can only be represented by the CF data model in a second
domain that comprised 1-d cells defining each edge and how it's connected to
others." What role do the edge coordinates have in the domain which refers
to the faces?
Best wishes
Jonathan

Dear @JonathanGregory,
Thanks for your comments.

I think that "unlike" implies it's somehow inconsistent ...

Agreed. With your new text, that last paragraph becomes
In CF-netCDF a domain topology can only be provided for a domain defined by a UGRID mesh topology variable. In this case, the connectivity array is supplied by a UGRID connectivity variable, such as a "face_face_connectivity" variable. The information represented by the symmetrical connectivity array of the domain topology construct in the CF data model is stored in a different but equivalent way in UGRID. The trailing dimension of a UGRID connectivity variable's data records, for each cell, the indices of the other cells to which it is connected (padded with missing data if a cell has fewer connections than some others).

What role do the edge coordinates have in the domain which refers to the faces?

That sounds like the right question, and I think the answer is "none", which clarifies for me that my initial concern about "round-tripping" is not actually relevant, here. How a software implementation decides, when writing multiple fields to a file, to avoid the duplication and proliferation of domain-related netCDF variables is up to it, as it always has been.
All the best,
David

thanks @JonathanGregory: that is better.
Final note on "axis" -- I think I'm being pedantic here, and it's probably bad to introduce new terms, but:
David wrote:

A "discrete axis" in CF is one which does not correspond to a continuous physical quantity, for example, this is the case for an axis that runs over ocean basins or area types, or for a domain axis that indexes a time series at scattered points. This also applies to the axis that stores the nodes of a UGRID mesh.

Does it though?

In many of these cases the discrete axis is sampling a higher dimensional space, as you say. This where the "discrete sampling geometries" of chapter 9 get their name (I always imagine a ship sailing along a sinuous course and recording SSTs at various time intervals). A domain that has such a discrete domain axis construct can also have other domain axis constructs (such as ones for time, level, etc.) which are indeed orthogonal to each other and to the discrete axis.

In all of these examples, the axis may be "wandering around" in a higher dimensional space, but as noted, they are still orthogonal to others, and more critically, they represent a continuum of some sort. That is, where along the axis they lie is meaningful -- the fact that a value comes before or after another value, or is "next to" a value is meaningful.
But in the case of a unstructured grid's node array, for instance, the order in which the node coordinates are in the array is completely (well, not completely, but ...) arbitrary. It is a mapping between a node index and the coordinates of that node, nothing more -- it could be completely re-ordered without its meaning changing. Which is why I don't think of it as an axis, even if it technically fits the definition.
But as I said, probably good enough, and I can't think of a better term.

Hi @ChrisBarker-NOAA,
This puzzled me for a bit, until I remembered that in the CF data model, DSGs are not  special. This is because what looks like a discrete axis in the encoding of a DSG is a really just a form of lossless packing for an orthogonal multidimensional array (with the data and coordinate arrays padded with missing data if required). This is indeed different to the UGRID mesh, as you rightly picked up on.
So bringing DSG's into this was misleading - sorry!
The all that the CF data model says about DSGs is:
When a collection of discrete sampling geometry (DSG) features has been combined in a data variable using the incomplete orthogonal or ragged representations to save space, the axis size has to be inferred, but this is an aspect of unpacking the data, rather than its conceptual description. In practice, the unpacked data array may be dominated by missing values (as could occur, for example, if all features in a collection of time series had no common time coordinates), in which case it may be preferable to view the collection as if each DSG feature were a separate variable, each one corresponding to a different field construct.
In my other examples (such as data stored on ocean basins) there is no assumption of (spatial) continuity, so I reckon that the CF data model discrete axis does apply to the UGRID case, after all.
Thanks,
David

Dear Chris and David
I agree that UGRID uses a discrete axis. There isn't necessarily any ordering implied by such an axis, just as in the case of UGRID. For example, ocean basins or other geographical regions may be arranged along a discrete axis in any order.
Best wishes
Jonathan

Thanks guys, i think I get in now -- the ocean basins example is a good one, I didn't really know what that meant the first time it was mentioned. "axis" it is :-)

Dear @davidhassell
Thanks for summarising the discussion up to now and writing the pull requests. As far as I know, this covers everything. The proposed changes look fine to me from a CF point of view. I noticed three small things:

In Appendix I (the data model), you describe the new CF-netCDF element as "Domain(s) with cell connectivity" and the new CF construct as "Connectivity of domain cells". I wonder if these should be the same.

In ch01 there is an accidental "accidential".

I note that many of the changes in https://github.com/cf-convention/cf-convention.github.io/pull/210/files are concerned with respelling github as GitHub. It's fine to put that right. In the unlikely event that this PR doesn't get accepted, we should remember to do that anyway.

Cheers
Jonathan

Dear Jonathan,
Thank you for your comments.

In Appendix I (the data model), you describe the new CF-netCDF element as "Domain(s) with cell connectivity" and the new CF construct as "Connectivity of domain cells". I wonder if these should be the same.

Good point, In this case I think they should different, though.
For instance, in the first new example in the new section 5.9 "Mesh Topology Variables" (which is adapted from https://ugrid-conventions.github.io/ugrid-conventions/#2d-triangular-mesh-topology) , the mesh topology variable Mesh2 contains  nodes, edges and faces, each of which corresponds to a self-contained CF data model domain construct. I.e. this mesh topology variable represents three domains (in the data model sense of "domain"):

(one domain for the 2 triangular faces, one for the 5 edges, and one for the 4 nodes.)
By contrast, in the data model we still do not formally recognise the inter-domain connectivity (e.g. the relationship between edges and faces; or the relationship between stagger locations on an Arakawa grid), but the existing domain construct does require the new domain topology construct component to represent the intra-domain connectivity required by UGRID.
I feel that this distinction should be better illuminated in the appendix I text  ... I'll see how I can work that in.

In ch01 there is an accidental "accidential".

Fixed (I'll push it up with the appendix I changes discussed above).

I note that many of the changes in https://github.com/cf-convention/cf-convention.github.io/pull/210/files are concerned with respelling github as GitHub. It's fine to put that right. In the unlikely event that this PR doesn't get accepted, we should remember to do that anyway.

You're quite right. (I shouldn't really do that, but I got carried away during a spell check :))

By the way, I very recently found out that GitHub now has nice image comparison feature when you do a "rich diff" on an image file, that includes side by side, and overlayed views (e.g. https://github.com/cf-convention/cf-conventions/pull/353/files?short_path=5a70e4b#diff-5a70e4b347e2935d49b12cfaff78556fb70319f62762e966b1c82042f022d463 and https://github.com/cf-convention/cf-conventions/pull/353/files?short_path=e4e016c#diff-e4e016c9bc64bd9cea792809a49d188836d6050352f0c24da156b896f8a3b6cf).

Dear Jonathan,
I have made some commits that hopefully make clearer the relationship between CF-netCDF mesh topology and location index variables, and CF data model Domain and Domain Topology constructs. What do you think?
https://github.com/cf-convention/cf-conventions/commit/3f9b4d53421f7da8c380c7e489e7713140edc60a?short_path=84395ff#diff-84395ffc95113b0c4c0cd22056e9b7461dc81bec9c89422b66c7b14f73d6c788
https://github.com/cf-convention/cf-conventions/commit/c401fae05d15f4aba7df1a2fe69163a8f00d389e?short_path=e4e016c#diff-e4e016c9bc64bd9cea792809a49d188836d6050352f0c24da156b896f8a3b6cf

Those changes look fine to me, thanks, David. The clarification is useful.

I believe that pull request #353 in this repo and pull request #210 in the website/governance repo can be merged, which make pointers from CF to UGRID. There was sufficient support according to the rules expressed when they were written early this year by @davidhassell and no remaining objections. I propose to merge them today.

Dear Jonathan,
I would very much like to do this, but there are perhaps a some loose ends: There has been no blessing, or otherwise, on the text from the UGRID folk (although the approach has been agreed), and the UGRID conformance document has not yet been completed (see https://github.com/ugrid-conventions/ugrid-conventions/issues/52#issuecomment-1158620865).
The first of these is not so much a problem, given that it's just an implementation of an agreed strategy and from the CF end we are happy with what we. But what about the conformance? Can we proceed without it?

Dear @davidhassell
Thanks for pointing that out and restraining my zeal. I don't think we can proceed without the UGRID conformance document being decided. @ChrisBarker-NOAA, are you able to help or advise us?
Best wishes
Jonathan

I took (an admittedly very quick) look at the PRs -- and it LGTM.
One comment though: Appendix K, Mesh Topology Attributes. seems to be duplicating some of what's in the UGRID spec, e.g. defining things like "face_face_connectivity". I see this has been explained with:
"However, a summary with examples can be found in <>, and to reduce the chance of ambiguities arising from their accidental re-use, all of the UGRID standardized attributes are specified in <> and <>."
which makes sense --  but if this is intended to be comprehensive, has it been reviewed for completeness?
As for the UGRID conformance document -- I think it's OK to have it as a draft, but we should at least publish it -- I've pinged that PR to see if we can get it done.

Hi Chris,
Thanks for taking a look!

which makes sense -- but if this [Appendix K] is intended to be comprehensive, has it been reviewed for completeness?

It is intended to be comprehensive. I don't know if has been reviewed for completeness, although Jonathan may have done so during his review of the PR and not mentioned it because it was found to be OK :) I will give it another check. Would you like to cast your eyes over it, Chris?
AS for keeping it up to date, well https://github.com/cf-convention/cf-convention.github.io/pull/210 should ensure that this happens (although humans are involved!)

As for the UGRID conformance document -- I think it's OK to have it as a draft, but we should at least publish it -- I've pinged that PR to see if we can get it done.

I would be happy proceeding on this basis. Someone tell me if that's too relaxed!

@ jypeterL thanks for the ping -- yes, we really need to get ha done!
Note: xarray doesn't "understand" UGRID -- though there is the https://github.com/UXARRAY/uxarray pacakge which is working on that.
And there is gridded (https://github.com/NOAA-ORR-ERD/gridded) which does work with UGRID, but doesn't use xarray (it could and should -- PR's accepted :-) )

And there is gridded (https://github.com/NOAA-ORR-ERD/gridded) which does work with UGRID, but doesn't use xarray (it could and should -- PR's accepted :-) )

Shout out for Iris, while we're on the subject 🙂

Does Iris have support for unstructured grids now?

Hi @jypeter, all of the references to UGRID are waiting to be merged into the latest document from PR #353 (TLDR: here and here) . We could merge this today if the draft UGRID conformance document were already  published ...
... if this still hasn't happened by the time CF-1.11 is due to be released (which is likely to be August/September 2023), then I will push for including UGRID into CF without the UGRID conformance, which is less than ideal but I think necessary for data creators and software providers.
Whilst we're shouting out, cf-python and cfdm will support UGRID by the time of CF-1.11.
Thanks,
David

Does Iris have support for unstructured grids now?

Yes, it absolutely does.
Enhanced the internal datamodel, and added several mesh-specific docs pages : https://scitools-iris.readthedocs.io/en/stable/further_topics/ugrid/index.html#ugrid
Here @MetOffice we put a lot of work into this 2021 as it's projected to have extensive internal use.
But real usage is actually only just beginning + API still experimental -- so all feedback welcome!

Hi, I realise that we need a little bit of extra text in the CF data model to account for the case that we have cell bounds but no representative coordinates. This would occur in the following case, for which the data is defined on faces, but the mesh only defines face vertices:
dimensions:
  nMesh2_node = 4 ; // nNodes
  nMesh2_face = 2 ; // nFaces
  Three = 3 ;
variables:
// Mesh topology
integer Mesh2 ;
    Mesh2:cf_role = "mesh_topology" ;
    Mesh2:long_name = "Topology data of 2D unstructured mesh" ;
    Mesh2:topology_dimension = 2 ;
    Mesh2:node_coordinates = "Mesh2_node_x Mesh2_node_y" ;
    Mesh2:face_node_connectivity = "Mesh2_face_nodes" ;
// Mesh node coordinates
double Mesh2_node_x(nMesh2_node) ;
    Mesh2_node_x:standard_name = "longitude" ;
    Mesh2_node_x:long_name = "Longitude of 2D mesh nodes." ;
    Mesh2_node_x:units = "degrees_east" ;
double Mesh2_node_y(nMesh2_node) ;
    Mesh2_node_y:standard_name = "latitude" ;
    Mesh2_node_y:long_name = "Latitude of 2D mesh nodes." ;
    Mesh2_node_y:units = "degrees_north" ;
// Mesh node connectivity
integer Mesh2_face_nodes(nMesh2_face, Three) ;
    Mesh2_face_nodes:cf_role = "face_node_connectivity" ;
    Mesh2_face_nodes:long_name = "Maps every triangular face to its three corner nodes." ;
    Mesh2_face_nodes:start_index = 1 ;
// Data
float tas(nMesh2_face) ;
    tas:mesh = "Mesh2" ;
    tas:location = "face" ;
    tas:units = "K" ;

The existing text says that the only way to encode cell bounds without coordinates is with simple geometries:
A CF-netCDF geometry container variable is used to store cell bounds without coordinates for a discrete axis.
This text should be modifed to:
A CF-netCDF geometry container variable or mesh topology variable is used to store cell bounds without coordinates for a discrete axis.
I'll update the pull request with this small change, pending further discussion.
Thanks,
Daivd

Dear @davidhassell
Thanks for noting this. In the current text of the data model section (at the link you helpfully provided) I wonder whether we have not distinguished sufficiently between the CF-netCDF encoding and the data model. We say, "A coordinate construct consists of an optional data array of the coordinate values spanning the subset of the domain axis constructs ...", and then, "The data array of the coordinate values is required, execpt for the special cases described below." The special cases are the two (formerly one) you have identified, if I understand correctly i.e. geometry container variable or mesh topology variable.
Those are two cases of encoding in netCDF. We do not logically distinguish simple geometry cells or UGRID cells from other kinds of cell. (UGRID has a domain topology construct in the data model, which contains extra information about cell connectivity, but it still has cells.) So I wonder if that, as far as the data model is concerned, the data array is unconditionally optional. We could replace the "execpt" sentence quoted above with something like: "Although the data array of the coordinate values is optional in the data model, it is mandatory in CF-netCDF, with two exceptions: simple geometries, and UGRID cells described by a mesh topology variable."
Cheers
Jonathan

So I wonder if that, as far as the data model is concerned, the data array is unconditionally optional.

Is it? Or is it optional in the data model? Don't you need SOME way of specifying the location of the data at hand?

I think that @ChrisBarker-NOAA is right - we do need some way of specifying the cell locations, and that can be with either coordinates (C), or bounds (B), or both coordinates and bounds. I.e. omitting both coordinates and bounds is disallowed, but any other combination is OK.
Other constructs, such as domain ancillary constructs, insist on having a data array present. So it makes sense that we insist on that for coordinate constructs - but noting that the data can be provided in one of three ways (i.e. C, B, or C & B).
Then also note, as @JonathanGregory said, "Although the data array of the coordinate values is optional in the data model, it is mandatory in CF-netCDF, with two exceptions: simple geometries, and UGRID cells described by a mesh topology variable."

@JonathanGregory wrote:

So I wonder if that, as far as the data model is concerned, the data array is unconditionally optional.

Maybe getting sidetracked here, but why is it unconditionally optional in the data model? Isn't a variable with no specification of its location kind of useless?
But I may be getting confused about what exactly is being talked about.

Dear @davidhassell and @ChrisBarker-NOAA
I can't think of an existing case where you would use a coordinate construct with neither coordinates nor bounds, so I agree with David that we must have one or the other although they are individually optional. So I was wrong to suggest that the coordinates are unconditionally optional. The first paragraph of the coordinate constructs in the data model could be something like this:

Coordinate constructs (Figure I.3) provide information which locate the cells of the domain and which depend on a subset of the domain axis constructs. A coordinate construct consists of an ~optional~ data array of the coordinate values spanning the subset of the domain axis constructs, properties to describe the coordinates (in the same sense as for the field construct), an ~optional~ data array of cell bounds recording the extents of each cell, and any extra arrays needed to interpret the cell bounds values. In the data model, all the components of the coordinate construct are optional, but it is mandatory to include either the coordinate array or the bounds array, and both may be included. In CF-netCDF, the coordinate array is mandatory, except for simple geometries and UGRID cells described by mesh topography variables.

Best wishes
Jonathan

Hello,
Since the beginning of May there has been a lot of discussion on this topic over at https://github.com/ugrid-conventions/ugrid-conventions/issues/65 and https://github.com/ugrid-conventions/ugrid-conventions/issues/66 and off-line - many thanks to @ChrisBarker-NOAA, @JonathanGregory, @hrajagers, @pp-mo and @drf5n for taking the time to think about this.
The result of all this is that we've had to change the nature of the CF data model Domain Topology construct, and need to make a note about "boundary_node_connectivity". None of this changes the agreed principles and approach of incorporating UGRID into CF.
It is very desirable to get this into CF-1.11 which will be released later this year (potentially as soon as the end of September), not least because a variety of general circulation models that need to archive in CF-netCDF are now starting to use UGRID to store their outputs.

Domain Topology construct description
Here is the new description of the Domain Topology construct (see https://github.com/cf-convention/cf-conventions/issues/153#issuecomment-888189611 for the original description):
Domain topology construct
A domain topology construct describes logically and explicitly the contiguity of domain cells indexed by a single domain axis construct, where two cells are described as contiguous if and only if they share at least one common boundary vertex. A domain construct allows contiguity to be ascertained without comparison of boundary vertices, which may be co-located for non-contiguous cells. A domain construct may contain at most one domain topology construct.
A domain topology construct contains an array that spans a single domain axis construct with the addition of an extra dimension that indexes the cell bounds for the corresponding coordinates. Identical array values indicate that the corresponding cell vertices map to the same node of the domain, but otherwise the array values are arbitrary.
In CF-netCDF a domain topology can only be provided for a domain defined by a UGRID mesh topology variable, supplied by a node connectivity variable, such as is named by a "face_node_connectivity" attribute. The indices contained in a node connectivity variable may be used directly to create a domain topology construct but the CF data model attaches no significance to the values, other than the fact that not all indices are the same.
The old version described a Boolean array which indicated which pairs of cells were contiguous, whilst the new version is an array that that has the exact form of a UGRID node connectivity variable.
The old version seemed at the time like a clear abstraction of what is going on, but there were still some subtleties which hadn't been considered (such as co-located but non-contiguous cells); and it turned out that with the old version it was not always possible to make the round trip from UGRID dataset -> CF data model constructs -> UGRID dataset. The first step was OK, but the second step still relied on inspection of cell bounds and examples were found which did not work, even given unambiguous bounds comparisons.

Domain Topology construct description

... carrying on from the last post!

Boundary Node Connectivity
The UGRID  boundary_node_connectivity attribute provides metadata towards describing the boundary conditions that constrain the data, and which are distributed over the same sampling domain as the field itself. The boundary conditions provide information on the processes that produced the data, and may inform operations on the the continuity of the data across them. Note that such  boundary conditions are constrained by the domain definition, but does not contribute to the domain's definition.
I propose to explicitly exclude Boundary Node Connectivity from being recognised by CF, given no current use cases for it, for the following reasons:

Practically, UGRID currently lacks a mechanism for associating the variable containing the nature of the boundary conditions with the variable containing their locations.

The nature of the boundary conditions is part of the model formulation. Whilst the formulation is clearly important, CF transcends this: it says "yes, you could compare/combine these two datasets and get a meaningful result", but that the interpretation of that result is a function of model formulation, experimental design, instrument type, etc., and these are up to the user to determine outside of CF. In this feature, UGRID is geared towards storing information needed to configure a model that uses the grid (https://github.com/ugrid-conventions/ugrid-conventions/issues/65#issuecomment-1520716314). I think that this is beyond CF at this time (but is clearly an area that could be explored later).

The current UGRID specification only allows for interface conditions along (N-1)d cell interfaces of Nd cells (e.g. edges of faces), but it is already being considered to extend this to interfaces at nodes of any cell type (edge/face/volume); at volume edges; and at volume faces. As this feature is under development, CF should consider incorporating it into the data model (for it would require a new data model construct) when it is more complete.

To that end, I suggest adding this new text to the new "Mesh topology" section of chapter 5 (see https://github.com/cf-convention/cf-conventions/pull/353/files#diff-3c189abe47ef902923e4a6126a2fe909ed568bcacae933778144094935c0a9d8 for the existing changes):
The UGRID conventions <<UGRID>> allow for the specification of boundary conditions that applied to the creation of the data, via the boundary_node_connectivity attribute, but this feature is not included in this version of CF.
Also, modifying the mention of boundary_node_connectivity from Appendix K (https://github.com/cf-convention/cf-conventions/pull/353/files#diff-d67e5a9a0f7dc06129dad9631f241b99f4fa8d962d5e81a15c2bb0776149f745).
| **`boundary_node_connectivity`**
| S
| MT
| Specifies an index variable identifying the nodes that each boundary element
(i.e. the nodes that define each edge of a face, or the nodes that define each face of a volume).
This attribute is not recognized by this version of CF.

@ChrisBarker-NOAA, is it indeed the case that data defined at nodes (which do not have bounds) are considered to be connected to the other nodes defined by whichever [edge|face|volume]_node_connectivity variables are present? If so, then the new Domain Topology construct definition does not work for this case, although the old definition would ....

Perhaps "contiguity" and "connectedness" are different concepts for which we need two CF data model constructs? The case I mentioned above maybe feels more "connected" than "contiguous".

Dear @davidhassell
Thanks for your work on this and the new text. Some aspect of it are unclear to me. They probably are clear given knowledge of the UGRID spec, which unfortunately I have forgotten! But the CF data model text should not depend on knowledge of the UGRID spec.

What does "which may be co-located for non-contiguous cells" mean? If two vertices are co-located - effectively identical - surely the cells which have these vertices are contiguous? Similarly, what are "co-located cells", mentioned later? Does this mean two cells which occupy exactly the same space (of the appropriate dimensionality: space, face, line or vertex)?

"an extra dimension that indexes the cell bounds for the corresponding coordinates". What are the "corresponding coordinates"? Does this mean the cells? Is the index over all the bounds i.e. any cell can refer to any bound (so this dimension is at least twice as large as the dimension of the domain axis), or is it a ragged array in which each cell refers to its own bounds only?

What is a "node of the domain"?

Excluding the boundary node connectivity from CF for the moment seems reasonable to me, since the definition is fluid. This information might later be treated as data in CF, or perhaps ancillary variables, rather than metadata of the domain.
Best wishes
Jonathan

Hello,
Since July, Chris Barker, Jonathan Gregory and myself have had many off-list discussions on getting UGRID within CF, and the three of us are now happy that we now have a data model extension which properly works for UGRID, with no unaccounted for corner cases.
I shall close the original PR and open a new one (#459) that contains the full integration of this new model into the conventions text, but for ease of reference, the new data model Domain topology and Cell connectivity constructs that we have devised are reproduced in this message.
We are considering the 3 week cooling-off period for merging this change as starting from now, but of course welcome feedback of any sort in the usual manner, and if that exposes anything we've missed and the clock is reset, so much the better!
All the best,
David
Domain topology construct
A domain topology construct defines the geospatial topology of cells arranged in two or three dimensions in real space but indexed by a single (discrete) domain axis construct, and at most one domain topology construct may be associated with any such domain axis.
The topology describes topological relationships between the cells - spatial relationships which do not depend on the cell locations - and is represented by an undirected graph, i.e. a mesh in which pairs of nodes are connected by links.
Each node has a unique arbitrary identity that is independent of its spatial location, and different nodes may be spatially co-located.
The topology may only describe cells that have a common spatial dimensionality, one of:

Point: A point is zero-dimensional and has no boundary vertices.
Edge: An edge is one-dimensional and corresponds to a line connecting two boundary vertices.
Face: A face is two-dimensional and corresponds to a surface enclosed by a set of edges.

Each type of cell implies a restricted topology for which only some kinds of mesh are allowed.
For point cells, every node corresponds to exactly one cell; and two cells have a topological relationship if and only if their nodes are connected by a mesh link.
For edge and face cells, every node corresponds to a boundary vertex of a cell; the same node can represent vertices in multiple cells; every link in the mesh connects two cell boundary vertices; and two cells have a topological relationship if and only if they share at least one node.

Figure I.5 A topology defined by a mesh with five nodes and six links.
For example, the mesh depicted in Figure I.5 may be used with any of three domain topology constructs for domains comprising two face cells (one triangle and one quadrilateral), six edge cells, and five point cells respectively.
A domain topology construct contains an array defining the mesh, and properties to describe it.
There must be a property indicating the spatial dimensionality of the cells.
The array values comprise the node identities, and all array elements that refer to the same node must contain the same value, which must differ from any other value in the array.
The array spans the domain axis construct and also has a ragged dimension, whose function depends on the spatial dimensionality of the cells.
For each point cell, the first element along the ragged dimension contains the node identity of the cell, and the following elements contain in arbitrary order the identities of all the cells to which it is connected by a mesh link.
For each edge or face cell, the elements along the ragged dimension contain the node identities of the boundary vertices of the cell, in the same order that the boundary vertices are stored by the auxiliary coordinate constructs.
Each boundary vertex except the last is connected by a mesh link to the next vertex along the ragged dimension, and the last vertex is connected to the first.
When a domain topology construct is present it is considered to be definitive and must be used in preference to the topology implied by inspection of any other constructs, which is not guaranteed to be the same.
In CF-netCDF a domain topology construct can only be provided for a UGRID mesh topology variable.
The information in the construct array is supplied by the UGRID "edge_nodes_connectivity" variable (for edge cells) or "face_nodes_connectivity" variable (for face cells).
The topology for node cells may be provided by any of these three UGRID variables.
The integer indices contained in the UGRID variable may be used as the mesh node identities, although the CF data model attaches no significance to the values other than the fact that some values are the same as others.
The spatial dimensionality property is provided by the "location" attribute of a variable that references the UGRID mesh topology variable, i.e. a data variable or a UGRID location index set variable.
A single UGRID mesh topology defines multiple domain constructs and defines how they relate to each other.
For instance, when "face_node_connectivity" and "edge_node_connectivity" variables are both present there are three implied domain constructs - one each for face, edge and point cells - all of which have the same mesh and so are explicitly linked (e.g. it is known which edge cells define each face cell).
The CF data model has no mechanism for explicitly recording such relationships between multiple domain constructs, however whether or not two domains have the same mesh may be reliably deternined by inspection, thereby allowing the creation of netCDF datasets containing UGRID mesh topology variables.
The restrictions on the type of mesh that may be used with a given cell spatial dimensionality excludes some meshes which can be described by an undirected graph, but is consistent with UGRID encoding within CF-netCDF.
UGRID also describes meshes for three-dimensional volume cells that correspond to a volume enclosed by a set of faces, but how the nodes relate to volume boundary vertices is undefined and so volume cells are currently omitted from the CF data model.
Cell connectivity construct
A cell connectivity construct defines explicitly how cells arranged in two or three dimensions in real space but indexed by a single domain (discrete) axis are connected.
Connectivity can only be provided when the domain axis construct also has a domain topology construct, and two cells can only be connected if they also have a topological relationship.
For instance, the connectivity of two-dimensional face cells could be characterised by whether or not they have shared edges, where the edges are defined by connected nodes of the domain topology construct.
The cell connectivity construct consists of an array recording the connectivity, and properties to describe the data.
There must be a property indicating the condition by which the connectivity is derived from the domain topology.
The array spans the domain axis construct with the addition of a ragged dimension.
For each cell, the first element along the ragged dimension contains the unique identity of the cell, and the following elements contain in arbitrary order the identities of all the other cells to which the cell is connected.
Note that the connectivity array for point cells is, by definition, equivalent to the array of the domain topology construct.
When cell connectivity constructs are present they are considered to be definitive and must be used in preference to the connectivities implied by inspection of any other constructs, apart from the domain topology construct, which are not guaranteed to be the same.
In CF-netCDF a cell topology construct can only be provided by a UGRID mesh topology variable.
The construct array is supplied either indirectly by any of the UGRID variables that are used to define a domain topology construct, or directly by the UGRID "face_face_connectivity" variable (for face cells).
In the direct case, the integer indices contained in the UGRID variable may be used as the cell identities, although the CF data model attaches no significance to the values other than the fact that some values are the same as others.
Restricting the types of connectivity to those implied by the geospatial topology of the cells precludes connectivity derived from any other sources, but is consistent with UGRID encoding within CF-netCDF.

Hello

We are considering the 3 week cooling-off period for merging this change as starting from now, but of course welcome feedback of any sort in the usual manner, and if that exposes anything we've missed and the clock is reset, so much the better!

Just a reminder that these changes will be merged in a couple of weeks unless any non-editorial questions are raised, so please do have a look at the PR #459 if you are interested.
Thanks,
David

Dear David
Thanks very much. I agree with the contents. I have two small editorial suggestions:

I stumbled on "however whatever". I suggest deleting "however" and starting a new sentence with "Whatever".

In the sentence

When cell connectivity constructs are present they are considered to be definitive and must be used in preference to the connectivities implied by inspection of any other constructs, apart from the domain topology construct, which are not guaranteed to be the same.

it's unclear what "which" refers to. I suggest

When cell connectivity constructs are present they are considered to define the connectivity of the cells. Exactly the same connectivity information could be derived from the domain topology construct. Connectivity information inferred from inspection of any other constructs is not guaranteed to be the same.

Cheers
Jonathan

Thanks @JonathanGregory and @taylor13, I have included your suggestions in PR #459.

I'm clearly quite late to the party here, but I have been carefully reading through the Issue here and the connected PR, and I wanted to register my support for this. I've also reviewed the PR #459 from a high-level perspective and it seems very sensible to me.
I particularly think it is good that we are not hard-coding aspects of the UGRID conventions into the CF Conventions, but instead referencing them, to loosen the coupling and making it less likely the two standards will get out-of-sync; and that the data models for both have been considered at the forefront for establishing and maintaining a formal association.
I raised a few questions as comments on the PR, https://github.com/cf-convention/cf-conventions/pull/459#pullrequestreview-1671888194 which occurred to me, but on further reading here might have been answered already in the above thread (there are quite a lot of comments to work through!).

Many thanks for your review, @sadielbartholomew - there's a lot of stuff here to wade through!
In response to your good point about the Conventions attribute, I have added a line to section 2.6.1:
"The UGRID conventions, which are fully incorporated into the CF conventions, do not need to be included in the Conventions attribute. "
This was always accepted as the case, as you also point out, so I don't think we need to reset the clock for this addition.

As far as I'm aware, there are no outstanding comments on this issue and pull request, so I'll give it another 24 hours just in case and then merge #459, alsongside https://github.com/cf-convention/cf-convention.github.io/pull/210 (update the rules for UGRID).
So many thanks to everyone who spent time on this (a lot of time, in some cases!) - we have, I think, arrived at a very satisfactory outcome.
David

I am glad to see this can be merged. :tada:
As some wash-up/tidying, I see that UGRID currently state in their README:

Note: NetCDF files using this convention can be be given the global attribute Conventions = 'CF-1.6, UGRID-1.0' if they are CF- and UGRID-compliant, or just Conventions = 'UGRID-1.0' if they are not CF-compliant.

which can be updated now in light of the new advice in https://github.com/cf-convention/cf-conventions/issues/153#issuecomment-1759132807. Should we open an Issue or PR to suggest this over on their repository?