What is netCDF?

Given recent discussions regarding additional C library features (for example, Unidata/netcdf-c#1545, Unidata/netcdf-c#1546, Unidata/netcdf-c#1548) a question that has been on my mind quite a bit is “what is netcdf?” Perhaps others are wondering as well. Here is my take, with the understanding that I started using netCDF in roughly 2004. If my take does not matter, that's ok - feel free to skip to the final paragraph. Note that none of this is intended to be a dig against any individual, project, group, etc., but rather outline a limited set of concerns regarding the current state of the netCDF ecosystem and how it moves forward, as well as a suggested path forward based on approaches that seem to work pretty well in other, somewhat similar situations (although not exactly the same by any means).
One take
Up until the addition of HDF5 as a persistence format (and the only persistence format supporting the extended data model), the situation seemed pretty clear, as the persistence formats were well defined, such that anyone could implement read/write without much ambiguity (and many did). I believe this helped gather a large, language agnostic community around netCDF and the associated persistence formats. The netCDF Data Model provided a way think about and access data in a relatively straightforward way, and netCDF files were a stable, portable container that was usable across multiple computing environments, including languages with different memory ordering (C and FORTRAN, specifically).
With the addition of HDF5 as a persistence format, as well as the extension of the netCDF data model, it feels like the answer to the question of “what is netCDF?” is less clear. The overall tone I get from the recent discussions mentioned above, and other similar github issues, is that netCDF, the data model, and persistence formats, are defined to be what the C library implementation supports and enables (as intended by direct implementation or unintended side effect of changes to the HDF5 library), and all other implementations need to keep pace. Honestly, that’s a perfectly valid choice, but not one that I ever considered was an option for this particular community driven endeavor. It's also not something that can be clearly found in the netCDF Users Guide, at least as far as I can tell, although one could argue it is hinted at in places (I'd say one gets a mixed message, at best).
netCDF-C is not the only library that attempts to read and/or write netCDF-4 files. The NCAR RAL developed Nujan library is a pure java writer for netCDF-4 and HDF files. More akin to netCDF-C, h5netcdf is a python library that both reads and writes netCDF-4 files using h5py (which wraps HDF5), and it also does not use the netCDF-C library (although it uses hdf5 python wrappers). Based on interactions with h5netcdf, I’d say maybe yes, the persistence format specification for the Enhanced Data Model can only be understood in light of netCDF-C implementation details, and really only in combination with the particular version of the HDF5 library in use by netCDF-C at the time of persistence. I know there are folks that would strongly agree with this view, and others that would not with equal intensity, but the current state of things with regards to the persistence format for the netCDF Enhanced Data Mode feels ambiguous at best.
With the C library addition of arbitrary filer support for use in writing, netCDF-4 as a persistence format has taken a concrete step towards GRIB in that now you need an open ended set of filters to read the actual data encoded by an application’s persistence layer (analogous to the way one can always read the octets of a GRIB message, with the question always being whether or not they can be understood and/or actually used). So now we have github issues discussing which filters are supported by netCDF (for the Enhanced Data Model persistence format HDF5), and the question is how are those filters defined? So far, for netCDF (specifically from the point of view of the C library), I’ve heard two methods:

a header file, which is analogous to a GRIB table ("can someone send me the libnetcdf header used by <insert agency acronym here>?")
filter specific function calls, which is similar to the situation in GRIB where one ends up crawling the code base of ecCodes or wgrib2 to figure out what’s really going on.

Both of these revolve solely around the netCDF-C library. The CCR will help reign in the possibilities with respect to filters (which is great!), but because compression is not explicitly specified as part of the netCDF Enhanced Data Model, and I assume the netCDF-4 persistence format spec is will not place limits on options now (currently the spec only lists zlib for writing), the options for writing now appear truly open ended (functionally making the default data type for a compressed variable in netCDF-4 opaque until a decompression technique is obtained). None of this is to say new compression methods for use in netCDF persistence are not needed. One such need driven by the operational community is clearly outlined here. However, we find ourselves in the situation where that while yes, netCDF-4 files produced by the C implementation are portable, they are not necessarily usable in the most basic sense, and therefore I would say not well suited for exchange or archiving as things currently stand, at least generally speaking ("generally" being outside of the current C library with specific filter support enabled). If you can’t read the actual data values, even with all of the metadata in the world, then what’s the point? Unless of course you can ensure the people you share your data file with use the exact same configurations and versions of netCDF-C and HDF5 libraries, plus any additional filters, used to produce the file..and all of these configurations/versions would be analogous to one or more GRIB tables. One could argue this was always the case for netCDF-4, and even netCDF-3 when combined with szip. I do not think that makes the where we currently stand ok, and in this case points to a deficiency in how well defined the netCDF-4 persistence format is. Given that the spec is currently written to be "sufficient to allow HDF5 users to create files that will be accessible from netCDF-4", I suppose it's not intended to be complete. One question would be, should it? And if so, does it belong in the C documentation, or elsewhere?
These thoughts lead me to ask, what exactly is netCDF?. Is the netCDF Enhanced Data Model and its currently associated single persistence format precisely defined to be anything a properly configured netCDF-C library will let you produce or consume (in addition to any unexpected side effects produced by the specific version of the HDF5 library in use, such as the under the hood change in the default superblock version). In that case, none of this particular github issue matters, as any other writer OR reader of data persisted in the current netCDF-4 format is, at best, a one off that should be avoided for real work. Do users of these libraries get what they deserve? If so, let's warn them and make if clear (I'm certainly not advocating this position, but if that's it, it should be put it in writing).
Where things really get hairy, and where these kinds of decisions regarding enhancements will have an incredibly wide spread impact even for the netCDF-C developers, will be when additional persistence formats options are added for the Extended Data Model (Zarr, for example). You can bet once the way in which the netCDF Enhanced Data Model is persisted using Zarr is finalized, it will only be a matter of time until a full pure python read/write implementation emerges, as well as other implementations that do not rely on libnetcdf (currently the plan for netCDF-Java does not involve relying on the netCDF-C library for this). I also see the likelihood of a JavaScript based full netCDF reader for the Zarr persistence format coming to life, and that will open up a whole new world of possibilities for getting data into the hands of decision makers, assuming the actual data chunks are readable. Once this happens, it will be critical that the development and enhancement of netCDF take a holistic view of the world, or draw a line in the sand, so to say.
When deciding how to advance netCDF, at least at the present time, there seems to be a tendency to think along the lines of “I think we should add feature X because we get it for free by using the HDF5 library”. When that feeds back into proposed changes to the Data Model, what that really says is, “I think we should let the features of a particular persistence format drive the data model”. If the only persistence format for the Enhanced Data Model is netCDF-4 (using HDF5), then you're good to go. What then if a feature from one persistence format ends up in the Data Model, but isn’t supported by another? Do we have a difference in Data Model features that can be persisted based on the choice of persistence format? Should netCDF files be limited to only those formats that can persist the data model? As I understand it, with Zarr as a persistence format as things currently stand, the answer is yes - the Zarr specification does not support all of the pieces of the Enhanced Data Model at the current time, so it will be an incomplete persistence format (at least initially). This has the potential to put users in a highly fragmented ecosystem. “Yeah, it’s a netCDF-4 file, but can only be read by the python stack because they used the python writer, which used X feature of Zarr, and the C Zarr reader does not support it yet.” One response would be “well, that’s not a real netCDF file”, but that will only work for so long.
Without a language agnostic way to move netCDF forward in innovative ways (data models and persistence layers alike), what is real, and how useful real actually is in practice, gets defined by the dominate writers...again, in many ways, just like GRIB. Already h5netcdf has the ability to write netCDF-4 files that are not readable by the C library, although not be default. Thankfully, h5netcdf code removes the _NC_PROPERTIES from those files, and will shout loudly at the user that they are doing something that is not "netCDF" (and in the future will throw an error), although is allowed by using invalid_netcdf=True in the file creation API call. Afterall, they are still valid HDF5 files (there is a GRIB analog with that, but I'll leave it be for now). From the users point of view, if they get a file with a .nc extension, and they use that library to read it in, they would never know the difference. And if they share that same file with someone who uses netCDF-C, the support question will end up here. Now these unsupported features might be perfectly nice to have, but how would h5netcdf developers go about proposing them - as a PR to the netCDF-C library?
Suggested Path Forward
I don’t believe that moving netCDF forward can be achieved in header file or a PR against the netCDF-C repository alone. Given the current state of netCDF, I feel we are very much in need of something like a netCDF Enhancement Proposal framework, following the likes of Python Enhancement Proposals or Java Specification Requests (and, I would argue, their community driven approval process as well) to prevent further fracturing. This can, and should, be an entirely separate discussion, but I'll throw it out there. However, it all depends on the question, what is netCDF?
Circling back
I suppose a good first place to start would be with the following question:
Is the netCDF Enhanced Data Model and its currently associated single persistence format precisely defined to be anything a specific netCDF-C library (specific to the set of configurations used with said library and dependencies) will let you produce or consume?

Sean, great discussion. I would like to start a separate issue about szip to discuss some of those specific points and how they bear on the software directly.
The answer to your thesis question is a (very carefully and thoughtfully crafted) sentence from the web page:

NetCDF (Network Common Data Form) is a set of software libraries and machine-independent data formats that support the creation, access, and sharing of array-oriented scientific data.

This is reflected in the backwards compatibility guarantee, which applies to both data files and C/Fortran code (not sure about Java - is backward compatibility fully supported in all releases? It was not when John was there - perhaps it is now).
Old files will always be readable. Old code will always continue to work and produce the same results.
This does not prevent, and never has prevented, new features being added, it does imply that we have to add the new features everywhere.
These challenges do not just arise in the HDF5 format. The CDF5 example is instructive and a great model for progress.
CDF5 was driven by operational requirements. Pnetcdf is popular, so plenty of data producers would have used CDF5, whether or not Unidata kept up. The alternative to CDF5 is not CDF1/2, it is binary files, because the limits of CDF1/2 had already been exceeded by many science codes. CDF5 was a great advance, and making it part of netCDF was a great decision. It strengthened the ecosystem because the two projects collaborated to bring the benefits to all users.
Similarly, if netCDF/HDF5 was not available, other giant data producers would not be using CDF1/2 or even CDF5. Instead, they would have just switched everything to HDF5. Their decisions are driven by operational requirements. If it takes more programming to convert all their code to HDF5, they will hire  another 100 contractors for a measly extra 10 million dollars.
Unidata users, on the other hand, would be in trouble. They just have to deal with what NASA, NOAA, and ESA are putting out. Because we wrap the benefits of HDF5 in netCDF in a way that does not violate our backward compatibility pledge, Unidata users can now use these data sets easily.
But now we are once again hitting limitations that were unthinkable 20 years ago. Never did I consider that one day, NOAA would be operationally attempting to compress and write files of 10s or even hundreds of GB. But here we are.
The answer is to move forward together, not to attempt to hold netCDF back. When a better solution becomes available within our ecosystem, and there's a strong need for it in the community, we must embrace it and provide it to netCDF users.
It is widely agreed that compression is an issue, and HDF5 has some well-developed solutions. Let's work together to bring those solutions to all our users. Otherwise large data producers will start producing important data files that cannot be read without extra filter installation efforts.

@lesserwhirls to specifically address the question of a single software stack to read HDF5 files.
Users currently have have a choice between the simple binary format, and the much more complex (and evolving) HDF5 format.
Large data producers have clearly chosen HDF5, either directly, or through netCDF. Their operational requirements drive the decision. It's not a choice between HDF5 and simple netCDF classic. It's a choice between HDF5 of a similarly complex custom format. The complexity is not accidental - it's there for performance.
The HDF5 format is not secret or proprietary. It's open, as are the software libraries. As long as the implementation is open-source, that meets requirements at NASA, NOAA, etc.

We also need to take into account the netcdf API (as defined
primarily by netcdf_xxxx.h files).
Ever since DAP2 support was added, a goal has been to allow
users to access other data formats as if they were  (subsets of)
the netcdf API. So in a  sense, the API defines "what is netcdf".
The nczarr implementation is a good case in point. It started with
the existing zarr  cloud format and extended it by attempting
to make the minimal changes necessary to support the netcdf API.
The result is a cloud format that is a superset of the zarr cloud format
but designed so that existing zarr code can read the content
as long as it ignores the nczarr extensions.

The answer is to move forward together, not to attempt to hold netCDF back.

100% agree. But I'm still trying to figure out what netCDF is, exactly, to understand what holding it back would be. For simplicity, we can limit our consideration of netCDF to the point of view of "netCDF-4 the file" (there is a Spaceballs meme in here somewhere). Is netCDF-4 the file only fully understood from the point of view of the C library?

The nczarr implementation is a good case in point. It started with the existing zarr cloud format and extended it by attempting to make the minimal changes necessary to support the netcdf API.

And similarly for the netCDF-4/HDF5 relationship? If so, maybe now I'm beginning to see my internal mismatch. I have viewed netCDF-4 files as well defined, standalone containers of data, and have been desiring a specification that includes a well defined subset of HDF5 file features utilized by netCDF-4 files, but that's not quite the right way of looking at things. In this case, netCDF-4 is any valid HDF5 file that follows a set of conventions with a few restrictions, and those conventions and restrictions are in place such that they allow HDF5 files to be read by the netCDF-C library (and associated wrappers)? That is, netcdf-4 isn't so much a file format based on HDF5, but rather an HDF5 convention?

@lesserwhirls you are correct about netCDF and HDF5.
NetCDF can read and understand almost all existing HDF5 files, and that is intentional. There are things now in the netCDF API (NC_STRING, for example) that are only there so that existing HDF5 files can be understood by netCDF. NetCDF-4 is a set of HDF5 conventions that allows us to express shared dimensions in HDF5. (Otherwise the models match pretty well and we need no conventions.) Even without those conventions in place, netCDF can do a pretty decent job of reading HDF5 files.
A big feature is that existing netCDF codes can be applied to existing HDF5 (and HDF4) data sets.

Thanks for starting this discussion @lesserwhirls!  One thing to keep in mind as we move forward is the work we are doing to uncouple the data model from data storage.  The Zarr/object storage access work is a great example of this, and us needing to address community concerns that were unthinkable a decade ago (as @edhartnett pointed out in a different thread, much has changed!).  The idea that a particular data model requires a specific storage format is no longer necessarily true, and this project will provide a roadmap for adding additional storage options in the future.  This isn't to say we are dropping HDF5 support, but rather reflects the broader needs of the community, and they want alternatives to the storage option(s) currently offered by netCDF.  So what is netCDF in this context? It is the data model and not the underlying storage format.   This will also let users select the storage format that works best for their purposes, be it HPC applications, cloud access, local storage, etc..  It will also let us select a subset of functionality offered by underlying libraries/storage formats, as dictated by the needs and expressed desires of our community.  There's no need to try to implement the kitchen, bathroom and utility sinks when our community has asked only for the one in the kitchen!
This also means we have to make sure we think carefully about implementation details, as @lesserwhirls points out.  It is relatively easy to focus on the C library (and I've been guilty of this in the past) without giving due consideration to the C++/Fortran interfaces, or considering what the impact will be on projects like netCDF-Java, netcdf4python, NCO, etc.  This risks creating a schism between user bases that might not be quickly resolved.  While there are certainly situations where this could be acceptable, we need to maintain a measured approach and make sure we think it through for even relatively small changes.  I have nightmares about the unintended consequences of making a small change with negative effects, as it ripples through the netCDF ecosystem.  Or I think about the 1.10.0 HDF5 release, and how for a few days we had 'netCDF4' files that only some netCDF4-capable software would be able to read.  Fortunately, we were able to knock out the changes on our end pretty quickly, but that was a pretty tense time.
Regarding the Suggested Path Forward, it bears some thinking about, and I imagine it will come up at the netCDF team meeting tomorrow.  Let me think about this more; I'll also follow up here as I organize my thoughts.  Taking a break from preparing EGU materials, and thought I'd respond to this thread in the meantime.  Hooray for drafting abstracts ;).

@lesserwhirls, I also thank you for starting this discussion.  I think file compatibility needs to focus on policies.  Other conversations were bouncing around between policy, implementation details, and other amazing things, partly with my help.
What is netcdf?  I agree with the overarching description that Ed quoted from the User Guide.  However, you asked to focus on persistence formats, which I think means physical file formats.  So for discussion, I offer a deliberately broadened definition to accommodate variations within reason.  I am avoiding opaque storage through translation layers, such as opendap and cloud; it seems those are different topics.
Netcdf file format may be considered to be approximately conforming to the limited set of published and recognized specifications of known netcdf file types.  Currently I see two official types: the netcdf-3 family, and the netcdf-4 family as an overlay or wrapper on HDF5 format.  As noted, a third type, nczarr, may be up and coming.
By "approximately conforming" I mean that internal structural conventions are the same or similar to the official known specs.  Examples of variants for netcdf-4 would be a new filter, multiple filters, or a newer HDF5 internal object version.  An example for netcdf-3 would be adding new data types, as was done for CDF5/pnetcdf.
For "approximately conforming", file identification and the overall internal file structure would remain the same as in official versions.  The key here is that a new construct inside the file would not make such file "not netcdf" under this broad definition.  I would call it "a netcdf file with a new object type" or a new filter type, or whatever.  But if it looked and smelled like netcdf, and the "file" command said netcdf or HDF5, and hackers -- er, I mean smart people -- could drill and parse through it like netcdf, I would still call it "netcdf" with a variation.  But lines can appropriately be drawn between "official" versus "unofficial" or "experimental" netcdf.
Unidata is concerned with file compatibility for the broad user community.  This is good.  As a way forward, it would help to think about this in terms of tiers of support or sanctioning for existing and future format variants.  This follows some natural competing purposes, such as broad format compatibility versus high performance.  Here is a suggested list to start.

Format variants that are known to be broadly supported by netcdf-C, netcdf-java, and other well known packages across the community.
Variants that are supported by Unidata software:  netcdf-C and netcdf-java.
Variants supported by netcdf-C.
Experimental formats and variants.

The lines are fuzzy.  The first category would be considered best for universal compatibility.  The next two might be candidates for high performance.  The last would be unsanctioned variants to be used at your own risk.  Good variants would be promoted when they meet some degree of approval, for example full szip support.  Variants might also be distinguished here, by subsets of a data model, such as CDM versus Extended.
Please consider this as one way to manage variants, evolution, sanctioning, and diverse needs.

Excellent discussion all! I have a feeling this thread will be useful far into the future.
From what I gather at this point, a netCDF-4 file is not something that can be understood apart from the specific version of the HDF5 C library used to produce it. That is to say, it's not enough to say "it's a netCDF-4 file, open it up, and get your science on!" That's a very different view than I previously had for what it meant to be a netCDF file. With the other netCDF formats, and certainly during my time in graduate school, it felt sufficient to say "the data are in netCDF" and we knew we could access the data and do the work we needed to do (metadata issues aside), using any number of libraries or tools (built upon the netCDF-C stack or not).
Circling back, with the scope of the question "what is netCDF" limited to netCDF-4 files (I promise we will return to that larger question), is it both necessary and sufficient to define a netCDF-4 file as any valid HDF5 file that conforms to these conventions?

I should have refreshed before submitting, as I just saw @Dave-Allured's excellent reply. I completely agree that we need a way to mark experimental formats, otherwise there is no moving forward without breakage. This would be a great candidate for the first netCDF Enhancement Proposal. I believe if we had a netCDF Enhancement Proposal framework, we would only need two (maybe three) categories for data formats:

Format variants that are known to be broadly supported by netcdf-C, netcdf-java, and other well known packages across the community.
(maybe) Fledgling formats
Experimental formats and variants.

We'd need to discuss what it would mean to be accepted as a format through the netCDF Enhancement Proposal framework. What would it take to, say, move from the third to the second category. I think part of that acceptance should include at least two independent implementations going forward (which ncZarr would have pretty quickly between the python and C implementations). That said, I think we should be very cautious, and very precise, when adding new formats.

@lesserwhirls It is quite easy to say "it's a netCDF-4 file, open it up, and get your science on!" That's what thousands of users do every day. There are no exceptions or conditions - netCDF guarantees full backward compatibility. A scientist who wrote data files with netcdf-c-4.0 can still open them with netcdf-c-4.7.3. And the same is true for HDF5 files.
If anyone wants to fund an independent implementation of HDF5, they may. John Caron, working alone, wrote one. But no one else sees the need. Even having an independent implementation, you guys do not develop it to keep up with changes in HDF5.
A fully independent implementation is a new requirement which, IMO, is not a wise requirement.
NetCDF-4 would not have met this requirement, and still does not, although it is extremely helpful to scientists around the world.

@edhartnett The existence of h5netcdf would seem to disprove your point. This means there are two additional implementations of support for HDF5-based netCDF, beyond netCDF-c: h5netcdf (reading and writing) and NetCDF-java (read-only). In the days of netcdf3, I know of 3 independent implementations within Python alone. When Zarr becomes a supported format, I can promise you that the use of this format will be based around an independent Python implementation, not netCDF-c.
So I would argue that the idea that the world doesn’t want independent implementations of netCDF is demonstrably false. In fact, I would say that the complicated on-disk format HDF5 may have reduced the feasibility of independent implementations, but they have made them no less desirable. Yes, this makes our world more complicated, but that’s just too bad.
Also, I’m not sure how you have an “archivable” data format, without a full, software-independent specification. And I’m not sure how you validate this specification without having independent implementations of this specification.

HDF5 does not seem to have an independent implementation. It would be great if it did, but it does not (as far as I am aware).
Despite this, there already are tremendously important Earth science archives of HDF5 data. These will continue to grow.
John Caron developed a full independent implementation of HDF5 in less than a year, so I guess you can have an independent implementation if you want one...
Sorry I don't have any help to offer here. Although this would no doubt benefit all netCDF users, I am particularly concerned with operational forecasting efforts, which are all about C and Fortran.  ;-)

I am particularly concerned with operational forecasting efforts, which are all about C and Fortran. ;-)

To get data out of a numerical model and onto disk, yes. But, take one step further. Send those files out to forecast offices where they will be ingested into an EDEX server for access via AWIPS CAVE. From what I can see, EDEX uses netCDF-Java to decode the data.

Even having an independent implementation, you guys do not develop it to keep up with changes in HDF5.

As a team with an effective FTE less than one (of which I am grateful that we have that, as other implementations are not so fortunate!), my priorities for HDF5 support in netCDF-Java are to maintain, at a minimum, what is needed to read netCDF-4. Second to that, we will try to address missing features in our HDF5 support as our users encounter them. The issue with that approach is that, as far as I can tell, there is no way to separate out what features of HDF5 are needed to read netCDF-4, outside of supporting read for everything enabled by the choice of superblock used when writing (because the netCDF-4 on disk spec punts to HDF5). NetCDF-Java currently support features up to and including superblock 2.

When first released, netCDF was completely incompatible with all other Unidata software and ways of doing things. Fortunately, at that time, Russ did not tell Glenn it should not be done. ;-)
When I wrote netCDF-4, no committee ruled out compression because it was not present in other implementations. And that's good, because compression has turned out to be one of the most popular features of netCDF-4.
No one prevented Dennis from adding new features to the DAP code. Russ never told John that he should not add IOSPs to netCDF-Java, though they certainly violate the philosophy of this "What is netCDF" thread.
And so we have the modern netCDF.
Russ did good work not forming committees and not telling us what software to not write! That was first rate technical leadership.
I trust that kind of thinking still has a place at Undiata and on the netCDF project. The cautions raised in this tread are valid and need to be considered. But these cautions should not be allowed to stop all new features. They should be weighed with each new idea, but not weighed so heavily that every idea must sink.
New development will not stop. If you attempt to dam the flow, you will simply isolate netCDF, and the features will be developed elsewhere. As the years go by and the great new features are added to other packages, then users will gradually switch away from netCDF, just as the giant data producers would have already switched away if netCDF-4 had not provided the new features they needed before they built the giant software systems that now depend on netCDF.
What is netCDF? It is a bit of a chameleon, changing to suit user needs in different eras. Historically is has grown disruptively, but with backward compatibility. Each advance has been a big leap, rather than an incremental improvement.
No one knows what great new features many be proposed tomorrow, and be considered an essential part of netCDF 10 years from now, just like all the features I've listed above. No doubt these new features will cause disruption and extra work. It's our role and privilege to cope with that work, and I'm quite confident we can continue to do so.
I hope and believe that working together is the best way to meet and surmount these difficulties, and ensure that the science world receives the most benefit and best I/O features from netCDF. I hope that netCDF can continue to incorporate the best ideas from it's active and energetic user base, and to remember that we do not have to have had an idea first, for it to be a good one.

@edhartnett I appreciate your fervor and enthusiasm for seeing netCDF advance and continue to serve its user base, as well as your continued contributions to see its use expand, especially for NOAA. I feel your chosen examples of netCDF history miss a few things:

DAP2 support relies on an external spec that defines how to communicate with other DAP2 servers. This is literally an example of what's being suggested to do for netCDF-4, and why it's necessary.
IOSPs are an implementation of read support for various file formats within netCDF-Java; they only serve to figure out how to map things into the netCDF data model, they don't define a serialization format for this data model.
You don't want committees in the way of doing new, greenfield development, like when you were creating netCDF-4. It's now 12 years after the release of netCDF 4.0. There are tons of users writing code to read data using it and lots of data producers saving their data in it. The netCDF-4 format has been a huge success story. netCDF-4 is all grown up now, and with that comes maturity, and a guarantee on our part of stability and dependability.

netCDF-4 has been so successful that other users are figuring out how to create netCDF-4 formatted data without using the netCDF-C library. The problem with that is that due to the lack of a netCDF-4 specification, users can create files that can reasonably be considered valid netCDF-4 files but are unable to be read by any of the predominant, Unidata-based netCDF library implementations. Do we want netCDF-4 files in the wild that require people to update their libraries to use? Or conversely, should there be valid netCDF-4 files that aren't readable on e.g. CentOS 6? The best case scenario here is that we end up with odd bug reports. Worst case is we end up like GRIB with a plethora of variants of the netCDF-4 format. How is that serving the best interests of the netCDF community?
I'm not proposing blocking any or all advancement. What I am proposing is we need a document that defines what makes a valid netCDF-4 file. Zarr has a spec, ncZarr has (or will have) a spec. netCDF-4, the file format, needs a spec--one that defines it completely, including what features from HDF5 are allowable. That document should then be updated through some process to allow an orderly advancement that doesn't create confusion for our users.

@dopplershift the good news is the document you propose for netCDF-4 has already been written. It is hosted by NASA here: https://earthdata.nasa.gov/esdis/eso/standards-and-references/netcdf-4hdf5-file-format, the standard document is here: https://cdn.earthdata.nasa.gov/conduit/upload/497/ESDS-RFC-022v1.pdf. A completely up to date explanation of the performance enhancements I added to this can be found in my 2018 AMS paper "NETCDF-4 PERFORMANCE IMPROVEMENTS OPENING COMPLEX DATA FILES" found here: https://ams.confex.com/ams/2019Annual/webprogram/Manuscript/Paper350021/NetCDF-4_Performance_2018.pdf
I will leave it to you guys to work up a similar documents on the Zarr format. I agree that you have some documentation work ahead of you there.
It seems unlikely that anyone is having trouble producing a netCDF compatible file with just HDF5. NetCDF-4 can read almost every HDF5 file without modification, so creating one that netCDF cannot takes some real effort. Anyone who is having trouble creating netCDF-4 files without the netcdf-c library should contact me with any questions or problems.
As always I welcome contributions and help netCDF and related packages. Please let me know if there is anything I can do to help you in your understanding and documenting what has already been achieved with netCDF.
And there is much more to come!
I hope we will have some good new compression options available from the CCR this year. Hopefully this will bring some long-needed new compression algorithms to the community. Interested users should follow progress here: https://github.com/ccr/ccr
The PIO library is also gaining users, providing the scalable HPC capabilities that netcdf-c lacks. PIO combines all known netCDF flavors, allowing the user to transparenely switch between netCDF classic, netCDF/HDF5, and paralllel-netcdf. The upcoming PIO release will include more compression and better integration with netCDF, allowing the use of existing netCDF C and Fortran code to use the PIO library. For more on PIO see: https://github.com/NCAR/ParallelIO.
There are many exciting developments in the netCDF world!

CCR has done a 1.1.0 release, which includes:

bzip2 compression filter
BitGroom lossy compression filter
Zstandard compression filter

Here's why these will be popular (figures from a poster Charlie and I are presenting at AMS in 2 weeks):

Some things to note:

bzip2, though the slowest, results in the best compression when used with the BitGroom filter.
zstandard yields same compression as zlib, but is an order of magnitude faster writing, 3 times faster reading.

Why to these matter?
The final size matters for long-term archival purposes. Many NASA missions, for example, use AWS S3 for backup storage. All their data gets uploaded to S3 to meet NASA requirements for an off-site data backup. Reducing that by 50% would save 50% of the money that AWS charges to the project. Since the data is backup only, slow read/write time is acceptable.
In other cases, it's the speed of write/read that matters. NOAA produces the massive UFS output files using parallel I/O with zlib, and by doing so they make their operational time budget. But when resolution increases again, they may overrun their time budget to write the data. That would be very challenging to fix by reducing the stored data, but Zstandard gives them a drop-in replacement which will be 10 times faster. That will give them plenty of headroom for further resolution increases.
There has been some trepidation expressed by some at Unidata with respect to these new compression options. I suggest a pro-active approach which supports the needs of data producers to cope with the ever-increasing size of netCDF datasets.
I will conclude with this sobering graph from NASA:

Faster and better compression is needed. The CCR project provides it, but netCDF-Java and netCDF on windows are going to need help keeping up...

CCR has done a 1.1.0 release, which includes:

bzip2 compression filter
BitGroom lossy compression filter
Zstandard compression filter

Here's why these will be popular (figures from a poster Charlie and I are presenting at AMS in 2 weeks):

Some things to note:

bzip2, though the slowest, results in the best compression when used with the BitGroom filter.
zstandard yields same compression as zlib, but is an order of magnitude faster writing, 3 times faster reading.

Why to these matter?
The final size matters for long-term archival purposes. Many NASA missions, for example, use AWS S3 for backup storage. All their data gets uploaded to S3 to meet NASA requirements for an off-site data backup. Reducing that by 50% would save 50% of the money that AWS charges to the project. Since the data is backup only, slow read/write time is acceptable.
In other cases, it's the speed of write/read that matters. NOAA produces the massive UFS output files using parallel I/O with zlib, and by doing so they make their operational time budget. But when resolution increases again, they may overrun their time budget to write the data. That would be very challenging to fix by reducing the stored data, but Zstandard gives them a drop-in replacement which will be 10 times faster. That will give them plenty of headroom for further resolution increases.
There has been some trepidation expressed by some at Unidata with respect to these new compression options. I suggest a pro-active approach which supports the needs of data producers to cope with the ever-increasing size of netCDF datasets.
I will conclude with this sobering graph from NASA:

Faster and better compression is needed. The CCR project provides it, but netCDF-Java and netCDF on windows are going to need help keeping up...

@edhartnett Really impressive work, and it's a huge :+1: from me to improve the quality of what netCDF offers its community. My only trepidation has been (what seems to me to be) a laissez-faire attitude about including such features in the library. In my opinion, features like this need to be thoughtfully considered as an addition to a netCDF standard--including a risk analysis with regards to implementation and support. Then, and only then, should such features enter into netCDF-C (with support for all platforms already supported) and netCDF-java.
This does not inhibit proactive features (just look at what's going on with the team's zarr work), but does require a methodical approach with all the time penalties that result. That IMO is our duty as stewards of such an important part of the scientific software stack.

@edhartnett Really impressive work, and it's a huge :+1: from me to improve the quality of what netCDF offers its community. My only trepidation has been (what seems to me to be) a laissez-faire attitude about including such features in the library. In my opinion, features like this need to be thoughtfully considered as an addition to a netCDF standard--including a risk analysis with regards to implementation and support. Then, and only then, should such features enter into netCDF-C (with support for all platforms already supported) and netCDF-java.
This does not inhibit proactive features (just look at what's going on with the team's zarr work), but does require a methodical approach with all the time penalties that result. That IMO is our duty as stewards of such an important part of the scientific software stack.

I too would like some clarification. I have been assuming that CCR would
provide the source (and possibly some binaries) for these compressors.
Then users would put the binaries in their HDF5_PLUGINS_DIR so that
netcdf could make use of them.
However, I am not sure I understand where, for example, the implementation
of nc_def_var_bzip2 resides; can you elaborate?
Also, I would prefer a different name for these functions. How about something like
nccr_def_var_bzip2?

I too would like some clarification. I have been assuming that CCR would
provide the source (and possibly some binaries) for these compressors.
Then users would put the binaries in their HDF5_PLUGINS_DIR so that
netcdf could make use of them.
However, I am not sure I understand where, for example, the implementation
of nc_def_var_bzip2 resides; can you elaborate?
Also, I would prefer a different name for these functions. How about something like
nccr_def_var_bzip2?

@DennisHeimbigner the nc_def_var_bzip2() function is in libccr, which must also be linked to the user application. The CCR also provides the source for the plugins, which are built and installed in HDF5_PLUGIN_PATH as you note. If the user wants to use nc_def_var_filter() directly, then they don't need to link to the CCR library and they don't need nc_def_var_bzip2(). CCR installs the plugins in the normal way, and they can be used by HDF5 without CCR. The CCR function names are designed to fit well with the netCDF API. Since we've already done two releases, it's a bit late to change the names.
@dopplershift I'm sorry can't take the time to help netcdf-java, but there's only one of me. ;-) I understand you are getting some help from John Caron again. That's great news, and I strongly suggest you start taking a look at zstandard compression in java. According to this post, there is a complete Java port: https://stackoverflow.com/questions/60974402/lz4-and-zstd-for-java.
I believe that the CCR obviates the need and desire to put these directly into the netcdf-c library, which was my original suggestion, so it seems that's no longer a concern. Since filters are already accommodated by both netCDF and HDF5, we are all good to go. Much thanks to Dennis for all his hard work getting filters, and multiple filters, to work well, and, of course, to the hard working programmers at the HDF Group.

@DennisHeimbigner the nc_def_var_bzip2() function is in libccr, which must also be linked to the user application. The CCR also provides the source for the plugins, which are built and installed in HDF5_PLUGIN_PATH as you note. If the user wants to use nc_def_var_filter() directly, then they don't need to link to the CCR library and they don't need nc_def_var_bzip2(). CCR installs the plugins in the normal way, and they can be used by HDF5 without CCR. The CCR function names are designed to fit well with the netCDF API. Since we've already done two releases, it's a bit late to change the names.
@dopplershift I'm sorry can't take the time to help netcdf-java, but there's only one of me. ;-) I understand you are getting some help from John Caron again. That's great news, and I strongly suggest you start taking a look at zstandard compression in java. According to this post, there is a complete Java port: https://stackoverflow.com/questions/60974402/lz4-and-zstd-for-java.
I believe that the CCR obviates the need and desire to put these directly into the netcdf-c library, which was my original suggestion, so it seems that's no longer a concern. Since filters are already accommodated by both netCDF and HDF5, we are all good to go. Much thanks to Dennis for all his hard work getting filters, and multiple filters, to work well, and, of course, to the hard working programmers at the HDF Group.

Echoing @dopplershift, the results you have shown in your post are great! However, this is about more than netCDF-Java (I have that covered), as you have suggested that netCDF-C on Windows will need help "keeping up" as well. And even then, this is about more than compression. The trepidation from my end is locking out entire portions of the ecosystem without even having a discussion or a plan. Now that I have a list of three specific filters (bzip2 compression filter, BitGroom lossy compression filter, Zstandard compression filter - none of which appear to be show-stoppers), I have a shot an getting them into netCDF-Java...but what about the windows support on the C side? What if there isn't a straight forward path for Windows support without major work? Was that topic discussed somewhere? Again, echoing @dopplershift, this is our duty as stewards of such an important part of the scientific software stack.

Echoing @dopplershift, the results you have shown in your post are great! However, this is about more than netCDF-Java (I have that covered), as you have suggested that netCDF-C on Windows will need help "keeping up" as well. And even then, this is about more than compression. The trepidation from my end is locking out entire portions of the ecosystem without even having a discussion or a plan. Now that I have a list of three specific filters (bzip2 compression filter, BitGroom lossy compression filter, Zstandard compression filter - none of which appear to be show-stoppers), I have a shot an getting them into netCDF-Java...but what about the windows support on the C side? What if there isn't a straight forward path for Windows support without major work? Was that topic discussed somewhere? Again, echoing @dopplershift, this is our duty as stewards of such an important part of the scientific software stack.

@lesserwhirls and @dopplershift I'm delighted with your sense of stewardship towards this vital software stack, into which Glenn, Russ, Steve, John, myself, Dennis, and Ward, and many others have poured much time, effort and care. Let's continue to work together to preserve and enhance that legacy.
Not sure what you mean by not having a discussion or plan. Seems like we've talked this topic a lot. This issue was started almost exactly a year ago! However, I'm always happy to answer questions about my work, so we can keep talking it out as long as you need to. We have a poster and paper for the AMS next week, which further discusses CCR in detail.
In terms of planning, CCR was proposed by Charlie in 2019. We told everyone our plans on the CCR website, a year ago, and we have been doing what we said we would do. Seems like we have a good plan and are following it well.
A Windows port of CCR and all it's filters is on the list of things to do in the New Year. I'll start with a CMake build and see how far that takes me. I agree that a windows port is a requirement.

@lesserwhirls and @dopplershift I'm delighted with your sense of stewardship towards this vital software stack, into which Glenn, Russ, Steve, John, myself, Dennis, and Ward, and many others have poured much time, effort and care. Let's continue to work together to preserve and enhance that legacy.
Not sure what you mean by not having a discussion or plan. Seems like we've talked this topic a lot. This issue was started almost exactly a year ago! However, I'm always happy to answer questions about my work, so we can keep talking it out as long as you need to. We have a poster and paper for the AMS next week, which further discusses CCR in detail.
In terms of planning, CCR was proposed by Charlie in 2019. We told everyone our plans on the CCR website, a year ago, and we have been doing what we said we would do. Seems like we have a good plan and are following it well.
A Windows port of CCR and all it's filters is on the list of things to do in the New Year. I'll start with a CMake build and see how far that takes me. I agree that a windows port is a requirement.

Not sure what you mean by not having a discussion or plan. Seems like we've talked this topic a lot. This issue was started almost exactly a year ago!

As I said, this is about more than compression (and by this, I mean this specific github issue). It just happens to be the case that discussions around compression led me to create this issue. My trepidation is more general than the question of new compression methods being potentially added to the netCDF-C library (e.g. bumping HDF5 H5F_LIBVER_LATEST versions, adding new persistence layer formats, etc.). My comment of "locking out entire portions of the ecosystem without even having a discussion or a plan" was meant to be more general. The CCR work is great, and definitely solves a problem. My concerns are not at all directed at the work that has been and is being done there. CCR intentions and plans have been communicated, although sometimes lost in the background signal of all of the implementation specific related issues on this repository.

A Windows port of CCR and all it's filters is on the list of things to do in the New Year. I'll start with a CMake build and see how far that takes me. I agree that a windows port is a requirement.

Excellent news! It was not clear that the work was planned, and the CCR repo didn't mention it until 20 minutes ago (and thus my last question asking if that topic, more specifically, was discussed somewhere).

Not sure what you mean by not having a discussion or plan. Seems like we've talked this topic a lot. This issue was started almost exactly a year ago!

As I said, this is about more than compression (and by this, I mean this specific github issue). It just happens to be the case that discussions around compression led me to create this issue. My trepidation is more general than the question of new compression methods being potentially added to the netCDF-C library (e.g. bumping HDF5 H5F_LIBVER_LATEST versions, adding new persistence layer formats, etc.). My comment of "locking out entire portions of the ecosystem without even having a discussion or a plan" was meant to be more general. The CCR work is great, and definitely solves a problem. My concerns are not at all directed at the work that has been and is being done there. CCR intentions and plans have been communicated, although sometimes lost in the background signal of all of the implementation specific related issues on this repository.

A Windows port of CCR and all it's filters is on the list of things to do in the New Year. I'll start with a CMake build and see how far that takes me. I agree that a windows port is a requirement.

Excellent news! It was not clear that the work was planned, and the CCR repo didn't mention it until 20 minutes ago (and thus my last question asking if that topic, more specifically, was discussed somewhere).

The issue for me is a namespace issue. Since C does not have C++ style namespaces,
it is important to be cognizant of what names you give to functions.
The  nc_dev_var_XXX names technically intrude on the netcdf-c namespace, so it
makes me unhappy.
Re the Java issue. It is possible to use these compressors via the JNA implementation.
But assuming that in many cases you took the compressor implementation from
elsewhere, you might see if those same sources by any chance have Java implementations
and publish a pointer to any that you find.

The issue for me is a namespace issue. Since C does not have C++ style namespaces,
it is important to be cognizant of what names you give to functions.
The  nc_dev_var_XXX names technically intrude on the netcdf-c namespace, so it
makes me unhappy.
Re the Java issue. It is possible to use these compressors via the JNA implementation.
But assuming that in many cases you took the compressor implementation from
elsewhere, you might see if those same sources by any chance have Java implementations
and publish a pointer to any that you find.

This was an interesting discussion, but is not really a netcdf-c issue. I recommend this issue be closed, or moved to the discussion section of GitHub.

The issue could be transferred to Unidata/netcdf if @WardF agrees.

Agreed.


Updates Openweather for Precipitation

Small fix, will probably fix other issues later.

whoops, added all the commits to the pr.


Refactor dataset CONFIG dict into models

Throughout the v03 pipeline code we rely heavily on the CONFIG dictionary defined in v03_pipeline/lib/reference_data/config.py, which holds external dataset configurations.
Refactor this dict into something more object oriented (like enums or python dataclasses) and replace all the references to the dict, (including the many mock config dicts in tests, like this), to make the code cleaner and more maintainable. (One issue with the dict is that we have no way to enforce requirements like if one key is present, another must also be.)
This is also a good time to add ReferenceGenome as an attribute of the dataset (suggested in a code review here).

Review when we are doing the database migration.


CDN s3 url not compatible with subdomains

Hello, it seems like the added support for AWS S3 URLs for CDNs broke support for path style urls with nested subdomain (where hostname has more than 2 dots) e.g. minio.east.domain.com/mybucket.
A workaround is to set S3_ENDPOINT to the wanted origin and set the hostname in the s3 url to a dummy value without dots e.g. https://-/mybucket
Let me know if I missed something

Hi @YassineElbouchaibi  thanks for pointing that out,
we are already working on the issue and will provide a fix.

Hi @YassineElbouchaibi,
https://github.com/wundergraph/cosmo/pull/1145
it is on review.

Hi @YassineElbouchaibi,
can you please try using the new release: https://github.com/wundergraph/cosmo/releases/tag/controlplane%400.105.1

It works great, thanks for the quick turnaround!


New feature: project manager

Proposed changes:

Collect project loading / preloading / unloading functions into a new class: ProjectManager

Status (please check what you already did):

[x] made PR ready for code review
[x] added some tests for the functionality
[x] updated the documentation
[x] updated the changelog

Hey @howl-anderson,
Not sure what status the review of this PR was in, but it doesn't make sense for us to merge this any more because we will soon be removing projects altogether.  More info and discussion on our forum here.
We thank you for the time you spent writing up this code, do you believe there are any features you implemented here that should still be merged?

@MetcalfeTom It's OK to me. User can build higher level architecture (beyond the Rasa stack) to simulate project behavior for PaaS scenes (different group of people have different projects in one platform)


(JAVA-W1037) Local variables should not be assigned in return statements

Description
A local variable is being assigned within a return statement. This may be either a typo in a comparison or an unnecessary operation. Review this code and either directly return the local variable or convert the assignment into a comparison as intended.
Occurrences
There is 1 occurrence of this issue in the repository.
See all occurrences on DeepSource → deepsource.io/gh/Verisimilitude11/Vigenere-Cipher/issue/JAVA-W1037/occurrences/

Hi VerisimilitudeX 🫡
Do you need some help with this issue?
I am avaiable to do that. 🚀
Could you assign this issue to me? I'd like to work on this issue, please.
Many thanks in advance 😄
Thank you 👍


DTS Plugin, generate Types path

Clear and concise description of the problem
When defining Typescript remotes we can set a path for mf-manifest.json but types are always created in dist without a sub path.
Withing modernjs server those files cannot be accessed.
Please enable a param for dts like for manifest
name: 'my_provider',
          exposes: {
            './Container': './src/components/MyContainer.tsx',
          },
          manifest: {
            filePath: '/modules/container/static/js',
          },
          dts: {
            generateTypes: {
              typesPath: '/modules/container/static/js
              typesFolder: 'mf-container-types',
            },
          },
          shared: ['react', 'react-dom'],
        }),

Suggested solution
Use the same path as given for mf-manifest.json by default and enable developers to define a custom path via
dts generateTypes.typesPath
Alternative
provide the types in js asset folder
Additional context
No response
Validations

[X] Read the Contributing Guidelines.
[X] Check that there isn't already an issue that request the same feature to avoid creating a duplicate.

#2550 fixed this :)


Remove stray copy of a file

I blame the reviewer.  :-D

+1
On Wed, Apr 11, 2018, 12:42 PM Tom Swirly<EMAIL_ADDRESS>wrote:

I blame the reviewer. :-D

You can view, comment on, or merge this pull request online at:
https://github.com/ManiacalLabs/BiblioPixel/pull/765
Commit Summary

Remove stray copy of a file

File Changes

D bibliopixel/util/pid.py
https://github.com/ManiacalLabs/BiblioPixel/pull/765/files#diff-0
(27)

Patch Links:

https://github.com/ManiacalLabs/BiblioPixel/pull/765.patch
https://github.com/ManiacalLabs/BiblioPixel/pull/765.diff

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/ManiacalLabs/BiblioPixel/pull/765, or mute the thread
https://github.com/notifications/unsubscribe-auth/AA6a6ux4cg39vUNawsFg1Wsf96H6mPT0ks5tnjKSgaJpZM4TQYAK
.


Feature: Add power factor calculation

Pls, consider adding a feature to calculate Power-Factor out of current UIP meterings.
I've made some tests for different loads and seems it works pretty fine most of the time.
Here is a sample graph.
I've implemented this a new method to the lib and an example code. If it looks reasonable, I can make a PR.
Thanks.

Thanks. I'll take a look at it soon.
I see you've added yield in reading method. Does it solve issue described here https://github.com/olehs/PZEM004T/issues/8 ? Are there any side effects?

Hello!
I use this library with ESP8266 (Wemos D1 mini)
errors after library update:
`C:...\Arduino\libraries\PZEM004T-master\PZEM004T.cpp: In member function 'bool PZEM004T::recieve(uint8_t, uint8_t)':*
C:...\Arduino\libraries\PZEM004T-master\PZEM004T.cpp:137:37: error: 'class SoftwareSerial' has no member named 'listen'
     ((SoftwareSerial *)serial)->listen();`

@outrullers see https://github.com/olehs/PZEM004T/issues/14

Yes, yield() makes a great difference for esp platform. Putting it between consecutive calls to pzem outside of the library solves the watchdog problem, but esp is still unresponsive to networking during serial IO.
Putting it inside the receive() lib call solves this issue too.
As for the side effects - I do not see any. Now I'm running it in software-serial mode, do not see any read errors, no data skips, so far so good.

@pieman64 To your comment, it is not so simple unfortunately. While it is easy to spot the problem in such a cases like only one pure active load - it's either switched off or on and you have reading current value 0 or >0. But it becomes more complicated if power meter monitors main line of your house with lots of loads simultaneously. You always have non zero current&power values and pf is usually <1 because of lots of switching PSU's. PF calculation works fine for steady loads but could give spikes in both directions if measurement had taken place during heavy transient power changes.  And which of the pzem readings you trust more - the current or the power? That is an open question to discuss and it would be nice to have more feedback on this.
For my meter project I'm fixing current value based on power readings if pf spikes to >1, but for the library implementation I think that it's better not to tackle actual pmeter readings but let user decide how deal with it.
I'm open to suggestions on this topic.

@vortigont yes you guessed right we are not currently using the Pzem for "whole house" monitoring. In fact some of the monitoring is down at just 70mA and 1 or 2W. Not sure what the accuracy is at this level when the Pzem only offers integer Watts etc.
In our sketch as well as v & I > 0 we have p > 30. Anything less than 30W gives a widely inaccurate pf.

@pieman64  To rise sensitivity for such a low power solutions I've tested setup with increased number of wire turns over CT and correcting read values in sketch, it could help but requires some calibration.


Add proper cursor support

Is your feature request related to a problem? Please describe
When I play some games with mouse support (e.g. Minecraft), it stays on touchscreen mode because of the touchscreen emulation.
Describe the solution you'd like
Add an option to disable touchscreen emulation, or downright disable touchscreen emulation because android supports mouse+keyboard natively.
Describe alternatives you've considered
No response
Please specify the version of Windows Subsystem for Android
2307.40000.6.0

Most Android apps don't support mouse, so we have to provide touch emulation by default.  Apps can opt out of the touch emulation by declaring that they support PC input modalities in their manifest as follows (this is the same approach used by ChromeOS):
<manifest ...>

...

We can consider providing users a setting to override the default, but this GitHub is for developer requests.  Please submit end user requests in Feedback Hub.

Ok, noted.
Thanks anyway 👍


Does mio have gitter channel yet?

It seems there are more and more project building on mio and I think mio as base-project should take care of organizing child-projects in some form of community.
It would be good to have a place to hang out and discuss ideas, to have some notion of what's going on in other projects building on top of mio. Also following mio development is important for people building on top of mio. Unfortunately IRC, without chat history persistence, is good only for quick discussion. So I think it would be nice if mio could have a gitter channel. I would personally like to direct mioco README there instead of mioco's one, as it's just way to small project ATM.

@zonyitoo What do you think?

I did an informal survey and there wasn't a huge amount of interest. I will leave this issue around for a bit to see if there are further thoughts.
Another option would be to figure out how to get #mio logged.

Gitter is a good start.

I like gitter because it has persistent chat, and a mobile app that behaves sanely and does not draw battery like irc clients. And it seems to be popular enough in Rust community.

I'm new to mio and wouldn't mind a more informal channel to ask easy questions. +1 for gitter from me

@KodrAus We are on IRC in the #mio channel on the mozilla server.
I'm going to close this for now. I don't know if gitter is right for mio right now. I think more important would be a discourse install or something more async.

Appears the mozilla IRC server commonly bans Tor users btw : https://bugzilla.mozilla.org/show_bug.cgi?id=802788


Long Video Support

when i try to put a video length 6min,it suddenly memoryleak. can model using cv2 directly read video flow and track the movment of the object?

If you switch to using async_loading_frames=True when calling the init_state function, and then disable the frame caching inside the async loader by commenting out line 165, then you should be able to get through a 6 minute video without memory issues (it's also worth commenting out the use of threading as well). You can find more information about this in issue #264.
If you did want to use opencv to load from a video file instead, it's not supported in the code as-is, but it wouldn't take much to modify the async loader to work with video (you'd need to setup the video capture, i.e. cv2.VideoCapture(video_path) inside the __init__ function, then read frames inside the __getitem__ function).

Hello, I've tried your suggestions above, it helps a lot, but still it's leaking memory, it's growing much slower, but once in a while it will just OOM.

Aside from the async loading stuff, there is another modification that can be made to stop the per-frame results caching from using up memory over time (it's mentioned in issue #196).
Aside from that, it's possible the whole model might be reloading as well? At least based on the yellow memory plot jumps around 1:00, 4:30 and 7:00. It might just be very similarly sized videos but those jumps seem to be consistently around 1.5-2GB, which is roughly the amount of memory needed to initialize the model itself. So it might be worth double checking that only one instance of the model is being used (if that's possible for your use case).


chore(deps): update vim/pack/plugins/start/fzf.vim digest to 556f45e

fzf should no longer get updated.


[Fix bug 1119510] Fixed the search overflow css issue.

Coverage remained the same at 93.0% when pulling 10ecfa54c3b1903c05403f50f225b8848af4b059 on akasantony:fix-search-css into 4a747df5f7a73bf7f1c95a8cf1efc2aa07ba4d29 on mozilla:master.

@comzeradd Can you please take a look?

r+


[BUG] executors died with too much off heap in yarn UCX CI udf_test

This has nothing to do with UCX, but the CI job failed due to yarn OOM (host out of memory) killing executors during the -k udf_test run:
23/04/02 17:44:52 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Requesting driver to remove executor 1 for reason Container killed by YARN for exceeding physical memory limits. 23.6 GB of 22 GB physical memory used. Consider boosting spark.executor.memoryOverhead.

 The memory manager is allocating an array of unsafe MemoryBlock. This needs to be fixed, and it may be the underlying cause here.


Allow Vault variables provider to retry

Maybe it will mitigate that flaky integration test.  Please let it mitigate that flaky integration test.

Please let it mitigate that flaky integration test.

It looks like testing-framework supports Docker HEALTHCHECKs. I wonder that might solve your flakiness more directly?

This seems to work:
HEALTHCHECK --interval=3s CMD vault status -address=http://<IP_ADDRESS>:8200

https://github.com/fermyon/spin/pull/2382

Closing in favour of #2382


[ALLUXIO-2623]Fix comment style in LRUEvictor.java (line 57)

https://alluxio.atlassian.net/browse/ALLUXIO-2623

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/14424/
Test PASSed.


Feature : Enable Intel GPU/XPU finetuning and inference

What does this PR do?
Motivation:  Thanks for creating this repository. At Intel, we are using this repository for finetuning and inference for our GPUs /XPU (Ipex backend) and hence the reason for this PR.
This change would help us run LLama v2 without issues on our GPU devices. (Since accelerate/peft already has support of XPUs enabled)
Fixes # (issue) -NA
Feature/Issue validation/testing
Feature for finetuning and inference tested on our Data Centre GPUs with the change
for CLA purposes this would be on behalf of Intel.
Before submitting

[ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
[ ] Did you read the contributor guideline,
Pull Request section?
[ ] Was this discussed/approved via a Github issue? Please add a link
to it if that's the case.
[ ] Did you make sure to update the documentation with your changes?
[ ] Did you write any new necessary tests?

Thanks for contributing 🎉!

@abhilash1910 thanks the PR, very interesting, would love to learn more about how we can test Intel GPUs/XPUs. I sent you a message on LD.

@abhilash1910 Thanks for submitting this PR. Please attach the logs for the tests for verification that things are working on the Intel GPUs with the changes. We don't have a mechanism to test this.
Can you also please check if torch.set_default_device api can be used to simply the device setting logic for the XPU devices?

@chauhang  sure I will attach the logs; also as discussed with @HamidShojanazeri  there is a plan to support device for integration testing for any future PRs from Intel side. I am closing on it this week internally , and eventually would like to discuss on the next steps.

@chauhang @HamidShojanazeri logs for distributed finetuning on 8 ranks :
7B_8ranks_intel_xpu.log
Could you also share a tentative timing for a discussion sometime this week or next on possible next steps?

Thanks very much @abhilash1910, I am wondering if there is a way to add support/ get intel GPU memory stats as well the way we are getting for nvidia.

@HamidShojanazeri yes the memory stats are also added , this can be found in the new log here:
7B_8ranks_intel_xpu.log
Thanks


left_sidebar: Restore righthand padding to DM partners.

This removes a now-unnecessary style block that restores space to the righthand side of DM partners, so that collisions with unreads as shown in the Before screenshot below are no longer possible.
Fixes part of #30359

Screenshots and screen captures:
Note the 1:1 DM to Prospero.

Before
After

Self-review checklist

[x] Self-reviewed the changes for clarity and maintainability
(variable names, code reuse, readability, etc.).

Communicate decisions, questions, and potential concerns.

[x] Explains differences from previous plans (e.g., issue description).
[x] Highlights technical choices and bugs encountered.
[x] Calls out remaining decisions and concerns.
[x] Automated tests verify logic where appropriate.

Individual commits are ready for review (see commit discipline).

[x] Each commit is a coherent idea.
[x] Commit message(s) explain reasoning and motivation for changes.

Completed manual review and testing of the following:

[x] Visual appearance of the changes.
[x] Responsiveness and internationalization.
[x] Strings and tooltips.
[x] End-to-end functionality of buttons, interactions and flows.
[x] Corner cases, error conditions, and easily imagined bugs.

Looks like a nice improvement!

Merged, thanks @karlstolley!


[Notification] User does not receive push notifications when a ride is created on request

Environment: OnePlus 8 Pro, Android 11
Reproducible: Always
Build found: 1.0.332
Preconditions

Open the app.
Login as a passager.
Create a ride request.

Steps to reproduce

Login as a driver.
Create a ride according to the previously created request.

Actual result
Passenger does not receive push notifications regarding the creation of a trip on request.
Expected result
The passenger receives push notifications regarding the creation of a trip on request.
Labels to be added
"Bug" ("bug"), Priority ("pri: ")

Fixed in previous pull request


Added async version of join_task_result

Added async version of join_task_result function.
We may use it to solve a couple of captcha simultaneously, or in case we do not want to wait until we get result and do some other stuff at the same time

Can you delete the blank in third line? Also if you can change the method's name like "***_async", I would be glad.

sure, changed that


Add support for multiline string output

Co-authored-by: Karsten Heinze<EMAIL_ADDRESS>

I'm interested in this feature as well, is there anything we can do to have this reviewed & merged?

:+1: for this PR, I hope it gets merged -- this was nicely implemented and would be a fine improvement.
Would it help to rebase against the latest to eliminate the CI fails?  I don't know if that's why this hasn't been looked at, but perhaps that would eliminate one blocker.  FWIW the tests pass fine when I merge this branch into the latest (https://github.com/davvid/yaml-rust/commit/d97f36287c1bf267ef020578481607fc8c999509), so a rebase should do the same.

:+1: for this PR, I hope it gets merged -- this was nicely implemented and would be a fine improvement.
Would it help to rebase against the latest to eliminate the CI fails?  I don't know if that's why this hasn't been looked at, but perhaps that would eliminate one blocker.  FWIW the tests pass fine when I merge this branch into the latest (https://github.com/davvid/yaml-rust/commit/d97f36287c1bf267ef020578481607fc8c999509), so a rebase should do the same.

@chyh1990, is there any chance to merge this PR? Multiline strings a main benefit over JSON in configuration files as it allow normal versioning of configs. Is there any what I can do to move forward?

I'm using the fork of davvid (https://crates.io/crates/yaml-rust-davvid) so I could use this feature. This repo seems to be missing a maintainer?


org.opencv.DocumentScannerPackage

could not get DocumentScannerPackage from opencv library

any update, getting same issue

any update ??

I think it could be related to 0.61 RN version, this package is not updated to use AndroidX, I recomend to exclude this package from jetify by seting:
create a file react-native.config.js and paste it:
module.exports = {
  dependencies: {
    'react-native-documentscanner-android': {
      platforms: {
        android: null, // disable Android platform, other platforms will still autolink if provided
      },
    }
  }
};

i'm not having time to fix the issues in this package. So helps are welcome.


Crash when copying type in change type popup

Hi,
i recently get regular crashes when copying field types.
This crash happens from time to time and is not 100% reproducable, it takes minutes to an hour of copying stuff around until it happens. After a restart the same copy command on the same field normally works again, but it will crash some time later with a different field.
The last time it crashed i did the following:
Select a Type -> select a field -> press Y to change type -> Ctrl + C to copy -> crash -> windows is unable to use the clipboard, while the VS debugger is paused. Aka. you cannot copy and paste anything in any program on the machine.
Exception:
Exception thrown at 0x0007FFF0239B04C (binaryninjacore.dll) in binaryninja.exe: 0xC0000005: Access violation reading location 0x00007FFF06D72B0.

Stacktrace:
>    binaryninjacore.dll!00007fff0239b04c()  Unknown
     binaryninjacore.dll!00007fff0239a91d()  Unknown
     binaryninjacore.dll!00007ffefd5130c3()  Unknown
     binaryninjacore.dll!00007ffefe412811()  Unknown
     binaryninjacore.dll!00007ffefd6cf4ec()  Unknown
     binaryninjacore.dll!00007ffefd739bbf()  Unknown
     binaryninjacore.dll!00007ffefe44ce6e()  Unknown
     binaryninjacore.dll!00007ffefe44ef5d()  Unknown
     binaryninjaui.dll!00007fff272b3a49()    Unknown
     binaryninjaui.dll!00007fff2717315b()    Unknown
     Qt6Core.dll!00007fff2297a0da()          Unknown
     kernel32.dll!BaseThreadInitThunk()      Unknown
     ntdll.dll!RtlUserThreadStart()          Unknown

binaryninja modules:
binaryninja.exe    binaryninja.exe    C:\Program Files\Vector35\BinaryNinja\binaryninja.exe    N/A    Yes    Cannot find or open the PDB file.        1    4.03.6539.0    05/12/2024 20:29    00007FF7A82C0000-00007FF7A9896000    [0x3900] binaryninja.exe
binaryninjacore.dll binaryninjacore.dll C:\Program Files\Vector35\BinaryNinja\binaryninjacore.dll   N/A Yes Cannot find or open the PDB file.       6       05/12/2024 20:22    00007FFEFCF60000-00007FFF064D7000   [0x1B58] binaryninja.exe
binaryninjaui.dll   binaryninjaui.dll   C:\Program Files\Vector35\BinaryNinja\binaryninjaui.dll N/A Yes Cannot find or open the PDB file.       12      05/12/2024 20:27    00007FFF227F0000-00007FFF2359B000   [0x1B58] binaryninja.exe

Every crash has the same stacktrace.
Stacktrace with relative values (manually calculated):
>    binaryninjacore.dll!0x543B04C()    Unknown
     binaryninjacore.dll!0x543A91D()    Unknown
     binaryninjacore.dll!0x5B30C3()     Unknown
     binaryninjacore.dll!0x14B2811()    Unknown
     binaryninjacore.dll!0x76F4EC()     Unknown
     binaryninjacore.dll!0x7D9BBF()     Unknown
     binaryninjacore.dll!0x14ECE6E()    Unknown
     binaryninjacore.dll!0x14EEF5D()    Unknown
     binaryninjaui.dll!0x4AC3A49()      Unknown
     binaryninjaui.dll!0x498315B()      Unknown
     Qt6Core.dll!00007fff2297a0da()     Unknown
     kernel32.dll!BaseThreadInitThunk() Unknown
     ntdll.dll!RtlUserThreadStart()     Unknown

The Log does not show anything about a crash :disappointed:
Version and Platform (required):

Binary Ninja Version: 4.3.6539-dev Personal (f984e126)
OS: Windows 10
OS Version: 22H2 (19045.5131)
CPU Architecture: x64

Update: The crash also happens sometimes when just writing in the popup window. I don't have a crashlog for that yet :(

Stacktrace for the crash when typing, happens every few minutes in the last hour :(
It looks like the exact same error than the one above. I guess it is an issue with the qt input handling? Really annoying for sure.
>	binaryninjacore.dll!00007fff4615a78c()	Unknown
 	binaryninjacore.dll!00007fff4615a05d()	Unknown
 	binaryninjacore.dll!00007fff412d2803()	Unknown
 	binaryninjacore.dll!00007fff421d1f51()	Unknown
 	binaryninjacore.dll!00007fff4148ec2c()	Unknown
 	binaryninjacore.dll!00007fff414f92ff()	Unknown
 	binaryninjacore.dll!00007fff4220c5ae()	Unknown
 	binaryninjacore.dll!00007fff4220e69d()	Unknown
 	binaryninjaui.dll!00007fff8ca54b39()	Unknown
 	binaryninjaui.dll!00007fff8c91424b()	Unknown
 	Qt6Core.dll!00007fff8b6aa0da()	Unknown
 	kernel32.dll!BaseThreadInitThunk()	Unknown
 	ntdll.dll!RtlUserThreadStart()	Unknown

binaryninja.exe	binaryninja.exe	C:\Program Files\Vector35\BinaryNinja\binaryninja.exe	N/A	Yes	Cannot find or open the PDB file.		1	4.03.6541.0	06/12/2024 17:49	00007FF693630000-00007FF694C06000	[0x39E8] binaryninja.exe
binaryninjacore.dll	binaryninjacore.dll	C:\Program Files\Vector35\BinaryNinja\binaryninjacore.dll	N/A	Yes	Cannot find or open the PDB file.		45		06/12/2024 17:42	00007FFF40D20000-00007FFF4A297000	[0x39E8] binaryninja.exe
binaryninjaui.dll	binaryninjaui.dll	C:\Program Files\Vector35\BinaryNinja\binaryninjaui.dll	N/A	Yes	Cannot find or open the PDB file.		12		06/12/2024 17:47	00007FFF8C2D0000-00007FFF8D07F000	[0x39E8] binaryninja.exe

Thanks for the info, sorry this is happening to you. 😞  Are you using any plugins you could try disabling? Just curious because, as far as I'm aware, you're the only user reporting a crash like this.
Will see if we can have someone take a look at these logs this week.

More info on disabling plugins: https://docs.binary.ninja/guide/troubleshooting.html

When you say "While the VS debugger is paused" -- do you mean this only happens when you've attached vscode as a python debugger? Are you using the visual studio real debugger to debug binary ninja? Trying to better understand what's triggering this.
Also, definitely let us know if you're able to reproduce the crash without any plugins.

This crash happens without any debugger attached. I attached the debugger to get a proper stacktrace a few times, but all the other crashes where without.
The crash also happens without plugins.
Attached a small video-clip of the crash (was created a few days ago when i still had plugins enabled). It is also WITHOUT any kind of debugger. I press "Y" to change the type, the popup opens, binja gets unresponsive and closes after a few seconds.
https://github.com/user-attachments/assets/cc25f775-ea68-4f3a-8d84-19907c2af8c7

Have you added any user types or platform types to your user folder?
Additionally, can you try backing up your Qsettings (https://docs.binary.ninja/guide/index.html#qsettings-locations) and then using the "clear qsettings" action" to clear them and see if that impacts the frequency of crashing?

@knoxfighter does your workflow involve any type archives, especially ones created by you? If so, is there any chance you can share those with us (in public or in private). The stack trace suggests it is related to the type archive, and the crash you are seeing just pulls the trigger on it

V35 folks should search for Mountain Swift Candle Echo Breeze to find the user provided type archive file


all selection helpers tests pass

DO NOT MERGE, JUST REVIEW :D
This PR Addresses:

Added tests and made all tests pass for selectionHelpers in VerseCheck.
This does not include UI/browser text selection functions yet.

To Test:

cd into tC_apps/VerseCheck
npm install to setup the tool to run tests
npm test to run the tests
ensure all tests pass
review tests to see if new ones should be added.

This change is

Reviewed 8 of 8 files at r1.
Review status: all files reviewed at latest revision, all discussions resolved.

Comments from Reviewable


device implementation checklist additions

Take a look in particular at the OmniPod checklist and the Tandem pump one for substantive (i.e., non-template) additions. @gniezen @pazaan

Ooops, yeah, will merge those dups into the datatype-specific sections in Tandem.


Please add a constructor function for Docket.class

Here is the reason:
For now users can only register Docket using Spring @Bean annotation. And if you want to register the Docket by hand using BeanDefinition, you will fail because Docket creates instances using builder-mode, rather than the normal way (setter and getter). Spring context can not inject values for Docket's member variables.
@dilipkrish

To solve this problem for temporary, I have to create a new class DocketWrapper to enhance the Docket class in my code. Following are the codes, just for reference to those people who encounter the same issue.
       @Data
	public static class DocketInfo{
		private String groupName;
		private ApiInfo apiInfo;
		private String apiPackagePath;
		private Tag [] tags;
	}

	public static class DocketWrapper extends Docket{

		public DocketWrapper(DocketInfo docketInfo) {
			super(dt);
			super.apiInfo(docketInfo.getApiInfo())
            .groupName(docketInfo.getGroupName())
            .forCodeGeneration(true)
            .useDefaultResponseMessages(false)
            .select()
                .apis(RequestHandlerSelectors.basePackage(docketInfo.getApiPackagePath()))
                .apis(RequestHandlerSelectors.withMethodAnnotation(ApiOperation.class))
                .build()
            .tags(docketInfo.tags[0], copyArray(docketInfo.tags, 1, docketInfo.tags.length));
		}
	}

Note: @Data is a lombok annotation, you can remove it as you like and use getter and setter instead.

@tbwork that is what I would do as well. Thanks for the solution!

Has it been implemented?

With functional bean creation supported in spring 5 is this even needed?

OK. BTW, providing a easy way to create Docket bean would be nice.

What in your mind is an easy way. There is a default docket if one isn't specified and there is a builder to customize its creation. It has too many options to use a meaningful constructor. Open to ideas

This is simplely my method to optimize the usage of swagger in terms of convenience.
1. Define a new swagger definition annotation:
/**
 * Please note that one module is corresponding to one controller.
 * This annotation is used to define a module for a certain group of APIs.
 *
 * @author tbwork
 */
@Target(value = {ElementType.TYPE})
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Inherited
public @interface SwaggerModule {

	/**
	 * The name of module. Please notice that the module names of different
	 * controllers should also differ from each other. The name will be shown
	 * in the "spec" drop-list options on the swagger-ui.html page.
	 */
	public String name() default "";

	/**
	 * The summary title of APIs in this module.
	 */
	public String title() default "";

	/**
	 * The detailed description of this module.
	 */
	public String description() default "";

	/**
	 * The version of APIs.
	 */
	public String version() default "New" ;

	/**
	 * Tag this module with summary words so that users can find them easily
	 * and friendly.
	 */
	public String [] tags() default {};

	/**
	 * The author or person in charge of this module. Please provider the full
	 * Chinese name so that others can contact you easily.
	 */
	public String author() default "stranger";

	/**
	 * The email of the author. Other users can contact the author by this email.
	 */
	public String email() default "";

}

## 2. Providing an annotation as "SwaggerConfiguration", and use it to specify one package path for searching those swagger modules. Following is only an example:

private void collect() {
	List<Class<?>> docketClasses = getClassWithSpecifiedAnnotation(SwaggerModule.class);
	for (Class<?> klass : docketClasses) {
		SwaggerModule annotation = klass.getAnnotation(SwaggerModule.class);
		addDocket(annotation.name(), annotation.title(), annotation.description(),
				  annotation.version(), annotation.author(), annotation.email(), klass,
				  annotation.tags());
	}
}

private List<Class<?>> getClassWithSpecifiedAnnotation(Class<? extends Annotation> annotation){
	List<Class<?>> result = new ArrayList<Class<?>>();
	ClassPathScanningCandidateComponentProvider scanner = new ClassPathScanningCandidateComponentProvider(false);
    scanner.addIncludeFilter(new AnnotationTypeFilter(annotation));
    for (BeanDefinition bd : scanner.findCandidateComponents("xxx.xxxx")) {
    	try {
			result.add(Class.forName(bd.getBeanClassName()));
		} catch (ClassNotFoundException e) {
			// TODO Auto-generated catch block
			throw new RuntimeException(e.getMessage());
		}
    }
    return result;
}

Hope it is helpful. Have a nice day!


Auth during sign up doesn't work when AdBlock is on

Github, Facebook, Github all showed a toast that said "A network error has occurred"

Moved to SMS Login


extend certificates with 100Y

Description
Extend certificates  with approx 100Y
Self Check:
Strike through any lines that are not applicable (~~line~~) then check the box

[ ] Attached issue to pull request
[x] Changelog entry
[ ] Type annotations are present
[ ] Code is clear and sufficiently documented
[ ] No (preventable) type errors (check using make mypy or make mypy-diff)
[ ] Sufficient test cases (reproduces the bug/tests the requested feature)
[x] Correct, in line with design
[ ] End user documentation is included or an issue is created for end-user documentation (add ref to issue here: )

Processing this pull request

Pull request rejected by merge tool. No change entry file found

Processing this pull request

Processing #3424.

Processing #3423.


[SPARK-25117][R] Add EXEPT ALL and INTERSECT ALL support in R

What changes were proposed in this pull request?
SPARK-21274 added support for EXCEPT ALL and INTERSECT ALL. This PR adds the support in R.
How was this patch tested?
Added test in test_sparkSQL.R

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2184/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2188/
Test PASSed.

Test build #94764 has started for PR 22107 at commit 7e88c9d.

Test build #94760 has finished for PR 22107 at commit 426ffed.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94760/
Test PASSed.

Test build #94764 has finished for PR 22107 at commit 7e88c9d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94764/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2196/
Test PASSed.

Test build #94777 has started for PR 22107 at commit 5247ab5.

Test build #94777 has finished for PR 22107 at commit 5247ab5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94777/
Test PASSed.

Merged build finished. Test PASSed.

Seems fine.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2203/
Test PASSed.

Merged build finished. Test PASSed.

Test build #94789 has started for PR 22107 at commit 1d93304.

Test build #94789 has finished for PR 22107 at commit 1d93304.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94789/
Test FAILed.

retest this please

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2204/
Test PASSed.

Test build #94791 has started for PR 22107 at commit 1d93304.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2205/
Test PASSed.

Test build #94791 has finished for PR 22107 at commit 1d93304.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94791/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2240/
Test PASSed.

Test build #94844 has started for PR 22107 at commit 528050d.

Test build #94844 has finished for PR 22107 at commit 528050d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94844/
Test PASSed.

@felixcheung I have incorporated the comments.

merged to master


Broken inside v-for loop

Hey,
I'm having an issue using the component inside an v-for loop of a reactive property.
I have a main app component with a data that returns axes: [], and I bind those on my "axes" component: <Axes v-bind:axes="axes" />.
Inside the Axes component axes is defined as a prop, in its template I am using:
<div v-for="axis in axes" v-bind:key="axis.tag">
        AXIS {{ axis }}

        <RangeSlider
          min="axis.min"
          max="axis.max"
          step="1"
          v-model="axis.value"
        >
        </RangeSlider>

        <input type="number" v-model="axis.value" min="axis.min" max="axis.max" />

        val: {{ axis.value }}

</div>

When the app axes prop updates to an array of several axis objects with min, max etc. attributes, the inputs in my Axes component work correctly and generated for however many axes there are, and they are updating from the model, yet the RangeSlider does not initiate correctly not does it update — although they both use the same axis.value as model inside the v-for. The RangeSlider UI gets rendered, but the knob is not functional, and the axis.value gets output as NaN when dragging the knob. Updating the input field does not update the RangeSlider knob position.
Any idea why this would not be working?

Not a 100% sure how this all figures, but making the component props axes validate as Array solve the linking and they now render and are reactive. Maybe when initially the prop is an empty array, the for loop initiates the RangeSliders in an odd way which causes later updating and re-rendering as loop over an actual Array to fail, somehow.

Not a 100% sure how this all figures, but making the component props axes validate as Array solve the linking and they now render and are reactive. Maybe when initially the prop is an empty array, the for loop initiates the RangeSliders in an odd way which causes later updating and re-rendering as loop over an actual Array to fail, somehow.