[BUG] ERA5 downloader failing (dev branch)

download-toolbox version: v0.0.2 (icenet_0.4 branch)
Python version: [3.9.20, 3.12.7]
Operating System: Rocky 9

Description
Attempting to use download_era5.
What I Did
❯ download_era5 north 2020-1-1 2020-1-1 uas,vas ',' -f DAY -o DAY -d
[13-11-24 00:01:09 :INFO    ] - ERA5 Data Downloading
[13-11-24 00:01:09 :INFO    ] - Creating path: ./data/era5/day/hemi.north
[13-11-24 00:01:09 :INFO    ] - Downloading between 2020-01-01 and 2020-01-01
2024-11-13 00:01:09,448 INFO [2024-09-28T00:00:00] **Welcome to the New Climate Data Store (CDS)!** This new system is in its early days of full operations and still undergoing enhancements and fine tuning. Some disruptions are to be expected. Your
[feedback](https://jira.ecmwf.int/plugins/servlet/desk/portal/1/create/202) is key to improve the user experience on the new CDS for the benefit of everyone. Thank you.
[13-11-24 00:01:09 :INFO    ] - [2024-09-28T00:00:00] **Welcome to the New Climate Data Store (CDS)!** This new system is in its early days of full operations and still undergoing enhancements and fine tuning. Some disruptions are to be expected. Your
[feedback](https://jira.ecmwf.int/plugins/servlet/desk/portal/1/create/202) is key to improve the user experience on the new CDS for the benefit of everyone. Thank you.
2024-11-13 00:01:09,448 INFO [2024-09-26T00:00:00] Watch our [Forum](https://forum.ecmwf.int/) for Announcements, news and other discussed topics.
[13-11-24 00:01:09 :INFO    ] - [2024-09-26T00:00:00] Watch our [Forum](https://forum.ecmwf.int/) for Announcements, news and other discussed topics.
2024-11-13 00:01:09,449 INFO [2024-09-16T00:00:00] Remember that you need to have an ECMWF account to use the new CDS. **Your old CDS credentials will not work in new CDS!**
[13-11-24 00:01:09 :INFO    ] - [2024-09-16T00:00:00] Remember that you need to have an ECMWF account to use the new CDS. **Your old CDS credentials will not work in new CDS!**
2024-11-13 00:01:09,449 WARNING [2024-06-16T00:00:00] CDS API syntax is changed and some keys or parameter names may have also changed. To avoid requests failing, please use the "Show API request code" tool on the dataset Download Form to check you are using the correct syntax for your API request.
[13-11-24 00:01:09 :WARNING ] - [2024-06-16T00:00:00] CDS API syntax is changed and some keys or parameter names may have also changed. To avoid requests failing, please use the "Show API request code" tool on the dataset Download Form to check you are using the correct syntax for your API request.
[13-11-24 00:01:09 :INFO    ] - Request frequency set to DAY
[13-11-24 00:01:09 :INFO    ] - Building request(s), downloading and averaging from ERA5 API
[13-11-24 00:01:09 :INFO    ] - Filtering 1 dates against 1 destination files
[13-11-24 00:01:09 :INFO    ] - Processing single download for uas with 1 dates
[13-11-24 00:01:09 :INFO    ] - Filtering 1 dates against 1 destination files
[13-11-24 00:01:09 :INFO    ] - Processing single download for vas with 1 dates
[13-11-24 00:01:09 :INFO    ] - Creating thread pool with 2 workers to service 2 batches
[13-11-24 00:01:09 :ERROR   ] - Thread failure: Cannot compare Timestamp with datetime.date. Use ts == pd.Timestamp(date) or ts.date() == date instead.
Traceback (most recent call last):
  File "/data/hpcdata/users/username/git/environmental-forecasting/download-toolbox/download_toolbox/download.py", line 198, in download
    files_downloaded = future.result()
                       ^^^^^^^^^^^^^^^
  File "/data/hpcdata/users/username/miniconda3/envs/canari_ml/lib/python3.12/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/data/hpcdata/users/username/miniconda3/envs/canari_ml/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/data/hpcdata/users/username/miniconda3/envs/canari_ml/lib/python3.12/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/hpcdata/users/username/git/environmental-forecasting/download-toolbox/download_toolbox/data/cds.py", line 130, in _single_api_download
    req_dates = [date for date in req_dates if date <= date_end]
                                               ^^^^^^^^^^^^^^^^
  File "timestamps.pyx", line 377, in pandas._libs.tslibs.timestamps._Timestamp.__richcmp__
TypeError: Cannot compare Timestamp with datetime.date. Use ts == pd.Timestamp(date) or ts.date() == date instead.
[13-11-24 00:01:09 :ERROR   ] - Thread failure: Cannot compare Timestamp with datetime.date. Use ts == pd.Timestamp(date) or ts.date() == date instead.
Traceback (most recent call last):
  File "/data/hpcdata/users/username/git/environmental-forecasting/download-toolbox/download_toolbox/download.py", line 198, in download
    files_downloaded = future.result()
                       ^^^^^^^^^^^^^^^
  File "/data/hpcdata/users/username/miniconda3/envs/canari_ml/lib/python3.12/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/data/hpcdata/users/username/miniconda3/envs/canari_ml/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/data/hpcdata/users/username/miniconda3/envs/canari_ml/lib/python3.12/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/hpcdata/users/username/git/environmental-forecasting/download-toolbox/download_toolbox/data/cds.py", line 130, in _single_api_download
    req_dates = [date for date in req_dates if date <= date_end]
                                               ^^^^^^^^^^^^^^^^
  File "timestamps.pyx", line 377, in pandas._libs.tslibs.timestamps._Timestamp.__richcmp__
TypeError: Cannot compare Timestamp with datetime.date. Use ts == pd.Timestamp(date) or ts.date() == date instead.
[13-11-24 00:01:09 :INFO    ] - 0 files downloaded
[13-11-24 00:01:09 :WARNING ] - No data provided as data object or source files, not doing anything

Commented out the offending statement to proceed:
https://github.com/environmental-forecasting/download-toolbox/blob/303221188ab4dcfd9a5e30e6d776c94d593c9111/download_toolbox/data/cds.py#L125
But, getting a Seg Fault:
[13-11-24 00:08:51 :INFO    ] - status has been updated to successful
[13-11-24 00:08:51 :INFO    ] - Downloading https://object-store.os-api.cci2.ecmwf.int:443/cci2-prod-cache/de58026bc2cfd205521e09f4891f8b2e.nc
[13-11-24 00:08:55 :INFO    ] - Download completed: ./data/era5/uas/hemi.north/temp.20200101.nc
[13-11-24 00:08:55 :WARNING ] - expver in coordinates, new cdsapi returns ERA5 and ERA5T combined, this needs further work: expver needs storing for later overwriting
[13-11-24 00:08:57 :INFO    ] - 1 files downloaded
2024-11-13 00:08:58,943 INFO status has been updated to running
[13-11-24 00:08:58 :INFO    ] - status has been updated to running
2024-11-13 00:09:10,379 INFO status has been updated to successful
[13-11-24 00:09:10 :INFO    ] - status has been updated to successful
[13-11-24 00:09:10 :INFO    ] - Downloading https://object-store.os-api.cci2.ecmwf.int:443/cci2-prod-cache/49cf5a1fe86f6be07c82704c53bb2d8b.nc
[13-11-24 00:09:13 :INFO    ] - Download completed: ./data/era5/vas/hemi.north/temp.20200101.nc
[13-11-24 00:09:13 :WARNING ] - expver in coordinates, new cdsapi returns ERA5 and ERA5T combined, this needs further work: expver needs storing for later overwriting
[13-11-24 00:09:15 :INFO    ] - 1 files downloaded
[13-11-24 00:09:15 :INFO    ] - 2 files downloaded
Segmentation fault (core dumped)

It did manage to download the data before segfault.
❯ tree data/
data/
└── [   55]  era5/
    ├── [   32]  day/
    │   └── [   40]  hemi.north/
    │       ├── [   10]  uas/
    │       └── [   10]  vas/
    ├── [   32]  uas/
    │   └── [   33]  hemi.north/
    │       └── [  31M]  20200101.nc
    └── [   32]  vas/
        └── [   33]  hemi.north/
            └── [  31M]  20200101.nc

9 directories, 2 files

If I run the command again, it runs without segfault, and outputs a config json file, but it refers to non-existent files:
data/
└── [  101]  era5/
    ├── [   32]  day/
    │   └── [   40]  hemi.north/
    │       ├── [   10]  uas/
    │       └── [   10]  vas/
    ├── [   32]  uas/
    │   └── [   33]  hemi.north/
    │       └── [  31M]  20200101.nc
    ├── [   32]  vas/
    │   └── [   33]  hemi.north/
    │       └── [  31M]  20200101.nc
    └── [ 1.2K]  dataset_config.day.hemi.north.json

9 directories, 3 files

Subset of dataset_config.day.hemi.north.json:
    "_var_files": {
      "uas": [
        "./data/era5/day/hemi.north/uas/20200101.nc"
      ],
      "vas": [
        "./data/era5/day/hemi.north/vas/20200101.nc"
      ]
    },

Hi @bnubald maybe try doing one at a time, -w 1 to see what's going on. The segfault is not something I've seen before, it could be a library compatibility issue: the Rocky 9 installation is a difference that could have a discernible effect from testing I've done before.

Just thinking. Make sure the call to get_era5_available_date_range is not happening whilst the caller has the files open, also making sure that there is not a chance that it's picking up the wrong file and causing any threading issues: I can now fathom that there's a potential issue with multiple threads trying to open the same file for different batches if the logic isn't right 😉

Hi @bnubald maybe try doing one at a time, -w 1 to see what's going on. The segfault is not something I've seen before, it could be a library compatibility issue: the Rocky 9 installation is a difference that could have a discernible effect from testing I've done before.

Tried on bslws??, but same segfault error (including -w 1 flag on both CentOS and Rocky 9) - though, am using conda/mamba.

Hi @bnubald maybe try doing one at a time, -w 1 to see what's going on. The segfault is not something I've seen before, it could be a library compatibility issue: the Rocky 9 installation is a difference that could have a discernible effect from testing I've done before.

Tried on bslws??, but same segfault error (including -w 1 flag on both CentOS and Rocky 9) - though, am using conda/mamba.

Interesting, getting different behaviour with -w 1 flag on CentOS vs Rocky 9 as you'd guessed:
❯ download_era5 north 2020-1-1 2020-1-1 uas,vas ',' -f DAY -o DAY -d -w 1

Rocky 9
[13-11-24 09:53:26 :WARNING ] - expver in coordinates, new cdsapi returns ERA5 and ERA5T combined, this needs further work: expver needs storing for later overwriting
[13-11-24 09:53:27 :INFO    ] - 1 files downloaded
[13-11-24 09:53:27 :INFO    ] - 2 files downloaded
Traceback (most recent call last):
  File "/data/hpcdata/users/username/miniconda3/envs/canari_ml_3.9/lib/python3.9/site-packages/xarray/backends/file_manager.py", line 211, in _acquire_with_cache_info
    file = self._cache[self._key]
  File "/data/hpcdata/users/username/miniconda3/envs/canari_ml_3.9/lib/python3.9/site-packages/xarray/backends/lru_cache.py", line 56, in __getitem__
    value = self._cache[key]
KeyError: [<class 'netCDF4._netCDF4.Dataset'>, ('/data/hpcdata/users/username/git/CANARI-ML/CANARI-ML-notebooks/data_visualisation/data/era5/vas/hemi.north/20200101.nc',), 'r', (('clobber', True), ('diskless', False), ('format', 'NETCDF4'), ('persist', False)), '94d36a9c-d769-4657-900c-20c3eb5bfd9b']

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/hpcdata/users/username/miniconda3/envs/canari_ml_3.9/bin/download_era5", line 8, in <module>
    sys.exit(main())
  File "/data/hpcdata/users/username/git/environmental-forecasting/download-toolbox/download_toolbox/data/cds.py", line 270, in main
    dataset.save_data_for_config(
  File "/data/hpcdata/users/username/git/environmental-forecasting/download-toolbox/download_toolbox/dataset.py", line 239, in save_data_for_config
    ds = xr.open_mfdataset(source_files,
  File "/data/hpcdata/users/username/miniconda3/envs/canari_ml_3.9/lib/python3.9/site-packages/xarray/backends/api.py", line 1085, in open_mfdataset
    datasets, closers = dask.compute(datasets, closers)
  File "/data/hpcdata/users/username/miniconda3/envs/canari_ml_3.9/lib/python3.9/site-packages/dask/base.py", line 662, in compute
    results = schedule(dsk, keys, **kwargs)
  File "/data/hpcdata/users/username/miniconda3/envs/canari_ml_3.9/lib/python3.9/site-packages/xarray/backends/api.py", line 588, in open_dataset
    backend_ds = backend.open_dataset(
  File "/data/hpcdata/users/username/miniconda3/envs/canari_ml_3.9/lib/python3.9/site-packages/xarray/backends/netCDF4_.py", line 645, in open_dataset
    store = NetCDF4DataStore.open(
  File "/data/hpcdata/users/username/miniconda3/envs/canari_ml_3.9/lib/python3.9/site-packages/xarray/backends/netCDF4_.py", line 408, in open
    return cls(manager, group=group, mode=mode, lock=lock, autoclose=autoclose)
  File "/data/hpcdata/users/username/miniconda3/envs/canari_ml_3.9/lib/python3.9/site-packages/xarray/backends/netCDF4_.py", line 355, in __init__
    self.format = self.ds.data_model
  File "/data/hpcdata/users/username/miniconda3/envs/canari_ml_3.9/lib/python3.9/site-packages/xarray/backends/netCDF4_.py", line 417, in ds
    return self._acquire()
  File "/data/hpcdata/users/username/miniconda3/envs/canari_ml_3.9/lib/python3.9/site-packages/xarray/backends/netCDF4_.py", line 411, in _acquire
    with self._manager.acquire_context(needs_lock) as root:
  File "/data/hpcdata/users/username/miniconda3/envs/canari_ml_3.9/lib/python3.9/contextlib.py", line 119, in __enter__
    return next(self.gen)
  File "/data/hpcdata/users/username/miniconda3/envs/canari_ml_3.9/lib/python3.9/site-packages/xarray/backends/file_manager.py", line 199, in acquire_context
    file, cached = self._acquire_with_cache_info(needs_lock)
  File "/data/hpcdata/users/username/miniconda3/envs/canari_ml_3.9/lib/python3.9/site-packages/xarray/backends/file_manager.py", line 217, in _acquire_with_cache_info
    file = self._opener(*self._args, **kwargs)
  File "src/netCDF4/_netCDF4.pyx", line 2521, in netCDF4._netCDF4.Dataset.__init__
  File "src/netCDF4/_netCDF4.pyx", line 2158, in netCDF4._netCDF4._ensure_nc_success
OSError: [Errno -101] NetCDF: HDF error: '/data/hpcdata/users/username/git/CANARI-ML/CANARI-ML-notebooks/data_visualisation/data/era5/vas/hemi.north/20200101.nc'

CentOS 7.9.2009
[13-11-24 09:36:59 :WARNING ] - expver in coordinates, new cdsapi returns ERA5 and ERA5T combined, this needs further work: expver needs storing for later overwriting
[13-11-24 09:37:01 :INFO    ] - 1 files downloaded
[13-11-24 09:37:01 :INFO    ] - 2 files downloaded
Segmentation fault

That's really odd. I'd start without the existing files in there, but I don't think it'll make a difference. Something wonky is going on and I can't tell you off the top of my head, but one might be indicative of the other. Happy to chat, but this isn't something I've seen before.

Hm... I did clear out previous downloads before running the command each time.
Chat when you have some time?

Based on offline discussion:
Was able to run through using the conda env file in icenet-pipeline here, but downgrading netCDF to older version.
Without downgrading (it installs netCDF=1.7.2), get following error:
[13-11-24 11:30:21 :INFO    ] - Saved dataset config ./data/era5/dataset_config.day.hemi.north.json
Exception ignored in: <function CachingFileManager.__del__ at 0x7fc76b105d30>
Traceback (most recent call last):
  File "/data/hpcdata/users/username/miniconda3/envs/canari_ml_3.9_icn_pipeline/lib/python3.9/site-packages/xarray/backends/file_manager.py", line 250, in __del__
    self.close(needs_lock=False)
  File "/data/hpcdata/users/username/miniconda3/envs/canari_ml_3.9_icn_pipeline/lib/python3.9/site-packages/xarray/backends/file_manager.py", line 234, in close
    file.close()
  File "src/netCDF4/_netCDF4.pyx", line 2669, in netCDF4._netCDF4.Dataset.close
  File "src/netCDF4/_netCDF4.pyx", line 2636, in netCDF4._netCDF4.Dataset._close
  File "src/netCDF4/_netCDF4.pyx", line 2164, in netCDF4._netCDF4._ensure_nc_success
RuntimeError: NetCDF: Not a valid ID

Updated environment file with netCDF pinned to 1.6.0:
channels:
  - conda-forge
dependencies:
  - cartopy
  - cudatoolkit
  - cudnn
  - eccodes
  - ffmpeg
  - hdf5
  - ipykernel
  - netcdf4=1.6.0
  - openh264
  - python=3.9
  - xarray

With this, its working!


how to transform obj line（l）to gltf

I have some models ,they are made up of lines，obj line（l）.Obj2gltf can't do this work,do you have some better methods.

I know COLLADA2GLTF has support for lines so if you are able to convert your obj to COLLADA and then run it through that tool you should get a glTF with lines in it.
I'm going to close this for now because there's another issue open about adding line support in obj2gltf. See https://github.com/AnalyticalGraphicsInc/obj2gltf/issues/106.


Cannot start dev: Error -5640 in supR3HardenedWinReSpawn

Environment:
VirtualBox 5.2.10
Windows 10 host
pcfdev-v0.30.0+PCF1.11.0
Issuing the command: cf dev start fails with:
Error: failed to start VM: failed to execute 'C:\Program Files\Oracle\VirtualBox\VBoxManage startvm pcfdev-v0.549.0 --type headless': exit status 1: Waiting for VM "pcfdev-v0.549.0" to power on...
VBoxManage.exe: error: The virtual machine 'pcfdev-v0.549.0' has terminated unexpectedly during startup with exit code 1 (0x1).  More details may be available in 'C:\Users\paul.stanley.pcfdev\vms\pcfdev-v0.549.0\Logs\VBoxHardening.log'
VBoxManage.exe: error: Details: code E_FAIL (0x80004005), component MachineWrap, interface IMachine
The VBoxHardening.log contains this:
92c.730: supR3HardenedWinInitAppBin(0x0): '\Device\HarddiskVolume3\Program Files\Oracle\VirtualBox'
92c.730: Calling main()
92c.730: SUPR3HardenedMain: pszProgName=VBoxHeadless fFlags=0x0
92c.730: supR3HardenedWinInitAppBin(0x0): '\Device\HarddiskVolume3\Program Files\Oracle\VirtualBox'
92c.730: '\Device\HarddiskVolume3\Program Files\Oracle\VirtualBox\VBoxHeadless.exe' has no imports
92c.730: supHardenedWinVerifyImageByHandle: -> 0 (\Device\HarddiskVolume3\Program Files\Oracle\VirtualBox\VBoxHeadless.exe)
92c.730: SUPR3HardenedMain: Respawn #2
92c.730: Error (rc=-5640):
92c.730: More than one thread in process
92c.730: Error -5640 in supR3HardenedWinReSpawn! (enmWhat=1)
92c.730: More than one thread in process
c5c.fd0: supR3HardNtChildWaitFor[1]: Quitting: ExitCode=0x1 (rcNtWait=0x0, rcNt1=0x0, rcNt2=0x103, rcNt3=0x103, 21 ms, the end);
This looks similar to #188, but there is no resolution given for that issue.

Seems like your installation of Virtualbox is broken. Can you uninstall Virtualbox and try it with the latest Virtualbox 5.1.x? (not 5.2).
Also see: https://stackoverflow.com/questions/34778147/cannot-boot-windows-guest-in-virtualbox-without-kernel-module-error


Archive: [Bug] when archiving, archived person is still shown in the list

When archiving, archived person is still shown in the list
same for unarchiving with unarchived person.
Can only counteract when using the 'opposite command': archive x -> listarchive -> list  will produce the correct result and vice versa

This bug still happens when pairing with other commands
Suggested bug fix: Cache the command / getPredicate from filteredList
Look into ArchiveCommand::execute::model.updateFilteredList()l


print warning if no resolver files match glob

warn if no files match the glob pattern
would have saved me half an hour...

Thanks @abenhamdine for your contribution!
Unfortunately this is a con of glob pattern loading 😞
I think that we should throw error when no resolvers matched the pattern, as it's definitely error in the setup. So I will move this to #12, thanks anyway for pointing this issue 😉

yes, better to check globally, you're right.
it was more of a hint than a real soution.
thx for your response.


Add ngx_link_func_ctx_t.uri

if (r->uri.len > 0) may not be needed here, I'm not sure.

@vikaig
Thanks for the contribution, I don't see any issue with merging this request, just had no time to do a full integration test
Have you been using it on live?  Have you ran Test Nginx?

Now I switched to Lua, before that I used it live, I didn't conduct any Nginx tests

Do you mind to share any Issue that you are using with this module?

Unable to get request URI before this patch

hi @vikaig
Normally I won't implement it on there block due to every request will copy the uri data as it's not 100% necessary for every request.
instead, I implement a function as you need it, you may get it by calling the function.
void my_app_simple_get_uri(ngx_link_func_ctx_t *ctx) {
    ngx_link_func_log_info(ctx, "log from my_app_simple_get_uri");
    ngx_link_func_str_t uri;
    if (ngx_link_func_get_uri(ctx, &uri) == 0) {
        ngx_link_func_write_resp(
            ctx,
            200,
            "200 OK",
            "text/plain",
            uri.data,
            uri.len
        );
    } else {
        ngx_link_func_write_resp(
            ctx,
            404,
            "404 NOT FOUND",
            "text/plain",
            NULL,
            0
        );
    }
}


Vaults with no scheme restriction

User should be able to create a vault with any scheme (not just 2/3 or 3/5)

https://xd.adobe.com/view/489bcd8a-f669-4ecb-9d11-bd705d75a996-1dc8/screen/a3b691fd-e944-47ca-8c3d-bae959f1a837/


Assorted updates

Add Game Resolution column to tribe-gear.
Add favicon and theme colour.

Reviewed by Giga :)


Finished http tech report and made shrimp watch api

Theres the inspiration meme of the shrimp watch that I think is really funny included too. Have a good break!


Docs for configuration files

WIP
┆Issue is synchronized with this Asana task

I was going to suggest splitting things up a bit here, but it quickly got very complicated. So here is my suggestion:
Make a table listing all the different config files, and have columns for where they should be used, if they are required, and if they are provided by Dolittle when deployed in the platform.
So something like:

Configuration
Head
Runtime
Required
Provided

tenants.json
x
x
x
x

clients.json
x

x

Then we can describe each file once. Does that make sense?

Should we have a list for each of the configs describing each of the "<>" variables?