Update documentation to clarify terminology around block verification

Here is some of the confusion in this auditor note that we should resolve in our documentation:

Note that a checkpointed/finalized block can be a settled network upgrade or a block beyond the rollback/reorg limit. The state service processes the finalized block and queues it to be committed to state

This is the core of the confusion:

a settled network upgrade is specified as a network upgrade that has "social consensus": it has enough confirmations on enough nodes that it can't be rolled back
checkpointed blocks are a Zebra implementation detail: every release, we generate checkpoints for all finalized blocks in the chain, because those blocks can't be rolled back by Zebra, and are very unlikely to be be rolled back by zcashd. This includes settled network upgrades, but it also goes beyond them to include almost all settled blocks.
finalized blocks can come from checkpoints, or from a contextually verified block which is beyond the rollback limit
the state service processes:

checkpoint verified blocks direct to the finalized state
contextually verified blocks into the non-finalized state, and then into the finalized state once they reach the rollback limit (or if they are on a side chain that will never be committed to the finalized state, they get dropped)

(finalized blocks can be optionally saved to disk.)

All valid finalized blocks are saved to disk, that's what finalization means in Zebra. Saving to disk is not under user control, and it's not a choice the application makes automatically dynamically. The only thing that can stop a finalized block being saved to disk is invalid data, which causes a verification failure in the block commitment, authorization, or chain history roots.
Non-finalized blocks are saved to disk if they are part of the best chain, and beyond the rollback limit. If they are not part of the best chain and beyond the rollback limit, they are dropped.

the state service will validate and commit the non-finalized queued blocks whose parents have recently arrived

To do contextual validation, the state service will:

queue blocks whose ancestors have not arrived in the state yet
send queued blocks to the block write task after all their ancestors have been sent
validate blocks as they arrive in the block write task, as long as their ancestors were valid
commit valid blocks to the non-finalized state

Breadth-first order is not guaranteed, because a lower height block from another fork can arrive at any time. Forks are allowed from the finalized tip, and any non-finalized block.

For the first non-finalized block to be validated there must be at least 28 blocks in the finalized state.

The 28 most recent ancestor blocks must have been validated and committed by the state, regardless of whether they are finalized or non-finalized.

If an honest node fails to obtain enough finalized blocks from its peers during syncing, an attacker could send it a non-finalized block to verify, which will fail this assertion and crash the Zebra node.

This is incorrect, due to the confusion above.
Verification Order
The Zebra implementation chooses to verify blocks using checkpoints or semantic/contextual verification, based on their heights and its checkpoints. Checkpoint verification happens in strict height order, in both zebra-consensus and zebra-state. There is only one chain fork when checkpointing.
Semantic verification happens in parallel in the zebra-consensus verifiers, but contextual verification happens in chain order in zebra-state. Each chain fork is verified in height order, different forks can have different tip heights. (Different chain forks could also be contextually verified in parallel, but Zebra doesn't implement that.)
Guaranteed Chain Context
So blocks are not validated by the state until all their ancestors have been validated. If a contextually verified block's ancestors aren't validated, it will wait in the queue and timeout, or be rejected with an error when it is received by the block write task. So validate_and_commit_non_finalized() won't be called unless a block has at least 28 ancestors, because it is only called during contextual validation. And before contextual validation starts, Zebra commits at least 1 million blocks using checkpoints.
No Cross-Module Dependencies
In most cases, the syncer or gossip downloaders will reject blocks that are too far in front of the tip. This prevents CPU and memory denial of service. But peers can't make Zebra panic by submitting blocks out of order, even if they convince these services to submit those blocks by the state. If that happens, the state will timeout or reject the block.
Originally posted by @teor2345 in https://github.com/ZcashFoundation/zebra/discussions/6620#discussioncomment-5841629

Hey team! Please add your planning poker estimate with Zenhub @arya2 @conradoplg @dconnolly @oxarbitrage @teor2345 @upbqdn

@mpguerra I think these changes will be easier after the renames in #6680, they are also likely to cause merge conflicts if we do them at the same time.

What do we actually need to do for this issue?

I think we should finish the missed renames in #6793 first, then add the information in this ticket to the state requests/service/queues/non-finalized state/finalized state. (Wherever each paragraph is most relevant.)

[ ] This documentation on CheckpointVerifiedBlock is incorrect, the note below it lists the checks that are actually required:
https://github.com/ZcashFoundation/zebra/blob/2e379811c3a8f4dd74c8e0e735738dcbe52d4288/zebra-state/src/request.rs#L214-L215

The documentation and variable names in these types and their methods:

[ ] Request
[ ] StateService (and service.rs)

Uses:

[ ] "non-finalized block verifier/verification" where it should use "semantic block verifier/verification"
[ ] "non-finalized block[s]" where it should use "semantically-verified block[s]"
[ ] "finalized block[s]" where it should use "checkpoint-verified block[s]"
[ ] finalized (by itself as a variable name) where it should use checkpoint_verified

Some uses are correct, if they are referring to a block that has actually been committed to the state, a queue, a chain, or a state. But most of them need to be manually updated.

What do we actually need to do for this issue?

The verification parts of the request, state service, and consensus rule checks use incorrect terms that are very confusing for developers and auditors. The storage parts use those terms correctly and should be left alone.

The documentation and variable names in these types and their methods:

@arya2 can you do a quick check of the list in this comment:
https://github.com/ZcashFoundation/zebra/issues/6681#issuecomment-1586389438
Feel free to change what we should rename, or what the new names should be.
If you think any of these renames can be done automatically, feel free to add them to #6793. But the old names have to be unique enough that there's no mistakes, and the replacement must be universal.


Support for PostgreSQL pg_basebackup

Currently, backup support PostgreSQL databases backup via pg_dump utility.
For backups that can be used for PITR, there is a different tool, called pg_basebackup (http://www.postgresql.org/docs/9.4/static/app-pgbasebackup.html)
I would like to add support for this type of backups, but a I need some guidance regarding how it should be added to the codebase: as a different database backup provider, or as an option to the existing postgresql.

We've found this workaround, in config.rb:
Utilities.configure do
   pg_dumpall '/usr/bin/pg_basebackup'
end

and in the model:
database PostgreSQL do |db|
  db.sudo_user = 'postgres'
  db.additional_options = ['-D -', '-Ft']
end

It seems to work fine, although the performance is a bit lower than just using pg_basebackup -D /path -Ft -z. Any hints on when can the performance be improved?

Hi @alexef, not really sure what would cause the performance to be lower. It can be the pg_basebackup tool itself, the amount of data backup has to process. You can narrow down if the problem is with backup by running it on the command line outside backup.


Add support for Iris 3.0

This (draft) PR adds support to IMPROVER for Iris 3.0 at the expense of supporting Iris 2.*. I have marked this PR as a draft so it can be used as a place for further discussion.
Here is a summary of the main changes:

removed usage of CubeList.extract_strict and use of strict keyword argument to CubeList.extract and replaced them with CubeList.extract_cube (for more, see here in the Iris docs changelog)
removed unavailable import from iris._cube_coord_common import LimitedAttributeDict which was introduced for type annotations
added a Conda environment file environment_py37_iris30.yml which has the same environment as environment_py37_iris24.yml except for replacing Iris 2.4 for 3.0
added this environment to GitHub Actions and removed Iris 2.* environments
updated the Iris version in setup.cfg
updated the readthedocs environment file
updated some docs

Further issues to be addressed:

[x] this PR only tests Python 3.7, whereas Iris 3.0 supports Python 3.6 and 3.8 as well (note: I could not build a Conda environment with the exactly pinned versions as environment_py37_iris24.yml with Iris 3.0 and Python 3.8)
[x] we may need to changed pinned (or want to unpin) package versions of cftime and scipy
[x] patching Iris for Nimrod support is no longer required (see this PR for more)

Addresses #1437, #1517
Testing:

[x] Ran tests and they passed OK

Currently the unit tests are passing in the py37_iris30 environment. However, I marked one test Test_lat_lon_determine in improver_tests/utilities/test_spatial.py as xfail. I did some digging but could not find why it was failing.

Although the unit tests are passing, 36 acceptance tests are failing with errors like:
_______________________ test_normal_point_by_point_sites _______________________
[gw4] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw4/test_normal_point_by_point_sit0')

    def test_normal_point_by_point_sites(tmp_path):
        """Test using a normal distribution when coefficients have been calculated
        independently at each site (initial guess and minimisation)."""
        kgo_dir = acc.kgo_root() / "apply-emos-coefficients/sites/point_by_point"
        kgo_path = kgo_dir / "kgo.nc"
        input_path = kgo_dir / ".." / "input.nc"
        emos_est_path = kgo_dir / "coefficients.nc"
        output_path = tmp_path / "output.nc"
        args = [
            input_path,
            emos_est_path,
            "--random-seed",
            "0",
            "--output",
            output_path,
        ]
        run_cli(args)
>       acc.compare(output_path, kgo_path, atol=LOOSE_TOLERANCE)
E       AssertionError: different attributes of /wmo_id - ['units']

improver_tests/acceptance/test_apply_emos_coefficients.py:99: AssertionError
----------------------------- Captured stdout call -----------------------------
improver apply-emos-coefficients /mnt/data/dmentipl/improver_tests/apply-emos-coefficients/sites/point_by_point/../input.nc /mnt/data/dmentipl/improver_tests/apply-emos-coefficients/sites/point_by_point/coefficients.nc --random-seed 0 --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw4/test_normal_point_by_point_sit0/output.nc

It seems some datasets are missing attributes, mostly units, e.g.
name is '/wmo_id'
desired is {'units': '1', 'long_name': 'wmo_id'}
actual is {'long_name': 'wmo_id'}

Here is the full output from pytest -n 16 -k 'acc':

============================= test session starts ==============================
platform linux -- Python 3.7.10, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: /home/dmentipl/repos/improver, configfile: pytest.ini, testpaths: improver_tests
plugins: cov-2.12.0, forked-1.3.0, xdist-2.2.1
gw0 I / gw1 I / gw2 I / gw3 I / gw4 I / gw5 I / gw6 I / gw7 I / gw8 I / gw9 I / gw10 I / gw11 I / gw12 I / gw13 I / gw14 I / gw15 I
gw0 [347] / gw1 [347] / gw2 [347] / gw3 [347] / gw4 [347] / gw5 [347] / gw6 [347] / gw7 [347] / gw8 [347] / gw9 [347] / gw10 [347] / gw11 [347] / gw12 [347] / gw13 [347] / gw14 [347] / gw15 [347]

..............F................F.....s.....s............................ [ 20%]
.........................F......F..F.....F.F....F..F..............F..... [ 41%]
.........F...FF........F.FF..F..F..FF....FFF..FF..FF..FssF........F..... [ 62%]
..........................x.......x..................................... [ 82%]
..........................F...........FF.....F...FF........              [100%]
=================================== FAILURES ===================================
__________________________________ test_basic __________________________________
[gw1] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw1/test_basic0')

    def test_basic(tmp_path):
        """Test basic create grid with halo"""
        kgo_dir = acc.kgo_root() / "create-grid-with-halo/basic"
        kgo_path = kgo_dir / "kgo.nc"
        input_path = kgo_dir / "source_grid.nc"
        output_path = tmp_path / "output.nc"
        args = [input_path, "--output", output_path]
        run_cli(args)
>       acc.compare(output_path, kgo_path)
E       AssertionError: different attributes of /grid_with_halo - ['units']

improver_tests/acceptance/test_create_grid_with_halo.py:50: AssertionError
----------------------------- Captured stdout call -----------------------------
improver create-grid-with-halo /mnt/data/dmentipl/improver_tests/create-grid-with-halo/basic/source_grid.nc --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw1/test_basic0/output.nc
_______________________ test_normal_point_by_point_sites _______________________
[gw4] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw4/test_normal_point_by_point_sit0')

    def test_normal_point_by_point_sites(tmp_path):
        """Test using a normal distribution when coefficients have been calculated
        independently at each site (initial guess and minimisation)."""
        kgo_dir = acc.kgo_root() / "apply-emos-coefficients/sites/point_by_point"
        kgo_path = kgo_dir / "kgo.nc"
        input_path = kgo_dir / ".." / "input.nc"
        emos_est_path = kgo_dir / "coefficients.nc"
        output_path = tmp_path / "output.nc"
        args = [
            input_path,
            emos_est_path,
            "--random-seed",
            "0",
            "--output",
            output_path,
        ]
        run_cli(args)
>       acc.compare(output_path, kgo_path, atol=LOOSE_TOLERANCE)
E       AssertionError: different attributes of /wmo_id - ['units']

improver_tests/acceptance/test_apply_emos_coefficients.py:99: AssertionError
----------------------------- Captured stdout call -----------------------------
improver apply-emos-coefficients /mnt/data/dmentipl/improver_tests/apply-emos-coefficients/sites/point_by_point/../input.nc /mnt/data/dmentipl/improver_tests/apply-emos-coefficients/sites/point_by_point/coefficients.nc --random-seed 0 --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw4/test_normal_point_by_point_sit0/output.nc
____________________________ test_nearest[uk-ukvx] _____________________________
[gw7] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw7/test_nearest_uk_ukvx_0')
domain = 'uk', model = 'ukvx'

    @pytest.mark.parametrize("domain,model", UK_GLOBAL)
    def test_nearest(tmp_path, domain, model):
        """Test basic neighbour finding"""
        kgo_dir = acc.kgo_root() / "neighbour-finding"
        kgo_path = kgo_dir / f"outputs/nearest_{domain}_kgo.nc"
        sites_path = kgo_dir / f"inputs/{domain}_sites.json"
        orography_path = kgo_dir / f"inputs/{model}_orography.nc"
        landmask_path = kgo_dir / f"inputs/{model}_landmask.nc"
        output_path = tmp_path / "output.nc"
        args = [orography_path, landmask_path, sites_path, "--output", output_path]
        run_cli(args)
>       acc.compare(output_path, kgo_path)
E       AssertionError: different attributes of /wmo_id - ['units']

improver_tests/acceptance/test_neighbour_finding.py:59: AssertionError
----------------------------- Captured stdout call -----------------------------
improver neighbour-finding /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/ukvx_orography.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/ukvx_landmask.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/uk_sites.json --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw7/test_nearest_uk_ukvx_0/output.nc
_________________________ test_nearest[global-global] __________________________
[gw7] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw7/test_nearest_global_global_0')
domain = 'global', model = 'global'

    @pytest.mark.parametrize("domain,model", UK_GLOBAL)
    def test_nearest(tmp_path, domain, model):
        """Test basic neighbour finding"""
        kgo_dir = acc.kgo_root() / "neighbour-finding"
        kgo_path = kgo_dir / f"outputs/nearest_{domain}_kgo.nc"
        sites_path = kgo_dir / f"inputs/{domain}_sites.json"
        orography_path = kgo_dir / f"inputs/{model}_orography.nc"
        landmask_path = kgo_dir / f"inputs/{model}_landmask.nc"
        output_path = tmp_path / "output.nc"
        args = [orography_path, landmask_path, sites_path, "--output", output_path]
        run_cli(args)
>       acc.compare(output_path, kgo_path)
E       AssertionError: different attributes of /wmo_id - ['units']

improver_tests/acceptance/test_neighbour_finding.py:59: AssertionError
----------------------------- Captured stdout call -----------------------------
improver neighbour-finding /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_orography.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_landmask.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_sites.json --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw7/test_nearest_global_global_0/output.nc
________________________________ test_halo_size ________________________________
[gw2] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw2/test_halo_size0')

    def test_halo_size(tmp_path):
        """Test basic create grid with halo"""
        kgo_dir = acc.kgo_root() / "create-grid-with-halo/halo_size"
        kgo_path = kgo_dir / "kgo.nc"
        input_path = kgo_dir / "../basic/source_grid.nc"
        output_path = tmp_path / "output.nc"
        args = [input_path, "--halo-radius", "75000", "--output", output_path]
        run_cli(args)
>       acc.compare(output_path, kgo_path)
E       AssertionError: different attributes of /grid_with_halo - ['units']

improver_tests/acceptance/test_create_grid_with_halo.py:61: AssertionError
----------------------------- Captured stdout call -----------------------------
improver create-grid-with-halo /mnt/data/dmentipl/improver_tests/create-grid-with-halo/halo_size/../basic/source_grid.nc --halo-radius 75000 --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw2/test_halo_size0/output.nc
______________________________ test_invalid_site _______________________________
[gw8] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw8/test_invalid_site0')

    def test_invalid_site(tmp_path):
        """Test invalid global site coordinates"""
        kgo_dir = acc.kgo_root() / "neighbour-finding"
        kgo_path = kgo_dir / "outputs/nearest_global_invalid_site_kgo.nc"
        sites_path = kgo_dir / "inputs/global_sites_invalid.json"
        orography_path = kgo_dir / "inputs/global_orography.nc"
        landmask_path = kgo_dir / "inputs/global_landmask.nc"
        output_path = tmp_path / "output.nc"
        args = [orography_path, landmask_path, sites_path, "--output", output_path]
        with pytest.warns(UserWarning, match=".*outside the grid.*"):
            run_cli(args)
>       acc.compare(output_path, kgo_path)
E       AssertionError: different attributes of /wmo_id - ['units']

improver_tests/acceptance/test_neighbour_finding.py:223: AssertionError
----------------------------- Captured stdout call -----------------------------
improver neighbour-finding /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_orography.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_landmask.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_sites_invalid.json --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw8/test_invalid_site0/output.nc
___________________________ test_coord_beyond_bounds ___________________________
[gw8] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw8/test_coord_beyond_bounds0')

    def test_coord_beyond_bounds(tmp_path):
        """Test coordinates up to 360 degrees"""
        kgo_dir = acc.kgo_root() / "neighbour-finding"
        kgo_path = kgo_dir / "outputs/nearest_global_kgo.nc"
        sites_path = kgo_dir / "inputs/global_sites_360.json"
        orography_path = kgo_dir / "inputs/global_orography.nc"
        landmask_path = kgo_dir / "inputs/global_landmask.nc"
        output_path = tmp_path / "output.nc"
        args = [orography_path, landmask_path, sites_path, "--output", output_path]
        run_cli(args)
>       acc.compare(output_path, kgo_path, exclude_vars=["longitude"])
E       AssertionError: different attributes of /wmo_id - ['units']

improver_tests/acceptance/test_neighbour_finding.py:236: AssertionError
----------------------------- Captured stdout call -----------------------------
improver neighbour-finding /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_orography.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_landmask.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_sites_360.json --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw8/test_coord_beyond_bounds0/output.nc
______________________ test_unset_wmo_ids_with_unique_ids ______________________
[gw8] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw8/test_unset_wmo_ids_with_unique0')

    def test_unset_wmo_ids_with_unique_ids(tmp_path):
        """Test sites with missing WMO ID numbers, but with valid unique IDs for
        all sites"""
        kgo_dir = acc.kgo_root() / "neighbour-finding"
        kgo_path = kgo_dir / "outputs/nearest_uk_kgo_some_unset_wmo_ids_unique_ids.nc"
        sites_path = kgo_dir / "inputs/uk_sites_missing_wmo_ids_with_unique_ids.json"
        orography_path = kgo_dir / "inputs/ukvx_orography.nc"
        landmask_path = kgo_dir / "inputs/ukvx_landmask.nc"
        output_path = tmp_path / "output.nc"
        args = [
            orography_path,
            landmask_path,
            sites_path,
            "--unique-site-id-key",
            "met_office_site_id",
            "--output",
            output_path,
        ]
        run_cli(args)
>       acc.compare(output_path, kgo_path)
E       AssertionError: different attributes of /wmo_id - ['units']

improver_tests/acceptance/test_neighbour_finding.py:258: AssertionError
----------------------------- Captured stdout call -----------------------------
improver neighbour-finding /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/ukvx_orography.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/ukvx_landmask.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/uk_sites_missing_wmo_ids_with_unique_ids.json --unique-site-id-key met_office_site_id --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw8/test_unset_wmo_ids_with_unique0/output.nc
__________________________ test_nearest_land[uk-ukvx] __________________________
[gw7] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw7/test_nearest_land_uk_ukvx_0')
domain = 'uk', model = 'ukvx'

    @pytest.mark.parametrize("domain,model", UK_GLOBAL)
    def test_nearest_land(tmp_path, domain, model):
        """Test neighbour finding with a land constraint"""
        kgo_dir = acc.kgo_root() / "neighbour-finding"
        kgo_path = kgo_dir / f"outputs/nearest_land_{domain}_kgo.nc"
        sites_path = kgo_dir / f"inputs/{domain}_sites.json"
        orography_path = kgo_dir / f"inputs/{model}_orography.nc"
        landmask_path = kgo_dir / f"inputs/{model}_landmask.nc"
        output_path = tmp_path / "output.nc"
        args = [
            orography_path,
            landmask_path,
            sites_path,
            "--land-constraint",
            "--output",
            output_path,
        ]
        run_cli(args)
>       acc.compare(output_path, kgo_path)
E       AssertionError: different attributes of /wmo_id - ['units']

improver_tests/acceptance/test_neighbour_finding.py:80: AssertionError
----------------------------- Captured stdout call -----------------------------
improver neighbour-finding /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/ukvx_orography.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/ukvx_landmask.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/uk_sites.json --land-constraint --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw7/test_nearest_land_uk_ukvx_0/output.nc
_______________________ test_nearest_land[global-global] _______________________
[gw7] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw7/test_nearest_land_global_globa0')
domain = 'global', model = 'global'

    @pytest.mark.parametrize("domain,model", UK_GLOBAL)
    def test_nearest_land(tmp_path, domain, model):
        """Test neighbour finding with a land constraint"""
        kgo_dir = acc.kgo_root() / "neighbour-finding"
        kgo_path = kgo_dir / f"outputs/nearest_land_{domain}_kgo.nc"
        sites_path = kgo_dir / f"inputs/{domain}_sites.json"
        orography_path = kgo_dir / f"inputs/{model}_orography.nc"
        landmask_path = kgo_dir / f"inputs/{model}_landmask.nc"
        output_path = tmp_path / "output.nc"
        args = [
            orography_path,
            landmask_path,
            sites_path,
            "--land-constraint",
            "--output",
            output_path,
        ]
        run_cli(args)
>       acc.compare(output_path, kgo_path)
E       AssertionError: different attributes of /wmo_id - ['units']

improver_tests/acceptance/test_neighbour_finding.py:80: AssertionError
----------------------------- Captured stdout call -----------------------------
improver neighbour-finding /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_orography.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_landmask.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_sites.json --land-constraint --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw7/test_nearest_land_global_globa0/output.nc
____________ test_normal_point_by_point_default_initial_guess_sites ____________
[gw13] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw13/test_normal_point_by_point_def0')

    @pytest.mark.slow
    def test_normal_point_by_point_default_initial_guess_sites(tmp_path):
        """
        Test estimate-emos-coefficients for diagnostic with assumed
        normal distribution where coefficients are computed independently at each
        site location (minimisation only).
        """
        kgo_dir = acc.kgo_root() / "estimate-emos-coefficients/normal/sites"
        kgo_path = kgo_dir / "point_by_point_default_initial_guess" / "kgo.nc"
        history_path = kgo_dir / "history/*.nc"
        truth_path = kgo_dir / "truth/*.nc"
        output_path = tmp_path / "output.nc"
        est_emos_tol = str(0.01)
        compare_emos_tolerance = 0.1
        args = [
            history_path,
            truth_path,
            "--distribution",
            "norm",
            "--truth-attribute",
            "mosg__model_configuration=uk_det",
            "--tolerance",
            est_emos_tol,
            "--point-by-point",
            "--use-default-initial-guess",
            "--output",
            output_path,
        ]
        run_cli(args)
        acc.compare(
>           output_path, kgo_path, atol=compare_emos_tolerance, rtol=compare_emos_tolerance
        )
E       AssertionError: different attributes of /wmo_id - ['units']

improver_tests/acceptance/test_estimate_emos_coefficients.py:366: AssertionError
----------------------------- Captured stdout call -----------------------------
improver estimate-emos-coefficients '/mnt/data/dmentipl/improver_tests/estimate-emos-coefficients/normal/sites/history/*.nc' '/mnt/data/dmentipl/improver_tests/estimate-emos-coefficients/normal/sites/truth/*.nc' --distribution norm --truth-attribute mosg__model_configuration=uk_det --tolerance 0.01 --point-by-point --use-default-initial-guess --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw13/test_normal_point_by_point_def0/output.nc
_______________________ test_nearest_minimum_dz[uk-ukvx] _______________________
[gw12] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw12/test_nearest_minimum_dz_uk_ukv0')
domain = 'uk', model = 'ukvx'

    @pytest.mark.slow
    @pytest.mark.parametrize("domain,model", UK_GLOBAL)
    def test_nearest_minimum_dz(tmp_path, domain, model):
        """Test neighbour finding with minimum altitude difference"""
        kgo_dir = acc.kgo_root() / "neighbour-finding"
        kgo_path = kgo_dir / f"outputs/nearest_minimum_dz_{domain}_kgo.nc"
        sites_path = kgo_dir / f"inputs/{domain}_sites.json"
        orography_path = kgo_dir / f"inputs/{model}_orography.nc"
        landmask_path = kgo_dir / f"inputs/{model}_landmask.nc"
        output_path = tmp_path / "output.nc"
        args = [
            orography_path,
            landmask_path,
            sites_path,
            "--similar-altitude",
            "--search-radius",
            "50000",
            "--output",
            output_path,
        ]
        run_cli(args)
>       acc.compare(output_path, kgo_path)
E       AssertionError: different attributes of /wmo_id - ['units']

improver_tests/acceptance/test_neighbour_finding.py:104: AssertionError
----------------------------- Captured stdout call -----------------------------
improver neighbour-finding /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/ukvx_orography.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/ukvx_landmask.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/uk_sites.json --similar-altitude --search-radius 50000 --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw12/test_nearest_minimum_dz_uk_ukv0/output.nc
__________________________ test_all_methods[uk-ukvx] ___________________________
[gw0] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw0/test_all_methods_uk_ukvx_0')
domain = 'uk', model = 'ukvx'

    @pytest.mark.slow
    @pytest.mark.parametrize("domain,model", UK_GLOBAL)
    def test_all_methods(tmp_path, domain, model):
        """Test neighbour finding using all methods"""
        kgo_dir = acc.kgo_root() / "neighbour-finding"
        kgo_path = kgo_dir / f"outputs/all_methods_{domain}_kgo.nc"
        sites_path = kgo_dir / f"inputs/{domain}_sites.json"
        orography_path = kgo_dir / f"inputs/{model}_orography.nc"
        landmask_path = kgo_dir / f"inputs/{model}_landmask.nc"
        output_path = tmp_path / "output.nc"
        args = [
            orography_path,
            landmask_path,
            sites_path,
            "--all-methods",
            "--output",
            output_path,
        ]
        run_cli(args)
>       acc.compare(output_path, kgo_path)
E       AssertionError: different attributes of /neighbour_selection_method_name - ['units']

improver_tests/acceptance/test_neighbour_finding.py:153: AssertionError
----------------------------- Captured stdout call -----------------------------
improver neighbour-finding /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/ukvx_orography.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/ukvx_landmask.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/uk_sites.json --all-methods --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw0/test_all_methods_uk_ukvx_0/output.nc
__ test_nearest_uk[all_methods_uk.nc-extra_args0-nearest_uk_temperatures.nc] ___
[gw14] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw14/test_nearest_uk_all_methods_uk0')
neighbour_cube = 'all_methods_uk.nc', extra_args = []
kgo_file = 'nearest_uk_temperatures.nc'

    @pytest.mark.parametrize(
        "neighbour_cube,extra_args,kgo_file",
        (
            ("all_methods_uk.nc", [], "nearest_uk_temperatures.nc"),
            ("all_methods_uk.nc", ["--similar-altitude"], "mindz_uk_temperatures.nc"),
            ("all_methods_uk_unique_ids.nc", [], "nearest_uk_temperatures_unique_ids.nc"),
        ),
    )
    def test_nearest_uk(tmp_path, neighbour_cube, extra_args, kgo_file):
        """Test spot extraction using nearest location"""
        kgo_dir = acc.kgo_root() / "spot-extract"
        neighbour_path = kgo_dir / f"inputs/{neighbour_cube}"
        diag_path = kgo_dir / "inputs/ukvx_temperature.nc"
        kgo_path = kgo_dir / f"outputs/{kgo_file}"
        output_path = tmp_path / "output.nc"
        args = [
            neighbour_path,
            diag_path,
            "--output",
            output_path,
            *extra_args,
            "--new-title",
            UK_SPOT_TITLE,
        ]
        run_cli(args)
>       acc.compare(output_path, kgo_path)
E       AssertionError: different attributes of /wmo_id - ['units']

improver_tests/acceptance/test_spot_extract.py:72: AssertionError
----------------------------- Captured stdout call -----------------------------
improver spot-extract /mnt/data/dmentipl/improver_tests/spot-extract/inputs/all_methods_uk.nc /mnt/data/dmentipl/improver_tests/spot-extract/inputs/ukvx_temperature.nc --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw14/test_nearest_uk_all_methods_uk0/output.nc --new-title 'IMPROVER UK Spot Values'
___ test_nearest_uk[all_methods_uk.nc-extra_args1-mindz_uk_temperatures.nc] ____
[gw14] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw14/test_nearest_uk_all_methods_uk1')
neighbour_cube = 'all_methods_uk.nc', extra_args = ['--similar-altitude']
kgo_file = 'mindz_uk_temperatures.nc'

    @pytest.mark.parametrize(
        "neighbour_cube,extra_args,kgo_file",
        (
            ("all_methods_uk.nc", [], "nearest_uk_temperatures.nc"),
            ("all_methods_uk.nc", ["--similar-altitude"], "mindz_uk_temperatures.nc"),
            ("all_methods_uk_unique_ids.nc", [], "nearest_uk_temperatures_unique_ids.nc"),
        ),
    )
    def test_nearest_uk(tmp_path, neighbour_cube, extra_args, kgo_file):
        """Test spot extraction using nearest location"""
        kgo_dir = acc.kgo_root() / "spot-extract"
        neighbour_path = kgo_dir / f"inputs/{neighbour_cube}"
        diag_path = kgo_dir / "inputs/ukvx_temperature.nc"
        kgo_path = kgo_dir / f"outputs/{kgo_file}"
        output_path = tmp_path / "output.nc"
        args = [
            neighbour_path,
            diag_path,
            "--output",
            output_path,
            *extra_args,
            "--new-title",
            UK_SPOT_TITLE,
        ]
        run_cli(args)
>       acc.compare(output_path, kgo_path)
E       AssertionError: different attributes of /wmo_id - ['units']

improver_tests/acceptance/test_spot_extract.py:72: AssertionError
----------------------------- Captured stdout call -----------------------------
improver spot-extract /mnt/data/dmentipl/improver_tests/spot-extract/inputs/all_methods_uk.nc /mnt/data/dmentipl/improver_tests/spot-extract/inputs/ukvx_temperature.nc --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw14/test_nearest_uk_all_methods_uk1/output.nc --similar-altitude --new-title 'IMPROVER UK Spot Values'
_________________________ test_lapse_rate_adjusted_uk __________________________
[gw1] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw1/test_lapse_rate_adjusted_uk0')

    def test_lapse_rate_adjusted_uk(tmp_path):
        """Test spot extraction with lapse rate adjustment"""
        kgo_dir = acc.kgo_root() / "spot-extract"
        neighbour_path = kgo_dir / "inputs/all_methods_uk.nc"
        diag_path = kgo_dir / "inputs/ukvx_temperature.nc"
        lapse_path = kgo_dir / "inputs/ukvx_lapse_rate.nc"
        kgo_path = kgo_dir / "outputs/lapse_rate_adjusted_uk_temperatures.nc"
        output_path = tmp_path / "output.nc"
        args = [
            neighbour_path,
            diag_path,
            lapse_path,
            "--output",
            output_path,
            "--apply-lapse-rate-correction",
            "--new-title",
            UK_SPOT_TITLE,
        ]
        run_cli(args)
>       acc.compare(output_path, kgo_path)
E       AssertionError: different attributes of /wmo_id - ['units']

improver_tests/acceptance/test_spot_extract.py:94: AssertionError
----------------------------- Captured stdout call -----------------------------
improver spot-extract /mnt/data/dmentipl/improver_tests/spot-extract/inputs/all_methods_uk.nc /mnt/data/dmentipl/improver_tests/spot-extract/inputs/ukvx_temperature.nc /mnt/data/dmentipl/improver_tests/spot-extract/inputs/ukvx_lapse_rate.nc --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw1/test_lapse_rate_adjusted_uk0/output.nc --apply-lapse-rate-correction --new-title 'IMPROVER UK Spot Values'
_ test_nearest_uk[all_methods_uk_unique_ids.nc-extra_args2-nearest_uk_temperatures_unique_ids.nc] _
[gw14] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw14/test_nearest_uk_all_methods_uk2')
neighbour_cube = 'all_methods_uk_unique_ids.nc', extra_args = []
kgo_file = 'nearest_uk_temperatures_unique_ids.nc'

    @pytest.mark.parametrize(
        "neighbour_cube,extra_args,kgo_file",
        (
            ("all_methods_uk.nc", [], "nearest_uk_temperatures.nc"),
            ("all_methods_uk.nc", ["--similar-altitude"], "mindz_uk_temperatures.nc"),
            ("all_methods_uk_unique_ids.nc", [], "nearest_uk_temperatures_unique_ids.nc"),
        ),
    )
    def test_nearest_uk(tmp_path, neighbour_cube, extra_args, kgo_file):
        """Test spot extraction using nearest location"""
        kgo_dir = acc.kgo_root() / "spot-extract"
        neighbour_path = kgo_dir / f"inputs/{neighbour_cube}"
        diag_path = kgo_dir / "inputs/ukvx_temperature.nc"
        kgo_path = kgo_dir / f"outputs/{kgo_file}"
        output_path = tmp_path / "output.nc"
        args = [
            neighbour_path,
            diag_path,
            "--output",
            output_path,
            *extra_args,
            "--new-title",
            UK_SPOT_TITLE,
        ]
        run_cli(args)
>       acc.compare(output_path, kgo_path)
E       AssertionError: different attributes of /wmo_id - ['units']

improver_tests/acceptance/test_spot_extract.py:72: AssertionError
----------------------------- Captured stdout call -----------------------------
improver spot-extract /mnt/data/dmentipl/improver_tests/spot-extract/inputs/all_methods_uk_unique_ids.nc /mnt/data/dmentipl/improver_tests/spot-extract/inputs/ukvx_temperature.nc --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw14/test_nearest_uk_all_methods_uk2/output.nc --new-title 'IMPROVER UK Spot Values'
_____________________________ test_new_spot_title ______________________________
[gw1] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw1/test_new_spot_title0')

    def test_new_spot_title(tmp_path):
        """Test spot extraction with external JSON metadata"""
        kgo_dir = acc.kgo_root() / "spot-extract"
        neighbour_path = kgo_dir / "inputs/all_methods_uk.nc"
        diag_path = kgo_dir / "inputs/ukvx_temperature.nc"
        kgo_path = kgo_dir / "outputs/nearest_uk_temperatures_amended_metadata.nc"
        output_path = tmp_path / "output.nc"
        args = [
            neighbour_path,
            diag_path,
            "--new-title",
            UK_SPOT_TITLE,
            "--output",
            output_path,
        ]
        run_cli(args)
>       acc.compare(output_path, kgo_path)
E       AssertionError: different attributes of /wmo_id - ['units']

improver_tests/acceptance/test_spot_extract.py:179: AssertionError
----------------------------- Captured stdout call -----------------------------
improver spot-extract /mnt/data/dmentipl/improver_tests/spot-extract/inputs/all_methods_uk.nc /mnt/data/dmentipl/improver_tests/spot-extract/inputs/ukvx_temperature.nc --new-title 'IMPROVER UK Spot Values' --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw1/test_new_spot_title0/output.nc
___________________________ test_lapse_rate_mismatch ___________________________
[gw14] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw14/test_lapse_rate_mismatch0')

    def test_lapse_rate_mismatch(tmp_path):
        """Test lapse rate adjustment mismatch between datasets"""
        kgo_dir = acc.kgo_root() / "spot-extract"
        neighbour_path = kgo_dir / "inputs/all_methods_uk.nc"
        diag_path = kgo_dir / "inputs/ukvx_temperature.nc"
        lapse_path = kgo_dir / "inputs/ukvx_lapse_rate_2m.nc"
        kgo_path = kgo_dir / "outputs/nearest_uk_temperatures.nc"
        output_path = tmp_path / "output.nc"
        args = [
            neighbour_path,
            diag_path,
            lapse_path,
            "--output",
            output_path,
            "--apply-lapse-rate-correction",
            "--new-title",
            UK_SPOT_TITLE,
        ]
        with pytest.warns(UserWarning, match=".*height.*not adjusted.*"):
            run_cli(args)
>       acc.compare(output_path, kgo_path)
E       AssertionError: different attributes of /wmo_id - ['units']

improver_tests/acceptance/test_spot_extract.py:139: AssertionError
----------------------------- Captured stdout call -----------------------------
improver spot-extract /mnt/data/dmentipl/improver_tests/spot-extract/inputs/all_methods_uk.nc /mnt/data/dmentipl/improver_tests/spot-extract/inputs/ukvx_temperature.nc /mnt/data/dmentipl/improver_tests/spot-extract/inputs/ukvx_lapse_rate_2m.nc --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw14/test_lapse_rate_mismatch0/output.nc --apply-lapse-rate-correction --new-title 'IMPROVER UK Spot Values'
_______________________ test_percentile_percentile_input _______________________
[gw1] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw1/test_percentile_percentile_inp0')

    def test_percentile_percentile_input(tmp_path):
        """Test extracting percentiles from percentile input"""
        kgo_dir = acc.kgo_root() / "spot-extract"
        neighbour_path = kgo_dir / "inputs/all_methods_uk.nc"
        threshold_path = kgo_dir / "inputs/enukx_temperature_percentiles.nc"
        kgo_path = kgo_dir / "outputs/extract_percentile_kgo.nc"
        output_path = tmp_path / "output.nc"
        args = [
            neighbour_path,
            threshold_path,
            "--output",
            output_path,
            "--extract-percentiles",
            "50",
            "--new-title",
            UK_SPOT_TITLE,
        ]
        run_cli(args)
>       acc.compare(output_path, kgo_path)
E       AssertionError: different attributes of /wmo_id - ['units']

improver_tests/acceptance/test_spot_extract.py:261: AssertionError
----------------------------- Captured stdout call -----------------------------
improver spot-extract /mnt/data/dmentipl/improver_tests/spot-extract/inputs/all_methods_uk.nc /mnt/data/dmentipl/improver_tests/spot-extract/inputs/enukx_temperature_percentiles.nc --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw1/test_percentile_percentile_inp0/output.nc --extract-percentiles 50 --new-title 'IMPROVER UK Spot Values'
________________________ test_percentile_deterministic _________________________
[gw14] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw14/test_percentile_deterministic0')

    def test_percentile_deterministic(tmp_path):
        """Test extracting percentiles from deterministic input"""
        kgo_dir = acc.kgo_root() / "spot-extract"
        neighbour_path = kgo_dir / "inputs/all_methods_uk.nc"
        diag_path = kgo_dir / "inputs/ukvx_temperature.nc"
        kgo_path = kgo_dir / "outputs/nearest_uk_temperatures.nc"
        output_path = tmp_path / "output.nc"
        args = [
            neighbour_path,
            diag_path,
            "--output",
            output_path,
            "--extract-percentiles",
            "50",
            "--new-title",
            UK_SPOT_TITLE,
        ]
        with pytest.warns(
            UserWarning, match="Diagnostic cube is not a known probabilistic type."
        ):
            run_cli(args)
>       acc.compare(output_path, kgo_path)
E       AssertionError: different attributes of /wmo_id - ['units']

improver_tests/acceptance/test_spot_extract.py:303: AssertionError
----------------------------- Captured stdout call -----------------------------
improver spot-extract /mnt/data/dmentipl/improver_tests/spot-extract/inputs/all_methods_uk.nc /mnt/data/dmentipl/improver_tests/spot-extract/inputs/ukvx_temperature.nc --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw14/test_percentile_deterministic0/output.nc --extract-percentiles 50 --new-title 'IMPROVER UK Spot Values'
__________________________________ test_spot ___________________________________
[gw5] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw5/test_spot0')

    def test_spot(tmp_path):
        """Test extraction from spot file"""
        kgo_dir = acc.kgo_root() / "extract/sites"
        kgo_path = kgo_dir / "kgo_spot.nc"
        input_path = kgo_dir / "input_spot.nc"
        output_path = tmp_path / "output.nc"
        args = [
            input_path,
            "--constraints",
            "wmo_id=['700', '845', '996', '3346', '3382']",
            "--output",
            output_path,
        ]
        run_cli(args)
>       acc.compare(output_path, kgo_path)
E       AssertionError: different attributes of /wmo_id - ['units']

improver_tests/acceptance/test_extract.py:168: AssertionError
----------------------------- Captured stdout call -----------------------------
improver extract /mnt/data/dmentipl/improver_tests/extract/sites/input_spot.nc --constraints 'wmo_id=['"'"'700'"'"', '"'"'845'"'"', '"'"'996'"'"', '"'"'3346'"'"', '"'"'3382'"'"']' --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw5/test_spot0/output.nc
__________________ test_multiple_percentile_thresholded_input __________________
[gw1] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw1/test_multiple_percentile_thres0')

    def test_multiple_percentile_thresholded_input(tmp_path):
        """Test extracting multiple percentiles from thresholded input"""
        kgo_dir = acc.kgo_root() / "spot-extract"
        neighbour_path = kgo_dir / "inputs/all_methods_uk.nc"
        threshold_path = kgo_dir / "inputs/enukx_temperature_thresholds.nc"
        kgo_path = kgo_dir / "outputs/extract_multiple_percentiles_kgo.nc"
        output_path = tmp_path / "output.nc"
        args = [
            neighbour_path,
            threshold_path,
            "--output",
            output_path,
            "--extract-percentiles",
            "25, 50, 75",
            "--new-title",
            UK_SPOT_TITLE,
        ]
        run_cli(args)
>       acc.compare(output_path, kgo_path)
E       AssertionError: different attributes of /wmo_id - ['units']

improver_tests/acceptance/test_spot_extract.py:354: AssertionError
----------------------------- Captured stdout call -----------------------------
improver spot-extract /mnt/data/dmentipl/improver_tests/spot-extract/inputs/all_methods_uk.nc /mnt/data/dmentipl/improver_tests/spot-extract/inputs/enukx_temperature_thresholds.nc --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw1/test_multiple_percentile_thres0/output.nc --extract-percentiles '25, 50, 75' --new-title 'IMPROVER UK Spot Values'
_____________________ test_percentile_deterministic_quiet ______________________
[gw14] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw14/test_percentile_deterministic_0')

    def test_percentile_deterministic_quiet(tmp_path):
        """Test extracting percentiles from deterministic input. In this case the
        --suppress-warnings flag is enabled. This excludes the warning raised when
        spot-extract is set to extract percentiles and used with deterministic data.
        This is intended to reduce output in logs when this warning is expected and
        thus not useful."""
        kgo_dir = acc.kgo_root() / "spot-extract"
        neighbour_path = kgo_dir / "inputs/all_methods_uk.nc"
        diag_path = kgo_dir / "inputs/ukvx_temperature.nc"
        kgo_path = kgo_dir / "outputs/nearest_uk_temperatures.nc"
        output_path = tmp_path / "output.nc"
        args = [
            neighbour_path,
            diag_path,
            "--output",
            output_path,
            "--extract-percentiles",
            "50",
            "--new-title",
            UK_SPOT_TITLE,
            "--suppress-warnings",
        ]
        with pytest.warns(None) as collected_warns:
            run_cli(args)

        msg = "Diagnostic cube is not a known probabilistic type."
        assert all([msg not in str(warning.message) for warning in collected_warns])
>       acc.compare(output_path, kgo_path)
E       AssertionError: different attributes of /wmo_id - ['units']

improver_tests/acceptance/test_spot_extract.py:333: AssertionError
----------------------------- Captured stdout call -----------------------------
improver spot-extract /mnt/data/dmentipl/improver_tests/spot-extract/inputs/all_methods_uk.nc /mnt/data/dmentipl/improver_tests/spot-extract/inputs/ukvx_temperature.nc --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw14/test_percentile_deterministic_0/output.nc --extract-percentiles 50 --new-title 'IMPROVER UK Spot Values' --suppress-warnings
__________________ test_multiple_percentile_percentile_input ___________________
[gw1] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw1/test_multiple_percentile_perce0')

    def test_multiple_percentile_percentile_input(tmp_path):
        """Test extracting multiple percentiles from percentile input"""
        kgo_dir = acc.kgo_root() / "spot-extract"
        neighbour_path = kgo_dir / "inputs/all_methods_uk.nc"
        threshold_path = kgo_dir / "inputs/enukx_temperature_percentiles.nc"
        kgo_path = kgo_dir / "outputs/extract_multiple_percentiles_kgo.nc"
        output_path = tmp_path / "output.nc"
        args = [
            neighbour_path,
            threshold_path,
            "--output",
            output_path,
            "--extract-percentiles",
            "25, 50, 75",
            "--new-title",
            UK_SPOT_TITLE,
        ]
        run_cli(args)
>       acc.compare(output_path, kgo_path)
E       AssertionError: different attributes of /wmo_id - ['units']

improver_tests/acceptance/test_spot_extract.py:375: AssertionError
----------------------------- Captured stdout call -----------------------------
improver spot-extract /mnt/data/dmentipl/improver_tests/spot-extract/inputs/all_methods_uk.nc /mnt/data/dmentipl/improver_tests/spot-extract/inputs/enukx_temperature_percentiles.nc --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw1/test_multiple_percentile_perce0/output.nc --extract-percentiles '25, 50, 75' --new-title 'IMPROVER UK Spot Values'
__________________________ test_multiple_constraints ___________________________
[gw9] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw9/test_multiple_constraints0')

    def test_multiple_constraints(tmp_path):
        """Test use of multiple constraints"""
        kgo_dir = acc.kgo_root() / "spot-extract"
        neighbour_path = kgo_dir / "inputs/all_methods_uk.nc"
        diag_path = kgo_dir / "inputs/ukvx_temperature.nc"
        kgo_path = kgo_dir / "outputs/mindz_land_constraint_uk_temperatures.nc"
        output_path = tmp_path / "output.nc"
        args = [
            neighbour_path,
            diag_path,
            "--output",
            output_path,
            "--similar-altitude",
            "--land-constraint",
            "--new-title",
            UK_SPOT_TITLE,
        ]
        run_cli(args)
>       acc.compare(output_path, kgo_path)
E       AssertionError: different attributes of /wmo_id - ['units']

improver_tests/acceptance/test_spot_extract.py:219: AssertionError
----------------------------- Captured stdout call -----------------------------
improver spot-extract /mnt/data/dmentipl/improver_tests/spot-extract/inputs/all_methods_uk.nc /mnt/data/dmentipl/improver_tests/spot-extract/inputs/ukvx_temperature.nc --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw9/test_multiple_constraints0/output.nc --similar-altitude --land-constraint --new-title 'IMPROVER UK Spot Values'
______________________ test_percentile_realization_input _______________________
[gw14] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw14/test_percentile_realization_in0')

    def test_percentile_realization_input(tmp_path):
        """Test extracting percentiles from realization input"""
        kgo_dir = acc.kgo_root() / "spot-extract"
        neighbour_path = kgo_dir / "inputs/all_methods_uk.nc"
        realization_path = kgo_dir / "inputs/enukx_temperature_realizations.nc"
        kgo_path = kgo_dir / "outputs/extract_percentile_kgo.nc"
        output_path = tmp_path / "output.nc"
        args = [
            neighbour_path,
            realization_path,
            "--output",
            output_path,
            "--extract-percentiles",
            "50",
            "--new-title",
            UK_SPOT_TITLE,
        ]
        run_cli(args)
>       acc.compare(output_path, kgo_path)
E       AssertionError: different attributes of /wmo_id - ['units']

improver_tests/acceptance/test_spot_extract.py:396: AssertionError
----------------------------- Captured stdout call -----------------------------
improver spot-extract /mnt/data/dmentipl/improver_tests/spot-extract/inputs/all_methods_uk.nc /mnt/data/dmentipl/improver_tests/spot-extract/inputs/enukx_temperature_realizations.nc --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw14/test_percentile_realization_in0/output.nc --extract-percentiles 50 --new-title 'IMPROVER UK Spot Values'
______________________ test_percentile_thresholded_input _______________________
[gw9] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw9/test_percentile_thresholded_in0')

    def test_percentile_thresholded_input(tmp_path):
        """Test extracting percentiles from thresholded input"""
        kgo_dir = acc.kgo_root() / "spot-extract"
        neighbour_path = kgo_dir / "inputs/all_methods_uk.nc"
        threshold_path = kgo_dir / "inputs/enukx_temperature_thresholds.nc"
        kgo_path = kgo_dir / "outputs/extract_percentile_kgo.nc"
        output_path = tmp_path / "output.nc"
        args = [
            neighbour_path,
            threshold_path,
            "--output",
            output_path,
            "--extract-percentiles",
            "50",
            "--new-title",
            UK_SPOT_TITLE,
        ]
        run_cli(args)
>       acc.compare(output_path, kgo_path)
E       AssertionError: different attributes of /wmo_id - ['units']

improver_tests/acceptance/test_spot_extract.py:240: AssertionError
----------------------------- Captured stdout call -----------------------------
improver spot-extract /mnt/data/dmentipl/improver_tests/spot-extract/inputs/all_methods_uk.nc /mnt/data/dmentipl/improver_tests/spot-extract/inputs/enukx_temperature_thresholds.nc --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw9/test_percentile_thresholded_in0/output.nc --extract-percentiles 50 --new-title 'IMPROVER UK Spot Values'
__________________ test_multiple_percentile_realization_input __________________
[gw14] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw14/test_multiple_percentile_reali0')

    def test_multiple_percentile_realization_input(tmp_path):
        """Test extracting multiple percentiles from realization input"""
        kgo_dir = acc.kgo_root() / "spot-extract"
        neighbour_path = kgo_dir / "inputs/all_methods_uk.nc"
        realization_path = kgo_dir / "inputs/enukx_temperature_realizations.nc"
        kgo_path = kgo_dir / "outputs/extract_multiple_percentiles_kgo.nc"
        output_path = tmp_path / "output.nc"
        args = [
            neighbour_path,
            realization_path,
            "--output",
            output_path,
            "--extract-percentiles",
            "25, 50, 75",
            "--new-title",
            UK_SPOT_TITLE,
        ]
        run_cli(args)
>       acc.compare(output_path, kgo_path)
E       AssertionError: different attributes of /wmo_id - ['units']

improver_tests/acceptance/test_spot_extract.py:417: AssertionError
----------------------------- Captured stdout call -----------------------------
improver spot-extract /mnt/data/dmentipl/improver_tests/spot-extract/inputs/all_methods_uk.nc /mnt/data/dmentipl/improver_tests/spot-extract/inputs/enukx_temperature_realizations.nc --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw14/test_multiple_percentile_reali0/output.nc --extract-percentiles '25, 50, 75' --new-title 'IMPROVER UK Spot Values'
_______________ test_percentile_from_threshold_with_realizations _______________
[gw1] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw1/test_percentile_from_threshold0')

    def test_percentile_from_threshold_with_realizations(tmp_path):
        """Test requesting a percentile from a cube with thresholds where the realizations
        need collapsing first"""
        kgo_dir = acc.kgo_root() / "spot-extract"
        neighbour_path = kgo_dir / "inputs/all_methods_uk.nc"
        diag_path = kgo_dir / "inputs/enukx_preciprate_realizations_thresholds.nc"
        kgo_path = kgo_dir / "outputs/with_realization_collapse.nc"
        output_path = tmp_path / "output.nc"
        args = [
            neighbour_path,
            diag_path,
            "--output",
            output_path,
            "--realization-collapse",
            "--extract-percentiles",
            "50",
            "--new-title",
            "MOGREPS-UK Spot Values",
        ]
        run_cli(args)
>       acc.compare(output_path, kgo_path)
E       AssertionError: different attributes of /wmo_id - ['units']

improver_tests/acceptance/test_spot_extract.py:480: AssertionError
----------------------------- Captured stdout call -----------------------------
improver spot-extract /mnt/data/dmentipl/improver_tests/spot-extract/inputs/all_methods_uk.nc /mnt/data/dmentipl/improver_tests/spot-extract/inputs/enukx_preciprate_realizations_thresholds.nc --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw1/test_percentile_from_threshold0/output.nc --realization-collapse --extract-percentiles 50 --new-title 'MOGREPS-UK Spot Values'
_______________________ test_normal_point_by_point_sites _______________________
[gw10] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw10/test_normal_point_by_point_sit0')

    @pytest.mark.slow
    def test_normal_point_by_point_sites(tmp_path):
        """
        Test estimate-emos-coefficients for diagnostic with assumed
        normal distribution where coefficients are computed independently at each
        site location (initial guess and minimisation).
        """
        kgo_dir = acc.kgo_root() / "estimate-emos-coefficients/normal/sites"
        kgo_path = kgo_dir / "point_by_point" / "kgo.nc"
        history_path = kgo_dir / "history/*.nc"
        truth_path = kgo_dir / "truth/*.nc"
        output_path = tmp_path / "output.nc"
        est_emos_tol = str(0.01)
        compare_emos_tolerance = 0.1
        args = [
            history_path,
            truth_path,
            "--distribution",
            "norm",
            "--truth-attribute",
            "mosg__model_configuration=uk_det",
            "--tolerance",
            est_emos_tol,
            "--point-by-point",
            "--output",
            output_path,
        ]
        run_cli(args)
        acc.compare(
>           output_path, kgo_path, atol=compare_emos_tolerance, rtol=compare_emos_tolerance
        )
E       AssertionError: different attributes of /wmo_id - ['units']

improver_tests/acceptance/test_estimate_emos_coefficients.py:296: AssertionError
----------------------------- Captured stdout call -----------------------------
improver estimate-emos-coefficients '/mnt/data/dmentipl/improver_tests/estimate-emos-coefficients/normal/sites/history/*.nc' '/mnt/data/dmentipl/improver_tests/estimate-emos-coefficients/normal/sites/truth/*.nc' --distribution norm --truth-attribute mosg__model_configuration=uk_det --tolerance 0.01 --point-by-point --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw10/test_normal_point_by_point_sit0/output.nc
____________________ test_nearest_minimum_dz[global-global] ____________________
[gw12] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw12/test_nearest_minimum_dz_global0')
domain = 'global', model = 'global'

    @pytest.mark.slow
    @pytest.mark.parametrize("domain,model", UK_GLOBAL)
    def test_nearest_minimum_dz(tmp_path, domain, model):
        """Test neighbour finding with minimum altitude difference"""
        kgo_dir = acc.kgo_root() / "neighbour-finding"
        kgo_path = kgo_dir / f"outputs/nearest_minimum_dz_{domain}_kgo.nc"
        sites_path = kgo_dir / f"inputs/{domain}_sites.json"
        orography_path = kgo_dir / f"inputs/{model}_orography.nc"
        landmask_path = kgo_dir / f"inputs/{model}_landmask.nc"
        output_path = tmp_path / "output.nc"
        args = [
            orography_path,
            landmask_path,
            sites_path,
            "--similar-altitude",
            "--search-radius",
            "50000",
            "--output",
            output_path,
        ]
        run_cli(args)
>       acc.compare(output_path, kgo_path)
E       AssertionError: different attributes of /wmo_id - ['units']

improver_tests/acceptance/test_neighbour_finding.py:104: AssertionError
----------------------------- Captured stdout call -----------------------------
improver neighbour-finding /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_orography.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_landmask.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_sites.json --similar-altitude --search-radius 50000 --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw12/test_nearest_minimum_dz_global0/output.nc
____________________ test_nearest_land_minimum_dz[uk-ukvx] _____________________
[gw12] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw12/test_nearest_land_minimum_dz_u0')
domain = 'uk', model = 'ukvx'

    @pytest.mark.slow
    @pytest.mark.parametrize("domain,model", UK_GLOBAL)
    def test_nearest_land_minimum_dz(tmp_path, domain, model):
        """Test neighbour finding with land constraint plus min altitude diff"""
        kgo_dir = acc.kgo_root() / "neighbour-finding"
        kgo_path = kgo_dir / f"outputs/nearest_land_constraint_minimum_dz_{domain}_kgo.nc"
        sites_path = kgo_dir / f"inputs/{domain}_sites.json"
        orography_path = kgo_dir / f"inputs/{model}_orography.nc"
        landmask_path = kgo_dir / f"inputs/{model}_landmask.nc"
        output_path = tmp_path / "output.nc"
        args = [
            orography_path,
            landmask_path,
            sites_path,
            "--land-constraint",
            "--similar-altitude",
            "--search-radius",
            "50000",
            "--output",
            output_path,
        ]
        if domain == "uk":
            args += ["--node-limit", "100"]
        run_cli(args)
>       acc.compare(output_path, kgo_path)
E       AssertionError: different attributes of /wmo_id - ['units']

improver_tests/acceptance/test_neighbour_finding.py:131: AssertionError
----------------------------- Captured stdout call -----------------------------
improver neighbour-finding /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/ukvx_orography.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/ukvx_landmask.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/uk_sites.json --land-constraint --similar-altitude --search-radius 50000 --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw12/test_nearest_land_minimum_dz_u0/output.nc --node-limit 100
_________________ test_nearest_land_minimum_dz[global-global] __________________
[gw12] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw12/test_nearest_land_minimum_dz_g0')
domain = 'global', model = 'global'

    @pytest.mark.slow
    @pytest.mark.parametrize("domain,model", UK_GLOBAL)
    def test_nearest_land_minimum_dz(tmp_path, domain, model):
        """Test neighbour finding with land constraint plus min altitude diff"""
        kgo_dir = acc.kgo_root() / "neighbour-finding"
        kgo_path = kgo_dir / f"outputs/nearest_land_constraint_minimum_dz_{domain}_kgo.nc"
        sites_path = kgo_dir / f"inputs/{domain}_sites.json"
        orography_path = kgo_dir / f"inputs/{model}_orography.nc"
        landmask_path = kgo_dir / f"inputs/{model}_landmask.nc"
        output_path = tmp_path / "output.nc"
        args = [
            orography_path,
            landmask_path,
            sites_path,
            "--land-constraint",
            "--similar-altitude",
            "--search-radius",
            "50000",
            "--output",
            output_path,
        ]
        if domain == "uk":
            args += ["--node-limit", "100"]
        run_cli(args)
>       acc.compare(output_path, kgo_path)
E       AssertionError: different attributes of /wmo_id - ['units']

improver_tests/acceptance/test_neighbour_finding.py:131: AssertionError
----------------------------- Captured stdout call -----------------------------
improver neighbour-finding /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_orography.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_landmask.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_sites.json --land-constraint --similar-altitude --search-radius 50000 --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw12/test_nearest_land_minimum_dz_g0/output.nc
_______________________ test_all_methods[global-global] ________________________
[gw0] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw0/test_all_methods_global_global0')
domain = 'global', model = 'global'

    @pytest.mark.slow
    @pytest.mark.parametrize("domain,model", UK_GLOBAL)
    def test_all_methods(tmp_path, domain, model):
        """Test neighbour finding using all methods"""
        kgo_dir = acc.kgo_root() / "neighbour-finding"
        kgo_path = kgo_dir / f"outputs/all_methods_{domain}_kgo.nc"
        sites_path = kgo_dir / f"inputs/{domain}_sites.json"
        orography_path = kgo_dir / f"inputs/{model}_orography.nc"
        landmask_path = kgo_dir / f"inputs/{model}_landmask.nc"
        output_path = tmp_path / "output.nc"
        args = [
            orography_path,
            landmask_path,
            sites_path,
            "--all-methods",
            "--output",
            output_path,
        ]
        run_cli(args)
>       acc.compare(output_path, kgo_path)
E       AssertionError: different attributes of /neighbour_selection_method_name - ['units']

improver_tests/acceptance/test_neighbour_finding.py:153: AssertionError
----------------------------- Captured stdout call -----------------------------
improver neighbour-finding /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_orography.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_landmask.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_sites.json --all-methods --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw0/test_all_methods_global_global0/output.nc
______________________ test_alternative_coordinate_system ______________________
[gw0] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw0/test_alternative_coordinate_sy0')

    def test_alternative_coordinate_system(tmp_path):
        """Test use of an alternative coordinate system"""
        kgo_dir = acc.kgo_root() / "neighbour-finding"
        # same KGO as test_nearest global domain
        kgo_path = kgo_dir / "outputs/nearest_global_kgo.nc"
        sites_path = kgo_dir / "inputs/LAEA_grid_sites.json"
        orography_path = kgo_dir / "inputs/global_orography.nc"
        landmask_path = kgo_dir / "inputs/global_landmask.nc"
        output_path = tmp_path / "output.nc"
        coord_opts = {
            "central_latitude": 54.9,
            "central_longitude": -2.5,
            "false_easting": 0.0,
            "false_northing": 0.0,
            "globe": {"semimajor_axis": 6378137.0, "semiminor_axis": 6356752.314140356},
        }
        coord_opts_json = json.dumps(coord_opts)
        args = [
            orography_path,
            landmask_path,
            sites_path,
            "--site-coordinate-system",
            "LambertAzimuthalEqualArea",
            "--site-coordinate-options",
            coord_opts_json,
            "--site-x-coordinate",
            "projection_x_coordinate",
            "--site-y-coordinate",
            "projection_y_coordinate",
            "--output",
            output_path,
        ]
        run_cli(args)
>       acc.compare(output_path, kgo_path)
E       AssertionError: different attributes of /neighbour_selection_method_name - ['units']

improver_tests/acceptance/test_neighbour_finding.py:189: AssertionError
----------------------------- Captured stdout call -----------------------------
improver neighbour-finding /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_orography.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_landmask.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/LAEA_grid_sites.json --site-coordinate-system LambertAzimuthalEqualArea --site-coordinate-options '{"central_latitude": 54.9, "central_longitude": -2.5, "false_easting": 0.0, "false_northing": 0.0, "globe": {"semimajor_axis": 6378137.0, "semiminor_axis": 6356752.314140356}}' --site-x-coordinate projection_x_coordinate --site-y-coordinate projection_y_coordinate --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw0/test_alternative_coordinate_sy0/output.nc
=========================== short test summary info ============================
FAILED improver_tests/acceptance/test_create_grid_with_halo.py::test_basic - ...

FAILED improver_tests/acceptance/test_spot_extract.py::test_lapse_rate_mismatch
FAILED improver_tests/acceptance/test_spot_extract.py::test_percentile_percentile_input
FAILED improver_tests/acceptance/test_spot_extract.py::test_percentile_deterministic
FAILED improver_tests/acceptance/test_extract.py::test_spot - AssertionError:...
FAILED improver_tests/acceptance/test_spot_extract.py::test_multiple_percentile_thresholded_input
FAILED improver_tests/acceptance/test_spot_extract.py::test_percentile_deterministic_quiet
FAILED improver_tests/acceptance/test_spot_extract.py::test_multiple_percentile_percentile_input
FAILED improver_tests/acceptance/test_spot_extract.py::test_multiple_constraints
FAILED improver_tests/acceptance/test_spot_extract.py::test_percentile_realization_input
FAILED improver_tests/acceptance/test_spot_extract.py::test_percentile_thresholded_input
FAILED improver_tests/acceptance/test_spot_extract.py::test_multiple_percentile_realization_input
FAILED improver_tests/acceptance/test_spot_extract.py::test_percentile_from_threshold_with_realizations
FAILED improver_tests/acceptance/test_estimate_emos_coefficients.py::test_normal_point_by_point_sites
FAILED improver_tests/acceptance/test_neighbour_finding.py::test_nearest_minimum_dz[global-global]
FAILED improver_tests/acceptance/test_neighbour_finding.py::test_nearest_land_minimum_dz[uk-ukvx]
FAILED improver_tests/acceptance/test_neighbour_finding.py::test_nearest_land_minimum_dz[global-global]
FAILED improver_tests/acceptance/test_neighbour_finding.py::test_all_methods[global-global]
FAILED improver_tests/acceptance/test_neighbour_finding.py::test_alternative_coordinate_system
= 36 failed, 305 passed, 4 skipped, 2 xfailed, 492 warnings in 137.60s (0:02:17) =

Thanks @dmentipl. This looks like good progress on support for Iris 3 and the changes needed look a lot less difficult than I was worried might be needed.
I think it'd be good to add tests/support for Python 3.8, but tackle that in a separate follow up PR to this Iris 3 work. I expect that the main constraint there will be library builds available on conda-forge rather than any breaking changes in the Python language.
I think the main outstanding questions are:

Is a hard changeover in Iris version (no overlap period with support for both Iris 2 and Iris 3) OK for Met Office usage?
The 36 failing acceptance tests all relate to the 'units' attribute. I'm not too worried by these differences as the NetCDF variables where the 'units' attribute is no longer present all seem to be coordinate variables where units aren't well defined. For example, there aren't really any physical units for the WMO ID number of a site, or the neighbour selection method. Is this metadata change OK for Met Office usage?
Changes to the pinned scipy library version will change some acceptance test outputs. See #1398 and #1397 for details. I think the change to Iris 3 is a good opportunity to move to a newer scipy at the same time. Are the Met Office team happy to move to a newer scipy or are there other python stack limitations?
Test that the Nimrod changes in Iris 3 cover the existing monkeypatches in IMPROVER and it's safe to remove iris_nimrod_patcher. We don't use this at BOM and the relevant tests get skipped by pytest if this library isn't available, so Nimrod handling will likely need some testing from the MO team.

I have added a Python 3.8 environment. It is passing the unit tests. It fails the same acceptance tests as the Python 3.7 environment. The changes from the 3.7 env (other than Python version) are:

unpin cftime
use cf-units rather than cf_units

Hard changeover to Iris 3 - in principle fine, as so far it looks like migrating to it is mandatory this year.
wmo_id units - what are the changes exactly? You are right that units just don't make sense for this, and as far as I can tell they are optional in CF for this kind of variable anyway.
scipy version - we'd expect to move to ~1.6.2 with Iris 3 at the moment?

Hard changeover to Iris 3 - in principle fine

That's good to know.

what are the changes exactly?

For '/wmo_id', and some others, the desired attribute dictionary is
{'units': '1', 'long_name': 'wmo_id'}

whereas the actual is
{'long_name': 'wmo_id'}

So it's just missing the units. I'm not sure why this is the case.

scipy version - we'd expect to move to ~1.6.2 with Iris 3

I've updated the scipy version in both environments. It leads to a few (8) more acceptance test failures (as @tjtg said above).

I've now removed nimrod patching. Unit tests still pass, and the acceptance test failures are the same as prior.

@dmentipl I have (scandalously) force pushed over your branch after rebasing and making some changes. The changes I have made are as follows:

Updated the version of pysteps to match our HPC environment.

This required a small tweak to the code to set the interpolation mode
the environments were updated too.

Updated the checksums for the acceptance test data.

Tom can hopefully give you access to the new data in branch im1517 of the acceptance test data repo.
There are details about what changed here https://github.com/metoppv/improver/issues/1517 where most were the unit change you noted, but we are also jumping forward to newer scipy in our HPC env, which affects interpolation as @tjtg noted above.

I have removed an IMPROVER function called equalise_cube_attributes as there is now functionality to do this within Iris which I have adopted.
Removed the iris2 environments from /envs as these will no longer work with IMPROVER, and added pytest-xdist to the envs as well to ensure acceptance tests can be run in parallel (lest any user grows old waiting for them to run sequentially).

I've run all the tests with both iris3 environments and everything passes as expected, as well as with our local environment. I still need to do a bit more testing on the HPC before this can all be merged at our end, but hopefully none of the changes I've made cause you any problems.
I've promoted this PR to ready for review. Could you check that what I've done is okay for you guys. I'll finish up the HPC changes and get someone to look at this from our end as well. Then when it gets merged we will finally leave iris 2 behind for good.

Thanks for making those changes @bayliffe 👍
They look good, in principle, from my perspective. However, even though the tests are passing in GitHub actions, I'm getting four unit test failures locally (using the conda environment files) in nbhood/circular_kernel. I need to investigate further. I suspect there is an unpinned package causing problems.

Tom can hopefully give you access to the new data in branch im1517 of the acceptance test data repo.

Yes, I've got access, and I'm in the process of testing these.
 ],
        [1. , 1. , 1. , 1. , 1. ],
        [1. ,...      [1. , 1. , 1. , 1. , 1. ],
        [1. , 1. , 1. , 1. , 1. ],
        [1. , 1. , 1. , 1. , 1. ]]], dtype=float32)
b = array([[[0. , 0.4, 1. , 1. , 1. ],
        [0.4, 1. , 1. , 1. , 1. ],
        [1. , 1. , 1. , 1. , 1. ],
        [1. ,..., 1. , 1. ],
        [1. , 1. , 1. , 1. , 1. ],
        [1. , 1. , 1. , 1. , 1. ],
        [1. , 1. , 1. , 1. , 1. ]]])
decimal = 6

    def assertArrayAlmostEqual(self, a, b, decimal=6):
>       np.testing.assert_array_almost_equal(a, b, decimal=decimal)
E       AssertionError:
E       Arrays are not almost equal to 6 decimals
E
E       Mismatched elements: 1 / 75 (1.33%)
E       Max absolute difference: 1.
E       Max relative difference: 0.00000001
E        x: array([[[0. , 0.4, 1. , 1. , 1. ],
E               [0.4, 1. , 1. , 1. , 1. ],
E               [1. , 1. , 1. , 1. , 1. ],...
E        y: array([[[0. , 0.4, 1. , 1. , 1. ],
E               [0.4, 1. , 1. , 1. , 1.  ],
        [1. , 1. , 1. , 1. , 1. , 1. , 1. ],
        [1. , 1. , 1. , 0.2,..., 1. , 1. ],
        [1. , 1. , 1. , 1. , 1. , 1. , 1. ],
        [1. , 1. , 1. , 1. , 1. , 1. , 1. ]]], dtype=float32)
b = array([[[1. , 1. , 1. , 1. , 1. , 1. , 1. ],
        [1. , 1. , 1. , 1. , 1. , 1. , 1. ],
        [1. , 1. , 1. , 0.2,..., 1. , 1. , 1. , 1. , 1. ],
        [1. , 1. , 1. , 1. , 1. , 1. , 1. ],
        [1. , 1. , 1. , 1. , 1. , 1. , 1. ]]])
decimal = 6

    def assertArrayAlmostEqual(self, a, b, decimal=6):
>       np.testing.assert_array_almost_equal(a, b, decimal=decimal)
E       AssertionError:
E       Arrays are not almost equal to 6 decimals
E
E       Mismatched elements: 1 / 245 (0.408%)
E       Max absolute difference: 0.6
E       Max relative difference: 1.5
E        x: array([[[1. , 1. , 1. , 1. , 1. , 1. , 1. ],
E               [1. , 1. , 1. , 1. , 1. , 1. , 1. ],
E               [1. , 1. , 1. , 0.2, 1. , 1. , 1. ],...
E        y: array([[[1. , 1. , 1. , 1. , 1. , 1. , 1. ],
E               [1. , 1. , 1. , 1. , 1. , 1. , 1.  ],
        [1. , 1. , 1. , 1. , 1. , 1. , 1. ],
        [1. , 1. , 1. , 0.1,..., 1. , 1. ],
        [1. , 1. , 1. , 1. , 1. , 1. , 1. ],
        [1. , 1. , 1. , 1. , 1. , 1. , 1. ]]], dtype=float32)
b = array([[[1. , 1. , 1. , 1. , 1. , 1. , 1. ],
        [1. , 1. , 1. , 1. , 1. , 1. , 1. ],
        [1. , 1. , 1. , 0.1,..., 1. , 1. , 1. , 1. , 1. ],
        [1. , 1. , 1. , 1. , 1. , 1. , 1. ],
        [1. , 1. , 1. , 1. , 1. , 1. , 1. ]]])
decimal = 6

    def assertArrayAlmostEqual(self, a, b, decimal=6):
>       np.testing.assert_array_almost_equal(a, b, decimal=decimal)
E       AssertionError:
E       Arrays are not almost equal to 6 decimals
E
E       Mismatched elements: 9 / 490 (1.84%)
E       Max absolute difference: 0.8
E       Max relative difference: 4.
E        x: array([[[1. , 1. , 1. , 1. , 1. , 1. , 1. ],
E               [1. , 1. , 1. , 1. , 1. , 1. , 1. ],
E               [1. , 1. , 1. , 0.1, 1. , 1. , 1. ],...
E        y: array([[[1. , 1. , 1. , 1. , 1. , 1. , 1. ],
E               [1. , 1. , 1. , 1. , 1. , 1. , 1.  ],
        [1. , 1. , 0.2, 1. , 1. ],
        [1. , 0.2, 0.2, 0.2, 1. ],
        [1. ,...      [1. , 0.8, 0.8, 0.8, 1. ],
        [1. , 1. , 0.8, 1. , 1. ],
        [1. , 1. , 1. , 1. , 1. ]]], dtype=float32)
b = array([[[1. , 1. , 1. , 1. , 1. ],
        [1. , 1. , 0.2, 1. , 1. ],
        [1. , 0.2, 0.2, 0.2, 1. ],
        [1. ,..., 1. , 1. ],
        [1. , 0.8, 0.8, 0.8, 1. ],
        [1. , 1. , 0.8, 1. , 1. ],
        [1. , 1. , 1. , 1. , 1. ]]])
decimal = 6

    def assertArrayAlmostEqual(self, a, b, decimal=6):
>       np.testing.assert_array_almost_equal(a, b, decimal=decimal)
E       AssertionError:
E       Arrays are not almost equal to 6 decimals
E
E       Mismatched elements: 2 / 75 (2.67%)
E       Max absolute difference: 0.6
E       Max relative difference: 1.5
E        x: array([[[1. , 1. , 1. , 1. , 1. ],
E               [1. , 1. , 0.2, 1. , 1. ],
E               [1. , 0.2, 0.2, 0.2, 1. ],...
E        y: array([[[1. , 1. , 1. , 1. , 1. ],
E               [1. , 1. , 0.2, 1. , 1. ],
E               [1. , 0.2, 0.2, 0.2, 1. ],...

../../conda/envs/improver_py38_iris30/lib/python3.8/site-packages/iris/tests/__init__.py:670: AssertionError
========================================= short test summary info =========================================
FAILED improver_tests/nbhood/circular_kernel/test_GeneratePercentilesFromACircularNeighbourhood.py::Test_pad_and_unpad_cube::test_single_point_on_corner
FAILED improver_tests/nbhood/circular_kernel/test_GeneratePercentilesFromACircularNeighbourhood.py::Test_run::test_number_of_percentiles_equals_number_of_points
FAILED improver_tests/nbhood/circular_kernel/test_GeneratePercentilesFromACircularNeighbourhood.py::Test_run::test_number_of_points_half_of_number_of_percentiles
FAILED improver_tests/nbhood/circular_kernel/test_GeneratePercentilesFromACircularNeighbourhood.py::Test_run::test_single_point_low_percentiles
================= 4 failed, 2428 passed, 319 skipped, 1 xfailed, 3568 warnings in 48.88s ==================

I've determined that the errors in the unit tests reported above are due to the version of NumPy. In my local testing environment I am using 1.21.0, whereas the version in the conda environment on GitHub is 1.20.3. See below for a diff of the conda environments.
We can pin numpy to 1.20.* if that satisfies Met Office requirements.   1.  1.  1.  1. ]
  [1.  1.  0.2 1.  1. ]
  [1.  0.2 0.2 0.2 1. ]
  [1.  1.  0.2 1.  1. ]
  [1.  1.  1.  1.  1. ]]

 [[1.  1.  1.  1.  1. ]
  [1.  1.  0.4 1.  1. ]
  [1.  0.4 0.4 0.4 1. ]
  [1.  1.  0.4 1.  1. ]
  [1.  1.  1.  1.  1. ]]

 [[1.  1.  1.  1.  1. ]
  [1.  1.  0.8 1.  1. ]
  [1.  0.8 0.8 0.8 1. ]
  [1.  1.  0.8 1.  1. ]
  [1.  1.  1.  1.  1. ]]]

and the result (note the misplaced 0.8 at [1, 0, 2] and 1.0 at [1, 1, 2]):
[[[1.  1.  1.  1.  1. ]
  [1.  1.  0.2 1.  1. ]
  [1.  0.2 0.2 0.2 1. ]
  [1.  1.  0.2 1.  1. ]
  [1.  1.  1.  1.  1. ]]

 [[1.  1.  0.8 1.  1. ]
  [1.  1.  1.  1.  1. ]
  [1.  0.4 0.4 0.4 1. ]
  [1.  1.  0.4 1.  1. ]
  [1.  1.  1.  1.  1. ]]

 [[1.  1.  1.  1.  1. ]
  [1.  1.  0.8 1.  1. ]
  [1.  0.8 0.8 0.8 1. ]
  [1.  1.  0.8 1.  1. ]
  [1.  1.  1.  1.  1. ]]]

@dmentipl thanks for looking into this. Let's pin the version of numpy in the envs for now and move this work forward. If you could then create a ticket in this repo explaining what you found (or linking to the comments above), we can address this issue in slower time before we are compelled to move to numpy 1.21.0. Sound okay?

@bayliffe no worries.
I've pinned numpy to 1.20.*, and raised an issue for dealing with moving to 1.21 in the future (#1522).
Also, I've pinned cf-units to 2.1.5. Version 3.0.0 was released overnight and leads to 48 unit test failures. I've raised an issue for this too (#1523).

We (Mostly @MoseleyS) have found a problem between Iris 3 and  wind_components.py: line 212.
Iris adds status_flag to the coord name so it is no longer wind_speed.
This then throws the error ValueError: Wind speed and direction cubes have unmatched coordinates.
This is now being looked into so it can be replicated with acceptance tests.

@tjtg @dmentipl Would it be possible to update improver branch iris3 to include changes in master (in particular I am interested in including #1528)? We are using improver/iris3 branch in raincal CIs but need #1528 - thanks.
cc @benowen-bom

@lucyleeow I've rebased against master and force pushed.

I have pinned cftime to < 1.5 as compatibility with cf-units 2.1.5 breaks.


Output Kubernetes Cluster Network Profile

Describe your changes
Small PR to output the network_profile attribute of the azurerm_kubernetes_cluster resource.
This is so that it is easier to discover the public IP address associated with the cluster's load balancer, e.g. to follow examples from the Azure Terraform provider like this.
Issue number
#000
Checklist before requesting a review

[ ] The pr title can be used to describe what this pr did in CHANGELOG.md file
[ ] I have executed pre-commit on my machine
[ ] I have passed pr-check on my machine

Thanks for your cooperation!

Hi @joshua-giumelli-deltatre thanks for opening this pr! We have a CI pipeline for this repo, would you please run the following command on your side to update readme file? Thanks:
docker run --rm -v $(pwd):/src -w /src mcr.microsoft.com/azterraform:latest make pre-commit

For Windows:
docker run --rm -v ${pwd}:/src -w /src mcr.microsoft.com/azterraform:latest make pre-commit

Hi @joshua-giumelli-deltatre thanks for the update, all have been settled down and we're good to go, the only thing I'd have to ask you to do us a favor is please read policy bot's comment and sign the agreement, thanks!

@microsoft-github-policy-service agree company="Deltatre"

Apologies for the hold up @lonegunmanb, is there anything more that needs to happen before this can be merged?

Apologies for the hold up @lonegunmanb, is there anything more that needs to happen before this can be merged?

No worries @joshua-giumelli-deltatre, we'll release this feature in our next release, but the next release would be 7.0.0 and it would be a major version with some breaking changes. If you're eager to use this feature ASAP and don't want to take any risk, I think I can release a patch version for this pr first. WDYT?


Soundtrack: Last bit of song gets cut off even with dnf

Reported by DaggerAce. Requires replication.

Can confirm exists. Cuts off last 1.5s- bread5029 also reported it.


Removed old AWS testing cluster stuff including docs

Removed the old (RethinkDB-based) stuff for deploying a testing cluster on AWS, including the related docs.
Removed the .gitattributes file because its only purpose was for use with the AWS testing cluster. (It was used to make a small Git archive of just-the-stuff-needed to send over the wire to each AWS virtual machine.)
Updated .gitignore and codecov.yml to remove all references to the deploy-cluster-aws/ and ntools/. (ntools/ was an old directory that is long gone. Its removal was incidental.)
Updated some docs pages to remove the reference to the deleted aws-testing-cluster.html page.
Updated the index of the Appendices.

Codecov Report

Merging #1917 into tendermint will not change coverage.
The diff coverage is n/a.

@@             Coverage Diff             @@
##           tendermint    #1917   +/-   ##
===========================================
  Coverage       45.18%   45.18%
===========================================
  Files              69       69
  Lines            3326     3326
===========================================
  Hits             1503     1503
  Misses           1823     1823


Please Offer SQL DDL to Clean Up the Cache On-Demand

Feature request
Is your feature request related to a problem? Please describe.
It's great to see that the latest version of Starrocks can run a SQL statement to pre-warm up the cache.
But because of security reason, we don't want to leave any footprint of data at anywhere, unless it is encrypted.
Data at cache is not encrypted, so we want to remove the cache from memory on demand.
Describe the solution you'd like
Please offer a SQL solution like, "CLEAR CACHE FOR {TABLE XXX}", so user can execute and clear the cache.
Describe alternatives you've considered

Additional context

Hi xiaobingxia, thanks for your advice! just want to fully understand your requirement, how is your data encrypted?  are you using hive or iceberg table and encrypted orc or something? I just curious of how the whole process is

So we want all data persisted on disk / s3 to be encrypted. But we also want to leverage the caching. So the process here is:

We pre-warm up the cache
We do queries towards cache (unencrypted) + tables on s3 (encrypted)
We clear the cache.
We can use the cache data without being encrypted, we just need to clean it on time.

I'm not quite understand. if you are using sr's data cache, it should only cache the original data, which in your case is encrypted.

To clarify, we have SR persist data on S3 bucket. And then we apply the s3 server side encryption: https://docs.aws.amazon.com/AmazonS3/latest/userguide/serv-side-encryption.html , which means, data on s3 is encrypted by S3. When any client like SR writes data to S3, S3 will encrypt data first, when SR fetching data from S3, S3 will decrypt data first and then return the data to the SR. So SR will get the decrypted data.

thanks for your explanation! In this case starrocks need to provide a evict cache SQL API.  This is a good suggestion.


A11y_SQL Azure Data Studio_New Terminal_Keyboard:Tab key or shift+tab does not move to link "https://aka.ms/pscore6" is not working.

“Check out Accessibility Insights! – Identify accessibility bugs before check-in and make bug fixing faster and easier.”
GitHubTags: #A11yMAS;#A11yTCS;#A11ySev1;#BM_SQLAzureDataStudio_Oct2020;#SQL Azure Data Studio;#Win10;#FTP;#DesktopApp;#WCAG2.1.1;#Benchmark;#Keyboard;#AINotInScope;#AILimited;
Environment Details:
Application Name: SQL Azure Data Studio
Version: 1.42.0-insider (user setup)
Windows Version: Windows11
Repro Steps:

Launch Azure Data Studio Insiders application.
Tab to server drop down and tab again to button new connection press enter key and add the server details.
Tab to view menu item in the menu bar and use down key arrow to select item "terminal"
Use shift+tab to navigate to https://aka.ms/pscore6

Actual:
On navigating using shift+tab or tab to link https://aka.ms/pscore6 keyboard key tab cannot access it.
Expected:
Tab and shift+tab from the first line of command should move to link https://aka.ms/pscore6
User Impact:
Keyboard users will not be able to access the link.
Attachment:
22333_A11y_SQL.Azure.Data.Studio_New.Terminal_Keyboard.zip

Opening this bug as per @alanrenmsft comment in https://github.com/microsoft/azuredatastudio/issues/12984

@Roopesh-T Could you please open an issue in VSCode repo, this is an issue we inherited from VSCode.

Logged External bug:https://github.com/microsoft/vscode/issues/182849

@Roopesh-T VSCode team has closed the external bug and provided a way to access the links, please refer to https://github.com/microsoft/vscode/issues/182849#issuecomment-1553560118.
Please test it in ADS and close the issue.


fine-tuned the GUI

Coverage remained the same at 79.039% when pulling e925248fab7b58a95bdd7052ca8cc42a6e5e34da on qiyunzhu:html into 7fc18ed3a5a55edc31f0626f825684a9b4b1e7f4 on amnona:develop.

Coverage decreased (-0.06%) to 78.979% when pulling c0cafdd8689a616cfe7738c3b887c6d8c4a68e69 on qiyunzhu:html into 7fc18ed3a5a55edc31f0626f825684a9b4b1e7f4 on amnona:develop.

Coverage decreased (-0.06%) to 78.979% when pulling c0cafdd8689a616cfe7738c3b887c6d8c4a68e69 on qiyunzhu:html into 7fc18ed3a5a55edc31f0626f825684a9b4b1e7f4 on amnona:develop.


Create an assembly dependency view

Create a view that is specific to an assembly, that shows all children and their dates as a gantt chart.

Done.


[WIP] Serialization and storage changes to Querier, Query, and Response.

Introduce serialization services that can flatten objects into a
variety of different formats.  Currently just reusing the Java
serialization code until the project is ready to adopt other
serializers.

Introduce storage services that can store objects and retrieve them.
Currently just reusing the local file system and HDFS.  The storage
service uses one of the serialization services to flatten the objects.

Remove code from Querier, Query, and Response that was serializing
and storing each of them; and update the callers to use the storage
services.

@smarthi   Sorry, not groking "try-all" can you explain?
These follow the method overloading pattern seen in a variety of places like  Arrays.fill(byte[], byte), Arrays.fill(char[], char), etc.

Ha, managed to take down the Travis VM ;-)

*** buffer overflow detected ***: /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java terminated

Yet it worked on my Travis branch before I opened the PR.  Maybe an admin can retry it pls.

@tellison meant to say use  - try-all-resources
//    FileOutputStream fos = new FileOutputStream(file);     try(FileOutputStream fos = new FileOutputStream(file)){ .... }

@smarthi  Ah, ok.  Yep, I had mistakenly thought that try-with-resources was a Java 8 feature :-)  I'll update in due course (not today).

Very nice - much cleaner.
One very minor thing that you may want to go ahead and change in ExpTableMapper:
query = new HadoopFileSystemStore(FileSystem.newInstance(ctx.getConfiguration())).recall(queryDir, Query.class);
to
query = new HadoopFileSystemStore(fs).recall(queryDir, Query.class);
as the FileSystem fs has already been grabbed a couple of lines above.
Otherwise, we will need to make sure that custom StorageServices and SerializationServices have corresponding tests.

fs is defined in setup of ExpTableMapper:
public void setup(Context ctx) throws IOException, InterruptedException
 {
   super.setup(ctx);

   valueOut = new Text();

   FileSystem fs = FileSystem.newInstance(ctx.getConfiguration());
   String queryDir = ctx.getConfiguration().get("pirMR.queryInputDir");
   query = Query.readFromHDFSFile(new Path(queryDir), fs);

   dataPartitionBitSize = query.getQueryInfo().getDataPartitionBitSize();
   maxValue = (int) Math.pow(2, dataPartitionBitSize) - 1;

   NSquared = query.getNSquared();
 }

Ah, I guess you are looking at the current code (pre-PR).  I in-lined that fs definition as it is only used once.

Correct - I was looking at the old code
+1


UI usability

[ ] "Edit" dropdown has no function in header

[ ] One can create a new pod, even automount it which should be default btw but
not delete a pod

[ ]  "Edit" dropdown has no function in header

We need the edit menu on macos platform, to enable Cmd+C,Cmd+V,Cmd+Z... shortcut

[ ]  One can create a new pod, even automount it which should be default btw but
not delete a pod

We will not add delete pod ATM. also automount is disabled for now.
Closing this

as agreed.


slow downloading

Checklist

[ ] I'm reporting a broken site support issue
[ ] I've verified that I'm running youtube-dl version 2021.12.17
[ ] I've checked that all provided URLs are alive and playable in a browser
[ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[ ] I've searched the bugtracker for similar bug reports including closed ones
[ ] I've read bugs section in FAQ

Verbose log

PASTE VERBOSE LOG HERE

Description

WRITE DESCRIPTION HERE

If your problem is as in the issue title, you have this problem, fixed in git master. Or please open and complete an issue properly.

hi, since issue #30583 is locked, i'm writing here.
I installed youtube_dl using pip, and the issue is indeed still present.
Is there any plan to publish a new release? I see the last one is from december 2021, so many updates will probably be included.
Honestly, even just considering this fixed issue is definitely worth it for the community! Keep up the good work :)

thank you for the quick reply! I understand the process, I'm just wondering why such a relevant fix has not been pushed to a release yet

Indeed: definitive statement.


File eval.py missing in the evaluation.

Hello, the file eval.py in the evaluation cannot be found.

Sorry we got the script name wrong.
eval.sh actually refers to infer_and_save.sh.
We have updated the README. Thank you for your correction.

Thank you for your reply!