Added recipe filler.

Added a script that automates filling out the recipe with available data, it's called recipe_filler.py, in utils.
You write your diagnostic in the unfilled recipe like this:
    thetao_global: # Temperature ocean 3D
        short_name: thetao
        preprocessor: prep_global
        mip: Omon
        exp: historical
        project: CMIP6
        dataset: SAM0-UNICON
        ensemble: r1i1p1f1
        start_year: 1995
        end_year: 1999

and it will add an additional_diagnostics section:
    thetao_global: # Temperature ocean 3D
        additional_datasets:
          - {dataset: SAM0-UNICON, project: CMIP6, exp: historical, mip: Omon, ensemble: r1i1p1f1, grid: gn, start_year: 1995, end_year: 1999, }
        short_name: thetao
        preprocessor: prep_global
        exp: historical
        project: CMIP6
        dataset: SAM0-UNICON
        ensemble: r1i1p1f1
        start_year: 1995
        end_year: 1999

So, it works okay. This example is fairly trivial. The more specific your list of requirements, the faster it runs and the fewer datasets it requires. If you wanted to see all datasets, just remove this line from your unfilled recipe:
        dataset: SAM0-UNICON

The problems with this tool:

The additional_datasets is added at the top of the preprocessor, which is ugly.
@valeriupredoi fixed this: it is now added under each variable using correct yaml syntax
you still need to edit the recipe to remove the repeated lines (ie dataset, ensemble, exp in the example above).
it doesn't understand yml anchors.
@valeriupredoi fixed this by using yaml dump and save appropriately
you most likely only want CMIP6 data from one grid, so there's no way to have a preference.
@valeriupredoi fixed this - you can ask for the mother of all grids now
Hard wired path to the configurtation file ./config-user.yml.
@valeriupredoi fixed this too: you can input your own file or you can not and the default ~/.esmvaltoon/config-user.yml will be used instead
It doesn't check whether the requested years are available.
@valeriupredoi fixed this too, start_year and end_year are mandatory fields that will allow file filtering
I added a hashable dictionary class and made a dictionary where the index was another dictionary. lol. very cool and very xzibit/pimp my ride. This was from before christmas 2019, so who knows why I did that!
@valeriupredoi has totally unpimped your ride because it's not needed :grin:

Before you start, please read our contribution guidelines.
Please discuss your idea with the development team before getting started, to avoid disappointment later. The way to do this is to open a new issue on GitHub. If you are planning to modify an existing functionality, please discuss it with the original author(s) by tagging them in the issue.

Tasks

[x] Create an issue to discuss what you are going to do, if you haven't done so already (and add the link at the bottom)
[ ] Give this pull request a descriptive title that can be used as a one line summary in a changelog
[ ] Make sure your code is composed of functions of no more than 50 lines and uses meaningful names for variables
[ ] Circle/CI tests pass. Status can be seen below your pull request. If the tests are failing, click the link to find out why.
[ ] Preferably Codacy code quality checks pass, however a few remaining hard to solve Codacy issues are still acceptable. Status can be seen below your pull request. If there is an error, click the link to find out why. If you suspect Codacy may be wrong, please ask by commenting.
[ ] Please use yamllint to check that your YAML files do not contain mistakes
[ ] (Only if really necessary) Add any additional dependencies needed for the diagnostic script to setup.py, esmvaltool/install/R/r_requirements.txt or esmvaltool/install/Julia/Project.toml (depending on the language of your script) and also to package/meta.yaml for conda dependencies (includes Python and others, but not R/Julia)
[ ] If new dependencies are introduced, check that the license is compatible with Apache2.0

If you need help with any of the tasks above, please do not hesitate to ask by commenting in the issue or pull request.

Closes #671
Closes #1170
EDIT by @valeriupredoi on 28/09/2020
Cheers @ledm for starting this and for the code you wrote! I have taken your draft and rewritten most of it, but the core functionality remains the same; I have fixed a number of outstanding issues which you nicely pointed out in this message and edited the message accordingly. I think it's in an okay shape now, although it still needs testing in anger and actual unit tests for each of the functions plus most of the functions will have to become private (for I am lazy and I really don't want to write a lot of api documentation :grin: ). Can I please ask you and @SarahAlidoost @bouweandela @jvegasbsc @nielsdrost @hb326 @mattiarighi and anyone else who wants to test this and report any issues they encounter, please :beer: I will then fix the issues, write more documentation and tests and then, Bouwe-willing, we can merge :)

@ledm I am wondering if you have had a chance to look at my review comment? It would be great if we could update the status of the pull request. Thanks.

Hi Sarah, hope you're well! I won't really have time to work on this for a while, but I believe that @valeriupredoi talked about taking over this utility. Guess he's pretty busy too though!

I am a bit less busy these days so I can defo have a stab at this next week :+1:

OK I started working on this, more tomorrow :+1: :beer:

Hi @valeriupredoi , I just tried this on some of my recipes, including the example from @SarahAlidoost , but everytime I get:
Traceback (most recent call last):
  File "/home/stef/miniconda3/envs/esmvaltool/bin/recipe_filler", line 33, in <module>
    sys.exit(load_entry_point('ESMValTool', 'console_scripts', 'recipe_filler')())
  File "/home/stef/r/esmvaltool/esmvaltool/utils/recipe_filler.py", line 377, in run
    files = list_all_files(recipe_dict, cmip_era)
  File "/home/stef/r/esmvaltool/esmvaltool/utils/recipe_filler.py", line 169, in list_all_files
    basepaths = determine_basepath(cmip_era)
  File "/home/stef/r/esmvaltool/esmvaltool/utils/recipe_filler.py", line 111, in determine_basepath
    if isinstance(get_site_rootpath(cmip_era)[1], list):
  File "/home/stef/r/esmvaltool/esmvaltool/utils/recipe_filler.py", line 93, in get_site_rootpath
    return yamlconf['drs'][cmip_era], yamlconf['rootpath'][cmip_era]
KeyError: 'drs'

hahahah, @stefsmeets serious debugging there mate, awesome :beer: I was about to reply to your first issue related to drs - but yeah, it does indeed need a lot of defaults setters. I will plug them in at some point soon, note that there is a recipe checker function in there, so I can easily add a config file checker in :+1:

or alternatively, if you find yourself with spare time on your hands - you can do it and I will buy pints at the next (real) meeting :beer:

@stefsmeets cheers for trying this out mate! Can you give it one more shot please, when you have some time (actually everyone else is invited to test) - I believe I fixed most of the bugs: you can run it just by invoking its basic call recipe_filler recipe_for_filling.yml where recipe_for_filling.yml is the recipe to be filled out. Cheers muchly :beer:

@valeriupredoi thank you. I tested the recipe_filler using this recipe (it is made based on recipe_python.yml):
# ESMValTool
# recipe_python.yml
---
documentation:
  description: |
    Example recipe that plots a map and timeseries of temperature.

  authors:
    - andela_bouwe
    - righi_mattia

  maintainer:
    - schlund_manuel

  references:
    - acknow_project

  projects:
    - esmval
    - c3s-magic

preprocessors:

  select_january:
    extract_month:
      month: 1

  annual_mean_amsterdam:
    extract_point:
      latitude: 52.379189
      longitude: 4.899431
      scheme: linear
    annual_statistics:
      operator: mean
    multi_model_statistics:
      statistics:
        - mean
      span: overlap

  annual_mean_global:
    area_statistics:
      operator: mean
    annual_statistics:
      operator: mean

diagnostics:

  timeseries:
    description: Annual mean temperature in Amsterdam and global mean since 1850.
    themes:
      - phys
    realms:
      - atmos
    variables:
      pr_amsterdam:
        short_name: pr
        mip: Amon
        dataset: CanESM2
        ensemble: r1i1p1
        exp: historical
        project: CMIP5
        preprocessor: annual_mean_amsterdam
        start_year: 2000
        end_year: 2000
    scripts: null

It resulted in the error: Could not find CMIP5 CMOR table for variable pr_amsterdam with mip Amon, see the log file here
main_log_debug.txt
I replaced pr_amsterdam with pr. Then  recipe_filler  was successful and the output recipe is:
diagnostics:
  timeseries:
    description: Annual mean temperature in Amsterdam and global mean since 1850.
    realms:
    - atmos
    scripts: null
    themes:
    - phys
    variables:
      pr:
        dataset: CanESM2
        end_year: 2000
        ensemble: r1i1p1
        exp: historical
        mip: Amon
        preprocessor: annual_mean_amsterdam
        project: CMIP5
        short_name: pr
        start_year: 2000
documentation:
  authors:
  - andela_bouwe
  - righi_mattia
  description: 'Example recipe that plots a map and timeseries of temperature.

    '
  maintainer:
  - schlund_manuel
  projects:
  - esmval
  - c3s-magic
  references:
  - acknow_project
preprocessors:
  annual_mean_amsterdam:
    annual_statistics:
      operator: mean
    extract_point:
      latitude: 52.379189
      longitude: 4.899431
      scheme: linear
    multi_model_statistics:
      span: overlap
      statistics:
      - mean
  annual_mean_global:
    annual_statistics:
      operator: mean
    area_statistics:
      operator: mean
  select_january:
    extract_month:
      month: 1

As can be seen, there is no entry for either dataset or additional_datasets in this output. Also, running this recipe with esmvaltool fails.

@SarahAlidoost cheers muchly for testing :beer: Your conclusions are both correct: pr_amsterdam would not work coz there is no CMOR variable called that way - I will put in a clause to look for the short_name and use that if present; the second run you did outputs the expected result - if you plug in
    variables:
      pr_amsterdam:
        short_name: pr
        mip: Amon
        dataset: CanESM2
        ensemble: r1i1p1
        exp: historical
        project: CMIP5
        preprocessor: annual_mean_amsterdam
        start_year: 2000
        end_year: 2000

you are fully describing the dataset with the given parameters and that means you want to use a single, fully determined dataset - the code actually went ahead and looked for it and it found it hence you still have that entry and no other additional entries, try commenting out dataset or esmemble or exp or all three, and then you'll see a whole lot more additional datasets, all different for different datasets, exps, and ensembles, projects etc :+1:

@valeriupredoi Thank you. As a test, I fully described the data to see if the output recipe will work as the original one. Also, I did some tests based on your explanations. I hope that it helps. Here a summary:
1- run recipe_filler using recipe_python that has a fully described diagnostic: the output recipe cannot be run by esmvaltool.

see recipe_python
# ESMValTool
# recipe_python.yml
---
documentation:
  description: |
    Example recipe that plots a map and timeseries of temperature.

  authors:
    - andela_bouwe
    - righi_mattia

  maintainer:
    - schlund_manuel

  references:
    - acknow_project

  projects:
    - esmval
    - c3s-magic

# datasets:
#   - {dataset: CanESM2, project: CMIP5, exp: historical, ensemble: r1i1p1}

preprocessors:

  select_january:
    extract_month:
      month: 1

  annual_mean_amsterdam:
    extract_point:
      latitude: 52.379189
      longitude: 4.899431
      scheme: linear
    annual_statistics:
      operator: mean
    multi_model_statistics:
      statistics:
        - mean
      span: overlap

diagnostics:
  timeseries:
    description: Annual mean temperature in Amsterdam and global mean since 1850.
    themes:
      - phys
    realms:
      - atmos
    variables:
      pr:
        short_name: pr
        mip: Amon
        dataset: CanESM2
        ensemble: r1i1p1
        exp: historical
        project: CMIP5
        preprocessor: annual_mean_amsterdam
        start_year: 2000
        end_year: 2000
    scripts: null

see output recipe
diagnostics:
  timeseries:
    description: Annual mean temperature in Amsterdam and global mean since 1850.
    realms:
    - atmos
    scripts: null
    themes:
    - phys
    variables:
      pr:
        dataset: CanESM2
        end_year: 2000
        ensemble: r1i1p1
        exp: historical
        mip: Amon
        preprocessor: annual_mean_amsterdam
        project: CMIP5
        short_name: pr
        start_year: 2000
documentation:
  authors:
  - andela_bouwe
  - righi_mattia
  description: 'Example recipe that plots a map and timeseries of temperature.

    '
  maintainer:
  - schlund_manuel
  projects:
  - esmval
  - c3s-magic
  references:
  - acknow_project
preprocessors:
  annual_mean_amsterdam:
    annual_statistics:
      operator: mean
    extract_point:
      latitude: 52.379189
      longitude: 4.899431
      scheme: linear
    multi_model_statistics:
      span: overlap
      statistics:
      - mean
  select_january:
    extract_month:
      month: 1

See esmvaltool main_log.text
main_log.txt
2- run recipe_filler using recipe_python that has a some information about dataset, but not project info: it results in an error by recipe_filler. It seems that the entry project cannot be missing!

see input diagnostics
diagnostics:
  timeseries:
    description: Annual mean temperature in Amsterdam and global mean since 1850.
    themes:
      - phys
    realms:
      - atmos
    variables:
      pr:
        short_name: pr
        mip: Amon
        dataset: CanESM2
        ensemble: r1i1p1
        # exp: historical
        # project: CMIP5
        preprocessor: annual_mean_amsterdam
        start_year: 2000
        end_year: 2000
    scripts: null

see recipe_filler error
Traceback (most recent call last):
  File "/home/sarah/GitHub/ESMValTool/esmvaltool/utils/recipe_filler.py", line 441, in <module>
    run()
  File "/home/sarah/GitHub/ESMValTool/esmvaltool/utils/recipe_filler.py", line 421, in run
    for cmip_era in cmip_eras:
UnboundLocalError: local variable 'cmip_eras' referenced before assignment

3-  run recipe_filler using recipe_python that has a some information about dataset and project info: the output recipe does not include entry of ensemble or exp!

see input diagnostics
diagnostics:
  timeseries:
    description: Annual mean temperature in Amsterdam and global mean since 1850.
    themes:
      - phys
    realms:
      - atmos
    variables:
      pr:
        short_name: pr
        mip: Amon
        dataset: CanESM2
        # ensemble: r1i1p1
        # exp: historical
        project: CMIP5
        preprocessor: annual_mean_amsterdam
        start_year: 2000
        end_year: 2000
    scripts: null

see output diagnostics
diagnostics:
  timeseries:
    description: Annual mean temperature in Amsterdam and global mean since 1850.
    realms:
    - atmos
    scripts: null
    themes:
    - phys
    variables:
      pr:
        dataset: CanESM2
        end_year: 2000
        mip: Amon
        preprocessor: annual_mean_amsterdam
        project: CMIP5
        short_name: pr
        start_year: 2000

@SarahAlidoost cheers for testing! good to this put through the grind - about the first issue, it's a bug in the main code that has been fixed, so I'll merge master here; second, it is my bug that I will fix right now; third used case: are you sre you have access to the full data or you passing the correct rootpath/DRS? I am running it on jasmin and I am finding loads of extra data, can you maybe post the log file please? Thanks again, Sarah :beer:

OK I fixed the issue with cmip_eras undefined and now you can use custom var names like pr_amsterdam :grin:

@valeriupredoi thank you for the quick responses. I see that issues of project and pr_amsterdam are now fixed. The first bug (first test) is still there.
Regarding the third bug, here the log file and extra information about rootpath and drs:
The recipe_filler log files: main_log.txt, and main_log_debug.txt.
Please see config-user:
# Rootpaths to the data from different projects (lists are also possible)
rootpath:
  CMIP5: [~/temp/ForRecipe/CMIP5]

# Directory structure for input data: [default]/BADC/DKRZ/ETHZ/etc
# See config-developer.yml for definitions.
drs:
  CMIP5: default

Please see files in the rootpath ~/temp/ForRecipe/CMIP5
├── pr_Amon_CanESM2_historical_r1i1p1_185001-200512.nc
├── pr_Amon_CCSM4_historical_r1i1p1_185001-200512.nc
├── pr_Amon_CCSM4_historical_r1i2p1_185001-200512 copy.nc
└── pr_Amon_CCSM4_historical_r1i2p1_185001-200512.nc

@SarahAlidoost - nice one! I have never tested it with local data (default drs), I did it now to replicate the issue you see and indeed it can't find the data - will have to fix it, cheers! Note that I was mistaken about the first (run-related issue) - I forgot this is in esmvaltool and not in esmvalcore, so that has been fixed in master in esmvalcore (duh! :grin: ) so if you update esmvalcore you won't see it no more. Also - careful with the 3rd used case - you are fixing the dataset to CanESM2 in it and the only file for CanESM2 you have in that path is the one that's already coded in the original recipe so it won't find anything else :+1:

thanks for the detailed debugging and attaching all the needed files and info BTW :beer:

Sarah, I have now added the functionality for default (local) paths and tested, if you could give it a test :+1:

@valeriupredoi thank you. My installation and environment are updated.  I found the problem. The rootpath in my conifig-user.yml started with syntax  ~. The glob in this line does not work with this syntax. I changed it and here the results:
1- Now the recipe_filler works fine with drs=DKRZ and these settings :
Config-user:
rootpath:
  CMIP5: [/home/sarah/temp/ForRecipe]

drs:
  CMIP5: DKRZ

The directory structure in the rootpath /home/sarah/temp/ForRecipe is: /home/sarah/temp/ForRecipe/NCAR/CCSM4/historical/mon/atmos/Amon/r1i1p1/latest/pr
I noticed that the DKRZ drs in the config-developers.yml is defined as:
'{institute}/{dataset}/{exp}/{frequency}/{modeling_realm}/{mip}/{ensemble}/{latestversion}/{short_name}'.
Where the {latestversion} can be a number e.g. 1 whereas the recipe_filler accepts the latest. This might not be related to this pull request, but I wanted to report it.
2- The recipe_filler doesn't work with drs=default and these settings:
rootpath:
  CMIP5: [/home/sarah/temp/ForRecipe/CMIP5]

drs:
  CMIP5: default

Because in this line, it looks for 'default' in the rootpath. When drs:CMIP5: default, it indicates that the data lies in the rootpath: CMIP5: [/home/sarah/temp/ForRecipe/CMIP5] that contains all the files without any structure. There is no need to have a rootpath: default:..... So, lines 96-102 need a fix or can be removed.

@SarahAlidoost cheers again! The problem with rootpath: default:anddrs: CMIPX: defaultis a dodgy one from the configuration point of view (esmvalcore) - you are right to try and use the setup you tried but my take on it (and how I wrote the data finder in Core and its configuration) is that if you userootpath: CMIPX:you expect some structure and if you userootpath: default:` then all the stuff is thrown in there with no structure. I will tweak the filler to work both ways, I was lazy and didn't think of all the cases :grin: Will do that tomorrow! :beer:

@SarahAlidoost - fixed! I also implemented the version lookup and selection of the latest but not hardcoding it to latest - good catch, that was legacy from @ledm from back in Medieval code times :laughing:

@valeriupredoi Thanks a lot. I see that the issue of "latest" is solved now. Just some minor comments:
1-  The recipe filler works fine with rootptah = default and drs = default, but not with roothpath=cmipx: and drs = default. I understood your explanations about the issue. So, it should be fixed in the recipe_filler or in esmvalcore? Shall we make an issue to not forget about it?
2- The recipe structure is not preserved in the output. Can we use  ruamel.yaml ? As an example, in the function add_datasets_into_recipe as:
from ruamel.yaml import YAML

    def add_datasets_into_recipe(additional_datasets, output_recipe):
    """Add the datasets into a new recipe."""
    yaml = YAML()
    yaml.default_flow_style = False
    with open(output_recipe, 'r') as yamlfile:
        # cur_yaml = yaml.safe_load(yamlfile)
        cur_yaml = yaml.load(yamlfile)
        ...........
    if cur_yaml:
        with open(output_recipe, 'w') as yamlfile:
            # yaml.safe_dump(cur_yaml, yamlfile)
            yaml.dump(cur_yaml, yamlfile)

3- There are duplicates entries in the output. Please see in the example below that entries dataset, end_year, start_year, ensemble, mip are repeated:
variables:
      pr_amsterdam:
        additional_datasets:
        - dataset: CCSM4
          end_year: 2000
          ensemble: r1i1p1
          exp: historical
          mip: Amon
          project: CMIP5
          start_year: 2000
        dataset: CCSM4
        end_year: 2000
        ensemble: r1i1p1
        mip: Amon
        preprocessor: annual_mean_amsterdam
        short_name: pr
        start_year: 2000

4- It will help to add a raise Error statement after this line if the roothpath isnot correct, e.g. when there was ~ in my roothpath.

@SarahAlidoost

1- The recipe filler works fine with rootptah = default and drs = default, but not with roothpath=cmipx: and drs = default

That works for me OK, I have this in my config:
rootpath:
  # Rootpath to CMIP5 data
  CMIP5: /home/users/valeriu/esmvaltool_var_test/cnesm
drs:
  CMIP5: default

and that dir contains
(esmvaltool-21) [valeriu@sci2 esmvaltool_var_test]$ ls -la cnesm/
total 122614
drwxr-xr-x 2 valeriu users        0 Nov  5 12:17 .
drwxr-xr-x 1 valeriu users        0 Nov 10 12:22 ..
-rw-r--r-- 1 valeriu users 64155620 Nov  5 10:39 pr_Amon_CanESM2_historicalNat_r1i1p1_185001-201212.nc
-rw-r----- 1 valeriu users 61401088 Nov  5 12:17 pr_Amon_CanESM2_historical_r1i1p1_185001-200512.nc

and my resulting autofilled recipe is:
diagnostics:
  diag_pr:
    scripts: null
    variables:
      pr_amsterdam:
        additional_datasets:
        - dataset: CanESM2
          end_year: 2000
          ensemble: r1i1p1
          exp: historicalNat
          mip: Amon
          project: CMIP5
          start_year: 2000
        - dataset: CanESM2
          end_year: 2000
          ensemble: r1i1p1
          exp: historical
          mip: Amon
          project: CMIP5
          start_year: 2000
        dataset: CanESM2
        end_year: 2000
        ensemble: r1i1p1
        mip: Amon
        preprocessor: annual_mean_amsterdam
        project: CMIP5
        short_name: pr
        start_year: 2000

so it looks like it's doing its job - why are you seeing a non-functional case? :beer:

good call about 4.! I will add that now. About 2. and 3. -- they don't really matter in the grand scheme of things, I mean - the recipe is still perfectly machine-readable and duplicates are not affecting the data finding process and will not be used as input (the data finding applies a set and will use just one valid entry); I'll try see if I can fix the duplication though, but adding a new package as dependency is something I'd be happy to avoid, at the cost of the recipe not being pretty :+1:

actually - the duplication you observe is not an actual duplication, unless in the specific case where either additional_datasets or datasets contain the specified dataset originally, not after it got autofilled; your case:
variables:
      pr_amsterdam:
        additional_datasets:
        #- dataset: CCSM4
        #  end_year: 2000
         # ensemble: r1i1p1
        #  exp: historical
        #  mip: Amon
        #  project: CMIP5
        #  start_year: 2000
        dataset: CCSM4
        end_year: 2000
        ensemble: r1i1p1
        mip: Amon
        preprocessor: annual_mean_amsterdam
        short_name: pr
        start_year: 2000

will not work - it will result in that variable[0] index out of bounds` cryptic error :+1:

@SarahAlidoost I have to apologize for selling you fake news about that variables[0] index out of bounds issue that you reported and I see it too if no datasets (or additional_datasets) are present - I thought it got fixed in ESMValCore - but it looks like the issue is still open and I might have a stab at it right now https://github.com/ESMValGroup/ESMValCore/issues/843 -- but yeah, the duplication is not an actual problem as per my comment above :+1:

@SarahAlidoost

1- The recipe filler works fine with rootptah = default and drs = default, but not with roothpath=cmipx: and drs = default

That works for me OK, I have this in my config:
rootpath:
  # Rootpath to CMIP5 data
  CMIP5: /home/users/valeriu/esmvaltool_var_test/cnesm
drs:
  CMIP5: default

and that dir contains
(esmvaltool-21) [valeriu@sci2 esmvaltool_var_test]$ ls -la cnesm/
total 122614
drwxr-xr-x 2 valeriu users        0 Nov  5 12:17 .
drwxr-xr-x 1 valeriu users        0 Nov 10 12:22 ..
-rw-r--r-- 1 valeriu users 64155620 Nov  5 10:39 pr_Amon_CanESM2_historicalNat_r1i1p1_185001-201212.nc
-rw-r----- 1 valeriu users 61401088 Nov  5 12:17 pr_Amon_CanESM2_historical_r1i1p1_185001-200512.nc

and my resulting autofilled recipe is:
diagnostics:
  diag_pr:
    scripts: null
    variables:
      pr_amsterdam:
        additional_datasets:
        - dataset: CanESM2
          end_year: 2000
          ensemble: r1i1p1
          exp: historicalNat
          mip: Amon
          project: CMIP5
          start_year: 2000
        - dataset: CanESM2
          end_year: 2000
          ensemble: r1i1p1
          exp: historical
          mip: Amon
          project: CMIP5
          start_year: 2000
        dataset: CanESM2
        end_year: 2000
        ensemble: r1i1p1
        mip: Amon
        preprocessor: annual_mean_amsterdam
        project: CMIP5
        short_name: pr
        start_year: 2000

so it looks like it's doing its job - why are you seeing a non-functional case?

My bad. I should comment/remove default in section rootpath.  You are right, it works fine :-)

@valeriupredoi thank you for fixes and explanations. I see that the issue of using ~ is now fixed, great. No further comment, as a tester, I am happy with  recipe_filler :-)

@SarahAlidoost thank you lots for the thorough testing, most helpful! :beer:  I will ad a bunch of tests and documentation and then hopefully this will be ready for code review - @bouweandela am looking at you, bro :grin:  :+1:

I gave this a quick test because it is very useful for us (working on  #1902 soon).

It still changes the order of things in the recipe I tried to fill at the moment (one diagnostic out of recipe_collins13ipcc.yml)
diagnostics and additional_datasets at the top, documentation and preprocessors at the bottom.
It changes things in the recipe that should not be touched, e.g. some optional settings therein
seasons: [0, 1, 2, 3]
to

seasons:
        - 0
        - 1
        - 2
        - 3

It would be very helpful if one could give two scenarios and it would find all datasets available for both of them, e.g. historical plus a scenario. Any chance this could be implemented?
It would also be helpful if one could find all datasets available for multiple variables.

I know 3 and 4 might be too much to ask but I try anyway ;-)

cheers for testing @ruthlorenz :beer:

I gave this a quick test because it is very useful for us (working on #1902 soon).
1. It still changes the order of things in the recipe I tried to fill at the moment (one diagnostic out of recipe_collins13ipcc.yml)
   diagnostics and additional_datasets at the top, documentation and preprocessors at the bottom.

indeed, I may go with ruamel as @SarahAlidoost suggested, the way it is now is perfectly fine for machine reading but I'm aware this is not nice for the user

2. It changes things in the recipe that should not be touched, e.g. some optional settings therein
   `seasons: [0, 1, 2, 3] `
   to

seasons:
        - 0
        - 1
        - 2
        - 3

that's fine - that is a list in yaml, so the functionality doesn't change, it's just ugly

1. It would be very helpful if one could give two scenarios and it would find all datasets available for both of them, e.g. historical plus a scenario. Any chance this could be implemented?

so a list for exp:?

2. It would also be helpful if one could find all datasets available for multiple variables.

this works fine, give it a try!

I know 3 and 4 might be too much to ask but I try anyway ;-)

OK I have now yaml dumping using the rouamel yaml as @SarahAlidoost suggested above - the style is preserved; note that for cases where null pointers are used eg scripts: null the rouamel yaml dumper doesn't dump the null string but rather keeps it empty, but in yaml.safe_load that's perfectly fine :+1:

Thanks already :-)
Yes, for example a list for exp, so it would find only those datasets which are available for both scenarios.
Same for variables. So far I think it finds all datasets for variable1 (here pr) and all datasets for variable2 (here tas), what we would need would be all datasets which are available for BOTH variables. I attached what I tried and what the outcome was. But let me know if I need to do this differently.
recipe_test_recipe_filler.yml.txt
recipe_autofilled.yml.txt

Yes, for example a list for exp, so it would find only those datasets which are available for both scenarios.

OK, I'll see what I can do about it.
About common datasets to multiple variables: very good point, a bit of a pain in the rear to implement :grin:

About common datasets to multiple variables: very good point, a bit of a pain in the rear to implement 😁

I was afraid it might be...

OK @ruthlorenz you can now run with a list of experiments for each variable eg exp: [piControl, rcp26, rcp85] :+1:

I tried:
variables:
      tas:
        short_name: tas
        start_year: 1960
        end_year: 2100
        mip: Amon
        project: CMIP6
        exp: [historical, ssp585]
        preprocessor: temperature_anomalies

Does not throw an error but also does not find any files (but I know they exist).
main_log_debug.txt
Do I need to adjust start_year, end_year? now they cover both scenarios, do we need start and end for each scenario?

this thing doesn't allow for partial data eg if the dates requested are [2000, 2010] and the available data is only [2005, 2010] that will not be added. Do you want to combine historical and ssp into one contiguous dataset, say historical ends in 2000 and ssp starts in 2000 and you want your files to be [1850, 2100]? Coz at the moment it can't do like this, it just looks for files individually with time bounds specified in start_year and end_year

@ruthlorenz I have now implemented aggregation of datasets for a list of experiments as you initially suggested ie [piControl, historical], start_year+1900, end_year=2200 will pick up the start bit from historical and whatever historical doesnt will be taken from piControl and will returns something like:
        additional_datasets:
        - project: CMIP5
          dataset: CanESM2
          exp:
          - historical
          - piControl
          ensemble: r1i1p1
          mip: Amon
          start_year: 1900
          end_year: 2200

Could you pls test when you have some? Cheers for the feedback :beer:

Thanks V!
If screening for two scenarios would work would be great. We often use time ranges which span two scenarios, e.g. historical and rcp85. And because not every historical run has a rcp counterpart (and vice versa) we need to only include runs in the recipe which are available for both (but all available runs, so filling the recipe by hand is a pain).
So far I receive a
Traceback (most recent call last):
  File "utils/recipe_filler.py", line 544, in <module>
    run()
  File "utils/recipe_filler.py", line 512, in run
    files = filter_years(files,
  File "utils/recipe_filler.py", line 189, in filter_years
    d_y = [
  File "utils/recipe_filler.py", line 190, in <listcomp>
    yr_pairs[j][1] - yr_pairs[j + 1][0]
IndexError: list index out of range

Recipe section to fill (anything wrong in here?):
  weighted_temperature_graph:
    variables:
      tas:
        short_name: tas
        start_year: 1960
        end_year: 2050
        mip: Amon
        project: CMIP6
        exp: [historical, ssp585]
        preprocessor: temperature_anomalies
    scripts:
      weighted_temperature_graph:
        script: weighting/weighted_temperature_graph.py
        weights: 'weights.nc'

main_log_debug attached.
main_log_debug.txt

cheers for re-testing @ruthlorenz lemme have a look now :beer:

OK @ruthlorenz - lots of side implementations for that to work - it works now, but only if you declare an ensemble, it's really twisted to make it work otherwise, can you pls give it a test when you have some time? Sothis entry:
      tas:
        short_name: tas
        start_year: 1960
        end_year: 2050
        mip: Amon
        ensemble: r1i1p1f1
        project: CMIP6
        exp: [historical, ssp585]

in your pilot recipe will return all the additional datasets that have both those exps and data between 1960 and 2050 for all CMIP6 models

note that this tool really needs decent data structures, can deal with gaps, but not absolute turkeys like this (on BADC):
lrwxrwxrwx 1 badc badcint 84 Nov  5 19:11 tas_Amon_EC-Earth3-Veg-LR_historical_r1i1p1f1_gr_194001-194012.nc -> ../files/d20200217/tas_Amon_EC-Earth3-Veg-LR_historical_r1i1p1f1_gr_194001-194012.nc
lrwxrwxrwx 1 badc badcint 84 Nov  5 19:11 tas_Amon_EC-Earth3-Veg-LR_historical_r1i1p1f1_gr_195901-195912.nc -> ../files/d20200217/tas_Amon_EC-Earth3-Veg-LR_historical_r1i1p1f1_gr_195901-195912.nc
lrwxrwxrwx 1 badc badcint 84 Nov  5 19:11 tas_Amon_EC-Earth3-Veg-LR_historical_r1i1p1f1_gr_196201-196212.nc -> ../files/d20200217/tas_Amon_EC-Earth3-Veg-LR_historical_r1i1p1f1_gr_196201-196212.nc
lrwxrwxrwx 1 badc badcint 84 Nov  5 19:11 tas_Amon_EC-Earth3-Veg-LR_historical_r1i1p1f1_gr_196301-196312.nc -> ../files/d20200217/tas_Amon_EC-Earth3-Veg-LR_historical_r1i1p1f1_gr_196301-196312.nc
lrwxrwxrwx 1 badc badcint 84 Nov  5 19:11 tas_Amon_EC-Earth3-Veg-LR_historical_r1i1p1f1_gr_197201-197212.nc -> ../files/d20200217/tas_Amon_EC-Earth3-Veg-LR_historical_r1i1p1f1_gr_197201-197212.nc
lrwxrwxrwx 1 badc badcint 84 Nov  5 19:11 tas_Amon_EC-Earth3-Veg-LR_historical_r1i1p1f1_gr_198701-198712.nc -> ../files/d20200217/tas_Amon_EC-Earth3-Veg-LR_historical_r1i1p1f1_gr_198701-198712.nc
lrwxrwxrwx 1 badc badcint 84 Nov  5 19:11 tas_Amon_EC-Earth3-Veg-LR_historical_r1i1p1f1_gr_198901-198912.nc -> ../files/d20200217/tas_Amon_EC-Earth3-Veg-LR_historical_r1i1p1f1_gr_198901-198912.nc
lrwxrwxrwx 1 badc badcint 84 Nov  5 19:11 tas_Amon_EC-Earth3-Veg-LR_historical_r1i1p1f1_gr_201201-201212.nc -> ../files/d20200217/tas_Amon_EC-Earth3-Veg-LR_historical_r1i1p1f1_gr_201201-201212.nc

Weird result...
I tried both, CMIP6 and CMIP5 data (both main_debug log files attached, but I do not think they help). At some point it looks in the wrong directory. The rootpath is correct, but then it goes wrong.
CMIP6:
2020-11-19 19:36:46,579 UTC [45956] DEBUG   CMIP6 root directory /net/atmos/data/cmip6/
Traceback (most recent call last):
  File "utils/recipe_filler.py", line 663, in <module>
    run()
  File "utils/recipe_filler.py", line 629, in run
    datasets = _get_datasets(recipe_dict, cmip_eras)
  File "utils/recipe_filler.py", line 524, in _get_datasets
    datasets = _find_all_datasets(cmip_eras)
  File "utils/recipe_filler.py", line 502, in _find_all_datasets
    institutes = os.listdir(os.path.join(site_path, activity))
FileNotFoundError: [Errno 2] No such file or directory: '/net/atmos/data/cmip6/CMIP'

CMIP5:
2020-11-19 19:40:48,579 UTC [906] DEBUG   CMIP5 root directory /net/atmos/data/cmip5/
Traceback (most recent call last):
  File "utils/recipe_filler.py", line 663, in <module>
    run()
  File "utils/recipe_filler.py", line 629, in run
    datasets = _get_datasets(recipe_dict, cmip_eras)
  File "utils/recipe_filler.py", line 524, in _get_datasets
    datasets = _find_all_datasets(cmip_eras)
  File "utils/recipe_filler.py", line 505, in _find_all_datasets
    os.listdir(os.path.join(site_path, activity, institute)))
NotADirectoryError: [Errno 20] Not a directory: '/net/atmos/data/cmip5/README-rsync.txt'

Be aware that I have ETHZ datastructure, might be the issue, its
CMIP6: '{exp}/{mip}/{short_name}/{dataset}/{ensemble}/{grid}/'
CMIP5: '{exp}/{mip}/{short_name}/{dataset}/{ensemble}/'
so no "institute" folder. This should actually make it easier BUT
if the multiple scenarios cause that much issues and it then depends on the datastructure we might be better off leaving it aside? In that case we might need individual tools for the different institutes?
main_log_debug.txt
main_log_debug.txt

hi @ruthlorenz thanks a bunch for testing! It works fine on Jasmin, so this means there is a tiny issue with the path, I think I am adding a CMIP to the path that shouldn't be, can you pls post here the full path to any one of your files? :beer:

/net/atmos/data/cmip6/historical/Amon/tas/ACCESS-CM2/r1i1p1f1/gn/tas_Amon_ACCESS-CM2_historical_r1i1p1f1_gn_185001-201412.nc

@ruthlorenz you are correct - the ETHZ drs is well different than all others (had forgotten about it) - I customized it for ETHZ too, I believe all should work now for you (famous last words :grin: )

OK @ruthlorenz I finally fixed the ETHZ case, you'd be proud to know I have named a dummy dataset after you so I can test thoroughly; I have a now a multitude of test cases that reproduce all the scenarios (known to me), so I am fairly sure your case will now work fine, cheers for sticking with it :beer:

OK this thing has taken way more time than I was planning to put in it, but with the help of you guys @SarahAlidoost @ruthlorenz @stefsmeets I reckon we have a solid tool now; I switched to review readiness, I still need to add documentation that I will do right now, but I have a strong test suite (88% coverage) that should take care of all possible scenarios; @SarahAlidoost and @ruthlorenz you guys tested this thoroughly, I owes you a beer, if you would like to give it one last test, if not, maybe give it an Approve (provided Ruth's last test will work); @bouweandela - it's a lot of code to review so I'm not expecting you to do it, maybe just have a look at the tests and run a test yourself? Also - any clue why the CI test is failing? Cheers, all :beer:

@ledm thank you for this utility. I had a look at the script, pull request checklist, and its related issues. It seems that this pull request is still a work in progress. Could you please either fix the remaining issues or change it to a draft pull request? Also, to review this task, I run the script on recipe_python using the following diagnostics settings:
pr:
        preprocessor: preprocessor1
        mip: Amon
        start_year: 2000
        end_year: 2002
        dataset: CanESM2
        exp: historical
        ensemble: r1i1p1
        project: CMIP5

It results in an error TypeError: sequence item 0: expected str instance, list found related to the function get_rootpath(). Because there is a list of CMIP5: [~/cmip5_inputpath1, ~/cmip5_inputpath2] in rootpath section in my config_user.yml file. This needs to be fixed somehow.

this is historical and I fixed this - I think you also tested for it, so I am going to dismiss (ugly word, sorry) the review :+1:

@ruthlorenz and @SarahAlidoost you guys are legends, cheers very much for testing heavily and sorry you had to come back to it quite a few times - taking care of all sorts of these non-standard corner cases is a pain in the rear :grin:

I still keep getting errors, mainly reading the config-user.yml.
I just have:
rootpath:
  default: ~/default_inputpath

and it errors out with:
Traceback (most recent call last):
  File "/home/stef/miniconda3/envs/esmvaltool/bin/recipe_filler", line 33, in <module>
    sys.exit(load_entry_point('ESMValTool', 'console_scripts', 'recipe_filler')())
  File "/home/stef/r/esmvaltool/esmvaltool/utils/recipe_filler.py", line 377, in run
    files = list_all_files(recipe_dict, cmip_era)
  File "/home/stef/r/esmvaltool/esmvaltool/utils/recipe_filler.py", line 169, in list_all_files
    basepaths = determine_basepath(cmip_era)
  File "/home/stef/r/esmvaltool/esmvaltool/utils/recipe_filler.py", line 111, in determine_basepath
    if isinstance(get_site_rootpath(cmip_era)[1], list):
  File "/home/stef/r/esmvaltool/esmvaltool/utils/recipe_filler.py", line 93, in get_site_rootpath
    return yamlconf['drs'][cmip_era], yamlconf['rootpath'][cmip_era]
KeyError: 'CMIP5'

Should it fallback to default?

@stefsmeets you have a CMIPX in your drs and you don't have one in rootpath - that's not a conformal config-file - this thing is not supposed to check and fix your config file :grin:

@stefsmeets you have a CMIPX in your drs and you don't have one in rootpath - that's not a conformal config-file - this thing is not supposed to check and fix your config file 😁

Yes, that's a good point and fixes that problem! Although ESMValTool does not complain and I'm lazy so I dump all my data in 1 directory 😅

@stefsmeets did you pull the latest version? Because by looking at your error above it looks to me you haven't :+1:

@stefsmeets did you pull the latest version? Because by looking at your error above it looks to me you haven't 👍

Yeah, good point 🤦♂ It works with the latest version, nvm my comments!

cheers dude @stefsmeets :beer: @ruthlorenz you mentioned that this doesn't work on Mistral or was I too sleepy to catch the right comment :coffee:

Worked fine on our machine (which is neither Mistral nor Jasmin but the ones at ETHZ...). And it would be very unfortunate if it only worked on Mistral and Jasmin because its integrated into the JupyterHub ;-) that was the comment....

cheers @ruthlorenz - that's encouraging, I reckon it's best we merged this soon so people can start using it in anger and find issues so we can fix them before @Peter9192 and his gang start integrating it in the Jupyter business. @bouweandela @schlunma  @stefsmeets would you be willing to gloss over the code so there is some sort of technical review plss - am aware it's a lot of code and would not expect you to go in detail :beer:

Seems to me like this PR could do with a lot of refactoring. The code is fairly fragile with so many branches and special cases, which will make it a pain to maintain. It also seems like it repeats a bunch of code that is already in ESMValCore, or should be implemented there instead. In that case, we can also make use of the code for the API.
Some comments:

Are you familiar with pathlib -> https://docs.python.org/3/library/pathlib.html ? I can highly recommend having a look at it, because it makes working with paths a breeze
Can you also have a look at the codacy issues? There are a lot of warnings related to logger statements. Usually logger statements are constructed with % for lazy evaluation at runtime.
Do we really need another yaml library for loading config files / recipes?

@schlunma and @stefsmeets cheers muchly for your review! I have addressed all the comments/suggestions in one way or another. Stef, here some answers to your q's:

Seems to me like this PR could do with a lot of refactoring. The code is fairly fragile with so many branches and special cases, which will make it a pain to maintain

unfortunately it needs to deal with the lack of standardization that we face in terms of all the DRS's, rootpaths etc, have a look at the _data_finder we have in Core, same sort of treatment

. It also seems like it repeats a bunch of code that is already in ESMValCore, or should be implemented there instead. In that case, we can also make use of the code for the API.

not quite, there are subtle differences and this thing needs a bit less strictness than the Core checks (see my comments above)

Some comments:
* Are you familiar with `pathlib` -> https://docs.python.org/3/library/pathlib.html ? I can highly recommend having a look at it, because it makes working with paths a breeze

the use of os.path is basic here, the problem is with the variety of path structures that is not solved by using Pathlib

* Can you also have a look at the codacy issues? There are a lot of warnings related to logger statements. Usually logger statements are constructed with % for lazy evaluation at runtime.

yeah good point, but we do use the f-string formatting pretty much everywhere

* Do we really need another yaml library for loading config files / recipes?

see my comment above and the discussion with @SarahAlidoost - that's needed for correct fromatting of the output recipe, not sure why pyyaml doesn't do that :+1:

Seems to me like this PR could do with a lot of refactoring. The code is fairly fragile with so many branches and special cases, which will make it a pain to maintain

unfortunately it needs to deal with the lack of standardization that we face in terms of all the DRS's, rootpaths etc, have a look at the _data_finder we have in Core, same sort of treatment

In my opinion, _data_finder could also use some work. Maybe some of the problems you encounter implementing this could be used to identify problems that need to be addressed in ESMValCore to make the interface more flexible and expose the necessary functionality from there.

. It also seems like it repeats a bunch of code that is already in ESMValCore, or should be implemented there instead. In that case, we can also make use of the code for the API.

not quite, there are subtle differences and this thing needs a bit less strictness than the Core checks (see my comments above)

If the _data_finder needs to be adjusted to enable this use case, then it would be better in my opinion to open an issue in ESMValCore and address the root cause to expose the required functionality. With this PR we are repeating similar, but subtly different code in two places, which will cause the code to diverge and make it difficult to fix one thing without breaking the other.

Some comments:
* Are you familiar with `pathlib` -> https://docs.python.org/3/library/pathlib.html ? I can highly recommend having a look at it, because it makes working with paths a breeze

the use of os.path is basic here, the problem is with the variety of path structures that is not solved by using Pathlib

To me it does not matter if the os.path use is basic, because it will make the code much clearer. In my encounters with os.path and pathlib.Path is that the latter often leads to better and more concise design, which, again, will make the code easier to maintain.

* Can you also have a look at the codacy issues? There are a lot of warnings related to logger statements. Usually logger statements are constructed with % for lazy evaluation at runtime.

yeah good point, but we do use the f-string formatting pretty much everywhere

Not being able to use f-strings with the logger surely is annoying, but the reason is for better performance. f-strings are executed at runtime, whereas %-strings are lazy so they can be bunched up by the logger and formatted written in one go. Also, it adds a lot of noise in the Codacy report which may hide other warnings (Some of them are actually useful 😅).

* Do we really need another yaml library for loading config files / recipes?

see my comment above and the discussion with @SarahAlidoost - that's needed for correct fromatting of the output recipe, not sure why pyyaml doesn't do that 👍

Would it be an idea to have a dedicated recipe writer in ESMValCore? We are going to need that in the future anyway, PyYaml is extremely flexible with regards to defining custom dumpers and representers.

Seems to me like this PR could do with a lot of refactoring. The code is fairly fragile with so many branches and special cases, which will make it a pain to maintain

unfortunately it needs to deal with the lack of standardization that we face in terms of all the DRS's, rootpaths etc, have a look at the _data_finder we have in Core, same sort of treatment

In my opinion, _data_finder could also use some work. Maybe some of the problems you encounter implementing this could be used to identify problems that need to be addressed in ESMValCore to make the interface more flexible and expose the necessary functionality from there.

I agree, but due to time constraints (people kinda need this) and lack of manpower, I wil defer this to later, maybe a nice project for a future release

. It also seems like it repeats a bunch of code that is already in ESMValCore, or should be implemented there instead. In that case, we can also make use of the code for the API.

not quite, there are subtle differences and this thing needs a bit less strictness than the Core checks (see my comments above)

If the _data_finder needs to be adjusted to enable this use case, then it would be better in my opinion to open an issue in ESMValCore and address the root cause to expose the required functionality. With this PR we are repeating similar, but subtly different code in two places, which will cause the code to diverge and make it difficult to fix one thing without breaking the other.

this deals with a slightly different approach - data finder starts with a lot of information the user provides and looks specifically for matching files, this implementation deals with a bull in a china shop only to harness the bull depending what data specifics the user needs; I believe at some point we'd want to generalize the data finder with this type of functionality, but until then it's best if we do it piecemeal and allow users to use new developments faster than if we kept them in the lab longer for comprehensive changes

Some comments:
* Are you familiar with `pathlib` -> https://docs.python.org/3/library/pathlib.html ? I can highly recommend having a look at it, because it makes working with paths a breeze

the use of os.path is basic here, the problem is with the variety of path structures that is not solved by using Pathlib

To me it does not matter if the os.path use is basic, because it will make the code much clearer. In my encounters with os.path and pathlib.Path is that the latter often leads to better and more concise design, which, again, will make the code easier to maintain.

agreed with Path from pathlib! but let's not squabble over clear and performant code here, mate - this is a tool that runs serially and does no computations whatsoever, low memory footprint and 99.9% of the time it's using is globbing through directories; plus os.path is called 16 times out of which 9 times is using join which is nye short of a string concatenation :grin:

* Can you also have a look at the codacy issues? There are a lot of warnings related to logger statements. Usually logger statements are constructed with % for lazy evaluation at runtime.

yeah good point, but we do use the f-string formatting pretty much everywhere

Not being able to use f-strings with the logger surely is annoying, but the reason is for better performance. f-strings are executed at runtime, whereas %-strings are lazy so they can be bunched up by the logger and formatted written in one go. Also, it adds a lot of noise in the Codacy report which may hide other warnings (Some of them are actually useful sweat_smile).

Yeah - I might change this - @bouweandela what are our standards for logger stringing again? I've seen both % lazy and f"{}" formatting

* Do we really need another yaml library for loading config files / recipes?

see my comment above and the discussion with @SarahAlidoost - that's needed for correct fromatting of the output recipe, not sure why pyyaml doesn't do that +1

Would it be an idea to have a dedicated recipe writer in ESMValCore? We are going to need that in the future anyway, PyYaml is extremely flexible with regards to defining custom dumpers and representers.

Good question! May be interesting to think of that :+1:

OK I'm hot on the trail of that pkg issue with ruamel:
(esmvaltool) valeriu@valeriu-PORTEGE-Z30-C:~/ESMValTool$ conda list ruamel
# packages in environment at /home/valeriu/miniconda3/envs/esmvaltool:
#
# Name                    Version                   Build  Channel
(esmvaltool) valeriu@valeriu-PORTEGE-Z30-C:~/ESMValTool$ pip install ruamel.yaml
Requirement already satisfied: ruamel.yaml in /home/valeriu/miniconda3/lib/python3.7/site-packages (0.15.87)

which is total bs since it's not actually installed correctly and conda doesn't see it - same as https://github.com/conda/conda/issues/10178
I will switch to pure conda install for it :+1:

Maybe the situation will improve once #1934 is merged, because then the image we use for ci/circleci: test job will be more up to date (it's now based on a 1 month old version of the master branch).

Yeah - I might change this - @bouweandela what are our standards for logger stringing again? I've seen both % lazy and f"{}" formatting

What @stefsmeets says

cheers fellas, OK - I'll change the f-stringies and will merge tomorrow - am honestly reaching a point that each change to this PR is equivalent to a GBU-12 Paveway II guided bomb exploding in my to-do list :laughing:

OK we're ready to get this in @stefsmeets would be so kind and approve or dismiss the change request, mate - we'll tackle your suggestions in a future overhaul of both the data finder and this, for now let's just allow people to use this and report issues that I'll fix on the go, rather than get major overhaul work started :+1: :beer:

Maybe the situation will improve once #1934 is merged, because then the image we use for ci/circleci: test job will be more up to date (it's now based on a 1 month old version of the master branch).

yeh man am liking what I'm seeing in #1934 :+1:

@bouweandela @stefsmeets I would very much like to merge this asap (today or tomorrow) since I would be very happy to include this in the release: Stef, I took care of your suggestions - the ones that dealt with possible changes (like the string formatting etc) and I staled your review buy you time to just approve this, Bouwe - I got a lot of feedback from both Sarah and Ruth and they both tested it heavily. This is a very useful bit of tool for users, so that's why it'd be very nice to have it released. The only snag is that ruamel is not in the test environment, still waiting on develop to finish when I am writing this but it should be fine :+1:

@valeriupredoi Can you have a look at the failing tests? I noticed you forgot to add the ruamel.yaml package as a dependency to to setup.py, maybe that will already solve some of the issues? The private imports are causing the other failing tests.

It would be really great if we could get this merged soon so diagnostic developers can start using it, but maybe it's not needed for the release, because diagnostic developers will almost always use a development installation of ESMValTool.

It would be really great if we could get this merged soon so diagnostic developers can start using it, but maybe it's not needed for the release, because diagnostic developers will almost always use a development installation of ESMValTool.

sounds good, lemme fix the private imports and the tests, then I'll merge and leave it for the next release, there may be bugs to fix in it anyway, well- not bugs - but lack of features, that users may unearth by using it in anger

@valeriupredoi Can you have a look at the failing tests? I noticed you forgot to add the ruamel.yaml package as a dependency to to setup.py, maybe that will already solve some of the issues? The private imports are causing the other failing tests.

OK @bouweandela  man, I fixed all tests, migrated code from private esmvalcore to here, and the env solves well both on my machine and the CI one, the only caveat is that when the new config module will get in the next core release the config developer file path needs to change here, but we'll bridge that bridge when we get to it. You wouldn't object too much about merging would you? :beer:

Thanks for fixing the private imports @valeriupredoi. It would be fine with me to merge this now.

terrific, cheers muchly @bouweandela and the other hard working reviewers, finalmente this one's in :grin:


How to Connect UART to ANT BMS

Hey,
I have a ANT BMS S16, but i have no Idea where i can Connect UART ?
Can you Help me please ?

Hi,
Please post a photo of a PCB board with a bluetooth adapter so I can help and draw the connectons.
Best regards,
Fedor.

Thanks for your help.

Take a look at http://fab.cba.mit.edu/classes/863.15/doc/tutorials/programming/bluetooth/bluetooth40_en.pdf
You should solder pins to RX/TX instead of Bluetooth module. I had a slightly different module, but still

It should be that signals are on these contacts.
You may not desolder these contacts from Bluetooth module, but you should send requests via your smartphone in this case.  In this case you need only TX from MCU on BMS board, just connect it to your receiver.
Otherwise you should send them via UART.
For the connection to the PC I used cheap converters like CH340 from Aliexpress.

So the two Pins in the left Side?
Thats what i extract from the Datasheet.
I tried to get Data via Bluetooth but the BMS ist disconnecting every few Seconds If i Connect from Other device than my phone - which ist working fine.

You should try to extract signals from RX pin and replicate them. This project only sends one command(a query for all data), but it may depend on hardware you use. I've used a slightly similar, more powerful BMS for 24S, but seems, they all have the same manufacturer. And, likely, the firmware.

I have the command, they published the protocol. It is the same which you send.

Most likely, they are not used. The pads are soldered on an opposite side to achieve a good mechanical connection.


Our game is about a toaster who hated all types of bread. You are a slice of bread that is trying to escape from the evil toaster and all of its toaster friends! Dodge toasters and collect jams and see how far you can go :)
How do you play your game?
The game will start with a storyline introduction. After the intro, press J and you will be put directly into the game. Use WASD keys to dodge the toasters. Go as far as you can and collect jam to earn extra points.
Code
Check off the items that are true.

[ x ] The game was made using the Sprig editor.
[ x ] The game is placed in the in the /games directory.
[ x ] The code is significantly different from all other games in the Sprig gallery (except for games labeled "demo").
[ x ] The game runs without errors.
[ x ] The name of the file/game contains only alphanumeric characters, -s, or _s.
[ x ] The game name is not the same as the others from gallery

Image (If an image is used)

[ x ] The image is in the /games/img directory.
[ x ] The name of the image matches the name of your file.

Thanks for your PR!

Check out our game and help Crummington the Ninth escape! We had a lot of fun making the game and hope you will have just as much playing it!

Great game to play

Note: AngelHacks Toronto Game (Team of 5)

I like that you used the case function, it's definitely not your first time with js haha, how did you learn this language?

Hello again! I remember that this was a team of 5, do the other 2 team members have github? If not, we can figure something out to contact them about their sprig.

Thank you @LucasHT22 but it actually was our first time using js! We had background knowledge in python and java which we used for some of the code. For complex parts, we referenced numerous games from the hack club gallery and had awesome help from our hackathon organizers :)

Wow that's awesome! Congrats!


Support for csr sparse matrices.

Currently, KernelExplainer only supports pandas dataframe and numpy matrices. It should be extended to support scipy.sparse.csr.csr_matrix

Okay, cool. I'll give that a try.
I also have one more doubt. Let's say I have a text data featured as BoW or TF-IDF. So, I have a separate list of mappings of feature number to the word that it corresponds to. Is there any way that I can pass this to get SHAP values of those words?

Not sure I fully understand the question. You have SHAP values for the features represented as a number and you want to get the SHAP value for the words? If there is a one-to-one mapping then SHAP values are the same.

Having sparse functionality would be extremely valuable. At the moment I run in a lot of bottlenecks working with larger datasets in TreeExplainer, e.g., with BOW models.

@psinger yeah, that would be really nice. The core part would require updating the code tree_shap.h to handle a sparse feature vector and return a sparse attribution vector.

@slundberg Wouldn't it be already enough to return a sparse matrix in shap_values and then update the plotting function accordingly?

+1 for sparse support for TreeExplainer!

It depends on if we are talking about the XGBoost LightGBM or sklearn implementations. the XGBoost and LightGBM currently are embedded in those packages. So the whole phi matrix is returned before we have the chance to make it sparse. That means the C++ implementations should be updated. You are right that if we could just import the XGBoost models and use the tree_shap.h implementation we could do dense rows but make a sparse matrix. I don't know if building a sparse matrix in python/numpy will be fast or not. Could be worth a try :) ...I can't pursue it at this moment though, so if anyone else wants to take a shot at it that would be great!

You are right, your implementations in xgboost and lightgbm return dense matrices. One could transform them to sparse matrices in this repo though, to at least somehow allow to work with sparse matrices. In the end, one better would return sparse matrices directly in xgboost and lightgbm though.
As a side question regarding this issue: I have struggles understanding why calculations on shap values are done in respective tree repos like xgboost or lightgbm but that then some further calculations are necessary within this package via tree_shap.h. Could you please shortly elaborate on that?

Initially tree_shap.h was used for sklearn-models since merging with the
core sklearn library for all their models was going to be problematic. I
also am not sure I want to maintain separate implementations in every tree
package, so I figured one common implementation would be better. The
XGBoost and LightGBM implementations came before tree_shap.h. tree_shap.h
is not used for XGBoost or CatBoost, and is only used for interaction
values with LightGBM.
On Fri, Jul 20, 2018 at 4:50 AM Philipp Singer<EMAIL_ADDRESS>wrote:

You are right, your implementations in xgboost and lightgbm return dense
matrices. One could transform them to sparse matrices in this repo though,
to at least somehow allow to work with sparse matrices. In the end, one
better would return sparse matrices directly in xgboost and lightgbm though.
As a side question regarding this issue: I have struggles understanding
why calculations on shap values are done in respective tree repos like
xgboost or lightgbm but that then some further calculations are necessary
within this package via tree_shap.h. Could you please shortly elaborate
on that?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/slundberg/shap/issues/138#issuecomment-406577714, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ADkTxRpFAR13yBtpT6FdmH5TqkJzq6UUks5uIcPrgaJpZM4U92im
.

Thanks for the elaboration, makes sense! After a quick glance, it seems to me though that tree_shap is also called via _tree_shap_ind which is called also for non-interaction shap values.
Anyways, I quickly implemented a dirty batch processing function that takes a large-scale sparse matrix and subsequently calculated the shap values for it and transforms the results to sparse matrices. The good thing is that the current shap_values  function already works out of the box with providing sparse features, but it returns a dense matrix, so one needs to transform it to sparse afterwards. The batch processing was necessary for me as I also ran into https://github.com/Microsoft/LightGBM/issues/1334.
Then, one needs to transform the dense shap values matrix to sparse and the plotting functions need a few re-implementation to work with sparse shap values and features. To try it out, I adapted the summary_plot code for dot and bar plots. One can see the changes here: https://github.com/psinger/shap/commit/2fe13faf050fd185dbcf5e781c6a90061e4c66f2
Maybe this is helpful to someone.

Thanks @psinger! We should really support returning a csr at some point, but would you did seems like a good stopgap.

@slundberg: "This would be good. It will take some work though so I can't promise any time line. You could work around it by selecting just the non-zero entries for the background and the instance you are explaining and passing those as a dense vector. This would effectively assume the sparse entries will have zero SHAP values."
I have a question how this is exactly done for sparse data sets? I already use as reference values all zeros to speed up the algorithm and the input is a numpy array with only a small fraction of non-zero values. Is this what you mean with "working around it by selecting only the non-zero entries for instance"? Or did you mean something else as to enter the input to make KernelShap more efficient?

@YanouRamon #200 has now added sparse support to KernelExplainer. So just passing a sparse matrix is enough.

@slundberg
When I pass the dense format (numpy array), the shap function works. However, when I pass the sparse matrix, I get the following error:
shap_values = explainer.shap_values(instance_idx, nsamples=5000)
0%|          | 0/1 [00:00<?, ?it/s]
Traceback (most recent call last):
File "", line 1, in
shap_values = explainer.shap_values(instance_idx, nsamples=5000)
File "C:\Users\yramon\AppData\Local\Continuum\anaconda3\lib\site-packages\shap\explainers\kernel.py", line 202, in shap_values
explanations.append(self.explain(data, **kwargs))
File "C:\Users\yramon\AppData\Local\Continuum\anaconda3\lib\site-packages\shap\explainers\kernel.py", line 331, in explain
self.addsample(instance.x, mask, w)
File "C:\Users\yramon\AppData\Local\Continuum\anaconda3\lib\site-packages\shap\explainers\kernel.py", line 490, in addsample
self.synth_data[offset:offset+self.N, groups] = x[0, groups]
ValueError: setting an array element with a sequence.

@yramon, I have the same error. Did you manage to circumvent the issue?

@yramon @ferlocar sorry for the late response, can you please send a repro of the issue - eg a code snippet that reproduces the error or the learner + dataset?  Also, what version of shap are you using?  Some bug fixes have been added to the sparse code since the original PR, especially for sparse background case.

@ferlocar  @imatiach-msft
I use the latest version of shap.
I managed to work with the sparse instance format by changing the reference value to a sparse matrix as well. Hope this works for you as well. More specifically, I did the following:
Reference_value=np.zeros(np.shape(x_test)[1])
RV=np.reshape(Reference_value,(1,len(Reference_value)))
RV=sparse.csr_matrix(RV)
explainer = shap.KernelExplainer(f, RV, link="identity")

@imatiach-msft @yramon
Sorry for the late reply. I stopped working on my project for a while.
@yramon Your solution works! That's what I was missing. Thanks a lot!


accept Buffer as image input

Warnings

:warning:
This pull request may need a CHANGELOG entry.

:warning:

Source/dom/models/ImageData.js changed, but not:

📚 its docs
🧪 its tests

That's OK as long as you're refactoring.

  Generated by :no_entry_sign: dangerJS


Global Shortcut Numpad

I can't seem to get any numpad shortcuts .. tried '1' and it really just fires on the '1'-key, not num1.
Someone has an idea?

That was actually pretty stupid, sorry.
NumPad[0..9]

This is a duplicate of #3332.


writing tfrecord with multithreading is not fast as expected

System information
Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No
OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
Ubuntu 16.04 and MacOS High Sierra 10.13.4
TensorFlow installed from (source or binary): docker and virtualenv
TensorFlow version (use command below): v1.8.0-0-g93bc2e2072
Python version: 2.7.12
Bazel version (if compiling from source):
GCC/Compiler version (if compiling from source):
CUDA/cuDNN version:
GPU model and memory:
Exact command to reproduce: N/A
Describe the problem
Tried to write tfrecord w/ and w/o multithreading, and found the speed difference is not much (w/ 4 threads: 434 seconds; w/o multithread 590 seconds).  Not sure if I used it correctly. Is there any better way to write tfrecord faster?
Source code / logs
import tensorflow as tf
import numpy as np
import threading
import time

def generate_data(shape=[15,28,60,1]):
	return np.random.uniform(size=shape)

def _bytes_feature(value):
	return tf.train.Feature(bytes_list=tf.train.BytesList(value=[value]))

def _int64_feature(value):
	return tf.train.Feature(int64_list=tf.train.Int64List(value=[value]))

def write_instances_to_tfrecord(tfrecord_file, filenames):
	tfrecord_writer = tf.python_io.TFRecordWriter(tfrecord_file)
	for i, filename in enumerate(filenames):
		curr_MFCC = generate_data()
		curr_MFCC_raw = curr_MFCC.tostring()
		curr_filename_raw = str(filename)+'-'+str(i)
		example = tf.train.Example(features=tf.train.Features(
			feature={
			'MFCC': _bytes_feature(curr_MFCC_raw),
			'filename': _bytes_feature(curr_filename_raw)
			})
		)
		tfrecord_writer.write(example.SerializeToString())
	tfrecord_writer.close()

def test():
	threading_start = time.time()
	coord = tf.train.Coordinator()
	threads = []
	for thread_index in xrange(4):
		args = (str(thread_index), range(200000))
		t = threading.Thread(target=write_instances_to_tfrecord, args=args)
		t.start()
		threads.append(t)
	coord.join(threads)
	print 'w/ threading takes', time.time()-threading_start

	start = time.time()
	write_instances_to_tfrecord('5', range(800000))
	print 'w/o threading takes', time.time()-start

if __name__ == '__main__':
	test()

This question is better asked on StackOverflow since it is not a bug or feature request. There is also a larger community that reads questions there.
If you think we've misinterpreted a bug, please comment again with a clear explanation, as well as all of the information requested in the issue template. Thanks!


Relax bloomfilter to allow non-power-of-2 sizes

... size is rounded up to next 64 bits though.

Fixed the literal


Natural only

Deserializes search_start and search_end directly into Naturals. Effectively removes the limit on input numbers.
Seeing a minor performance bump on detailed benchmarks, minor degradation on niceonly.
Should add a u128 version for public use.
Fixes #20

Detailed: 37% slowdown
Niceonly: 56% slowdown
Considering a branching method

Fixed the niceonly slowdown, pushing to main with a task to speed up the detailed branch or break into u128/Natural branches


composer.json:

1.) updated: description
2.) added: author
3.) changed: required for more flexible approach

Thank you for your effort. New release already includes following changes.


Code block not correctly escaped when using pygments

Putting
```bash
sudo patch -p1 < ~/auth.patch
```

in a markdown file causes the remainder of the document to be rendered inside of a <pre> tag. Removing the < character from the code block corrects the problem. This problem is not encountered if pygments.rb is not installed.

This line appears to be the culprit. I honestly can't see a reason for taking a different path on nil or "bash". If that differentiation has to stay, though, code needs to be passed through a function to encode HTML special characters.

I dont see the point of filtering out bash either, since pygments supports it. Maybe lang needs to be non-nil though? @bartkamphorst?

@dometto The lang does need to be non-nill and a language that pygments.rb recognizes I believe, which is why we default to text if one cannot be found.
Regardless, the line mentioned by @mchill still needs to be HTML-Escaped.

Not sure what the reasons were to put bash syntax between pre-tags. And yes, lang indeed has to be non-nil. Thanks @techwiz24 for offering to work on this!


introduce eslintrc.json and fix lint warnings

Updates

Introduce config file for eslint
Fix detected warnings

@kachick
Here's small refactoring with introducing eslintrc. Please review and merge if there's no concern! 😋

@kachick
Improved. Please review them again! 💪

Thank you for reviewing! 😋


Support React 16

@oliviertassinari
This is to support the official React 16
not sure should we add react in peerDependencies or dependencies though.

It's already supported https://github.com/oliviertassinari/react-swipeable-views/blob/5835f94383ddbc7d16baea4df8404d3f70262830/packages/react-swipeable-views/package.json#L32

I need to upgrade react-native and react-dom at the same time for the documentation.

@roderickhsiao Thanks for the effort, but you changed the demo dependencies.

Thanks man! I will just wait


Update Icon.svelte

Svelte 5 [vite-plugin-svelte] /node_modules/@iconify/svelte/dist/Icon.svelte:116:2 Self-closing HTML tags for non-void elements are ambiguous — use <span ...></span> rather than <span ... />

Thanks for noticing and creating pull request!
Merged. Published updated version as 4.0.2


Very slow IO with WSL2 even on SSD.  Running updatedb.
It takes 6 minutes the first time and 2 minutes the second run. On an SSD this should take a couple of seconds at most, at least for the second run.
root@BORKASON:~# time updatedb

real    5m50.360s
user    0m7.647s
sys     0m50.115s
root@BORKASON:~# time updatedb

real    1m55.429s
user    0m2.723s
sys     0m16.396s

What's wrong / what should be happening instead:
On an SSD this should take a couple of seconds at most.

Try adding /mnt to the PRUNEPATHS variable in /etc/updatedb.conf
Accessing /mnt files in WSL2 is slow right now #4197

Try adding /mnt to the PRUNEPATHS variable in /etc/updatedb.conf

That's the catch. I didn't consider updatedb(8) would traipse all over /mnt (an strace log would have made that obj). Thanks!

that happened much faster after trying to keep this running for almost 7 days


Error in test case method which has a parameter.

TestNG Version

Note: only the latest version is supported

I have a TestNG test case -
   @Test(enabled = false)
   public void testSignPutUrlWithTimeExpired() throws Exception {
       testSignPutUrlWithTime(-getSignedUrlTimeout());
   }

   public void testSignPutUrlWithTime(final long timeout)  {
//Some code
}

I am getting an error -

Cannot inject @Test annotated Method [testSignPutUrlWithTime] with [long].

If it is disabled then why is it giving this error in the first place?
This used to work in version 6.8.21. This error is coming since we switched to the latest 6.14.3 version.
Is the issue reproductible on runner?

[ ] Shell
[Yes ] Maven
[ ] Gradle
[ ] Ant
[ ] Eclipse
[ Yes] IntelliJ
[ ] NetBeans

Test case sample

Please, share the test case (as small as possible) which shows the issue

@noviceamateur - I don't quite understand your sample. Your error is suggesting that the method was testSignPutUrlWithTime but it doesn't have any annotation on it. Please provide a clear sample that can be used to reproduce this issue.

I suppose the annotation is on the class, so testSignPutUrlWithTime is considered as a test method and it fails because TestNG is not able to understand how to inject the param.
If true, the error message is not explicit enough.

@juherr

If true, the error message is not explicit enough.

Why do you say the error message is not explicit enough? The error states that the test method is expecting a long to be injected into it, but its not happening. What do you think should be changed here?

First, Cannot inject @Test annotated Method is wrong here.
Then, we may explain what is the expected fix (define a dataprovider or a parameter).

@juherr - Nope. That doesn't recreate the error as well.
Test class
import org.testng.annotations.Test;

@Test
public class ErrorTest {
  @Test
  public void testSignPutUrlWithTimeExpired() throws Exception {
    testSignPutUrlWithTime(100);
  }

  @Test(enabled = false)
  public void testSignPutUrlWithTime(final long timeout) {
    // Some code
  }
}

Same test runner as before.
Output
===== Invoked methods
    ErrorTest.testSignPutUrlWithTimeExpired()[pri:0, instance:com.rationaleemotions.github.issue1762.ErrorTest@16022d9d] 369241501
=====
PASSED: testSignPutUrlWithTimeExpired

===============================================
    Command line test
    Tests run: 1, Failures: 0, Skips: 0
===============================================

===============================================
Command line suite
Total tests run: 1, Failures: 0, Skips: 0
===============================================

===== Invoked methods
    TestRunner.testMethod()[pri:0, instance:com.rationaleemotions.github.issue1762.TestRunner@cb51256] 213193302
=====
===============================================
Default Suite
Total tests run: 1, Failures: 0, Skips: 0
===============================================

Process finished with exit code 0

My supposed sample
@Test
public class ErrorTest {

  @Test(enabled = false)
  public void testSignPutUrlWithTimeExpired() throws Exception {
    testSignPutUrlWithTime(100);
  }

  public void testSignPutUrlWithTime(final long timeout) {
    // Some code
  }
}

@juherr - How is the sample that you shared an example of a broken functionality in TestNG ?
The @Test is added at the class level. So all methods in the class are basically test methods. Since enabled=false is not added on top of testSignPutUrlWithTime(). Its being considered as a test method. And since its missing either a data provider or a @Parameters annotation and since there's no native injection involved here, the error comes up. Please let me know what I am missing here ?

It's not a broken functionality and it is working as expected but the error message may be more explicit.

@juherr - So you are basically saying that we need to improve the error verbiage when Tests fail due to parameter injection missing, when the use case doesn't involve a native injection. Correct ? I am just trying to ensure I understand the scope of the issue, so that the fix can be applied properly.

Yes, check my first message: https://github.com/cbeust/testng/issues/1762#issuecomment-383491384
But we can just close the issue without any 'fix' too.

@juherr - I am inclined to closing this as working as designed.


Usage of input type synonyms

I'm submitting a...

[ ] Bug report
[ ] Feature request
[ ] Documentation issue or request
[x] Other... Please describe:

I'm defining the following input type:
  "inputTypes": [
    {
      "name": "deviceInputType",
      "values": [
        {
          "value": "iPhone XS",
          "synonyms": [
            "iPhone 10 S",
            "iPhone zehn S",
            "iPhone ten S",
            "iPhone X S"
          ]
        }
      ]
    }
  ]

Then I use it in a handler:
'SomeIntent': function(device) {
    console.log(device.value)

Expected Behavior

When I say in German iPhone zehn S it gets detected by Alexa as iPhone XS.
I want to be able to use the param device.value and get the input type with the matching synonym as defined in the model.
Current Behavior

When I say in German iPhone zehn S it gets detected by Alexa as iPhone zehn S and the translation of the synonym doesn't work.
Error log
"intent": {
			"name": "SomeIntent",
			"confirmationStatus": "NONE",
			"slots": {
				"device": {
					"name": "device",
					"value": "iPhone zehn S",
					"resolutions": {
						"resolutionsPerAuthority": [
							{
								"authority": "amzn1.er-authority.echo...",
								"status": {
									"code": "ER_SUCCESS_MATCH"
								},
								"values": [
									{
										"value": {
											"name": "iPhone XS",
											"id": "57035d69cca71b5827ba8f97f7ff5060"
										}
									}
								]
							}
						]
					},
					"confirmationStatus": "NONE",
					"source": "USER"
				}
			}

Your Environment

Jovo Framework version used: 1.x.x
Operating System:

Alright I found a way to access the correct value.
I can just do device.key to find the matching input type. Closing the issue :)


Create upgrade strategy for safe kubelet upgrade without losing pods

Problem:
Kargo upgrade of kubelet is unsafe because pods may be rescheduled during the brief period of unavailability.
Proposed solution:
We should create a cluster-upgrade.yml playbook that adds new tasks for cordon+drain+wait for each kube-node host before running kubernetes/node role. After this role is evaluated, uncordon the host.
Additionally, we should ensure this is run in a way such that the cluster can support all pods currently running. We shouldn't cordon+drain all kube-nodes at the same time, but instead use batch mode to evacuate, for example, 20% of the nodes.

Hey guys. here's a quick way that I am doing this. Would love to see it get solidified and improved. I basically just have a separate Jenkins job that runs an upgrade.yml file that looks like:
---
- hosts: all
  gather_facts: true

- hosts: all
  roles:
    - { role: kubernetes/preinstall, tags: preinstall }

- hosts: etcd
  serial: 1
  roles:
    - { role: etcd, tags: etcd }

- hosts: kube-node
  serial: 1
  roles:
    - { role: etcd, tags: etcd }
    - { role: upgrade/pre-upgrade, tags: upgrade/pre-upgrade }
    - { role: kubernetes/node, tags: node }
    - { role: network_plugin, tags: network }
    - { role: upgrade/post-upgrade, tags: upgrade/post-upgrade }

- hosts: kube-master
  serial: 1
  roles:
    - { role: etcd, tags: etcd }
    - { role: kubernetes/node, tags: node }
    - { role: kubernetes/master, tags: master }

Long story short, it goes through one-by-one and upgrades etcd, kube-nodes, then kube-masters. You'll also notice pre and post upgrades, which just cordon and drain a node, then uncordon it after the upgrade. Those look like:
pre-upgrade:
---
- name: Cordon node
  command: kubectl cordon {{ inventory_hostname }}
  delegate_to: "{{groups['kube-master'][0]}}"

- name: Drain node
  command: kubectl drain --force --ignore-daemonsets --delete-local-data {{ inventory_hostname }}
  delegate_to: "{{groups['kube-master'][0]}}"

- name: Sleep for grace period for draining
  pause: seconds=30

post-upgrade:
---
- name: Uncordon node
  command: kubectl uncordon {{ inventory_hostname }}
  delegate_to: "{{groups['kube-master'][0]}}"

for reference:
--- ant31 [3:17 PM]
@mattymo shouldn't we add a special cluster-upgrade.yaml ?
it can be the same that the current one for now, but it may move over time.
the first immediate thing I see is the usage of serial
to apply changes 1 node after another
and or do drain/cordon  before doing the upgrade
I don't know I think the maintenance lifecycle is a little separated from the cluster deployment
also it would give more control over what we want to upgrade.
like I don't care about upgrading docker, it always have been a mess when I did.
Same for network, if it work it work.
So from my side most of my maintenance would only rollout master/node kubernetes components.
And I think that cluster.yaml is the the file to express taht (edited)

so what about `cp cluster.yaml cluster-upgrade.yaml`
to begin, and then think how to tweak it?

---- mattymo [3:24 PM]
in ansible 2.2 we could do batch mode instead of serial
and we would add pre-upgrade post-upgrade tasks for drain/cordon/wait and uncordon (edited)
batch lets you specify # of nodes or % of nodes which is really cool
I would assume 20% per batch is safe
on a 10 node cluster, it will upgrade 2 at a time

I can PR what I currently have if it jives with what you guys were thinking for a first cut.

Is cordon/uncordon really required all the time? For minor version upgrades, it should be ok to replace the components in-place without rescheduling. As I understand it, all components try its best to keep backwards compatibility to some degree.
Also, containers are not restarted when kubelet restarts. This should even work when kubelet is upgraded in the meantime.
I would suggest to make cordon/uncordon optional. Not sure however what should be the default.

@codablock w/o worlkoads evacuation, if something in the service units dependencies bothers docker daemon, all containers crash on docker restart, which is not good.

That's true. Isn't docker already (at least in theory) able to handle daemon restarts without container restarts?


--force option is invalid, but knife client delete tells you to use it

jkeiser@johns-mbp-3 ~/o/c/o/c/playground> knife client delete chef-validator
Do you really want to delete chef-validator? (Y/N)Y
FATAL: You must specify --force to delete the validator client chef-validator
jkeiser@johns-mbp-3 ~/o/c/o/c/playground> knife client delete chef-validator --force
Error: invalid option: --force

My version is master (though I imagine this happens on other versions):
jkeiser@johns-mbp-3 ~/o/c/o/c/playground> knife --version
Chef: 11.16.0.alpha.0

@stevendanna Am I correct this can be closed with the merge of #1667

@dpnl87 yup!


walletconnect is unstable

i am working with the version2 plugin of yours now since the first alpha came out and tried every version since.
the problem seems to be, that the walletconnect part seems to be unstable somehow, as it most of the time is does not connect the wallet, but lets it be in a loading screen (tried several wallets for this, all the same, either no reaction or a infinite loading). sometimes it works, but to get to this point i need to close, reopen and rescan the modal several times.
same for the example scene as well as selfmade scenes with own scripts and the thirdweb manager.

Identified the issue, will be fixed next release, thank you Morisander!


On-demand import of external stock market list

for example...
http://www.londonstockexchange.com/statistics/companies-and-issuers/companies-and-issuers.htm
This needs to be handled by the turbine-furnace and ultimately added to elasticsearch for retrieval by the turbine-condenser and probably atacama too.

use turbine-inlet for this

Issue #30 raised when trying to download LSE XLS.

Possible alternatives:
http://www.londonstockexchange.com/exchange/prices-and-markets/stocks/indices/summary/summary-indices-constituents.html?index=UKX&page=1 (also gives 403!)
http://www.lse.co.uk/index-constituents.asp?index=IDX:UKX&indexname=ftse_100

Other sources (no longer needed as issue #30 is resolved)
http://shares.telegraph.co.uk/indices/?index=UKX
http://shareprices.com/ftse100

Done and committed. the JMX operation(s) might need a little more polish.


Is this package still maintained?

Since there are no updates since July, I want to ask, do you @mahnunchik still maintain this package?

@aslafy-z sorry I don't have time to maintain this package 😢
It still works (and I'm using it) but sharp library has too many new features so it is required almost full refactoring.

This full refactor, is it something you're interested in or are you pleased with the plugin as it is? Do you have ideas about what a rewrite would look like? Would you change anything about the API?

Hello @TheDancingCode
I've planed the following:

Full compatibility with sharp features.
Simplified config.

But I don't have a time now 😢

@mahnunchik Can I help you? I would like to work on Node.js 12 support and some code refactorings. I also can help with PR reviews.

If I don't get any feedback in a appropriate range I will fork it with https://github.com/mahnunchik/gulp-responsive/pull/129


UPS International rate is not correct

@kylewest
For address:
InternationalAddress = new DotNetShipping.Address("345A Forbes Road Zone 6 -  Meadowlands", "", "", "Johannesburg", "", "1852", "ZA");
it returns incorrect UPS shipping rate.
Package weight: 1.4 pounds
From address:
DomesticAddress = new DotNetShipping.Address("3701 Algonquin Road", "", "", "Rolling Meadows", "IL", "60008", "US");

Which rate is incorrect and what is the correct amount? Some things to check:

Are you using negotiated rates?
Have you implemented an IRateAdjuster that is manipulating the rates?

I used the same addresses and got different rates but I also don't know your package dimensions or insured value.

var packages = new List<Package>();
packages.Add(new Package(12, 12, 4, 1.4M, 0));

var origin = new DotNetShipping.Address("3701 Algonquin Road", "", "", "Rolling Meadows", "IL", "60008", "US");
var destination = new DotNetShipping.Address("345A Forbes Road Zone 6 - Meadowlands", "", "", "Johannesburg", "", "1852", "ZA");

@kylewest
Do you think it costs $200+ to send 1.4 pounds package to South Africa?
Answers: 1. I am not using negotiated rate.
2. Adjuster IRateAdjuster is not used/implemented.
Package:
private readonly Package PackageInternational;
PackageInternational = new Package(13, 1, 2, (decimal)1.4, 0);

@kylewest
By the way correct rate should be $87.22

I have no idea how much it costs to ship to south africa, but yes it looks expensive:

Your screenshot shows:

No dimensions which can dramatically change the rate.
Negotiated rate which also dramatically changes the rate. I asked if you were using negotiated rates and you said no. Set UseNegotiatedRates = true and you should see a difference.


Uploading to maven central

Do you know of any techniques that would allow the executable jar to be uploaded to a Maven Repository with the rest of the artifacts?

it should "just work". Set attachProgramFile in the configuration to true and it will upload a .sh artifact with your other artifacts.
                   <plugin>
                        <groupId>org.skife.maven</groupId>
                        <artifactId>really-executable-jar-maven-plugin</artifactId>
                        <executions>
                            <execution>
                                <id>basepom.executable.default</id>
                                <phase>package</phase>
                                <goals>
                                    <goal>really-executable-jar</goal>
                                </goals>
                                <configuration>
                                    <classifier>shaded</classifier>
                                    <flags>${basepom.executable.flags}</flags>
                                    <programFile>${basepom.executable.name}</programFile>
                                    <attachProgramFile>true</attachProgramFile>
                                </configuration>
                            </execution>
                        </executions>
                    </plugin>

you can also use programFileType to change the type from sh.

Works great!


Update Subsegment Type Definitions to Includenamespace

Background
AWS XRay will consider segments with a namespace of "remote"  calls to a remote service. This allows the service map show which services are calling the remote service. It also allows you to select the remote service and view traces that called it.
Setting the namespace to "remote" is also done within this codebase. https://github.com/aws/aws-xray-sdk-node/blob/043be345d24dc537463a58999dd8651fefa12349/packages/mysql/lib/mysql_p.js#L251
The type definition for the Subsegment class does not contain the property namespace, meaning TypeScript projects that use this type definition are unable to override the namespace property.
Proposed Resolution
Add the namespace property as an optional string to the Subsegment type definition.

Released in 3.3.4


[Papercut][SW-12745] Fix voucher wrong enddate

This PR now compares the voucher validdate with the Date instad of the datetime from today so that the last day is now included to the days that you can use the voucher.

Questions
Answers

BC breaks?
no

Tests pass?
yes

Related tickets?
https://issues.shopware.com/#/issues/SW-12745 https://issues.shopware.com/#/issues/SW-15604

How to test?
Set the valid_from date to something and the valit_to time of a voucher to the current date and check if you can use the voucher in the frontend. You should also make a check if its available on the next date at 0:00 .

I think this is even more confusing the user, because in the backend you configure a till date. As voucher user I would suggest my voucher is still usable on the day it ends and not the day before.

I think I described it wrong. With this PR if you configure a till date it is now possible to use that voucher until the day ends. Without the PR it isnt possible to use it at the last (configured) day

Updated it.


[diem-framework] Port DiemVersion and Roles tests #9054 - (108)

Motivation
This PR ports over the tests for DiemVersion and Roles modules (bottom and top commit resp.). Coverage of these modules based solely on the new unit tests are:
DiemVersion: 100%
Roles: 98%
Test Plan

cargo test --package diem-framework --tests

/canary

:broken_heart: Test Failed - ci-test

/canary

:sunny: Canary successful

/land


tests: add prewrite before max ts is synced failpoints tests

What is changed and how it works?
This PR adds integration tests for prewriting when max ts is not synced. It is expected to receive MaxTimestampNotSynced error in prewrite when there is a leader transfer or region merge but max TS is not updated with PD TS yet.
Check List
Tests

Integration test

Release note

No release note

/merge

/merge


Implement a conversation buffer with automated summaries

Ideally, a bot like MiniAGI would keep all previous interactions in memory but summarize old interactions once the context window is full. It appears like Langchain ConversationSummaryBufferMemory implements that idea. Feature request: Provide a wrapper or alternative implementation to ConversationSummaryBufferMemory.

Voting for that!


Added account_id and query parameters to get_signature_request_list()

https://github.com/hellosign/hellosign-python-sdk/issues/75

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Calculate and cache signatures for introspection

[x] tests and linter show no problems (npm t)
[x] code is properly formatted (npm run fmt)

Landed in https://github.com/metarhia/impress/commit/19d378ea947878c7216b84c571723f1fa1a1955a


IsElement doesn't supply a default method for getElement()

In the Errai UI section of the manual (https://github.com/errai/errai/blob/master/errai-docs/src/main/asciidoc/reference.asciidoc#implement-orgjbosserraiuiclientlocalapiiselement), it's stated that IsElement provides a default implementation of getElement() but there's no default method in the code.  This section also describes the IsElement interface as being in two different packages.

Hi Steve,
Perhaps this is not clear from the documentation but there are 2 IsElement interfaces. The one in errai-common does not have a default implementation. The one in errai-ui extends the the former and it does have a default implementation.
Cheers.

Ouch! ... I created a View interface with a few other methods I want all my components to include, so when I imported the IsElement I extended my interface from, I caused the problem for all my views!
Thanks for the pointer!


ENH: adjust default build options

Switch from debug=false, optimization=2 to buildtype=release. Despite what the Meson documentation says, the former does not change the default buildtype=debug, which undesired effects on other settings derived from the value of the buildtype option.
Whit the default b_vscrt=from_buildtype option, Meson instructs the MSVC compiler to use the debug version of the VS runtime library when buildtype=debug. This causes the linker look for the debug build of all the linked DLLs. The Python distribution for Windows does not contain a debug version of the python.dll and linking fails. To avoid this issue when the user explicitly asks for a debug build, also set b_vscrt=mt.
The b_ndebug=if-release option set by meson-python also looks at the value of the buildtype and not to the value of the debug options. Setting buildtype=release has the desired effect of disabling assertions.
The prefix, python.purelibdir, and python.platlibdir build options are no more necessary after the heuristic for determining the location where installed files need to be placer from their installation location has been removed in #280. Remove them.
Fixes #381.

I'm updating the documentation. I wanted to check that this does not break anything before finishing the documentation.

Documentation updated. I still feel that the "Default build options" page belongs to "Reference" and not to "Explanations". I also feels that splitting the documentation is these sections is silly, but this is a longer story...

Running the tests with MSVC bumped into a bug in Meson https://github.com/mesonbuild/meson/pull/11655. I added a work around.


feat: convert project to typescript

Type: refactor
Description:
Objective: Convert the whole project to Typescript
Task

[x] convert codebase
[x] No tsc errors
[x] fix eslint errors
[x] update docker image
[x] update build stack to typescript
[x] update @verdaccio/types
[x] Convert tests

[x] Convert unit tests (WIP)
[x] Convert functional tests
[x] ~Convert e2e tests~ I skip this step, JS is fine for now.

[x] add logging to local-storage (this help for convert test) https://github.com/verdaccio/local-storage/pull/172

Context
I'm using the same setup we uses in all projects already converted:

common babel configuration (https://github.com/verdaccio/babel-preset)
common eslint (https://github.com/verdaccio/eslint-config-verdaccio)
common error handlers (https://github.com/verdaccio/commons-api)
add types (https://github.com/verdaccio/flow-types)

I tried as much as possible to do not refactor anything not related with this migration.
Most likely unless is a critical or potential bug future tickets should be created if something does not look nice. Otherwise this will take for ever. I rely in all test now are passing, so I guess there are no regression issues.
I plan to squash all the commits. This might affect current open PR, but I cannot wait any longer.
What's next

Fix all warnings
Remove @ts-ignore as possible
Remove // @flow or @prettier (those comments are harmless)
Improve types (there are a lot of room for improvement here)
Ideas 🤔

Resolves https://github.com/verdaccio/verdaccio/issues/1166

Thanks @sergiohgz @griffithtp for the feedback. Just one more thing, many types were chosen based on research what previous contributors tried to achieve, and the abusing of coercion or mutability of objects were and are common rule. With this migration (and previously Flow) we have caught plenty of potential issues. However, there are a lot room for improvement, so future PR are very welcome 👍