Splitting of flato13ipcc.yml into separate recipes and adding recipes for regional Figures

Description
The recipe recipe_flato13ipcc.yml is split into different parts for one or few figures for easier and faster processing:

recipe_flato13ipcc_figures_92_95.yml
recipe_flato13ipcc_figure_96.yml
recipe_flato13ipcc_figure_98.yml
recipe_flato13ipcc_figure_914.yml
recipe_flato13ipcc_figure_924.yml
recipe_flato13ipcc_figures_926_927.yml
recipe_flato13ipcc_figure_942.yml
recipe_flato13ipcc_figure_945a.yml

under the common folder: recipes/ipccwg1ar5ch9/
Non working data sets are converted to comments, the data set length it fixed in one case.
The NCEP data are updated to the new data version NCEP-NCAR-R1.
Additional, recipes which cover different Regions,  similar to figures of the IPCC AR5 (Flato et al., 2013) are added:

Seasonal cycle over land within defined regions (like Fig. 9.38)
Downscaling: Seasonal bias box plot within defined regions (like Fig. 9.39)
Downscaling: Seasonal bias box plot within defined polar and ocean regions (like Fig. 9.40)
Downscaling: observations versus models within defined regions (like Fig. 9.41)

The figures are add in two versions: one used to create
Figure 13-16 in https://gmd.copernicus.org/articles/14/3159/2021/
(recipe_weigel21gmd_figures_13_16.yml)
and one closer to the IPCC version
(recipe_flato13ipcc_figures_938_941_cmip6.yml)

Closes #2155
Link to documentation: https://github.com/ESMValGroup/ESMValTool/blob/regional_downscaling_gmddraft/doc/sphinx/source/recipes/recipe_flato13ipcc.rst
Now in: https://esmvaltool--2156.org.readthedocs.build/en/2156/recipes/recipe_ipccwg1ar5ch9.html

Before you get started

[x] ☝ Create an issue to discuss what you are going to do

Checklist
It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.

[x] 🛠 This pull request has a descriptive title
[x] 🛠 Code is written according to the code quality guidelines
[x] 🛠 Documentation is available
[x] 🛠 Tests run successfully
[x] 🛠 The list of authors is up to date
[x] 🛠 Any changed dependencies have been added or removed correctly
[x] 🛠 All checks below this pull request were successful

New or updated recipe/diagnostic

[x] 🧪 Recipe runs successfully
[x] 🧪 Recipe is well documented
[x] 🧪 Figure(s) and data look as expected from literature
[x] 🛠 Provenance information has been added

New or updated data reformatting script
- [ ] 🛠 Documentation is available
- [ ] 🛠 The dataset has been added to the CMOR check recipe
- [ ] 🧪 Numbers and units of the data look physically meaningful

To help with the number of pull requests:

🙏 We kindly ask you to review two other open pull requests in this repository

@katjaweigel the provenance should be done. Maybe you could test it...
Shall I inlcude the recipe in the overall recipe_flato13ipcc.yml?

@katjaweigel the provenance should be done. Maybe you could test it...
Shall I inlcude the recipe in the overall recipe_flato13ipcc.yml?

Hi Lisa,
thanks, I'll test it now, if this runs ok it would be great if you could include it in recipe_flato13ipcc.yml.

It is also running also for me.

@katjaweigel I ran the new part in the flato recipe, works fine!
I think the pull request is ready for review...

Hi @katjaweigel, I gave a first try to this PR but I'm having trouble to run the recipe_flato13ipcc.yml on Mistral. Running it with max_parallel_task=1 leads to time limit exception and using more leads to memory issues even with the prepost nodes. Have you identified a configuration that allows to run the recipe in one go? Otherwise, would it be an option to split the recipe into smaller ones?
A second problem is that I couldn't find a suitable version of the Core to run the recipe. The stable version (v2.3) leads to provenance issues:
  File "/work/bd0854/b309192/soft/miniconda3/envs/tool/lib/python3.9/site-packages/esmvalcore/_provenance.py", line 248, in _include_provenance
    write(filename, attributes)
  File "/work/bd0854/b309192/soft/miniconda3/envs/tool/lib/python3.9/site-packages/esmvalcore/_provenance.py", line 231, in _include_provenance_png
    with Image.open(filename) as image:
  File "/work/bd0854/b309192/soft/miniconda3/envs/tool/lib/python3.9/site-packages/PIL/Image.py", line 2953, in open
    im = _open_core(fp, filename, prefix, formats)
  File "/work/bd0854/b309192/soft/miniconda3/envs/tool/lib/python3.9/site-packages/PIL/Image.py", line 2939, in _open_core
    im = factory(fp, filename)
  File "/work/bd0854/b309192/soft/miniconda3/envs/tool/lib/python3.9/site-packages/PIL/ImageFile.py", line 121, in __init__
    self._open()
  File "/work/bd0854/b309192/soft/miniconda3/envs/tool/lib/python3.9/site-packages/PIL/PngImagePlugin.py", line 694, in _open
    s = self.png.call(cid, pos, length)
  File "/work/bd0854/b309192/soft/miniconda3/envs/tool/lib/python3.9/site-packages/PIL/PngImagePlugin.py", line 187, in call
    return getattr(self, "chunk_" + cid.decode("ascii"))(pos, length)
  File "/work/bd0854/b309192/soft/miniconda3/envs/tool/lib/python3.9/site-packages/PIL/PngImagePlugin.py", line 545, in chunk_zTXt
    v = _safe_zlib_decompress(v[1:])
  File "/work/bd0854/b309192/soft/miniconda3/envs/tool/lib/python3.9/site-packages/PIL/PngImagePlugin.py", line 133, in _safe_zlib_decompress
    raise ValueError("Decompressed Data Too Large")
ValueError: Decompressed Data Too Large

It looks to me that this issue was addressed in #1307. Could you please confirm @bouweandela? To circumvent that, I tried to comment out some datasets from the recipe but this leads to diagnostic issues as some datasets cannot be used optionally in some diagnostics. Also, switching to the latest version of the Core in development mode leads to issues with the write_plots option still in place in quite a few NCL diagnostics who then crash.
@ESMValGroup/tech-reviewers, I'm open to suggestions on how to handle such a technical review, thanks.

In this setting now also the complete recipe_flato13ipcc.yml started to run, but the time I set was too short (5h), I try with 23h now (using 47CPUs on shared to have enough memory).

@axel-lauer thanks! I didn't know that interactive jobs can access the whole memory (and I doubt it is a good setting for the stability of Mistral, rather the reason why the pp nodes have trouble, when it is too crowded there).

I think there are reasons why nobody tested that for the last release, as far as I'd read: #2198

I tried running it, but it failed because of missing CMIP3 and CMIP5 data, see https://github.com/ESMValGroup/ESMValTool/issues/2198#issuecomment-872966389.

@remi-kazeroni unfortunately it also did not run with 23 hours. But I guess the time is not the problem, it seems nothing really happend after 2.5 hours any more, so I think it got stuck for some reason (but there is no obvious error). I tell you if I find out more (I'll start it again in an other queue now, but I'm on holidays coming week so I'll be slow to test what happens.)

Thanks @LisaBock ! Now it produces at least an understandable error:

File "/mnt/lustre02/work/bd1083/b380216/CORE202102/ESMValTool/esmvaltool/diag_scripts/ipcc_ar5/ch09_fig09_14.py", line 177, in multi_model_merge
equalise_attributes(cube_list)
File "/work/bd0854/b380216/anaconda3/envs/esmvaltool202106/lib/python3.9/site-packages/iris/experimental/equalise_cubes.py", line 30, in equalise_attributes
raise Exception(emsg)
Exception: The function "iris.experimental.equalise_cubes.equalise_attributes" has been moved.
Please replace "iris.experimental.equalise_cubes.equalise_attributes()" with "iris.util.equalise_attributes()".

I try now if it runs if I change that function.

I don't think that equalise_attributes has changed substantially. It is rather either that with the updated version of iris the checks on time have tightened, or that the data has changed since the last time this was checked.
Perhaps you can track down which data input files exactly are affected and try to merge them in a jupyter notebook or an ipython console directly using iris to explore the problem with the time coordinate?

@katjaweigel Release v2.5 is approaching quickly (the feature freeze will be on 2022-02-21). Since this pull request is marked for v2.5, could you briefly comment if you further intend to include this in the new release and/or if you are facing any problems with it? Thanks!!

I agree that this recipe is too large and thus a challenge for testing (and further extensions). In order to move forward from the now stale discussion at the workshop, I would therefore like to propose to create a folder in recipes e.g. called flato13ipcc and then split recipe_flato13ipcc.yml by figure. The new smaller recipes are then put in this folder. That would make handling of this recipe a lot easier, for users as well as for developers yet things are still grouped and clearly marked as parts of flato13ipcc.

I like this idea @axel-lauer!

+1 from me as well. I suppose there is a place for big recipes where a lot of different figures are generated for the same set of datasets. But this isn't the case here and needs a separate discussion.

Yes, that would be good solution for this large recipe!

@katjaweigel I noticed that there are still considerable changes to recipe_flato13ipcc.yml in this PR. From the discussion above I think it would be best to

Revert all the changes you did to recipe_flato13ipcc.yml
Create a new directory recipes/recipe_flato13ipcc
Move the original recipe_flato13ipcc.yml into this directory
Move the new recipe_flato13ipcc_regional_downscaling.yml into this and rename it to something like recipe_flato13ipcc_figures_938-941.yml (maybe there is a better title, but I think it would be nice to include the figure numbers to the recipe).
Adapt the doc correspondingly

In subsequent PRs (for v2.6) we can then split the original recipe_flato13ipcc.yml into smaller pieces.

@schlunma Yes, the recipe should not to be merged like that. I had started to work my way through the changes from Remi, but I'm rather delayed to to other things I needed to do. So far I did not manage to get a running version of the recipe again, due to the missing fx data. I'll do the changes you propose above and the remaining corrections from Remi, but I'm not sure how fast I'll manage. What is the current time line since we are already well after the planed feature freeze for the tool?

We will publish a final release candidate of the core sometime this week, and then do another round of testing with this. If you can finish this PR by the end of this week I think we could include it into v2.5, if not then I propose to move it to v2.6.

@remi-kazeroni I hope I managed to answer all open issues above. One led to a larger change, because I'd unfortunately never tested, if the option with two multi model means is working (and due to your corrections of the header I got the idea, that I actually should check what it is doing). I needed to add two NCL functions, which where part of ESMValTool v1 but I didn't find them in the current version of v2 anymore (get_unique_difference and get_unique_intersection from ESMValTool v1 diag_scripts/lib/ncl/contributed_new.ncl).
@schlunma I split the recipes as suggested (recipe_flato13ipcc/recipe_flato13ipcc.yml and recipe_flato13ipcc/recipe_flato13ipcc_figures_938_941.yml ) and reverted recipe_flato13ipcc/recipe_flato13ipcc.yml to main. I updated doc/sphinx/source/recipes/recipe_flato13ipcc.rst (but I did not split it, only mentioned which recipe produces which figures within it).
I commented bcc-csm1-1 and inmcm4 out in  recipe_flato13ipcc/recipe_flato13ipcc.yml due to https://github.com/ESMValGroup/ESMValTool/issues/2408
I tested recipe_flato13ipcc/recipe_flato13ipcc_figures_938_941.yml (for one and two multi model means), but not recipe_flato13ipcc/recipe_flato13ipcc.yml.
I'll open a new issue for the splitting of the recipe.
I'm not sure, if it is a good idea, to merge it now or better iterate it more/wait for the new release?

Thanks for all the changes @katjaweigel!

I'm not sure, if it is a good idea, to merge it now or better iterate it more/wait for the new release?

Since I want to do the release today I think it's best to merge this afterwards. We already postponed the release by quite some time and I don't want to postpone it any further. We can thoroughly test this PR afterwards and then get it merged pretty soon.
Would that be okay for you?

While checking the PR, I also tried to run the main recipe_flato13ipcc a few times using v2.5. For each failure due to a dataset, I removed all occurrences of the dataset just to get a feeling of how many distinct problems remain. Here is what I got:

Impossible to download the dataset below, even manually on ESGF. I think this is an ESGF issue. If the areacello file is really needed, we may have to comment out the dataset. Otherwise, another file could be used for the calculations, I guess.

ERROR   esmvalcore.esgf._download:445 Failed to download ESGFFile:CMIP6/CMIP/NCAR/CESM2-WACCM/historical/r1i1p1f1/Ofx/areacello/gn/v20190227/areacello_Ofx_CESM2-WACCM_hist\
orical_r1i1p1f1_gn.nc on hosts ['esgf-data.ucar.edu'], error message Failed to download file /scratch/b/b309192/data/CMIP6/CMIP/NCAR/CESM2-WACCM/historical/r1i1p1f1/Ofx/areacello/gn/v20190227/areacello_Ofx_\
CESM2-WACCM_historical_r1i1p1f1_gn.nc, errors:
http://esgf-data.ucar.edu/thredds/fileServer/esg_dataroot/CMIP6/CMIP/NCAR/CESM2-WACCM/historical/r1i1p1f1/Ofx/areacello/gn/v20190227/areacello_Ofx_CESM2-WACCM_historical_r1i1p1f1_gn.nc: 404 Client Error: 40\
4 for url: http://esgf-data.ucar.edu/thredds/fileServer/esg_dataroot/CMIP6/CMIP/NCAR/CESM2-WACCM/historical/r1i1p1f1/Ofx/areacello/gn/v20190227/areacello_Ofx_CESM2-WACCM_historical_r1i1p1f1_gn.nc

After commenting out the model CESM2-WACCM, I got problems with the file /mnt/lustre02/work/ik1017/CMIP6/data/CMIP6/CMIP/BCC/BCC-ESM1/1pctCO2/r1i1p1f1/Ofx/areacello/gn/v20190613/areacello_Ofx_BCC-ESM1_1pctCO2_r1i1p1f1_gn.nc which contains multiple latitude and longitude coordinates:

esmvalcore.cmor.check.CMORCheckError: There were errors in variable areacello:
 There are multiple coordinates with standard_name "latitude": ['lat', 'latitude']
 There are multiple coordinates with standard_name "longitude": ['lon', 'longitude']
 areacello: does not match coordinate rank
in cube:
cell_area / (m2)                    (latitude: 232; longitude: 360)
    Dimension coordinates:
        latitude                             x               -
        longitude                            -               x
    Auxiliary coordinates:
        latitude                             x               x
        longitude                            x               x

To get passed this problem, I had to comment out all BCC-* models. I reckon a data fix is probably needed but I wanted to identify the different possible issues. I finally ran into a diagnostic issue. I don't know the recipe enough to say if removing datasets triggered that:

2022-03-24 18:27:39,287 [30746] INFO     tcr.py,140     Calculating multi-model mean anomaly
[...]
ValueError: operands could not be broadcast together with shapes (140,) (150,)

in tcr_cmip6/tcr.
I'm not sure if this PR aims also at fixing the recipe_flato13ipcc.yml. If not, we could open separate issues once this PR is merged.

Is this ready to be merged?

No, sorry, I didn't manage to continue it and this week I'm in holidays.
Best regards,
Katja
BlueMail for Android herunterladen
Am 18. Juli 2022, 09:26, um 09:26, sloosvel @.***> schrieb:

Is this ready to be merged?
--
Reply to this email directly or view it on GitHub:
https://github.com/ESMValGroup/ESMValTool/pull/2156#issuecomment-1186856140
You are receiving this because you were mentioned.
Message ID: @.***>

@katjaweigel the 2.7.0 freeze is fast approaching (Monday October 17, next Monday, that is) - if you think this could be ready by then, please make it so (as Jean-Luc Picard says) :beer:

OK bumping this to Milestone 2.8 @katjaweigel pls let me know if anything of interest for 2.7 happens and I'll move it back to this release cycle :+1:

@valeriupredoi thanks, but although I had started last week I don't think I can do it. (and @remi-kazeroni has no time to review it, we are on the same meeting the whole week, so we probably have to move it to the next release again. Sorry, I hope I can gather some more time to work on ESMValTool later in Autumn.)

not a worry, take your time, and have fun in Spain! :es:

To get passed this problem, I had to comment out all BCC-* models. I reckon a data fix is probably needed but I wanted to identify the different possible issues. I finally ran into a diagnostic issue. I don't know the recipe enough to say if removing datasets triggered that:
2022-03-24 18:27:39,287 [30746] INFO     tcr.py,140     Calculating multi-model mean anomaly
[...]
ValueError: operands could not be broadcast together with shapes (140,) (150,)
in tcr_cmip6/tcr.

This one is due to a wrong length of time series in the recipe for MIROC-ES2L, it should be fixed now.

@remi-kazeroni thanks for your comments! I think I solved most now, but there are possible still issues with the calendar and it does not completely work to use two projects.

@remi-kazeroni and @axel-lauer
From my side, it should be ready now (the issue in the documentation is caused by something else). So far, it only splits the recipe_flato13ipcc.yml in the part, which was there before (recipe_flato13ipcc/recipe_flato13ipcc.yml) and a part with the new figures from @irenecionni (recipe_flato13ipcc/recipe_flato13ipcc_figures_938_941.yml).
recipe_flato13ipcc/recipe_flato13ipcc.yml
Because there are many interdependencies, I did not split it further. But because it needs to be part of this PR to add the new figures, I made a running version by commenting out all data sets, which are currently not available on Levante (taking care of all the interdependencies, so that is really runs with the remaining data sets). I could also reset it to the original version of recipe_flato13ipcc.yml but then it would not run again(the option skip_nonexistent doesn't help, because if two experiments are needed  for a diagnostic and only one is missing the diagnostic will fail). Or make a reduced, running version and a complete non running one?
recipe_flato13ipcc/recipe_flato13ipcc_figures_938_941.yml
Also here some now no longer existing data sets and a model causing a strange calendar error deep in NCL are commented out compared the version used for the ESMValTool paper. I also tested it with two projects (CMIP5 and CMIP6), adding some CMIP6 models to check, that the options described for this case in the documentation and headers are really working. But the version included in this PR only contains CMIP5 (like in the paper). I could make an additional version including CMIP6, but that would be again one extra recipe?

I started looking at the output of the regional downscaling figures (9.38 - 9.41). The good news is, that I was able to run the recipe for these figures right away. Yay! I do have, however, a few questions regarding these four figures:

Fig. 9.38: AR5 shows absolute values while our figure shows the bias. Is this intended? If so, we might want to consider renaming this figure. Also, the AR5 figure 9.38 includes pr, which is missing in our recipe. Do we want to add pr?
Fig. 9.39: AR5 also includes pr. Do we want to add pr?
Fig. 9.40: In contrast to our implementation, AR5 shows the relative bias (in %) and for different seasons (October-March and April-September). Is this indented?
Fig. 9.41: For some reason, the AR5 figure also shows values for T < 5°C while ours does not. Not sure why, but we might want to double check this.

The masked areas found by @remi-kazeroni e.g. in fig. 9.4 are caused by a bug in the masking preprocessor. @schlunma wrote a bug fix for this (https://github.com/ESMValGroup/ESMValCore/pull/1823) solving the problem. Thanks @schlunma !
As a side note, I would prefer to put the flato13ipcc recipe(s) into a folder called e.g. ipccwg1ar5ch9 (as done for ch3 of AR6) instead of "recipe_flato13ipcc". @katjaweigel if you are fine with it, I would be happy to help splitting the flato recipe into individual recipes for each figure. I think that would be much easier to maintain and to test. Just let me know what you think.

Fig. 9.41: For some reason, the AR5 figure also shows values for T < 5°C while ours does not. Not sure why, but we might want to double check this.

So far I don't find a reason - one setting, which could cause this that a different time period and probably a different observational data set is used here compared to the IPCC. I'll try to find that when Levante runs again.

Fig. 9.41: For some reason, the AR5 figure also shows values for T < 5°C while ours does not. Not sure why, but we might want to double check this.

So far I don't find a reason - one setting, which could cause this that a different time period and probably a different observational data set is used here compared to the IPCC. I'll try to find that when Levante runs again.

The reasons is the obs data set and the time range. The IPCC plot starts in 1961, this one later (the paper says 1979, but actually 1986 as currently in the recipe is probably correct - but the difference between these years is small). If I change to 1961-2000 (like in IPCC) using CRU instead of ERA-Interim the plot looks different (see file).

As a side note, I would prefer to put the flato13ipcc recipe(s) into a folder called e.g. ipccwg1ar5ch9 (as done for ch3 of AR6) instead of "recipe_flato13ipcc". @katjaweigel if you are fine with it, I would be happy to help splitting the flato recipe into individual recipes for each figure. I think that would be much easier to maintain and to test. Just let me know what you think.
@axel-lauer I started to make a suggestion how to split the main flato recipe into single figures and small groups of figures (dependent on shared preprocessors and further settings). Testing is still ongoing and I'll continue next year. What do you thing of the figure groups? I'd also need your help to assign who of the flato authors should be author in the subgroups.

Test results on Levante:  /work/bd1083/b380216/output/
recipe_flato13ipcc_figures_92_95_20221221_163935
recipe_flato13ipcc_figure_914_20221221_164129
recipe_flato13ipcc_figure_98_20221221_164130
recipe_flato13ipcc_figure_945a_20221221_171030
recipe_flato13ipcc_figure_924_20221221_171036
recipe_flato13ipcc_figures_938_941_20221221_171024
recipe_flato13ipcc_figure_942_20221221_170958
recipe_flato13ipcc_figures_926_927_20221221_171024
recipe_flato13ipcc_figure_96_20221221_172808

Found (and hopefully now solved) a bug in the setting for the time for Fig 9.39 and 9.40 - I plan to clean up and upload the corrected files later today.

Explanation of the time errors:
To use shared functions, these NCL scripts try convert month number to a date. To do that, one needs to define the starting time for the calendar. This was set wrong in the current version because:

The starting point has to be set with date and time (instead "months since 1-1-1 00:00:00" of "months since 1-1-1"), else it will get a strange value for the (not initialized) time, leading to the error:"CDMS error: Error on character time conversion: invalid hour = XX" in few cases but enough to make the program crash regularly, if many models and regions are used.
To convert month number to month since January of some year one has to substract 1.0, because January is 0 in this case but month number starts counting at 1. This error unfortunately causes the seasons to be wrongly assigned in all former versions of Fig9.39 and 9.40.
I'm actually not sure, if it is best to fix it this way, also because there is another implementation of using this function "time_operations" form esmvaltool/diag_scripts/shared/statistics.ncl on month numbers although it is made for time series: The output is also used to get weights for the months depending on the number of days in each month (with the NCL function days_in_month). But since the year is set arbitrary (because the month numbers are not assigned to a certain year but belong to the mean of a time period of several year), the number of days is not correct in all cases (due to leap years). But probably that effect is very small.

The CDMS error was also that hard to find, because:

It does not come with a line number
It appears in stderr of the NCL run
The python scrip calling the NCL script (in esmvalcore/_task.py) mixes stderr and stdout
Part of the error message is lost because it still can be in the buffer when the error is detected and is often not written completely into the log file
I found a way at least to change 3) and improve 4) but I'm not sure if it is worth to make a Core issue/PR for that?

@remi-kazeroni Updated recipe_flato13ipcc.yml and recipe_flato13ipcc_figure_96.yml to NCEP-NCAR-R1, running recipe_flato13ipcc_figure_96.yml to test.

After a strange crash probably caused by Levante issues ("slurmstepd: error") on Friday, it run successful for  NCEP-NCAR-R1 now.

@axel-lauer would be great, if you could have a look again to have a chance to finish this for the upcoming release?

@axel-lauer
Thanks a lot for checking! I'll go through the individual points later. But the split was also your idea (in several comments in this PR), so I really wonder why you now would prefer not to split it any more in this PR? The only changes to the original recipe are the commenting out of data sets, which don't work (I don't know, why that many are commented for figs. 9.26 and 9.27 anymore, but I'll try to find out, in the running version of the complete recipe I have they are also commented out, so this was done before the split), the split and the addition of Fig9.38-9.41. The largest advantages of the split of the former recipes is in my view, that everything despite Fig9.6 (and possibly the new Figures for the _cmip3 version) can be run without max_parallel_tasks:1, because they contain no CMIP3 data and it is much faster to check issues for individual figures
I'll also check, if/why the _cmip3 version does not run anymore (it did in my last try, but since that I merged several times "main" into this branch). Did you remember to set max_parallel_tasks to 1, which is necessary for CMIP3 data?

Fig 9.38: many CMIP6 models in the legend are black lines. Maybe these models are not defined in the style file used?
Yes, the diagnostic can only use one style file (which is set to the one from CMIP5 through the recipe, as in AR5, like this the other project, regardless if it is CMIP3 or CMIP6 appears in black). It should be possible to set is to CMIP6, than the CMIP5 models would be shown in black.

But the split was also your idea (in several comments in this PR), so I really wonder why you now would prefer not to split it any more in this PR?

I still think the split is a good idea but in general it is not a good idea to mix different things in a single pull request. This makes reviewing the changes very hard.

The only changes to the original recipe are the commenting out of data sets, which don't work (I don't know, why that many are commented for figs. 9.26 and 9.27 anymore,

Things like this make reviewing this mixture hard as the results look quite different from previous runs and thus requires figuring out why this is the case. Commenting this in the PR would also help.

I'll also check, if/why the _cmip3 version does not run anymore (it did in my last try, but since that I merged several times "main" into this branch). Did you remember to set max_parallel_tasks to 1, which is necessary for CMIP3 data?

Yes.

Yes, the diagnostic can only use one style file (which is set to the one from CMIP5 through the recipe, as in AR5, like this the other project, regardless if it is CMIP3 or CMIP6 appears in black). It should be possible to set is to CMIP6, than the CMIP5 models would be shown in black.

I would then recommend creating new style files that combine CMIP3/CMIP5 and CMIP5/CMIP6.

The only changes to the original recipe are the commenting out of data sets, which don't work (I don't know, why that many are commented for figs. 9.26 and 9.27 anymore,

Things like this make reviewing this mixture hard as the results look quite different from previous runs and thus requires figuring out why this is the case. Commenting this in the PR would also help.

It was comment on, on Nov 3, 2022:

katjaweigel commented on Nov 3, 2022
@remi-kazeroni and @axel-lauer
From my side, it should be ready now (the issue in the documentation is caused by something else). So far, it only splits the recipe_flato13ipcc.yml in the part, which was there before (recipe_flato13ipcc/recipe_flato13ipcc.yml) and a part with the new figures from @irenecionni (recipe_flato13ipcc/recipe_flato13ipcc_figures_938_941.yml).
recipe_flato13ipcc/recipe_flato13ipcc.yml
Because there are many interdependencies, I did not split it further. But because it needs to be part of this PR to add the new figures, I made a running version by commenting out all data sets, which are currently not available on Levante (taking care of all the interdependencies, so that is really runs with the remaining data sets). I could also reset it to the original version of recipe_flato13ipcc.yml but then it would not run again(the option skip_nonexistent doesn't help, because if two experiments are needed for a diagnostic and only one is missing the diagnostic will fail). Or make a reduced, running version and a complete non running one?

The corresponding commit is: https://github.com/ESMValGroup/ESMValTool/pull/2156/commits/14b6875b8ce0ec3b3e2e70301c53f0ca210eba25
You just need to open a lot of hidden conversations to see it. This PR just grew too large ...
I also forgot in my answer before, that was at some other changes to make recipe_flato13ipcc.yml run again/keep it running: Fixing a wrong length of a data set, commit: https://github.com/ESMValGroup/ESMValTool/pull/2156/commits/32cae8df173fea1415bf0edc078da8611adf2fa7
Changing to NCEP-NCAR-R1, commit: https://github.com/ESMValGroup/ESMValTool/pull/2156/commits/964b0303c000b7f90001d6f544105cb14aee1e4d
there might be more which I don't remember right now, but the only reason to change things in recipe_flato13ipcc.yml despite the split was, to get/keep a running version.
And which data sets are working for which figure is also much easier to check in the splitted version. If I put all figures despite 9.38-9.41 together again (what is easy to do, that recipe is still here, I even did not delete it from the pull request so far...) but that would not help to get these data sets back. At some point they probably did not work. I could check, if some of them work now, but that need new tests again. Or we just keep the non working version of recipe_flato13ipcc.yml with all data sets, but then I wonder why I had to spend so much time to test I to get it running ... (and we should at least keep the fix of the time series length and of the NCEP Version.)
Because the PR did already grew to large, I also don't like the idea to add even further changes, especially to produce new style files (changing the unit for pr is ok, I ask, which would be better, and this only affects the actual diagnostic). New style files would be again something of relevance beyond these figures and this would be in making things more different from the IPCC version.
If I remember right @LisaBock is working on combined style files, if they are available, I could include them?

If I remember right @LisaBock is working on combined style files, if they are available, I could include them?

@katjaweigel I am sorry I haven't used any combined stylesets yet.

Here is how a combined style file could look like: cmip356.style.txt
I am not sure I understand how using this style in the new recipes recipe_flato13ipcc_figures_938_941_cmip3.yml and recipe_flato13ipcc_figures_938_941_cmip6.yml could affect anything else...

Here is how a combined style file could look like: cmip356.style.txt
I am not sure I understand how using this style in the new recipes recipe_flato13ipcc_figures_938_941_cmip3.yml and recipe_flato13ipcc_figures_938_941_cmip6.yml could affect anything else...

@axel-lauer thanks for the style file. I'll try to use it. My concern towards adding it is, that I think I'd need to add it into the common folder for style files to be picked. This could raise discussions, if people like the file or would like further changes. (And I don't really see the point, why this needs to be solved differently then in the IPCC version of the figure.)

The new recipe recipe_flato13ipcc_figures_938_941_cmip3.yml did not run at all (ValueError: File tas_A1_2.nc dates do not match a recognized pattern and time can not be read from the file), my review comments apply to recipe_flato13ipcc_figures_938_941_cmip6.yml only.

I ran recipe_flato13ipcc_figures_938_941_cmip3.yml again, and it worked without problems, see:
/work/bd1083/b380216/output/recipe_flato13ipcc_figures_938_941_cmip3_20230214_112611/

The new recipe recipe_flato13ipcc_figures_938_941_cmip3.yml did not run at all (ValueError: File tas_A1_2.nc dates do not match a recognized pattern and time can not be read from the file), my review comments apply to recipe_flato13ipcc_figures_938_941_cmip6.yml only.

I ran recipe_flato13ipcc_figures_938_941_cmip3.yml again, and it worked without problems, see: /work/bd1083/b380216/output/recipe_flato13ipcc_figures_938_941_cmip3_20230214_112611/

The reason is probably that you're using a stable version of the Core (v2.7.0/1) while @axel-lauer must be using a development installation of the Core which leads to error when reading CMIP3 datasets. I've just opened an issue about that in the Core, see https://github.com/ESMValGroup/ESMValCore/issues/1929. I think this issue can be ignored here since the recipe runs with the last stable release of the Core. We'll fix the issue this CMIP3 data before/during the upcoming release

@remi-kazeroni thanks! At least this PR helps to find Core issues :)

My concern towards adding it is, that I think I'd need to add it into the common folder for style files to be picked.

Just like other style files that have been added for specific papers...

(And I don't really see the point, why this needs to be solved differently then in the IPCC version of the figure.)

The plots we are talking about here are not included in the IPCC AR5. But as a scientific reviewer, I think all plots produced by (new) diagnostics should make scientific sense, which - in my opinion - is not the case for a plot that shows half of the models clearly distinguishable in different colors and line styles while the other half of models is all black lines. I think there would be three options:

The same line style is used for all individual models (e.g. black lines) if it is not needed (or desired) to distinguish the individual models. In that case, also no legend showing all models is needed.
All models use different line styles (that's what I would recommend).
The plot is removed since it is not included in the IPCC AR5 anyways (would be a shame in my opinion).

With the splitting of flato13ipcc.yml into separate recipes, I think the "old" recipe_flato13ipcc.yml can now be removed to avoid confusion.

Hi @katjaweigel, I plan to do the feature freeze for the Tool on March 8. Do you think you would have time to address the remaining points from @axel-lauer's review in time? I can do another technical review this week. I've dismissed my previous "green" review as the PR has changed a lot in the meantime.
In particular, it would be great if you could update the title and description of this PR to reflect the latest changes. The scope of this PR has changed quite a lot since the initial description. Thanks!

@remi-kazeroni thanks for you message! I plan to continue the work on the remaining issues tomorrow (Wednesday 1st) and hope to finish them this week. I'll also test for the new Core then.

@remi-kazeroni thanks for you message! I plan to continue the work on the remaining issues tomorrow (Wednesday 1st) and hope to finish them this week. I'll also test for the new Core then.

Thanks @katjaweigel! Testing with the new Core is not necessary here. We recommend recipe/diagnostic developers to do their developments with the stable release of the Core (v2.7.1). The good news is: I just finished running successfully all recipes of this PR using Core v2.7.1. The testing against the new Core will be done by me during the release process later in March.
If you could address Axel's comments from the scientific review over the next days, I think we should be close to get this PR merged. Please also remove the old recipe_flato13ipcc.yml, it was a bit confusing for me when running the recipes of this PR.

@remi-kazeroni ok, thanks. Yes, I'm working on the Axel's comments. I'll push some changes now (removing recipe_flato13ipcc.yml, including the merged style file, include more models for recipe_flato13ipcc_figures_926_927.yml), but I'm still on the issues with the legends and differences in the values and have to test the unit change.

Documentation updated.

I tested all flato recipes and recipe_weigel21_gmd_figures_13_16 once again. The following two recipes crash with ESMValCore v2.8:

recipe_flato13ipcc__figure_98: This seems to be related to ESMValCore issue https://github.com/ESMValGroup/ESMValCore/issues/1962
recipe_flato13ipcc_figures_926_927: This also seems to be related to an ESMValCore problem giving the following error message:

ERROR   [1268254] Failed to run preprocessor function 'add_supplementary_variables' on the data
[...]
    data_dims = self._check_multi_dim_metadata(
  File "/work/bd0854/b380103/mambaforge/envs/esmvaltool-test/lib/python3.10/site-packages/iris/cube.py", line 1191, in _check_multi_dim_metadata
    raise iris.exceptions.CannotAddError(
iris.exceptions.CannotAddError: Unequal lengths. Cube dimension 1 => 64; metadata 'sea_area_fraction' dimension 0 => 192.

@remi-kazeroni Would you have an idea about the problem with recipe_flato13ipcc_figures_926_927? Here is the debug file: main_log_debug.txt
I also noticed the following:

Fig. 9.39pr: all entries seem to be close to zero. This might be related to a unit problem (kg m-2 s-1 vs. %)
Fig. 9.41: the axis labels are  illegibly small

In addition, I strongly recommend to add a note (e.g. using .. note::) to the documentation (in the beginning) along the lines of:
"Please note that most recipes have been modified to include only models that are (still) readily available via ESGF. Plots produced may therefore look different than the original figures from IPCC AR5."
Please also accept or decline my suggestions for

recipe_flato13ipcc_figure_924.yml
Figure9.83.ncl
Figure9.39.cnl
Figure9.40.ncl

given three weeks ago.

Dear @axel-lauer thanks for checking!
According to Remis comment I tested with the old core and I did not see that:

Fig. 9.39pr: all entries seem to be close to zero. This might be related to a unit problem (kg m-2 s-1 vs. %)

Can a difference like this be caused by the Core?

Thanks for the reminder, I accepted:

Please also accept or decline my suggestions for
recipe_flato13ipcc_figure_924.yml
Figure9.83.ncl
Figure9.39.cnl
Figure9.40.ncl
given three weeks ago

.
now, sorry that I overlooked them before (GitHub or my browser tends to hide them together with more than 200 other items in this PR).

I tested all flato recipes and recipe_weigel21_gmd_figures_13_16 once again. The following two recipes crash with ESMValCore v2.8:

recipe_flato13ipcc__figure_98: This seems to be related to ESMValCore issue https://github.com/ESMValGroup/ESMValCore/issues/1962

Yes, indeed. That seems to be the same issue with the same OBS dataset.

I fixed the font size for Fig. 41b again and added the note to the documentation.
I'll try the new Core next, to see if I can reproduce the issue in Fig 9.39pr.

@katjaweigel this is how the plots look like when I run the recipes. Please notice the differences between CMIP3 (kg m-2 s-1) and the CMIP6 version of the recipe (%). If both of your figures look fine, we could ask e.g. @LisaBock to run recipe_flato13ipcc_figures_938_941_cmip3.yml as another test...
CMIP3

CMIP6

Ok, thanks, I only checked the CMIP6 version, I thought I updated CMIP3 as well, but maybe I overlooked something (it is selected through a setting in the recipe)

Sorry, I did not know, that it was about CMIP3, the last commit should solve this (test is still running).

With that change Fig9.38pr looks also ok for recipe_flato13ipcc_figures_938_941_cmip3.yml.

Great, thanks a lot @remi-kazeroni!


How to add -q parameter using package.json with lint-staged key?

Can't figure it out :(

Hello,
when you configure a git hook (like pre-commit), you supply it the command to run. This is where you put the parameter. For example, when using husky in package.json:
"husky": {
    "hooks": {
        "pre-commit": "lint-staged -q"
    }
}

@iiroj thanks, great idea


ClimateMapped: Improve site speed

Improve the site's speed.
This will require collaboration with the backend team to ensure API calls return only the required data.
Request Link: Feature 8

Let's see the upstream work on wazimap by @DavidTheProgrammer changes things

itemize gyhh
Ticket


missing file in Cha_Bell_WCD2024_initialization.ipynb notebook

One of the cells in the Cha_Bell_WCD2024_initialization.ipynb notebook contains this code: include("scythe_func.jl").    However, 'scythe_func.jl' does not appear to be in the repository.

The notebook has been updated to remove that dependency in the v1.0.1 release.


[Feature Request] Add a "What's New" component

Overview
Notifies users about new features, fixes, and developments.
Proposal
Add a card/component in the tutorial/help menu that also shows what's new, with a summary of changes and a link to the most recent release

I'd like for the summary and latest release link to be automatically updated

https://docs.github.com/en/rest/releases/releases?apiVersion=2022-11-28#get-the-latest-release
To dynamically call the latest release. I want to display the Overview text


WIP [release-4.9] Use MC kernelArguments field instead of /etc/pivot/kernel-args file

Rework #247

/test e2e-aws-upgrade

/test e2e-aws-upgrade

Still no luck eradicating double reboots - 4.8-okd-nightly -> 4.9 + this PR

Alternative solution: https://github.com/openshift/okd-machine-os/pull/257

The e2e install here seems to have the same symptom as reported in https://github.com/openshift/okd/issues/963

Closed in favor of #257


Adds support for "blinking" matching delimiters ({[

Briefly moves the cursor to a matching delimiter when one is typed, then returns the cursor to its original position either after a configurable timeout or the user supplies more input.
Resolves #111

Thanks for your PR.
The Windows implementation of next_char_ready is missing.
And the associated declaration must not be tagged with #[cfg(unix)].
Linefeed uses WaitForSingleObject to implement next_char_ready on Windows.
Alternatively, it seems possible to highlight matching brace without depending on next_char_ready.
Regards.

Thanks for the feedback!  I have added Windows support via WaitForSingleObject.  I looked at linenoise-ng but when I ran the example, it did not seem to have this feature.
CI is passing although I do not have a Windows machine handy to try it out.

Indeed, linenoise-ng has a different behaviour: matching brace is highlighted only when the cursor is on brace (not when a brace is inserted).

See #163

Would you mind closing the PR ?


.*: Remove useless settings join_concurrent_build

What problem does this PR solve?
Issue Number: close #xxx

Remove useless settings join_concurrent_build
Use std::max(build_pipeline.streams.size(), build_pipeline.streams_with_non_joined_data.size()) as join_build_concurrency because of https://github.com/pingcap/tiflash/blob/master/dbms/src/Interpreters/Join.cpp#L637-L638

Problem Summary:
What is changed and how it works?
Check List
Tests

[ ] Unit test
[ ] Integration test
[ ] Manual test (add detailed scripts or steps below)
[ ] No code

Side effects

[ ] Performance regression: Consumes more CPU
[ ] Performance regression: Consumes more Memory
[ ] Breaking backward compatibility

Documentation

[ ] Affects user behaviors
[ ] Contains syntax changes
[ ] Contains variable changes
[ ] Contains experimental features
[ ] Changes MySQL compatibility

Release note

None

/run-all-tests

/merge

/run-unit-test


docs: add tke and ack deployment docs

Please rename ack to Alibaba Cloud Kubernetes and rename tke to Tencent Kubernetes Engine


App not updating and showing several "ENErrorDomain-Fehler 5" dialogs

Avoid duplicates

[ X] Bug is not mentioned in the FAQ
[ ] Bug is specific for Android only, for general issues / questions that apply to iOS and Android please raise them in the documentation repository
[ X] Bug is not already reported in another issue

Describe the bug
After starting or reopening the app several "ENErrorDomain-Fehler 5" dialogs appear. Reproducable after closing and restarting the app.
App is not refreshing it´s update date. Last update was before 2 das.
Expected behaviour
No error dialog and updating status.
Steps to reproduce the issue
Start app. 3-5 error dialogs appearing  "ENErrorDomain-Fehler 5"
Technical details

Mobile device: iphone xs, latest updates.

Duplicate of corona-warn-app/cwa-app-ios#766


resolves STREAMS-350

:+1: assuming the Travis failure is the same as it is in #244 (looks like it is).


Theme: Update to crate-docs-theme>=0.37.0.dev0, introducing dark mode

About
Use the new theme that introduces dark mode. Thanks, @msbt.
Preview
https://crate-cloud--87.org.readthedocs.build/en/87/
References

https://github.com/crate/crate-docs-theme/pull/546
https://github.com/crate/crate/pull/17001

crate-docs-theme 0.37.0 has been released.

crate-docs-theme 0.37.1.dev0 has been released, for preview purposes.

Publishing of 0.37.1 is around the corner.


Add create-collector endpoint for API

First, I want to start with a thank you for building this in public.  I was thrilled to run into this project on X a couple weeks ago, it was a much cleaner and well written version of something I felt I would need to build myself for my small suite of personal projects.
This PR adds the localhost:5775/create-collector endpoint.  My personal projects are mainly API based that often don't have a JS integration.  By adding this endpoint, users can create a collector via API without pulling the JS and leverage the IP lookup functionality.
Rust is not my strongest language so please let me know if anything needs to be reformatted or fixed!

Hey!  :)
Adding this makes a lot of sense, thanks for contributing! – I'll check it out this week and follow up.


fixes boundary param match for proper multipart request detection

Fixes issue #6

Wow, good work! I never would have caught this.


Image filename parsing assumes date case in wrong case

I'm trying to process the search-first zebrafish test case and created a subset of the data that doesn't have any overlap issues. In this subset, something in the lib_parse_filename_metadata.py fails during the parse_filename call.
The test data is here: /data/active/fractal/3D/PelkmansLab/zebrafish_subset/

 Here is an example_run.sh script
# For the demo: Run everything as the current user
USERNAME="$(whoami)"
PORT=8001
echo -e "FRACTAL_USER=$USERNAME@me.com\nFRACTAL_PASSWORD=test\nFRACTAL_SERVER=http://localhost:$PORT" > .fractal.env

###############################################################################
# THINGS TO BE CHANGED BY THE USER
# Adapt to settings to where you run the example
BASE_FOLDER_EXAMPLE="/data/homes/$USERNAME/03_fractal/fractal-demos/examples"
###############################################################################

LABEL="zebrafish_subset_test2"

###############################################################################
# IMPORTANT: modify the following lines, depending on your preferences
# 1. They MUST include a `cd` command to a path where your user can write. The
#    simplest is to use `cd $HOME`, but notice that this will create many sh
#    scripts in your folder. You can also use `cd $HOME/.fractal_scripts`, but
#    first make sure that such folder exists
# 2. They MAY include additional commands to load a python environment. The ones
#    used in the current example are appropriate for the UZH Pelkmans lab setup.
TARGET_DIR=~/.tmp_fractal
rm -r $TARGET_DIR
mkdir $TARGET_DIR
WORKER_INIT="\
export HOME=$HOME; \
cd $TARGET_DIR; \
source /opt/easybuild/software/Anaconda3/2019.07/etc/profile.d/conda.sh; \
conda activate /data/homes/fractal/sharedfractal; \
"
###############################################################################

# Set useful variables
PRJ_NAME="proj-"$USERNAME"-"$LABEL
DS_IN_NAME="input-ds-"$USERNAME"-"$LABEL
DS_OUT_NAME="output-ds-"$USERNAME"-"$LABEL
WF_NAME="WF-2x2-"$USERNAME"-"$LABEL

# Set cache path and remove any previous file from there
export FRACTAL_CACHE_PATH=`pwd`/".cache"
rm -v ${FRACTAL_CACHE_PATH}/session
rm -v ${FRACTAL_CACHE_PATH}/tasks

# Define/initialize empty project folder and temporary file
PROJ_DIR=$BASE_FOLDER_EXAMPLE/tmp_${LABEL}
rm -r $PROJ_DIR
mkdir $PROJ_DIR
TMPJSON=${PROJ_DIR}/tmp.json

###############################################################################
# IMPORTANT: modify the following lines so that they point to absolute paths
INPUT_PATH="/data/active/fractal/3D/PelkmansLab/zebrafish_subset/"
OUTPUT_PATH='/data/active/jluethi/Fractal/20221107-zebrafish-subset/'
###############################################################################

# Define useful auxiliary command (this will be removed in the future)
CMD_JSON="python aux_extract_id_from_project_json.py $TMPJSON"

# Create project
fractal -j project new $PRJ_NAME $PROJ_DIR > $TMPJSON
PRJ_ID=`$CMD_JSON project_id`
DS_IN_ID=`$CMD_JSON dataset_id "default"`
echo "PRJ_ID: $PRJ_ID"
echo "DS_IN_ID: $DS_IN_ID"

# Update dataset name/type, and add a resource
fractal dataset edit --name "$DS_IN_NAME" -t image --read-only $PRJ_ID $DS_IN_ID
fractal dataset add-resource -g "*.tif" $PRJ_ID $DS_IN_ID $INPUT_PATH

# Add output dataset, and add a resource to it
DS_OUT_ID=`fractal --batch project add-dataset $PRJ_ID "$DS_OUT_NAME"`
fractal dataset edit -t zarr --read-write $PRJ_ID $DS_OUT_ID
fractal dataset add-resource -g "*.zarr" $PRJ_ID $DS_OUT_ID $OUTPUT_PATH

# Create workflow
WF_ID=`fractal --batch task new "$WF_NAME" workflow image zarr`
echo "WF_ID: $WF_ID"

# Add subtasks

echo "{\"num_levels\": 5, \"executor\": \"cpu-low\", \"coarsening_xy\": 2, \"channel_parameters\": {\"A01_C01\": {\"label\": \"Channel 1\",\"colormap\": \"00FFFF\",\"start\": 110,\"end\": 700 }, \"A02_C02\": {\"label\": \"Channel 2\",\"colormap\": \"FF00FF\",\"start\": 110,\"end\": 1500 }, \"A03_C03\": {\"label\": \"Channel 3\",\"colormap\": \"FFFF00\",\"start\": 110,\"end\": 1500 }}}" > ${PROJ_DIR}/args_create.json
fractal task add-subtask $WF_ID "Create OME-ZARR structure" --args-file ${PROJ_DIR}/args_create.json

echo "{\"executor\": \"cpu-mid\"}" > ${PROJ_DIR}/args_yoko.json
fractal task add-subtask $WF_ID "Yokogawa to Zarr" > ${PROJ_DIR}/args_yoko.json

echo "{\"executor\": \"cpu-low\"}" > ${PROJ_DIR}/args_replicate.json
fractal task add-subtask $WF_ID "Replicate Zarr structure" --args-file ${PROJ_DIR}/args_replicate.json

echo "{\"executor\": \"cpu-low\"}" > ${PROJ_DIR}/args_mip.json
fractal task add-subtask $WF_ID "Maximum Intensity Projection" --args-file ${PROJ_DIR}/args_mip.json

# echo "{\"labeling_level\": 2, \"executor\": \"cpu-low\", \"ROI_table_name\": \"well_ROI_table\"}" > ${PROJ_DIR}/args_labeling.json
# fractal task add-subtask $WF_ID "Cellpose Segmentation" --args-file ${PROJ_DIR}/args_labeling.json

# echo "{\"level\": 0, \"measurement_table_name\": \"nuclei\", \"executor\": \"cpu-low\", \"ROI_table_name\": \"well_ROI_table\",\"workflow_file\": \"$BASE_FOLDER_EXAMPLE/03_cardio_dataset_one_2x2_well_3_channels_10_Z/regionprops_from_existing_labels_feature.yaml\"}" > ${PROJ_DIR}/args_measurement.json
# fractal task add-subtask $WF_ID "Measurement" --args-file ${PROJ_DIR}/args_measurement.json

# Apply workflow
fractal task apply $PRJ_ID $DS_IN_ID $DS_OUT_ID $WF_ID --worker_init "$WORKER_INIT"

It fails here: https://github.com/fractal-analytics-platform/fractal-tasks-core/blob/bd0408455de8c6447dc4216344ef2a765356643d/fractal_tasks_core/lib_parse_filename_metadata.py#L69
With the following error message:
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/data/homes/fractal/.conda/envs/pelkman_demo_fractalserver/lib/python3.8/site-packages/parsl/dataflow/dflow.py", line 782, in _unwrap_futures
    kwargs[key] = dep.result()
  File "/data/homes/fractal/.conda/envs/pelkman_demo_fractalserver/lib/python3.8/concurrent/futures/_base.py", line 437, in result
    return self.__get_result()
  File "/data/homes/fractal/.conda/envs/pelkman_demo_fractalserver/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/data/homes/fractal/.conda/envs/pelkman_demo_fractalserver/lib/python3.8/site-packages/parsl/dataflow/dflow.py", line 288, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
  File "/data/homes/fractal/.conda/envs/pelkman_demo_fractalserver/lib/python3.8/site-packages/parsl/dataflow/dflow.py", line 483, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/data/homes/fractal/.conda/envs/pelkman_demo_fractalserver/lib/python3.8/site-packages/parsl/app/errors.py", line 137, in reraise
    reraise(t, v, v.__traceback__)
  File "/data/homes/fractal/.conda/envs/pelkman_demo_fractalserver/lib/python3.8/site-packages/six.py", line 719, in reraise
    raise value
  File "/data/homes/fractal/.conda/envs/pelkman_demo_fractalserver/lib/python3.8/site-packages/parsl/app/errors.py", line 175, in wrapper
    return func(*args, **kwargs)  # type: ignore
  File "/data/homes/fractal/sharedfractal/lib/python3.8/site-packages/fractal_server/app/runner/parsl/__init__.py", line 57, in _task_fun
    metadata_update = _callable(
  File "/data/homes/fractal/sharedfractal/lib/python3.8/site-packages/fractal_tasks_core/create_zarr_structure.py", line 167, in create_zarr_structure
    metadata = parse_filename(fn.name)
  File "/data/homes/fractal/sharedfractal/lib/python3.8/site-packages/fractal_tasks_core/lib_parse_filename_metadata.py", line 70, in parse_filename
    raise Exception(
Exception: Failure in metadata parsing of AssayPlate_Greiner_#655090, with img_date={img_date} and img_time={img_time}

It appears the filenames run into an issue:
This is an example filename, it's quiet typical of the Yokogawa, but we didn't have it in the initial test cases:
AssayPlate_Greiner_#655090_B02_T0001F004L01A01Z01C01.tif
I would assume the Zarr file would be called AssayPlate_Greiner_#655090, which isn't actually a relevant name for the Zarr file. But the way the current parsing works, it assumes this is a barcode reading and then fails.

This bug is quite important, because many people may come with data formatted this way and then our parsing fails!
The high level discussion we should have for this:
How should we handle plate naming in general? Should we
a) Try to be fancy and create a good plate name from the filename (current), with the risk of failing for some cases we haven't seen (=> then catch those somehow)
b) Just do much simpler parsing, e.g. always just take the string before the _well (e.g. anything before _B02 here). @gusqgm the time parsing idea came from you in the beginning, right? Is it critical?
c) Allow users to name the OME-Zarr file (via an optional input? AssayPlate_Greiner_#655090 isn't a meaningful name for the Zarr file after all and will actually occur in many settings when users don't change defaults at the microscope).
My intuition is:
Go with b) for a quick-fix, optionally implement either a & c or one of them.

(also, the error message formatting doesn't work anymore, because line 72 is missing the f before the string)

I've looked at the filenames we support more closely here:
    Three kinds of filenames are supported:
    1) Filenames from UZH:
       20200812-Cardio[...]Cycle1_B03_T0001F036L01A01Z18C01.png
       with plate name 20200812-Cardio[...]Cycle1
    2) Filenames from FMI, with successful barcode reading:
       210305NAR005AAN_210416_164828_B11_T0001F006L01A04Z14C01.tif
       with plate name 210305NAR005AAN
    3) Filenames from FMI, with failed barcode reading:
       yymmdd_hhmmss_210416_164828_B11_T0001F006L01A04Z14C01.tif
       with plate name RS{yymmddhhmmss}

While I didn't spot this on initial implementation (sorry @mfranzon ), these assumptions are clearly to narrow, as this test-case shows. I don't think we can make assumptions on the filename structure based on the number of _ before the well. We could check whether a filename belongs to some known type and then process that in some fancy way.
But thinking bigger picture: What do we really need this initial processed part for? Is it only to name the OME-Zarr?
In that case, I think we should not do any fancy processing and just name the OME-Zarr by the project name or something like that. The filename could sometimes be useful, but not always. And we anyway already make the user specify a project name.
Alternatively, we could pick the output dataset name as the relevant string of how our OME-Zarr file is named.
Let's briefly discuss this on Wednesday

This is unfortunate indeed!
The point in b) did not really come from me per se, but rather from an error handling of yokogawa once the barcode scanner does not work. If the barcode is not working properly the time_stamp is written instead.
It is an annoying feature but does happen from time-to-time during image acquisitions.
If strictly necessary, we could still go for a quick fix for the time being, but keeping in mind that a more stable solution that can still encompass this possibility of the barcode scanning failure output will eventually need to be found, at least until we have Fractal running in the lab more generally.

Hmm, ok, good to know. Let's discuss then what should happen with these timestamps and this Yokogawa formats. Currently, we parse img_date & img_time, but the function doesn't do anything with them. It just removes them from the OME-Zarr name that is then assigned.
So if there's a broader use-case for them, we don't currently cover this and should think about what should actually happen with this information

For the record, the list of supported filenames was discussed in https://github.com/fractal-analytics-platform/fractal/issues/48 (see also test_unit_parse_metadata_from_filename.py). Looking at it after a few months, it doesn't look so robust - agreed.
@jluethi @gusqgm if you let us know a list of intended uses (i.e. filename and expected plate name), we will quickly update it. And while we are at it we can probably also streamline the part of metadata extraction (instead of using all those regex).

(also, the error message formatting doesn't work anymore, because line 72 is missing the f before the string)

Fixed.

Yokogawa filenames can basically be:
ANYTHING_B02_T0001F004L01A01Z01C01.tif
In some cases, the Z position can have more than 2 digits, I have never seen any of the other positions have extra digits (i.e. we can have ANYTHING_B02_T0001F004L01A01Z125C01.tif). ANYTHING can be any possible string that is valid as part of a filename.
The part starting from B02 has important information we need to extract. The ANYTHING part is often just flavor, can either be a default from the microscope, a name the user entered or a readout (successful or failed) from a plate reader.
There's a discussion to be had whether we need to parse the ANYTHING part more specifically and what we'd do with this information. So far, we used ANYTHING (or a subset of it in the case of the FMI optimized parsing) as the the name for the OME-Zarr file. As far as I'm aware, it's not really used for more (the plate_prefix, which is the first string before the first _ is used in some of the parsing as well though).
Thus, we can discuss whether extra info needs to be extracted from the ANYTHING part and what would need to be done with it (I'll discuss this further with Gustavo). As far as the general parsing is concerned, the OME-Zarr file needs to have some name, but it doesn't necessarily need to be part of the filename string. Easiest "quick" fix would be to just have the OME-Zarr name be the ANYTHING string. And then we can have a debate on whether it should be the dataset name, project name, additional string, specially parsed version of ANYTHING or something different.

OK, then in the meanwhile we can prepare a new function that just splits ANYTHING_B02_T0001F004L01A01Z01C01 into ANYTHING + the set of standard metadata attributes. And then we can decide what to do with ANYTHING.

OK, then in the meanwhile we can prepare a new function that just splits ANYTHING_B02_T0001F004L01A01Z01C01 into ANYTHING + the set of standard metadata attributes. And then we can decide what to do with ANYTHING.

Sounds good, thanks! :)

That's great! With that, we can close this issue. And I'll open a discussion issue about whether we need to do something with potential infos we parse from the filename + how we want to name OME-Zarr files in general. But neither of those questions are urgent.


[VMR] Add support for join point vertical job in vmr-build.yml

Contributes to https://github.com/dotnet/source-build/issues/3739
Depends on https://github.com/dotnet/source-build/issues/4215
A join point vertical job isn't much different to a normal vertical vmr-build job aside from:
Two new parameters

reuseBuildArtifactsFrom: This parameter lists the jobs that the join point job depends on. This parameter should support multiple values and the order is important. Artifacts from the job name listed first are preferred in case of duplicates with other jobs listed here.
buildPhase: This parameter identifies the build phase of the join point job. This should be a value > 1.

The existing job steps need to be updated to also work in a non-source-only build mode: https://github.com/dotnet/installer/blob/986cd8e87e84681dfa2dc11efa85ce2a2b15176e/eng/pipelines/templates/jobs/vmr-build.yml#L168-L210
The artifacts from the previously built verticals should be copied into the VMR's root artifacts folder. If a file already exists then it should not overwritten. This enables the priority system so that i.e. win-x64 artifacts are preferred over win-x86.

I'm assuming use of "Phase" here is a typo, the other issues (including the design doc) seem to have settled on "Pass"

reuseBuildArtifactsFrom is already a (single string) parameter, I assume the intent here is to replace that with an array so things scale, and fix up all existing usages to work with the modified syntax


DateFormatter issue

Hi, thanks for the useful library.
I think I found some bug in the code NmeaParser.swift. The dateFormat should represent hours by HH (not hh).
With hh, it was not using the GPS timestamp but it kept using current time which is not from GPS data.
So, it should be changed like below. I just changed hh to HH and it works fine now.
if rawDate.isEmpty {
    dateFormatter.dateFormat = "HHmmss.SSS" // 025816.16
} else {
    dateFormatter.dateFormat = "ddMMyyHHmmss.SSS"
}

Hi, thanks for the hint! Will check and change 👍


Pre-v3 release bugfixes

The fixes include:

Better documentation for the blocking functionality.
Unification of celebrate()/wiggle() between V1 and V2
Dance movement added for Marty V1
stand_straight() command added for both V1 and V2
Typos

TODO:

[ ] Update changelog

This all looks good, agree with all the comments and I like the changes to the documentation


[embedded] Add executable tests for array builtins

The array builtins (copyArray, takeArrayBackToFront, destroyArray, etc.) have custom IRGen in embedded Swift that avoids runtime calls. Let's cover these with an executable test (code taken from existing test/stdlib/Builtins.swift executable test).

@swift-ci please test


Consumer Group support for Kafka 0.8.x?

How does this library support Consumer Groups for legacy versions of Kafka older than 0.9.
Before the introduction of consumer offset tracking in 0.9, consumer groups used ZooKeeper for offset tracking. Is this supported in the library.

This library does not natively work with Zookeeper. You'll have to manage the offsets yourself.
I don't think any Go-based Kafka library natively talks to ZK and handles groups that don't use CG APIs.


3.0 packaging - RPM

This adds in support for generating the RPM package. DEB to follow.

PR 1000 :champagne:

Merging as this is just bringing in new files.


polymer test: module is not define

When I test the modulizered elements I get this error:

And I don't use lodash... I supose is something from the modulizer and/or the WCT.

I run into the same problem today after running the modulizer on mu components

Closing and old issue.


Invert conditional expression assist

Fixes https://github.com/dart-lang/sdk/issues/45853

[x] I’ve reviewed the contributor guide and applied the relevant portions to this PR.

Contribution guidelines:

See our contributor guide for general expectations for PRs.
Larger or significant changes should be discussed in an issue before creating a PR.
Contributions to our repos should follow the Dart style guide and use dart format.

Note that this repository uses Gerrit for code reviews. Your pull request will be automatically converted into a Gerrit CL and a link to the CL written into this PR. The review will happen on Gerrit but you can also push additional commits to this PR to update the code review.

Requested @bwilkerson to review on Gerrit, as he wrote the assist for inverting if.


Versioning for all protocols & components

Due to a raising set of dependencies, we should discuss a common versioning for IDS components (both technical components as well as IDS entities). This is there for the info model, but lacking for e.g., DATs and functional entities such as the clearing house.
Maybe better suited to be part of the RAM, this is still a good place to discuss that.
There is also the point of a compatibility matrix. Which versions are supported? Will v4 info model connector work together with a v2 info model? When does a component have to be upgraded?

Will be closed with Organizational Handbook V2 section releases


More forgiving wasi:http version detection

We detect a couple of supported RCs of wasi:http, plus releases 0.2.0 and 0.2.1.  However, the wasi crate (https://docs.rs/wasi/latest/wasi/) has already moved on to 0.2.2, and will presumably move on to 0.2.3 etc. in other minor versions.  This means users of the wasi crate in Spin apps have to pin to an exact patch version instead of the 0.13 minor version (see https://github.com/fermyon/leptos-spin/pull/48#discussion_r1847541217).
What would be the practicalities of enabling all 0.2 versions of wasi:http, present and future?
cc @alexcrichton

Closed via #2935.


Can't create shops

Whenever you try to create a shop, it just gives an error message and a stacktrace in the console.
[19:14:34] [Server thread/ERROR] [Sponge]: Error occurred while executing command 'buyshop 5 32' for source EntityPlayerMP['Mayor_Mike'/232, l='world', x=-642.79, y=66.00, z=141.12]: org.spongepowered.api.util.blockray.BlockRay$BlockRayBuilder.blockLimit(I)Lorg/spongepowered/api/util/blockray/BlockRay$BlockRayBuilder; java.lang.NoSuchMethodError: org.spongepowered.api.util.blockray.BlockRay$BlockRayBuilder.blockLimit(I)Lorg/spongepowered/api/util/blockray/BlockRay$BlockRayBuilder; 	at me.nentify.servershops.commands.ShopCommand.execute(ShopCommand.java:44) ~[ShopCommand.class:?] 	at org.spongepowered.api.command.spec.CommandSpec.process(CommandSpec.java:351) ~[CommandSpec.class:1.11-6.0.0-BETA-0] 	at org.spongepowered.api.command.dispatcher.SimpleDispatcher.process(SimpleDispatcher.java:333) ~[SimpleDispatcher.class:1.11-6.0.0-BETA-0] 	at org.spongepowered.common.command.SpongeCommandManager.process(SpongeCommandManager.java:291) [SpongeCommandManager.class:1.11-6.0.0-BETA-0] 	at net.minecraft.command.ServerCommandManager.func_71556_a(SourceFile:82) [bd.class:?] 	at net.minecraft.network.NetHandlerPlayServer.func_147361_d(SourceFile:811) [mi.class:?] 	at net.minecraft.network.NetHandlerPlayServer.func_147354_a(SourceFile:798) [mi.class:?] 	at net.minecraft.network.play.client.CPacketChatMessage.func_148833_a(SourceFile:37) [ip.class:?] 	at net.minecraft.network.play.client.CPacketChatMessage.func_148833_a(SourceFile:9) [ip.class:?] 	at org.spongepowered.common.network.PacketUtil.onProcessPacket(PacketUtil.java:125) [PacketUtil.class:1.11-6.0.0-BETA-0] 	at net.minecraft.network.PacketThreadUtil$1.redirect$onProcessPacket$zim000(SourceFile:39) [fo$1.class:?] 	at net.minecraft.network.PacketThreadUtil$1.run(SourceFile:13) [fo$1.class:?] 	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [?:1.8.0_101] 	at java.util.concurrent.FutureTask.run(Unknown Source) [?:1.8.0_101] 	at net.minecraft.util.Util.func_181617_a(SourceFile:46) [h.class:?] 	at net.minecraft.server.MinecraftServer.func_71190_q(SourceFile:142) [MinecraftServer.class:?] 	at net.minecraft.server.dedicated.DedicatedServer.func_71190_q(SourceFile:335) [lh.class:?] 	at net.minecraft.server.MinecraftServer.func_71217_p(SourceFile:562) [MinecraftServer.class:?] 	at net.minecraft.server.MinecraftServer.run(SourceFile:466) [MinecraftServer.class:?] 	at java.lang.Thread.run(Unknown Source) [?:1.8.0_101]

I am having the same issue. 23 days without a fix is disheartening.

Hi, as I've posted elsewhere, I have tried to update the plugin but I just don't have the time or interest anymore, sorry. The plugin is open source if anyone does want to fix the plugin themselves.

Same here


Setting BasePath properties in csproj breaks dotnet cli commands

I personally dislike having the bin and obj folder in the project directory, especially in solutions with multiple projects in them. Because of that, I have always since the beginning of time modified the BaseOutputPath and BaseIntermediateOutputPath properties in my csproj files like this:
<Project>
  <PropertyGroup Label="BasePath">
    <BaseOutputPath>$(SolutionDir)bin\</BaseOutputPath>
    <BaseIntermediateOutputPath>$(SolutionDir)obj\$(MSBuildProjectName)\</BaseIntermediateOutputPath>
  </PropertyGroup>
</Project>

When running just MSBuild MySolution.sln this works fine, and if I just build a single project in my solution, well either I set the SolutionDir property manually, or simply face the fact that SolutionDir is empty. All this works fine when running MSBuild from the developer command line and runs fine from building in Visual Studio.
However, neither the dotnet restore, dotnet build or even dotnet msbuild commands work with this setup. The restore command still creates the obj files in the project directory (instead of the solution directory) and the build command fails, because dependencies are not resolved correctly.
Dotnet Environment:
> dotnet --info
.NET Command Line Tools (1.0.1)

Product Information:
 Version:            1.0.1
 Commit SHA-1 hash:  005db40cd1

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.14393
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\1.0.1

This issue was moved to dotnet/sdk#1018


Add Schwab bank mapping, LEDGERBAL tag support, MS Money compatible output

This PR adds the ability to convert CSV files downloaded from Charles Schwab checking accounts to OFX files supported by MS Money.
For convenience, I broke the changes down into the following logically-distinct commits:

Ignore vim temp files
Add option for MS Money compatible output

Deal with MS Money idiosyncrasies RE: OFX formatting
MS Money specific OFX header
Provide dummy value for TRNUID
Don't include blank CHECKNUM
Limit payee length to 32 chars
Use noon for dates lacking a time (Recommended by MS Money troubleshooting OFX guide)

Include LEDGERBAL in non-split OFX output if possible

These are required fields per OFX spec & their absence breaks MS Money import.

Add mapping for Charles Schwab Bank
Add tests for ending balance analysis
Update example --help output

Confirmed no new errors from manage lint.  Confirmed all manage test tests pass.
Tested with >18 months of real-world data and manually verified transaction sanity in MS Money.
(Please let me know if anything doesn't meet coding standards, etc.  Unfortunately, I'm not a native Python speaker.  😃)

I don't have Charles Schwab, but my CC stopped creating OFX files and now just provides CSV files. So I locally patched with 69140f3, and the resulting OFX file does seem to generally match what I historically had for my CC account OFX files before, especially the header. The biggest difference is that the new csv2ofx output closes all tags, while the CC original leaves many tags open. However, I'm not able to import the generated OFX into MS Money because it says it contains invalid or corrupt data. I'm not sure if it's a problem with csv2ofx or my mapping file.
This is the original CSV from my CC:
Trans. Date,Post Date,Description,Amount,Category
12/18/2021,12/18/2021,"ATMOS ENERGY<PHONE_NUMBER>",151.75,"Services"
12/11/2021,12/13/2021,"KOHL'S",-66.22,"Payments and Credits"

Trans. Date
Post Date
Description
Amount
Category

12/18/2021
12/18/2021
ATMOS ENERGY<PHONE_NUMBER>
151.75
Services

12/11/2021
12/13/2021
KOHL'S
-66.22
Payments and Credits

I luckily still had the original OFX from the CC, and comparing it to the generated OFX, the transaction type seems backward. All purchases are listed as CREDIT with a positive value, while payments are DEBIT with a negative value. This is backwards from the original.
Original OFX
<STMTTRN>
<TRNTYPE>DEBIT
<DTPOSTED>20211218170000.000
<TRNAMT>-151.75
<FITID>FITID20211219-151.75Q65G3
<NAME>ATMOS ENERGY<PHONE_NUMBER> TX
</STMTTRN>

csv2ofx OFX
<STMTTRN>
<TRNTYPE>CREDIT</TRNTYPE>
<DTPOSTED>20211218120000</DTPOSTED>
<TRNAMT>151.75</TRNAMT>
<FITID>5ff442c1347782c556acb7fc52c477cc</FITID>
<NAME>ATMOS ENERGY<PHONE_NUMBER> TXATMO</NAME>
</STMTTRN>

The wrapping tags for the whole OFX are also different. The original uses <CREDITCARDMSGSRSV1><CCSTMTTRNRS>, while the new one uses <BANKMSGSRSV1><STMTTRNRS>.
So I'm not sure if the import failure is specific to the MS money patch from here or general to csv2ofx. Will happily create a new issue if it's general so as to not hijack this PR.

@nieder You'll also want to grab commit 893e6a8531ed1f2aa2f57bddb174eda08ccb08c5 if you want to produce an OFX file that MS Money is happy with.  Also, since it looks like your CSV doesn't have a column with balances in it, you'll need to specify an ending balance at the command line (with the -B or --ending-balance option) when running csv2ofx.  Without an ending balance in the OFX file, MS Money rejects the file (or at least, it does for bank statements).
This is a tool that I used to check the OFX files for errors while attempting to produce something that will satisfy MS Money: http://moneymvps.org/faq/article/366.aspx.  Unfortunately, the tool doesn't appear to check everything, but it was a good start.  I think the ledger-balance requirement wasn't actually checked by the tool, IIRC.
As for the incorrect transaction types, it looks like by default, csv2ofx treats negative values as debits and positive values as credits.  If I'm interpreting things correctly, you should be able to correct both the <TRNTYPE> and the sign in <TRNAMT> by adding a "type" line to your mapping.  For example:
    'type': lambda tr: 'debit' if tr.get('Amount') >= 0 '' else 'credit',

It looks like the <BANKMSGSRSV1> tags may be hard-coded.  If MS Money refuses to accept the OFX as a credit card statement with the <BANKMSGSRSV1> tags, you may need to open a new issue or PR for changing that.

Thanks. The extra commit for the balances does help in passing validation.
I was able to previously fix the debit/credit issue with this line in the mapping:
    "amount": lambda r: convert_amount(r["Amount"]) * -1,

Now, purchases show up as DEBIT, with a negative value in the OFX. This is clearly a problem with the CSV from Discover that uses  logic backwards from what Money expects, but so be it (as long as they stay consistent).
For the  BANKMSGSRSV1 tags, etc, if I keep them as is, the OFX analyzer complains. Money takes the file but asks me to identify the account during import (which it didn't with the original files before they switched to CSV). But if I do the following changes on the OFX:
1) Change:
STMTTRNRS -> CCSTMTTRNRS
BANKMSGSRSV1 -> CREDITCARDMSGSRSV1
BANKACCTFROM -> CCACCTFROM
STMTRS -> CCSTMTRS

then Money imports fine.
The OFX analyzer also complains about these lines
2) BANKID in CCACCTFROM
3) ACCTTYPE in CCACCTFROM

But I can leave them in and Money seems happy.
Lastly, csv2ofx converted literal ampersands (&) to &amp (missing ending semicolon), and Money complained about that. If I edit them to & everything is fine. Yay! Now to confirm the import imported everything correctly...

Just getting a chance to look this over. Can you sep it into two PRs? I'd like to avoid having library changes and new mappings in the same PR.

@reubano Thanks for considering this PR.  It looks like when split, there will still be a dependency between the two PRs.  Namely, the tests for the ending-balance code (commit e0143e9695c2bd880ade20a24973364a7d09dbe9) use the new Schwab Bank mapping.  (There is also a test in the new-mapping commit (8a4e7b40a8df79cbaafd51cce52a5d74de2f885d) that uses the new MS-Money mode, but that test could easily be split-out / moved to the new-features PR.)
Should I just create the new-mapping PR first (minus the one test that uses MS-Money mode), wait for that to merge, then create another for the other features?  Or is there a specific way you'd like me to structure the two PRs so that they could both be created at once?

@tstabrawa if you don't need any new lib features for the mapping then I suggest a PR just for the mapping. So what you said in your second to last sentence. That will be very easy to review and approve. Then new PR(s) with the features so we can discuss the problems and proposed solutions. Thanks!

Replaced by #114 and #115.  Closing this PR in favor of those.


Joint axes are incorrect.

Hi! Thank you for this Addin!
It saves a lot of time, as before we had to export each mesh file in a separate format(via fusion2urdf) in-order for Pybullet to accept the mesh file format. This takes care of it instantaneously.
Although I ran into a joint axes mismatch (below). I think that you can use the mesh exporter that you have here, and fusion2urdf's original code for joint axes and urdf creation and it would be perfect
I have also provided the video of how it should actually look like (which was made with fusion2urdf)

fusion2urdf's URDF exporter:
V14_URDF_f2u.zip
Fusion2Pyullet URDF exporter:
V14_URDF_f2p.zip

Output: (Please excuse the music. This is my first time using screenrecord application )
Fusion to Pybullet Video:
https://user-images.githubusercontent.com/37873142/107810131-33ffff80-6d92-11eb-9963-c85f4fce3e16.mp4
Fusion to URDF Video:
https://user-images.githubusercontent.com/37873142/107812529-d372c180-6d95-11eb-8b6b-d6acb15d1c80.mp4
My modified hello_bullet.py code, video output:
#!/usr/bin/env python3
import pybullet as p
import time
import pybullet_data
physicsClient = p.connect(p.GUI)#or p.DIRECT for non-graphical version
p.setAdditionalSearchPath(pybullet_data.getDataPath()) #optionally
p.setGravity(0,0,0)
# planeId = p.loadURDF("plane.urdf")
cubeStartPos = [0,0,0]
cubeStartOrientation = p.getQuaternionFromEuler([0,0,0])
robotId = p.loadURDF("V14_URDF.urdf",cubeStartPos, cubeStartOrientation,
                   # useMaximalCoordinates=1, ## New feature in Pybullet
                   useFixedBase=True,
                   flags=p.URDF_USE_INERTIA_FROM_FILE)
val = 0

for i in range (10000):
    p.stepSimulation()
    time.sleep(1./240.)
    # for j in range(p.getNumJoints(robotId)):
    	# p.setJointMotorControl2(robotId,j,controlMode=p.POSITION_CONTROL,targetPosition=val)
    	# val = val +0.001
    p.setJointMotorControl2(robotId,2,controlMode=p.POSITION_CONTROL,targetPosition=val)
    val = val +0.001
    p.setJointMotorControl2(robotId,3,controlMode=p.POSITION_CONTROL,targetPosition=val)
    val = val +0.001

cubePos, cubeOrn = p.getBasePositionAndOrientation(robotId)
print(cubePos,cubeOrn)
p.disconnect()

Warm Regards,
Rohit Kumar J

Please ping me for any further clarifications. :smile:

Thank you for pointing this out.
I am currently busying with some other staff, and I had some problem running PyBullet with my WSL, so fix might not be able to be applied very soon.
Welcome to send a pull request if you successfully fix this.

I feel like It looks like there is an update of xyz calculation in Fusion 360 and caused this problem. Here is some quick fix I would recommend you to try.
I checked syuntoku14's repo, and he had an update on core/Joint.py in 2020/01/19, core/Joint.py
It might work (I had to set up my PyBullet again to test it. But.... I might not have time in these few weeks..)

I have applied the xyz calculation fix from syuntoku14/fusion2urdf.
Actually I have no wrong working demo to reproduce the problem you faced with, so I'm simply pull the changes from the original repo.

I will try this out again on my model and get back.
Warm Regards,
Rohit Kumar J


Unknown:  RuntimeError: Cannot make context: this context is already current on another thread.

Hi.
Thanks for releasing your code. I'm trying to run the code for cartpole_swingup with the default parameters and the command from README. However, there is an error:
Unknown: RuntimeError: Cannot make context <dm_control._render.pyopengl.egl_renderer.EGLContext object at 0x7f72e010e970> current on thread <_DummyThread(Dummy-10, started daemon<PHONE_NUMBER>70400)>: this context is already current on another thread <_DummyThread(Dummy-9, started daemon<PHONE_NUMBER>77696)>.
Traceback (most recent call last):

  File "/opt/miniconda/envs/dreamer/lib/python3.8/site-packages/tensorflow/python/ops/script_ops.py", line 244, in __call__
    ret = func(*args)

  File "/opt/miniconda/envs/dreamer/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py", line 302, in wrapper
    return func(*args, **kwargs)

  File "/opt/miniconda/envs/dreamer/lib/python3.8/site-packages/tf_agents/environments/tf_py_environment.py", line 318, in _isolated_step_py
    return self._execute(_step_py, *flattened_actions)

  File "/opt/miniconda/envs/dreamer/lib/python3.8/site-packages/tf_agents/environments/tf_py_environment.py", line 214, in _execute
    return fn(*args, **kwargs)

  File "/opt/miniconda/envs/dreamer/lib/python3.8/site-packages/tf_agents/environments/tf_py_environment.py", line 314, in _step_py
    self._time_step = self._env.step(packed)

  File "/opt/miniconda/envs/dreamer/lib/python3.8/site-packages/tf_agents/environments/py_environment.py", line 203, in step
    self._current_time_step = self._step(action)

  File "/opt/miniconda/envs/dreamer/lib/python3.8/site-packages/tf_agents/environments/batched_py_environment.py", line 166, in _step
    time_steps = self._envs[0].step(actions)

  File "/opt/miniconda/envs/dreamer/lib/python3.8/site-packages/tf_agents/environments/py_environment.py", line 203, in step
    self._current_time_step = self._step(action)

  File "/opt/miniconda/envs/dreamer/lib/python3.8/site-packages/tf_agents/environments/wrappers.py", line 117, in _step
    time_step = self._env.step(action)

  File "/opt/miniconda/envs/dreamer/lib/python3.8/site-packages/tf_agents/environments/py_environment.py", line 203, in step
    self._current_time_step = self._step(action)

  File "/home/hue/Documents/anhdd14/pisac/pisac/dm_control_env.py", line 194, in _step
    time_step = self._env.step(action)

  File "/opt/miniconda/envs/dreamer/lib/python3.8/site-packages/tf_agents/environments/py_environment.py", line 203, in step
    self._current_time_step = self._step(action)

  File "/opt/miniconda/envs/dreamer/lib/python3.8/site-packages/tf_agents/environments/wrappers.py", line 213, in _step
    time_step = self._env.step(action)

  File "/opt/miniconda/envs/dreamer/lib/python3.8/site-packages/tf_agents/environments/py_environment.py", line 203, in step
    self._current_time_step = self._step(action)

  File "/opt/miniconda/envs/dreamer/lib/python3.8/site-packages/tf_agents/environments/dm_control_wrapper.py", line 91, in _step
    return convert_time_step(self._env.step(action))

  File "/opt/miniconda/envs/dreamer/lib/python3.8/site-packages/dm_control/suite/wrappers/pixels.py", line 98, in step
    return self._add_pixel_observation(time_step)

  File "/opt/miniconda/envs/dreamer/lib/python3.8/site-packages/dm_control/suite/wrappers/pixels.py", line 115, in _add_pixel_observation
    pixels = self._env.physics.render(**self._render_kwargs)

  File "/opt/miniconda/envs/dreamer/lib/python3.8/site-packages/dm_control/mujoco/engine.py", line 213, in render
    camera = Camera(

  File "/opt/miniconda/envs/dreamer/lib/python3.8/site-packages/dm_control/mujoco/engine.py", line 664, in __init__
    with self._physics.contexts.gl.make_current() as ctx:

  File "/opt/miniconda/envs/dreamer/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)

  File "/opt/miniconda/envs/dreamer/lib/python3.8/site-packages/dm_control/_render/base.py", line 116, in make_current
    raise RuntimeError(

RuntimeError: Cannot make context <dm_control._render.pyopengl.egl_renderer.EGLContext object at 0x7f72e010e970> current on thread <_DummyThread(Dummy-10, started daemon<PHONE_NUMBER>70400)>: this context is already current on another thread <_DummyThread(Dummy-9, started daemon<PHONE_NUMBER>77696)>.

Do you know why this is happening? I'm using Ubuntu 18.04.4 LTS and Python 3.8.3 with Conda. I have tried to change the isolation parameter of TFPyEnvironment call to True to make the env run on a thread, and the training process can run. However when it comes to the eval env, the error appear again.

This is a known issue with dm control. You can follow the Requirements section of README and set --gin_bindings=train_pisac.train_eval.drivers_in_graph=False. Please reopen if it doesn't work.


CO-1845_REST_API_username_is_a_two_word_prefix

Transform CO name in a lower case letter word with no spaces

Looks good - I approve this request.


Finally when i added the ressource plugin to the buildconfig i avoided this error
but now i'm facing with a permgen error when i run the command =>   rest-api-doc
Can someone tell me please  if grails 2.4.3 is comptaible with the 0.6.1 version of rest api doc

RestApiDoc works with Grails 2.4.3.
It seems to be an issue with one dependencie.
Do you already have this issue?

Finally i changed the configuration. Now i'm using rest-api-doc like a inline plugin instead of including it in the buildconfig.groovy of my project.
This issue can be closed!


lost pubsub connection

I use redisson to subscribe message in my project, I found the pubsub connection will lost and message will be not received any more. No any exceptions can be found in log, so I tured log level to DEBUG on org.redisson. Then I found following logs, and I not sure is it normal or not:
on project start
2022/03/15 15:37:07.368 [ty-2-14] DEBUG ClientConnectionsEntry [-|-|-] - new pubsub connection created: RedisPubSubConnection@1163063656 [redisClient=[addr=redis://myredis-servrer:6379], channel=
[id: 0xdbb90202, L:/<IP_ADDRESS>:47226 - R:myredis-servrer/<IP_ADDRESS>:6379], currentCommand=null]
2022/03/15 15:37:07.373 [ty-2-10] INFO  MasterConnectionPool [-|-|-] - 5 connections initialized for myredis-servrer/<IP_ADDRESS>:6379
2022/03/15 15:37:07.376 [ty-2-14] INFO  MasterPubSubConnectionPool [-|-|-] - 1 connections initialized for myredis-servrer/<IP_ADDRESS>:6379
almost 4 hours later
2022/03/15 19:36:55.840 [ty-2-32] DEBUG ClientConnectionsEntry [-|-|-] - new pubsub connection created: RedisPubSubConnection@1656211046 [redisClient=[addr=redis://myredis-servrer:6379], channel=[id: 0xf3a494a2, L:/<IP_ADDRESS>:37412 - R:myredis-servrer/<IP_ADDRESS>:6379], currentCommand=CommandData [promise=RedissonPromise [promise=ImmediateEventExecutor$ImmediatePromise@6c702440(incomplete)], command=(PING), params=[], codec=org.redisson.client.codec.StringCodec]]
and  13 seconds later
2022/03/15 19:37:09.041 [tty-2-8] DEBUG IdleConnectionWatcher [-|-|-] - Connection [id: 0xdbb90202, L:/<IP_ADDRESS>:47226 ! R:myredis-servrer/<IP_ADDRESS>:6379] has been closed due to idle timeout. Not used for 13167 ms
It seems the pubsub connection on port 47226 is replaced by an new pubsub connection on port 37412.
But CLIENT LIST with redis-cli show:
id=4824 addr=<IP_ADDRESS>:37412 laddr=<IP_ADDRESS>:6379 fd=19 name= age=12092 idle=22 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=ping user=default redir=-1
flags is N not P.
This is my redisson.yaml:
singleServerConfig:
idleConnectionTimeout: 10000
connectTimeout: 10000
timeout: 3000
retryAttempts: 3
retryInterval: 1500
password: null
clientName: null
address: "redis://myredis-server:6379"
subscriptionConnectionMinimumIdleSize: 1
subscriptionConnectionPoolSize: 50
subscriptionsPerConnection: 5
connectionMinimumIdleSize: 5
connectionPoolSize: 64
database: 0
dnsMonitoringInterval: 60000
pingConnectionInterval: 30000
keepAlive: true
threads: 16
nettyThreads: 32
codec: !<org.redisson.codec.FstCodec> {}
transportMode: "NIO"

You don't mention your version but 3.16.5 has this fix "Fixed - IdleConnectionWatcher shouldn't close RedisPubSubConnection if it's in subscribed state"

thanks, your comment is helpful! my version is 3.16.1.


Ability to swap all content of one workspace with another at once

Description
Sometimes  user may want to arrange workspaces with many windows in them, e.g. move the contents of one workspace to another, but doing this manually for each window with movetoworkspace dispatcher and arranging them afterwards is TOO TIRESOME.
Proposition
Add swapworkspacecontent or swapworkspacecontentsilent dispatchers, which take workspace param and swap content of active workspace with content of specified workspace.  swapworkspacecontent will change active workspace to specified one,  swapworkspacecontentsilent - not.
This will allow basically rearrange workspaces on same monitor without pain it some places.

Oops, I miss already closed issue https://github.com/hyprwm/Hyprland/issues/3554. But scripting, really?

or a plugin. This makes zero sense.


How to pass key/value pair username/passwords

How should i extract to username/password to something similar here?
Example:
locals {
db_creds            = jsondecode(data.aws_secretsmanager_secret_version.creds.secret_string)
}
CODE:
resource "random_password" "random_db_password" {
length           = 16
special          = true
override_special = "_!%^"
}
module "rds_aws_secret" {
source = "./modules/secrets"
secrets = {
"tf-${var.env_type}-dbsecrets" = {
description = "This is a key/value secret for RDS admin account, used for provisioning"
secret_key_value = {
username = "${var.db_username}"
password = "${random_password.random_db_password.result}"
}
}
}
}
Terraform state shows:
module.rds_aws_secret.aws_secretsmanager_secret_version.sm-sv["tf-dev-dbsecrets"]:
resource "aws_secretsmanager_secret_version" "sm-sv" {
arn            = "arn:aws:secretsmanager:us-east-1:******:secret:tf-dev-dbsecrets-S9tkKc"
id             = "tf-dev-dbsecrets|D4FF8F82-B9D2-430E-9FE6-A1F0C69F52F2"
secret_binary  = (sensitive value)
secret_id      = "tf-dev-dbsecrets"
secret_string  = (sensitive value)
version_id     = "D4FF8F82-B9D2-430E-9FE6-A1F0C69F52F2"
version_stages = [
"AWSCURRENT",
]
}

@usuryadevara please check the key-value example here.
I'm going to close the issue, but if you need further help you can comment here.


Managed Identity for App Service indicates Linux Consumption only supports older endpoint

The note in this document specifies that Linux Consumption plans require using the 2017-09-01 endpoint version. Based on information we've received from @Hamster-Huey, it looks like this is outdated information.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: f57d3177-ce9e-ebfa-03a7-45beb8ca3212
Version Independent ID: a59df23d-f3ac-9809-61d3-5741e31f020a
Content: Managed identities - Azure App Service
Content Source: articles/app-service/overview-managed-identity.md
Service: app-service
GitHub Login: @cephalin
Microsoft Alias: cephalin

Thanks for the feedback! I have assigned the issue to the content author to investigate further and update the document as appropriate.

@Hamster-Huey can you let us know what the updated information is?

Linux consumption supports the "latest" 2019-08-01 api version as described in the rest api spec here: https://docs.microsoft.com/en-us/azure/app-service/overview-managed-identity?tabs=portal%2Chttp#connect-to-azure-services-in-app-code

#ready-to-doc


Deleted Git workflow, moved to wiki

Now that we have a wiki, I thought it would be better to move some of the more basic information off CONTRIBUTING.md
I left the section with a little blurb telling people to go to the wiki if they need help. closes #23

@fedGL, for future merges, I think @Vic-ST wants to do the merges himself so because he needs to replicate it to master as well.


[Enhancement]: Add a typewriting animation in the landing page

Enhancement: Add a typewriting animation to the landing page to improve user engagement and visual appeal. This animation will create a dynamic introduction, enhancing the overall user experience.

I will love to element this in the code.

@anjalit03 assign it to me "Find New Experiences " text will be animated to increase visual appeal

and make the search boxes more better in UI


Fixed Symfony 4.1 deprecations.

Patch applied in the same format as https://github.com/stof/StofDoctrineExtensionsBundle/commit/ee653c8079ebb7dcf391ed5876357cca41f848b1

Status? My phpunit test returns deprecation notices about those.

Could you give a look to why the tests are red?

Looks like it's an issue with PHP spec failing with "unexpected method call" on the container builder. I'll try and take a look.

@dunglas All tests are passing now thanks to the test fixes from @firstred

Thanks @leevigraham

Great job everyone! Nice to see that the changes made it into the bundle. I'll close my redundant PRs.

@dunglas Whats need to create a new release?


Add License

Hi,
first of all thanks for creating this awesome project :)
Could you please also add a license to this repo so that I can use it in an official project?
Kind regards :)

Duplicate of #3


Include siblings syntactic sugar with AndInclude

Currently we have to write this:
db.Posts
    .Include(x => x.Author).ThenInclude(x => x.FavoriteBook)
    .Include(x => x.Author).ThenInclude(x => x.FavoriteMovie);

It would look better and reduce redundancy if we could do this:
db.Posts
    .Include(x => x.Author)
    .ThenInclude(x => x.FavoriteBook)
    .AndInclude(x => x.FavoriteMovie);

@isonil This is an interesting idea and similar to some suggestions that have been made in issue #4490. It's not clear that this can be implemented without breaking changes using the current types because the generics for the previous level in the hierarchy are not necessarily available. If you find a way to make this work we would be interested to consider it more. For now, closing as a duplicate of #4490


Add documentation for custom plugin

Codecov Report

Merging #356 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #356   +/-   ##
=======================================
  Coverage   31.64%   31.64%
=======================================
  Files          14       14
  Lines         986      986
=======================================
  Hits          312      312
  Misses        623      623
  Partials       51       51

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 29274da...5b17c6b. Read the comment docs.

@aerokite , the plugin doc is missing the most important part. How we convert the vars to json format. There is no mention of that.


Metadata API: Remove Signed.bump_version() method

Related to #1736 (see https://github.com/theupdateframework/python-tuf/pull/1736#pullrequestreview-837214238)
Description of the changes being introduced by the pull request:
Remove bump_version() method, which is just an alias for "+= 1" on the version attribute. For a slim low-level API it seems okay to just directly access/modify the attribute.
The extra level of abstraction of "bumping a version" is more appropriate for a repository library (see #1136).
This patch also removes a related unit test and updates another one to directly do (...).version += 1.
Please verify and check that the pull request fulfills the following
requirements:

[ ] The code follows the Code Style Guidelines
[ ] Tests have been added for the bug fix or new feature
[ ] Docs have been added for the bug fix or new feature

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-0.0008%) to 97.701%

Totals

Change from base Build<PHONE_NUMBER>:
-0.0008%

Covered Lines:
4093

Relevant Lines:
4173

💛  - Coveralls

Thanks for the reviews!


feat: add sepolia support

Closes: https://github.com/snapshot-labs/sx-ui/issues/558
Depends on: https://github.com/snapshot-labs/sx-ui/pull/649
It works, but there seems to be an issue with Graph's file data source where we can't load existing DB entries inside IPFS handlers and we end up with unrecognized executions on spaces.
Contacted Graph about it.

On space creation network tab we should have a hand cursor on the network block, otherwise it works well tACK!