Revised CMIP6 and CORDEX Jupyter Notebooks for Extreme Temperature Indices (C3S2_520 WP4)

Notebook description
In this notebooks, data from five models from CORDEX or CMIP6, as well as ERA5 reanalysis, are downloaded. Two maximum-temperature-based ECA&D indices (one of physical nature and the other of statistical nature) are calculated using the icclim Python package. The first index, labeled with the ETCCDI short name 'SU', represents the count of summer days (i.e., with daily maximum temperature above 25°C) occurring in a year or a season (JJA in the example presented here). The second index, labeled with the ETCCDI short name 'TX90p', describes the number of days with daily maximum temperatures exceeding the daily 90th percentile of maximum temperature for a 5-day moving window. This calculation is performed over a reference period from 1971 to 1995 in this example.
After calculating these indices for the reference period (resulting in index values per year), the anomalies, temporal means, and trends are calculated. Following this, the bias of the temporal mean values and the trend bias are calculated, using ERA5 as the reference forecast. These biases are displayed for each model and for the ensemble median.
Additionally, maps of the ensemble dispersion are calculated and displayed for the mean values and for the trends.
Notebook link or upload
drafts_CMIP6_CORDEX_tasmax.zip
Anything else we need to know?
The primary goal of these notebooks is to compute two daily-based maximum temperature indices from ECA&D. The first index, labeled 'SU' following the ETCCDI short name, signifies the count of summer days (daily maximum temperature above 25°C) occurring annually or within a specific season (e.g., JJA in the example presented here). The second index, labeled 'TX90p,' describes the number of days with daily maximum temperatures exceeding the daily historical period 90th percentile within a 5-day moving window.
In our analysis, we have assessed both MEAN VALUES and TRENDS over a reference historical period (1971-1995) to streamline calculations and downloading. The evaluation includes the BIAS OF the MEAN values and the TREND BIAS, utilizing ERA5 as the reference forecast. Additionally, we have displayed these biases for each model and the ensemble median, accompanied by the dispersion of the ensemble for both mean values and trends.
While we aimed for homogeneity between the two notebooks, there are some differences that we would like to highlight:
"draft_cordex_tasmax_extremes" NOTEBOOK
For the Mean:

Calculated using indices regridded to ERA5.
Values displayed for individual models use the ERA5 grid.

For the Trends:

Trend calculation is performed using indices regridded to ERA5.
Values displayed for individual models use the ERA5 grid.

"draft_CMIP6_tasmax_extremes" NOTEBOOK:
For the Mean:

Calculated using indices regridded to ERA5.
Values shown for individual models use the model's grid.

For the Trends:

Trends are calculated using the original grid of each model, and then the resulting trends are regridded.
Values shown for individual models use the model's grid.

THE RATIONALE BEHIND THESE DIFFERENCES IS AS FOLLOWS:
1.  Mean Calculation:

In both cases, mean values are calculated using indices regridded to ERA5.

2.  Trend Calculation:

For Cordex notebook: To prevent noisy outputs, the indices are regridded to ERA5, and then the trend is calculated.

For CMIP6 notebook: To prevent noisy outputs, trends are calculated using the model's grid, and then regridded to ERA5.

3.  Displaying Trends of CORDEX Individual Models:

Due to issues with rotated latitudes, we opted to display only regridded trends.

Another thing left to do is to check the agreement on the sign in the multimodel ensemble when the significance of the ensemble is calculated.
We hope this clarification provides insight into the specificities of each notebook. Please feel free to reach out if you have any questions or require further details.
The next steps from our side will be to derive future projections and the related uncertainty of the above-mentioned temperature extreme indices.
 We usually work step by step with the evaluators, but these notebooks are already quite long and complex.
Are you available for a call tomorrow morning? I'd like to go through the notebook together before I start the review.

Hi @malmans2
I agree that it may be necessary to go through the notebook together before you start reviewing it.
Tomorrow morning I am available from 11 on

Cool, I'll send you a google meet link via email.
One more thing, looks like your notebooks use external data. It's hard for us to work on those notebooks because we have no way to reproduce your results (btw, most of the data you need should be already available on the VM).
Could you please share notebooks that only use publicly available data? E.g., use download_and_transform or directly cdsapi, and make sure the notebooks can be executed on any machine. Also, please execute the notebooks using a small subsample of data (e.g., just a couple of years). That way, we can compare our results with yours while we optimize the code (this is what I meant here).

Hi @almarbo,
Could you please let me know which years you are going to use in the notebook you are about to run?

chi @malmans2
I am using from 1971 to 1975

Hi @malmans2
as agreed, I am attaching here the notebook of CMIP6 going from 1971 to 1975
draft_CMIP6_tasmax_extremes_1971-1975.ipynb.zip
best

@almarbo are you going to run the analysis on various regions and/or the whole global as well?
If yes, we will download the whole area, but we need to use smaller chunks because you are working with hourly data (e.g., monthly chunks). The original notebook was designed to work with yearly data.
If you are only going to need a single region, we can download cutouts instead (and we can probably use yearly chunks).

Hi @malmans2,
for this variable the final domain that we will need is the same as the euro-codex domain which is a little bit larger than the one specified in the notebook. The final needed domain is 22°W to 45°E longitude and 27°N to 72°N latitude.

Also, you said you are only going to need temperatur, right?
I.e., no need to parametrize the variable? Just trying to keep it as simple as possible.

exactly, for this notebooks just maximum temperature.

No, I made it in the prospective of adding future projections, but you can remove it and we will include it if needed in future steps

Hi @almarbo,
It looks like you had some job running that was consuming a lot of memory.
The VM is shared among all users, so make sure you don't run heavy jobs (e.g., use small samples when working to your code and ask us to run and cache heavy jobs).

Let me double check that I'm understanding everything correctly. These are the steps I'm implementing so far:

Resample to daily max (needed for ERA5 only)
Compute indexes for each model and ERA5
Compute anomalies for each model and ERA5 (the anomaly is defined as ds - ds.mean("time"))
Compute the trends of the anomalies

Correct?

That's correct @malmans2

OK, awesome. Next step is apply a bilinear interpolation to the model output.

Not exactly for the model output. We apply it twice:
(1) for the indices values (necessary to obtain temporal means and anomalies using regridded data)
(2) the trend is calculated using the indices (in this case not regridded, i.e. with the original grid of the model), and then the regridding is applied to the trend.
I understand it's a little bit messy and difficult to understand. Maybe it could be nice to do a quick meeting when you have time to talk about it.
thanks

Mmmm I didn't get the last part. We can have a quick chat right now if you are available or in the afternoon.
This is the implementation I have right now:
def select_timeseries(ds, timeseries, year_start, year_stop):
    if timeseries == "annual":
        return ds.sel(time=slice(str(year_start), str(year_stop)))
    return ds.sel(time=slice(f"{year_start-1}-12", f"{year_stop}-11"))

def compute_indexes(ds, index_names, timeseries):
    with tempfile.TemporaryDirectory(suffix=".zarr") as zarr_store:
        ds.to_zarr(zarr_store)
        datasets = [
            icclim.index(
                index_name=index_name,
                in_files=zarr_store,
                slice_mode="year" if timeseries == "annual" else timeseries,
            )
            for index_name in index_names
        ]
    return xr.merge(datasets).drop_dims("bounds")

def compute_trends(ds):
    with xr.set_options(keep_attrs=True):
        ds -= ds.mean("time")
    datasets = []
    coords_name = {"time": "time", "y": "latitude", "x": "longitude"}
    for index, da in ds.data_vars.items():
        ds = Mann_Kendall_test(
            da,
            alpha=0.05,
            method="theilslopes",
            coords_name=coords_name,
        ).compute()
        ds = ds.rename({k: v for k, v in coords_name.items() if k in ds.dims})
        ds = ds.assign_coords({dim: da[dim] for dim in ds.dims})
        datasets.append(ds.expand_dims(index=[index]))
    return xr.concat(datasets, "index")

def compute_index_trends(ds, index_names, timeseries, year_start, year_stop, resample, **interp_kwargs):
    ds = select_timeseries(ds, timeseries, year_start, year_stop)
    if resample:
        ds = ds.resample(time="1D").max(keep_attrs=True)
    ds = compute_indexes(ds, index_names, timeseries)
    ds = compute_trends(ds)
    if interp_kwargs:
        ds = diagnostics.regrid(ds, **interp_kwargs)
    return ds

thank you for the work, it looks way more efficient
I am available from 15.00 to 17.00

OK, let's go for 3pm. I'll send the link via email.

if both of them are in xclim I think it's okay to go with xclim. I don't know though if for other future notebooks that we have to do regarding climate extremes it will be necessary, but we'll see. For now it's okay!

I have a doubt about tx90p. Here is the documentation: https://xclim.readthedocs.io/en/stable/api_indicators.html#xclim.indicators.atmos.tx90p
For the seasonal output, tasmax_per should be the 90% considering all data, or only the relevant season?

Nevermind, I think I figured it out:
(tasmax, ) = ds.data_vars.values()
tasmax_per = tasmax.chunk(time=-1).groupby("time.dayofyear").quantile(0.9, keep_attrs=True)
tx90p = xclim.indicators.icclim.TX90p(tasmax, tasmax_per)

in theory it should be the number of days per time period (so the number of days  per JJA season) in which the daily maximum temperature is above the corresponding calendar day 90th percentile of maximum temperature for a 5-day moving window in the base period. I think that's why I take a while for computing it. The other index indeed does not take that much.
here you can find the definition:
https://cds.climate.copernicus.eu/cdsapp#!/dataset/sis-extreme-indices-cmip6?tab=overview

OK. I think this the snippet I sent you is correct.
Could you please check? I don't have icclim on the VM anymore.
If you do this, you should get similar (or even identical) results:
tasmax_per = (
    tasmax.chunk(time=-1).groupby("time.dayofyear").quantile(0.9, keep_attrs=True)
)
xclim_result = xclim.indicators.icclim.TX90p(tasmax, tasmax_per)

icclim_result = icclim.index(
    index_name="TX90p",
    in_files=tasmax,
    slice_mode="year" if timeseries == "annual" else timeseries,
)

I will check later. My only doubt here is if for the calculation of the daily percentile is used a 5-day moving window or not. A priori,  this could be important to get consistent daily percentiles

Oh right, I missed the 5-day moving window. I'll make a couple of tests, we might be able to use icclim, it looks like I found a good way to optimise it.
BTW, could you please confirm that you want to use years starting from January for the "annual" timeseries, and years starting from December for DJF? It doesn't mattar for other seasons.

OK, we can use icclim (not ideal as it adds a lot of dependencies and it pins various packages).
Almost done, I need to check a few things and test conservative interpolation.

Looks like conservative interpolations is working fine, I didn't even have to add bounds (probably because I keep all attrs in good shape and cf-xarray + xesmf do that automatically).
Is conservative interpolation the method you'd like to use?

I need to test it before deciding. For CMIP6 I neither had problems when implementing the conservative method and I did not have to include the bounds. However, when dealing with the CORDEX models, I was not able to  implement it (even though I added the bounds)

OK, I'll cache both of them

@almarbo,
I have a first draft ready.

Here is the template: https://github.com/bopen/c3s-eqc-toolbox-template/blob/main/notebooks/wp4/WIP-extreme_temperature_indices.ipynb
Here is the notebook executed: https://gist.github.com/malmans2/3309ea180f316a1b7a1a35669426efbe

If you execute the notebook as it is on the VM, everything is already cached. I also cached the same notebook using the conservative interpolation (e.g., if you only change the interpolation_method to "conservative", the results are cached)
I still have to tune a few things, are you available for a quick chat? The sooner the better, but today I'm only around until 4PM.

Nice, thank you!
I am available now for a quick chat.

Please don't run the notebook on the VM, I'm playing with the cache.
I'll let you know when the new version is ready.

@almarbo, what's the criteria to hatch non-significant values?
Also, do you want to only hatch single models or the ensemble as well?
If the latter, please let me know that criteria as well.

Hi @malmans2
We would like to hatch the NON-SIGNIFICANT VALUES for both the single models and the ensemble median. For the ensemble median, we consider significant values of the trend when more than 66% of the models are statistically significant and there's an agreement on their sign of the trend.
thank you

OK. I need the definition of non-significant. p-value > 0.05?

Exactly

Actually, this is not clear:

there's an agreement on their sign of the trend.

What do you mean? Do you consider significant grid points where 2/3 of the models have p<=0.05 and trend of the same sign as the ensamble mean?

OK, I'll take a look.
The notebook you attached is different than the notebook I reviewed and added to the templates.

@almarbo I think everything is implemented now.
Please take a look at the latest notebook, and let me know if we can cache the whole period.
Let me know which time period you need, and which interpolation method to cache first.
Here is the latest notebook: https://gist.github.com/malmans2/3309ea180f316a1b7a1a35669426efbe
BTW, masking is still missing. Yesterday I downloaded the wrong mask by mistake, and today I've been waiting forever in the queue.

Hi @malmans2
Sorry I completely missed the comment about the reference period. We would like that the reference period goes from 1971 to 2005.
Thank you so much for your work. I will check later the results and let you know. But as you said, I did some different things probably (I was wrongly showing the significant values, for instance). Let me have a view, check more in detail what I was doing differently and I will let you know so you can proceed for the other seasons and new period

OK, I will cache until 2005 in the meantime.
Unlikely it will be ready today, the CDS is very busy during weekdays.

Ok, no worries.
Thanks!

Hi @malmans2
I have checked the notebooks. There's a couple of details that I would like to comment with you. Do you have time tomorrow for a 10-minutes chat? I am available from 11 on.

Tomorrow is a busy day for me. I'll invite you to a meeting from 11 to 11.30am, but we might have to postpone if my previous meeting doesn't finish on time.

Perfect! Thanks. And no worries if we finally have to postpone it

Hi @almarbo,

Before plotting the boxplot the data should be spatially averaged.

Are spatial weighted mean OK, or do you want unweighted means?

Note: In our case we prefer representing the ensmble median rather than the ensemble mean

Do you mean that you don't want to show the mean? The median should be the green line in the boxplot, so it's already shown.
Other than the doubts above, i think we are all set. Here it is: https://gist.github.com/malmans2/3309ea180f316a1b7a1a35669426efbe
If everything is OK I will remove the WIP status and will cache all seasons.

Hi @almarbo,
All CMIP6 is cached, which means that if you run the notebook on the VM and you only change the timeseries parameter, the results are already cached.

Here is the template: https://github.com/bopen/c3s-eqc-toolbox-template/blob/main/notebooks/wp4/extreme_temperature_indices.ipynb
Here is JJA executed: https://gist.github.com/malmans2/3309ea180f316a1b7a1a35669426efbe

I'll take a look at CORDEX in the next few days, hopefully before the end of the week.

Hi @almarbo,
I made a couple of changes so we can use the same code for CMIP6 and CORDEX. I'm about to invalidate the cache.
Please don't run the notebook on the VM until I'm done.

Hi @malmans2
can we have a quick chat one of these days? We were said to include the future projections and other models in the final version of the notebook.

yes! tomorrow it's okay for me. I am free from 11 to 13 and from 15 to 17.30
the finals models could be:
models_cordex = [
    "clmcom_clm_cclm4_8_17",
    "clmcom_eth_cosmo_crclim",
    "cnrm_aladin63",
    "dmi_hirham5",
    "knmi_racmo22e",
    "mohc_hadrem3_ga7_05",
    "mpi_csc_remo2009",
    "smhi_rca4",
    "uhoh_wrf361h",
]

models_cmip6 = [
    "access_cm2",
    "awi_cm_1_1_mr",
    "canesm5",
    "cmcc_esm2",
    "cnrm_cm6_1_hr",
    "ec_earth3_cc",
    "gfdl_esm4",
    "miroc6"
    "mpi_esm1_2_lr",
]

Regarding the projections I will think a little bit and I let you know. It's more likely that we finally use two different notebooks (one for historical and the other for future projections

Hi @almarbo,
Sorry butt unfortunately I can't meet today. How about Monday next week. Would that work for you?

Perfect!

Hi @almarbo,
Here is JJA CORDEX: https://gist.github.com/malmans2/3309ea180f316a1b7a1a35669426efbe
Let me know if it looks OK.
In the meantime, I'll cache the other seasons (only JJA is cache on the VM right now)

Hi @malmans2
It looks nice, thank you.
Just a question, is CMIP6 still on the cache? Because I am trying to see it for CMIP6 (I only changed the collection_id) but I think it's downloading the data. If I am overloading or something, please let me know

Probably something is missing or not up to date.
If you can wait until tomorrow, please stop your notebook.
I was planning to run the scripts to check and cache everything tonight (both CORDEX and CMIP6)

CMIP6 is all cached. Please use the latest template from now on: https://github.com/bopen/c3s-eqc-toolbox-template/blob/main/notebooks/wp4/extreme_temperature_indices.ipynb
I'm still caching CORDEX, I'll be in touch.

Hi @malmans2
thank you so much

All CORDEX sesonal analysis are also available.
Please don't try to use annual yet. I've been struggling to keep the memory under control on the VM, I'll take a look later today. It might be just a hiccup as the VM is quite stressed at the moment.

Perfect, thank you.
I also wanted to let you know that we have already defined the way we would like to proceed with future projections notebooks. It would be really similar to the ones from the historical. Would you be up for a meeting the upcoming days? If not we can do next week. It will be really short (15 mins)

It's okay for me!

I also write down here the details regarding the future projections notebooks:

Models: same as previous notebooks
Period: 2015 to 2099
Experiment (scenario): exp="ssp5_8_5"
Regridding model: for CMIP6: "gfdl esm4". For Cordex: "clmcom_eth_cosmo_crclim"
The mean values over the period is not needed anymore, the only needed thing is the trend
Plotting part: similar to the notebooks provided in my last comment. For every index, two layouts. The first, a 2x1 layout showing the ensemble median and the ensemble standard deviation. The second, a  3x3 layout showing the trend of every model in their original grid.
No need of ERA5 (maybe only for the sea mask).
Boxplot: same as my two last notebooks. Boxplot showing the trend of the non-re gridded models

You can leave the plotting part to me if you prefer
Thank you again

Hi @malmans2
I have created the first draft version of the notebook for future projections. I'm leaving it here in case you want to take a look:
future_cmip6_extreme_temperatue_indices.ipynb.zip
I have not applied the sea mask yet (downloading it from ERA5 takes a while). Besides, I am experiencing some problems:

I am not being able to adjust the colorbar size for the layout that includes the ensemble+model dispersion
The model ec-earth shows some problems when calculating the trend for periods that go beyond 2080 (and I am not able to understand why)

Hi @malmans2
I upload here, as agreed in our recent meeting, the two notebooks already executed containing the final display and results for the historical period (1971-2005)
historical_extreme_temperatue_indices.zip
thank you

Hi @almarbo,
Can you please check the latest version of the notebook? https://gist.github.com/malmans2/3309ea180f316a1b7a1a35669426efbe
I tried to improve and simplify the plotting, hopefully we can reuse most of it in the future projections notebook.
You might want to edit a bit titles/captions, but other than that everything should be in good shape.
I'll work on future projection either later today or tomorrow.

Thank you so much.

I tried to improve and simplify the plotting, hopefully we can reuse most of it in the future projections notebook.
You might want to edit a bit titles/captions, but other than that everything should be in good shape.

it is clearer now, thank you. Would it be possible to homogenise styles between the plot models and plot bias sections plots? We would like to have a single color bar (as in the bias plots). We would also like, only if possible, (1) a shorter color bar, (2) in a plot layout, the distance between one raw of plots and the next one to be smaller, (3) a more centered title and (4) avoid overwriting in the longitude labels.
thank you again

All done, I think.
The figures are a bit smaller now, but I'm using the default. You can easily tune that with the figsize parameter in the plotting functions.
https://gist.github.com/malmans2/3309ea180f316a1b7a1a35669426efbe

That's perfect! Thank you so much. I will only change a little bit the titles and will add captions to the figures for delivering purposes. I will also add a boxplot for the trend bias (now there's only one for the trend). This last boxplot was specified in the feedback that was given to us but we forgot to include it.

OK, it's easy with the current setup. I can add the boxplot right away.
Give me 5 min.

Done: https://gist.github.com/malmans2/3309ea180f316a1b7a1a35669426efbe

really nice, thanks!
Let me know if you need something for the future period or if there's any problem

@almarbo,
There's a new template for future projections: https://github.com/bopen/c3s-eqc-toolbox-template/blob/main/notebooks/wp4/extreme_temperature_indices_future.ipynb
Note that I made the ensemble plotting functions more general in both templates.
I only tried CMIP6 so far, could you please check whether everything is in good shape?
You can find the executed notebooks here: https://gist.github.com/malmans2/3309ea180f316a1b7a1a35669426efbe

Hi @malmans2, thank you.
It looks nice; I noticed that you made the functions more general. By doing that, is there no need to have different functions for the future and historical?
I only have one doubt: I found that the domain shown in the maps for the future notebook is slightly smaller than for the historical case. Any clue on that? I also noticed that there is a band of 'NA's in the northern part of the map for the ensemble plots.

It looks nice; I noticed that you made the functions more general. By doing that, is there no need to have different functions for the future and historical?

Yes, the plotting functions are identical

I only have one doubt: I found that the domain shown in the maps for the future notebook is slightly smaller than for the historical case. Any clue on that? I also noticed that there is a band of 'NA's in the northern part of the map for the ensemble plots.

I guess they're both due to the fact that you are interpolating on a different grid, and the CMIP6 domain is likely a little smaller. It's up to you, we can either show the exact same domain of ERA5 (with more nans), or we can make the domain slightly smaller removing the NaNs line.
I think we just need to tune this line:
# Remove boundaries
isel = {coord: slice(5, -5) for coord in ("latitude", "longitude")}
ds_interpolated = ds_interpolated.isel(isel)

We can also easily interpolate to ERA5 if you change your mind.

 I will tune it a little bit for the delivering version then.

OK. I'm caching CORDEX now.
(If you are using the VM, I had to restart jupyter. You'll have to re-do the usual jupyter_lab procedure. Watch out that you'll likely get a different port)

Okey perfect!
No worries, I will proceed with the tests on Monday
Buon weekend

Hi @malmans2
I performed some tests with the set_extent function, and as you suggested, by adjusting the parameters slightly for the future, the NAs can be handled. Is the CORDEX future still being cached, right?
I also wanted to inform you that we are ready to start working on Use Case 2, where some precipitation indices (which can also be calculated using icclim) need to be computed. The structure of the notebook remains essentially the same, but the names of the indices to be used now are: index_names = ("rx1day", "rx5day", "rr1", "r20mm", "cwd"), and the variable names for the requests need to be adapted to precipitation. Should I open a new issue, or should we continue the discussion here since the structure remains the same?

Yes, please open a new issue.
I'm having troubles caching CORDEX, I get memory errors raise by the package you are using to compute indexes.
I tried various chunking, but I couldn't get it to work.
Have you been able to compute indices for the whole future projections of CORDEX on other machines?

The regridding model was downloaded and computed with no problems (slow, but fine).  I am having problems with the regrid, though. Did you have to include something new to the "Download and transform model" section?  :
grid_out = ds_regrid[["latitude", "longitude"]].reset_coords(drop=True)
grid_out.attrs = {}

interpolated_datasets = []
model_datasets = {}
for model in models:
    print(f"{model=}")
    # Original model
    model_datasets[model] = download.download_and_transform(
        requests=[request | {model_key: model} for request in request_sim[1]],
        **kwargs,
        transform_func_kwargs=transform_func_kwargs,
    )

    # Interpolated model
    ds = download.download_and_transform(
        requests=[request | {model_key: model} for request in request_sim[1]],
        **kwargs,
        transform_func_kwargs=transform_func_kwargs
        | {
            "grid_out": grid_out,
            "method": interpolation_method,
            "skipna": True,
        },
    )
    interpolated_datasets.append(ds.expand_dims(model=[model]))

ds_interpolated = xr.concat(interpolated_datasets, "model")

the error was the following:

KeyError                                  Traceback (most recent call last)
Cell In[8], line 16
      9     model_datasets[model] = download.download_and_transform(
     10         requests=[request | {model_key: model} for request in request_sim[1]],
     11         **kwargs,
     12         transform_func_kwargs=transform_func_kwargs,
     13     )
     15     # Interpolated model
---> 16     ds = download.download_and_transform(
     17         requests=[request | {model_key: model} for request in request_sim[1]],
     18         **kwargs,
     19         transform_func_kwargs=transform_func_kwargs
     20         | {
     21             "grid_out": grid_out,
     22             "method": interpolation_method,
     23             "skipna": True,
     24         },
     25     )
     26     interpolated_datasets.append(ds.expand_dims(model=[model]))
     28 ds_interpolated = xr.concat(interpolated_datasets, "model")

File ~/anaconda3/envs/DEVELOP/lib/python3.10/site-packages/c3s_eqc_automatic_quality_control/download.py:573, in download_and_transform(collection_id, requests, chunks, split_all, transform_func, transform_func_kwargs, transform_chunks, n_jobs, invalidate_cache, cached_open_mfdataset_kwargs, **open_mfdataset_kwargs)
    569     if invalidate_cache:
    570         cacholote.delete(
    571             func.func, *func.args, request_list=request_list, **func.keywords
    572         )
--> 573     ds = func(request_list=request_list)
    575 ds.attrs.pop("coordinates", None)  # Previously added to guarantee roundtrip
    576 return ds

File ~/anaconda3/envs/DEVELOP/lib/python3.10/site-packages/cacholote/cache.py:85, in cacheable.<locals>.wrapper(*args, **kwargs)
     82             warnings.warn(str(ex), UserWarning)
     83             clean._delete_cache_entry(session, cache_entry)
---> 85 result = func(*args, **kwargs)
     86 cache_entry = database.CacheEntry(
     87     key=hexdigest,
     88     expiration=settings.expiration,
     89     tag=settings.tag,
     90 )
     91 try:

File ~/anaconda3/envs/DEVELOP/lib/python3.10/site-packages/c3s_eqc_automatic_quality_control/download.py:450, in _download_and_transform_requests(collection_id, request_list, transform_func, transform_func_kwargs, **open_mfdataset_kwargs)
    448 if transform_func is not None:
    449     with cacholote.config.set(return_cache_entry=False):
--> 450         ds = transform_func(ds, **transform_func_kwargs)
    451     if not isinstance(ds, xr.Dataset):
    452         raise TypeError(
    453             f"`transform_func` must return a xr.Dataset, while it returned {type(ds)}"
    454         )

Cell In[6], line 87, in compute_indices_and_trends(ds, index_names, timeseries, year_start, year_stop, resample, **regrid_kwargs)
     85 ds = ds.merge(ds_trends)
     86 if regrid_kwargs:
---> 87     ds = diagnostics.regrid(
     88         ds.merge({da.name: da for da in bounds}),
     89         **regrid_kwargs,
     90     )
     91 return ds

File ~/anaconda3/envs/DEVELOP/lib/python3.10/site-packages/c3s_eqc_automatic_quality_control/diagnostics.py:98, in regrid(obj, grid_out, method, **kwargs)
     95     call_kwargs[key] = kwargs.pop(key)
     96 call_kwargs.setdefault("keep_attrs", True)
---> 98 regridder = _regrid.cached_regridder(obj, grid_out, method, **kwargs)
     99 obj = regridder(obj, **call_kwargs)
    100 return obj

File ~/anaconda3/envs/DEVELOP/lib/python3.10/site-packages/c3s_eqc_automatic_quality_control/_regrid.py:41, in cached_regridder(grid_in, grid_out, method, **kwargs)
     39         grid_in[var].attrs = {}
     40 dict_in = grid_to_dict(grid_in)
---> 41 dict_out = grid_to_dict(grid_out)
     42 with cacholote.config.set(return_cache_entry=True, io_delete_original=True):
     43     cache_entry = regridder_weights(dict_in, dict_out, method, **kwargs)

File ~/anaconda3/envs/DEVELOP/lib/python3.10/site-packages/c3s_eqc_automatic_quality_control/_regrid.py:26, in grid_to_dict(grid)
     24 coords = set()
     25 for coord in ("longitude", "latitude"):
---> 26     coords |= set(grid.cf.coordinates[coord] + grid.cf.bounds.get(coord, []))
     27 return xr.Dataset({coord: grid[coord] for coord in coords}).to_dict()

KeyError: 'longitude'

Is the period 2015 to 2099?
Maybe the problem is just that the VM is busy right now.
Have you tried all models?
I run into memory issue with clmcom_eth_cosmo_crclim. Can you try that one if you haven't tried yet.
(The regrid issue is easy to fix)

Yes, the period is 2015 to 2099.
I am running it in my personal iMac and I had no problems for clmcom_eth_cosmo_crclim . For getting the CORDEX period I used the get_cordex_years function:
def get_cordex_years(
    year_start,
    year_stop,
    start_years=[2016, 2021, 2026, 2031, 2036, 2041, 2046, 2051, 2056, 2061, 2066, 2071, 2076, 2081, 2086, 2091, 2096],
    end_years=[2020, 2025, 2030, 2035, 2040, 2045, 2050, 2055, 2060, 2065, 2070, 2075, 2080, 2085, 2090, 2095, 2100],
):

Good news, it's running now.
If everything goes smooth, we can have a look at the interpolation issue tomorrow and close this one.

BTW, I forgot to ask, is the conservative interpolation issue only happening with CORDEX/future proj?
CMIP6 is always OK, and CORDEX/historical is OK?

I don't know why, but now I am finding problems for every case (CMIP6 historical and future included). Could you check with your last version of the notebook? I think I didn't mess it up with the versions, but just in case

OK. Just conservative right?
I though you said that bilinear was fine, so I stopped caching and checking conservative a while ago. Very likely there's a bug.
(Please don't run uncached notebooks on the VM, your notebooks are quite heavy and I think your other notebook I'm caching crashed because of that)

Actually, bilinear is fine. Sorry the misunderstanding, I thought the conservative method was still considered as an option. I wanted to have the option just in case it was necessary to use it or do some checks when regridding the future projections (since the regrid is not done to ERA5 but to a model's grid). Besides I wanted to keep the conservative method as an option as it may be useful for future notebooks.
Apologies, I realised later it was not cached

(Please don't run uncached notebooks on the VM, your notebooks are quite heavy and I think your other notebook I'm caching crashed because of that)

OK, let's finish to cache bilinear, then we'll look into conservative.
I'm pretty sure it's gonna be easy to fix.

Hi @almarbo,
I revised a bit the future projection notebook to deal with CORDEX and CMIP6 using the same code.
I made I couple of quick tests, and I think it's pretty much good to go (I haven't tried conservative yet).
I was able to cache CMIP6, but right now the VM is too busy to cache CORDEX (I will run the scripts overnight).
If you need the output soon (or you want to test it out), you could try to run it locally.
If you do that, please let me know how it goes.
Here is the notebook: https://github.com/bopen/c3s-eqc-toolbox-template/blob/main/notebooks/wp4/extreme_temperature_indices_future.ipynb

Hi @malmans2 ,
I have tested for CORDEX a little bit and I already have cached it on my personal computer, it looks pretty fine. I have tried to plot it without land-sea mask (the land-sea mask downloading is taking a lot) and it plots it the whole domain. I leave a screenshot here.

I have also tested the conservative method, and I confirm that it is still not working.

Hi @almarbo,
bilinear is cached: https://gist.github.com/malmans2/3309ea180f316a1b7a1a35669426efbe
I'll look into conservative now.

Thanks. I think, for now, there's no need to caching the whole period for the conservative, but just ensure it would work.
Regarding the CORDEX future plots. Is there a way to only show in the map the selected area (in order to get an output similar to CORDEX historical)? I guess these differences are due to the fact that for the historical we were regridding for ERA5 and for the future to a CORDEX model

Yes, you can hardcode the extent here:
def set_extent(da, axs):
    extent = [
        da["longitude"].min().values,
        da["longitude"].max().values,
        da["latitude"].min().values,
        da["latitude"].max().values,
    ]
    for ax in axs:
        ax.set_extent(extent)

So substitute extent with [x0, x1, y0, y1]. If you prefer, we can infer it from the area parameter.

Hi @almarbo,
I think I fixed conservative, and I changed the extent behaviour.
Could you please test? https://gist.github.com/malmans2/3309ea180f316a1b7a1a35669426efbe

conservative is also cached for CMIP6 future projection.
Unless you need it, I'm not going to cache CORDEX future projection as it's quite expensive and the VM is too busy right now.
Let me know if we can close this issue.

Hi @malmans2
sorry for the late answer, I have been doing some tests.
No need to cache it for future CORDEX. However, could you check that it worked for that particular case? I tested it in local and I got an error.
I also have doubts on the CORDEX future notebook. I have the feeling something is not going well. There are some longitudinal bands for the trends that are quite strange:

I don't really know the reason why it generates this lines, but it has something to do with the computing of the trends and with the curvilinear latitudes, I guess. For the indices it does not do this. I do not think it has to do with the regridding (the lines can also be seen for the individual models trends - not as evident as the ensemble but it can be easily seen if you remove the land-sea mask ).

I also have doubts on the CORDEX future notebook. I have the feeling something is not going well.

Not sure, if it's a bug it might have to do with dask chunks. I'll check.

I tested it in local and I got an error.

Please share the error traceback

Please don't run CORDEX future on the VM right now.
I'm debugging.
You can clearly see the stripes if you just do this:
ds_regrid["trend"].isel(index=0).plot(robust=True)

Let's see if it's numeric in our python code.

Thank you for checking it!
I confirm that the conservative method is also working for the CORDEX future notebook (previously, I mixed up versions of the notebooks by mistake).

Cool!
I found the bug. Not sure why, but .persist() does not play well with our setup.
I need to replace it with .compute().
I'm going to invalidate the cache of all notebooks, so we are sure that everything is in good shape.
Please don't use the VM until I'm done (very likely tomorrow morning).
Do you know how to invalidate the cache locally? You can just use invalidate_cache=True or set this global variable at the very beginning of your code:
from c3s_eqc_automatic_quality_control import download
download.INVALIDATE_CACHE = True

Please don't do it on the VM though.

Super! thank you so much.
I was using invalidate_cache=True, didn't know about the other, thanks!

hi @malmans2
I have run future CORDEX locally and now the trend has a pretty good shape. Thank you!

Nice! It's getting cached on the VM as well, halfway through.

Hi @malmans2
would you have time these days for a quick chat?
Unfortunately, we have to include a fifth (and last one) notebook to the set. I already have a draft but I would like to comment it with you. The notebook will deal with global warming levels and will be mostly composed of existing code.

How about today at noon?
I'm going to make a minor change in the template meanwhile.
Let's pass the request to get the grid in the cached function rather than the grid dataset. Looks like the grid of codex is not properly cached (probably because of numerics in the hash is not unique)

Today at noon is perfect!

Hi @malmans2
I leave here the global warming levels notebook (only will be used for cmip6) as discussed:
future_gloabal_warm_extreme_temperature_indices.ipynb.zip
As mentioned in the previous meeting, there are some differences compared to the previous notebooks. I have highlighted some of them here and have attempted to improve and comment the code to make it easier for you to understand:

we have two distinct time periods: one historical and fixed, spanning from 1850 to 1900, and the other variable and dependent on the model, representing a future period of 30 years different for each model. Consequently, we have defined a dictionary of requests called request_cmip6_exp, which contains a different request for every model.

The calculation of the 'TX90p' index proceeds as follows: first, the historical and future periods are downloaded and merged. Subsequently, the number of days where the maximum temperature exceeds the 90th daily percentile is calculated for the future period. In this case, the 90th daily percentile is calculated based on the historical period.

For the 'SU' index it is slightly different. For this case, the index is calculated for the historical period and then calculated for the future period.

After these calculations are performed the land-sea mask is applied. Subsequently, the climate signal is calculated. For the 'SU' index, this is calculated as the 'SU' index values for the future period minus the 'SU' index for the historical period. For the 'TX90p', the climate signal is already what was calculated previously so no operation needed.

The plotting part remains the same (the climate signals are shown for 'SU' and 'TX90p').

If you encounter any problem let me know and we can see it together.
Thank you again

@almarbo, I didn't work on the new stuff yet, but I made the modification I told you yesterday.
Everything is now cached on the VM.
See: https://gist.github.com/malmans2/3309ea180f316a1b7a1a35669426efbe

@almarbo let's start with SU.
Do we need to compute separately historical and future?
If we feed to icclim a dataset with both historical and future is it OK?

Sorry, I did not answer your last response, thank you for caching it.

Do we need to compute separately historical and future?
If we feed to icclim a dataset with both historical and future is it OK?

I think we should feed icclim first with the historical and then with the future period, so that we have two mean values and we can get the difference between them

OK, got it

Do you want to use ERA5 for the interpolation?

I rather prefer using the same model as we use for the future notebook

Oh right, was looking at the wrong NB.

I'm working on it, but I realised that pre-industrial data are not cached on the VM.
It looks like the queue on the CDS to access CMIP6 data is quite long right now.

Hi @malmans2 , I was about to write to you. No need to cache preindustrial. Let's keep the classical historical period used until now (actually a little change, if you could make it from 1971 to 2000). Thanks

What do you mean?

we have two distinct time periods: one historical and fixed, spanning from 1850 to 1900, and the other variable and dependent on the model, representing a future period of 30 years different for each model. Consequently, we have defined a dictionary of requests called request_cmip6_exp, which contains a different request for every mode

Do you want to change this from 1850/1900 to 1971/2000?

Exactly! sorry for letting you know so late. We have been discussing about this topic...

mmm ok, I need to revise a little the new code I was working on.
It was based on a single request for all data, then the cached functions were automatically detecting pre vs post industrial data, using 1900 as separator.

@almarbo no trends for this notebook?

no needed

Unfortunately not (I have only for the preindustrial baseline). As you said the CDS for the CMIP6 is extremely slow today and I am not managing to retrieve anything.

You do have the data cached on the VM

in theory the values should be lower than these:

okey,

You do have the data cached on the VM

you are right

Just change the name of the cached function so you don't affect the cache

do you get something like that?:

Yes I think so (I'm getting an outlier on top as well for SU though).
Give me 5min and I'll share the template

Here it is:

Template: https://github.com/bopen/c3s-eqc-toolbox-template/blob/main/notebooks/wp4/extreme_temperature_indices_historical_vs_future.ipynb
NB executed: https://gist.github.com/malmans2/8ece63f5e6e60cd212289724eb86d3f2

Let me know if it looks OK!

Oops, I need to fix a little bug. I think I'm skipping the very first year right now.

OK, I fixed it and the plots are now identical.
Let me know if we can close this issue.

Hi @malmans2 , thank you so much for the notebook, it looks nice.
unfortunately we have found that there are some things for these set of notebooks that should be changed before submitting them. I can leave the desired changes here or we can do a really quick meeting if you prefer (or both). We would like to:

Change the historical period from 1971 to 2005 to 1971 to 2000. (for the historical notebooks)

Change the canesm5 model to inm_cm5_0. This change should apply to the three cmip6 notebooks. Additionally, we have had to change the way of calculating the global warming level ,and, consequently, we should slightly change the slice for the future in the last notebook being them:  # Models time post

models = {
"access_cm2": slice(2022 ,2051),
"awi_cm_1_1_mr": slice(2021,2050),
"cmcc_esm2": slice(2024,2053),
"cnrm_cm6_1_hr": slice(2015,2044),
"ec_earth3_cc": slice(2019,2048),
"gfdl_esm4": slice(2037,2066),
"inm_cm5_0": slice(2031,2060),
"miroc6": slice(2039,2068),
"mpi_esm1_2_lr": slice(2034,2063),
}

For the future notebooks of cmip6 and cordex, the way of calculating the 'TX90p' was done wrongly. It should follow the same approach as the one done in this last notebook, i.e., the index is calculated for the time_range of the future (2015-2099) but the 90th percentile is calculated for the base_period_time_range corresponding to the historical (1971-2000). For the 'SU' index (since it is a physical value) no change is needed.

The first 2 are easy, the last one need a refactor of the current code.
We can not just reuse the work I've done this morning as I thought the TX90p procedure historical vs future was going to be applied to CMIP6 only.

That was our initial idea. However, this morning, while writing some take-home messages and analyzing the results, we realized that it didn't really make sense to calculate the 90th percentile for the same future period. Instead, it should be calculated for the historical period. I apologise for so many iterations and so many changes

The new template is ready and cached for CMIP6: https://gist.github.com/malmans2/3309ea180f316a1b7a1a35669426efbe#file-cmip6-bilinear_future-ipynb
I'm still using the old list of models, I will cache the new models overnight.
Please take a look at it and let me know if you see anything that doesn't look right.
I'm going to update the template we did this morning so it's consistent with the new template for future projections.

okay, thank you. Everything looks right. In fact the TX90p results for the future are now way more consistent.
I noticed, though, that the historical period for the historical notebooks is still the old one instead of the one that we are using now (1971-2000):
# Time period
year_start = 1971
year_stop = 2005

and by the way, aren't we missing five years for the CORDEX historical?
when I print the model_kwargs["request"] I get this:
model='clmcom_clm_cclm4_8_17'
[{'format': 'zip', 'domain': 'europe', 'experiment': 'historical', 'horizontal_resolution': '0_11_degree_x_0_11_degree', 'temporal_resolution': 'daily_mean', 'variable': 'maximum_2m_temperature_in_the_last_24_hours', 'gcm_model': 'mpi_m_mpi_esm_lr', 'ensemble_member': 'r1i1p1', 'area': [72, -22, 27, 45], 'start_year': 1971, 'end_year': 1975, 'rcm_model': 'clmcom_clm_cclm4_8_17'}, {'format': 'zip', 'domain': 'europe', 'experiment': 'historical', 'horizontal_resolution': '0_11_degree_x_0_11_degree', 'temporal_resolution': 'daily_mean', 'variable': 'maximum_2m_temperature_in_the_last_24_hours', 'gcm_model': 'mpi_m_mpi_esm_lr', 'ensemble_member': 'r1i1p1', 'area': [72, -22, 27, 45], 'start_year': 1976, 'end_year': 1980, 'rcm_model': 'clmcom_clm_cclm4_8_17'}, {'format': 'zip', 'domain': 'europe', 'experiment': 'historical', 'horizontal_resolution': '0_11_degree_x_0_11_degree', 'temporal_resolution': 'daily_mean', 'variable': 'maximum_2m_temperature_in_the_last_24_hours', 'gcm_model': 'mpi_m_mpi_esm_lr', 'ensemble_member': 'r1i1p1', 'area': [72, -22, 27, 45], 'start_year': 1981, 'end_year': 1985, 'rcm_model': 'clmcom_clm_cclm4_8_17'}, {'format': 'zip', 'domain': 'europe', 'experiment': 'historical', 'horizontal_resolution': '0_11_degree_x_0_11_degree', 'temporal_resolution': 'daily_mean', 'variable': 'maximum_2m_temperature_in_the_last_24_hours', 'gcm_model': 'mpi_m_mpi_esm_lr', 'ensemble_member': 'r1i1p1', 'area': [72, -22, 27, 45], 'start_year': 1986, 'end_year': 1990, 'rcm_model': 'clmcom_clm_cclm4_8_17'}, {'format': 'zip', 'domain': 'europe', 'experiment': 'historical', 'horizontal_resolution': '0_11_degree_x_0_11_degree', 'temporal_resolution': 'daily_mean', 'variable': 'maximum_2m_temperature_in_the_last_24_hours', 'gcm_model': 'mpi_m_mpi_esm_lr', 'ensemble_member': 'r1i1p1', 'area': [72, -22, 27, 45], 'start_year': 1991, 'end_year': 1995, 'rcm_model': 'clmcom_clm_cclm4_8_17'}]

Yes, good catch.
start/end_years are not listing all years. I'll fix that and make it more general now that I know that cordex start/end always have a 5 year gap.

OK, I think I'm ready to run all of them again.
Could you please check whether the parameters of the notebooks here are correct?

They look okey

They look perfect, thanks.

Great! Let's close this so we can start looking at the other one.


Obsidian protocol handler: Don't install beta plugins right away, just fill in the form

Context: https://github.com/TfTHacker/obsidian42-brat/pull/69#issuecomment-1846581591

Thank you for this additional PR.


Proposal: Filter events to sync

There may be calender events that I do not want to sync, like duplicates or reminders of private events. We could add an optional section similar to transformations, e.g.:
filters:
  - name: Regex
    config:
      Pattern: "(foo|bar)"
  - name: TimeFrame
    config:
      From: "0:00"
      Until: "8:00"

Implemented in #62


Add route support

This adds support for route
Since route and pathname are always exactly the same we can use the same values for both.

Published as 0.9.3


Fix go custom marshal function to correctly handle null cases

Custom marshal functions need to return valid JSON.

https://github.com/grafana/cog/pull/288 should fix the failing "Run examples" CI step


Added a line to force STDOUT output every line

This fixes issues with build systems like Jenkins where you don't see any output until the command is finished.

+1

+1 can we get this in the next release?

Hope this gets merged in at some point.


travis build broken

The travis CI build for p4factory is broken. This is because travis recently added --recursive to their git submodule update --init command, which means we do not have an opportunity to replace ssh access by https access in the floodlight submodules before cloning them. I don't see an easy solution for this.

Found this hidden on stackoverflow:
git:
  submodules: false

fixed by #134


Add updates for October 2020 publish

Initial work for the October 2020 update, only involving Sysmon.

Docs Build status updates of commit 1d00b6c:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

sysinternals/downloads/index.md
:white_check_mark:Succeeded
View

sysinternals/downloads/sysinternals-suite.md
:white_check_mark:Succeeded
View

sysinternals/downloads/sysmon.md
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel

Docs Build status updates of commit 04367ff:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

sysinternals/downloads/diskview.md
:bulb:Suggestion
View
Details

sysinternals/downloads/accesschk.md
:white_check_mark:Succeeded
View

sysinternals/downloads/index.md
:white_check_mark:Succeeded
View

sysinternals/downloads/rammap.md
:white_check_mark:Succeeded
View

sysinternals/downloads/sysinternals-suite.md
:white_check_mark:Succeeded
View

sysinternals/downloads/sysmon.md
:white_check_mark:Succeeded
View

sysinternals/downloads/vmmap.md
:white_check_mark:Succeeded
View

sysinternals/downloads/diskview.md

Line 28, Column 1: [Suggestion-alt-text-bad-value] Alt text 'DiskView' is the same as the image file name. This redundancy adds no value for users of screen readers. For accessibility, add more descriptive alt text. NOTE: This Suggestion will become a Warning on 1/29/21.

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel

Docs Build status updates of commit b335b8b:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

sysinternals/downloads/diskview.md
:bulb:Suggestion
View
Details

sysinternals/downloads/accesschk.md
:white_check_mark:Succeeded
View

sysinternals/downloads/index.md
:white_check_mark:Succeeded
View

sysinternals/downloads/rammap.md
:white_check_mark:Succeeded
View

sysinternals/downloads/sysinternals-suite.md
:white_check_mark:Succeeded
View

sysinternals/downloads/sysmon.md
:white_check_mark:Succeeded
View

sysinternals/downloads/vmmap.md
:white_check_mark:Succeeded
View

sysinternals/index.md
:white_check_mark:Succeeded
View

sysinternals/downloads/diskview.md

Line 28, Column 1: [Suggestion-alt-text-bad-value] Alt text 'DiskView' is the same as the image file name. This redundancy adds no value for users of screen readers. For accessibility, add more descriptive alt text. NOTE: This Suggestion will become a Warning on 1/29/21.

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel

Docs Build status updates of commit 5086575:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

sysinternals/downloads/accesschk.md
:white_check_mark:Succeeded
View

sysinternals/downloads/diskview.md
:white_check_mark:Succeeded
View

sysinternals/downloads/index.md
:white_check_mark:Succeeded
View

sysinternals/downloads/rammap.md
:white_check_mark:Succeeded
View

sysinternals/downloads/sysinternals-suite.md
:white_check_mark:Succeeded
View

sysinternals/downloads/sysmon.md
:white_check_mark:Succeeded
View

sysinternals/downloads/vmmap.md
:white_check_mark:Succeeded
View

sysinternals/index.md
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel

Docs Build status updates of commit 1555002:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

sysinternals/downloads/accesschk.md
:white_check_mark:Succeeded
View

sysinternals/downloads/diskview.md
:white_check_mark:Succeeded
View

sysinternals/downloads/index.md
:white_check_mark:Succeeded
View

sysinternals/downloads/rammap.md
:white_check_mark:Succeeded
View

sysinternals/downloads/sysinternals-suite.md
:white_check_mark:Succeeded
View

sysinternals/downloads/sysmon.md
:white_check_mark:Succeeded
View

sysinternals/downloads/vmmap.md
:white_check_mark:Succeeded
View

sysinternals/index.md
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel

Docs Build status updates of commit a693fa3:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

sysinternals/downloads/accesschk.md
:white_check_mark:Succeeded
View

sysinternals/downloads/diskview.md
:white_check_mark:Succeeded
View

sysinternals/downloads/index.md
:white_check_mark:Succeeded
View

sysinternals/downloads/rammap.md
:white_check_mark:Succeeded
View

sysinternals/downloads/sysinternals-suite.md
:white_check_mark:Succeeded
View

sysinternals/downloads/sysmon.md
:white_check_mark:Succeeded
View

sysinternals/downloads/vmmap.md
:white_check_mark:Succeeded
View

sysinternals/index.md
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel

Docs Build status updates of commit 5453a12:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

sysinternals/downloads/accesschk.md
:white_check_mark:Succeeded
View

sysinternals/downloads/diskview.md
:white_check_mark:Succeeded
View

sysinternals/downloads/index.md
:white_check_mark:Succeeded
View

sysinternals/downloads/rammap.md
:white_check_mark:Succeeded
View

sysinternals/downloads/sysinternals-suite.md
:white_check_mark:Succeeded
View

sysinternals/downloads/sysmon.md
:white_check_mark:Succeeded
View

sysinternals/downloads/vmmap.md
:white_check_mark:Succeeded
View

sysinternals/index.md
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel

Docs Build status updates of commit d328c28:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

sysinternals/downloads/accesschk.md
:white_check_mark:Succeeded
View

sysinternals/downloads/diskview.md
:white_check_mark:Succeeded
View

sysinternals/downloads/index.md
:white_check_mark:Succeeded
View

sysinternals/downloads/rammap.md
:white_check_mark:Succeeded
View

sysinternals/downloads/sysinternals-suite.md
:white_check_mark:Succeeded
View

sysinternals/downloads/sysmon.md
:white_check_mark:Succeeded
View

sysinternals/downloads/vmmap.md
:white_check_mark:Succeeded
View

sysinternals/index.md
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel

Docs Build status updates of commit ad599f8:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

sysinternals/downloads/accesschk.md
:white_check_mark:Succeeded
View

sysinternals/downloads/diskview.md
:white_check_mark:Succeeded
View

sysinternals/downloads/index.md
:white_check_mark:Succeeded
View

sysinternals/downloads/rammap.md
:white_check_mark:Succeeded
View

sysinternals/downloads/sysinternals-suite.md
:white_check_mark:Succeeded
View

sysinternals/downloads/sysmon.md
:white_check_mark:Succeeded
View

sysinternals/downloads/vmmap.md
:white_check_mark:Succeeded
View

sysinternals/index.md
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel


Picker control does not close [sometimes]

Hi,
I am experiencing an issue within nw.js (node webkit) where the picker control sometimes does not close after the mouse has been clicked out side of it. In that case there is now other way of getting this done other than closing the window or destroying the element.
Did this occur before, any idea what might cause this? I suspect some internal event handling issue.
One more bit of info: I am using the picker tool within UI-bootstrap accordions.
Thanks a lot.

I have never used nw.js, are you able to share some code? That would help me greatly.

                                Stroke colour:

                                    <color-picker ng-model="graph.stroke" color-picker-format="'hex8'" color-picker-alpha="false"></color-picker>

I have the same issue (and i'm not using the 'nw.js' lib)
Did you found the problem?

@Nudelsieb @ido567 I cannot replicate your problem, so are you able to give me some environment information like OS, Browser, angular color picker version etc.

@ruhley my machine is Windows 7 64 bit and latest updates. nw.js uses chromium. According to the changelog (https://github.com/nwjs/nw.js/blob/master/CHANGELOG.md) it uses version 41.0.2272.76.
I don't know if this problem might be related to nw.js, but if @ido567 experiences the same issue and runs a different environment, then it may be a problem in color picker?
Thanks

I'v experienced the issue on both Windows and Linux (chrome and safari)
I don't know what triggers it, but when the picker is stuck, nothing cannot remove it

@Nudelsieb @ido567 Can you please uncomment Line 365 and try to reproduce it. If it is being hidden correctly you should see Color Picker: Document Click Event and Color Picker: Hide Event.
The only thing that I can think might be causing it is this if not being run.

I made the changes you asked, next time I will post the logs next time I will have the issue

@ruhley will do so, thanks. I will let you know how it turns out.

@ruhley I got a log output while the problem occured. Basically it shows the correct event but nothing happens.
angularjs-color-picker.js:361 Color Picker: Hide Event```

Those two lines appear over and over every time I click somewhere on the document. however, the color picker is stuck and does not disappear!

@ruhley
The log is as follows when the picker gets stuck.
Color Picker: Document Click Event
Color Picker: Hide Event

Those two lines repeat over and over again every time I click in the document. However, the picker stays open.
Attached is also a screenshot.

Why did you close it? Did you fixed it?

@ido567 good point, didn't see it. I hope it's just a mistake ;-).

@Nudelsieb Sorry to keep going back and forth like this but can you change the line
$scope.log('Color Picker: Hide Event');
to
$scope.log('Color Picker: Hide Event', $scope.visible);,
and tell me what the output is again.
@ido567 I think Close and comment was accidentally clicked.

@ruhley I did. However, sometimes, when the picker gets stuck, there is no log.
I wonder about line 59: ```javascript
if ($scope.find(evt.target).length === 0) {

When you click the picker first this if evaluates to "true" and therefore fires "$scope.hide();". This shouldn't happen since at the same time you also fire "$scope.show();". If you have a timing problem in the event order then it may be possible that you set visible to false shortly after it has been set to true. I found in general, that the "$scope.hide();" fires every time I click something in the document. This can be any element, even buttons. You may need some more checks, say only fire hide if show has been fired before. I hope this makes sense.

I have released v0.6.11 which hopefully fixes your problem. Please tell me how it goes.

Awesome work, thanks. I will see how it goes. Do I see it right that scope.destroy was fired in some cases and removed the click event?
When will the version be available on bower? I still get 0.6.8.

Thanks! I'v updated the package, and will update if the problem will occur again

Closing the issue now as I have not heard anything more. Feel free to open it again if the issue reappears.

So far I haven't noticed any problems, therefore many thanks for the fix, highly appreciated :).


Attendance isn't shown on the admin control panel

Right now the values are dummy values.

In the code its pulling the statistic from the attendance value for each user?


:bug: fix response time improperly configured for micro-seconds.

Description
Microseconds were not properly being output into the log via the common_log() function.
This PR corrects the coversion of seconds to microseconds and clarifies the units used.
GitHub Issues
https://github.com/zappa/Zappa/issues/1229

coverage: 74.571% (+0.007%) from 74.564% when pulling 02187e2aa02e7b515c124671a9133fc9184b151e on fix/wrong-microseconds-log-display into 67b137784d4faa13da9d83a332b17d9e1d46aaad on master.


Prevent clicks when looping in AudioStreamWAV

Partially fixes #64775 (namely, the WAV part. OGG is addressed by #80452).
AudioStreamWAV supports resampling and variable playback speeds by using sub-sample accuracy in the playback position and linear interpolation between two neighboring samples.
When forward or backward looping is enabled between samples s and e (inclusive) and the playback pointer is between samples e and e + 1, interpolation is done between samples e and e + 1. The next/previous sample to be played after/before e, however, is s, so interpolation should instead be done between e and s to prevent discontinuities/clicks in the signal. For pingpong looping, interpolation in that same case should be done between e and e - 1, because the stream is essentially mirrored at the end.
Ensuring this border condition requires one additional check per output sample. I don't know how performance sensitive we are in the audio thread so just to be sure, I added one more template parameter to the do_resample method so that there's a separate code path for mixing blocks near the end of the loop region. That produces quite the cascade of function calls, but allows skipping the border checks for the vast majority of blocks.

I don't know how performance sensitive we are in the audio thread

The audio thread isn't really performance constrained in the traditional sense. Outside of godot I've been doing a lot of embedded programming and it's nothing like that. The main thing is we avoid the use of any mutexes and allocation (because most allocators require mutexes). Audio mixing is the most cache-friendly operation I can think of, and branch predictors on the kinds of processors that Godot runs on tend to be pretty good at their jobs, so I wouldn't sweat an additional branch or two if the condition you're branching on is cheap to evaluate.
Do we have any test files for this bug so I could listen before and after?

I wouldn't sweat an additional branch or two if the condition you're branching on is cheap to evaluate.

Then there's the option to remove the check whether the special case handling is neccessary. Might be slightly less performant but would require less code (one less template parameter and thus half the function calls to do_resample).

Do we have any test files for this bug so I could listen before and after?

I used the cooking_S00 example from the initial post in the issue thread and converted it to WAV. That did click for me when looping:
cooking_S00.zip

For this change I'd say that as you're only adding O(1) cache misses/branch mispredicts (per mix, per WAV playback) you should do whatever is least complex and easiest to read and review.

Ah, yes, that makes a lot of sense. I've removed the template parameter, so all that's new now is a new if in the inner loop and two parameters to do_resample that control the border behavior.
I haven't touched the ADPCM path, by the way. As far as I can tell, that doesn't do interpolation, so there shouldn't be anything to fix there.
Would be great if you could take a look at it now.

I might need to go re-familiarize myself with how wav looping is actually implemented.

I had to do quite a bit of familiarization today, too. 😅 There's a lot going on here.

There's a lot going on here.

"It's just copying one PCM buffer into another, how hard could it be?" -- me, to myself, circa 2020 :laughing:
I'm still getting clicks in the wav file and loop points you provided but in audacity it doesn't seem to be looping at zero crossings, so I'm going to find my own loop points and then re-test.

I tried 396954 to 3571752 and I'm still getting clicks both before and after these changes. :/

Being able to do backwards and ping-pong looping is really neat, but it does open a whole new can of worms, that's true. 😄

I tried 396954 to 3571752 and I'm still getting clicks both before and after these changes. :/

Hm, let me check that again. 🤔 In any case, it's correct that there's no zero crossing at 9 seconds, but the waveform there should line up with the end of the stream.

Hm, let me check that again. 🤔 In any case, it's correct that there's no zero crossing at 9 seconds, but the waveform there should line up with the end of the stream.

That makes sense. And even if you loop at zero crossings I think you can still get sidebands because of abrupt frequency or amplitude changes so that could be what I'm hearing. I'm definitely not a competent sound engineer. Just trying to find a good test case.

So I don't get a click with the original test case at 396900. I do get a click with your 396954, but I suppose that's expected, because the stream ends on a positive value, so the jump to zero will click (should be the same in other applications). 3571752 is 8 ms before the end of the stream. That loop sounds... interesting. 😄
Can you set a breakpoint around line 285 and check what values are actually used for mixing? Mine are:
base_rate = 48000
srate = 44100
playback_speed_scale = 1.0
loop_begin_fp =<PHONE_NUMBER>
length_fp =<PHONE_NUMBER>0

Yeah I just picked zero crossings without listening to them because audacity takes exclusive control of my audio device and I was halfway through an album I like that I wanted to go back and finish.
I don't have an environment set up to actually debug Godot, which I should probably fix because I spend a lot of time working on the engine. I fell out of the habit of working with debuggers when I spent a few years working on an app notorious for crashing debuggers. I'm going to shelve this and work on some other stuff because I do work on Godot as a day job and this is my weekend so it feels like I should be doing something else. I'll try and pick it back up soon though.

Ah that's okay. I was just wondering if maybe we were using different mixing rates or something and if that could explain the difference, so it would have been interesting if the values that actually end up in the method match up. Enjoy your weekend! 🙂

Ok, I tested only with 3 WAV files (I'm pretty tired right now) which were clearly popping. They all have loop offset > 0. But I cut off the intro, so I had 6 files to test (looping back to 0 also popped).
With this PR: They're all still popping.
I only tested forward looping.
I was only skimming over your comments, but you seem to get some popping as well?

Thanks for testing!

I was only skimming over your comments, but you seem to get some popping as well?

I didn't before, but I just tried a few things and I do get a click when I set my system sampling rate to match the file sampling rate! In my mind that should've been unproblematic because this case does not require interpolation and thus should be unaffected by any issues with the interpolation, but clearly that's either not true or this case has a problem of its own. 😅 Thanks for pointing me in the right direction!

Funnily enough, there were indeed two separate issues. The one I fixed in my original commit affects only the resampling case while the other affects almost exclusively the non-resampling case. Specificly, this line:
aux = (limit - offset) / increment + 1
will return one sample too many if the remaining length is a whole multiple of the increment. Quite unlikely with resampling, but guaranteed to always be the case when the playback rate is 1 (file sampling rate matches mixing sampling rate).
I've changed this now to aux = (limit - offset + increment - 1) / increment (for positive increment only), which should give correct results in both cases:

limit
offset
increment
samples to be read
aux before
aux now

1000
0
700
2 (0 and 700)
2
2

1000
0
1000
1 (0)
2 ⚡
1

1000
0
500
2 (0 and 500)
3 ⚡
2

@bs-mwoerner: sorry for the late testing. But I can say that all 6 test WAV files are now looping smoothly (with looping at second 0 and > 0). They still pop in Godot 4.1.1 and 4.2.1 (tested there just for reference). Could you please squash the commits into 1 and rebase on latest on master?
@ellenhp: Ping, could you please review (if you have the time)

That's great! Thanks for testing. I've squashed all commits into one and rebased on the current master. 👍

@bs-mwoerner: I poked @reduz for a review, and he mentioned doing this once on import.
I only took a quick glance at the code, but so far it sounds like the most sensible approach.
What do you think?

@MJacred I had a look at the importer and, yes, as long as there's no way to play a file with a loop mode different from what was specified during import, I think we can hardcode the correct overflow sample into the imported copy of the stream (with a bit of special-case handling for when the loop ends at the end of the stream).
This would solve the issue with having to handle loop modes when picking the next sample (resampling case) and replace that part of this fix. The other part (non-resampling case), though, would still need to be fixed in the playback, I think. If I remember correctly, that was just a miscalculation when advancing the pointer.

I think we do support changing loop mode at runtime. At the very least it is possible and I've suggested people do it before to reduce export size.

@ellenhp: Thanks!
I can confirm. Just tested in Godot v4.2.1: while changing the loop mode in the editor requires a re-import, and the editor makes the UI widgets insensitive for clicks, you can still change the mode at runtime. Even disabling.

@bs-mwoerner: Hm… can we go hybrid with this? No idea if this can work (can also be used as inspiration for a different / more refined solution):

Do it once on the imported copy of the stream
IF the user changes the loop mode at runtime, we flag the AudioStreamPlayback as "loop_mode_is_dirty" and then process it the way your PR currently handles
when the AudioStreamWAV is stopped, check if "loop_mode_is_dirty", then do step 1. and set "loop_mode_is_dirty" to false again

@MJacred Hm, but this would then only work in the editor, since we can't re-import during runtime, can we? With exported builds, we don't even have access to the source media, so we definitely can't reimport anything there. I feel adding considerable code complexitiy for an optimization that doesn't affect release builds isn't really worth it.
However, it looks like the actual WAV data is mutable during runtime, so I suppose we could have a solution that modifies the stream in-memory in accordance with the loop mode. Then we wouldn't need the special case handling during mixing. We would, however, have to make sure that we keep track of these changes, so that we can undo them before a loop setting or the actual data is changed and then modify the stream anew for the new settings afterwards.
I'd like to hear ellenhp's input on this. I remember her saying that with branch prediction, the impact of a single "if" that's almost never taken in the mixing loop is negligible, so I'm not yet convinced that the additional management effort for replacing this with transient stream modifications is a good trade-off (we would add lots of small things that need to be done in a number of places). One example for an edge case, we'd have to consider: It's possible to save the stream to a file while it is being played, so we'd have to make sure that we undo the stream modification on the fly while saving without affecting what the playback sees.

Yeah, I'm not an expert on any particular microarchitecture, but processors are designed to be able to handle branches like the one in the inner loop here without stalling the pipeline. You could optimize out the arithmetic add/subtract by comparing pos against limit_minus_1 which only needs to be initialized on the stack once. At that point I'd imagine the compiler is going to leave everything in a register and you're looking at a single comparison and a conditional jump that will be speculatively executed. It's the kind of thing where I bet it took me longer to write this message than it will take to execute those two instructions across all end-user devices from now until the heat death of the universe.

@bs-mwoerner

Hm, but this would then only work in the editor, since we can't re-import during runtime, can we? […] However, it looks like the actual WAV data is mutable during runtime

Yeah. The re-import is more like a "save this default config".

We would, however, have to make sure that we keep track of these changes, so that we can undo them before a loop setting or the actual data is changed and then modify the stream anew for the new settings afterwards.

Yes, tracking the changes would be necessary.

so I'm not yet convinced that the additional management effort for replacing this with transient stream modifications is a good trade-off

If I understood everything correctly, then the trade-off should be agreeable, right?

One example for an edge case, we'd have to consider: It's possible to save the stream to a file while it is being played, so we'd have to make sure that we undo the stream modification on the fly while saving without affecting what the playback sees.

Hm… I don't get that. Shouldn't the user desire exactly this behaviour (edge case)?

If I understood everything correctly, then the trade-off should be agreeable, right?

Personally, I'd go the "handle this during playback" route, because patching the stream in-memory creates more possiblities to forget to do or undo the modifications at the various places that they need to be done and thus introduces more ways for things to go wrong. If it's true that there's no noticeable difference in performance (I didn't do any benchmarking, so that's an assumption for now), then I'd prefer the simpler solution for maintainability reasons.
Just for context: reduz's original suggestion of preprocessing the stream during import doesn't work with loop mode changes during runtime, because this preprocessing is destructive, so something needs to be done during runtime either way. The remaining options are handling looping during playback and modifying the stream in memory. The first one is simpler and has no side effects (the actual data is not touched), the second one saves one if per sample during mixing but requires some maintenance in the public interface functions.

Hm… I don't get that. Shouldn't the user desire exactly this behaviour (edge case)?

Hm, that's actually a good question. The docs for save_to_wav say "Saves the AudioStreamWAV as a WAV file to path" and doesn't state whether the saved file changes depending on the selected loop mode. Currently it always saves the original waveform, though, so that would be a breaking change, strictly speaking. And since the exported file would have a click where there was none in the original, I can imagine some might consider this more of a bug than a feature.
data is a similar case. Its docs just say "Contains the audio data in bytes." Currently, this returns the unaltered data. Changing this might break some very specific, existing use cases.
Thanks for moving this forward, by the way! I had almost forgotten about this issue.

Happy to be of help! And a big "thank you" for everyone who pitched in!
It's good that we went through our options. Now we have sth. to report back. I'll poke again in the contributor chat to get an official review on this.

@reduz: Can we get your review on this, please? It would be great to get one more audio fix into 4.3

@bs-mwoerner: I got a response from Juan on this issue (I'm late in relaying this)

not convinced to be honest given its so, so so corner case
Internpolation does just that, it should never go past the end sample, so even if there is a next sample it should not be audible
So to me it sounds like sample not being well done rather than a problem with interpolation

What is your take on this response?
If the PR is not acceptable in its current form, maybe there's an option to perform cleanups on import (just throwing this into the room).

Loop offsets >0 aren't an edge case, they're a core feature of the engine. The issue is that there's a bug in the interpolation code that shows up at the loop boundary. I understand having some objections to this PR. It adds branches in inner loops. I understand wanting to see profiling data to back it up, or asking for it to be commented better. But if Juan doesn't want the underlying bug to ever be fixed I don't really know what to say.
On a meta level: It's very frustrating that after putting in dozens or sometimes hundreds of developer-hours across all contributors into a fix, ~5 minutes of Juan's time can completely sink it because he either doesn't understand the problem fully or doesn't think it's important enough to review. This is the main reason I completely deprioritized contributing to and maintaining the project. I recognize that I'm rocking the boat by saying this, but it's because I still care about the project, and I want the best for it. Providing this feedback is important.

What is your take on this response?

I was a bit confused initially, because when Juan says "Interpolation does just that, it should never go past the end sample, so even if there is a next sample it should not be audible", that sounds like an exact description of what this PR does: It stops the interpolation from going past the end sample, which it previously did. After giving this some thought, I now think when Juan says "interpolation", he may refer to overlapping and fading between the end and the start of the loop (whereas we were talking about interpolating between individual samples in the stream to account for sample rate differences or non-standard playback rates).
I ran a new test with the official 4.3 build and as of now, the clicking problem is still there, but if there are plans to implement overlapping loops for WAV streams at some point in the future, then Juan may mean that we shouldn't bother fixing these issues now, because they will likely not be heard through the fade anyway?


Move WorkItem links.self to data.links.self

WorkItemList now include links.self to each WorkItem in the list
Related to #526

@sanbornsen This change allow you to use GET [n].links.self to move from the WorkItemList to the WorkItem details.
{
	"data": [
		{
			"attributes": {
				"system.created_at": "2016-10-27T10:16:03.829904Z",
				"system.creator": "DominicBoettger",
				"system.description": "TypeError: undefined is not a function\nangular-material.js:1643:27\n\ninputDirective.link.pre(scope, {\n        on: angular.noop,\n        0: {}\n      }, attr, [ngModelCtrl]);\n\nAngular 1.3 rc2\n",
				"system.remote_item_id": "https://api.github.com/repos/angular/material/issues/408",
				"system.state": "closed",
				"system.title": "Checkbox not working",
				"version": 0
			},
			"id": "104840",
			"links": {
				"self": "http://localhost:8080/api/workitems/104840"
			},
			"relationships": {
				"assignee": {},
				"baseType": {
					"data": {
						"id": "system.bug",
						"type": "workitemtypes"
					}
				}
			},
			"type": "workitems"
		}
	]
}

Current coverage is 68.17% (diff: 100%)

Merging #568 into master will increase coverage by <.01%

@@             master       #568   diff @@
==========================================
  Files            70         70
  Lines          3898       3899     +1
  Methods           0          0
  Messages          0          0
  Branches          0          0
==========================================
+ Hits           2657       2658     +1
  Misses          971        971
  Partials        270        270

Powered by Codecov. Last update d34333a...59807e3


Caching support

Originally created by Antoine Nguyen on 2012-10-21T09:54:22Z
To improve performance, Modoboa should use a local cache to store IMAP information. Maybe a Redis or MongoDB backend to store:

UID per mailbox
Body structures
etc.

This issue was moved to modoboa/modoboa-webmail#15


[问题] 在 win 上加载插件报错

在 win 上执行 测试报错：
{"level":"error","error":"connect grpc plugin failed: fork/exec E:\\self_work\\other\\lemon\\server\\debugtalk.bin: %1 is not a valid Win32 application.","time":"2022-08-05T15:06:24+08:00","message":"init plugin failed: E:\\s
elf_work\\other\\lemon\\server\\debugtalk.bin"}
{"level":"error","error":"init plugin failed: connect grpc plugin failed: fork/exec E:\\self_work\\other\\lemon\\server\\debugtalk.bin: %1 is not a valid Win32 application.","time":"2022-08-05T15:06:24+08:00","message":"[Run]
 init session runner failed"}

请问改怎么处理？

在 linux 上也会出错
{"level":"error","error":"connect grpc plugin failed: fork/exec /hrp/lemon/server/debugtalk.bin: exec format error","time":"2022-08-05T16:52:59+08:00","message":"init plugin failed: /hrp/lemon/server/debugtalk.bin"}
{"level":"error","error":"init plugin failed: connect grpc plugin failed: fork/exec /home/qiyouji/hrp/lemon/server/debugtalk.bin: exec format error","time":"2022-08-05T16:52:59+08:00","message":"[Run] init session runner failed"}


Flaky intellisense test

This is definitely flaky, i've run into this a few times, no idea whats going on here.
Seems to be a problem with the language server itself.

Test is enabled and I've seen no failures for this recently, closing.


Update to Glide v4?

Can you tell me a brief pros n cons? Or just a link 😄

I would suggest to read the release changelog: https://github.com/bumptech/glide/releases
And at my point - the changes a dev has to make in code are very small.

@Rainer-Lang will check on it ASAP 😄 👍

Please do it asap. Adding glide externally breaks compatibility.

@Rainer-Lang @ks419 updated and release in 1.9.0 🎉
Please kindly check 🙏


rear camera

How to use the rear camera on smartphones?

Ok! Tks!

@Sixbitunder Don't you mean if (cameras.length > 1) { ... } though?


2.x Remove legacy eviction code.

NetflixOSS » eureka » eureka-pull-requests #355 SUCCESS
This pull request looks good


[New API] : BBC News API

Brief Description 🤓
BBC, British Broadcasting Corporation is a Trustable News Site. It has coverage of 30 languages.
How will you develop it?🤩
BBC News API is the API for serving the news from all the BBC Services according to your need. This API has a coverage of 30 Languages (All supported languages by BBC)!
Deployment
None
Select the open source program you are contributing to
Other
Code of Conduct

[X] I follow CONTRIBUTING GUIDELINES of this project.

Greetings @Parth20GitHub🎀 ! , We are excited to have you dive into another issue with us!🚀
Your involvement in our project has been invaluable, and we're confident that your skills and insights will help us conquer this challenge😇.  Use command /gssoc24 to self assign the issue.  This issue represents an exciting opportunity for us to improve and enhance our project, and we are thrilled to have you on board.
We request you to follow CONTRIBUTING GUIDELINES.
Looking for your PR soon!
In case of any issues, you can contact us on Discord.❣

/gssoc24


Import all book content

Thanks for the plugin! Just great.
I would like to know if there is any chance to import all the book content to the note as well.
It would be very useful to me if I could read from the note the context where I did the highlight or comment.
Thanks!

With all the book content, you mean literally export the whole book?
The book's content is not stored in the database, but in an .epub file on the device.
I think this feature is out of scope for this plugin, and you're better of using an .epub to .md conversion tool like https://pandoc.org/

Thanks for the answer. Right after I wrote that I just thought about what you are saying. Also that there would be some sort of Copyright problem aside.
Thanks for creating the plugin! It's really helpful to me!


How to split one UXTO that contains 4 inscriptions?

Hi, I used Bitcoin-cli to inscribe 4 months ago, find that 1 UXTO(output value about 160,000 sats) contains 4 inscriptions.
When I try to transfer 1 of them, I get the error: "error: cannot send xxx without also sending inscription xxx at xxx".
How should I do?

You need to move the satpoints from the back of the output to the front.

Hi, thanks for your advice. But when I try
wallet send target_address inscription_4_id
still get the same error..

Hi @pdm1230
Can you see if https://sating.io/ will work for you?
Make sure you send them from the end to the beginning there too. Make sure you have another UTXO at the same address to use for fees.
To make ord do this you would need to patch the code to ignore any inscription checks and treat them as raw sats.

@raphjaph @casey How should I do? Thanks

You need to move the satpoints staring from the back of the output to the front.
E.g.:
UTXO1: [ ins1 ins2 ins3 ins4 ]
Find the sat point for ins4, use ord wallet send to move it to a new address.
Now your UTXOs look like:
UTXO1: [ ins1 ins2 ins3 ] UTXO2: [ ins4 ]
Repeat the process for ins3 and ins2.
The reason you work from the end to the beginning is so you can protect your inscription sat points from being used for fees or as padding for other inscriptions.

How do I achieve this? Can I achieve this using psbt?


code128 not Working, Only QR....

On pc and android barcodes wont work. It is possible to fix it? Only QR codes work good.

Hello,
I not understand why. It works for me.

But sometimes the decoding not get correct result.
Multiple decoding, comparing needed.
Regards


cbd init does not work with or wothout Profile

Whether I have a Profile or not, cbd init refuses to run.
[WARN] We can not guess your PUBLIC_IP, please run the following command: (replace <IP_ADDRESS> with a real IP)
echo '<IP_ADDRESS>' > Profile
If I follow the directions by running:
root@wiretux:~/cloudbreak-deployment # echo $(ifconfig eth0 |grep -F 'inet ' |awk '{print $2;}') >Profile
root@wiretux:~/cloudbreak-deployment # cat Profile
<IP_ADDRESS>
Now I have a Profile that satisfies the instructions I was given. Trying cbd init again:
./Profile: line 1: <IP_ADDRESS>: command not found
Assuming that you really meant that my Profile should contain:
export PUBLIC_IP=<IP_ADDRESS>
Manually made this change. Running cbd init again:
[WARN] Profile already exist, move away if you want to regenerate
It's a "damned if I do, damned if I don't" situation.

Hi Carlos,
Thanks for reporting. You are right the instructions should be:
echo export '<IP_ADDRESS>' > Profile

Once you have the Profile, there is nothing to init you start to
cbd generate
cbd pull

But the command should say something like:

You already have a Profile, you can run cbd generate now

You can test with a snapshot version:
cbd update master
cbd init

Reproducing
Here is my test (reproducing your issue with 0.1.0)
$ echo export PUBLIC_IP=<IP_ADDRESS>>Profile
$ cbd init
[WARN] Profile already exist, move away if you want to regenerate

testing the fix
$ cbd update master

Update /bin/cbd from: https://circle-artifacts.com/gh/sequenceiq/cloudbreak-deployer/123/artifacts/0/tmp/circle-artifacts.jXdpNpT/cbd-linux.tgz

$ cbd --version
CloudBreak Deployer: 0.1.0-master-d1eb594

$ cbd init
Profile already exist, now you are ready to run:
cbd generate

Much appreciated! Worked.


Extension UI is too narrow

When I click on the extension on Chrome this is what I see:

Can the extension display the full line and the relevant text in the same row ?

Thank you for your feedback. We didn't finish the UI work, and this issue was mentioned on https://github.com/os-scar/overlay/issues/27.
Closing as duplicate.


Transfer analytics

Hey Noah, please reach out to plural with your email that owns nrd analytics so we can transfer it into the new analytics account.

Yes i can absolutely do that for you.

oh, i guessed and upgraded the one you had as a reader for nisei to admin so if i guessed correctly, you should be able to move the NRDB property over.  https://support.google.com/analytics/answer/6370521


Add awesome-captcha

https://github.com/ZYSzys/awesome-captcha
This repo is a collection of awesome captcha libraries and captcha crack tools.
By submitting this pull request I confirm I've read and complied with the below requirements 🖖
Please read it multiple times. I spent a lot of time on these guidelines and most people miss a lot.
Requirements for your pull request

I have read and understood the contribution guidelines and the instructions for creating a list.
This pull request has a descriptive title.For example, Add Name of List, not Update readme.md or Add awesome list.
The entry in the Awesome list should:

Include a short description about the project/theme of the list. It should not describe the list itself.Example: - [Fish](…) - User-friendly shell., not - [Fish](…) - Resources for Fish..
Be added at the bottom of the appropriate category.

The list I'm submitting complies with these requirements:

Requirements for your Awesome list

Has been around for at least 30 days.That means 30 days from either the first real commit or when it was open-sourced. Whatever is most recent.
It's the result of hard work and the best I could possibly produce.
Non-generated Markdown file in a GitHub repo.
Includes a succinct description of the project/theme at the top of the readme. (Example)
The repo should have awesome-list & awesome as GitHub topics. I encourage you to add more relevant topics.
Not a duplicate.
Only has awesome items. Awesome lists are curations of the best, not everything.
Includes a project logo/illustration whenever possible.

Either fullwidth or placed at the top-right of the readme. (Example)
The image should link to the project website or any relevant website.
The image should be high-DPI. Set it to maximum half the width of the original image.

Entries have a description, unless the title is descriptive enough by itself. It rarely is though.
Includes the Awesome badge.

Should be placed on the right side of the readme heading.
Should link back to this list.

Has a Table of Contents section.

Should be named Contents, not Table of Contents.
Should be the first section in the list.
Should only have one level of sub-lists, preferably none.

Has an appropriate license.

That means something like CC0, not a code licence like MIT, BSD, Apache, etc.
WTFPL and Unlicense are not acceptable licenses.
If you use a license badge, it should be SVG, not PNG.

Has contribution guidelines.

The file should be named contributing.md. Casing is up to you.

Has consistent formatting and proper spelling/grammar.

The link and description are separated by a dash. Example: - [AVA](…) - JavaScript test runner.
The description starts with an uppercase character and ends with a period.
Consistent and correct naming. For example, Node.js, not NodeJS or node.js.

Doesn't include a Travis badge.You can still use Travis for list linting, but the badge has no value in the readme.

Go to the top and read it again.

The description starts with an uppercase character and ends with a period.

Not done

And I correct the character now ?

Same on list itself, describe what a Captcha is, not what the list is.

And then is it ok now ?


Yet another starter template 😊

Hello fine folks 👋 ,
I would love to add this one to the list in the hope that it can be of some help. Cheers.

Thank you!


N15: rg3d

Part of #320 ("Newsletter 15")

Thanks for the PR!
Note this line from CONTRIBUTING.md though:

Don't send PRs from your main branch, create a unique branch (named like n14_zemeroth, n12_veloren, etc) for each PR. This allows sending multiple simultaneous PRs and simplifies the creation of the next PRs.


Integration with Spark ,Mesos, Akka, Cassandra, Kafka stack crashes my Windows XP server

This should work, but unfortunately it's not working.
I think it's this configuration option:
`
            `

Or perhaps the windows registry

Try the following steps:
1.
2.

This worked for me as a quick-and-dirty workaround. Please keep in mind that I'm not responsible if your fusion core suddenly collapses - you are on your own here.
(Keep in mind that you need to install Windows in the Lojban language and have at least SP2 installed)


Service Network option is unavailable under "create network" section of opsman configuration

Using pcf-vsphere-2.1-build.326.ova for opsmanger deployment
https://docs.pivotal.io/runtimes/pks/1-1/vsphere-om-config.html
Step 5: Create Networks Page
Doc says - Service Network	Leave Service Network unchecked for network creation
but with the above mentioned build I don't see the option of "service network"

@shausy Thanks for raising this issue. We've updated the docs to reflect the removal of the "Service Network" checkbox.


update to support ruby 3.0

unf_ext-<IP_ADDRESS>-x64-mingw32 requires ruby version >= 2.2, < 2.8.dev, which is
incompatible with the current version, ruby 3.0.0p0

Any update on this? I'm running into this as well.

@knu can we ask your help on this please? Ruby 3.0 looks not supported on x64-mingw32

looking at the non mingw32 version the required ruby version is >= 0. Can this gem simply be updated with the same requirement.

looks like PR #64 should fix this issue

This is done -- can be closed.


Update extension-based-hybrid-runbook-worker-install.md

Added info, not to add DCs into hybrid worker groups. This way you can elevate domain admin f.e. to everyone. From my pov a major security issue. MS support team informed.

https://msrc.microsoft.com/report/vulnerability/VULN-131943/

@SnehaSudhirG
Can you review the proposed changes?
Important: When the changes are ready for publication, adding a #sign-off comment is the best way to signal that the PR is ready for the review team to merge.
#label:"aq-pr-triaged"
@MicrosoftDocs/public-repo-pr-review-team

@Court72 @SnehaSudhirG are there already any updates?

I sent an email to the content owner today.

@Court72 - Thank you for your patience. Awaiting confirmation from PM/Engg team

I sent an email to the content owner today.
@MicrosoftDocs/public-repo-pr-review-team

This MR is now 3+ months open.
This request is not about some small version shit, this is a huge security issue. Could you please approve this request? @SnehaSudhirG

@IrgendwasMitSchnittlauch - Thank you for your patience, the info is updated as a note and here is the live link - https://learn.microsoft.com/en-us/azure/automation/extension-based-hybrid-runbook-worker-install?tabs=windows%2Cbicep-template