Dealing with experiments with multiple sponsoring MIPs

The discussion in #937 has diverged from the issue title and is now focused on better clarifying how modeling groups need to identify data contributed to an experiment which is a contribution other than the primary sponsoring MIP.
To attempt to keep discussions contained in a suitably identified discussion, I have opened this new issue with an appropriate title. @MartinaSt @nedavid @taylor13 @wachsylon @sjmarsland @martinjuckes @sol1105 pinging you here for discussion continuity.
The example which led to #937 being submitted was the ssp370 experiment which is sponsored by ScenarioMIP, AerChemMIP.
Just for context, in the experiment_id CV of the CMIP6_CVs, the number of experiments that list more than a single activity include:

experiment_id
activity_id (primary)
activity_id (additional)

dcppC-forecast-addPinatubo
DCPP
VolMIP

esm-1pct-brch-1000PgC
C4MIP
CDRMIP

esm-1pct-brch-2000PgC
C4MIP
CDRMIP

esm-1pct-brch-750PgC
C4MIP
CDRMIP

esm-1pctCO2
C4MIP
CDRMIP

esm-bell-1000PgC
C4MIP
CDRMIP

esm-bell-2000PgC
C4MIP
CDRMIP

esm-bell-750PgC
C4MIP
CDRMIP

land-hist
LS3MIP
LUMIP

piClim-aer
RFMIP
AerChemMIP

piClim-control
RFMIP
AerChemMIP

ssp370
ScenarioMIP
AerChemMIP

For reference, https://github.com/WCRP-CMIP/CMIP6_CVs/issues/937#issuecomment-644243564 is where discussion about this separate issue begins. I will close #937 as that particular issue has been resolved

The above is not really a statement of an "issue"; it's simply extracting the special cases from the CV.  The issue(s) relate apparently to confusion resulting from improper handling or interpretation of these special cases. I'll try to enumerate these issues:

The modeling groups have recorded their aspiration to participate in a MIP in their entry labeled activity_participation in the CMIP6_source_id.json
Even though the activities they listed there were originally not meant to be definitive, the citation service relies on activity_participation in generating a potential list of citations that is needed once data have been published before the service can actually automatically create a citation.  [@MartinaST  please check that I've described this correctly.]  So, as I understand it, if the primary sponsor of an experiment (i.e., the MIP that is supposed to be recorded first in activity_id in the file, and as activity_drs on ESGF, and used in generating directory structures, and communicating throughout the infrastructure) has not been included in the activity_participation list, then a citation cannot be created.
The ESGF publisher does not check, apparently, whether the first MIP listed in the activity_id is consistent with the first MIP listed for the experiment_id CV; it just checks that the activity_id value(s) correspond with those found in the experiment_id CV (and if only one of multiple values is found, the data is published anyway).  This means data (not written through CMOR, which enforces the correct number of entries and their ordering) can get published with the wrong activity_drs value. In the future, I think we should guard against this happening. Also some data providers think that if they are not participating in the MIP that is the primary sponsor of an experiment, then they should publish the data   using activity_id (and activity_drs) set to the secondary MIP (which they do participate in).  This is not correct.  They need to publish their experiment under the primary sponsoring MIP even if it isn't included in their list of activity_participation.  I think we may have steered @nedavid wrong in https://github.com/WCRP-CMIP/CMIP6_CVs/issues/937#issuecomment-629993922 .
The use of activity_id in the ESGF search is meant to allow users to narrow consideration to only the experiments that are sponsored (or co-sponsored) by an activity.    It is not meant to allow users to narrow consideration of models that are participating in a particular MIP. But it is understandable that some have misinterpreted this search facet. In any case if a dataset has properly recorded all the activities responsible for an experiment in the activity_id, then these datasets will be found searching for any of the activities (even though only the first activity is used as part of the DRS).

In summary,

no one should rely on the activity_participation for definitive information.  Modeling groups may indicate in activity_participation that they are participating in a MIP that is a secondary sponsor of a particular experiment, but that same group must publish the data under the activity designated as the primary sponsor.  If I understand the present state of things, this somehow prevents the citation service from producing a citation.  Is that true?  If so, the proper remedy is to 1) make sure the data lists the primary MIP sponsor first in the global attribute activity_id, and 2) ask the modeling group to add the primary sponsors name to their activity_participation list, even though they may not really be participating in that activity.  Alternatively, the citation service could be altered so that it doesn't consult activity_participation for the purpose of creating citations.

In the global attribute activity_id the string can be a list of activities and it was meant to be a list in the same order as it appears in activity_id of the experiment_id CV.  CMOR ensures that it will be, but PrePARE and the ESGF publisher do not enforce the ordering, so some data has gotten published with the wrong activity being recorded in activity_drs (and presumably in the directory structure and elsewhere in the CMIP6 infrastructure).  PrePARE and/or the publisher should implement a check that all (co-)sponsoring activities are listed in the correct order in activity_id.  The data from all models that have run a given experiment will be found regardless of which sponsoring MIP one filters against. [We can consider whether to change the approach for CMIP7.]

@taylor13 Thanks for writing this up.
Two comments:

"citation service relies on activity_participation":  The citation service indeed uses the activity_participation information. The reason is that it would be impractical to provide all possible MIPs and experiments for every modeling center. I provide a GUI, which is used be the majority of the modeling centers. The use of a GUI gets cumbersome for a large number of entries and impractical if the majority of those entries is of no interest.
"primary sponsor or order of sponsors": I don't know if a PrePARE check for the order of the sponsors is the right check to catch the inconsistency between registered information in the CV on the one hand and DRS and global attributes of the files on the other hand. Excluding the activity from the DRS for CMIP7 might be an option as activity is not required to make dataset IDs unique.

No, the citation service cannot be altered this way, as it checks against master_id and not activity. It relies on the unique ESGF master_id or the DRS_id of the data to connect the citation to the right data collection. This DRS_id makes the citation independent of the current portal solution, which is essential on the long-term. These citation entries with their DRS_ids are defined by the modeling center in their registration of activity_participations for a source_id. The modeling center prepares the data for the registered activities and publishes them. The citation connects its metadata using the unique ESGF master_id (DRS_id) The master_id is also used to create the citation_url in the ESGF index. If a modeling center decides to prepare its data for an activity, they do not intend to participate in (in terms of global attributes in the files and DRS), the ESGF publishes the data with these wrong master_ids, for which no citation entry exists (and according to the modeling center's registration should not exist).

@MartinaSt @martinjuckes @durack1 @sashakames For CMIP6, will the following solution work for ES-DOC, CMOR, ESGF, data citation services, the data request, the CV's etc. (at least going forward)?

For any given experiment, the ESGF publisher (perhaps relying on PrePARE?) should check that the correct primary activity (defined to be the activity that is listed first in the experiment_id CV under "activity_id") has been recorded by the data provider in the file's activity_id.  If it doesn't, we could

reject the file as being non-compliant (my choice), or
accept the file, but in the ESGF catalog enter a corrected the activity_id and activity_drs, consistent with the experiment_id CV.

If the activity_id contains entries that are ordered incorrectly, but otherwise consistent with the list in the experiment_id CV, we could

accept the file, but in the ESGF catalog, set activity_drs to the first activity listed for the experiment in the experiment_id CV, ignoring the ordering that the activities appear in the activity_id global attribute (my choice), or
reject the file.

The citation service should invariably associate an experiment with the activity that is the primary sponsor of the experiment (i.e., listed as the first activity for that experiment in the experiment_id CV).  If the citation service relies on activity_participation to generate citations, then we would need to insist that the modeling group list in activity_participation all activities that are the "primary" activities for experiments they have performed (whether or not they actually are carrying out all the tier-1 experiments called for by that activity).

A question is: how difficult would it be for the ESGF publisher to implement the above (i.e., check and possibly correct activity_id?
As I understand it, nothing would have to be done by @MartinaSt , but how would we (she) find out that a group had failed to include a needed primary activity in their activity_participation list?  Would ESGF throw an error?  Would Martina receive some sort of notice that her service was unable to associate a citation with a data set?  or what?
Please say if the above won't work and why.
thanks.

@wachsylon the intention of activity_participation was to collect intentions (to participate in satellite MIPs), and so we don't want to lose that at the point that the model is registered.
I am also not keen on the idea of a new issue being created automagically, rather, an error would be thrown, providing the URL to create a new issue so that whoever is trying to publish data is very aware of what is causing the hold-up and takes ownership over this registration step.
We have been trying to ascertain how much checking we can get the publication step to do, and I think that validation, not registration is the only role of the publisher checks

@sashakames I have 2 questions:

Does the ESGF publisher check the directory structure for consistency with the CMIP6 specs?  If so, does it check that the "activity" is the same as the first entry in the activity_id global attribute?
When the ESGF publisher attempts to publish a dataset, does it know if @MartinaSt citation service is able to link it to a citation, or does that happen later?  If the publisher learns that the citation information is missing, does it currently error exit or what?

thanks.

Hi All, some observations (I've only just seen this conversation);
I've published MOHC data [1] via the CEDA esgf node for the two shared RFMIP-AerChemMIP experiments noted above which have the activity_id AerChemMIP in their dataset ids [2]. If I search for these on ESGF they all have links to file paths within THREDDS that use RFMIP for the activity_id, i.e. some part of ESGF uses the primary activity_id for the construction of directory paths, which is in conflict with the dataset id.
Searching back through the standard documentation (CMIP6 guidance and global attributes) the only reference to this issue I can find is in a bullet point in the global attributes document under the description of the DRS structure.  My assumption was that if you haven't registered a model against an activity_id (UKESM1-0-LL wasn't initially registered for submission to RFMIP) that you should not attempt to submit to it.
I'd be tempted to suggest that where a group has published data using the additional activity_id that a "won't fix" errata is issued (it would be a little disruptive to correct this now). From what I can see on ESGF [3] there are three institutes that have done this for the two RFMIP-AerChemMIP experiments; MOHC, BCC and NOAA-GFDL. Similar searches should quickly highlight where else this has happened.
[1] with activity_id attribute set to RFMIP AerChemMIP in each file by CMOR.
[2] search for id:CMIP6.AerChemMIP.MOHC.UKESM1-0-LL.piClim-control.* or id:CMIP6.AerChemMIP.MOHC.UKESM1-0-LL.piClim-aerl.* on ESGF
[3] search for id:CMIP6.AerChemMIP.*.piClim-control.*

@matthew-mizielinski : Thanks for engaging on this.  We need your perspective/advice.  I'm confused about the terminology being used.  From the ESGF page, when I select one of your piClim-control files and ask ESGF to "show metatdata", some of the information shown is:
d = CMIP6.RFMIP.MOHC.HadGEM3-GC31-LL.piClim-control.r1i1p1f3.Amon.hus.gn.v20191113|esgf-data3.ceda.ac.uk
version = 20191113
_timestamp = 2019-11-13T14:59:58.401Z
access = HTTPServer, GridFTP, OPENDAP, Globus
activity_drs = RFMIP
activity_ids = RFMIP, AerChemMIP
cf_standard_name = specific_humidity
citation_url = http://cera-www.dkrz.de/WDCC/meta/CMIP6/CMIP6.RFMIP.MOHC.HadGEM3-GC31-LL.piClim-control.r1i1p1f3.Amon.hus.gn.v20191113.json
data_node = esgf-data3.ceda.ac.uk
data_specs_version = 01.00.29
dataset_id_template_ = %(mip_era)s.%(activity_drs)s.%(institution_id)s.%(source_id)s.%(experiment_id)s.%(member_id)s.%(table_id)s.%(variable_id)s.%(grid_label)s
datetime_start = 1850-01-16T00:00:00Z
datetime_stop = 1879-12-16T00:00:00Z
directory_format_template_ = %(root)s/%(mip_era)s/%(activity_drs)s/%(institution_id)s/%(source_id)s/%(experiment_id)s/%(member_id)s/%(table_id)s/%(variable_id)s/%(grid_label)s/%(version)s
.
.
.
instance_id = CMIP6.RFMIP.MOHC.HadGEM3-GC31-LL.piClim-control.r1i1p1f3.Amon.hus.gn.v20191113
institution_id = MOHC
master_id = CMIP6.RFMIP.MOHC.HadGEM3-GC31-LL.piClim-control.r1i1p1f3.Amon.hus.gn
.
.
.

The only place I see AerChemMIP is as the second MIP listed in activity_id.  I think this is  done correctly and is exactly what I hoped to see.  You mention a dataset i.d. and as an example give id:CMIP6.AerChemMIP.MOHC.UKESM1-0-LL.piClim-aerl.*.   How do you get this i.d.?
To be sure the ESGF search is smart enough to find piClim-aerl experiment data under either of the two activities listed in activity_id, but I don't think it uses the second activity in defining any fundamental i.d.  To go further, do you know where the dataset i.d., you refer to is stored  and what it is used for?

Let me get my terms correct; I'm used to using "dataset id" for what is recorded in ESGF as the instance_id (my fault) and I'll try to correct this forthwith.
Apologies for the typo in the instance_id above (now corrected); it should have included piClim-aer not piClim-aerl. The syntax for searching by this id is from the search page of esgf (under more search options).
The dataset you pick out is from the HadGEM3-GC31-LL "physical" model, which was processed and published with the primary activity_id RFMIP in the instance_id.  For our earth system model UKESM1-0-LL things were done differently as it (initially*) wasn't registered in the CVs to contribute to RFMIP, so we used AerChemMIP for the shared experiments that were wanted by the AerChemMIP community.
I'll pick out one dataset from the UKESM1-0-LL submission;
id = CMIP6.AerChemMIP.MOHC.UKESM1-0-LL.piClim-control.r1i1p1f4.AERday.ua10.gn.v20200214|esgf-data3.ceda.ac.uk
version = 20200214
_timestamp = 2020-02-21T17:35:51.735Z
access = HTTPServer, GridFTP, OPENDAP, Globus
activity_drs = AerChemMIP
activity_ids = RFMIP, AerChemMIP
cf_standard_name = eastward_wind
citation_url = http://cera-www.dkrz.de/WDCC/meta/CMIP6/CMIP6.AerChemMIP.MOHC.UKESM1-0-LL.piClim-control.r1i1p1f4.AERday.ua10.gn.v20200214.json
data_node = esgf-data3.ceda.ac.uk
data_specs_version = 01.00.29
dataset_id_template_ = %(mip_era)s.%(activity_drs)s.%(institution_id)s.%(source_id)s.%(experiment_id)s.%(member_id)s.%(table_id)s.%(variable_id)s.%(grid_label)s
.
.
.
instance_id = CMIP6.AerChemMIP.MOHC.UKESM1-0-LL.piClim-control.r1i1p1f4.AERday.ua10.gn.v20200214
institution_id = MOHC
master_id = CMIP6.AerChemMIP.MOHC.UKESM1-0-LL.piClim-control.r1i1p1f4.AERday.ua10.gn
.
.
.

Note that AerChemMIP is  used in the id, activity_drs, citation_url and instance_id. If you follow the links to the THREDDS catalogue you'll eventually get to this page which has links to files that use RFMIP in the path name rather than AerChemMIP, e.g.
https://esgf-data3.ceda.ac.uk/thredds/fileServer/esg_cmip6/CMIP6/RFMIP/MOHC/UKESM1-0-LL/piClim-control/r1i1p1f4/AERday/ua10/gn/v20200214/ua10_AERday_UKESM1-0-LL_piClim-control_r1i1p1f4_gn_18500101-18941230.nc
Data is also stored on disk under RFMIP rather than AerChemMIP; files can be found under RFMIP in the CEDA archive, but not under AerChemMIP
So I think what I was trying to communicate is that we correctly have included both activity ids in the file metadata and that THREDDS appears to host the data using the primary activity_id RFMIP, but in all other aspects these datasets have AerChemMIP as their activity_id.
*UKESM1 was registered to take part in RFMIP via #731 a year ago, but we never got round to changing the processing configuration for the piClim-control and piClim-aer experiments as I had already published the first versions using AerChemMIP.

Thanks for pointing me to the model where the "problem" is.  I'm pretty sure I wouldn't recommend republishing the above dataset, since it seems to be discoverable and citable.  The question is, should we (for future publications) guard against publishing datasets with the secondary activity used to define activity_drs, instance_id, master_id, citation_url, and the like.
I thought that the publisher keyed on activity_id, and extracted the first activity to use as activity_drs, which then got subsequently used to define the master_id, instance_id and citation_url.  From the example above, however, that doesn't seem to be the case.
Does anyone know how activity_drs gets defined?

@taylor13 Sorry for the delayed answers.

For CMIP6, will the following solution work for ES-DOC, CMOR, ESGF, data citation services, the data request, the CV's etc. (at least going forward)?

For any given experiment, the ESGF publisher (perhaps relying on PrePARE?) should check that the correct primary activity (defined to be the activity that is listed first in the experiment_id CV under "activity_id") has been recorded by the data provider in the file's activity_id. If it doesn't, we could

reject the file as being non-compliant (my choice), or
accept the file, but in the ESGF catalog enter a corrected the activity_id and activity_drs, consistent with the experiment_id CV.

My choice is the same: reject.

If the activity_id contains entries that are ordered incorrectly, but otherwise consistent with the list in the experiment_id CV, we could

accept the file, but in the ESGF catalog, set activity_drs to the first activity listed for the experiment in the experiment_id CV, ignoring the ordering that the activities appear in the activity_id global attribute (my choice), or
reject the file.
Both are fine but for the accept option we need to ensure that the activity_drs in instance_id and master_id are also corrected  in the ESGF index.

Regarding the relation of ESGF Publisher and citation: The ESGF Publisher constructs the citation_url based on the instance_id, e.g.:
instance_id = CMIP6.RFMIP.MPI-M.MPI-ESM1-2-LR.piClim-control.r1i1p1f1.Amon.hur.gn.v20190710
citation_url = http://cera-www.dkrz.de/WDCC/meta/CMIP6/CMIP6.RFMIP.MPI-M.MPI-ESM1-2-LR.piClim-control.r1i1p1f1.Amon.hur.gn.v20190710.json
The citation is independent of the ESGF Publisher. Thus, the correctness of the connection between data and data citation relies on DRS (instance_id and that all its components are registered for this source_id in the CV).

There is a plan for a ScenarioMIP community paper which includes analysis of the ssp370 experiment with data of modeling groups that actually do not "want" to participate in ScenarioMIP but AerChemMIP. Related to this, it seems to be important to understand what activity_participation means.
@matthew-mizielinski
Should UKESM1-0-LL be for example excluded from RFMIP analysis? In that case i.e. if modeling groups did explicitly not want to be part of other MIP analysis/papers, it would be clear to me why the data should not get citation for ScenarioMIP.
I for myself have had another definition of that term activity_participation

@matthew-mizielinski
Should UKESM1-0-LL be for example excluded from RFMIP analysis? In that case i.e. if modeling groups did explicitly not want to be part of other MIP analysis/papers, it would be clear to me why the data should not get citation for ScenarioMIP.

My take on it is that anyone is welcome to analyse any experiment, but that the RFMIP community are principally interested in physical rather than earth system models, so our RFMIP leads are generally only running experiments to support their core analyses (i.e. RFMIP). The UKESM1 simulations submitted for RFMIP were run by our AerChemMIP group to support AerChemMIP activities.
Looking back on the MIPs structure I suspect we will need to think again about shared experiments for CMIP7 (perhaps we should create SharedMIP) and consider the purpose of the activity_participation field here (splitting it into something that means intending to submit and something that means involved in analyses.

I agree we should review the treatment of activity_id for CMIP7.  For CMIP6,

a single activity owns an experiment (and has responsibility for it).  That activity has usually defined the experiment conditions, and for each experiment in the experiment_id CV, this primary activity is listed first.  It should also appear first in the  global attribute, activity_id.
a second activity may have contributed to an experiment's design and that activity may in fact require the experiment results for its analysis.  That activity should be listed (second) in the activity_id global attribute.  In contrast and as an exception), although the deck and historical experiments may be essential to many of the activities, onlyCMIP appears under activity_id.

The activity_participation descriptors in the source_id CV were meant to be indications of interest, not as a way of constructing DRS identifiers for CMIP6.  It makes some sense for @MartinaSt to construct potential citation references based on the activity_id information in the CV, but it is a mistake to expect that the various ESGF identifiers (e.g., instance_id, master_id) will include the activity_id of interest to a modeling group.  The primary activity_id is uniquely determined by the experiment_id, as specified in the the experiment_id CV, and this is the i.d., that should appear in instance_id, master_id, etc.
So I think it is o.k. that @MartinaSt has generated her potential citations using activity_participation, but in communicating with the rest of the infrastructure (mainly the publisher), the primary activity_id should be used, not a group's intention of participating in a MIP.
As someone noted already, activity_id is not needed to uniquely identify a dataset (and it doesn't appear in filenames), so software could ignore activity_id as specified in files and catalogs and indexes, and simply determine the activity_id used by ESGF in creating instance_id, master_id, etc. from the experiment_id CV (i.e., given an experiment_id, simply extract the first item appearing in activity_id in the CMIP6_experiment_id.json dictionary).

Some input was provide offline, which I partially copy here so that it doesn't get forgotten:

The HAMMOZ-Consortium has reconfirmed that they want to publish the experiment "ssp370" as contribution to "AerChemMIP" (see https://github.com/WCRP-CMIP/CMIP6_CVs/issues/937). According to the Global Attributes Paper, the primary and DRS-relevant activity is specified first in the global attribute activity_id="AerChemMIP ScenarioMIP".
Currently, PrePare prevents this data from being published in the ESGF.
As it is the wish of the data provider to publish their ssp370 data as part of AerChemMIP, a change of PrePare is required. Any work-around solution outside of PrePare will potentially cause problems in other infrastructure components and on the long-term.
We need a short-term solution for the HAMMOZ-Consortium.
Do you see anything, which speaks against this PrePare change?

This is a response to the above (https://github.com/WCRP-CMIP/CMIP6_CVs/issues/951#issuecomment-672393353):

Is the issue the PrePARE software itself (some intrinsic logic), or how the metadata is structured in the CV?  Meaning the CV could be updated to accommodate the change (whether of course this is an acceptable change)?
Did you capture the specific error message generated by PrePARE?

And a subsequent response:

Right now, the order of the values of activities for experiments with co-sponsoring MIPs is strict in PrePARE/CMOR and must be followed, e.g.: https://github.com/PCMDI/cmip6-cmor-table/blob/c1810416ec67f48dca454b9ea69c5c8dbab856a9/Tables/CMIP6_CV.json#L9567  This is, as far as we can see, not part of the CMIP6 document: https://goo.gl/v1drZl    The note for handling multiple acitvity_ids does not strictly determine that there is one primary activity for experiments which cannot be changed.
The value for the activity within the DRS is used to link the data to the citation entry.
Since the first value of the activity_id from the CMOR/PrePARE CV is used to create the DRS, it is not possible to link a citation entry for a MIP listed as "secondary" in the CV with corresponding data because the data is saved under the activity listed first.
We propose that we allow to change the order of the values for activity_id when more than one MIP is defined for any experiment in the CV. This results in data for one experiment being in multiple MIP directories however it better aligns with the wishes and intentions of the data providers.

Is the issue the PrePARE software itself (some intrinsic logic), or

We could change the activity_id values in the CV of CMIP6-CMOR-tables to be a regex that allows both orders and we may need to tell CMOR/PrePARE to interprete it as a regex.

I want to correct what I said in https://github.com/WCRP-CMIP/CMIP6_CVs/issues/951#issuecomment-658480977.  It turns out that for any experiment, PrePARE will prevent publication of data if the activity_id is different from what is specified in the experiment_id CV.  When multiple activities appear, they must be in the correct order.  Here is a summary of the tests performed by Chris (@mauzey1):
 I downloaded some of the files from the ssp370 experiment to test them with PrePARE.

I tested a file that had the exact activity_id with PrePARE, and it passed.  I created a new version of this file with the activity_id reversed, which failed when processed by PrePARE.

C Traceback:
! In function: _CV_checkExperiment
!

!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Error: Your input attribute "activity_id" with value
! "AerChemMIP ScenarioMIP" needs to be replaced with value "ScenarioMIP AerChemMIP"
! as defined for experiment_id "ssp370".
!
!  See Control Vocabulary JSON file.(Tables/CMIP6_CV.json)
!
!
!!!!!!!!!!!!!!!!!!!!!!!!!

└──> :: CV FAIL    :: /Users/mauzey1/Desktop/github/cmor/ssp370_test/reversed_activity_ids/emibc_AERmon_CanESM5_ssp370_r15i1p1f1_gn_201501-210012.nc

I then tried another experiment_id=”ssp370” file with activity_id=”ScenarioMIP", which also failed when processed by PrePARE.

C Traceback:
! In function: _CV_checkExperiment
!

!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Error: Your input attribute "activity_id" with value
! "ScenarioMIP" needs to be replaced with value "ScenarioMIP AerChemMIP"
! as defined for experiment_id "ssp370".
!
!  See Control Vocabulary JSON file.(Tables/CMIP6_CV.json)
!
!
!!!!!!!!!!!!!!!!!!!!!!!!!

└──> :: CV FAIL    :: /Users/mauzey1/Desktop/github/cmor/ssp370_test/emibc_AERmon_CESM2-WACCM_ssp370_r1i1p1f1_gn_206501-210012.nc

Yes, both examples will generate an error when processed by PrePARE.

@taylor13 @mauzey1 I believe this is the correct behavior, so there is nothing to fix on the CMOR/PrePARE end

I agree.  Will close.

This thread is relevant for documenting the CMIP6 status, and consideration of changes for CMIP6+:
From: Taylor, Karl E.
Sent: 27 August 2020 05:50
To: david.neubauer
Cc: WIP List; Martin Schupfner; Fabian Wachsmann
Subject: publishing ssp370 experiments on ESGF and linking to citations

Dear David,

As co-chair of the WGCM Infrastructure Panel (WIP) supporting CMIP6, I am writing to seek clarification of your reluctance to publish the ssp370 experiment  with identifying descriptors that indicate that the ScenarioMIP activity (MIP) specified the experiment design.   First I want to provide a bit of background, and then I have a couple of specific questions.

There is likely some misunderstanding of the purpose of the data set identifiers used throughout the CMIP6 infrastructure.  These identifiers (based on what is known as the Data Reference Syntax, DRS) are unique collections of descriptors that include the name of the model (i.e. “source”), the institution responsible for running the experiments, the name of the experiment and the activity that has primary responsibility for that experiment.  Driven by scientific investigation, each CMIP activity has designed experiments to meet their research goals, and they have described these in a GMD CMIP paper.  Except for the DECK and historical runs, the experiments designed by an activity will usually be of little interest to other MIPs.  In CMIP6 there are fewer than a dozen cases where a second MIP has enough interest in an experiment that they request that the modeling groups participating in their MIP run that other MIP’s experiment, regardless of their intent to participate in that activity.  In these cases, the activity_id contains the names of both MIPs, with the “owner” of the experiment listed first.  An example of dual interest is experiment ssp370, which was designed by ScenarioMIP (the “owner”) but is also of interest to AerChemMIP.    By including the names of both MIPs in the activity_id, users accessing CMIP data through the ESGF portals can search either for experiments in the ScenarioMIP activity or the AerChemMIP activity, and they will find all the ssp370 simulations.  [They could just as well simply search directly for all the ssp370 experiments and get the same collection.]

The unique identifiers of experiments, which are used in various forms across the entire CMIP6 infrastructure, are simply descriptions of who designed the experiment, what the experiment is, who performed the experiment and what model they used.  These identifiers are not meant to imply anything about a modeling group’s interest in a particular activity.  For example, “CMIP6.ScenarioMIP.BCC.BCC-CSM2-MR.ssp370” indicates that under the CMIP6 umbrella, the ScenarioMIP-designed experiment named ssp370 was performed by the BCC institution using the BCC-CSM2-MR model.  This says nothing about BCC’s interest or lack there-of in ScenarioMIP.

The citation reads:
Xin, Xiaoge; Wu, Tongwen; Shi, Xueli; Zhang, Fang; Li, Jianglong; Chu, Min; Liu, Qianxia; Yan, Jinghui; Ma, Qiang; Wei, Min (2019). BCC BCC-CSM2MR model output prepared for CMIP6 ScenarioMIP ssp370. Version YYYYMMDD[1].Earth System Grid Federation. https://doi.org/10.22033/ESGF/CMIP6.3035

This accurately reflects the information provided by the data set identifier. “CMIP6 ScenarioMIP ssp370” should be considered the full name of the experiment (i.e. its extended name, including both CMIP6 and ScenarioMIP).  “Prepared for CMIP6 ScenarioMIP ssp370” is meant to be an abbreviation of “produced by a simulation conforming to the CMIP6 ScenarioMIP ssp370 experiment design.”    It does not imply that the modeling group actually participated in ScenarioMIP.

Another aspect of CMIP6 that is also at first difficult to understand is what is meant by the “activity_participation” descriptor that is created when a model is registered.  In registering a model, a group indicates its potential interest in participating in a number of MIPs.  Subsequently, they may add additional names to the list if they decide to participate in some other MIPs.  Sometimes a group’s intention to participate in a particular MIP isn’t realized.  Thus, at any point in time the activity_participation list cannot serve as a commitment or definitive interest in the MIPs listed there.

The citation service must not slow down the publication process, so the citations for all potential datasets expected in the CMIP6 archive must be created prior to publication of the dataset.  The citation service creates its potential list of citations based on what modeling groups have recorded in its “activity_participation” list.   [It would also not be practical to produce the list assuming all models will perform all experiments.]   If data from an experiment is published but the modeling group has not included in its activity_participation list the activity that “owns” that experiment, then the dataset will not be citable.  This means that modeling groups must include in their activity_participation list all activities that are responsible for the experiments they have performed.  In the case of ssp370, a modeling group must include ScenarioMIP in its activity_participation list.  When this system of creating citations was originally set up, we did not anticipate that there would be more than one activity asking its participants to perform a particular experiment.  If we had, we would have probably changed the name of the descriptor from “activity_participation” to something like “activities_with_primary_responsibility_for_the_experiments_expected_to_be_performed_with_this_model”.   I would note that very few users or contributors to CMIP are even aware of the activity_participation lists, so there should be very little concern that by including ScenarioMIP in this list, anyone might assume the modeling group will actually participate in ScenarioMIP.   It is essential, however, to include ScenarioMIP if you want to publish one of ScenarioMIP’s experiments: for example, ssp370.  If you do not include the activity, then there will be no proper citation or DOI created.

With the above background (apologies for the length, but CMIP6 is complex and involves lots of infrastructure parts that must communicate efficiently), I have a couple of questions.

1.	Would you still be reluctant to publish your ssp370 datasets and have them identified as being run in conformance with the ScenarioMIP  design?
2.	What would you recommend we do differently for CMIP7?  (in that eventuality).

Any other thoughts/comments/suggestions would be most welcome.

Best regards,
Karl

This is the view of the group who published ssp370 under AerChemMIP.
Subject: 	Re: Questions about BCC ssp370 simulations
Date: 	Fri, 28 Aug 2020 10:09:00 +0800 (GMT+08:00)
From: 	Dr. Jin Ba<EMAIL_ADDRESS>To: 	Karl E<EMAIL_ADDRESS>CC<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
Dear Karl,

I am the liaison for DOC. About your questions I asked my colleagues Jinghui and Xueli.

The answers are as below,

1. Yes, the documentations were not clear about this. Because the ssp370 experiment was listed in both ScenarioMIP(O'Neill et al., GMD2016) and AerChemMIP (Table 4 in Collins et al., GMD2016, attached). The ssp370 and ssp370-lowNTCF was comapred in AerChemMIP in Table 4, and stated "ssp370 is also specified as Tier 1 in ScenarioMIP", which was very easy to misunderstand.

Additionally, in the CMIP6_CV.json of the Climate Model Output Rewriter (CMOR) system, the activity_ids of ssp370 include ScenarioMIP and AerChemMIP, which might mean that each of the activity is acceptable, therefore we've finished the datasets under AerChemMIP in 2019.

Besides, also as stated in the CMIP6_CV.json, BCC-CSM2-MR is the major model version participating ScenarioMIP, while the activity_participation of BCC-ESM1 is AerChemMIP and CMIP.  Therefore we put the BCC-ESM1 simulated ssp370 under the AerChemMIP.

2. As response in question 1, we know that the ssp370 was one of the experiments in ScenarioMIP. Actually, we've published the datasets of another model version, i.e., BCC-CSM2-MR for different scenarios (ssp126,ssp245,ssp370 and ssp585). But there is only one scenario experiment conducted with BCC-ESM1, and several comparison experiments ( ssp370-lowNTCF, ssp370SST,ssp370SST-lowNTCF) are under AerChemMIP activity. Therefore it should be more convenient to put the ssp370 data under the same MIP.

We wouldn't object to publish the BCC-ESM1 ssp370 data under ScenarioMIP, would we?  If you think it is really necessary, we could modify the user-input information and processing again for the experiment.

3. No, we haven't done anything special to CMORing and publish the data.  Because the data was processing in 2019, we need to check the related information of publishing data.

4. The prePARE was not updated in order to ensure the data release.

Hopefully these answers are clear for you.

Best regards,

Jin

    -----原始邮件-----
    发件人:"Taylor, Karl E."<EMAIL_ADDRESS>    发送时间:2020-08-27 04:09:10 (星期四)
    收件人<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>    抄送: "WIP List"<EMAIL_ADDRESS>    主题: Questions about BCC ssp370 simulations

    Dear Jin, Tongwen, and Jinghui,

    I’m writing on behalf of the WGCM Infrastructure Panel (WIP).  We seek your input on one of the confusing aspects of CMIP6 metadata, specifically assignment of the activity_id.  Except for the DECK and historical runs, most CMIP6 experiments are of direct interest to a single activity.  There are a few experiments, however, that are of interest to multiple activities.  One example is the ScenarioMIP ssp370 experiment, which is also of interest to the AerChemMIP activity.  When you published your BCC-ESM1 ssp370 output on ESGF, you identified it with the AerChemMIP activity, rather than the ScenarioMIP activity, which has primary responsibility for this experiment and is therefore considered its “owner”.    In contrast, when you published your BCC-CSM2-MR ssp370 output, you identified it with the ScenarioMIP activity.   Recent versions of the ESGF publisher that use PrePARE to check the metadata will not permit ssp370 data to be published under the AerChemMIP activity.

    So we want to understand what led to your choosing to publish the BCC-ESM1 ssp370 output under the AerChemMIP activity, rather than the ScenarioMIP activity. We do not want you to retract or republish this data because that would be confusing to those who have already accessed it and would involve work from lots of people, including your scientists and those of us supporting CMIP6 infrastructure.   What we would like to find out though is:

        When you published the ssp370 data, why did you choose to publish under the AerChemMIP rather than ScenarioMIP?  Although our documentation was not explicit nor clear about this, ssp370 is supposed to be invariably published under the ScenarioMIP activity.  Were you aware of this?  I’m guessing not.
        Had you been aware that ScenarioMIP is considered the owner of ssp370, would you have objected to publishing your BCC-ESM1 data under that activity_id?
        When you specified AerChemMIP as the activity for ssp370, did you have to do anything special to successfully publish?  Do you recall any errors raised in your first attempts?

    Thanks for all your help with this.  We hope to improve the clarity of our documentation and consider for CMIP7 alternative ways of handling experiments of interest to multiple activities.

    best regards,

    Karl on behalf of the WIP.

--

Dr. Jin Ba

Division of Climate System Modelling

National Climate Center

China Meteorological Administration

No.46 Zhongguancun Nandajie, Haidian

100081, Beijing, China

Tel. +86 10 6840 8605

Email<EMAIL_ADDRESS>

Table 4.png

@MartinaSt thanks for that, I have just edited the above to remove complete email addresses, just so we're not the source of complete email addresses for spambots

@durack1 Sure, thanks!

And here is a further correspondence from David Neubauer, responding to https://github.com/WCRP-CMIP/CMIP6_CVs/issues/951#issuecomment-682099506:
Dear Karl,

sorry for my late response. Thank you for the clarifications. I can only provide my personal perspective as a
 member of a modelling group taking part in CMIP6.

It's good that all MIPs are included in the activity_id and that an experiment (e.g. ssp370) can be found
 under any activity_id that requests the experiment.

For AerChemMIP ssp370 and variants e.g. ssp370-lowNTCF, ssp370SST have to be performed. The time
 period requested for ssp370 differs between ScenarioMIP (2015-2100) and AerChemMIP (2015-2055).
Also the diagnostics requested by AerChemMIP and ScenarioMIP differ. Therefore I would find it more
intuitive for data users if the ssp370-experiment would be published under AerChemMIP. If CMIP rather
wants to MIP designing the experiment in the filename and citation then that's fine for me too. My primary
interest is that our data is published and findable. And that there is a clear process how data needs to be
CMORized and uploaded.

In answer to your questions:
1. We are a small modelling group and already had to re-process and re-publish ssp370 data because of this
inconsistency in id's. We followed the instructions that where given to us. We don't have the resources to
re-publish the data again.
2. If a modelling group registers for a MIP it should automatically be registered for all MIPS that designed
experiments for this MIP. E.g. in CMIP6 registration for AerChemMIP would automatically lead to registration
for ScenarioMIP and RFMIP. Or the modelling groups need to be made aware of that a registration for the
MIPS designing an experiment is required. In my understanding this was not the case for CMIP6 and led to
the issue we have had. It would be great if also smaller modelling groups can take part in CMIP, that don't
have the resources to take part in all the MIPs, without unnecessary struggles.

Best regards,
David

Reviewing the current ESGF archive, I found only a few cases where models had incorrectly published data when 2 activities were supposed to be listed as the activity_id:

For the ssp370 experiment, the BCC-ESM1 model omitted the primary activity, ScenarioMIP, and incorrectly published under the secondary activity, AerChemMIP.  Their data is citable under AerChemMIP.

for the ssp370 experiment, the MPI-ESM-1-2-HAM model republished their data correctly with activity_id="ScenarioMIP AerChemMIP", but the ESGF search interface link pointing to the citation information is broken.  I think the reference, however, is found is the citation database by replacing in the citation_url "ScenarioMIP" with "AerChemMIP".

for the esm-1pct-brch-1000PgC experiment, the CESM2 model omitted the secondary activity, CDRMIP.   Their data is citable under C4MIP.

for the land-hist experiment, the BCC-CSM2-MR model and the GFDL-ESM4 model omitted the primary activity, LS3MIP, and published incorrectly under the secondary activity, LUMIP.  Their data is citable under LUMIP.

for the land-hist experiment, the CESM2 model omitted the secondary activity.  Their data is citable under LS3MIP.

I didn't check whether the activity_ids were listed in the correct order when both were  listed in activity_ids (and so I didn't check whether the correct activity was used in generating the DRS i.d.'s.).

I suggest that nothing be done about the current deviations from the data requirements described in  https://github.com/WCRP-CMIP/CMIP6_CVs/issues/951#issuecomment-701711008 .  In all cases the data can be found on ESGF under at least one of the activity_id's associated with an experiment; if a user enters the name of the experiment (without specifying activity_id), all available output will be listed.  With one exception all citation_url's can be successfully followed to the citation service, although in a few cases the citation will indicate the wrong activity (i.e., the secondary activity) is responsible for the experiment.  I don't think this is a big deal.
The one exception to the above is that for the ssp370 experiment, the MPI-ESM-1-2-HAM model data is now correctly republished with activity_id="ScenarioMIP AerChemMIP", but the citation_url recorded in the ESGF catalog points to a non-existent reference.  To get to a citable reference to this data a user must replace "ScenarioMIP" with "AerChemMIP" in the citation_url.
I don't think it worth the effort to correct this small blemish.  If we adopt this course, HAMMOZ won't have to republish their data (David's main concern expressed in https://github.com/WCRP-CMIP/CMIP6_CVs/issues/951#issuecomment-701673162 above) and no one involved in the infrastructure will have to do anything.
We should, however, make sure that in the future everyone includes PrePARE checks as part of their publication procedure and all groups include all activities in their activity_participation list in the source_id registration.  How can we make sure that's the case?

@taylor13 The citation_url published in the ESGF index is a tiny part of the citation service and thus cannot be discussed individually but requires the discussion of the citation service for this specific case of the Hammoz data, where it is broken. We have no way to mend this broken citation just two options how to deal with it and how to document it. As there are no other cases among the CMIP6 data where the citation is broken, I am interested also in David's opinion.

I would favor option 2.  Under the downside you mentioned ssp370 data can't be specifically cited for this model.  However, to give credit to HAMMOZ in a publication, couldn't an author include the following reference available from the citation service?  The DOI is a valid one.

With regard to DKRZ possibly assisting HAMMOZ in reprocessing and republishing the data, I don't recall why that is needed?  The ssp370 data has already be published with the correct global attribute activity_id = "ScenarioMIP AerChemMIP" and all the dataset identifiers are correct on ESGF.

Thank you for your help @MartinaSt and @taylor13. What @taylor13 suggests seems reasonable. A note on the AerChemMIP landing-page to explain that counter-intuitively one experiment namely ssp370 is not included and has to be cited as:  https://doi.org/10.22033/ESGF/CMIP6.1621 would be very helpful.
So far this only concerns the ssp370 data. But we will publish also data for the piClim-aer and piClim-control experiments, which are hosted by RFMIP and co-sponsored by AerChemMIP (see the comment by @durack1 above).
What is your recommendation to publish these two experiments? As I understand we should register for RFMIP. But will it be possible to get a citation for these two experiments?

Thanks @taylor13 and @nedavid for your feedback.
If we go with option 2, no registration for ScenarioMIP nor RFMIP is required. The piClim-aer and piClim-control experiments will add another two exceptions with broken data citations, which will be not citable. Sorry, David, this is the same case as ssp370.
@taylor13 I have mentioned the support that DKRZ could provide for re-processing and re-publication of the data to make clear that the resource constraint mentioned by David is not an obstacle to the solution, which Michael favored in his email from 08.09.2020:

... But I think my position to open the ESGF publication restrictions is clear from the related email discussion. Modelling groups which only participate for an experiment in the co-sponsoring MIP should have the opportunity to publish and to cite in this MIP.

So, for completeness of all possible options, there is the option to break the strict co-sponsoring rule and republish ssp370 under AerChemMIP. DKRZ could do the required data processing. This option would not break the data citation (all experiments and MIPs provided are citable in an intuitive way), and this would also make transparent to data users that the ssp370 data from Hammoz was created with AerChemMIP diagnostics. Severe problems in other infrastructure parts like the data replication have not been raised, so far, at least I am not aware of such a problem.
We all have spent quite some time on this issue already, so we should come to a final decision on Tuesday between the possibilities:

keep the strict co-sponsoring rule and deal with the broken data citations by leaving the co-sponsored experiments without data references or non-citable.
allow for the publication of the three co-sponsored experiments ssp370, piClim-aer and piClim-control under AerChemMIP for the Hammoz-Consortium; DKRZ will do the data re-processing for Hammoz in this case.

As Michael wrote: We at DKRZ shall accept the WIP decision.

@taylor13 I am a little confused if there is something to be done here in the CMIP6_CVs, or whether this has been reopened so the citation tweaks can be completed, and once this is done, closed?

I reopened this so that we could capture some of the discussion that affects what activities should be recorded in source_id CV.  There is no need for any changes to the CV processing, although I think HAMMOZ may need to add and activity or two to their activity_participation list.

@taylor13 @durack1 No, there is no need to register additional activity_participations for MIPs, in which HAMMOZ does not participate, for both remaining possibilities:

keep the strict co-sponsoring rule and deal with the broken data citations by leaving the co-sponsored experiments ssp370, piClim-aer and piClim-control without data references or non-citable.
allow for the publication of the three co-sponsored experiments ssp370, piClim-aer and piClim-control under AerChemMIP for the Hammoz-Consortium; DKRZ will do the data re-processing for Hammoz in this case.

The WIP provided its final guidance on this in a recent email to @wachsylon and @nedavid:

Your ScenarioMIP experiments should remain unmodified in the CMIP6 archive and will remain uncitable.
Your forthcoming piClim-aer and piClim-control experiments should be published and will be citable following the current CMIP6 approach, which requires
a) assigning activity_id = "RFMIP AerChemMIP" to the global attribute in each of your output files.
b) adopting the recommended directory structure for your output files, which uses "RFMIP" rather than "AerChemMIP" as the activity_id sub-directory.  (This may not be absolutely necessary, but it is safer, more convenient, and consistent with what most other modeling groups have done, so we urge you to adopt that structure..)
Your ScenarioMIP experiments should remain unmodified in the CMIP6 archive and will remain uncitable.
Your forthcoming piClim-aer and piClim-control experiments should be published and will be citable following the current CMIP6 approach, which requires
a) assigning activity_id = "RFMIP AerChemMIP" to the global attribute in each of your output files.
b) adopting the recommended directory structure for your output files, which uses "RFMIP" rather than "AerChemMIP" as the activity_id sub-directory.  (This may not be absolutely necessary, but it is safer, more convenient, and consistent with what most other modeling groups have done, so we urge you to adopt that structure..)

I'll close this issue now.


Not possible to specify shadow branches at SCM

I've started to use your multibranch plugin for feature branch testing. Currently, PR shadow branches should be tested the same way as usual heads. Basic Git SCM allows to specify refspec i.e like following
Remote name: origin
Refspec: +refs/*:refs/remotes/origin/*
Branch specifier: :.*pull-requests/.*/merge
And PR shadow branch will be used.
Currently even * branch specification sees only master and production branches (so, all the general ref/heads/* branches).

I'd also like PRs to be included. Im currently having to use a separate job to watch for PRs and it would be awesome to be able to do both in a single job.

SCM options, including those on the multi-branch project configuration page, are implemented by the SCM plugins themselves.

Well, but SCM on the multibranch projects differs with one on the freeconfig project.

vs.

If we have the save grade config for multibranch config - that would be nice. at least to solve this issue
Branches to build and Refspec fields should replace given include branches and exclude branches options

Sure, they look different, but this plugin still uses APIs that are provided by other plugins.  There is no code in this plugin for any SCM configuration settings.

Yeah, I look through the sources and see that. Thank you for clarifying, I'll try to do something with it


Be able to configure the player controls

Some people use a Dvorak keyboard and it can be uncomfortable for some people.

#45 Sorry.


Update log4j2 to 2.17.1

They fixed another critical security vulnerability.

This only affects if you allow an attacker to control your log4j configuration so priority is low for this.


rNA report VOOM error with test data

[Wed Jan  3 11:39:49 2024]
rule rna_report:
    input: /data/Ziegelbauer_lab/circRNADetection/tmp_kopardevn/RENEE_testing_240103/DEG_ALL/RSEM.genes.expected_counts.all_samples.reformatted.tsv, /data/Zieg
elbauer_lab/circRNADetection/tmp_kopardevn/RENEE_testing_240103/DEG_ALL/combined_TIN.tsv, /data/Ziegelbauer_lab/circRNADetection/tmp_kopardevn/RENEE_testing_24
0103/Reports/multiqc_matrix.tsv
    output: /data/Ziegelbauer_lab/circRNADetection/tmp_kopardevn/RENEE_testing_240103/Reports/RNA_Report.html
    jobid: 0
    reason: Missing output files: /data/Ziegelbauer_lab/circRNADetection/tmp_kopardevn/RENEE_testing_240103/Reports/RNA_Report.html
    resources: mem_mb=1000, mem_mib=954, disk_mb=1000, disk_mib=954, tmpdir=/tmp

    # Avoids inheriting $R_LIBS_SITE
    # from local env variables
    R_LIBS_SITE=/usr/local/lib/R/site-library
    # Generate RNA QC Dashboard
    workflow/scripts/rNA.R         -m workflow/scripts/rNA_flowcells.Rmd         -r /data/Ziegelbauer_lab/circRNADetection/tmp_kopardevn/RENEE_testing_240103/D
EG_ALL/RSEM.genes.expected_counts.all_samples.reformatted.tsv         -t /data/Ziegelbauer_lab/circRNADetection/tmp_kopardevn/RENEE_testing_240103/DEG_ALL/comb
ined_TIN.tsv         -q /data/Ziegelbauer_lab/circRNADetection/tmp_kopardevn/RENEE_testing_240103/Reports/multiqc_matrix.tsv         -o /data/Ziegelbauer_lab/c
ircRNADetection/tmp_kopardevn/RENEE_testing_240103/Reports         -f RNA_Report.html

Activating singularity image /data/CCBR_Pipeliner/SIFS/ccbr_rna_v0.0.1.sif

processing file: rNA_flowcells.Rmd
Quitting from lines 31-88 (rNA_flowcells.Rmd)
Error in voom(deg, normalize = "quantile", plot = TRUE, save.plot = TRUE) :
  Need at least two genes to fit a mean-variance trend
Calls: <Anonymous> ... withCallingHandlers -> withVisible -> eval -> eval -> voom
In addition: Warning message:
In filterByExpr.DGEList(deg) :
  All samples appear to belong to the same group.
Execution halted
[Wed Jan  3 11:39:57 2024]
Error in rule rna_report:

I opened this up in an interactive RStudio session on biowulf using the dataset in /scratch/RENEE_testing_240103. I see that only one gene passes the filterByExpr function, causing this error in voom. However, on a different run of the test data I get 6 genes passing the filter. So it seems for some reason RSEM is giving different expected counts on the same dataset.
works - RENEE git commit hash 6987849
rawcounts <- read.table(file = '/data/sovacoolkl/renee_test/DEG_ALL/RSEM.genes.expected_counts.all_samples.reformatted.tsv', sep = '\t', header = TRUE, row.names = 1, quote = "")
deg <- edgeR::DGEList(counts = rawcounts)
keep_genes <- edgeR::filterByExpr(deg)
keep_genes %>% Filter(function(x) isTRUE(x),.) %>% dput()

c(`ENSG00000185658.13|BRWD1` = TRUE, `ENSG00000157540.21|DYRK1A` = TRUE,
`ENSG00000160294.11|MCM3AP` = TRUE, `ENSG00000141959.17|PFKL` = TRUE,
`ENSG00000182670.13|TTC3` = TRUE, `ENSG00000160201.11|U2AF1` = TRUE
)

fails - RENEE git commit hash 3c747aa
rawcounts <- read.table(file = '/data/sovacoolkl/RENEE_testing_240103/DEG_ALL/RSEM.genes.expected_counts.all_samples.reformatted.tsv', sep = '\t', header = TRUE, row.names = 1, quote = "")
deg <- edgeR::DGEList(counts = rawcounts)
keep_genes <- edgeR::filterByExpr(deg)
keep_genes %>% Filter(function(x) isTRUE(x),.) %>% dput()

c(`ENSMUSG00000106106.2|CT010467.1` = TRUE)

I used this to launch a test run with v2.5.10:
module load ccbrpipeliner
renee run --input /data/CCBR_Pipeliner/Pipelines/RENEE/develop/.tests/*.R1.fastq.gz --output /data/sovacoolkl/renee_test_v2.5.10 --genome hg38_30 --mode slurm --sif-cache /data/CCBR_Pipeliner/SIFS

@kopardev is launching another test run with --genome hg38_30, he previously used the default (mouse) genome. This is almost certainly the cause of the issue, but let's wait to close it until Vishal's new test run succeeds.

@kopardev did your new test run succeed?

Vishal's test succeeded


int4 x int4 matmul from StableHLO causes illegal 'memref.subview'

What happened?
Compiling the following IR from JAX for a fully quantized matmul causes the error below to be thrown:
module @jit_quantized_matmul attributes {mhlo.num_partitions = 1 : i32, mhlo.num_replicas = 1 : i32} {
  func.func public @main(%arg0: tensor<128x256xi4> {mhlo.sharding = "{replicated}"}, %arg1: tensor<256x512xi4> {mhlo.sharding = "{replicated}"}) -> (tensor<128x512xi32> {jax.result_info = ""}) {
    %0 = stablehlo.dot_general %arg0, %arg1, contracting_dims = [1] x [0], precision = [DEFAULT, DEFAULT] : (tensor<128x256xi4>, tensor<256x512xi4>) -> tensor<128x512xi32>
    return %0 : tensor<128x512xi32>
  }
}

iree-compile --iree-hal-target-backends=llvm-cpu --iree-input-type=stablehlo /tmp/int4_int4_matmul.mlir -o /tmp/int4_int4_matmul.vmfb
/tmp/int4_int4_matmul.mlir:3:10: error: failed to legalize operation 'memref.subview' that was explicitly marked illegal
    %0 = stablehlo.dot_general %arg0, %arg1, contracting_dims = [1] x [0], precision = [DEFAULT, DEFAULT] : (tensor<128x256xi4>, tensor<256x512xi4>) -> tensor<128x512xi32>
         ^
/tmp/int4_int4_matmul.mlir:2:3: note: called from
  func.func public @main(%arg0: tensor<128x256xi4> {mhlo.sharding = "{replicated}"}, %arg1: tensor<256x512xi4> {mhlo.sharding = "{replicated}"}) -> (tensor<128x512xi32> {jax.result_info = ""}) {
  ^
/tmp/int4_int4_matmul.mlir:3:10: note: see current operation: %46 = "memref.subview"(%27, %45, %42) <{operandSegmentSizes = array<i32: 1, 2, 0, 0>, static_offsets = array<i64: -9223372036854775808, -9223372036854775808>, static_sizes = array<i64: 8, 16>, static_strides = array<i64: 1, 1>}> : (memref<128x256xi4>, index, index) -> memref<8x16xi4, strided<[256, 1], offset: ?>>
    %0 = stablehlo.dot_general %arg0, %arg1, contracting_dims = [1] x [0], precision = [DEFAULT, DEFAULT] : (tensor<128x256xi4>, tensor<256x512xi4>) -> tensor<128x512xi32>
         ^
/tmp/int4_int4_matmul.mlir:3:10: error: failed to run translation of source executable to target executable for backend #hal.executable.target<"llvm-cpu", "embedded-elf-x86_64", {cpu = "generic", cpu_features = "", data_layout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128", native_vector_size = 16 : index, target_triple = "x86_64-unknown-unknown-eabi-elf", ukernels = false}>
    %0 = stablehlo.dot_general %arg0, %arg1, contracting_dims = [1] x [0], precision = [DEFAULT, DEFAULT] : (tensor<128x256xi4>, tensor<256x512xi4>) -> tensor<128x512xi32>
         ^

Full error.
Steps to reproduce your issue
Tested compiling to the llvm-cpu backend.
What component(s) does this issue relate to?
Frontends, Compiler
Version information
2f72249fcbc310d41a179e76344f1a853e0547b9
Additional context
No response

cc @MaheshRavishankar
This fails in the EmulateNarrowTypePass.

I'm just going to create an auto responder which posts that on every issue, and I think it will be right a lot :)
Would be good to robustify this. It isn't clear to me if it just needs more work/tests or has design issues that are making it hard to be comprehensive.

Would be good to robustify this. It isn't clear to me if it just needs more work/tests or has design issues that are making it hard to be comprehensive.

I cannot really comment on the design aspect of it, I didn't follow how any of this was plumbed through, but I am surprised that we care about memref operations other than load and store and hence choke on them.
Aside from load/store, everything should be just pointer arithmetic (and make the type system happy), which IMHO should not need to be massaged. But I may be missing something.

I am not sure there are any design issues. We do care only about memref.load/memref.store. Will repro and look at it.

Hmmm... I cant repro this error on  ToT.
$ ./repro.sh
+ IREE_ROOT_DIR=/home/mahesh/iree
+ BUILD_TYPE=RelWithDebInfo
+ IREE_SOURCE_DIR=/home/mahesh/iree/iree
+ IREE_BUILD_DIR=/home/mahesh/iree/build/RelWithDebInfo
+ IREE_TOOLS_DIR=/home/mahesh/iree/build/RelWithDebInfo/tools
+ IREE_COMPILE=/home/mahesh/iree/build/RelWithDebInfo/tools/iree-compile
+ /home/mahesh/iree/build/RelWithDebInfo/tools/iree-compile --iree-hal-target-backends=llvm-cpu --iree-input-type=stablehlo -o output.vmfb repro.mlir
mahesh@mahesh-dev:~/iree/scratchspace/issues/14989$ ls
output.vmfb  repro.mlir  repro.sh  repro.sh~

Hmmm... I cant repro this error on ToT.
$ ./repro.sh
+ IREE_ROOT_DIR=/home/mahesh/iree
+ BUILD_TYPE=RelWithDebInfo
+ IREE_SOURCE_DIR=/home/mahesh/iree/iree
+ IREE_BUILD_DIR=/home/mahesh/iree/build/RelWithDebInfo
+ IREE_TOOLS_DIR=/home/mahesh/iree/build/RelWithDebInfo/tools
+ IREE_COMPILE=/home/mahesh/iree/build/RelWithDebInfo/tools/iree-compile
+ /home/mahesh/iree/build/RelWithDebInfo/tools/iree-compile --iree-hal-target-backends=llvm-cpu --iree-input-type=stablehlo -o output.vmfb repro.mlir
mahesh@mahesh-dev:~/iree/scratchspace/issues/14989$ ls
output.vmfb  repro.mlir  repro.sh  repro.sh~

This was incidentally fixed by https://github.com/openxla/iree/pull/14988.
If the failure is further down the stack and fixing it is still be relevant you can use 2f72249fcbc310d41a179e76344f1a853e0547b9 to reproduce it. Otherwise feel free to close this!

Also, would it make sense for me to add this as a test somewhere?

I think this was not relevant type emulation, the error was just caught there. I am closing this. Adding a test case would be nice, but it needs to be tracked down to the root issue (I am not looking into it).


Allow disabling of X-Axis calibration

Is your feature request related to a problem? Please describe.
By default, Carbon updates axis limits, either to accomodate out of range datapoints or to have nicer axis limits such as 10, 75 or 100 instead of 7, 68 or 90. The allowCalibration property can be used to prevent this behavior, however it is limited to the Y-Axis. This functionality needs to be extended to the X-axis so consumer-defined limits will not be automatically updated.
Describe the solution you'd like
Update the logic of when allowCalibration is set to false, so that both X and Y axis limits are not automatically calibrated.
Describe alternatives you've considered
There is no other convenient alternative for preventing calibration of the X-axis.
Additional context
N/A

This issue is resolved by:
https://github.com/cerner/terra-graphs/pull/285
https://github.com/cerner/terra-graphs/pull/285
https://github.com/cerner/terra-graphs/pull/285
https://github.com/cerner/terra-graphs/pull/285

This issue is resolved by:
https://github.com/cerner/terra-graphs/pull/285
https://github.com/cerner/terra-graphs/pull/286
https://github.com/cerner/terra-graphs/pull/287
https://github.com/cerner/terra-graphs/pull/288
https://github.com/cerner/terra-graphs/pull/289
https://github.com/cerner/terra-graphs/pull/290
https://github.com/cerner/terra-graphs/pull/291
https://github.com/cerner/terra-graphs/pull/292
https://github.com/cerner/terra-graphs/pull/293


Displaying multiple when a notification arrives

Hello , Displaying multiple when a notification arrives on IOS. My code is
`PushNotification.configure({
// (optional) Called when Token is generated (iOS and Android)
onRegister: function (token) {
//console.log("TOKEN:", token);
},
// (required) Called when a remote or local notification is opened or received
onNotification: function (notification) {
// process the notification

// required on iOS only (see fetchCompletionHandler docs: https://github.com/react-native-community/react-native-push-notification-ios)
if (!notification.foreground) {
 //notification.finish(PushNotificationIOS.FetchResult.NoData);
}
notification.finish(PushNotificationIOS.FetchResult.NoData);

},
// ANDROID ONLY: GCM or FCM Sender ID (product_number) (optional - not required for local notifications, but is need to receive remote push notifications)
senderID: "503601710565",
// IOS ONLY (optional): default: all - Permissions to register.
permissions: {
alert: true,
badge: true,
sound: true
},
// Should the initial notification be popped automatically
// default: true
popInitialNotification: true,
/**

(optional) default: true

Specified if permissions (ios) and token (android and ios) will requested or not,

if not, you must call PushNotificationsHandler.requestPermissions() later
*/
requestPermissions: true
});
`

ComponentDidMount:
` messaging().onMessage(notification => {
  console.log('2');
  PushNotification.localNotification({
    title:notification.notification.title,
    message:notification.notification.body, // (required),
  });
  console.log('Notification received!', notification);
});`

Fixed in v1.7.0
Notification with specific id should only display one at a time.

@Naturalclar  I also facing this issue and I'm using version 1.5.0
Once I build and run the app from the codebase it will not be duplicated
But once I publish to Test flight it will duplicate same message twice

I'm getting 3 notifications when I sent a sample notification from firebase console. Anybody fixed this issue ?


Shell Sort added

Related Issue
Info about the related issue
I have added the python program for shell sort algorithm
✅ DevIncept Participant
Contributor
Closes: #118
Describe the changes you've made
Type of change
Created separate files to and implemented the shell sort algorithm
What sort of change have you made:
I have added a new feature i.e the program for Shell Sort Algorithm.
How Has This Been Tested?
Describe how have you verified the changes made
The programs was run on VS Code multiple time and checked for issues and error.
Checklist:
✅ My code follows the guidelines of this project.
✅ I have performed a self-review of my own code.
✅ I have commented my code, particularly whereever it was hard to understand.
✅ I have made corresponding changes to the documentation.
✅ My changes generate no new warnings.

Your PR has conflicting files due to which I am unable to check on your PR. Kindly correct it from your end.  @neha030

I think it's resolved now @prathimacode-hub
Please review


Super Auditor

Based on: https://github.com/molnard/WalletWasabi/pull/47
https://github.com/zkSNACKs/CoinVerifier/issues/7
CSV file, from left to right

DateTime of creating the audit
Outpoint
Address
Did we ban it or not (boolean)
Did we remove it or not (boolean)
Amount
Reason (Remix, Whitelisted, OneHop, RemoteApiChecked, Inmature, Exception, NotChecked)
Details (in case of we did an API request: report_id, report_type, risk_ids, risk_categories)

Commit history looks bad. Opening new PR.

It just got even worse.

History is a mess. Here you can find the same but with clean history https://github.com/adamPetho/WalletWasabi/pull/8


add part3 to mkdocs

Coverage remained the same at 49.123% when pulling b878733097cf1f3c1b6fa549f96a452853226e84 on develop into 167f7f1b2181300b3febe529d78b1b87abdff777 on part3_model-and-database.


app works only with develop branch

After installing wiki to my site the default route was home when I click on see on website its show my homepage
then I change route to wiki/home now when I click on see on website i get this error:
Traceback (most recent call last):
  File "/home/frappe/frappe-bench/apps/frappe/frappe/website/render.py", line 49, in render
    data = render_page_by_language(path)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/website/render.py", line 178, in render_page_by_language
    return render_page(path)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/website/render.py", line 194, in render_page
    return build(path)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/website/render.py", line 201, in build
    return build_page(path)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/website/render.py", line 224, in build_page
    html = frappe.get_template(context.template).render(context)
  File "/home/frappe/frappe-bench/env/lib/python3.6/site-packages/jinja2/environment.py", line 1090, in render
    self.environment.handle_exception()
  File "/home/frappe/frappe-bench/env/lib/python3.6/site-packages/jinja2/environment.py", line 832, in handle_exception
    reraise(*rewrite_traceback_stack(source=source))
  File "/home/frappe/frappe-bench/env/lib/python3.6/site-packages/jinja2/_compat.py", line 28, in reraise
    raise value.with_traceback(tb)
  File "/home/frappe/frappe-bench/apps/wiki/wiki/./wiki/doctype/wiki_page/templates/wiki_page.html", line 1, in top-level template code
    {% extends "wiki/doctype/wiki_page/templates/wiki_doc.html" %}
  File "/home/frappe/frappe-bench/apps/wiki/wiki/./wiki/doctype/wiki_page/templates/wiki_doc.html", line 2, in top-level template code
    {%- from "templates/includes/navbar/navbar_items.html" import render_item -%}
  File "/home/frappe/frappe-bench/apps/frappe/frappe/./templates/base.html", line 83, in top-level template code
    {% block content %}
  File "/home/frappe/frappe-bench/apps/wiki/wiki/./wiki/doctype/wiki_page/templates/wiki_doc.html", line 77, in block "content"
    {% block page_container %}
  File "/home/frappe/frappe-bench/apps/wiki/wiki/./wiki/doctype/wiki_page/templates/wiki_doc.html", line 83, in block "page_container"
    {%- block page_content -%}{%- endblock -%}
  File "/home/frappe/frappe-bench/apps/wiki/wiki/./wiki/doctype/wiki_page/templates/wiki_page.html", line 19, in block "page_content"
    {% include "wiki/doctype/wiki_page/templates/show.html" %}
  File "/home/frappe/frappe-bench/apps/wiki/wiki/./wiki/doctype/wiki_page/templates/show.html", line 31, in top-level template code
    {{ include_script('wiki.bundle.js') }}
  File "/home/frappe/frappe-bench/env/lib/python3.6/site-packages/jinja2/sandbox.py", line 460, in call
    if not __self.is_safe_callable(__obj):
  File "/home/frappe/frappe-bench/env/lib/python3.6/site-packages/jinja2/sandbox.py", line 360, in is_safe_callable
    getattr(obj, "unsafe_callable", False) or getattr(obj, "alters_data", False)
jinja2.exceptions.UndefinedError: 'include_script' is undefined

same issue here!

Hi,
Can you please share your frappe, nodejs and python versions?

Hi @hasnain2808,
Node v14.17.4
Frappe 13.7.0
ERPNext 13.7.1

Hi @hasnain2808,
Node v14.9.0
Python  3.6.9
Frappe 13.8.0

Hey,
Was able to replicate the issue on version 13(13.8.0)
This is happening since the version13(13.8.0, 13.7.0...) branch still uses rollup, and frappe/wiki is already ported to esbuild constructs(include_script). You should not face these issues on develop. I am having talks with the other members to see if we can find a middle ground.

Hi @hasnain2808 , any news of this for Frappe? Thanks!

@federicocalvo
There are two options

Maintain a separate version of wiki for version13
Add the missing functions in frappe v13, add rollup based build in wiki. If we do this same version of wiki would work with both v13 and develop

@hasnain2808 hi! any news of this for the frappe team? thanks!

There were a bunch of new changes that were added to the develop branch, which are not backported
The Wiki uses some of these new features
So simply adding forward compatibility for esbuild in v13 would not be enough.

@hasnain2808 I need to use this can you please let me know what is a workaround? I'm running Frappe in production.

A workaround I personally did, works but not the most elegant: I created a new bench matching what the wiki app required and served it on a different port. (On the same server)

From: Osama Asif @.>
Sent: Wednesday, September 15, 2021 8:19:00 AM
To: frappe/wiki @.>
Cc: Chris Kateera @.>; Manual @.>
Subject: Re: [frappe/wiki] app works only with develop branch (#10)
@hasnain2808https://github.com/hasnain2808 I need to use this can you please let me know what is a workaround? I'm running Frappe in production.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHubhttps://github.com/frappe/wiki/issues/10#issuecomment-919732840, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AFOB46OARYSSHELBRW2JOXLUCA3FJANCNFSM5BOI53XQ.
Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

Today I installed the Wiki App on a v13 site and started fixing the issues that were coming up due to the incompatibility and then pushed the changes to the version-13 branch. Looks like everything is working apart from the route redirects
There are some routing rules so that the route for edit, new, and revisions do not look bad
https://github.com/frappe/wiki/blob/version-13/wiki/hooks.py#L16
Idk why these are not working on v13 so the routes look a bit bad
http://test:8000/home/edit
vs
http://test:8000/edit?wiki_page=home7
The issue looks like it is in Framework v13, so will need a fix there.

What is the actual URL to see the wiki? I tried /home and it shows the default home and the explore products button. I also tried /wiki/test after creating the test page, and it says error processing page with this traceback.
Traceback (most recent call last):
  File "/home/meofcorse/frappe-bench/apps/frappe/frappe/website/render.py", line 49, in render
    data = render_page_by_language(path)
  File "/home/meofcorse/frappe-bench/apps/frappe/frappe/website/render.py", line 178, in render_page_by_language
    return render_page(path)
  File "/home/meofcorse/frappe-bench/apps/frappe/frappe/website/render.py", line 194, in render_page
    return build(path)
  File "/home/meofcorse/frappe-bench/apps/frappe/frappe/website/render.py", line 201, in build
    return build_page(path)
  File "/home/meofcorse/frappe-bench/apps/frappe/frappe/website/render.py", line 224, in build_page
    html = frappe.get_template(context.template).render(context)
  File "/home/meofcorse/frappe-bench/env/lib/python3.8/site-packages/jinja2/environment.py", line 1090, in render
    self.environment.handle_exception()
  File "/home/meofcorse/frappe-bench/env/lib/python3.8/site-packages/jinja2/environment.py", line 832, in handle_exception
    reraise(*rewrite_traceback_stack(source=source))
  File "/home/meofcorse/frappe-bench/env/lib/python3.8/site-packages/jinja2/_compat.py", line 28, in reraise
    raise value.with_traceback(tb)
  File "/home/meofcorse/frappe-bench/apps/wiki/wiki/./wiki/doctype/wiki_page/templates/wiki_page.html", line 1, in top-level template code
    {% extends "wiki/doctype/wiki_page/templates/wiki_doc.html" %}
  File "/home/meofcorse/frappe-bench/apps/wiki/wiki/./wiki/doctype/wiki_page/templates/wiki_doc.html", line 2, in top-level template code
    {%- from "templates/includes/navbar/navbar_items.html" import render_item -%}
  File "/home/meofcorse/frappe-bench/apps/frappe/frappe/./templates/base.html", line 83, in top-level template code
    {% block content %}
  File "/home/meofcorse/frappe-bench/apps/wiki/wiki/./wiki/doctype/wiki_page/templates/wiki_doc.html", line 77, in block "content"
    {% block page_container %}
  File "/home/meofcorse/frappe-bench/apps/wiki/wiki/./wiki/doctype/wiki_page/templates/wiki_doc.html", line 83, in block "page_container"
    {%- block page_content -%}{%- endblock -%}
  File "/home/meofcorse/frappe-bench/apps/wiki/wiki/./wiki/doctype/wiki_page/templates/wiki_page.html", line 19, in block "page_content"
    {% include "wiki/doctype/wiki_page/templates/show.html" %}
  File "/home/meofcorse/frappe-bench/apps/wiki/wiki/./wiki/doctype/wiki_page/templates/show.html", line 32, in top-level template code
    {{ include_script('wiki.bundle.js') }}
  File "/home/meofcorse/frappe-bench/env/lib/python3.8/site-packages/jinja2/sandbox.py", line 460, in call
    if not __self.is_safe_callable(__obj):
  File "/home/meofcorse/frappe-bench/env/lib/python3.8/site-packages/jinja2/sandbox.py", line 360, in is_safe_callable
    getattr(obj, "unsafe_callable", False) or getattr(obj, "alters_data", False)
jinja2.exceptions.UndefinedError: 'include_script' is undefined

@MrKovalski you're probably installing the master branch on v13
The master branch or the release does not support Frappe v13

@hasnain2808 Thanks, its working on version-13 branch.

I think this needs to be changed on the Frappecloud service as well. It's currently pointing to the master branch.

thought exactly the same, that it must be unable to be installed also from frappecloud

I installed the wiki on my V.13 installation using the described installation description:
$ bench --site sitename install-app wiki    # Used the same sitename as the ERPNext installation

When I tried to define a first page, I came across this error and found this thread. I guess I have to use the Version 13-branch, but how do I uninstall this one now and then install the other branch? Thanks for your help.


Add SAoC banner

Disclaimer: I don't like this banner (and I didn't create it).
There's a thing called Corporate Identity (and imho that applies to Open Source organizations too).
Our Corporate Identity has been created by an actual designer -> https://github.com/dlang-community/artwork
In short

Font: Montserrat
the new logo (it has been decided over a year ago that this is the new logo for new projects)
Background color: #532360 (or the nice stars)
Main Font color: white

CC @aldacron

Unfortunately this logo has the same problem as the old one. It's not a recognizable shape, it doesn't work in a single color. Which means the logo is very inflexible. We would be stuck with a single color scheme. There will be cases where the logo will not work for these reasons.

@jacob-carlborg I'm aware of this problem (and that's the unfortunately main reason why we haven't swapped the icon on the menu and the favicon).
Anyhow, I'm more criticizing the fact that the pink font on blue background doesn't fit at all with the existing dlang.org design.

Oh, I didn't see that image before. That Symmetry Autumn of Code image looks absolutely horrible. Did the designer came up with that as well?

Did the designer came up with that as well?

No. He is absolutely innocent (actually I think since 2017's DConf at Sociomantic he hasn't done any D-related work and he doesn't even work at Sociomantic anymore).

Come on, we cannot put this on our web site. The more I look at the image the more upset I become.

The new banner Mike send me today:

(PR updated with it)

Looks better but it still looks a bit out of place.

Updated again (thanks @mdparker)

I still think it could be more beautiful, but the application deadline is very soon, so I think we should move forward with this and we can always improve it over the next days.
Are you okay with this as well @CyberShadow?

It's not bad, but if I'm allowed to nitpick:

Should it take as much vertical space? Maybe something in the shape of the 2nd banner but aesthetics of the 3rd?
Font rendering looks a bit weird and pixelated
Maybe use one of the fonts we are already using on dlang.org?
What is the license of the laptop-on-autumn-leaves photo, are we free to use it?

That version used Montserrat Black as the font
. I paid for a royalty-free license for the photo via Depositphotos.
Here's a leaderboard version using Roboto Slab.

Ping @mdparker @CyberShadow
I think we should try to get this to the frontpage of dlang.org ASAP (it's only two weeks to the application deadline) and I'm inclined to merge this as we don't seem to be capable to come with something better :/

Here's a leaderboard version using Roboto Slab.

Hmm, the font shadowing is imho way too big. Also the earlier mentioned problem that no one knows what the Symmetry Autumn of Code is applies here.

Oh I am far from happy, but as mentioned I think it's important to get something up on the front page for increased visibility of this campaign.


Rouge-L result is very low

I am working on a model of a text summary. For the output of the model, the test is performed using pyrouge. The test results show that the rouge-1 and rouge2 indicators are normal and consistent with the effect of the paper. However, the results of rouge-L differ greatly from the results of the paper. What is doubtful is how the rouge-L is calculated. What could be the cause of this?

Hi Hanghang,
Did you figure out about the reason?

Any update on this ? @ShirleyHan6 @hanghang2333
How much the ROUGE-L results differ ?


Extra badge variant

Provide a general summary of the feature here
A badge is really handy for communicating status or counts. There are a few variants but we are missing a neutral (white variant).
🤔 Expected Behavior?
In our case we are using the badge in tree list. Per item we are showing the amount of items that are in this collection. This shouldn't get extra attention. But when a tree list item is selected or is in a hover state the background of the badge gives the proper contrast.

😯 Current Behavior
If we had used the primary subtle variant this would have pretty much the same color as the active or hover state of the tree list item. So contrast would be bit of an issue there
💁 Possible Solution
Provide an inverted or neutral variant of the badge
🔦 Context
See screenshots. This pattern is being used in several functionalities now
💻 Examples
No response
👤 Your name
Jeremy Talen
🧢 Your Product/Team
Magister
### Tasks

@jtalen we don't think it should be part of the badge component itself but can (maybe) an option in the tree view component once we start building it. I'll add this issue to the tree view component.

We are using those numbers also in other list components

So maybe we can re-use that new specific part as a sub-component or something


Incorrect Signature attach added

Support added for https://portswigger.net/kb/issues/00200900_jwt-signature-not-verified, incorrect signature verification attack.

CHANGELOG.md updated with new 1.0.3 release

Code updated according to you suggestions @preetkaran20

@killshotrevival LGTM, merging the PR. Thank you :)


UI elements are misaligned in API Revision (Create & Deploy a new revision)

Description
Drop-downs are misaligned in API Deployments page.

Drop-down for delete a revision in the 'Deploy new revision' pop up dialog

Drop-down for delete a revision in the 'Create new revision' pop up dialog

Steps to Reproduce

Login to publisher portal
Click on any API
Deploy -->  Deployments
To navigate 'Deploy new revision' pop up dialog click on 'Deploy new revision'
To navigate 'Create new revision' pop up dialog click on the plus icon (+) in the revision timeline.

Affected Component
APIM
Version
4.3.0
Environment Details (with versions)
No response
Relevant Log Output
No response
Related Issues
No response
Suggested Labels
No response

Following misalignments in deployments  page needs to be fixed
.


More documentation for rubber stamps

Hey Howdy team!
First of all, this program works amazingly well. I've been using it alongside open-fprintd and using biometrics to sign in, unlock and authorise is great!
I noticed the "rubber stamps" section in your wiki, but I don't quite understand it. The guide doesn't explain how or where to put the rubber stamp rules (it mentions stamp_rules, but it's not clear where & how that should go in the config file).
Could you please explain how to use rubber stamps more thoroughly? I'm happy to edit the wiki if you want to explain it in this issue - I can edit the page myself.
Thanks again for your work :)

Good to hear that Howdy works well for you! Sorry for not being clear about it on the wiki page, but rubber stamps will be introduced in 3.0.0 which is not ready for release yet

Is there any timeframe for the v3? I see commit activities, but no "some kind of indication" on the timeline.


Fix Ember.merge deprecations

Addressing #79.

Thanks again @trentmwillis!

FYI - This deprecation has been removed from canary/beta/release and will be released as 2.5.1 sometime Monday.

@rwjblue interesting, looking at the PR to turn it off, sounds like it will still be turned back on in a future release though, correct? The ecosystem just isn't ready for it to be deprecated right now.

@trentmwillis - Yep, I was definitely not suggesting that this PR was invalid or anything.  I'm mostly just pointing out, that many many things are spewing this deprecation at the moment so we decided to disable.


Move Essentials economics into MySQL

As far as we have to use nosql db files - we have a block with integration the server with the site

Won't fix


api: set up sqlalchemy

[!WARNING]
This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

#130  👈
#123 : 1 other dependent PR (#124 )
#122
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @isaacharrisholt and the rest of your teammates on  Graphite


[e2e]: nodejs push branch1 default slsa3

Repo: https://github.com/slsa-framework/example-package/tree/branch1
Run: https://github.com/slsa-framework/example-package/actions/runs/11791701091
Workflow file: https://github.com/slsa-framework/example-package/tree/main/.github/workflows/e2e.nodejs.push.branch1.default.slsa3.yml
Workflow runs: https://github.com/slsa-framework/example-package/actions/workflows/e2e.nodejs.push.branch1.default.slsa3.yml
Trigger: push
Branch: branch1
Date: Tue Nov 12 06:16:53 UTC 2024

Repo: https://github.com/slsa-framework/example-package/tree/branch1
Run: https://github.com/slsa-framework/example-package/actions/runs/11811637039
Workflow file: https://github.com/slsa-framework/example-package/tree/main/.github/workflows/e2e.nodejs.push.branch1.default.slsa3.yml
Workflow runs: https://github.com/slsa-framework/example-package/actions/workflows/e2e.nodejs.push.branch1.default.slsa3.yml
Trigger: push
Branch: branch1
Date: Wed Nov 13 06:12:59 UTC 2024
Tests are passing now. Closing this issue.


Font files are classified as Scala or HolyC

Font source files included in repositories are being misidentified as Objective-J/HolyC/Scala.
URL of the affected repository:
https://github.com/ValveSoftware/Proton/tree/proton_5.0
https://github.com/adobe-fonts/source-han-mono
Expected language:
none
Detected language:
Scala/HolyC/SuperCollider/etc

As you've already found, this is because of the .sc extension which is only associated with those three languages and no font "languages" or otherwise explicitly ignored.  As a result, things fall through to the heuristic and then on to the classifier, but given Linguist doesn't know anything about the font "language" you're expecting it to find, it'll never find it.
As Linguist relies upon community contributions to address such things, we'd welcome a PR that either adds the language or ignores the files.

Oh, this applies to the other extensions too.

A month is a long time? Yes of course it's still an issue. Come on, bot. Am I going to have to do this every month?

A month is a long time? Yes of course it's still an issue. Come on, bot. Am I going to have to do this every month?

Yes, or you can submit a PR to add support 😉, after all Linguist relies almost exclusively on community contributions.

https://github.com/github/linguist/issues/4870#issuecomment-650159950

Asdf

@aeikum Could you enlighten us on what .SC files are? I googled SC font file" but it (naturally) brings up results for small-cap` variants of actual typefaces.

I believe they are variants of the font files for different languages. So they're different file types, despite the same final extension. See here: https://github.com/adobe-fonts/source-han-mono/tree/master/Heavy/OTC
J - Japanese, K - Korean, SC - Simplified Chinese, TC - Traditional Chinese. Not sure about HC.

SC - Simplified Chinese, TC - Traditional Chinese. Not sure about HC

Hong Kong and Taiwan have different variations of traditional Chinese, so TC and HC probably stand for "Taiwan Chinese" and "Hong Kong Chinese", respectively.
Anyway. From what I see in adobe-han-mono/Heavy/OTC, the .HC files are a mix of PostScript (Type 1 fonts, essentially specialised PostScript programs), OpenType feature definitions, and some ad hoc-looking format for CID font metadata. Since most of these files are too large to be indexed or displayed on GitHub, it's gonna be difficult to accurately determine how many repositories use .HC and .TC as file extensions (see CONTRIBUTING.md if you're unsure why that's relevant).
@aBARICHELLO Aside from the two repositories you linked to, how many others have you encountered where this is obviously an issue?

Note that there was quite some discussion about font files: https://github.com/github/linguist/issues/2516

That issue's from early/mid 2015. Barely any of it is relevant anymore — several missing font formats were added by yours truly, and being the resident font-nerd, I'm always eager to add support for a font format. 😉

@aBARICHELLO Aside from the two repositories you linked to, how many others have you encountered where this is obviously an issue?
Advanced search returned hundreds of results for .SC files being identified as Scala. .HC is harder to find.

Well, if you or anybody else is interested, here's an (unsorted) stash of .sc files harvested from search results. They don't appear to have much (or anything) in common... 😕

Yup


Make the text in the word comparison mode left-aligned

Make the text in the word comparison mode left-aligned instead of the current center alignment.

@davelab6, the text only appears center aligned in the opacity review mode when you switch to the word comparison mode. The text is not actually center aligned, though. The entire review mode is center aligned, but the text within the review mode is left aligned. I did this because it looks more visually pleasing.
Here is what the review mode looks like when it is entirely center aligned:

And here is what it looks like when it is left aligned:

I'm closing this issue. Please reopen if you want to discuss it further.


sver validate does not detect default profile missing.

It the default profile is not on the sver.toml and you refer it, sver calc fails.
Example:
.
├── data1
│   └── sver.toml (1)
└── data2
     └── sver.toml (2)

# sver.toml(1)
[default]
dependencies = [
  "data2",
]

# sver.toml(2)
[non-default]
dependencies = [
   ...
]

Commands:
% sver --version
sver 0.1.14
% sver validate
[OK]	data1/sver.toml:[default]
[OK]	data2/sver.toml:[no-default]
% sver calc data1
profile[default] is not found

It seems correct that sver calc raises an error.
sver validate should detect it.

@kna
Thank you for the report. This behavior is a bug and has been fixed in #122.

Thank you for fixing!


save_overwrite conflicts with autoresume=true

When must set autoresume=true,
if I set save_overwrite = false，it says:

FileExistsError: /home/mnt/diffmodels/ep0-ba1000-rank0.pt may conflict with a future checkpoint of the current run.Please
delete that file, change to a new folder, or set overwrite=True.
wandb: Waiting for W&B process to finish... (failed 1). Press Control-C to abort syncing.
wandb: ð View run sd-model at: https://wandb.ai/xxxx/diffusion/runs/hpzkfedp
wandb: Synced 6 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
wandb: Find logs at: ./wandb/run-20230701_152030-hpzkfedp/logs

if I set save_overwrite = true, it says:

InstantiationException: Error in call to target 'composer.trainer.trainer.Trainer':
ValueError('The flag save_overwrite must be False when autoresume is enabled as autoresume always loads the latest existing checkpoint in save_folder. ')
full_key: trainer

Unfortunately, this is a requirement for using autoresume. You are welcome to turn off autoresume if you would like, but in the future you will have to manually specify the load path when continuing a training run.


Pulsar Scaler: fix msgBacklogThreshold field being named wrongly as msgBacklog

Hi team,
This PR aims to ix msgBacklogThreshold field being named wrongly as msgBacklog . I pass down the logger into the method  parsePulsarMetadata as well
Checklist

[x] I have verified that my change is according to the deprecations & breaking changes policy
[x] Tests have been added
[x] Changelog has been updated and is aligned with our changelog requirements
[x] Commits are signed with Developer Certificate of Origin (DCO - learn more)

Fixes #4681

Thanks for the PR, ÇI have left some small nits

/run-e2e pulsar

Ah ha, no wonder I have a feeling that I was missing something :laughing:

/run-e2e pulsar


Makefile: allow passing CFLAGS/LDFLAGS

Originally part of #2, this patch has been carried in Buildroot since almost 8 years now.
@tpetazzoni, thanks for the original patch!

I think this should be safe. Merging.


encoding/xml: typo in docs

The documentation of func Unmarshal states:

Unmarshal uses a case-sensitive comparison to match XML element names to tag values and struct field names.

Unless I'm really confused about something, that should be "case-insensitive" instead. At least, that's my experience with using encoding/xml and encoding/json.

The documentation is accurate. The field won't match if the XML element's case doesn't match. See http://play.golang.org/p/9oZIgl9n3X where Name is an empty string after unmarshalling.


Add recent packages

Add nim7z, nimarchive, nimpcre and nimdeps to nimble packages.

Ping me when you've fixed this problem

Nimpcre is fixed after our discussion. Was there something else?


doc: Add invatation link in docs

Fixes #

Proposed Changes

Release Note

Thank you for this great downloading tool!

Thank you for your contribution. We look forward to receiving more contributions from you in the future.


add bound check to rustic-cargo-auto-add-missing-dependencies

Hi :wave: When I set rustic-load-optional-libraries to nil, I got this error on M-x rustic-mode:

Symbol's value as variable is void: rustic-cargo-auto-add-missing-dependencies

I think it's because rustic-cargo.el is not loaded and their custom variables are not set.
This PR fixes it by adding bound-and-true-p check.

Thanks :+1:
I think the option was supposed to be removed after rust-mode was added as dependency(but we obviously forgot it). Out of curiosity, why do you use rustic if you disable loading all rustic features. Why not just use rust-mode instead ?

Right, I would switch back to rust-mode, thanks! I forgot why I was using rustic 🤔


[8.x] Fix Eager loading partially nullable morphTo relations

This pull request fixes errors when 1 model has multiple polymorphic relationships sharing same "type" column.
In case we have a polymorphic relationship, where _type column is empty, script will work great. But if the _id column is empty, script will throw an exception.
For an example, if model has the following columns:

related_type (non-nullable)
related_old_id (nullable)
related_new_id (nullable)

and the following relationships:
    public function oldRelatedModel(): MorphTo
    {
        return $this->morphTo(name: null, type: 'related_type', id: 'related_old_id');
    }

    public function newRelatedModel(): MorphTo
    {
        return $this->morphTo(name: null, type: 'related_type', id: 'related_new_id');
    }

When we eager load both polymorphic relationships and one of the columns (related_old_id or related_new_id) are null, execution will fail (especially if we are dealing with uuids) because the function will convert all values to string (null => '') and '' is invalid uuid syntax.
 ERROR:  invalid input syntax for type uuid: \"\" (SQL: select * from \"user\" where \"user\".\"id\" in (, 68504945-959d-4e13-b1b4-7a250523cc24))"
This pull requests filters out nullable values and eager loads only non-nullable values.

Do you really need the callback at all? Doesn't array_filter with no callback already remove empty items.

Do you really need the callback at all? Doesn't array_filter with no callback already remove empty items?

You are right, it is not needed. I have removed it.


test if toastr is loaded

Hello,
I'm facing a little issue.
I'm using gritter but I want to migrate to toastr.
To avoid any problem, I write this in my javascript
`
if( $.isFunction($.fn.toastr) ){
        toastr[type](text);

    }

    else{

        text = typeof text !== 'undefined' ? text : '';

        type = typeof type !== 'undefined' ? type : 'success';

        time = typeof time !== 'undefined' ? time : 2;

        $.gritter.add({

            title: title,

            text: text,

            class_name: 'clean',

            image: '/assets/images/' + type + '.png',

            time: time

        });

    }

`
but $.isFunction($.fn.toastr) return false even if toastr is loader.
Is there another way to check if toaster is loaded ?
Thanks for your time.
Regards,
Francois

I use if (typeof toastr == 'object') { /* do something */ }

Looks like toastr is not defined under jQuery(or whatever $ means)'s fn namespace, so maybe
if(window.toastr != undefined) { console.log('toastr has been loaded'); }


Default to info level logs in prod

This is largely due to the fact that the Gnat library logs connection information at the debug level (e.g. jwt + seed used for connecting)

@connorsmith256 figured it out!


Add Nexus 6 and 9

Device information should be added for the latest of the Nexus family.
I can add this myself once I get some time to get a UA string from the actual devices I can get a hold of. But if someone else can get in faster then that's fine too.

Fixed by commit: 3f770e9b758ca198b67ad49b5b57a344ef898cdc and 5a8f42f59f92b5652d2ee15a7476457a9eb0ef2a


ef core

Can I integrated marten to  ef core , so that I can both use ef and marten's jsonb for pgsql

Yes, you can. For more detailed answer, I'd need more detailed question :) I was using both of them and I didn't find any bigger issues or blockers