Not all files of the MCPileup are added to the configuration

following up on https://hypernews.cern.ch/HyperNews/CMS/get/prep-ops/4066.html where the software needs all file of the secondary input to be listed in the mix module, @thongonary and I were looking at a specific job log
/eos/cms/store/logs/prod/2017/05/WMAgent/rverma_PR_newco_RelVal_276950_170428_191716_3771/rverma_PR_newco_RelVal_276950_170428_191716_3771-Task1MergeDQMoutputMergedEndOfRunDQMHarvestLogCollect-cmswn2203-1-logs.tar
/WMTaskSpace/logCollect1/2a1532f6-465e-11e7-a342-a0369f23d01e-488-1-logArchive.tar.gz
and the PSet has 2423 files in mixing module while the full dataset does have 5660 files in total, and is in full at the site showing errors about this workflow.
the suspicion is that that secondary file list was truncated at pset tweak time one way or the other.
It's the first time we (at least I) encouter such type of request requiring exactly all files of the secondary to be present.

FYI @prozober

I just looked a the sandbox and the pileupconf.json file.
T1_ES_PIC_Disk 133
T1_FR_CCIN2P3_MSS 5660
T2_CH_CSCS 157
T1_FR_CCIN2P3_Disk 134
T2_TW_NCHC 5660
T2_UK_SGrid_RALPP 60
T2_FR_GRIF_LLR 126
T2_FR_IPHC 226
T0_CH_CERN_Disk 764
T2_DE_DESY 2189
T2_IT_Legnaro 176
T1_DE_KIT_Disk 5660
T2_UK_London_Brunel 2382
T2_IT_Pisa 5660
T1_UK_RAL_Disk 4475
T1_US_FNAL_Disk 124
T2_FR_GRIF_IRFU 104
T1_IT_CNAF_Disk 248
T1_RU_JINR_Disk 282
T2_UK_London_IC 4972
T2_IT_Bari 136
T1_FR_CCIN2P3_Buffer 5660
T2_FR_CCIN2P3 174
T2_ES_CIEMAT 162
T2_US_MIT 5660
T2_BE_IIHE 78
T2_CH_CERN 5660
T2_PT_NCG_Lisbon 65
T2_US_Vanderbilt 348
is the number of files per site, as seen during the sandbox creation.
I can confirm from unified logs that the workflow was assigned with 100% of the dataset at desy (and kit and mit).
So something went astray in creating the sandbox

@ticoann @amaltaro we will need your insight to this, as it has spread over many workflows. Is there anything in the agent log that can give a hint at what went wrong with the sandbx creation ? if not, can we add some logging.

I had a look at that sandbox and the pileupconf.json file actually contains 2423 files as present at DESY
>>> filesInDESY = 0
>>> for block in data['mc']:
...   if 'T2_DE_DESY' in data['mc'][block]['PhEDExNodeNames']:
...     filesInDESY += len(data['mc'][block]['FileList'])
...
>>> filesInDESY
2423

which confirms what you saw in one of the job logs "and the PSet has 2423 files in mixing module".
I was also playing with the phedex APIs  trying to find subscriptions, dataset/block completion and etc and the closest I got was that there was a dataset subscription on the same day that that workflow got assigned (~12h before):
Id: 1014153	Requestor: Matteo Cremonesi	Type: xfer	Status: approved	Node: T2_DE_DESY
	Time-created: 2017-05-31 12:57:28	Time-decided: 2017-05-31 12:57:28

Thus, my question goes to the transfer team @sidnarayanan and Sebastian (dunno his GH user). Is there any API which tells when a transfer request got 100% completed? Are you able to track when this dataset:
/ReggeGribovPartonMC_EposLHC_pPb_4080_4080_DataBS/pPb816Spring16GS-MB_80X_mcRun2_pA_v4-v2/GEN-SIM
got completely transferred to T2_DE_DESY?
About the pileup json file, as you could see there are a few PNNs where the dataset is fully available (5660 files), so I doubt there would be any truncation or issue like that building it.
In addition to that, the ACDC workflow prozober_ACDC0_task_HIN-pPb816Summer16DR-00079__v1_T_170607_094539_6258 has all 5660 PU files in the json file (for DESY).
In summary, my suspicious is that something went wrong on
"I can confirm from unified logs that the workflow was assigned with 100% of the dataset at desy (and kit and mit)." and the data was not completely there, unless Sid can prove me wrong.

Hi Alan,
The best way I know how to tell when it was complete is to check when each of the replicas made it to the endpoint. This can be done via filereplicas [1] and looking at the time_create field of the replicas. For this subscription, it looks like the last file replica was created at<PHONE_NUMBER>.8.
[1] https://cmsweb.cern.ch/phedex/datasvc/json/prod/filereplicas?dataset=/ReggeGribovPartonMC_EposLHC_pPb_4080_4080_DataBS/pPb816Spring16GS-MB_80X_mcRun2_pA_v4-v2/GEN-SIM&node=T2_DE_DESY

Thanks, Sid! Would you by any chance know whether the phedex timestamps are local unix epoch time, or always UTC? BTW, who publishes that replica info? Does it come from every site phedex agent?
Taking into consideration a possible problem with local x utc epoch timestamps, I listed 493 files that were created after that workflow got assigned<PHONE_NUMBER> -7200), using the filereplicas api
Looking at the blockreplicas api and the time_update replica attribute, I listed 23 blocks and 989 files, also after the same timestamp. Where several of those blocks were updated yesterday (?)
These numbers don't really match with the lack of ~3k files during the assignment though. However, I'm pretty convinced this dataset wasn't fully available at DESY during the request assignment/acquisition.

Unfortunately I don't have definite answers to any of those 3 questions :). You might want to ask @nataliaratnikova . Certainly the site phedex agent has to tell the central agent that the replica was created, but then I'm not sure if it's the site or central agent that chooses the timestamp. I would hope we use UTC for everything, or things would get confusing...
Sorry, but I'm having a bit of trouble following this. Do I understand correctly:

You tried to run at DESY at some given time, and WMCore saw ~3k missing files at that site at that time

You found 493 files (989 files) that were created after the wf got assigned, when checking via filereplicas (blockreplicas). I think 493 vs 989 is not surprising: I assume when you check via blockreplicas, you count all the files in that block if time_update is after your cutoff timestamp. It's possible some of the files were produced before the cutoff, but the block wasn't finished and closed. There could even be a delay between finishing the file injections and closing the block (not sure how you do this in WMCore).

There is still the question of why ~3k files were missing. But then I have to ask how you came up with this 2423 number. Is it from PhEDEx, and if so, which API?

Thanks for the clarification.

No problem, there are too many details and it's easy to get lost.
In summary, when a workflow gets acquired by the agents (after it's assigned in reqmgr2), the agent builds a json for the PU dataset which contains a mapping of blocks x files x PNNs. This json doesn't change during the lifetime of this workflow.
Thus the suspicion is that agents acquired that workflow when not ALL blocks were completed at DESY. That's why we saw 2423 instead of 5660 files at DESY (the agent only considers complete blocks)

Wed May 31 23:58:32 2017 (GMT) (consistent in reqmgr2 and unified)
http://dabercro.web.cern.ch/dabercro/unified/showlog/?search=task_HIN-pPb816Summer16DR-00079&module=assignor
https://github.com/CMSCompOps/WmAgentScripts/blob/master/utils.py#L2697
returned
/ReggeGribovPartonMC_EposLHC_pPb_4080_4080_DataBS/pPb816Spring16GS-MB_80X_mcRun2_pA_v4-v2/GEN-SIM
{
"T1_ES_PIC_Disk": [
false,
93.3364270657362
...
"T2_DE_DESY": [
true,
100.0
],
"T2_IT_Legnaro": [
false,
3.02387718966739
],
"T1_DE_KIT_Disk": [
true,
100.0
],
...
I cannot see any flaw in the method, and if there was, we'd be seen much more failures all over the place in terms of workflow assigned to site that do not have the data, ... and we do not.

time.asctime(time.gmtime(1496271816.8))
'Wed May 31 23:03:36 2017'

so it looks like the last file replica at DESY was made ~1 hour before the workflow was assigned.

@amaltaro can you please point us to the spot where the PU location is queried ?

note also that the same code, at Wed May 31 22:59:11 2017 returned
],
"T2_DE_DESY": [
false,
87.77380013046422
],

Apologies, I missed one important detail:
https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/WMSpec/Steps/Fetchers/PileupFetcher.py#L167
so we do keep a cache of pileup type+dataset in the agent (24h caching).
Apparently there are ~10 workflows in the system using this PU dset.
This PU cache is very important for large campaigns using the same PU and it has a significant impact on how WorkQueueManager acquires work.
Maybe we should decrease it to 1h and live with possible inefficiencies on work acquisition? Of course, there will always be a chance that there is fresh PU data and it uses a stale cache.

I see. Makes sense. I assume you cannot make the cache update conditional on what site is going to be used in sitewhitelist.
from the current crash, 30min would be more adapted, but more impacted.
we could keep the 24h cache lifetime, and in addition have an systematic incremental call to something like
https://cmsweb.cern.ch/phedex/datasvc/xml/prod/filereplicas?dataset=/ReggeGribovPartonMC_EposLHC_pPb_4080_4080_DataBS/pPb816Spring16GS-MB_80X_mcRun2_pA_v4-v2/GEN-SIM&node=T2_DE_DESY&since_time=
N.B. since_time would filter on create_time, to update the cache (i.e. we assume that within 24h the dataset is incrementally getting to sites, and cache is flushed every 24h)
Should we get a faster dedicated api from phedex instead ?

Hi,
looking at the last_replica time<PHONE_NUMBER>) returned by
https://cmsweb.cern.ch/phedex/datasvc/perl/prod/blocklatencylog?dataset=/ReggeGribovPartonMC_EposLHC_pPb_4080_4080_DataBS/pPb816Spring16GS-MB_80X_mcRun2_pA_v4-v2/GEN-SIM&to_node=T2_DE_DESY
the last file from this dataset was transferred to DESY on Wed, 31 May 2017 16:44:45 GMT .

Actually, the above time is the completion time  for some random block I picked from the returned data. The last block in this dataset arrived a few hours later.
I added  an example:
https://github.com/nataliaratnikova/PHEDEX/blob/master/Contrib/phedex_dataset_arrival_time_example.ipynb
on how to get the arrival time for the last completed block in this dataset.

@amaltaro A similar issue happened at Purdue: https://hypernews.cern.ch/HyperNews/CMS/get/relval/7838.html
Given that this is a relatively small PU, could you please consider reducing the cache lifetime to a few minutes?

@thongonary we've just patched the production agents (except those old agents draining). Cache lifetime should be reduced to 30min now.

There is apparently another issue of this type, much longer after the pu timeout being 30min or 24h it seems https://cmsweb.cern.ch/reqmgr2/fetch?rid=vlimant_ACDC0_task_HIN-pPb816Spring16wmLHEGS-00001__v1_T_170721_103006_6723

Paola provided some ACDC workflows that were created after the patch got applied. I picked this one
mcremone_ACDC0_task_HIN-pPb816Spring16wmLHEGS-00006__v1_T_170724_151638_8337-Sandbox.tar.bz2
running on vocms0126 and took a look at its pileupconf.json file (inside the sandbox tarball):
WMSandbox/HIN-pPb816Summer16DR-00147_1/cmsRun1/pileupconf.json
According to DBS, this datasetL
/ReggeGribovPartonMC_EposLHC_pPb_4080_4080_DataBS/pPb816Spring16GS-MB_80X_mcRun2_pA_v4-v2/GEN-SIM
has 117 blocks, which indeed are all mapped to CERN:
>>> with open('pileupconf.json') as jo:
...   data = json.load(jo)
>>> blocksAtCERN = 0
>>> for block, info in data['mc'].iteritems():
...   if 'T2_CH_CERN' in info['PhEDExNodeNames']:
...     blocksAtCERN += 1
...
>>> blocksAtCERN
117

Thus, I see no issues from the WMCore side. The PU json configuration is correct and contains all blocks.
I don't know how CMSSW decides which event to pick from the pileup dataset (?), but it's probably trying to read an event that does NOT exist at all in the PU sample.
For the record, this is the wmstats job error that I was looking at:
Job Name: 2268d5ba-7077-11e7-b4db-02163e0184a6-357
WMBS job id: 1161634
Workflow: mcremone_ACDC0_task_HIN-pPb816Spring16wmLHEGS-00006__v1_T_170724_151638_8337
Task: /mcremone_ACDC0_task_HIN-pPb816Spring16wmLHEGS-00006__v1_T_170724_151638_8337/HIN-pPb816Summer16DR-00147_0
Status: jobfailed
Input dataset:
Site: T2_CH_CERN
Agent: vocms0126.cern.ch
ACDC URL: https://cmsweb.cern.ch/couchdb/acdcserver
State Transition
Exit code: 8026
Retry count: 0
Errors:
    cmsRun1
        Fatal Exception (Exit Code: 8026)

            An exception of category 'NotFound' occurred while
               [0] Processing run: 1 lumi: 17 event: 1441
               [1] Running path 'digitisation_step'
               [2] Calling event method for module MixingModule/'mix'
            Exception Message:
            RootEmbeddedFileSequence::readOneSpecified(): Secondary Input files do not contain specified event:
            run: 1 lumi: 51215 event: 4878241

and this lumi section maps to the following LFN:
https://cmsweb.cern.ch/dbs/prod/global/DBSReader/files?logical_file_name=/store/himc/pPb816Spring16GS/ReggeGribovPartonMC_EposLHC_pPb_4080_4080_DataBS/GEN-SIM/MB_80X_mcRun2_pA_v4-v2/90001/4CE2F999-0009-E711-B779-0025905C2CBC.root&detail=true
which I have no clue whether it contains that event or not? Does anyone know how to list the events in a file? @yuyiguo  ?

BTW, it was assigned to both CERN and MIT, but MIT doesn't have the complete dataset. It's missing one block.

I checked the file pointed to by Alan and confirmed that it contained the mentioned event:
$ root -l /eos/cms/store/himc/pPb816Spring16GS/ReggeGribovPartonMC_EposLHC_pPb_4080_4080_DataBS/GEN-SIM/MB_80X_mcRun2_pA_v4-v2/90001/4CE2F999-0009-E711-B779-0025905C2CBC.root

root [5] Events->Scan("EventAuxiliary.event()","EventAuxiliary.event()==4878241")
************************
*    Row   * EventAuxi *
************************
*     3345 *   4878241 *
************************
==> 1 selected entry

So this doesn't seem to be a dataset's problem either.

Ok, I have some interesting findings which I can't really explain ATM...Verbose mode ON
Looking at the PU datasets requested for this workflow (yes, there are two different datasets):
https://cmsweb.cern.ch/reqmgr2/data/request?name=mcremone_ACDC0_task_HIN-pPb816Spring16wmLHEGS-00006__v1_T_170724_151638_8337&mask=MCPileup
side note, from my experience, I have had issues in the past when setting AcqEra to Integ_Test or Integ_TEST. Maybe Yuyi @yuyiguo can confirm whether all dbs operations are case sensitive and thus not an issue in this case.
That task HIN-pPb816Summer16DR-00147_0 was using the following PU sample:
/ReggeGribovPartonMC_EposLHC_PbP_4080_4080_DataBS/pPb816Spring16GS-MB_80X_mcRun2_pA_v4-v2/GEN-SIM
which has 117 blocks:
https://cmsweb.cern.ch/dbs/prod/global/DBSReader/filesummaries?dataset=/ReggeGribovPartonMC_EposLHC_PbP_4080_4080_DataBS/pPb816Spring16GS-MB_80X_mcRun2_pA_v4-v2/GEN-SIM
Now, if we look at the cmsRun stdout log, this is the file and lumi + event it was looking for
24-Jul-2017 22:20:26 UTC  Initiating request to open file root://eoscms.cern.ch//eos/cms/store/himc/pPb816Spring16GS/ReggeGribovPartonMC_EposLHC_PbP_4080_4080_DataBS/GEN-SIM/MB_80X_mcRun2_pA_v4-v2/00002/A2A83E49-0F08-E711-84C7-02163E013767.root
24-Jul-2017 22:20:28 UTC  Successfully opened file root://eoscms.cern.ch//eos/cms/store/himc/pPb816Spring16GS/ReggeGribovPartonMC_EposLHC_PbP_4080_4080_DataBS/GEN-SIM/MB_80X_mcRun2_pA_v4-v2/00002/A2A83E49-0F08-E711-84C7-02163E013767.root
----- Begin Fatal Exception 24-Jul-2017 22:20:28 UTC-----------------------
An exception of category 'NotFound' occurred while
   [0] Processing run: 1 lumi: 17 event: 1441
   [1] Running path 'digitisation_step'
   [2] Calling event method for module MixingModule/'mix'
Exception Message:
RootEmbeddedFileSequence::readOneSpecified(): Secondary Input files do not contain specified event:
run: 1 lumi: 51215 event: 4878241
----- End Fatal Exception -------------------------------------------------
24-Jul-2017 22:20:29 UTC  Closed file root://eoscms.cern.ch//eos/cms/store/himc/pPb816Spring16wmLHEGS/DYtoMuMu_M-10to30_pPb-EmbEPOS_8p16_Powheg/GEN-SIM/pPbEmb_80X_mcRun2_pA_v4-v1/130000/0688E6D5-BE6C-E711-88A5-FA163E183C33.root
24-Jul-2017 22:20:29 UTC  Closed file root://eoscms.cern.ch//eos/cms/store/himc/pPb816Spring16GS/ReggeGribovPartonMC_EposLHC_PbP_4080_4080_DataBS/GEN-SIM/MB_80X_mcRun2_pA_v4-v2/00002/A2A83E49-0F08-E711-84C7-02163E013767.root
MemoryReport> Peak virtual size 1966.58 Mbytes

which corresponds to the correct dataset, however this file does NOT have the lumi section 51215
https://cmsweb.cern.ch/dbs/prod/global/DBSReader/filelumis?logical_file_name=/store/himc/pPb816Spring16GS/ReggeGribovPartonMC_EposLHC_PbP_4080_4080_DataBS/GEN-SIM/MB_80X_mcRun2_pA_v4-v2/00002/A2A83E49-0F08-E711-84C7-02163E013767.root
this lumi section can be found in this file:
https://cmsweb.cern.ch/dbs/prod/global/DBSReader/files?dataset=/ReggeGribovPartonMC_EposLHC_PbP_4080_4080_DataBS/pPb816Spring16GS-MB_80X_mcRun2_pA_v4-v2/GEN-SIM&run_num=1&lumi_list=[51215]
thus /store/himc/pPb816Spring16GS/ReggeGribovPartonMC_EposLHC_PbP_4080_4080_DataBS/GEN-SIM/MB_80X_mcRun2_pA_v4-v2/50001/B496A982-C007-E711-B0AC-1866DA7F90B8.root
Last but not least, I unpickled the PSet configuration for the job I mentioned in my previous comment  and ... as you can see, the correct file with lumi 51215 is there:
amaltaro@lxplus076:~/workarea/PU/CMSSW_8_0_28/src $ grep -r '/store/himc/pPb816Spring16GS/ReggeGribovPartonMC_EposLHC_PbP_4080_4080_DataBS/GEN-SIM/MB_80X_mcRun2_pA_v4-v2/50001/B496A982-C007-E711-B0AC-1866DA7F90B8.root' myPSet.py
            '/store/himc/pPb816Spring16GS/ReggeGribovPartonMC_EposLHC_PbP_4080_4080_DataBS/GEN-SIM/MB_80X_mcRun2_pA_v4-v2/50001/B496A982-C007-E711-B0AC-1866DA7F90B8.root',

Just to be safe, I picked the original (first) workflow sandbox, untarred and checked the pileupconf.json files and indeed the Task1 has a different json compared to Task2 and 3 (which use the same PU sample):
cmst1@vocms0116:/data/srv/wmagent/current/alandir $ diff -q ./WMSandbox/HIN-pPb816Summer16DR-00147_0/cmsRun1/pileupconf.json ./WMSandbox/HIN-pPb816Summer16DR-00147_1/cmsRun1/pileupconf.json
cmst1@vocms0116:/data/srv/wmagent/current/alandir $ diff -q ./WMSandbox/HIN-pPb816Spring16wmLHEGS-00006_0/cmsRun1/pileupconf.json ./WMSandbox/HIN-pPb816Summer16DR-00147_1/cmsRun1/pileupconf.json
Files ./WMSandbox/HIN-pPb816Spring16wmLHEGS-00006_0/cmsRun1/pileupconf.json and ./WMSandbox/HIN-pPb816Summer16DR-00147_1/cmsRun1/pileupconf.json differ

In summary, I can see 2 problems:

McM somehow mixed up different PU samples (unless that was intentional)
the framework got lost on which lumi section should be in a file

indeed the workflow is mis-configured
/ReggeGribovPartonMC_EposLHC_pPb_4080_4080_DataBS/pPb816Spring16GS-MB_80X_mcRun2_pA_v4-v2/GEN-SIM
/ReggeGribovPartonMC_EposLHC_PbP_4080_4080_DataBS/pPb816Spring16GS-MB_80X_mcRun2_pA_v4-v2/GEN-SIM
are requested on the two different steps.
thanks @amaltaro for the paranoiac check. apologies for doubting the "fix"


Detecting UNINITIALIZED_VALUE in $onChanges lifecycle hook

With a scope watcher, you can detect if it is the first time the watcher is running by checking if the new and old values are the same:
$scope.$watch('expr', function (newValue, oldValue) {
    if (newValue === oldValue) {
        // first time running
    }
});

However, with the Angular 1.5 $onChanges lifecycle hook, the placeholder object UNINITIALIZED_VALUE is the previousValue. However, there is no easy way to check for that. You can do it by checking the constructor's name:
ExampleController.prototype.$onChanges = function (changes) {
    if (changes.expr.previousValue.constructor.name === 'UNINITIALIZED_VALUE') {
        // first time running
    }
};

but that seems like a bit of a hack.
The best solution I can think of is having the UNINITIALIZED_VALUE object should be stored as a constant (read-only) somewhere (eg angular.UNINITIALIZED_VALUE) to compare against, though there are probably other solutions as well.
Thanks!!

Each change object (contained in the changes object) has an isFirstChange() method for this exact purpose.

I used the following code as isFirstChange() is not available in the changes object:
ctrl.$onChanges = function (changesObj) {   if (changesObj.fieldInfo.previousValue.constructor.name === 'UNINITIALIZED_VALUE') {     init();   } };

@saurajit it doesn't look like it's there (eg in the Dev tools console it appears to be missing when hovering on change.expr object) but that function is hidden on the object's prototype. Try executing changes.expr.isFirstChange().


Adds tar as dependency used for VMware to seed systemvmtemplate on mgmt server

Adds tar as dependency used for VMware to seed systemvmtemplate on mgmt server

@blueorangutan package

@rohityadavcloud a Jenkins job has been kicked to build packages. It will be bundled with  SystemVM template(s). I'll keep you posted as I make progress.

@rohityadavcloud a Jenkins job has been kicked to build packages. It will be bundled with
SystemVM template(s). I'll keep you posted as I make progress.

Packaging result: :heavy_check_mark: el7 :heavy_check_mark: debian. SL-JID 5323

@rohityadavcloud
systemvm templates are not bundled if @blueorangutan package is not the last line of your comment (e.g. blank/empty line after that).
@blueorangutan package

@weizhouapache a Jenkins job has been kicked to build packages. It will be bundled with` is not the last line of your comment (e.g. blank/empty line after that).
SystemVM template(s). I'll keep you posted as I make progress.

@blueorangutan package

@weizhouapache a Jenkins job has been kicked to build packages. It will be bundled with  KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Packaging result: :heavy_check_mark: el7 :heavy_check_mark: el8 :heavy_check_mark: el9 :heavy_check_mark: debian :heavy_check_mark: suse15. SL-JID 5324

Packaging result: :heavy_check_mark: el7 :heavy_check_mark: el8 :heavy_check_mark: el9 :heavy_check_mark: debian :heavy_check_mark: suse15. SL-JID 5325

Packaging result: :heavy_check_mark: el7 :heavy_check_mark: el8 :heavy_check_mark: el9 :heavy_check_mark: debian :heavy_check_mark: suse15. SL-JID 5326

@blueorangutan test

@blueorangutan help

@rohityadavcloud I understand these words: "help", "hello", "thanks", "package", "test"
Test command usage: test [mgmt os] [hypervisor] [keepEnv] [qemuEv]
Mgmt OS options: ['centos7', 'centos6', 'suse15', 'alma8', 'ubuntu18', 'ubuntu22', 'ubuntu20', 'rocky8']
Hypervisor options: ['kvm-centos6', 'kvm-centos7', 'kvm-rocky8', 'kvm-alma8', 'kvm-ubuntu18', 'kvm-ubuntu20', 'kvm-ubuntu22', 'kvm-suse15', 'vmware-55u3', 'vmware-60u2', 'vmware-65u2', 'vmware-67u3', 'vmware-70u1', 'vmware-70u2', 'vmware-70u3', 'xenserver-65sp1', 'xenserver-71', 'xenserver-74', 'xcpng74', 'xcpng76', 'xcpng80', 'xcpng81', 'xcpng82']
Note: when keepEnv is passed, you need to specify mgmt server os and hypervisor or use the matrix command.
when qemuEv is passed, it will deploy KVM hyperviosr hosts with qemu-kvm-ev, else it will default to stock qemu.
Package command usage: package [all(default value),kvm,xen,vmware,hyperv,ovm] - a comma separated list can be passed with package command to bundle the required hypervisor's systemVM templates. Not passing any argument will bundle all - kvm,xen and vmware templates.
Blessed contributors for kicking Trillian test jobs: ['rohityadavcloud', 'nvazquez', 'borisstoyanov', 'DaanHoogland', 'shwstppr', 'andrijapanicsb', 'rahulbcn27', 'davidjumani', 'harikrishna-patnala', 'vladimirpetrov', 'weizhouapache', 'NuxRo', 'mlsorensen']

@blueorangutan rocky8 kvm-rocky8

@blueorangutan test rocky8 kvm-rocky8

@rohityadavcloud a Trillian-Jenkins test job (rocky8 mgmt + kvm-rocky8) has been kicked to run smoke tests

Mentioned briefly in the description; I was trying to deploy Vmware 7 and 8 on an EL8 mgmt server when the routing-8.ova won't extract as "tar" pkg was missing on the mgmt server host. This fixes for only vmware env with EL8. @DaanHoogland

@DaanHoogland unsupported parameters provided. Supported mgmt server os are: centos7, centos6, suse15, alma8, ubuntu18, ubuntu22, ubuntu20, rocky8. Supported hypervisors are: kvm-centos6, kvm-centos7, kvm-rocky8, kvm-alma8, kvm-ubuntu18, kvm-ubuntu20, kvm-ubuntu22, kvm-suse15, vmware-55u3, vmware-60u2, vmware-65u2, vmware-67u3, vmware-70u1, vmware-70u2, vmware-70u3, xenserver-65sp1, xenserver-71, xenserver-74, xcpng74, xcpng76, xcpng80, xcpng81, xcpng82

Trillian test result (tid-5908)
Environment: kvm-rocky8 (x2), Advanced Networking with Mgmt server r8
Total time taken: 53963 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7113-t5908-kvm-rocky8.zip
Smoke tests completed. 101 look OK, 6 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test
Result
Time (s)
Test File

test_01_add_primary_storage_disabled_host
Error
0.78
test_primary_storage.py

test_01_primary_storage_nfs
Error
0.17
test_primary_storage.py

ContextSuite context=TestStorageTags>:setup
Error
0.31
test_primary_storage.py

test_01_non_strict_host_anti_affinity
Failure
157.74
test_nonstrict_affinity_group.py

test_02_non_strict_host_affinity
Error
82.50
test_nonstrict_affinity_group.py

test_create_pvlan_network
Error
0.08
test_pvlan.py

test_02_upgrade_kubernetes_cluster
Failure
673.65
test_kubernetes_clusters.py

test_03_deploy_and_scale_kubernetes_cluster
Failure
347.45
test_kubernetes_clusters.py

test_07_deploy_kubernetes_ha_cluster
Failure
189.53
test_kubernetes_clusters.py

test_08_upgrade_kubernetes_ha_cluster
Failure
82.48
test_kubernetes_clusters.py

test_09_delete_kubernetes_ha_cluster
Failure
81.46
test_kubernetes_clusters.py

ContextSuite context=TestKubernetesCluster>:teardown
Error
166.49
test_kubernetes_clusters.py

test_01_migrate_VM_and_root_volume
Error
122.43
test_vm_life_cycle.py

test_02_migrate_VM_with_two_data_disks
Error
48.81
test_vm_life_cycle.py

test_01_secure_vm_migration
Error
200.24
test_vm_life_cycle.py

test_02_unsecure_vm_migration
Error
273.69
test_vm_life_cycle.py

test_03_secured_to_nonsecured_vm_migration
Error
145.28
test_vm_life_cycle.py

test_08_migrate_vm
Error
45.08
test_vm_life_cycle.py

test_hostha_enable_ha_when_host_in_maintenance
Error
304.34
test_hostha_kvm.py

@blueorangutan test centos7 vmware-70u3

@DaanHoogland a Trillian-Jenkins test job (centos7 mgmt + vmware-70u3) has been kicked to run smoke tests

Trillian test result (tid-5916)
Environment: vmware-70u3 (x2), Advanced Networking with Mgmt server 7
Total time taken: 49017 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7113-t5916-vmware-70u3.zip
Smoke tests completed. 105 look OK, 2 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test
Result
Time (s)
Test File

test_create_pvlan_network
Error
0.06
test_pvlan.py

test_08_upgrade_kubernetes_ha_cluster
Failure
672.98
test_kubernetes_clusters.py

@blueorangutan test centos7 vmware-70u3

@rohityadavcloud a Trillian-Jenkins test job (centos7 mgmt + vmware-70u3) has been kicked to run smoke tests

@rohityadavcloud a Trillian-Jenkins test job (centos7 mgmt + vmware-70u3) has been kicked to run smoke tests

Trillian Build Failed (tid-5973)

@blueorangutan test centos7 vmware-70u3

@rohityadavcloud a Trillian-Jenkins test job (centos7 mgmt + vmware-70u3) has been kicked to run smoke tests

@DaanHoogland - this is ready but let's wait until the smoketests pass. (earlier env failed due to env issues)

Trillian test result (tid-5981)
Environment: vmware-70u3 (x2), Advanced Networking with Mgmt server 7
Total time taken: 50858 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7113-t5981-vmware-70u3.zip
Smoke tests completed. 104 look OK, 3 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test
Result
Time (s)
Test File

test_03_ping_in_ssvm_success
Failure
60.39
test_diagnostics.py

test_create_pvlan_network
Error
0.06
test_pvlan.py

test_02_upgrade_kubernetes_cluster
Failure
552.22
test_kubernetes_clusters.py

Trillian test result (tid-5986)
Environment: vmware-70u3 (x2), Advanced Networking with Mgmt server 7
Total time taken: 46801 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7113-t5986-vmware-70u3.zip
Smoke tests completed. 106 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test
Result
Time (s)
Test File

test_08_arping_in_ssvm
Failure
5.17
test_diagnostics.py

test_09_arping_in_cpvm
Failure
5.18
test_diagnostics.py


Fixes passing token/env into release task from group publish

Fixes in issue where distribute release with --token works but distribute groups publish does not (with a 403) due to not forwarding params from the groups/publish task.

Oh nice catch -- I guess we have the same issue with stores publish?

Possibly! We can fix as part of this PR. I'll glance at that command as well

Oh I didn't get the update on the page until I commented lol. Thanks for doing that stores change @anywherepilot!


Add AI model-powered translation to the webpage

Page Assist is a good plugin, similar to the open-source DeepSeekv3. Could you add some features to translate webpages, similar to the translation function of Monica?

Hey, this is a good idea! I will try to add it in the coming updates. :) Also, there are free alternatives like Immersive Translate


x/tools/gopls: support insertReplaceEdit results in completion items

What version of Go, VS Code & VS Code Go extension are you using?
Version Information

Run go version to get version of Go from the VS Code integrated terminal.

go version go1.17 darwin/amd64

Run gopls -v version to get version of Gopls from the VS Code integrated terminal.

Build info
----------
golang.org/x/tools/gopls v0.7.2
<EMAIL_ADDRESS>h1:kRKKdvA8GOzra8rhSFDClOR7hV/x8v0J0Vm4C/gWq8s=
<EMAIL_ADDRESS>h1:GaI7EiDXDRfa8VshkTj7Fym7ha+y8/XxIgD2okUIjLw=
<EMAIL_ADDRESS>h1:BKbKCqvP6I+rmFHt06ZmyQtvB8xAkWdhFyr0ZUNZcxQ=
<EMAIL_ADDRESS>h1:we8PVUC3FE2uYfodKH/nBHMSetSfHDR6scGdBi+erh0=
<EMAIL_ADDRESS>h1:Gz96sIWK3OalVv/I/qNygP42zyoKp3xptRVCWRFEBvo=
<EMAIL_ADDRESS>h1:5KslGYwFpkhGh+Q16bwMP3cOontH8FOep7tGV86Y7SQ=
<EMAIL_ADDRESS>h1:WUoyKPm6nCo1BnNUvPGnFG3T5DUVem42yDJZZ4CNxMA=
<EMAIL_ADDRESS>h1:C0nyHiBU2m0cR6hDiUORWqQIt3h37wsp1255QBSSXqY=
<EMAIL_ADDRESS>h1:go1bK/D/BFZV2I8cIQd1NKEZ+0owSTG1fDTci4IqFcE=
<EMAIL_ADDRESS>h1:ws8AfbgTX3oIczLPNPCu5166oBg9ST2vNs0rcht+mDE=
<EMAIL_ADDRESS>h1:bi/1aS/5W00E2ny5q65w9SnKpWEF/UIOqDYBILpo9rA=
<EMAIL_ADDRESS>h1:59Olnbt67UKpxF1EwVBopJvkSUBmgtb468E4GVWIZ1I=

Run code -v or code-insiders -v to get version of VS Code or VS Code Insiders.

1.60.2
7f6ab5485bbc008386c4386d08766667e155244e
x64

Check your installed extensions to get the version of the VS Code Go extension

v0.28.1

Describe the bug

When performing an autocomplete, vscode-go replaces the word under the cursor, even though the VSC editor.suggest.insertMode setting is set to "insert".
When editor.suggest.insertMode is set to "insert", the word under the cursor should remain in-place, as demonstrated in this video.

Example
Let's say I was editing the following line, where ✎ represents the current cursor position:
if ✎filepath.Ext(myPath) == ".txt" {

I want to change the comparison to lowercase, so I type strings. at which time the suggestions dropdown appears.  If I select ToLower, then vscode-go will replace it as follows,
if strings.ToLower(✎).Ext(myPath) == ".txt" {

where it has replaced filepath with the suggestion.
But because my VSCode editor.suggest.insertMode setting is set to "insert", I would have expected,
if strings.ToLower(✎)filepath.Ext(myPath) == ".txt" {

where the suggestion has been inserted at the original cursor position.
Bonus points
In the above example, if vscode-go could instead wrap the expression at the cursor, as follows, it would result in even greater productivity:
if strings.ToLower(✎filepath.Ext(myPath)) == ".txt" {

Possible dupe of #40871

Thanks @muirdm, I think it makes sense to dedupe to #40871


add filter support add win32 support

use this plugin,
rollup-plugin-scss will error,
so i add filter support,
it can exclude scss folder

Thanks I'll release a new version here soon

Thanks I'll release a new version here soon
When publish in npm?

Thanks I'll release a new version here soon

When publish in npm?

When publish in npm? @gabemeola


Add Feature to Put Default Filename into Inputbox @Editor

Actual behavior
For now, if we download the file without importing any file, then the file will download which contain the placeholder text in the body part of the file,
and second thing is ,The input box which is for the custom filename is empty
Expected behavior
I think we should not give permission to the user to download the file before importing any file, So we can show an alert message to the user.
if the user import file and then press a download button, then inside the download file dialog we should provide the default filename in the input box, So if the user don't want to change the filename then it will directly click the download button,
Check out the below video which I implement in code
https://user-images.githubusercontent.com/53515714/112942083-3a351a00-914d-11eb-9e00-20665c03aabc.mp4

Actually, in most cases, the user would just copy-paste content from the internet and the need to import a txt file won't arise

Oh, that's true , i forgot it😅 i think like only we can import file.
Ok ok but if user import file then we can add default file name in input field . What about this?


Confusing function name - currentSong

The currentSong function is only intended to express whether or not a song has started - it doesn't return a value associated with the currently selected song. I suggested changing it to something like "songHasStarted" which is more clear.

I'm gonna take my own issue!


Added support for large values for gradient_steps to SAC, TD3, and TQC

Description
closes #14
Addresses #14 by replacing the python loop in _train() with a jax.lax.fori_loop construct for SAC, TD3, and TQC.
This avoids explicitly unrolling the loop during compile time, which significantly reduces compile times and the size of the compiled program for large gradient_steps values.
Since performance was a concern in https://github.com/araffin/sbx/issues/14#issuecomment-1687684993, I did some benchmarking. To not exacerbate the influence of the compile time, I trained each agent for 1 million steps. I used different values for train_freq (with gradient_steps=-1) to highlight the differences between the implementations.
I used the following script for benchmarking.
import argparse
import time

from sbx import SAC, TD3, TQC

parser = argparse.ArgumentParser()
parser.add_argument("algo", type=str)
parser.add_argument("train_freq", type=int)
parser.add_argument("--steps", type=int, default=1_000_000)
args = parser.parse_args()

algo_class = eval(args.algo.upper())

start = time.time()
algo = algo_class(
    "MlpPolicy",
    "Walker2d-v4",
    train_freq=args.train_freq,
    gradient_steps=-1,
    tensorboard_log=f"tensorboard/{args.algo}_{args.train_freq}_{args.steps}",
)
algo.learn(args.steps)
print(f"{args.algo.upper()}, train_freq: {args.train_freq}, steps: {args.steps}: {time.time() - start:.2f}s")

I got the following results on Ubuntu 22.04 with an NVIDIA RTX A6000 GPU.
SAC:

train_freq and gradient_steps
Execution time in seconds (original)
Execution time in seconds (PR)
Relative speedup (torig / tpr)

1
2716
2713
100.1 %

4
1621
1659
97.7 %

16
1498
1490
100.5 %

64
1784
1360
130.2 %

128
2587
1256
206.0 %

TD3:

train_freq and gradient_steps
Execution time in seconds (original)
Execution time in seconds (PR)
Relative speedup (torig / tpr)

1
2281
2270
100.5 %

4
1315
1404
93.7 %

16
1094
1070
102.2 %

64
1103
950
116.1 %

128
1311
939
139.6 %

TQC:

train_freq and gradient_steps
Execution time in seconds (original)
Execution time in seconds (PR)
Relative speedup (torig / tpr)

1
2995
2936
102.0 %

4
2070
1926
107.5 %

16
2035
1721
118.2 %

64
2090
1443
144.8 %

128
3359
1418
236.9 %

For small gradient_steps values, the implementation is on par with the original implementation for all three algorithms. For large values, it leads to significant speed-ups due to faster compilation and smaller program sizes.
I also tried running the original implementation with gradient_steps=1000 but it ran out-of-memory during compilation on my 64GB RAM machine.
Motivation and Context

[x] I have raised an issue to propose this change (required for new features and bug fixes)

Types of changes

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] Documentation (update in the documentation)

Checklist:

[x] I've read the CONTRIBUTION guide (required)
[ ] I have updated the changelog accordingly (required).
I could not find the changelog (there is no docs/misc/changelog.rst)
[ ] My change requires a change to the documentation.
[ ] I have updated the tests accordingly (required for a bug fix or a new feature).
[ ] I have updated the documentation accordingly.
[x] I have reformatted the code using make format (required)
[x] I have checked the codestyle using make check-codestyle and make lint (required)
[x] I have ensured make pytest and make type both pass. (required)
[ ] I have checked that the documentation builds using make doc (required)

Note: You can run most of the checks using make commit-checks.
Note: we are using a maximum length of 127 characters per line

Hello,
thanks a lot for the PR =)
I'll try to look into it today.
I assume you also check that the performance was the same with this change?

I assume you also check that the performance was the same with this change?

I benchmarked the performance of the three algorithms (after the bugfix) on Walker2D-v4 for 10 random seeds. I used the default hyperparameter (the train_freq=1 case of the computation time benchmark above). Here are the results (mean and standard deviation across the seeds).

In all cases, the average performance is roughly the same for the original implementation and the PR. The standard deviation differs between the implementations for SAC and TD3 but I assume this is due to noise and that the difference would vanish for more seeds.

Hello,
I'm back =)
thanks for the benchmark, I will try to do one additional soon with train_freq=gradient_steps=12.

On-going report is here: https://wandb.ai/openrlbenchmark/sbx/reports/SBX-TD3-PR-21-perf-check--Vmlldzo2NDk2NDUy
Performance-wise, it looks good.
The compilation is much faster, however I observed a slower training after compilation compared to the current SBX version (we should probably play with loop unroll parameter that was introduced in newer jax version, but #22 must be fixed first).

Sorry for the late reply. I am a bit busy with deadlines at the moment, but I try to have a look at the loop unrolling next week.

After some local tests:

there seems to be a bug with newer jax versions (I get NaN when I try to unroll on CUDA)
the performance drop also depends on jax/cuda version, the compilation time too (latest jax version, 0.4.23 is slower to compile, yield NaNs when trying to unroll but is the fastest, no perf drop w.r.t. master)

Tested jax versions:

jaxlib==0.4.12+cuda11.cudnn86 (work with unroll but slower when using for i loo)
jaxlib==0.4.23+cuda11.cudnn86 (NaN with unroll, faster than master when using for i loop, couldn't test master though)

I would be happy if you could help isolate the NaN bug so we can send a minimal working example to Jax team.

After some local tests:

there seems to be a bug with newer jax versions (I get NaN when I try to unroll on CUDA)
the performance drop also depends on jax/cuda version, the compilation time too (latest jax version, 0.4.23 is slower to compile, yield NaNs when trying to unroll but is the fastest, no perf drop w.r.t. master)

Tested jax versions:

jaxlib==0.4.12+cuda11.cudnn86 (work with unroll but slower when using for i loo)
jaxlib==0.4.23+cuda11.cudnn86 (NaN with unroll, faster than master when using for i loop, couldn't test master though)

Hi @araffin, I was able to reproduce the NaN problem on my machine for jaxlib==0.4.23+cuda11.cudnn86. However, I noticed that the problem vanishes for jaxlib version 0.4.24. It also seems to be independent of whether the CUDA 11 or CUDA 12 version is used (works for both jaxlib==0.4.24+cuda11.cudnn86 or jaxlib==0.4.24+cuda12.cudnn89). So, I guess that the Jax people fixed the issue in the latest jaxlib version, but I could not find anything about it in the changelogs.
I will do some benchmarking of the performance with different values for the unroll parameter to figure out if it helps. I'll come back to you as soon as I have the data.

Thanks, I can confirm that the bug is gone with newest version of jax.
After some testing, it also seems that keeping the default unroll value is the best for speed.

I can confirm that observation. In my tests, the runs without loop unrolling were almost always faster (in terms of the time/fps metric). I also compared against the old implementation from before the PR. Previously, you reported that time/fps was slightly higher for the old implementation. In my experiments, this does not seem to be the case anymore. The new implementation was always faster, so perhaps the new version of Jax also improved the efficiency of the fori_loop construct in the meantime.
The results seem to be consistent across all three algorithms.

Previously, you reported that time/fps was slightly higher for the old implementation. In my experiments, this does not seem to be the case anymore.

yes, that was the case for older version of jax. Not the case anymore with newer versions.
Thanks for checking =)


CV::Merge binding

Hi,
I'm trying to add the binding for CV::Merge
cv::merge(InputArrayOfArrays mv, OutputArray dst)
But I don't know how to treat InputArrayOfArrays in coreBindings.h, I haven't found another function that uses them.
    class Merge : public CvClassMethodBinding<Mat> {
    public:
            void setup() {
                    auto mat = ???
                    auto dst = ret<Mat::Converter>("dst");
                    executeBinding = [=]() {
                            cv::merge(mat->ref(), dst->ref());
                    };
            };
    };

function as defined as: export function merge(mat: Mat[]): Mat;
Regards and thanks for your work

I do not remember how it woorks.
but As I told you find existing code with params and return type, and that will be easy.
the function is : void 	cv::merge (const Mat *mv, size_t count, OutputArray dst)
will be add to core_array.d.ts as
  export function merge (src: Mat, count?: number): Mat;
  export function mergeAsync(src: Mat, count?: number): Promise<Mat>;

or added in Mat.d.ts
export class Mat {
  merge (count?: number): Mat;
  mergeAsync(count?: number): Promise<Mat>;
}

The first argument is a Mat, so pick the first argument mapping from on another one.
like:
export function mulSpectrums(src1: Mat, src2: Mat, dftRows?: boolean, conjB?: boolean): Mat;
export function mulSpectrumsAsync(src1: Mat, src2: Mat, dftRows?: boolean, conjB?: boolean): Promise<Mat>;

this function also returns a Mat
or pick the code from:
  idct(flags?: number): Mat;
  idctAsync(flags?: number): Promise<Mat>;

the signature is the same.
From Mat.cc duplicate:
  Nan::SetPrototypeMethod(ctor, "idct", Idct);
  Nan::SetPrototypeMethod(ctor, "idctAsync", IdctAsync);

as
  Nan::SetPrototypeMethod(ctor, "merge ", Merge);
  Nan::SetPrototypeMethod(ctor, "mergeAsync", MergeAsync);

From Mat.cc duplicate:
NAN_METHOD(Mat::Idct) {
  FF::executeSyncBinding(
    std::make_shared<MatBindings::DCTWorker>(Mat::unwrapSelf(info)),
    "Mat::Idct",
    info
  );
}

NAN_METHOD(Mat::IdctAsync) {
  FF::executeAsyncBinding(
    std::make_shared<MatBindings::DCTWorker>(Mat::unwrapSelf(info)),
    "Mat::IdctAsync",
    info
  );
}

NAN_METHOD(Mat::Merge) {
  FF::executeSyncBinding(
    std::make_shared<MatBindings::MergeWorker>(Mat::unwrapSelf(info)),
    "Mat::Merge",
    info
  );
}

NAN_METHOD(Mat::MergeAsync) {
  FF::executeAsyncBinding(
    std::make_shared<MatBindings::MergeWorker>(Mat::unwrapSelf(info)),
    "Mat::MergeAsync",
    info
  );
}

From Mat.h duplicate:
  static NAN_METHOD(Idct);
  static NAN_METHOD(IdctAsync);

as
  static NAN_METHOD(Merge);
  static NAN_METHOD(MergeAsync);

MatBindings.h
Duplique the DCTWorker as MergeWorker, I see a // TODO in the code, which may cause some issues.
build It,
best It,
...
write a test...
I have some HTML stuff to write right now... So keep going.
add needed
#if CV_VERSION_GREATER_EQUAL(3, 2, 0)
if the function is not supper old.
#endif

I managed to bind another function without problems, but with Merge the first parameter is an array of Mats, not a Mat, and I can't get it to compile correctly:
../cc/core/coreBindings.h:266:80: error: no matching function for call to ‘merge(cv::Mat&, int&, cv::Mat&)’
266 |                                 cv::merge(mat->ref(), size_t->ref(), dst->ref());
|                                                                                ^
/node_modules/@u4/opencv4nodejs/node_modules/@u4/opencv-build/opencv-4.5.5-ecae4/build/include/opencv4/opencv2/core.hpp:945:17: note: candidate: ‘void cv::merge(const cv::Mat*, size_t, cv::OutputArray)’
945 | CV_EXPORTS void merge(const Mat* mv, size_t count, OutputArray dst);
|                 ^~~~~
/node_modules/@u4/opencv4nodejs/node_modules/@u4/opencv-build/opencv-4.5.5-ecae4/build/include/opencv4/opencv2/core.hpp:945:34: note:   no known conversion for argument 1 from ‘cv::Mat’ to ‘const cv::Mat*’
945 | CV_EXPORTS void merge(const Mat* mv, size_t count, OutputArray dst);
I can define the function with
export function merge (src: Mat[], count?: number): Mat;
But then I don't know how to bind the Array of Mats in coreBindings.h, doing it as if it were a Mat (auto mat = reqMat::Converter()) doesn't work.


deckstats: adding a \ if it is a , is XD

When importing cards into deckstats, the comma in the name is also used as a separator.
According to deckstats this can be escaped with \

obsolet in new version


Execute while planning

Description

Fixes #3562
Fixes #3563
Adds a second thread to the spinner in move_group.cpp, allowing two capabilities to run at once. This is intended to allow the TrajectoryExecutionManager to execute trajectories while planning is happening.
Makes PlanningSceneMonitor::updateSceneWithCurrentState return early if the mutex is locked, rather than waiting for the mutex. This allows the CurrentStateMonitor to continue updating while the PlanningSceneMonitor is locked, allowing the TrajectoryExecutionManager to grab fresh states while planning is happening.

Open questions:

Adding a second spinner to move_group.cpp is likely to allow a whole raft of new race conditions. Do we now need a set of mutexes around capabilities to prevent these?
Is this change to PlanningSceneMonitor the best way to avoid locking out the CurrentStateMonitor? Or do we want something a little more targeted to the specific callback the PlanningSceneMonitor adds to the CurrentStateMonitor?

Checklist

[ ] Required by CI: Code is auto formatted using clang-format
[ ] Extend the tutorials / documentation reference
[ ] Document API changes relevant to the user in the MIGRATION.md notes
[ ] Create tests, which fail without this PR reference
[ ] Include a screenshot if changing a GUI
[ ] While waiting for someone to review your request, please help review another open pull request to support the maintainers

:warning: Please install the  to ensure uploads and comments are reliably processed by Codecov.
Codecov Report
Attention: Patch coverage is 75.00000% with 1 line in your changes missing coverage. Please review.

Project coverage is 47.84%. Comparing base (6754954) to head (3b765a9).

Files with missing lines
Patch %
Lines

...nning_scene_monitor/src/planning_scene_monitor.cpp
66.67%
1 Missing :warning:

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #3676      +/-   ##
==========================================
+ Coverage   47.83%   47.84%   +0.02%
==========================================
  Files         604      604
  Lines       61018    61019       +1
  Branches     7018     7019       +1
==========================================
+ Hits        29183    29190       +7
+ Misses      31417    31411       -6
  Partials      418      418

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Sure, that's why I'm asking about mutexes. I'm thinking of something like a planning mutex and an executing mutex, so only planning+executing can happen concurrently, not e.g. planning+planning.


Dapr PubSub Usage

I need to use dapr with different server VMs, so not the same host.
I'm bit confused how pubsub uses redis or any other queue management tool
for now, let's assume I'm using declarative approach for subscribers.
We have a worker server, accessible from other servers, which doesn't need any component itself, just to be called when an event occurs and call the specified route.
let's have simple yaml under "./dapr-components " - subscription.yaml
apiVersion: dapr.io/v1alpha1
kind: Subscription
metadata:
  name: pubsub
spec:
  topic: files
  route: /files
  pubsubname: pubsub-vm

After running the worker server with following command (Rust app):
dapr run --app-id worker --app-port 3000 --components-path ./dapr-components -- cargo run

I assume it should be ready to receive events.

Now move to a publisher server (different VM), which is responsible to receive calls from outside world.
Here we have default config files:
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: pubsub-vm
spec:
  type: pubsub.redis
  version: v1
  metadata:
  - name: redisHost
    value: localhost:6379
  - name: redisPassword
    value: ""

So, I will just directly:
dapr run --app-id order --dapr-http-port 3501
Now, I'm using following url:
localhost:3501/v1.0/publish/pubsub-vm/files

but I'm not getting anything on the other end, worker server
Do,
I need pubsub config file on the other side as well?
Is there any good guide for non-localhost setup with multiple devices?

Yes, you need the pubsub config on the other side so that Dapr knows to connect to the pubsub and consume events from it.

Thanks Yaron,
So, having darp slim (without docket) would work in my case for subs?

Yes it should work for you.


Adding LogR

Older log msgs look inconsistent. Some start with lower case and others with upper case. All should start with the lower case. We will also fix this in the operator.


MNT: Added black to dev requirements

Resolves #14.

Codecov Report

Merging #15 (5a65b98) into main (9251ef2) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main      #15   +/-   ##
=======================================
  Coverage   87.50%   87.50%
=======================================
  Files           2        2
  Lines          16       16
=======================================
  Hits           14       14
  Misses          2        2

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9251ef2...5a65b98. Read the comment docs.


Coder should allow to define/change classes and comment besides methods

Current sketch:

Things left to do:

[ ] The meta view on objects does not show breadcrumbs. Maybe it should just be a full class coder anyway?
[ ] Definition and comment coders should have a collapsed view denoting the first line.
[ ] Definition and comment coders should listen to class change/add/remove events and update accordingly.
[ ] Comment coders should look the same as the comment view.
[ ] CRUD operations for all code elements

Currently, the test GToolkit.Coder.Examples.GtMethodsCoderExamples.validateNoMemoryLeaks fails. Interestingly, the subscriptions disappear after a while, i.e. after a while GtSearchMethodGroupResult allInstances is empty again. Not sure why this is not garbage-collected before, but whenever I look, I only see weak references in the system announcer that keep hold of the result.

I marked GToolkit.Coder.Examples.GtMethodsCoderExamples.validateNoMemoryLeaks as noTest for now.

Now it's green! Breadcrumbs in header save space as well.

Is there any reason the > in the breadcrumb are slightly lower than the content?

Is there any reason the > in the breadcrumb are slightly lower than the content?

@bjorn3 The reason is that text across different elements does not respect the text baseline of the parent yet. We will fix it as soon we have a shared baseline working.


Better use of descrition field

Is your feature request related to a problem? Please describe.
The workflow spec supports a description field but it doesn't seem to add much value at this point. It is only to describe the workflow. But on the overview of the workflow as well as on the workflow list I don't see any information about the workflow.
Describe the solution you'd like
The description should be shown on the overview page of the workflow but maybe it can be incorporated in the workflow list too so I don't have to click and look but just see what this workflow does.

Will be done in a bigger UI assessment


Single statement one-line control structures

If/else with one-line body can omit braces according to the style guide, but I get the following error:
Expected '{' after if clause

with the code:
modifier positive(uint256 amount) {
    if (amount <= 0)
       throw;
    _
}

solved in v0.1.9


[WIP] fix crash when using end-of-string symbol $ in regex VIRTUAL_HOST

This PR adds tests to show a case where the nginx configuration file produced is not valid and makes nginx fail to reload.

$ bats wildcard-hosts.bats
✓ [wildcard-hosts] start a nginx-proxy container
✓ [wildcard-hosts] VIRTUAL_HOST=*.wildcard.bats
✓ [wildcard-hosts] VIRTUAL_HOST=wildcard.bats.*
✓ [wildcard-hosts] VIRTUAL_HOST=~^foo\.bar\..*\.bats
✗ [wildcard-hosts] VIRTUAL_HOST=~^foo\.bar\..*\.bats$
  (from function `assert_output' in file lib/bats/batslib.bash, line 328,
   from function `assert_200' in file wildcard-hosts.bats, line 83,
   in test file wildcard-hosts.bats, line 65)
    `assert_200 foo.bar.whatever.bats' failed
    HTTP/1.1 503 Service Temporarily Unavailable
  ab2531a8a276
  container_name: bats-wildcard-hosts-4
  ports: 80
  options: -e VIRTUAL_HOST=~^foo.bar..*.bats$
  expose_option: --expose=80

  -- line differs --
  index    : 0
  expected : HTTP/1.1 200 OK
  actual   : HTTP/1.1 503 Service Temporarily Unavailable
  --

✓ [wildcard-hosts] stop all bats containers

 tests, 1 failure

$ docker start bats-nginx-proxy-wildcard-hosts
$ docker start bats-wildcard-hosts-4
$ docker logs bats-nginx-proxy-wildcard-hosts

dockergen.1 | 2017/01/07 22:48:10 Received event start for container 4aba463b7021
dockergen.1 | 2017/01/07 22:48:11 Generated '/etc/nginx/conf.d/default.conf' from 2 containers
dockergen.1 | 2017/01/07 22:48:11 Running 'nginx -s reload'
dockergen.1 | 2017/01/07 22:48:11 Error running notify command: nginx -s reload, exit status 1

$ docker exec bats-nginx-proxy-wildcard-hosts cat /etc/nginx/conf.d/default.conf
# If we receive X-Forwarded-Proto, pass it through; otherwise, pass along the
# scheme used to connect to this server
map $http_x_forwarded_proto $proxy_x_forwarded_proto {
  default $http_x_forwarded_proto;
  ''      $scheme;
}
# If we receive X-Forwarded-Port, pass it through; otherwise, pass along the
# server port the client connected to
map $http_x_forwarded_port $proxy_x_forwarded_port {
  default $http_x_forwarded_port;
  ''      $server_port;
}
# If we receive Upgrade, set Connection to "upgrade"; otherwise, delete any
# Connection header that may have been passed to this server
map $http_upgrade $proxy_connection {
  default upgrade;
  '' close;
}
gzip_types text/plain text/css application/javascript application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;
log_format vhost '$host $remote_addr - $remote_user [$time_local] '
                 '"$request" $status $body_bytes_sent '
                 '"$http_referer" "$http_user_agent"';
access_log off;
# HTTP 1.1 support
proxy_http_version 1.1;
proxy_buffering off;
proxy_set_header Host $http_host;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $proxy_connection;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $proxy_x_forwarded_proto;
proxy_set_header X-Forwarded-Port $proxy_x_forwarded_port;
# Mitigate httpoxy attack (see README for details)
proxy_set_header Proxy "";
server {
        server_name _; # This is just an invalid value which will never trigger on a real hostname.
        listen 80;
        access_log /var/log/nginx/access.log vhost;
        return 503;
}
upstream ~^foo.bar..*.bats$ {
                                ## Can be connect with "bridge" network
                        # bats-wildcard-hosts-4
                        server <IP_ADDRESS>:80;
}
server {
        server_name ~^foo.bar..*.bats$;
        listen 80 ;
        access_log /var/log/nginx/access.log vhost;
        location / {
                proxy_pass http://~^foo.bar..*.bats$;
        }
}

The issue is on one (or both) of those lines:
upstream ~^foo.bar..*.bats$ {

proxy_pass http://~^foo.bar..*.bats$;

If you remove the $ from the regular expression, nginx is happy with the conf.

I think I found a fix. Careful code review is needed: I'm very unfamiliar with the project's code.

Thanks @thomasleveil!


Getting error when navigating to Analytics section due to a issue in retrieving governance connector.

Describe the issue:

Getting error when navigating to Analytics section due to a issue in retrieving governance connector.
https://github.com/wso2/product-is/assets/40081059/235b8669-3e62-4a07-b48b-fba39e3f8f7e
How to reproduce:

1.Login to the console
2.Navigate to analytics section
Expected behavior:

Environment information (Please complete the following information; remove  any unnecessary fields) :

Product Version: [IS 7 legacy snapshot]
OS: [ Windows]
Database: [H2]

Optional Fields
Related issues:

Suggested labels:

This is reported with https://github.com/wso2/product-is/issues/18334


Allow to set empty graffiti

At the moment, it is not possible to set an empty graffiti value by providing --graffiti="" as omitting the graffiti or setting it to an empty string both results in the default graffiti being used which is Lodestar-<version>/<short-commit-sha>.
I think we should allow to explicitly set an empty graffiti as an operator might not want to use the default graffiti but also not a custom graffiti.
This is quite easy to change in terms of the implementation, it is just a matter of if we want this behavior or not.
Current code which disallows empty graffiti
https://github.com/ChainSafe/lodestar/blob/bc7759674f5c54013c238cb369a9d60c0635b5f9/packages/cli/src/cmds/validator/handler.ts#L184-L185
Fix to allow empty graffiti
 const defaultConfig = {
   graffiti: args.graffiti ?? getDefaultGraffiti(),

Sounds good to me


ffe-buttons-react: Make TertiaryButton also support rightIcon

Steps to reproduce:

Select a date in the calendar widget
Use the next or previous button to change month

Expected behavior: the date should be rendered as selected only for the actual month I selected
Actual behavior: the date is rendered as selected for all months

Hm, is the issue title a bit off here @wkillerud

Waaay off. Copy paste gone wrong. Will fix.

The bug title and description is not really correct here.
While it's not completely wrong, it's important to notice there are two modifiers for ffe-calendar__date: selected and focus.
When you navigate the months, after selecting a date, it's only the focus modifier that stays on the day-of-month you selected. The selected modifier correctly renders only on the month where you actually selected a date.
The following image shows the difference between the two modifiers:

Dark blue background for selected, blue border for focus.
Given this information

is this correct behaviour?
if not, should we fix it? I ask this question because I suspect it may cause more complexity for keyboard navigation.

I think the way you described it made sense - is this really a bug?

@henrikhermansen Aha, that makes sense. This issue is from the internal archives :female_detective: Sounds like it's working as intended. I'll close the issue as wontfix.


I can't delete the overlay layers in map like circle, rectangle ... etc. only markers is deleted ...

[x] I'm reporting a bug, not asking for help
[x] I've looked at the documentation to make sure the behaviour is documented and expected
[x] I'm sure this is a Leaflet Draw code issue, not an issue with my own code nor with the framework I'm using (Cordova, Ionic, Angular, React…)
[x] I've searched through the issues to make sure it's not yet reported

How to reproduce

Leaflet version I'm using:
Leaflet Draw version I'm using:
Browser (with version) I'm using:
OS/Platform (with version) I'm using:
step 1
step 2

What behaviour I'm expecting and which behaviour I'm seeing
Minimal example reproducing the issue

[ ] this example is as simple as possible
[ ] this example does not rely on any third party code

Using jsfiddle or another example site.

I have a similar issue. If I add some ESRI feature layers for example and one of the esri feature is overlapping on the draw layer, delete doesnot work for polygons etc and works for marker


⚠ Bofrost.de has degraded performance

In cebc533, Bofrost.de (https://www.bofrost.de) experienced degraded performance:

HTTP code: 200
Response time: 1754 ms

Resolved: Bofrost.de performance has improved in 0a5587f after 9 minutes.


TransformOperationTransition bug

Describe the bug
this bug occurs with TransformOperationTransition and :pointerover psudoclass, when the pointer left the control, the transition still executing. Occurs with 0.10.1 and 0.10.2, but not with previous versions.
To Reproduce
Use the follow styles:
<Style Selector="Control"> //any control can be affect
   <Setter Property="Transitions">
      	<Transitions>
		<TransformOperationsTransition Property="RenderTransform" Easing="ExponentialEaseOut"
                                                                     Duration="0:0:0.2"/>
	</Transitions>
   </Setter>
</Style>
<Style Selector="Control:pointerover">
   <Setter Property="RenderTransform" Value="scale(0.95)"/>
</Style>

Expected behavior

Current Behavior

Desktop (please complete the following information):

OS: Windows 10 20H2
Version: Avalonia 0.10.2

Transform transitions are a bit tricky currently, you might want to try using none instead of scale(1). Although both should work.
I will take a look sometime during this week.

THANKSSSSSS!!! @MarchingCube  Works very fine 👌


Update TINYMCE_DEFAULT_CONFIG like a dictionary

I wonder if there is a way I can update TINYMCE_DEFAULT_CONFIG without copying the default into my project's settings.py.
I would prefer not to copy the entire TINYMCE_DEFAULT_CONFIG so that when there is any update to this package, the TINYMCE_DEFAULT_CONFIG will always be the latest default configuration.
Specific custom change I want:  enable menubar.
I tried the following
# method 1
TINYMCE_DEFAULT_CONFIG["menubar"] = True

# method 2
TINYMCE_DEFAULT_CONFIG.update({"menubar": True})

The above attempt don't work as NameError: name 'TINYMCE_DEFAULT_CONFIG' is not defined.
I wonder if I can import the default TINYMCE_DEFAULT_CONFIG from some path in order for the above code to work.
Thanks!

You can try adding from tinymce.settings import TINYMCE_DEFAULT_CONFIG before updating it.
You might endup with a circular import though.


Add nanoseconds to block time

Merging this into master now. There is a new 0.10 branch for 0.10.x patches


CLI workspaces

This starts to adapt some of the logic from #1707 to do with swapping workspaces.
One of the issues I'm hitting is not quite knowing what logic is best to implement. Currently, I've implemented the logic from the issue, without the "workspace.autoDetectWorkspace" logic.
That is, in the current form of the PR, the logic is as follows, in order of priority:

CLI : Use first given folder, use first given file, use location oni was called from.
Non-CLI (unchanged): Use first given folder, use first given file.

Neither of these can be changed with a config option, is that something we'd want?
Things I'd potentially think are useful would be the ability to never use the location oni was called from, and the ability to always/never use the detection logic.
I'll tag @badosu since I know he had some thoughts about this over in #2163.
Anyone any ideas?

Neither of these can be changed with a config option, is that something we'd want?

The behavior sounds reasonable to me (and in-line with expections of CLI behavior for similiar apps). So I'd lean towards not adding configuration for now; if there ends up being a need for it, we can add it later.

Things I'd potentially think are useful would be the ability to never use the location oni was called from, and the ability to always/never use the detection logic.

There could potentially be the workaround today of adding logic in the plugin load to force a specific workspace, too - so it may be that we don't even need a configuration setting to accomodate these scenarios. For example, I could force a workspace via oni.workspace.changeDirectory("/my/permanent/folder") in the activate method of my plugin.
The plugin also has access to the arguments via process.argv (although this is serendipity, as we host the Oni plugins in the renderer process) - so it could do custom processing. I think it would make sense to make the CLI arguments available somewhere on our API too, though.

Hoping to get this updated, tested and then merged, since there wasn't any extra feedback that the other stuff was needed.
I'll also have a look into the dev setup for all this, since my scripts are all aimed at the production build, but I'm now realising they should also work for the users who run master but without building fully.

Okay, I've finally been able to tidy this up.
I'll need to stick up a PR for oni-api, and check the functionality a bit more, but this adds what the PR initial message says + support for using NETRW. (Ie passing a folder in and it will open netrw on that folder).

Okay, I've finally been able to tidy this up.

I'll need to stick up a PR for oni-api, and check the functionality a bit more, but this adds what the PR initial message says + support for using NETRW. (Ie passing a folder in and it will open netrw on that folder).

Thanks for all the work on this, @CrossR ! Excited to see it go in 👍


Creating a new project from the command-line

Hi,
I seem to recall (incorrectly?) that there's some sort of cmdline goodness you can use to get kha to generate a brand new empty project for you.
I can't find any documentation or notes on this. Am I wrong, or is there a way to do this?
I found this in your source, which seems to suggest haxelib run kha init generates a new project, but on my machine I just get an error:

Neither Kha directory nor project file (khafile.js) found.

(I uninstalled and reinstalled kha via haxelib to make sure it's not something environmental.)

You almost had it, it's --init.

D;'oh. At least it's now permanently documented in an issue  so I can look it up.
Thanks!

Hey @RobDangerous , I was about to add this to the Getting Started wiki page, but I notice that you removed it a couple of revisions ago. (There's a haxelib link but no haxelib section.)
Any particular reason why? Any objection if I reinstate it?

Using the Haxelib version of Kha is not presently advised, which is why all references to it have either been removed or annotated with a warning.
Haxelib Kha is purposefully kept outdated to ensure compatibility with Haxe 3.2.1, and will not be updated until the next stable release of Haxe.
At least that's how I understand it.

Aye, using the haxelib version of Kha is currently strongly discouraged.


Industry Sector - Harmonizing with FORECAST

too highly negative emissions in 2045 in industry (harmonise with FORECAST related to biomass, i.e. reduce biomass usage?)
too much coal in 2045 (esp. Industry)

Following possibilities to harmonize FORECAST with PyPSA-Ariadne:
Take the production volumes from FORECAST
This would leave the industry sector from PyPSA-Eur mostly untouched. The same sector ratios (energy demands per produced unit) would be used to calculate final energy demand in the industrial sector.
Problem: only small deviations between the scenarios from FORECAST since the production [Mt/yr] is expected to stay the same except for the Easy Ride and Low Demand scenarios.

Calculate FORECAST specific sector ratios for each year
The workflow in PyPSA-Eur would stay mostly the same, however the files industry_sector_ratios_{planning_horizon}.csv would change depending on the FORECAST results.
Alternative a)
Stay very closely to FORECAST and specify even the source of gas (biomass vs. natural gas) and liquids (biomass vs. petroleum vs synfuels).
Alternative b)
Give PyPSA-Ariadne more options to cover the demand by leaving it up to the model where e.g. the natural gas is coming from.
The sectors do not match entirely between the JRC IDEES and the FORECAST model.

PyPSA-Eur sector ratios
FORECAST

Electric arc
Steel

DRI + Electric arc
Steel

Integrated steelworks
Steel

HVC

HVC (mechanical recycling)

HVC (chemical recycling)

Ammonia
Chemicals

Chlorine
Chemicals

Methanol
Chemicals

Other chemicals
Chemicals

Pharmaceutical products etc.

Cement
Non-Metallic Minerals

Ceramics & other NMM
Non-Metallic Minerals

Glass production
Non-Metallic Minerals

Pulp production
Pulp and Paper

Paper production
Pulp and Paper

Printing and media reproduction
Pulp and Paper

Food, beverages and tobacco
Food and Tobacco

Alumina production
Non-Ferrous Metals

Aluminium - primary production
Non-Ferrous Metals

Aluminium - secondary production
Non-Ferrous Metals

Other non-ferrous metals
Non-Ferrous Metals

Transport Equipment

Machinery Equipment

Textiles and leather

Wood and wood products

Other Industrial Sectors

[ ] FORECAST would need to report more variables

Taking Demands from FORECAST
Most invasive way is taking the final energy demand Final Energy|Industry|Carrier from FORECAST and setting it as a demand in PyPSA-Ariadne. We would outsource the modelling of the industrial sector completely and would allow PyPSA-Ariadne only how to supply hydrogen, gas etc.

[ ] What does FORECAST mean by Final Energy|Industry|Heat

After consideration we decided on:

taking production volumes for Germany from FORECAST (which is already implemented)
taking our own assumption for the industry transformation
[ ] take DRI ratio from FORECAST
[ ] add config to vary the energy source for industry heat from currently only biomass to biomass, hydrogen and electricity (following scenario variations from FORECAST)


'Word2VecKeyedVectors' object has no attribute 'key_to_index'

Hi, how to fix  'Word2VecKeyedVectors' object has no attribute 'key_to_index' after loading xxx.bin file in w2v = KeyedVectors.load_word2vec_format(self.w2v_path, **self.w2v_kwargs)

update gensim发自我的iPhone------------------ Original ------------------From: chendeming @.>Date: Wed,May 26,2021 8:13 PMTo: shibing624/text2vec @.>Cc: Subscribed @.***>Subject: Re: [shibing624/text2vec] 'Word2VecKeyedVectors' object has no attribute'key_to_index' (#23)
Hi, how to fix  'Word2VecKeyedVectors' object has no attribute 'key_to_index' after loading xxx.bin file in w2v = KeyedVectors.load_word2vec_format(self.w2v_path, **self.w2v_kwargs)
—You are receiving this because you are subscribed to this thread.Reply to this email directly, view it on GitHub, or unsubscribe.
[
{
@.": "http://schema.org",
@.": "EmailMessage",
"potentialAction": {
@.": "ViewAction",
"target": "https://github.com/shibing624/text2vec/issues/23",
"url": "https://github.com/shibing624/text2vec/issues/23",
"name": "View Issue"
},
"description": "View this Issue on GitHub",
"publisher": {
@.": "Organization",
"name": "GitHub",
"url": "https://github.com"
}
}
]

更新gensim到4.0.0版本


Incorrect results about Go-to-definition and Go-to-declaration

Link: https://github.com/clangd/vscode-clangd/issues/634
Hi, I found a bug about the go to definition and go to declaration when used clangd in vscode. Please see the code example below:
#include  <stdio.h>
struct str {
    int v;
};
int main() {
    str s;
    s.v = 10;
    return 0;
}

I performed Go-to-def/decl on the variable s in declaration statement at line 6, but it jumps to the variable s at line 7. Actually, I think this is Go to Use or Find References, rather than Go-to-def or decl (should be unchanged at line 6). So, this is a bug.
System information
Clangd version:
Apple clangd version 15.0.0 (clang-15<IP_ADDRESS>)
Features: mac+xpc
Platform: x86_64-apple-darwin23.5.0; target=arm64-apple-darwin23.5.0
clangd extension version: Version 0.1.28: March 20, 2024
Operating system: MacOS

See the answer here: https://github.com/clangd/vscode-clangd/issues/634#issuecomment-2130749041
This is expected VS Code behavior. I will close the issue.


Colonies disappear when loading a quick save

Check why colonies disappear permanently when the user loads a quick save

The problem could not be replicated.


Support user-defined lifted types

This feature has several advantages over v0.6.1:

Magnolia would no longer need to be a runtime dependency of any libraries which use it,
implicit evidence (of some type other than the typeclass being derived) could be requested for each parameter,
runtime and compile-time performance could be improved by reducing garbage generation

The macro currently instantiates CaseClass, SealedTrait, Subtype and Param instances which it passes to the user-defined method, combine. Magnolia provides fully-functional versions of these types. That is, including all the features which may be utilized in any given derivation, regardless of whether they are actually used or not.
However, there is scope for alternative implementations of these types to be used, and it should be possible for users to abstract over the instantiation of these types in general ways, for example by resolve additional typeclass instances on the type of the parameter (which is not currently possible from within the combine method definition, as the parameter type is existential by that point).
I propose to give users the option of providing their own implementations of these types, to be instantiated by new methods provided on the derivation object, namely,

param,
caseClass,
sealedTrait, and
subtype

As with combine and dispatch, the signatures of these methods would not have to adhere to a fixed interface (in the object-oriented sense), and could take additional type parameters or implicit parameters, but would be invoked by the macro which would inspect their parameters' names, and supply values only to those parameters in the signature. Any parameters (e.g. support for default values) not listed as the parameter would not be generated.
The first advantage would be that a number of ephemeral objects which exist only for covariant or only for contravariant typeclasses would not need to be instantiated for all cases, even when they are not used in the derivation. This should be beneficial for both compile time and runtime performance.
The second advantage would be that user-defined classes could be used in preference to Magnolia's default implementations, thereby obviating the need for Magnolia to be a runtime dependency. Given the library is likely to be useful for a number of other projects, the risk of a diamond dependency problem could be eliminated, while for application (i.e. non-library) developers, the tersity of the status quo would apply.
The existing API would remain unchanged if users did not provide any of the custom methods.

Nice to see this being worked on :)


Finish updating core nodes for V2 on conda-forge

As far as I can tell, here's what I need to take care of tomorrow to make things look the way we'd like prior to submitting the core nodes recipe to conda-forge for approval:

[x] Core nodes library: update all "core" imports to "gpi_core"
[x] Remove .gpi_node_lib_config (switching to name recognition)
[x] Update the core nodes feedstock to have the appropriate name
[x] Build the core-nodes without testing
[x] Framework: change default libraries to search for anything called gpi_
[x] Make a new build of the updated framework
[x] Install and test to see if things work / how the nodes appear
[ ] Update framework to strip the "gpi_" at the start of condo-forge packaged node libs
[ ] Another framework build
[ ] Test and build test network, upload network to core-nodes
[ ] Final core-nodes build with testing added back
[ ] Restore build config matrix for linux/osx and python versions
[ ] Fork staged-recipes and upload our new recipe for testing.

@aganders3 I want to move discussion here rather than on a random commit.
If you want to test where we're at, you can install gpi=1.1.1rc4 and gpi_core from my channel (danielborup) on anaconda.
I'm playing with this now, and I actually like the idea of keeping the full library name (like gpi_core) when browsing nodes within GPI. It makes it clear which nodes are "official" in that they're packaged on conda-forge, and makes clear what name needs to be passed if someone wants to import those packages elsewhere. There are a few places in our internal MRTUD repo, for example, that have an import core statement that we'll need to change — this is less obvious if the node library still showed up as core within GPI.

I'm playing with this now, and I actually like the idea of keeping the full library name (like gpi_core) when browsing nodes within GPI. It makes it clear which nodes are "official" in that they're packaged on conda-forge, and makes clear what name needs to be passed if someone wants to import those packages elsewhere. There are a few places in our internal MRTUD repo, for example, that have an import core statement that we'll need to change — this is less obvious if the node library still showed up as core within GPI.

Yeah I guess that's fine - still a bit strange to me if you want to keep a dev version as well since it will have to go in ~/gpi/gpi_core but I can get over that.

Yeah I guess that's fine - still a bit strange to me if you want to keep a dev version as well since it will have to go in ~/gpi/gpi_core but I can get over that.

What if we have the framework create a directory like ~/gpi/packaged_nodes that contains symlinks to the site-packages folders without the gpi_ prefix? Then we could leave the node library names and import/include statements alone, right?

I think that's overcomplicating it. I think there are benefits to doing it this way and the only cons so far are the friction to change and a slight aesthetic preference.

Sounds good — will go forward with this!


Hot reload does not work when multiple features are loaded

I am running the following command to load multiple mock features:
java -jar karate-1.4.0.jar \
  --mocks=init/post-sample.feature,init/FileServerMock.feature \
  -W \
  -p 8100

Unfortunately, when I change the content of init/post-sample.feature it is not hot reloaded.
Instead, if I just load post-sample.feature it is correctly hot reloaded when its content changes.
java -jar karate-1.4.0.jar \
  -m init/post-sample.feature \
  -W \
  -p 8100

post-sample.feature before:
Feature:

Scenario: pathMatches('/sample') && methodIs('post')
* def responseStatus = 200
* def response =
  """
  {
    "code": "000",
    "message": "success",
    "result": true
  }

post-sample.feature after:
Feature:

Scenario: pathMatches('/sample') && methodIs('post')
* def responseStatus = 400
* def response =
  """
  {
    "code": "106",
    "message": "Invalid request",
    "result": false
  }

@gturi quite likely we don't support this use-case. we welcome code contributions to fix this


Correct integer types

Clean up a few more files using isize and usize in error.
This came up in #283. These were low-hanging fruit, there is still more to be done before #283 is fixed.

Grand, cheers


Issue on node-gyp rebuild trying to install jt400, can you help me ?

Hi there, I am trying to install jt400 and got this error on the node-gyp rebuild,  Do you know if there is any other solution ?
Thanks.
npm install jt400
npm WARN deprecated<EMAIL_ADDRESS>Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue

><EMAIL_ADDRESS>install /Users/jorgeramos/git/invoices-admin/node_modules/java
> node-gyp rebuild

gyp WARN download NVM_NODEJS_ORG_MIRROR is deprecated and will be removed in node-gyp v4, please use NODEJS_ORG_MIRROR
gyp WARN download NVM_NODEJS_ORG_MIRROR is deprecated and will be removed in node-gyp v4, please use NODEJS_ORG_MIRROR
gyp WARN download NVM_NODEJS_ORG_MIRROR is deprecated and will be removed in node-gyp v4, please use NODEJS_ORG_MIRROR
  CXX(target) Release/obj.target/nodejavabridge_bindings/src/java.o
In file included from ../src/java.cpp:1:
In file included from ../src/java.h:9:
In file included from ../../nan/nan.h:182:
../../nan/nan_maybe_43_inl.h:221:17: warning: 'CloneElementAt' is deprecated [-Wdeprecated-declarations]
  return array->CloneElementAt(GetCurrentContext(), index);
                ^
/Users/jorgeramos/.node-gyp/6.4.0/include/node/v8.h:3012:36: note: 'CloneElementAt' has been explicitly marked deprecated here
                MaybeLocal<Object> CloneElementAt(Local<Context> context,
                                   ^
In file included from ../src/java.cpp:1:
In file included from ../src/java.h:9:
In file included from ../../nan/nan.h:188:
In file included from ../../nan/nan_new.h:189:
../../nan/nan_implementation_12_inl.h:40:29: warning: 'New' is deprecated [-Wdeprecated-declarations]
  return v8::BooleanObject::New(value).As<v8::BooleanObject>();
                            ^
/Users/jorgeramos/.node-gyp/6.4.0/include/node/v8.h:3976:56: note: 'New' has been explicitly marked deprecated here
  V8_DEPRECATED("Pass an isolate", static Local<Value> New(bool value));
                                                       ^
In file included from ../src/java.cpp:1:
In file included from ../src/java.h:9:
../../nan/nan.h:590:20: error: no type named 'GCEpilogueCallback' in 'v8::Isolate'
      v8::Isolate::GCEpilogueCallback callback
      ~~~~~~~~~~~~~^
../../nan/nan.h:596:20: error: no type named 'GCEpilogueCallback' in 'v8::Isolate'
      v8::Isolate::GCEpilogueCallback callback) {
      ~~~~~~~~~~~~~^
../../nan/nan.h:601:20: error: no type named 'GCPrologueCallback' in 'v8::Isolate'
      v8::Isolate::GCPrologueCallback callback
      ~~~~~~~~~~~~~^
../../nan/nan.h:607:20: error: no type named 'GCPrologueCallback' in 'v8::Isolate'
      v8::Isolate::GCPrologueCallback callback) {
      ~~~~~~~~~~~~~^
../../nan/nan.h:1927:15: warning: 'SetAccessor' is deprecated [-Wdeprecated-declarations]
  return obj->SetAccessor(
              ^
/Users/jorgeramos/.node-gyp/6.4.0/include/node/v8.h:2736:22: note: 'SetAccessor' has been explicitly marked deprecated here
                bool SetAccessor(Local<Name> name,
                     ^
../src/java.cpp:897:40: warning: 'ToNumber' is deprecated [-Wdeprecated-declarations]
  v8::Local<v8::Number> val = info[0]->ToNumber();
                                       ^
/Users/jorgeramos/.node-gyp/6.4.0/include/node/v8.h:7962:22: note: 'ToNumber' has been explicitly marked deprecated here
Local<Number> Value::ToNumber() const {
                     ^
../src/java.cpp:927:40: warning: 'ToNumber' is deprecated [-Wdeprecated-declarations]
  v8::Local<v8::Number> val = info[0]->ToNumber();
                                       ^
/Users/jorgeramos/.node-gyp/6.4.0/include/node/v8.h:7962:22: note: 'ToNumber' has been explicitly marked deprecated here
Local<Number> Value::ToNumber() const {
                     ^
../src/java.cpp:956:40: warning: 'ToNumber' is deprecated [-Wdeprecated-declarations]
  v8::Local<v8::Number> val = info[0]->ToNumber();
                                       ^
/Users/jorgeramos/.node-gyp/6.4.0/include/node/v8.h:7962:22: note: 'ToNumber' has been explicitly marked deprecated here
Local<Number> Value::ToNumber() const {
                     ^
../src/java.cpp:983:42: warning: 'ToNumber' is deprecated [-Wdeprecated-declarations]
    v8::Local<v8::Number> val = info[0]->ToNumber();
                                         ^
/Users/jorgeramos/.node-gyp/6.4.0/include/node/v8.h:7962:22: note: 'ToNumber' has been explicitly marked deprecated here
Local<Number> Value::ToNumber() const {
                     ^
../src/java.cpp:1019:40: warning: 'ToNumber' is deprecated [-Wdeprecated-declarations]
  v8::Local<v8::Number> val = info[0]->ToNumber();
                                       ^
/Users/jorgeramos/.node-gyp/6.4.0/include/node/v8.h:7962:22: note: 'ToNumber' has been explicitly marked deprecated here
Local<Number> Value::ToNumber() const {
                     ^
../src/java.cpp:1044:40: warning: 'ToNumber' is deprecated [-Wdeprecated-declarations]
  v8::Local<v8::Number> val = info[0]->ToNumber();
                                       ^
/Users/jorgeramos/.node-gyp/6.4.0/include/node/v8.h:7962:22: note: 'ToNumber' has been explicitly marked deprecated here
Local<Number> Value::ToNumber() const {
                     ^
../src/java.cpp:1263:16: warning: 'TryCatch' is deprecated [-Wdeprecated-declarations]
  v8::TryCatch tryCatch;
               ^
/Users/jorgeramos/.node-gyp/6.4.0/include/node/v8.h:6583:40: note: 'TryCatch' has been explicitly marked deprecated here
  V8_DEPRECATED("Use isolate version", TryCatch());
                                       ^
10 warnings and 4 errors generated.
make: *** [Release/obj.target/nodejavabridge_bindings/src/java.o] Error 1
gyp ERR! build error
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/Users/jorgeramos/.nvm/versions/node/v6.4.0/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:276:23)
gyp ERR! stack     at emitTwo (events.js:106:13)
gyp ERR! stack     at ChildProcess.emit (events.js:191:7)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:204:12)
gyp ERR! System Darwin 16.4.0
gyp ERR! command "/Users/jorgeramos/.nvm/versions/node/v6.4.0/bin/node" "/Users/jorgeramos/.nvm/versions/node/v6.4.0/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /Users/jorgeramos/git/invoices-admin/node_modules/java
gyp ERR! node -v v6.4.0
gyp ERR! node-gyp -v v3.4.0
gyp ERR! not ok
npm WARN<EMAIL_ADDRESS>No repository field.
npm ERR! Darwin 16.4.0
npm ERR! argv "/Users/jorgeramos/.nvm/versions/node/v6.4.0/bin/node" "/Users/jorgeramos/.nvm/versions/node/v6.4.0/bin/npm" "install" "jt400"
npm ERR! node v6.4.0
npm ERR! npm  v3.10.10
npm ERR! code ELIFECYCLE

npm ERR<EMAIL_ADDRESS>install: `node-gyp rebuild`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the<EMAIL_ADDRESS>install script 'node-gyp rebuild'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the java package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     node-gyp rebuild
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs java
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!     npm owner ls java
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     /Users/jorgeramos/git/invoices-admin/npm-debug.log

how did you solve that problem?
I had to use the command : npm install node-jt400, but i don't know if is different.


how to export 3d file?

Hi, I successfully ran the launch_speedup.py file for training, but when I tried to export a 3D file by changing -- train to -- export, I found that the program could not run. I carefully checked the launch_speedup.py and launch_baseline.py in the project and found that there were export parameters inside, but never used. When I tried to export 3D files using launch. py in threshtudio, I found it to be incompatible with this project. So how can I export 3D files, or does the project not provide this function?

Sorry for the delay! You are correct I did not implement export 3D files. I only checked correctness of visualizing via video. If you manage to figure out how to export 3D model, please consider submit a pull request!


feat:Goctl can't generate the type of decimal.

As we all know, if using int or float instead of decimal, there may be precision loss, which is fatal for finance. We hope thatdevelopers can enable goct to generate the type of decimal.Decimal to resolve this question.

see features #4062


request to pin cfroutesync image

We're currently overriding your default with "gcr.io/cf-networking-images/cf-k8s-networking/cfroutesync@sha256:758abd1b6144596cef74ad4805bc7e1a6055142cab5719f0d42d0321a14a190d" but we'd prefer that y'all dictate the preferred image SHA, if that seems desirable to you.
The current line of code pointing to "latest" is here: https://github.com/cloudfoundry/cf-k8s-networking/blob/master/config/cfroutesync/values.yaml#L10
Best,
relint

This issue was fixed by this PR.

We now have a CI task that will keep that image sha up to date. I think it's okay for you to remove your override now.

Oh cool, looks like we first need to get cf-k8s-networking updated in cf-for-k8s. I think it'd make sense to remove our override as part of that PR.
Do y'all have plans to bump networking in cf-for-k8s anytime soon? If you have a story or want to kick that off, it might be a good idea to add a task/reminder to remove our override here: https://github.com/cloudfoundry/cf-for-k8s/blob/master/config/values.yml#L30


tflite model produces different results if using the CPU on a Ubuntu PC or a NPU+NNAPI on an embedded system

Please go to Stack Overflow for help and support:
https://stackoverflow.com/questions/tagged/tensorflow
If you open a GitHub issue, here is our policy:

It must be a bug, a feature request, or a significant problem with the
documentation (for small docs fixes please send a PR instead).
The form below must be filled out.
It shouldn't be a TensorBoard issue. Those go
here.

Here's why we have that policy: TensorFlow developers respond to issues. We want to focus on work that benefits the whole community, e.g., fixing bugs and adding features. Support only helps individuals. GitHub also notifies thousands of people when issues are filed. We want them to see you communicating an interesting problem, rather than being redirected to Stack Overflow.

System information

Have I written custom code (as opposed to using a stock example script
provided in TensorFlow): no
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 18.04 / Linux Yocto
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue
happens on a mobile device:
TensorFlow installed from (source or binary): binary
TensorFlow version (use command below): 2.4.1
Python version: 3.6
Bazel version (if compiling from source):
GCC/Compiler version (if compiling from source):
CUDA/cuDNN version:
GPU model and memory:
Exact command to reproduce:

Describe the problem
I trained a classification network and I performed quantization-aware training using Tensorflow Model Optimization. Then I converted the trained and quantized model to tflite format.
The model is supposed to be used in an embedded system running Yocto. The system is provided with a NPU and the inference is performed with the tflite_runtime package and the NNAPI.
The problem is that the same tflite model produces different results if tested on my PC with TF2.4.1 and on the Yocto board. The result are often very similar but there are some cases where they are completely different, resulting in an accuracy drop on the board. I don't know why this happens. Is there a way to prevent it during training?

@FSet89 ,
We see that the issue template has not been filled.
Also for analysing the issue can you please provide the stand alone code or share a colab gist with the issue reported.
Thanks!


Fix GLX creating only 3.0 contexts

Fix #16
Would be nice if someone could test this. I'm currently having some issues with my drivers, and only have OpenGL 2.1.

@vberger Reported getting a 3.3 context \o/
It would be nice if someone else could test it, otherwise I will merge.


refactor(new): naming schema: use format Org.Product.Client.Feature.Rest

This will be the first of a series of pull requests. This PR uses the org.product.client.feature.(...) as a basis for package names, namespaces and assembly names.
This had a direct impact on the business package assembly names.
This PR also adds numerous tests for the scaffolding operations. Some tests are tagged with the trait "LongRunning" as they take several minutes to run. These tests are exempt from being run on each commit, but they will still be executed in every pull request.

Hi Marco, please remove this
using Cmf.Navigo.BusinessObjects;
using Cmf.Navigo.BusinessOrchestration.LaborManagement.InputObjects;
from the base context as it's not needed nor used


Renderer's theme overriding app's styles

It has been reported that when an app is consuming the Smart Forms renderer library, the background colours in the parent app is overwritten.
Might be something to do with Material UI's themes which enforces styles.

Seems like <CssBaseline /> is the culprit.
Apparently it provides baseline MUI styles for the app, which seems to make sense that it overrides the styles defined in the app previously. For more reading: https://mui.com/material-ui/react-css-baseline/
Fixed it by removing <CssBaseline /> from RendererThemeProvider.