MSUnmerged: Track accumulated errors

Impact of the new feature
MSUnmerged
Is your feature request related to a problem? Please describe.
This is NOT a feature request NEITHER a bug
After we deployed the latest version of MSUnmerged in production yesterday, we are already capable of tracking all the errors we have per RSE with a single query directly to the database. [1]
The current issue is just for tracking purposes, in order to follow if any of those errors are critical, so that we can announce the service completely functional later on.
Describe the solution you'd like
Follow on all the critical Errors that  we observe and make sure we do not miss an RSE behind.
Describe alternatives you've considered
No alternative.
Additional context
[1]
In [47]: filter = {}

In [48]: projection = {
    ...:     "_id": False,
    ...:     "name": True,
    ...:     "isClean": True,
    ...:     "counters": {
    ...:         "gfalErrors": True,
    ...:         "dirsToDelete": True,
    ...:         "dirsDeletedSuccess": True,
    ...:         "dirsDeletedFail": True}}

In [49]: list(msUnmergedCollProd.find(filter, projection))
Out[49]:
[{'name': 'T1_US_FNAL_Disk',
  'isClean': True,
  'counters': {'dirsToDelete': 37,
   'dirsDeletedSuccess': 37,
   'dirsDeletedFail': 0,
   'gfalErrors': {}}},
 {'name': 'T2_TW_NCHC',
  'isClean': False,
  'counters': {'dirsToDelete': 57,
   'dirsDeletedSuccess': 55,
   'dirsDeletedFail': 2,
   'gfalErrors': {'No such file or directory': 720}}},
 {'name': 'T2_US_Vanderbilt',
  'isClean': True,
  'counters': {'dirsToDelete': 302,
   'dirsDeletedSuccess': 302,
   'dirsDeletedFail': 0,
   'gfalErrors': {}}},
 {'name': 'T2_PT_NCG_Lisbon',
  'isClean': True,
  'counters': {'dirsToDelete': 8,
   'dirsDeletedSuccess': 8,
   'dirsDeletedFail': 0,
   'gfalErrors': {'No such file or directory': 8}}},
 {'name': 'T2_UA_KIPT',
  'isClean': False,
  'counters': {'dirsToDelete': 70,
   'dirsDeletedSuccess': 69,
   'dirsDeletedFail': 1,
   'gfalErrors': {'No such file or directory': 279}}},
 {'name': 'T2_TR_METU',
  'isClean': True,
  'counters': {'dirsToDelete': 0,
   'dirsDeletedSuccess': 0,
   'dirsDeletedFail': 0,
   'gfalErrors': {}}},
 {'name': 'T2_IT_Rome',
  'isClean': True,
  'counters': {'dirsToDelete': 111,
   'dirsDeletedSuccess': 111,
   'dirsDeletedFail': 0,
   'gfalErrors': {'No such file or directory': 440}}},
 {'name': 'T2_UK_SGrid_RALPP',
  'isClean': False,
  'counters': {'dirsToDelete': 569,
   'dirsDeletedSuccess': 561,
   'dirsDeletedFail': 8,
   'gfalErrors': {'No such file or directory': 4820}}},
 {'name': 'T1_UK_RAL_Disk',
  'isClean': True,
  'counters': {'dirsToDelete': 194,
   'dirsDeletedSuccess': 194,
   'dirsDeletedFail': 1,
   'gfalErrors': {'Communication error on send': 61}}},
 {'name': 'T2_DE_DESY',
  'isClean': True,
  'counters': {'dirsToDelete': 194,
   'dirsDeletedSuccess': 194,
   'dirsDeletedFail': 0,
   'gfalErrors': {'No such file or directory': 1376}}},
 {'name': 'T2_UK_London_IC',
  'isClean': True,
  'counters': {'dirsToDelete': 291,
   'dirsDeletedSuccess': 291,
   'dirsDeletedFail': 0,
   'gfalErrors': {'No such file or directory': 3533}}},
 {'name': 'T2_DE_RWTH',
  'isClean': False,
  'counters': {'dirsToDelete': 268,
   'dirsDeletedSuccess': 264,
   'dirsDeletedFail': 4,
   'gfalErrors': {'No such file or directory': 2260}}},
 {'name': 'T2_IT_Bari',
  'isClean': False,
  'counters': {'dirsToDelete': 166,
   'dirsDeletedSuccess': 165,
   'dirsDeletedFail': 1,
   'gfalErrors': {'No such file or directory': 670}}},
 {'name': 'T2_RU_INR',
  'isClean': True,
  'counters': {'dirsToDelete': 31,
   'dirsDeletedSuccess': 31,
   'dirsDeletedFail': 0,
   'gfalErrors': {'No such file or directory': 31}}},
 {'name': 'T2_FR_GRIF_LLR',
  'isClean': False,
  'counters': {'dirsToDelete': 593,
   'dirsDeletedSuccess': 565,
   'dirsDeletedFail': 28,
   'gfalErrors': {'No such file or directory': 8054}}},
 {'name': 'T2_US_Wisconsin',
  'isClean': False,
  'counters': {'dirsToDelete': 673,
   'dirsDeletedSuccess': 666,
   'dirsDeletedFail': 7,
   'gfalErrors': {'Communication error on send': 7139}}},
 {'name': 'T2_FR_IPHC',
  'isClean': False,
  'counters': {'dirsToDelete': 241,
   'dirsDeletedSuccess': 238,
   'dirsDeletedFail': 3,
   'gfalErrors': {'No such file or directory': 1383}}},
 {'name': 'T2_RU_ITEP',
  'isClean': False,
  'counters': {'dirsToDelete': 1,
   'dirsDeletedSuccess': 0,
   'dirsDeletedFail': 1,
   'gfalErrors': {'Input/output error': 3}}},
 {'name': 'T2_IN_TIFR',
  'isClean': False,
  'counters': {'dirsToDelete': 176,
   'dirsDeletedSuccess': 171,
   'dirsDeletedFail': 5,
   'gfalErrors': {'No such file or directory': 3053}}},
 {'name': 'T1_DE_KIT_Disk',
  'isClean': True,
  'counters': {'dirsToDelete': 126,
   'dirsDeletedSuccess': 126,
   'dirsDeletedFail': 0,
   'gfalErrors': {'No such file or directory': 1639}}},
 {'name': 'T1_RU_JINR_Disk',
  'isClean': True,
  'counters': {'dirsToDelete': 10,
   'dirsDeletedSuccess': 10,
   'dirsDeletedFail': 0,
   'gfalErrors': {'No such file or directory': 5}}},
 {'name': 'T1_ES_PIC_Disk',
  'isClean': False,
  'counters': {'dirsToDelete': 367,
   'dirsDeletedSuccess': 366,
   'dirsDeletedFail': 1,
   'gfalErrors': {'No such file or directory': 827758}}},
 {'name': 'T2_US_Caltech',
  'isClean': True,
  'counters': {'dirsToDelete': 126,
   'dirsDeletedSuccess': 126,
   'dirsDeletedFail': 0,
   'gfalErrors': {'No such file or directory': 1317}}},
 {'name': 'T2_US_UCSD',
  'isClean': True,
  'counters': {'dirsToDelete': 54,
   'dirsDeletedSuccess': 54,
   'dirsDeletedFail': 0,
   'gfalErrors': {}}},
 {'name': 'T2_IT_Pisa',
  'isClean': True,
  'counters': {'dirsToDelete': 92,
   'dirsDeletedSuccess': 92,
   'dirsDeletedFail': 0,
   'gfalErrors': {'No such file or directory': 245}}},
 {'name': 'T2_KR_KISTI',
  'isClean': True,
  'counters': {'dirsToDelete': 64,
   'dirsDeletedSuccess': 64,
   'dirsDeletedFail': 0,
   'gfalErrors': {'No such file or directory': 196}}},
 {'name': 'T2_RU_JINR',
  'isClean': True,
  'counters': {'dirsToDelete': 32,
   'dirsDeletedSuccess': 32,
   'dirsDeletedFail': 0,
   'gfalErrors': {'No such file or directory': 85}}},
 {'name': 'T2_AT_Vienna',
  'isClean': True,
  'counters': {'dirsToDelete': 107,
   'dirsDeletedSuccess': 107,
   'dirsDeletedFail': 0,
   'gfalErrors': {'Communication error on send': 506}}},
 {'name': 'T2_BR_UERJ',
  'isClean': True,
  'counters': {'dirsToDelete': 167,
   'dirsDeletedSuccess': 167,
   'dirsDeletedFail': 0,
   'gfalErrors': {'Communication error on send': 1}}},
 {'name': 'T2_FI_HIP',
  'isClean': True,
  'counters': {'dirsToDelete': 28,
   'dirsDeletedSuccess': 28,
   'dirsDeletedFail': 0,
   'gfalErrors': {'No such file or directory': 76}}},
 {'name': 'T2_UK_SGrid_Bristol',
  'isClean': False,
  'counters': {'dirsToDelete': 94,
   'dirsDeletedSuccess': 0,
   'dirsDeletedFail': 94,
   'gfalErrors': {'Communication error on send': 282}}},
 {'name': 'T2_US_MIT',
  'isClean': True,
  'counters': {'dirsToDelete': 113,
   'dirsDeletedSuccess': 113,
   'dirsDeletedFail': 0,
   'gfalErrors': {'Communication error on send': 36438}}},
 {'name': 'T2_US_Purdue',
  'isClean': True,
  'counters': {'dirsToDelete': 18,
   'dirsDeletedSuccess': 18,
   'dirsDeletedFail': 0,
   'gfalErrors': {}}},
 {'name': 'T3_US_Rutgers',
  'isClean': True,
  'counters': {'dirsToDelete': 0,
   'dirsDeletedSuccess': 0,
   'dirsDeletedFail': 0,
   'gfalErrors': {}}},
 {'name': 'T2_CN_Beijing',
  'isClean': True,
  'counters': {'dirsToDelete': 42,
   'dirsDeletedSuccess': 42,
   'dirsDeletedFail': 0,
   'gfalErrors': {'No such file or directory': 64}}},
 {'name': 'T2_EE_Estonia',
  'isClean': False,
  'counters': {'dirsToDelete': 185,
   'dirsDeletedSuccess': 184,
   'dirsDeletedFail': 1,
   'gfalErrors': {'Communication error on send': 1666}}},
 {'name': 'T2_BE_IIHE',
  'isClean': True,
  'counters': {'dirsToDelete': 141,
   'dirsDeletedSuccess': 141,
   'dirsDeletedFail': 0,
   'gfalErrors': {'No such file or directory': 902}}},
 {'name': 'T2_IT_Legnaro',
  'isClean': True,
  'counters': {'dirsToDelete': 310,
   'dirsDeletedSuccess': 310,
   'dirsDeletedFail': 0,
   'gfalErrors': {'No such file or directory': 1881}}},
 {'name': 'T2_PL_Swierk',
  'isClean': True,
  'counters': {'dirsToDelete': 36,
   'dirsDeletedSuccess': 36,
   'dirsDeletedFail': 0,
   'gfalErrors': {'No such file or directory': 2}}},
 {'name': 'T2_CH_CSCS',
  'isClean': True,
  'counters': {'dirsToDelete': 68,
   'dirsDeletedSuccess': 68,
   'dirsDeletedFail': 0,
   'gfalErrors': {'No such file or directory': 328}}},
 {'name': 'T2_BR_SPRACE',
  'isClean': False,
  'counters': {'dirsToDelete': 263,
   'dirsDeletedSuccess': 254,
   'dirsDeletedFail': 9,
   'gfalErrors': {'No such file or directory': 982}}},
 {'name': 'T2_HU_Budapest',
  'isClean': True,
  'counters': {'dirsToDelete': 85,
   'dirsDeletedSuccess': 85,
   'dirsDeletedFail': 0,
   'gfalErrors': {'No such file or directory': 465}}},
 {'name': 'T2_CH_CERN',
  'isClean': False,
  'counters': {'dirsToDelete': 245,
   'dirsDeletedSuccess': 244,
   'dirsDeletedFail': 1,
   'gfalErrors': {'Communication error on send': 701}}},
 {'name': 'T2_FR_GRIF_IRFU',
  'isClean': False,
  'counters': {'dirsToDelete': 523,
   'dirsDeletedSuccess': 499,
   'dirsDeletedFail': 24,
   'gfalErrors': {'No such file or directory': 3811}}},
 {'name': 'T2_ES_CIEMAT',
  'isClean': False,
  'counters': {'dirsToDelete': 514,
   'dirsDeletedSuccess': 512,
   'dirsDeletedFail': 2,
   'gfalErrors': {'No such file or directory': 2586}}}]

Thanks for creating this issue, Todor. Do I read it right that there are only 2 problems reported:

'No such file or directory'; or
'Communication error on send'
?

If so, I guess what deserves attention is only these reporting communication error, right? It could be that this is somewhat a generic error from the gfal plugin though.

hi @amaltaro , yes I think you are correct. I'll  double check today.

The more I look into those logs the more confident I get that  we should exclude the /store/unmerged/SAM directory from the tree to be cleaned. The contents there are too volatile and the files are constantly written and deleted by the SAM tests, which in general seems to be doing their job of cleaning behind themselves quite well. Also the contents are usually small enough. While on the other hand if we  preserve this part of the tree we are going to be constantly blocked to delete one or two non-empty directories per site and hence not announcing some of them clean for a quite long.
We are going to have a deployment in production today due to another minor configuration error  I've  found, so I am about to make the relevant configuration PRs for excluding this part of the tree too.

And here are the relevant configuration changes:
cmsweb-k8s/services_config!127
cmsweb-k8s/services_config!126

@ivmfnal Igor, don't we have this path skipped from the Rucio ConMon system?
@todor-ivanov the deployment is just the deployment of the new configuration, right?

Hi @amaltaro,

the deployment is just the deployment of the new configuration, right?

Almost... here is another minor configuration error (a typo) we have found with @muhammadimranfarooqi  earlier today and which actually trigggered the need of this deployment, but this one should not have caused any troubles in the service run.
https://github.com/dmwm/CMSKubernetes/commit/0b09abbd4bb0ee3cc0811d1c921e7511edd1f3b8

And just for logging purposes here follows a short list of all the RSEs that are missing from MongoDB due to various of reasons, pasted in parallel bellow:

T1:

T1_FR_CCIN2P3_Disk      In non-final state in Rucio Consistency Monitor
T1_IT_CNAF_Disk         In non-final state in Rucio Consistency Monitor

T2T3:

T2_BE_UCL         In non-final state in Rucio Consistency Monitor
T2_ES_IFCA        In non-final state in Rucio Consistency Monitor
T2_FR_CCIN2P3     Missing in stats records at Rucio Consistency Monitor
T2_GR_Ioannina      In non-final state in Rucio Consistency Monitor
T2_PK_NCP        In non-final state in Rucio Consistency Monitor
T2_RU_IHEP        In non-final state in Rucio Consistency Monitor
T2_UK_London_Brunel    In non-final state in Rucio Consistency Monitor
T3_BG_UNI_SOFIA       Missing in stats records at Rucio Consistency Monitor
T3_CH_CERNBOX       Missing in stats records at Rucio Consistency Monitor
T3_CH_CERN_OpenData     Missing in stats records at Rucio Consistency Monitor
T3_CH_PSI       Missing in stats records at Rucio Consistency Monitor
T3_FR_IPNL     Missing in stats records at Rucio Consistency Monitor
T3_HR_IRB     Missing in stats records at Rucio Consistency Monitor
T3_IR_IPM     Missing in stats records at Rucio Consistency Monitor
T3_IT_MIB     Missing in stats records at Rucio Consistency Monitor
T3_IT_Trieste     Missing in stats records at Rucio Consistency Monitor
T3_KR_KISTI     Missing in stats records at Rucio Consistency Monitor
T3_KR_KNU     Missing in stats records at Rucio Consistency Monitor
T3_KR_UOS     Missing in stats records at Rucio Consistency Monitor
T3_MX_Cinvestav     Missing in stats records at Rucio Consistency Monitor
T3_TW_NCU     Missing in stats records at Rucio Consistency Monitor
T3_TW_NTU_HEP    Missing in stats records at Rucio Consistency Monitor

T2T3_US:

T2_US_Caltech_Ceph    is skipped due to a restriction set in msConfig
T2_US_Florida    In non-final state in Rucio Consistency Monitor
T2_US_Nebraska    In non-final state in Rucio Consistency Monitor
T3_US_Baylor    Missing in stats records at Rucio Consistency Monitor
T3_US_Brown     Missing in stats records at Rucio Consistency Monitor
T3_US_CMU     Missing in stats records at Rucio Consistency Monitor
T3_US_Colorado     Missing in stats records at Rucio Consistency Monitor
T3_US_FNALLPC     Missing in stats records at Rucio Consistency Monitor
T3_US_MIT     Missing in stats records at Rucio Consistency Monitor
T3_US_NERSC     Missing in stats records at Rucio Consistency Monitor
T3_US_NotreDame     Missing in stats records at Rucio Consistency Monitor
T3_US_OSU     Missing in stats records at Rucio Consistency Monitor
T3_US_Princeton_ICSE     Missing in stats records at Rucio Consistency Monitor
T3_US_PuertoRico     Missing in stats records at Rucio Consistency Monitor
T3_US_Rice     Missing in stats records at Rucio Consistency Monitor
T3_US_TAMU     Missing in stats records at Rucio Consistency Monitor
T3_US_UMD     Missing in stats records at Rucio Consistency Monitor
T3_US_UMiss     Missing in stats records at Rucio Consistency Monitor

Hi @ivmfnal Could we check if this informatin I am extracting from our dabase and logs is correct? Sorry if causing additional troubles, with such a request.
@amaltaro in order to fetch those errors I had to do a several set operations to figure out which are the RSEs actually missing from the database and upon that I had to parse all those long longs that we have in order to find the reasons. And this is because all those they exit the pipeline by  raising a MSUnmergedException quite early [1] and we have never considered for such cases to actually put a record in the database. Given how   difficult and time consuming it was to track all that I'd say we better record this as an actual error in the RSE object and preserve it in the database. I may create yet another bugifx for that too. What do you think?
[1]
https://github.com/dmwm/WMCore/blob/90a3cb7703bebe2957f94a1887183da7bd6d8a55/src/python/WMCore/MicroService/MSUnmerged/MSUnmerged.py#L491

Todor, you can see all the status and some history about CC here: https://cmsweb-k8s-prod.cern.ch/rucioconmon/index
Igor

From: todor-ivanov @.>
Sent: Friday, February 11, 2022 10:25 AM
To: dmwm/WMCore @.>
Cc: Igor V Mandrichenko @.>; Mention @.>
Subject: Re: [dmwm/WMCore] MSUnmerged: Track accumulated errors (Issue #10982)
And just for logging purposes here follows a short list of all the RSEs that are missing from MongoDB due to various of reasons, pasted in parallel bellow:

T1:

T1_FR_CCIN2P3_Disk      In non-final state in Rucio Consistency Monitor
T1_IT_CNAF_Disk         In non-final state in Rucio Consistency Monitor

T2T3:

T2_BE_UCL         In non-final state in Rucio Consistency Monitor
T2_ES_IFCA        In non-final state in Rucio Consistency Monitor
T2_FR_CCIN2P3     Missing in stats records at Rucio Consistency Monitor
T2_GR_Ioannina      In non-final state in Rucio Consistency Monitor
T2_PK_NCP        In non-final state in Rucio Consistency Monitor
T2_RU_IHEP        In non-final state in Rucio Consistency Monitor
T2_UK_London_Brunel    In non-final state in Rucio Consistency Monitor
T3_BG_UNI_SOFIA       Missing in stats records at Rucio Consistency Monitor
T3_CH_CERNBOX       Missing in stats records at Rucio Consistency Monitor
T3_CH_CERN_OpenData     Missing in stats records at Rucio Consistency Monitor
T3_CH_PSI       Missing in stats records at Rucio Consistency Monitor
T3_FR_IPNL     Missing in stats records at Rucio Consistency Monitor
T3_HR_IRB     Missing in stats records at Rucio Consistency Monitor
T3_IR_IPM     Missing in stats records at Rucio Consistency Monitor
T3_IT_MIB     Missing in stats records at Rucio Consistency Monitor
T3_IT_Trieste     Missing in stats records at Rucio Consistency Monitor
T3_KR_KISTI     Missing in stats records at Rucio Consistency Monitor
T3_KR_KNU     Missing in stats records at Rucio Consistency Monitor
T3_KR_UOS     Missing in stats records at Rucio Consistency Monitor
T3_MX_Cinvestav     Missing in stats records at Rucio Consistency Monitor
T3_TW_NCU     Missing in stats records at Rucio Consistency Monitor
T3_TW_NTU_HEP    Missing in stats records at Rucio Consistency Monitor

T2T3_US:

T2_US_Caltech_Ceph    is skipped due to a restriction set in msConfig
T2_US_Florida    In non-final state in Rucio Consistency Monitor
T2_US_Nebraska    In non-final state in Rucio Consistency Monitor
T3_US_Baylor    Missing in stats records at Rucio Consistency Monitor
T3_US_Brown     Missing in stats records at Rucio Consistency Monitor
T3_US_CMU     Missing in stats records at Rucio Consistency Monitor
T3_US_Colorado     Missing in stats records at Rucio Consistency Monitor
T3_US_FNALLPC     Missing in stats records at Rucio Consistency Monitor
T3_US_MIT     Missing in stats records at Rucio Consistency Monitor
T3_US_NERSC     Missing in stats records at Rucio Consistency Monitor
T3_US_NotreDame     Missing in stats records at Rucio Consistency Monitor
T3_US_OSU     Missing in stats records at Rucio Consistency Monitor
T3_US_Princeton_ICSE     Missing in stats records at Rucio Consistency Monitor
T3_US_PuertoRico     Missing in stats records at Rucio Consistency Monitor
T3_US_Rice     Missing in stats records at Rucio Consistency Monitor
T3_US_TAMU     Missing in stats records at Rucio Consistency Monitor
T3_US_UMD     Missing in stats records at Rucio Consistency Monitor
T3_US_UMiss     Missing in stats records at Rucio Consistency Monitor
Hi @ivmfnalhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_ivmfnal&d=DwMCaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=xVVABFB8tmUPsqeRvA-B6A&m=i7PzUhnPHjYZRrTL7FJbKoxY87KesnLnF3QoQ6AeiPz5HNBbcyJXfRFnfHcW-e4n&s=KAjoEyq2ZD8snZ4iJAcdvfJpezNpdmW8WsPZOVWpbRE&e= Could we check if this informatin I am extracting from our dabase and logs is correct? Sorry if causing additional troubles, with such a request.
@amaltarohttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_amaltaro&d=DwMCaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=xVVABFB8tmUPsqeRvA-B6A&m=i7PzUhnPHjYZRrTL7FJbKoxY87KesnLnF3QoQ6AeiPz5HNBbcyJXfRFnfHcW-e4n&s=B-eJA5gjFzTzlry6KO8ep17xQwbwO5mH1MLYFoQHPM4&e= in order to fetch those errors I had to do a several set operations to figure out which are the RSEs actually missing from the database and upon that I had to parse all those long longs that we have in order to find the reasons. And this is because all those they exit the pipeline by raising a MSUnmergedException quite early [1] and we have never considered for such cases to actually put a record in the database. Given how difficult and time consuming it was to track all that I'd say we better record this as an actual error in the RSE object and preserve it in the database. I may create yet another bugifx for that too. What do you think?
[1]
https://github.com/dmwm/WMCore/blob/90a3cb7703bebe2957f94a1887183da7bd6d8a55/src/python/WMCore/MicroService/MSUnmerged/MSUnmerged.py#L491https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_dmwm_WMCore_blob_90a3cb7703bebe2957f94a1887183da7bd6d8a55_src_python_WMCore_MicroService_MSUnmerged_MSUnmerged.py-23L491&d=DwMCaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=xVVABFB8tmUPsqeRvA-B6A&m=i7PzUhnPHjYZRrTL7FJbKoxY87KesnLnF3QoQ6AeiPz5HNBbcyJXfRFnfHcW-e4n&s=xBSV0doEJVGiQIpMKKHv3Lt3hkfMOGrN7PH-Iv922hQ&e=
—
Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_dmwm_WMCore_issues_10982-23issuecomment-2D1036385522&d=DwMCaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=xVVABFB8tmUPsqeRvA-B6A&m=i7PzUhnPHjYZRrTL7FJbKoxY87KesnLnF3QoQ6AeiPz5HNBbcyJXfRFnfHcW-e4n&s=rk0cApT-wk9f0jo5IU0E7X3_nkRmPuR28j8uBu5nypA&e=, or unsubscribehttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AFK4SQW2HWVVBXWTBPUKD5TU2UZ6FANCNFSM5N57BT5A&d=DwMCaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=xVVABFB8tmUPsqeRvA-B6A&m=i7PzUhnPHjYZRrTL7FJbKoxY87KesnLnF3QoQ6AeiPz5HNBbcyJXfRFnfHcW-e4n&s=eKnXVozzG2KWVuf4E7qFb8NfsC8R-Flqqu3VV3mzYww&e=.
Triage notifications on the go with GitHub Mobile for iOShttps://urldefense.proofpoint.com/v2/url?u=https-3A__apps.apple.com_app_apple-2Dstore_id1477376905-3Fct-3Dnotification-2Demail-26mt-3D8-26pt-3D524675&d=DwMCaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=xVVABFB8tmUPsqeRvA-B6A&m=i7PzUhnPHjYZRrTL7FJbKoxY87KesnLnF3QoQ6AeiPz5HNBbcyJXfRFnfHcW-e4n&s=iNtw5Fx4GixXK36a0aayiJhrTA8w0DTsb6fy9BskrQs&e= or Androidhttps://urldefense.proofpoint.com/v2/url?u=https-3A__play.google.com_store_apps_details-3Fid-3Dcom.github.android-26referrer-3Dutm-5Fcampaign-253Dnotification-2Demail-2526utm-5Fmedium-253Demail-2526utm-5Fsource-253Dgithub&d=DwMCaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=xVVABFB8tmUPsqeRvA-B6A&m=i7PzUhnPHjYZRrTL7FJbKoxY87KesnLnF3QoQ6AeiPz5HNBbcyJXfRFnfHcW-e4n&s=hujBgdSkzK9mikKmKSzzrkQ6AHG_zVby4vD-Bi-HGec&e=.
You are receiving this because you were mentioned.Message ID: @.***>

Thank you @ivmfnal , We will track the information provided there too. I am also creating another issue just to address the missing records in our database and I will put the link in the docstring inside the code so that the sites in error could be linked from both systems.

With the latest version of MSUnmerged deployed in production, we can easily check the result of the above query to MongoDB through the info=ALL_DOCS REST API of the service. The link is here.
From what I can see in the current run there are gfal errors mostly of the following two types:

Communication error on send:

T1_UK_RAL_Disk
T2_IN_TIFR
T2_CH_CERN
T2_US_Nebraska

No such file or directory:

All RSEs which has files or directories which vanished from /store/unmerged since last RucioConMon run.

The 4 RSEs above are actually marked as clean, because the number of directories found to be purged equals the number of directories which either have been successfully deleted or have been already gone at the time MSUnmerged retried the RSE.

@todor-ivanov JSON is a text file. Even if it is 800MB you can still open it with vim and look up the mailformed characted. The key line here is line 1 column 168804222, so in vim you just need to jump to column 168804222 and see your guy. The trick is to use <number>| in your case it would be 168804222| and vim cursor will jump to that column.
It is too bad that we have such enourmous JSON, few comments about it:

JSON was not designed for large data structures
large JSON leads to large RAM, usually 2-3 times of size of JSON
not formatted JSON (like this in single line) can be hard to read since you can scroll it with more/less tools
sending such huge JSON via HTTP is path to block entire service, we saw DBS usage of 10GB and you may likely hit this too if you'll use such large JSONs

I was able to download https://cmsweb-prod.cern.ch/rucioconmon/unmerged/files/T1_UK_RAL_Disk_wm_file_list.json?rse=T1_UK_RAL_Disk&format=json and parse it as JSON

$ curl -k -o file_list.json --cert ~/certs/CERN_crt.pem --key ~/certs/CERN_key.pem "https://cmsweb-prod.cern.ch/rucioconmon/unmerged/files/T1_UK_RAL_Disk_wm_file_list.json?rse=T1_UK_RAL_Disk&format=json"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  826M    0  826M    0     0  10.5M      0 --:--:--  0:01:18 --:--:-- 12.8M

$ python -m json.tool file_list.json | wc -l
 3790380

Thanks @vkuznet  and @ivmfnal . I continue  searching if there are any issues/limitations on our side.
Valentin, do we have any statistics about resource consumption at the K8 clusters.

@todor-ivanov , the short answer is we do have plenty, we monitor nodes and services. But it really depends on what service metrics are. For instance, all python based services under cmsweb reports some stats, e.g. here is reqmgr2 dashboard. It provides cpu, ram, fds, etc. Since you're querying rucioconmon it is up to this service to provide its metrics.
Just few days ago, we added node metrics of k8s cluster, see here and here. The question is do they useful for your use case I have no idea. The monitoring comes from services, if service provide relevant metrics we can easily plug them in. If nobody cares about service metrics then you have nothing.

I am going to implement ignore list filtering at the web server so that the file list is filtered before being sent to the client

I made changes to the web server:

Now the server can filter out some unwanted files from the output of a URL like https://cmsweb.cern.ch/rucioconmon/WM/files/T1_UK_RAL_Disk_wm_file_list.txt?rse=T1_UK_RAL_Disk&format=json. For now, by default it filters out /store/unmerges/logs/*.
The default can be overridden by URL parameters:

include=<path>[,<path>] - only files under specified paths will be included in the resulting list
exclude=<path>[,<path>] - files under the specified paths will be excluded from the list
if both include and exclude are present, the list will include only those paths matched by include minus those matched by exclude

format values:

raw (old) - gzip-compressed original list from the scanner, without any filters described above applied
text (new) - plain text, paths are separated with newlines
json (old) - JSON list of paths
zip-stream (new) - zip stream (see https://docs.python.org/3/library/zlib.html#module-zlib)

Just logging here something we have previously noticed, but it is now a question that needs to be answered.
While looking at the logs related to T1_UK_RAL_Disk we noticed that more than one (in this case 2) different errors on protocol level, with two different error messages, but with the same error code returned by gfal, has been mapped to the same posix error:
2022-03-11 18:17:17,450:DEBUG:MSUnmerged: RSE: T1_UK_RAL_Disk, Dir: /store/unmerged/RunIISummer20UL16DIGIPremixAPV/BsToMuMuPhi_PhiToKK_BMuonFilter_SoftQCDnonD_TuneCP5_13TeV-pythia8-evtgen/GE
N-SIM-DIGI/106X_mcRun2_asymptotic_preVFP_v8-v1, delResult: [GError('[gfal_gridftp_unlinkG][gfal_gridftp_unlinkG] an end-of-file was reached globus_xio: An end of file occurred ', 70),
 GError('[gfal_gridftp_unlinkG][gfal_gridftp_unlinkG] an end-of-file was reached globus_xio: An end of file occurred ', 70),
 GError('[gfal_gridftp_unlinkG][gfal_gridftp_unlinkG] an end-of-file was reached globus_xio: An end of file occurred ', 70),
 GError('[gfal_gridftp_unlinkG][gfal_gridftp_unlinkG] globus_xio: System error in send: Broken pipe globus_xio: A system call failed: Broken pipe ', 70),
 GError('[gfal_gridftp_unlinkG][gfal_gridftp_unlinkG] globus_xio: System error in send: Broken pipe globus_xio: A system call failed: Broken pipe ', 70),

And those are both mapped to the posix error: 70: ECOMM: Communication error on send, and reported as such in  MongoDB by our service [1].
This is a question that  has been asked to the gfal developers here. But we still have no answer.
FYI @amaltaro
[1]
https://cmsweb.cern.ch/ms-unmerged/data/info?rse=T1_UK_RAL_Disk&detail=False
{"result": [
 {
  "wmcore_version": "2.0.1.pre6",
  "microservice_version": "2.0.1.pre6",
  "microservice": "MSManager",
  "query": "rse=T1_UK_RAL_Disk&detail=False",
  "rseData": [
    {
      "name": "T1_UK_RAL_Disk",
      "pfnPrefix": "gsiftp://gridftp.echo.stfc.ac.uk:2811/cms:",
      "rucioConMonStatus": "done",
      "isClean": true,
      "timestamps": {
        "rseConsStatTime":<PHONE_NUMBER>,
        "prevStartTime":<PHONE_NUMBER>.8978102,
        "startTime":<PHONE_NUMBER>.5946178,
        "prevEndTime":<PHONE_NUMBER>.8980052,
        "endTime":<PHONE_NUMBER>.5947526
      },
      "counters": {
        "totalNumFiles": 20348,
        "totalNumDirs": 1052,
        "dirsToDelete": 235,
        "filesToDelete": 0,
        "filesDeletedSuccess": 5551,
        "filesDeletedFail": 0,
        "dirsDeletedSuccess": 235,
        "dirsDeletedFail": 2,
        "gfalErrors": {
          "Communication error on send": 408
        }
      }
    }
  ]
}]}

@todor-ivanov have I missed the different errors at the protocol level from your message above?
My naive and generic view of this error handling is the following though, if the posix error is good enough for us to track:
a) problems in MSUnmerged
b) problems at the storage level
then we should not try to micromanage those errors. To me, it looks fair to have a map of multiple protocol errors into the same posix error (since the posix errors are very limited as well).

hi @amaltaro
In general this should be the case  and that's exactly why I beleived this should be correct method at the first place. We should only watch for Permissions related errors from the protocol level, not to be masked inside the Communication Error on Send posix error. Here [1] is an example. This error itself is actually triggered because of a broken uri construction for the gsiftp protocol, but still we need to pay attention to those.
[1]
$ gfal-ls gsiftp://gridftp.echo.stfc.ac.uk:2811/cms/store/unmerged/RunIISummer20UL18DIGIPremix/DYJetsToEE_M-100To160_VBFFilter_TuneCP5_13TeV-amcatnloFXFX-pythia8/GEN-SIM-DIGI/106X_upgrade2018_realistic_v11_L1v1-v1
gfal-ls error: 70 (Communication error on send) - globus_ftp_client: the server responded with an error 500 Command failed : globus_l_gfs_ceph_stat: authorization error: 'MLST' operation not allowed

have I missed the different errors at the protocol level from your message above

I am not sure if you have... but, maybe I should have pointed them better. Here they are:

Error message:  ...An end of file occurred,  Error code: 70 =  Posix error: Communication error on send
Error message:  ...Broken pipe,  Error code: 70 =  Posix error: Communication error on send

So far all the full list of RSEs to have issues with deletions is:
T2_UK_London_Brunel
T2_BE_UCL

I am creating the relevant GGUS tickets  for them.

And here is the llist of GGUS tickets created for this [1]. I am closing the current issue here. We will follow with the sites and Site Support team. If any other changes are needed another WMcore issue is to be opened.
[1]
https://ggus.eu/?mode=ticket_info&ticket_id=156526
https://ggus.eu/?mode=ticket_info&ticket_id=156527


Does say.export work for Linux?

Hi I saw there is a PR already years ago to make say.export to work for Linux, but in the document, it still says say.export is not supported for Linux platform? is it difficult to make it work for Linux? if I install Festival on linux, will the say.export() work on linux?
Thanks

say.export does not currently work on Linux. Installing festival is required to do anything on Linux.
The text2wave command may include some hints on how to get such functionality working in say.js.


IMS Preprocess

IMS gz files were downloaded from PolarWatch website. The compressed files are in HDF5 format. The indexed keys in HDF5 used are timestamp and the image data. The data are values 0-4. The plots use colors used by the default IMS product to show land, sea ice, snow, etc. I sampled 10 days of each year and displayed the plots.
All data are stored locally. All told, compressed and uncompressed is around 6GB.

ah your CI workflow is failing still because Ruchi's notebook isn't compliant with the linters and your branch is based relative to hers. Since you're not using her notebook in your work, let's get your branch rebased off main.
I would do a squash and rebase if I were you -- git rebase -i origin/main and follow instructions online to only keep your first commit, since you have a high ratio of commits to files changes. It makes it easier to look at the repo's git history and understand progress if the commit log is relatively trim and every commit message is descriptive.
In the future, this is a nice resource on writing good commit messages -- I'm sure you just had the typical experience like below, but there are great tips at this link and it sets a good paradigm of what you should actually include in a message. For instance, I don't know what files were added (or why) and what it meant to upload them in the commit message you have.


fix: format output with markdown

fixes: #39

When will this PR merge? has an useful feature.
ALso, if possible, please highlight the code blocks too

@kskarthik wdym by highlight?

@kskarthik If Llama adds the language ext after the triple quotes then the code will automatically be highlighted
but It does not adds that

Hello. What's the status of this PR? Seems like a basic and much needed feature.

Hi @Shivansh-yadav13, I'd like to work on this issue. Could you please assign it to me? I'll fix the output formatting with Markdown and make the chat textarea sizable.


Support for Laravel 5.6

When can we expect support for Laravel 5.6?
Thanks in advance.

A PR for supporting Laravel 5.6 is underway : https://github.com/jenssegers/laravel-mongodb/pull/1428

Closed in https://github.com/jenssegers/laravel-mongodb/pull/1428


fix(validator): typo in isURL function call

When running the validator against an OpenAPI 3 document, the validator can fail with the error:
[Error] There was a problem with a validator.
stringValidator.isurl is not a function

TypeError: stringValidator.isurl is not a function
    at Object.module.exports.validate (~/.nvm/versions/node/v12.14.1/lib/node_modules/ibm-openapi-validator/src/plugins/validation/oas3/semantic-validators/security-definitions-ibm.js:168:26)
    at ~/.nvm/versions/node/v12.14.1/lib/node_modules/ibm-openapi-validator/src/cli-validator/utils/validator.js:65:45
    at Array.forEach (<anonymous>)
    at validateSwagger (~/.nvm/versions/node/v12.14.1/lib/node_modules/ibm-openapi-validator/src/cli-validator/utils/validator.js:64:35)
    at processInput (~/.nvm/versions/node/v12.14.1/lib/node_modules/ibm-openapi-validator/src/cli-validator/runValidator.js:238:17)

The call to isurl() should be isURL().

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

:tada: This PR is included in version 0.29.2 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


fix: bug that when UserIdentity is changed, the correct user identities are not returned afterwards

Fixes #450

Fix bug that when UserIdentity is changed (including when saving User object), the correct user identities does not be returned afterwards.
Now UserModel does not alter the passed User object state when saving, but when it is an insertion, add the user id to it.

I've changed my mind.
The following code is a misuse. Now it causes an exception.
$user           = new User();
$user->username = 'foo';
$user->email    =<EMAIL_ADDRESS>$user->password = 'password';
$user->active   = false;

$users->save($user);

// $user is not fetched from the database.
$identity = $user->getEmailIdentity();

Because CI4's Model::save() does not alter the passed object state.
We should not use incomplete User objects.
If we don't get User objects from the database,
these have no created/updated timestamps, or other fields coming from the database,
or effects of stored procedures and field defaults.
So if you do new User(), you should save it and get again.
// To get the complete user object, we need to get from the database
$user = $users->findById($users->getInsertID());


Failed prop type: Invalid prop form of type object supplied to Form(App), expected string

I have the following form:
It's almost same as Simple Form.
import React from 'react'
import { Field, reduxForm } from 'redux-form'

const App = (props) => {
  const { handleSubmit, pristine, reset, submitting } = props
  return (
    <form onSubmit={handleSubmit}>

        First Name

          <Field name="firstName" component="input" type="text" placeholder="First Name"/>

        Last Name

          <Field name="lastName" component="input" type="text" placeholder="Last Name"/>

        Email

          <Field name="email" component="input" type="email" placeholder="Email"/>

        Sex

          <Field name="sex" component="input" type="radio" value="male"/> Male
          <Field name="sex" component="input" type="radio" value="female"/> Female

        Favorite Color

          <Field name="favoriteColor" component="select">
            <option></option>
            <option value="ff0000">Red</option>
            <option value="00ff00">Green</option>
            <option value="0000ff">Blue</option>
          </Field>

        Employed

          <Field name="employed" id="employed" component="input" type="checkbox"/>

        Notes

          <Field name="notes" component="textarea"/>

        <button type="submit" disabled={pristine || submitting}>Submit</button>
        <button type="button" disabled={pristine || submitting} onClick={reset}>Clear Values</button>

    </form>
  )
}

export default reduxForm({
  form: 'app'  // a unique identifier for this form
})(App)

I get the following error:
warning.js:36Warning: Failed prop type: Invalid prop `form` of type `object` supplied to `Form(App)`, expected `string`.
    in Form(App) (created by Connect(Form(App)))
    in Connect(Form(App)) (created by ReduxForm)
    in ReduxForm (created by Connect(ReduxForm))
    in Connect(ReduxForm) (at index.js:12)
    in Provider (at index.js:11)

I checked the prop of the Connect(Form(App)).

Why is the prop 'form' recognized as Object (not string)?
And I want to know how to solve this problem.

I don't see anything wrong with the code you've shared. So the problem must be elsewhere. Like how you are setting up the reducer or in the render() method of the container of this component.

@gyarasu Did you solve this issue? Running into the same problem.

Never mind, found the answer :) in my case one of my props was also named "form", renamed it to something else.

@chanan  thanks, change reducer's name, not named 'form', is fixed!

if using follow code, remove {...this.props}
<AppForm {...this.props}></AppForm>

Thanks, I'll try it later:)

same issue was in such case:
====in component====
class NewBuilding extends React.Component {
....
}
NewBuilding = reduxForm({
form: 'building',
})(NewBuilding);
const mapStateToProps = ({ building, form }) => ({
building,
form,
});
=====
form reducer was exported such way:
import { reducer as form } from 'redux-form';
fixed part:
const mapStateToProps = ({ building, form }) => ({
building,
formReducer: form,
});
=====

Since folks are finding their way around this issue, I will close it.
Feel free to keep the discussion going.

Just to contribute.
I had the same problem because one of my fields had the same name as the form.
export default reduxForm({
   form: 'example',
  initialValues: { example: 'Testing'},
})

Renaming didn't work for me, but the following did
js const mapStateToProps = function(state) {     delete state.form; // hack     return state };

For those who use next.js and redux form
I have create.js in pages folder that called component with redux form
<Component {...props}/>
In some cases you props could hold form property as object form:{}
And redux form will use this passed prop from next.js
So just do not pass all props to you redux from component, pass only what you need. (Took me hakf a day to recognise it) :(


sortupdate event doesn't fire when removing a single remaining item in connected lists

When using connected lists if you remove a single remaining item from one list, leaving no items in that list, the sortupdate event isn't fired, it's only fired by the other connected lists.
Obviously you can listener for the event on both lists but this is messy as you'll get two events for every item move. It should really fire on both if there are connected lists.

Hmm, is it fired before on the list, the item is removed from? Maybe we should have separate events for item removed and item added, so you actually know what happend. But maybe thats an overkill. Feel free to send a PR to fix this.


Importer from oggDude is failing on foundry

Hi all,
when importing a character created in oggDude it fails with the message:
Starwars FFG |  Error while importing character TypeError: Cannot set property 'value' of undefined     at import-helpers.js:686     at Array.forEach (<anonymous>)     at Function.characterImport (import-helpers.js:676)     at HTMLButtonElement.<anonymous> (character-importer.js:48) Error     at Object.error (https://fvtt-ib.mooo.com/systems/starwarsffg/modules/helpers/common.js:15:53)     at Function.characterImport (https://fvtt-ib.mooo.com/systems/starwarsffg/modules/importer/import-helpers.js:1133:21)     at HTMLButtonElement.<anonymous> (https://fvtt-ib.mooo.com/systems/starwarsffg/modules/importer/character-importer.js:48:27) error	@	common.js:15
The template used is: this
is there anything that I´m losing in the character generation?
Thanks

Did you already import the oggdude data set?

Actually I see the problem, the XML file has characteristic names in non-english, which is currently not supported for the character import.  We map the name to the value in foundry,  I see that there is a key but foundry has no way to map the key to the correct characteristic.

We'll have to rework the character import to utilize the keys instead of the characteristic name

ok, thanks.
I´ll try using english.
Regards!

That's the workaround for now, and we'll add the fix in the upcoming major release.

Implemented this in the dev branch, will be included in the next release.


Ceph compatibility - is_dir() fails for directories when "/" included at end of path

Hello,
For directories that have a suffix such as .SAFE directories for Sentinel satellites data, pathlib recognizes the directories correctly, although cloudpathlib fails:
>>> # On disk
>>> from cloudpathlib import AnyPath
>>> A = AnyPath(r"D:\S2B_MSIL2A_20200114T065229_N0213_R020_T40REQ_20200114T094749.SAFE")
>>> A.is_file()
False
>>> A.is_dir()
True

>>> # In th Cloud
>>> client = S3Client(
    endpoint_url=f"https://{AWS_S3_ENDPOINT}",
    aws_access_key_id=os.getenv(AWS_ACCESS_KEY_ID),
    aws_secret_access_key=os.getenv(AWS_SECRET_ACCESS_KEY),
)
>>> client.set_as_default_client()
>>> B = AnyPath("s3://my-bucket/S2B_MSIL2A_20200114T065229_N0213_R020_T40REQ_20200114T094749.SAFE/")
>>> B.is_file()
True
>>> B.is_dir()
False

However, it works when removing the final \ !
>>> # In th Cloud
>>> client = S3Client(
    endpoint_url=f"https://{AWS_S3_ENDPOINT}",
    aws_access_key_id=os.getenv(AWS_ACCESS_KEY_ID),
    aws_secret_access_key=os.getenv(AWS_SECRET_ACCESS_KEY),
)
>>> client.set_as_default_client()
>>> B = AnyPath("s3://my-bucket/S2B_MSIL2A_20200114T065229_N0213_R020_T40REQ_20200114T094749.SAFE")
>>> B.is_file()
False
>>> B.is_dir()
True

Hi @remi-braun, I don't reproduce this based on what you specified. Here's an example with a public asset on S3:
>>> file = S3Path('s3://drivendata-public-assets/nested_dir/test.dot/a_file.txt')
>>> file.is_file()
True
>>> file.is_dir()
False
>>> directory = S3Path('s3://drivendata-public-assets/nested_dir/test.dot/')
>>> directory.is_file()
False
>>> directory.is_dir()
True

A few questions:

Can you print out B from your second example so we can see what kind of path that resolved to?
Is there anything in that directory or is it empty? There's not an actual concept of a "directory" on S3 so some of it is faked if the directory is empty, which may cause the behavior you are seeing: https://serverfault.com/questions/435827/what-is-the-difference-between-buckets-and-folders-in-amazon-s3
How was the directory created? Some tools like the console GUI can create these fake "folders" on s3: https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-folders.html
Is there a public S3 path you can point to that reproduces this for us to test?

Here's the logic that checks directories/files on S3. It's not too complicated, but bsaically we check if S3 lists the exact path when we ask it for files. If it does, we assume it is a file.

Hi !
Thanks for your quick reply :)

* Can you print out `B` from your second example so we can see what kind of path that resolved to?

>>> B
S3Path('s3://sertit-eoreader-ci/optical/S2B_MSIL2A_20200114T065229_N0213_R020_T40REQ_20200114T094749.SAFE/')

* Is there anything in that directory or is it empty? There's not an actual concept of a "directory" on S3 so some of it is faked if the directory is empty, which may cause the behavior you are seeing: https://serverfault.com/questions/435827/what-is-the-difference-between-buckets-and-folders-in-amazon-s3

This directory is full and the glob function works:
>>> [path.name for path in B.glob("**/*")]
['S2B_MSIL2A_20200114T065229_N0213_R020_T40REQ_20200114T094749.SAFE', 'AUX_DATA', 'DATASTRIP', 'DATASTRIP', 'DS_MTI__20200114T094749_S20200114T065245', 'DS_MTI__20200114T094749_S20200114T065245', 'MTD_DS.xml', 'QI_DATA', 'QI_DATA', 'FORMAT_CORRECTNESS.xml', 'GENERAL_QUALITY.xml', 'GEOMETRIC_QUALITY.xml', 'RADIOMETRIC_QUALITY.xml', 'SENSOR_QUALITY.xml', 'GRANULE', 'GRANULE', 'L2A_T40REQ_A014918_20200114T065245', 'L2A_T40REQ_A014918_20200114T065245', 'AUX_DATA', 'AUX_DATA', 'AUX_ECMWFT', 'IMG_DATA', 'IMG_DATA', 'R10m', 'R10m', 'T40REQ_20200114T065229_AOT_10m.jp2', 'T40REQ_20200114T065229_B02_10m.jp2', 'T40REQ_20200114T065229_B03_10m.jp2', 'T40REQ_20200114T065229_B04_10m.jp2', 'T40REQ_20200114T065229_B08_10m.jp2', 'T40REQ_20200114T065229_TCI_10m.jp2', 'T40REQ_20200114T065229_WVP_10m.jp2', 'R20m', 'R20m', 'T40REQ_20200114T065229_AOT_20m.jp2', 'T40REQ_20200114T065229_B02_20m.jp2', 'T40REQ_20200114T065229_B03_20m.jp2', 'T40REQ_20200114T065229_B04_20m.jp2', 'T40REQ_20200114T065229_B05_20m.jp2', 'T40REQ_20200114T065229_B06_20m.jp2', 'T40REQ_20200114T065229_B07_20m.jp2', 'T40REQ_20200114T065229_B11_20m.jp2', 'T40REQ_20200114T065229_B12_20m.jp2', 'T40REQ_20200114T065229_B8A_20m.jp2', 'T40REQ_20200114T065229_SCL_20m.jp2', 'T40REQ_20200114T065229_TCI_20m.jp2', 'T40REQ_20200114T065229_WVP_20m.jp2', 'R60m', 'R60m', 'T40REQ_20200114T065229_AOT_60m.jp2', 'T40REQ_20200114T065229_B01_60m.jp2', 'T40REQ_20200114T065229_B02_60m.jp2', 'T40REQ_20200114T065229_B03_60m.jp2', 'T40REQ_20200114T065229_B04_60m.jp2', 'T40REQ_20200114T065229_B05_60m.jp2', 'T40REQ_20200114T065229_B06_60m.jp2', 'T40REQ_20200114T065229_B07_60m.jp2', 'T40REQ_20200114T065229_B09_60m.jp2', 'T40REQ_20200114T065229_B11_60m.jp2', 'T40REQ_20200114T065229_B12_60m.jp2', 'T40REQ_20200114T065229_B8A_60m.jp2', 'T40REQ_20200114T065229_SCL_60m.jp2', 'T40REQ_20200114T065229_TCI_60m.jp2', 'T40REQ_20200114T065229_WVP_60m.jp2', 'MTD_TL.xml', 'QI_DATA', 'QI_DATA', 'FORMAT_CORRECTNESS.xml', 'GENERAL_QUALITY.xml', 'GEOMETRIC_QUALITY.xml', 'MSK_CLDPRB_20m.jp2', 'MSK_CLDPRB_60m.jp2', 'MSK_CLOUDS_B00.gfs', 'MSK_CLOUDS_B00.gml', 'MSK_DEFECT_B01.gfs', 'MSK_DEFECT_B01.gml', 'MSK_DEFECT_B02.gfs', 'MSK_DEFECT_B02.gml', 'MSK_DEFECT_B03.gfs', 'MSK_DEFECT_B03.gml', 'MSK_DEFECT_B04.gfs', 'MSK_DEFECT_B04.gml', 'MSK_DEFECT_B05.gfs', 'MSK_DEFECT_B05.gml', 'MSK_DEFECT_B06.gfs', 'MSK_DEFECT_B06.gml', 'MSK_DEFECT_B07.gml', 'MSK_DEFECT_B08.gfs', 'MSK_DEFECT_B08.gml', 'MSK_DEFECT_B09.gml', 'MSK_DEFECT_B10.gml', 'MSK_DEFECT_B11.gfs', 'MSK_DEFECT_B11.gml', 'MSK_DEFECT_B12.gfs', 'MSK_DEFECT_B12.gml', 'MSK_DEFECT_B8A.gfs', 'MSK_DEFECT_B8A.gml', 'MSK_DETFOO_B01.gfs', 'MSK_DETFOO_B01.gml', 'MSK_DETFOO_B02.gfs', 'MSK_DETFOO_B02.gml', 'MSK_DETFOO_B03.gfs', 'MSK_DETFOO_B03.gml', 'MSK_DETFOO_B04.gfs', 'MSK_DETFOO_B04.gml', 'MSK_DETFOO_B05.gfs', 'MSK_DETFOO_B05.gml', 'MSK_DETFOO_B06.gfs', 'MSK_DETFOO_B06.gml', 'MSK_DETFOO_B07.gml', 'MSK_DETFOO_B08.gfs', 'MSK_DETFOO_B08.gml', 'MSK_DETFOO_B09.gml', 'MSK_DETFOO_B10.gml', 'MSK_DETFOO_B11.gfs', 'MSK_DETFOO_B11.gml', 'MSK_DETFOO_B12.gfs', 'MSK_DETFOO_B12.gml', 'MSK_DETFOO_B8A.gfs', 'MSK_DETFOO_B8A.gml', 'MSK_NODATA_B01.gfs', 'MSK_NODATA_B01.gml', 'MSK_NODATA_B02.gfs', 'MSK_NODATA_B02.gml', 'MSK_NODATA_B03.gfs', 'MSK_NODATA_B03.gml', 'MSK_NODATA_B04.gfs', 'MSK_NODATA_B04.gml', 'MSK_NODATA_B05.gfs', 'MSK_NODATA_B05.gml', 'MSK_NODATA_B06.gfs', 'MSK_NODATA_B06.gml', 'MSK_NODATA_B07.gml', 'MSK_NODATA_B08.gfs', 'MSK_NODATA_B08.gml', 'MSK_NODATA_B09.gml', 'MSK_NODATA_B10.gml', 'MSK_NODATA_B11.gfs', 'MSK_NODATA_B11.gml', 'MSK_NODATA_B12.gfs', 'MSK_NODATA_B12.gml', 'MSK_NODATA_B8A.gfs', 'MSK_NODATA_B8A.gml', 'MSK_SATURA_B01.gfs', 'MSK_SATURA_B01.gml', 'MSK_SATURA_B02.gfs', 'MSK_SATURA_B02.gml', 'MSK_SATURA_B03.gfs', 'MSK_SATURA_B03.gml', 'MSK_SATURA_B04.gfs', 'MSK_SATURA_B04.gml', 'MSK_SATURA_B05.gfs', 'MSK_SATURA_B05.gml', 'MSK_SATURA_B06.gfs', 'MSK_SATURA_B06.gml', 'MSK_SATURA_B07.gml', 'MSK_SATURA_B08.gfs', 'MSK_SATURA_B08.gml', 'MSK_SATURA_B09.gml', 'MSK_SATURA_B10.gml', 'MSK_SATURA_B11.gfs', 'MSK_SATURA_B11.gml', 'MSK_SATURA_B12.gfs', 'MSK_SATURA_B12.gml', 'MSK_SATURA_B8A.gfs', 'MSK_SATURA_B8A.gml', 'MSK_SNWPRB_20m.jp2', 'MSK_SNWPRB_60m.jp2', 'MSK_TECQUA_B01.gfs', 'MSK_TECQUA_B01.gml', 'MSK_TECQUA_B02.gfs', 'MSK_TECQUA_B02.gml', 'MSK_TECQUA_B03.gfs', 'MSK_TECQUA_B03.gml', 'MSK_TECQUA_B04.gfs', 'MSK_TECQUA_B04.gml', 'MSK_TECQUA_B05.gfs', 'MSK_TECQUA_B05.gml', 'MSK_TECQUA_B06.gfs', 'MSK_TECQUA_B06.gml', 'MSK_TECQUA_B07.gml', 'MSK_TECQUA_B08.gfs', 'MSK_TECQUA_B08.gml', 'MSK_TECQUA_B09.gml', 'MSK_TECQUA_B10.gml', 'MSK_TECQUA_B11.gfs', 'MSK_TECQUA_B11.gml', 'MSK_TECQUA_B12.gfs', 'MSK_TECQUA_B12.gml', 'MSK_TECQUA_B8A.gfs', 'MSK_TECQUA_B8A.gml', 'RADIOMETRIC_QUALITY.xml', 'SENSOR_QUALITY.xml', 'T40REQ_20200114T065229_PVI.jp2', 'HTML', 'HTML', 'UserProduct_index.html', 'UserProduct_index.xsl', 'banner_1.png', 'banner_2.png', 'banner_3.png', 'star_bg.jpg', 'INSPIRE.xml', 'MTD_MSIL2A.xml', 'manifest.safe', 'rep_info', 'rep_info', 'S2_PDI_Level-2A_Datastrip_Metadata.xsd', 'S2_PDI_Level-2A_Tile_Metadata.xsd', 'S2_User_Product_Level-2A_Metadata.xsd']

* How was the directory created? Some tools like the console GUI can create these fake "folders" on s3: https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-folders.html

This folder has been uploaded to a S3 compatible storage as usual, and it can be used without the final /. Only this prevents the normal use. But what is problematic is that glob function gives path with this / in the end for directories.

* Is there a public S3 path you can point to that reproduces this for us to test?

I'm sorry, I've not found any public S3 enabling .SAFE products, but you could download as they are free: https://sentinelhub-py.readthedocs.io/en/latest/aws_cli.html
I could send you some product but they are quite heavy (several hundred of Mo minimum).

Here's the logic that checks directories/files on S3. It's not too complicated, but bsaically we check if S3 lists the exact path when we ask it for files. If it does, we assume it is a file.

I think I have the same problem in another place:
I am downloading a multi level directory, such as:

(the sub-directories are not empty)
And the downloaded product obtained is:

You can see that directories have been downloaded as files !
The code of the function does 2 different things according if we are in face of a file or a directory (and it adds a / if it is a directory !), so maybe it is related:
def download_to(self, destination: Union[str, os.PathLike]) -> Path:
    destination = Path(destination)
    if self.is_file():
        if destination.is_dir():
            destination = destination / self.name
        return self.client._download_file(self, destination)
    else:
        destination.mkdir(exist_ok=True)
        for f in self.iterdir():
            rel = str(self)
            if not rel.endswith("/"):
                rel = rel + "/"

            rel_dest = str(f)[len(rel) :]
            f.download_to(destination / rel_dest)

        return destination

All this is happening on a S3 compatible storage, I don't know if it is related.

All this is happening on a S3 compatible storage

What library/implementation? We do test MinIO, but maybe we are missing a test like this.
It does seem that the root cause is that our is_file_or_dir check does not work reliably for every set of files/metadata/s3 implementations. Just need to track down enough info to repro and fix.

Ouch I don't know, it is my university's storage...
I have no clue on this question 😢
If you want me to run additional tests on my side, just ask !

I'm told they are using Ceph and maybe OpenStack.
I don't know if it helps you...

🙇 thanks, I'll see if we can find a Ceph server to test against.

I have the same issue, but I'm using ordinary S3 (I think it is ordinary S3 at least)

@remi-braun @karolzlot The issues you were seeing should (hopefully) be fixed by #190.  I'm not sure about ceph since we test on MinIO and AWS S3.
If you get a chance, could you test the latest on master before we cut a release with these changes?
pip install git+https://github.com/drivendataorg/cloudpathlib

Hello,
Thanks for the update :)
Here is what I got:

Thanks @remi-braun. The recursion problem looks like a separate issue that we can investigate. Can you confirm that the is_dir calls that work here fail on the latest release?

Yes !

Thanks @remi-braun. The recursion problem looks like a separate issue that we can investigate. Can you confirm that the is_dir calls that work here fail on the latest release?

Do I need to open an issue on that ? Or you are investigating it ?

@pjbull @remi-braun  I also faced the same issue fixed it like this

Thanks @GeorgeSabu that fix looks good, and we'd accept a PR if you want to contribute!

@remi-braun @GeorgeSabu ^ filed #198 to track the recursion issue and the fix you pointed out. If you can contribute referencing that issue, it would be great. Thanks!


Bug in esp_hf_ag_api.c? (IDFGH-3614)

IDF release v4.1 in file esp-idf/components/bt/host/bluedroid/api/esp_hf_ag_api.c
Lines:

182
203
223
244
371
391
418
445
472
499

All lines read return (status = BT_STATUS_SUCCESS) ? ESP_OK : ESP_FAIL;
Am I right in saying BT_STATUS_SUCCESS is the first enum (i.e. zero), therefore the ternary operator will always evaluate to false, therefore ESP_FAIL?
Should the lines not be return (status == BT_STATUS_SUCCESS) ? ESP_OK : ESP_FAIL;

Hi
This status just indicates that ESP has successfully posted the task to BTC to process.
And BT_STATUS_SUCESS is the default status. There is nothing wrong with it, maybe you should run the hfp_ag example and add some log to verify the situation.
Thanks
Weitianhua

I have run the example and each of the above functions that are called in the example return ESP_FAIL because you are assigning BTC_STATUS_SUCCESS meaning whatever btc_transfer_context returned is overwritten.  Then the ternary operator checks, and since the value held by status is now BT_STATUS_SUCCESS (which is zero) it evaluates to false.  There is no comparison operator == at all to flip the logic.
I am 100% sure you're wrong here

@howroyd Is correct, it should read return (status == BT_STATUS_SUCCESS) ? ESP_OK : ESP_FAIL;
Here's a Compiler Explorer comparing the wrong and correct versions:
https://godbolt.org/z/9KoW8M

@howroyd
Oh.. sorry...probably I should go to the hospital to check my eyes...
It was my fault, I will fix it and sync this to Github ASAP.

@Wth-Esp no worries, easy mistake to make.  Glad I could help.
Interestingly, one of the MISRA rules is to always put the constant first when doing comparisons.  That way if you accidentally put = instead of == you get a compile time error for trying to assign to a constant. ie:
return (BT_STATUS_SUCCESS == status) ? ESP_OK : ESP_FAIL;

But the following won't compile:
return (BT_STATUS_SUCCESS = status) ? ESP_OK : ESP_FAIL;

@howroyd Oh! Thank you, I get the tip.

@howroyd
Oh.. sorry...probably I should go to the hospital to check my eyes...
It was my fault, I will fix it and sync this to Github ASAP.

The fix is not yet in esp-idf master tree.
No idea why it takes such long time for such trivial fix?

@AxelLin Thanks for the notes.

Sorry about this. Because of an internal problem with our CI checks, master branch has not pushed out fixes
We're working to deploy master ASAP.

Will update once the fix is available on GitHub.

will update once fix on release/4.1 is available, thanks.


Redirecting hardwaremodel output to a file

Signed-off-by: adarsh-zededa<EMAIL_ADDRESS>

Please provide details as to why is this required and what problem does it address @adarsh-zededa

Please provide details as to why is this required and what problem does it address @adarsh-zededa

I asked Adarash to split it out from PR #1317 even though it is required to make the single zedbox produce output since it runs everything in the background.


Missing add button on text blocks

When I add a new text block, it is not possible to add more blocks - neither above nor underneath.

I'll take a look

Did you already take a look on this topic?

I took a look and I think what you are seeing is the desired behaviour for now.
.st--hide-top-controls .st-top-controls .st-block-addition {
  display: none;
}

When we started on the redesign we tried to get rid of as much clutter as possible. So you are seeing the mode which could be described as "Get typing and then add extra content".
https://github.com/madebymany/sir-trevor-js/blob/master/src/editor.js#L161-L178

Whilst removing clutter is a noble cause, it is confusing from a UX standpoint as the hover isn't an obvious (also the '+' at the bottom isn't a high enough contrast, seriously, I missed it was there at first) interaction method since there's nothing to infer that hovering will cause something to happen.

Thank you for that information, @raffij!
My intention was the following: when I start with a blank editor and create a new text block, there is no chance to add any extra content - neither above nor underneath the new text block. You just pointed at the addition button at the top, for what I found a way to "reintroduce" it for the text block. Where is decided, if the button underneath is displayed?
Thanks also for your comment, @SgtOddball, which hits also my experience.

Thanks for your feedback both of you.
What you see currently is a way of getting something in place for our current working practices and then finding out how others want to use it and work towards a different solution or adding different functionality in through configuration changes.
@morpheus-87 something like this will do what you want.
.st-block:last-child .st-block-addition {
  display: block;
}


fix: csharp enum rendering associated values

Description
This PR adds an extension to enums, which allows the retrieval of the associated enum values.
Related issue(s)
fixes https://github.com/asyncapi/modelina/issues/336

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

17 of 17 (100.0%)  changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.02%) to 93.392%

Totals

Change from base Build<PHONE_NUMBER>:
0.02%

Covered Lines:
1880

Relevant Lines:
1919

💛  - Coveralls

These functions are primarily for #356, however, I feel they are needed in the standard enum generation 🤔 Let me know if you disagree!

@magicmatatjahu fix or feat? 🤔 Kinda thinking it should be feat


Refactor: Use nullable type hints for optional dependencies and param…

Fix Deprecation Notices in LexikJWTAuthenticationBundle
Summary
This pull request addresses multiple deprecation notices that occur when using the LexikJWTAuthenticationBundle with PHP 8.4. The fixes involve explicitly marking nullable parameters in constructors and methods to comply with PHP 8.4 requirements.
Fixed Issues

JWTAuthenticator::__construct
Implicitly marking parameter $translator as nullable is deprecated. The parameter is now explicitly marked as ?TranslatorInterface.

JWTAuthenticator::start
Implicitly marking parameter $authException as nullable is deprecated. The parameter is now explicitly marked as ?AuthenticationException.

JWTManager::__construct
Implicitly marking parameter $payloadEnrichment as nullable is deprecated. The parameter is now explicitly marked as ?PayloadEnrichmentInterface.

LcobucciJWSProvider::__construct
Implicitly marking parameter $clock as nullable is deprecated. The parameter is now explicitly marked as ?ClockInterface.

AuthenticationFailureHandler::__construct
Implicitly marking parameter $translator as nullable is deprecated. The parameter is now explicitly marked as ?TranslatorInterface.

Changes Made

Updated JWTAuthenticator to use ?TranslatorInterface and ?AuthenticationException where applicable.
Modified AuthenticationFailureHandler to use ?TranslatorInterface in the constructor.
Adjusted LcobucciJWSProvider to use ?Clock in the constructor for better nullability support.
Updated JWTManager to use ?PayloadEnrichmentInterface for the optional payload enrichment dependency.

Environment

PHP Version: 8.4
Symfony Version: 7.2
LexikJWTAuthenticationBundle Version: 3.1.0
PHPUnit: 9.6.21

Additional Context
These changes aim to ensure smooth operation in PHP 8.4 environments by addressing all known deprecation warnings related to nullable parameters.

There's already a PR on this topic : https://github.com/lexik/LexikJWTAuthenticationBundle/pull/1238

Fixed in #1238, thank you for the issue & PR @gzim324.


Add PyTorch to users.rst

I've seen PRs updating this file so adding PyTorch to the list as well as we're relying on this pretty heavily now!
Thanks for the hard work in maintaining this! It helps a lot to have reliable compat code!

Oups I did that from your fork instead of the main repo. Sorry for the noise!


[SPARK-44482][CONNECT] Connect server should can specify the bind address

What changes were proposed in this pull request?
When a machine has multiple network cards, we may only want users of a certain network segment to be able to connect to the connect server. I propose that in addition to specifying port, the connect server can also specify address.
https://github.com/apache/spark/blob/047fad5861daedbdb58111b223e76c05784f4951/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/config/Connect.scala#L26-L30
Why are the changes needed?
1.Connect server should can specify the bind address, improve flexibility.
2.As other Spark components, a bind address can also be specified, such as:
https://github.com/apache/spark/blob/047fad5861daedbdb58111b223e76c05784f4951/common/network-common/src/main/java/org/apache/spark/network/server/TransportServer.java#L147-L149
https://github.com/apache/spark/blob/047fad5861daedbdb58111b223e76c05784f4951/core/src/main/scala/org/apache/spark/SparkEnv.scala#L177-L196
Does this PR introduce any user-facing change?
No.
How was this patch tested?

Manually test.
Pass GA.

The manual testing process is as follows:
1.My local env as follows:
172.xxx.xxx.xxx
2.Set the following configuration in spark-defauls.conf

spark.connect.grpc.binding.address <IP_ADDRESS>
spark.connect.grpc.binding.port 5000`

3.Start connect server
sh sbin/stop-connect-server.sh
4.Specify the host name and port to connect to the server

Success Scenario:

Failure scenario:

@HyukjinKwon The GA finally turned green. 😄

Merged to master.


Dependency check fails with error message "Archive file was not found"

I recently integrated the plugin into the Gradle build. The dependency check is run as one of the steps in our CI build pipeline.
 I saw that there also was an issue with connection to the database earlier. Could this potentially be an issue with concurrency?
Caused by: org.owasp.dependencycheck.exception.ExceptionCollection: Unable to continue dependency-check analysis.
Unable to connect to the database

@bmuschko sorry for the extremely delayed response - the temp directory would be changed via temp.directory.  In 3.1.1 I also added some debug logging that should assist in tracking the problem down.

Circling back to really old issues - a lot of changes have been made since this was reported. Is this still an issue?

I have no more access to the project. You can close the issue if nobody else is running into it anymore.


Fix snapshot test environment variables

Our CI Test snapshots failed today, complaining that The workflow is not valid. .github/workflows/pytest-snapshots.yaml (Line: 18, Col: 5): A mapping was not expected.
Looking at the docs, a mapping is allowed, but apparently not a nested mapping. Thus, this PR removes the version level in the env mapping (and temporarily enables Test snapshots to see if they run now).
How to review

Read the diff and note that the CI checks all pass (in particular Test snapshot).

PR checklist

[x] Continuous integration checks all ✅
[x] Update tests; coverage checks both ✅
~[ ] Add, expand, or update documentation.~ Just CI.
~[ ] Update doc/whatsnew.~ Just CI.

Tests are all passing, happy to drop the TEMPORARY commit and merge this :)

Maybe we can add a pre-commit step that validates these.

e.g. https://github.com/mpalmer/action-validator?tab=readme-ov-file#using-pre-commit

The check is failing because of https://github.com/mpalmer/action-validator/issues/74, which confirms my suspicion: usually (see e.g. for ruff) you don't need to install the tool the hook is invoking yourself. The issue is just three weeks old, there's no associated PR yet. I'm not sure we want to do that ourselves, I don't know how long it would take to create that PR, have it merged and a new version released. It doesn't sound like a quick solution, whereas we might want to enable our snapshot tests again as soon as possible.
Maybe there's another workaround to install action-validator manually or maybe there's another action we could use, but we could also skip this for now.

we could also skip this for now.

Yes, to be clear: my "approve" review meant "LGTM as-is", and then that suggestion was a "P.S."/tangential thought on how to avoid this in the future.


Error in routing in Zuul Proxy Server

I changed my Zuul Proxy project from Dalston SR3 to Edgware.BUILD-SNAPSHOT and Edgware.M1 to use the FallbackProvider (which is not present in Edgware.M1), but when I go to the subsequent services routes, I'm getting the following log and then the error:

Log:

c.netflix.loadbalancer.BaseLoadBalancer  : Client: AnyService instantiated a LoadBalancer: DynamicServerListLoadBalancer:{NFLoadBalancer:name=AnyService,current list of Servers=[],Load balancer stats=Zone stats: {},Server stats: []}ServerList:null

Error:

2017-10-03 13:10:29.729  INFO 2596 --- [nio-8075-exec-9] o.apache.http.impl.execchain.RetryExec   : I/O exception (org.apache.http.conn.UnsupportedSchemeException) caught when processing request to {s}->https://localhost:8081: https protocol is not supported
2017-10-03 13:10:29.729  INFO 2596 --- [nio-8075-exec-9] o.apache.http.impl.execchain.RetryExec   : Retrying request to {s}->https://localhost:8081
2017-10-03 13:10:29.730  INFO 2596 --- [nio-8075-exec-9] o.apache.http.impl.execchain.RetryExec   : I/O exception (org.apache.http.conn.UnsupportedSchemeException) caught when processing request to {s}->https://localhost:8081: https protocol is not supported
2017-10-03 13:10:29.731  INFO 2596 --- [nio-8075-exec-9] o.apache.http.impl.execchain.RetryExec   : Retrying request to {s}->https://localhost:8081
2017-10-03 13:10:29.731  INFO 2596 --- [nio-8075-exec-9] o.apache.http.impl.execchain.RetryExec   : I/O exception (org.apache.http.conn.UnsupportedSchemeException) caught when processing request to {s}->https://localhost:8081: https protocol is not supported

I think it's something simple about HTTPS, but I need help.

Can you share your Zuul configuration?
The ZuulFallbackProvider interface was available in Dalston.SR3.

Oh I see you want the FallbackProvider interface, yes that is only in the Edgware.

(Disregard the issue of FallbackProvider, the problem is other and occurring with both versions)
My Zuul configuration:
zuul:
  routes:
    AnyService:
      path: /any/**
      sensitive-headers: Set-Cookie, Authorization

hystrix:
  command:
    default:
      execution:
        isolation:
          thread:
            timeoutInMilliseconds: 60000
        timeout:
          enabled: true

ribbon:
  ReadTimeout: 60000
  ConnectTimeout: 60000
  ServerListRefreshInterval: 30000

However, the error does not seem to be this. All my services only work with ssl enabled (https), and by log, the http client used does not seem to be able to do https requests.
PS: My certificate is being loaded in all instances, including Zuul Proxy, so it is not an error of this, it seems to be the http client.

This log
c.netflix.loadbalancer.BaseLoadBalancer  : Client: AnyService instantiated a LoadBalancer: DynamicServerListLoadBalancer:{NFLoadBalancer:name=AnyService,current list of Servers=[],Load balancer stats=Zone stats: {},Server stats: []}ServerList:null

would seem to indicate that Ribbon does not have any servers for AnyService.  Are you using service discovery as well?

Yes, it's all right. This error occurs when switching from the version to the ones I mentioned, the Dalston SR3 works normally. Using Dalston SR3, when accessing any route from AnyService via Zuul Proxy I get this log:
c.netflix.loadbalancer.BaseLoadBalancer  : Client: AnyService instantiated a LoadBalancer: DynamicServerListLoadBalancer:{NFLoadBalancer:name=AnyService,current list of Servers=[],Load balancer stats=Zone stats: {},Server stats: []}ServerList:null

But soon after:
c.n.l.DynamicServerListLoadBalancer      : DynamicServerListLoadBalancer for client AnyService initialized: DynamicServerListLoadBalancer:{NFLoadBalancer:name=AnyService,current list of Servers=[],Load balancer stats=Zone stats: {},Server stats: []}ServerList:org.springframework.cloud.netflix.ribbon.eureka.DomainExtractingServerList@5843c98c

And this works fine. Alread with the Edgware.*:
c.n.l.DynamicServerListLoadBalancer      : DynamicServerListLoadBalancer for client AnyService initialized: DynamicServerListLoadBalancer:{NFLoadBalancer:name=AnyService,current list of Servers=[],Load balancer stats=Zone stats: {},Server stats: []}ServerList:org.springframework.cloud.netflix.ribbon.eureka.DomainExtractingServerList@6bfd4734

Similar messages in log for #2375


docker-compose --project-name=ocean up fails

Expected Behavior
If i run
export OCEAN_VERSION=latest
docker-compose --project-name=ocean up

from https://github.com/oceanprotocol/docker-images/blob/develop/README.md
it should spin up ocean's default env. This leads to an error now. I think there is something missing in the documentation?
Current Behavior
Creating ocean_keeper-contracts_1 ... error
Creating ocean_mongodb_1          ...

ERROR: for ocean_keeper-contracts_1  Cannot create container for service keeper-contracts: create .: volume name is too short, names should be at least two alphCreating ocean_mongodb_1          ... done

ERROR: for keeper-contracts  Cannot create container for service keeper-contracts: create .: volume name is too short, names should be at least two alphanumeric characters
ERROR: Encountered errors while bringing up the project.

can confirm. The updated docker-compose.yml now requires a bunch of environment variables which are only set by using the new start_ocean.sh script. So guess would make sense to default to the script and extending it a bit more, instead of trying to figure out how to support the script and the raw commands. For that we should change script a bit:

default to stable versions of all components
make latest versions configurable with argument, e.g. --latest

Right now, the script can be used to get the latest versions of everything so it's essentially the same outcome as the currently documented commands to get latest of everything
I can look into that, currently in testing mode with all this, and in combination with pleuston

I removed the docker-compose commands from the readme in here so at least community doesn't run into this issue https://github.com/oceanprotocol/docker-images/pull/32
Don't know if it's worth trying to make everything work with no environment variables set and the raw docker-compose command

It should be fixed.


add dockerfile

add dockerfile

why PostgreSQL 9.6? can this be overridden easily?

why PostgreSQL 9.6? can this be overridden easily?

PostgreSQL 9.6 is just default version. I have updated the PR to make version overridable.
Thanks.

The base image (debian buster) only supports LLVM 7.0 which failed to build extensions for postgresql 11+.

The base image (debian buster) only supports LLVM 7.0 which failed to build extension for postgresql 11+.

Fixed. Adding the following code in Makefile:
COMPILE.cxx.bc = $(CLANG) -xc++ -Wno-ignored-attributes $(BITCODE_CXXFLAGS) $(CPPFLAGS) -emit-llvm -c

%.bc : %.cpp
	$(COMPILE.cxx.bc) -o $@ $<
	$(LLVM_BINPATH)/opt -module-summary -f $@ -o $@


purharm-db-doobie module

pureharm-doobie-db as a submodule of pureharm-db. It ought to be similar to the db-slick module, as it will provide qiimplementations of DaoAlgebra for ConnectionIO and for F.

closed in #104


How do you download all the river data

How do you download the whole river data

That is not currently possible. Although I would like to add that. The Makefile shows how the data is created, if you want to do it yourself.


[BZ#1951660] Add plan clone action for easier replacement of failed/canceled plans

https://bugzilla.redhat.com/show_bug.cgi?id=1951660
Ilanit:

Description of problem:
A migration plan can be edited only if it was not started yet.
In case the plan was cancelled,
or after a migration plan fails the plan is no longer editable.
Use cases for example:

If the failure is caused by the network/storage mapping,
users cannot edit the plan and change the mapping.

If a plan was cancelled since users didn't set the plan correctly,
it is not possible. Need to delete current plan and create a new one.

Can edit option be added in cases of plan failed/cancelled?

Vince:

After meeting with Jeff, Miguel, and Mike we propose adding a "plan clone" feature first because that will be easier to implement for 2.1. If a user wants to edit a plan that has already been started and canceled, or has completed and failed, they can clone it and create a new plan which can be edited. The user will have the option of whether to include already migrated VMs in the clone (default is no). If the user chooses to include already migrated VMs in the clone the UI will alert them that they must be manually removed from the target cluster before migration or they will fail.

@vconzola if we still plan to do this for v2.1 we'll need to finalize the design. I assume we'll add a kebab action (what should that be called? "Clone"? "Create copy"? "Duplicate"?) which will open a modal with either a yes/no choice to include already-migrated VMs or perhaps a checklist of which VMs to include (with already-migrated defaulting to unchecked), and the warning you mention if already-migrated VMs are selected.

@mturley I'd like to see some real requirements from Miguel before we jump into this.

Ok. I'll mark him as needinfo on the BZ.
cc @mperezco

Closing this because we decided not to have a plan clone feature. Instead we will consider later the impact of allowing the user to edit plans that have been run previously.


use angular.merge instead of utils.deepCopy

current: utils.deepCopy
recommended replacement: angular.merge (src) (meat of the implementation).

That function appears to be new in 1.4--I don't see an equivalent in 1.3?

ahh, That explains it.
perhaps do feature detection with a polyfill then? Or at least compare deepCopy to merge (they differ in some significant ways) and see if they are catching some corner cases we are not?

Gave this 30 mins today and couldn't get in running--test units fail with awkward errors and the data in $bindTo objects is not as expected. Perhaps will try again when 1.4 leaves beta. Closing for now as won't fix. Feel free to send a PR if you feel this is important.

polyfill for angular<1.4.0
    if (!angular.merge) {
      angular.merge = (function mergePollyfill() {
        function setHashKey(obj, h) {
          if (h) {
            obj.$$hashKey = h;
          } else {
            delete obj.$$hashKey;
          }
        }

        function baseExtend(dst, objs, deep) {
          var h = dst.$$hashKey;

          for (var i = 0, ii = objs.length; i < ii; ++i) {
            var obj = objs[i];
            if (!angular.isObject(obj) && !angular.isFunction(obj)) continue;
            var keys = Object.keys(obj);
            for (var j = 0, jj = keys.length; j < jj; j++) {
              var key = keys[j];
              var src = obj[key];

              if (deep && angular.isObject(src)) {
                if (isDate(src)) {
                  dst[key] = new Date(src.valueOf());
                } else {
                  if (!angular.isObject(dst[key])) dst[key] = angular.isArray(src) ? [] : {};
                  baseExtend(dst[key], [src], true);
                }
              } else {
                dst[key] = src;
              }
            }
          }

          setHashKey(dst, h);
          return dst;
        }

        return function merge(dst) {
          return baseExtend(dst, [].slice.call(arguments, 1), true);
        }
      })();
    }


Model checkpoints

Do you have any model checkpoints?

I apologize, but we are unable to provide the model weights because this paper's model and dataset directly benchmark Fujifilm's film simulations. Releasing the model weights could potentially lead to commercial applications infringing on Fujifilm's patents and copyrights. If you are interested, you can train the model yourself to obtain the weights.

Do you have any model checkpoints?

@tttonyalpha Hi! We've released the checkpoints for MIT-Adobe 5K and HDR+ for your convenience. You can find them here: https://github.com/CXH-Research/FilmNet/releases/tag/weights.


spint has pysal as a dependency, should be libpysal

Realizing that spreg is not yet a stand-alone package, this can't be fixed at present. Just raising it to flag issues we have to fix as part of the larger pysal refactoring.
  File "/home/serge/anaconda3/envs/pysal2/lib/python2.7/site-packages/nose/loader.py", line 418, in loadTestsFromName
    addr.filename, addr.module)
  File "/home/serge/anaconda3/envs/pysal2/lib/python2.7/site-packages/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/home/serge/anaconda3/envs/pysal2/lib/python2.7/site-packages/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/home/serge/Documents/p/pysal/src/pysal-reorg/spint/spint/__init__.py", line 1, in <module>
    from . import gravity
  File "/home/serge/Documents/p/pysal/src/pysal-reorg/spint/spint/gravity.py", line 21, in <module>
    from pysal.spreg import user_output as User
ImportError: No module named pysal.spreg

Sounds good. If I recall, spreg will eventually be stand-alone as well, so no need to make changes other than pysal.spreg-->spreg onces spreg ready?

spreg is ready

:smile: :tada:

This is very nice indeed!

Is this issue resolved then?

I think so


OSX build

Just a quick try out, and found that it does not build on OSX. There seems to be some linux-specfics used, e.g. ".../" instead of "../../". A quick fix might be required if OSX support gets a chance to your list of goals?
venus:spooky wc$ mkdir -p build/android && cd build/android
venus:android wc$ cmake -DCMAKE_TOOLCHAIN_FILE=[...]/toolchain-android.cmake -DANDROID_ABI="armeabi" -DCMAKE_BUILD_TYPE=Debug -DBUILD_SHARED=ON ../..
CMake Error at /usr/local/Cellar/cmake/3.5.2/share/cmake/Modules/CMakeDetermineSystem.cmake:104 (message):

It should build fine on OSX, however it was never tested.
[...] should be replaced with the actual path by the user. I forgot to mention it in the installation guide, I'll add it right away. Thanks

I replaced all [...] in INSTALL.md with "/path/to", so that nobody will get confused.


Add SVG placeholder support

It would be really neat to add https://github.com/EmilTholin/image-trace-loader an option for next-optimized-images. The benefit of that vs LQIP is more visual detail without sacrificing file size.

This looks really cool! I didn't know this technique, I'll definitely try to add an option for this

I just had to try this out and published v2.4.0 which supports this feature with the ?trace resource query. Thank you for the idea, I really like this one :)


The USER_CODE in device flow has to be unique

The USER_CODE in the device code has to be unique. Currently, the user_code has only 6 characters for usability. The queries in the code[1] need to be checked on whether the user_code is actually treated as a unique. Currently, the USER_CODE does not have a unique constraint in the DB.
[1] - https://github.com/wso2-extensions/identity-inbound-auth-oauth/blob/master/components/org.wso2.carbon.identity.oauth/src/main/java/org/wso2/carbon/identity/oauth2/device/dao/SQLQueries.java

A temporary unique constraint has been added to USER_CODE with this PR.

The 6-character USER_CODE can be made unique for a specific time interval. Say the validity period for the USER_CODE (T_val) is 10 minutes. Let's quantize time into intervals of length 2*t_val and call it T_q .
T_q = 2*T_val
To do this, define the quantized time intervals from the epoch time.

take the current time based upon epoch time and find the quanta which it resides in. Then, generate a USER_CODE that is unique in that quanta. Thus, we can guarantee that the generated USER_CODE is unique during its validity period. (T_val)
In the DB table, a new field should be created to store the T_q which the issued USER_CODE belongs to. A celan-up task service should be run periodically (every 12 hours or so) to delete the expired entries.
In the clean up service, it should have a limit as to the number of entries it should delete in one session (eg: 1000 entries max) since the table would have to be locked during this task.

As the device flow is already available in IS-5.10.0 we also need to look into handling any possible migration concerns that can come with the fix for this issue.

Related PRs:
Fix: https://github.com/wso2-extensions/identity-inbound-auth-oauth/pull/1569
Schema change: https://github.com/wso2/carbon-identity-framework/pull/3461
Migration: https://github.com/wso2-extensions/identity-migration-resources/pull/196


kubefwd communication behind a proxy

Hello,
I am trying to setup local port forwarding to a Kubernetes Cluster. I am behind a proxy. Is there a way to allow communication over a proxy. Here are the log details;

kforward
2018/12/24 15:39:24  _          _           __             _
2018/12/24 15:39:24 | | ___   | |_   ___ / | | |
2018/12/24 15:39:24 | |/ / | | | ' \ / _ \ |\ \ /\ / / _  |
2018/12/24 15:39:24 |   <| || | |) |  /  |\ V  V / (| |
2018/12/24 15:39:24 |_|_\,|./ _||   _/_/ _,_|
2018/12/24 15:39:24
2018/12/24 15:39:24 Version 1.4.10
2018/12/24 15:39:24 https://github.com/txn2/kubefwd
2018/12/24 15:39:24
2018/12/24 15:39:24 Press [Ctrl-C] to stop forwarding.
2018/12/24 15:39:24 'cat /etc/hosts' to see all host entries.
2018/12/24 15:39:24 Loaded hosts file /etc/hosts
2018/12/24 15:39:24 Hostfile management: Original hosts backup already exists at /etc/hosts.original
2018/12/24 15:39:24 Error forwarding service: Get https://gav-cu-ine-dev-aks-91bc5af0.hcp.centralus.azmk8s.io:443/api/v1/namespaces/default/services: dial tcp: lookup gav-cu-ine-dev-aks-91bc5af0.hcp.centralus.azmk8s.io on <IP_ADDRESS>:53: no such host
2018/12/24 15:39:24 Done...
Thanks
Sohil
Senior Staff Software Engineer, GE Aviation

hi,
i have the same problem. my API-server is only available through a bastion-host. to connect to it, i have a KUBECONFIG which only works if i go through the bastion. so i do:
ssh -D 8123 bastion.my.host

now i have a local SOCKS proxy which i export with
export https_proxy=socks5://localhost:8123

and voila, i can use kubectl commands as kubectl is using the proxy setting.
now i start kubefwd:
sudo https_proxy=socks5://localhost:8123 kubefwd svc -c my-kube-config

and now i get errors:
ERROR: error upgrading connection: error sending request: Post https://<my-ip>:6443/api/v1/namespaces/default/pods/nsqd-0/portforward?timeout=32s: EOF

it looks like kubefwd is using the proxy setting to query the information but not using the proxy when creating the socket connection?


Performance issue on Ubuntu 15.04

I have been seeing degraded performance lately on the master branch on my Ubuntu machine, as compared to my Macbook. I have been trying to bisect the issue, and I think I finally have: It appears to be caused by 61daeca. When I compile from 61daeca and run Butterflow on a 400x400x20sec video, it takes 4 minutes to run, but when I compile from c1003ed (the previous commit) and run it on the same video, it takes 0.26 minutes.
Although the commit is aptly named ("Use veryslow preset when encoding"), I never would have suspected it, for two reasons:

I have been running ffmpeg veryslow encoding on Butterflow output for awhile now and it is relatively performant, and more strangely:
I get the same bad performance even when running Butterflow in lossless mode!

So to sum up, on Ubuntu only, with or without the lossless flag:
butterflow -l -v -r 60 -o interpolated.mp4 -s full,spd=0.75 original.mp4
# takes 0.26 minutes compiled from c1003ed
# takes 3.97 minutes compiled from 61daeca
ffmpeg -y -i interpolated.mp4 -preset veryslow compressed.mp4
# takes 0.11 minutes

Thanks for the help!

PS: I'm now posting Butterflow-interpolated satellite videos fairly regularly, on the top 4 playlists on this youtube channel. I have been keeping up with the master branch for these renders, and they look great.
I have also been trading e-mails with the scientists at NOAA who created the processed images, and they're really interested in experimenting with motion interpolation. I'm helping them get set up with Butterflow, and they may be using it more in the future. Exciting!

Thanks for finding where this problem originated.
These are rendering times I'm getting using Butterflow compiled from the perf_regression branch with a 400x400 30fps 20sec test file on the same filesystem as the temporary directory. I'm running Windows and Linux on the same machine so both are using my GTX 760.
Windows 10
encoder preset		time			    lossless time
x264    veryslow	1.100 (0.32 MB)    	1.140 (6.26 MB)
x265    ultrafast   1.120 (1.41 MB)     1.190 (10.9 MB)
x264    ultrafast   1.130 (1.91 MB)     1.130 (10.2 MB)
x265    veryslow    1.420 (0.37 MB)     2.010 (7.34 MB)

Arch Linux
x264 	ultrafast	0.598 (1.60 MB)		0.609 (8.49 MB)
x264 	veryslow	0.603 (0.28 MB)		0.601 (5.07 MB)
x265 	ultrafast	0.604 (1.20 MB)		0.599 (9.20 MB)
x265	veryslow	1.090 (0.31 MB)		1.630 (5.99 MB)

I made the test video with ffmpeg -y -f lavfi -i testsrc=duration=20:size=400x400:rate=30:decimals=2 -c:v libx264 -crf 18 a.mp4 and ran butterflow -v -r 60 -s full,spd=0.75 a.mp4 against it.
It's clear that using x265 with the veryslow preset is the problem. Butterflow uses x264 on Windows and OS X by default. On Linux I set it to use x265. It explains why you're seeing a difference in performance between your machines.
I'm going to push a fix for this to master over the weekend but you can work around this on Ubuntu right now by setting default['cv'] = libx264 and preset to something that is not veryslow, like medium, in settings.py.

Thanks, that makes a huge difference for me! Large videos that were taking literally 2+ hours with x265 are now taking around 5.5 minutes with x264.
I did a quick test comparing ultrafast + lossless in Butterflow followed by a veryslow ffmpeg compression pass, vs. just letting Butterflow compress using veryslow. I expected them to be about the same, but was surprised to find that the latter creates a file about 2x as big. Any idea why this might be the case?
# with veryslow in settings.py
butterflow -v -r 60 -o interpolated-lossless -s full,spd=0.75 original.mp4
# creates 91.7MB file in 6.31 minutes

# with ultrafast in settings.py
butterflow -v -l -r 60 -o interpolated-lossless.mp4 -s full,spd=0.75 original.mp4
# creates 1.7GB file in 5.22 minutes
ffmpeg -y -i interpolated-lossless.mp4 -c:v libx264 -preset veryslow interpolated.mp4
# compresses to 48.1MB file in 5.25 minutes
ffmpeg -y -i interpolated-lossless.mp4 -c:v libx265 -preset veryslow interpolated.mp4
# ...got bored and ctrl+C'ed after <10% completed in 5 minutes

Butterflow uses crf=18 with the veryslow preset when not using the lossless flag. You didn't specify a crf value in your command when re-encoding/compressing so FFmpeg defaults to crf=23. Generally speaking, a lower crf value is a higher quality and will yield a bigger file size. You are getting a smaller size because you're using a higher crf (compressing more).
Try adding -crf 18 to your FFmpeg command. You should get a file size that is closer to Butterflow using veryslow.
These are my results with a 1920x1080 30fps 20sec test video.
Made with Butterflow on Windows 10
encoder     preset      quality     time & size
x264        veryslow    crf=18      7.89 mins (3.45MB)
x264        ultrafast   qp=0        7.90 mins (54.2MB)

Re-encoded qp=0 file with FFmpeg
x264        veryslow    crf=18 	    0.44 mins (3.46MB)
x264        veryslow    crf=23 	    0.44 mins (2.24MB)

By the way, those videos look awesome and thanks for sharing that tidbit of news :smile:.

Ah thanks, that makes sense. In that case, I'll just let Butterflow handle the compression, and skip the lossless step (I'm deleting the lossless ones anyway now to save HD space). I should probably be using crf 18 anyway.


Getting error "EIP-3855 is not supported in one or more of the RPCs used"

Component
Forge
Have you ensured that all of these are up to date?

[X] Foundry
[X] Foundryup

What version of Foundry are you on?
forge 0.2.0 (ba399ae5 2024-04-30T13:37:57.271979000Z)
What command(s) is the bug in?
forge script
Operating System
macOS (Apple Silicon)
Describe the bug
The issue arises when using the Foundry toolchain and running the forge script command on Moonbeam. The system returns an "EIP-3855 is not supported in one or more of the RPCs used" error.
This issue is related to fee estimation for EIP-1559. Foundry uses the eth_feeHistory method to calculate the gas fee, and it sends the following request:
{"method":"eth_feeHistory","params":[10,"latest",[20.0]],"id":1,"jsonrpc":"2.0"}
While this works fine with Ethereum nodes, on a Moonbeam node, it returns the following error:
"code": -32602,
"message": "Invalid params",
"data": "invalid type: integer `10`, expected a (both 0x-prefixed or not) hex string or byte array containing between (0; 32] bytes at line 2 column 10"

Essentially, Foundry expects the node to accept an integer as an argument, but Moonbeam (and some other chains) does not. The issue is not specific to Moonbeam alone.
To reproduce the issue, you can try the following curl commands:
curl https://rpc.api.moonbase.moonbeam.network/ \
-X POST \
-H "Content-Type: application/json" \
--data '{"method":"eth_feeHistory","params":[10,"latest",[20.0]],"id":1,"jsonrpc":"2.0"}'
bash

curl https://ethereum-sepolia-rpc.publicnode.com/ \
-X POST \
-H "Content-Type: application/json" \
--data '{"method":"eth_feeHistory","params":[10,"latest",[20.0]],"id":1,"jsonrpc":"2.0"}'

A quick fix for this issue is to add the --legacy flag when running the forge script command with Foundry. This will disable the use of eth_feeHistory and instead use the legacy fee estimation method, which should work with both Ethereum and Moonbeam nodes.
But according to Ethereum's API doc, the first parameter should be a hex string instead of a integer.
Would you consider fixing it please?

thanks for flagging, this is a client bug


chore(deps): revert tarteaucitronjs from 1.9.7 to 1.8.4

Linked to #1382
With tac > 1.8.4 (from 1.9.1) we got this error on Netlify:
09:43:12.735 Uncaught TypeError: document.getElementById(...) is null
    color https://deploy-preview-1388--boosted.netlify.app/docs/5.2/assets/js/docs.min.js:6
    color https://deploy-preview-1388--boosted.netlify.app/docs/5.2/assets/js/docs.min.js:6
    addService https://deploy-preview-1388--boosted.netlify.app/docs/5.2/assets/js/docs.min.js:6
    load https://deploy-preview-1388--boosted.netlify.app/docs/5.2/assets/js/docs.min.js:6
    addScript https://deploy-preview-1388--boosted.netlify.app/docs/5.2/assets/js/docs.min.js:6
    addInternalScript https://deploy-preview-1388--boosted.netlify.app/docs/5.2/assets/js/docs.min.js:6
    load https://deploy-preview-1388--boosted.netlify.app/docs/5.2/assets/js/docs.min.js:6
    onload https://deploy-preview-1388--boosted.netlify.app/docs/5.2/assets/js/docs.min.js:6
docs.min.js:6:45074

Let's revert this change while we don't know where the issue comes from.

No error in the console log for this URL https://deploy-preview-1389--boosted.netlify.app/ in Chrome nor Firefox. So LGTM!


Looking for detailed documentation

Is there detailed documentation anywhere? If I click F1 in the application hoping to see a context help for stencils and their properties and possible with example usage - I'm seeing an error "Unhandled exception" where the corresponding log file shows: "Threat Modeling Tool, Assembly version 'TMT7, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=69c3241e6f0468ca', today is 'Tuesday, March 17, 2020 4:48:13 PM'Exception information:
System.ComponentModel.Win32Exception (0x80004005): The system cannot find the file specified
at System.Diagnostics.Process.StartWithShellExecuteEx(ProcessStartInfo startInfo)
at System.Diagnostics.Process.Start()
at System.Diagnostics.Process.Start(ProcessStartInfo startInfo)
at ThreatModeling.ViewModel.HelpCommand.ExecuteImp(Object parameter)
at ThreatModeling.ViewModel.Commands.TrackedCommand.Execute(Object parameter)"
I'm using version 7.3..00206.1.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 60c188c4-9aee-f131-f9c4-2582fcc2db6e
Version Independent ID: bf54e8b2-be4c-2d8b-1bfb-d16fc95da11e
Content: Microsoft Threat Modeling Tool overview - Azure
Content Source: articles/security/develop/threat-modeling-tool.md
Service: security
Sub-service: security-develop
GitHub Login: @jegeib
Microsoft Alias: jegeib

@Pikhulya Thank you for the feedback. We are actively investigating and will get back to you soon.

@Pikhulya At this point there is limited documentation available on this tool. The getting started guide talks about building the threat model and we have some threat models on the TMT github template repo and Azure Architecture center Github covering the following services. The F1 help file is not built in on this and the current documentation is limited hence you see that exception. I would suggest you to engage with the team directly on tmtextsupport[at]microsoft[dot]com.  I will also inform the author @jegeib internally for the same.
Thank you.


Update rand dependency version

The dependency on rand appears to be out of date. It appears PR #7 addresses this.
https://crates.io/crates/rand

Fixed in version 0.2.1.


Helm deletes secrets too soon

Output of helm version:
Client: &version.Version{SemVer:"v2.7.2", GitCommit:"8478fb4fc723885b155c924d1c8c410b7a9444e6", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.7.2", GitCommit:"8478fb4fc723885b155c924d1c8c410b7a9444e6", GitTreeState:"clean"}
Output of kubectl version:
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.4", GitCommit:"5ca598b4ba5abb89bb773071ce452e33fb66339d", GitTreeState:"clean", BuildDate:"2018-06-06T08:13:03Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.3", GitCommit:"435f92c719f279a3a67808c80521ea17d5715c66", GitTreeState:"clean", BuildDate:"2018-11-26T12:46:57Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}
When adding a post-delete hook to a chart, with a secret to pull it from a private repo, the hook cannot run because Tiller has already deleted the secret by the time it tries to pull the post-delete hook image.
Helm should keep secrets around at least until it has loaded all images, including all hook job images.
If this is fixed in a later version of Helm, please say so and I'll see about getting a later version installed.

@Jamie-sas would you mind verifying if this is still a bug with a more recent version of Helm?

When adding a post-delete hook to a chart, with a secret to pull it from a private repo, the hook cannot run because Tiller has already deleted the secret by the time it tries to pull the post-delete hook image.

I think this is kind of intended by design, though I can see why it would occur and why this would be frustrating. By design, a post-delete hook runs after everything has been deleted, and therefore should not assume that it can access any resources created from the chart itself.
Perhaps a pre-delete hook may be better suited here for your use case?

I am attempting to get the OK to install a more recent version of Helm, as it is not entirely my decision.
Yes, switching to a pre-delete hook instead does work, but in my opinion shouldn't be necessary.  If Helm supports pulling images from private repos, it should support that for ALL images.  There is also the question of whether or not everything has quiesced enough for a pre-delete to work, where a post-delete definitely does.  I'm trying to figure out how to check that enough to satisfy my own curiosity.
At the very least, if you don't plan to make this work, it should be very clearly documented as an exception.

closing as inactive.