HADOOP-17689. Avoid Potential NPE in org.apache.hadoop.fs

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 33s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

-1 :x:
test4tests
0m  0s

The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

-1 :x:
mvninstall
33m 52s
/branch-mvninstall-root.txt
root in trunk failed.

+1 :green_heart:
compile
20m 57s

trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04

+1 :green_heart:
compile
18m  7s

trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08

+1 :green_heart:
checkstyle
1m  5s

trunk passed

+1 :green_heart:
mvnsite
1m 32s

trunk passed

+1 :green_heart:
javadoc
1m  7s

trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04

+1 :green_heart:
javadoc
1m 37s

trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08

+1 :green_heart:
spotbugs
2m 20s

trunk passed

+1 :green_heart:
shadedclient
15m 46s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
0m 52s

the patch passed

+1 :green_heart:
compile
20m 11s

the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04

+1 :green_heart:
javac
20m 11s

the patch passed

+1 :green_heart:
compile
18m  4s

the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08

+1 :green_heart:
javac
18m  4s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
1m  3s

the patch passed

+1 :green_heart:
mvnsite
1m 31s

the patch passed

+1 :green_heart:
javadoc
1m  3s

the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04

+1 :green_heart:
javadoc
1m 38s

the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08

+1 :green_heart:
spotbugs
2m 31s

the patch passed

+1 :green_heart:
shadedclient
15m 52s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
17m  1s

hadoop-common in the patch passed.

+1 :green_heart:
asflicense
0m 55s

The patch does not generate ASF License warnings.

178m  7s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3008/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/3008

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell

uname
Linux ebf78dde868b 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 2ebe30c4d148a0eba3aca0380648e40097b51c1f

Default Java
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3008/1/testReport/

Max. process+thread count
3152 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3008/1/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

QA failure is not relevant.

Merged it. Thanks for your contribution, @virajjasani!


fix(cira): cira config is case insensitive to pull the data from vault

are we lowercasing upon insert?


apic mcecm process metric exporter

Creating prometheus exporter to monitor new ACI Bug related to mcecm process.

@artherd42 @Kuckkuck Could you please review, approve and merge this PR?


Description
Open issues:
Design:

[x]  Integrated new typography
[ ]  Spacing is defined
[ ] Documentation final

Tokens:

[ ]  Typography tokens ready

Design
https://www.figma.com/design/JIT5AdGYqv6bDRpfBPV8XR/Foundations-%26-Components-Next-Level?node-id=26-49
Tokens

[ ] Tokens for this component are ready

DEVs proposal
Base styles: body
Token set name: Element/Label

post-label-font-weight (Label large bold (default))
post-label-gap-block-start https://github.com/swisspost/design-system/labels/needs%3A discussion
post-label-gap-block-end https://github.com/swisspost/design-system/labels/needs%3A discussion

Tasks
### 💻 Tasks
- [ ] Review Design (All states present? Possible to implement?)
- [ ] https://github.com/swisspost/design-system/issues/3496
- [ ] Use generic typography tokens from the elements collection
- [ ] HTML/CSS implementation
- [ ] Check if documentation is still ok
- [ ] Visual regression tests
- [ ] Design review

Default label styles can be inherited from the body. Other label variants are used in specific contexts, so the context should provide the overrides. Do we even need specific label styles or is it pure overhead?
-> If we define default label styles using the tokens, we'd get style changes to the default label without any change, but as long as the label has the same styles as default text, we'd ship code that does not do anything.

@oliverschuerch please add information about what's needed from token side

We will not create tokens to use in code for this. As @gfellerph described, these text styles are used in different contexts (buttons, inputs, etc.). and will be tokenized there, per component.
There might be some FigmaOnly typography tokens for this, to simplify things in Figma, but that's all.
@Cian77 we will also not provide the possibility to use those styles through utility css-classes, because we don't think this should be possible. Do you agree?
If @Cian77 agrees to the above question, we can close this issue.

@oliverschuerch @gfellerph @Cian77 Task unclear, can we close the ticket now or is there more needed? Ticket seems not ready for tokenization.

Closed according to discussion, no specific label tokens will exist. Instead each Component uses its own token.


Compatibility Fixups

This addresses a couple issues on dask & distributed master

removal of docs_wraps: https://github.com/dask/dask/pull/5871/files
gen_with_timeout -> asyncio.wait_for (https://github.com/dask/distributed/pull/3372/files).

cc @jrbourbeau, I suspect the changes needed in https://github.com/dask/dask-ml/commit/3309b33255c55b98dc707ce37f86dc7dcf8dafcf fall under "expected changes", but just to confirm.

Yeah, I recall needing to increase the disconnect timeout to reliably avoid timeout errors when making the gen.with_timeout -> asyncio.wait_for changes


Fix for pallete-based image(converts them to full RGB)

As per https://stackoverflow.com/questions/43267807/valueerror-cannot-filter-palette-images-during-pytesseract-conversion , this change allows pallet-based images to be converted to full RGB in order to use the PIL filters.

@pixelexel
You saved my life!


Questions: how to optimize the speed of landmark detection code

Hi Patrikhuber,
I would like to run only landmark detection on your SDM code, and I would like to make the prediction code run faster as long as the landmark accuracy is good for me.
as you might know that, the most time needed for SDM landmark is for HOG calculation, and the time is mostly related with the size of ROI image calculated in adaptive_vlhog.hpp file.
my questions are:

in your rcr-train.cpp file, you set hog_params as follows:
std::vectorrcr::HoGParam hog_params{
{ VlHogVariant::VlHogVariantUoctti, 5, 11, 4, 1.0f },
{ VlHogVariant::VlHogVariantUoctti, 5, 10, 4, 0.7f },
{ VlHogVariant::VlHogVariantUoctti, 5, 8, 4, 0.4f },
{ VlHogVariant::VlHogVariantUoctti, 5, 6, 4, 0.25f } };
why is num_cells 5? shouldn't it be an even number?

in order to keep the accuracy and promote the speed, what are the best hog_params according to your experience?

in rcr-train.cpp file, the regressor level is 4. should it be smaller? what is the best regressor level according to your experience?

Hi,
1: I think an odd number is fine ;-) But for this one you should really look at the VlHog documentation, it will have all the details!
2: The ones that are set are pretty good and also a good trade-off. You can search through the issue list here, I think in one of the issues I posted a table or XLS with some parameter values and results.
3: 4 is the best trade-off imho, 3 is too little, 5 gives slightly better result but is slower/bigger.

Thanks Patrikhuber for your answer!
I tested some parameters  and would like to share my test result with you and all developers:
I tried to reduce the cell size from big to small, and I found out that the following HOG parameters gave the best trade-off between and performance and speed:
rcr::HoGParam hogptemp1 = { VlHogVariant::VlHogVariantUoctti, 4, 6, 4, 0.6f };
rcr::HoGParam hogptemp2 = { VlHogVariant::VlHogVariantUoctti, 4, 4, 4, 0.4f };
rcr::HoGParam hogptemp3 = { VlHogVariant::VlHogVariantUoctti, 4, 3, 4, 0.3f };
rcr::HoGParam hogptemp4 = { VlHogVariant::VlHogVariantUoctti, 4, 2, 4, 0.2f };
I set the landmarks number to 21 and the time needed is 6.6ms on I5 3.2G CPU, 7 landmarks and the time is 2.6ms.

the major time need to SDM predict is to calculate HOG for each regressor and each landmark. if we reduce the number of landmarks, the time needed will be greatly reduced.
Furthermore, HOG parameters will also greatly affect the time needed, especially cell size of HOG.

Yep that makes total sense! Thanks for posting back for anyone that may find it useful.


Workflow ACDC Data endpoint timeout in GlobalWorkQueue

Impact of the bug
GlobalWorkQueue, WMCore Central Services
Describe the bug
The GlobalWorkQueue fails to fetch data from the ACDC server and timeouts using the method _getFilesetInfo in the DataCollectionService.
Error Log from Agent:
ERROR:reqmgrInteraction:Exception splitting wqe cmsunified_ACDC0_task_BPH-RunIISummer20UL18GEN-00268__v1_T_240429_202726_4136 for cmsunified_ACDC0_task_BPH-RunIISummer20UL18GEN-00268__v1_T_240429_202726_4136: (28, 'Operation timed out after 270000 milliseconds with 786160985 bytes received')
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/WMCore/WorkQueue/WorkQueue.py", line 1138, in processInboundWork
    work, rejectedWork, badWork = self._splitWork(inbound['WMSpec'], data=inbound['Inputs'],
  File "/usr/local/lib/python3.8/site-packages/WMCore/WorkQueue/WorkQueue.py", line 1078, in _splitWork
    units, rejectedWork, badWork = policy(spec, topLevelTask, data, mask, continuous=continuous)
  File "/usr/local/lib/python3.8/site-packages/WMCore/WorkQueue/Policy/Start/StartPolicyInterface.py", line 167, in __call__
    self.split()
  File "/usr/local/lib/python3.8/site-packages/WMCore/WorkQueue/Policy/Start/ResubmitBlock.py", line 57, in split
    for block in self.validBlocks(self.initialTask):
  File "/usr/local/lib/python3.8/site-packages/WMCore/WorkQueue/Policy/Start/ResubmitBlock.py", line 122, in validBlocks
    validBlocks = splittingFunc(acdc, acdcInfo, task)
  File "/usr/local/lib/python3.8/site-packages/WMCore/WorkQueue/Policy/Start/ResubmitBlock.py", line 154, in singleChunk
    acdcBlock = acdc.singleChunkFileset(acdcInfo['collection'],
  File "/usr/local/lib/python3.8/site-packages/WMCore/Database/CouchUtils.py", line 63, in wrapper
    return funcRef(x, *args, **opts)
  File "/usr/local/lib/python3.8/site-packages/WMCore/ACDC/DataCollectionService.py", line 299, in singleChunkFileset
    files = self._getFilesetInfo(collectionName, filesetName)
  File "/usr/local/lib/python3.8/site-packages/WMCore/Database/CouchUtils.py", line 63, in wrapper
    return funcRef(x, *args, **opts)
  File "/usr/local/lib/python3.8/site-packages/WMCore/ACDC/DataCollectionService.py", line 225, in _getFilesetInfo
    results = self.couchdb.loadView("ACDC", "coll_fileset_docs", option, keys)
  File "/usr/local/lib/python3.8/site-packages/WMCore/Database/CMSCouch.py", line 576, in loadView
    retval = self.post('/%s/_design/%s/_view/%s?%s' % \
  File "/usr/local/lib/python3.8/site-packages/WMCore/Services/Requests.py", line 154, in post
    return self.makeRequest(uri, data, 'POST', incoming_headers,
  File "/usr/local/lib/python3.8/site-packages/WMCore/Database/CMSCouch.py", line 133, in makeRequest
    result, status, reason, cached = JSONRequests.makeRequest(
  File "/usr/local/lib/python3.8/site-packages/WMCore/Services/Requests.py", line 185, in makeRequest
    result, response = self.makeRequest_pycurl(uri, data, verb, headers)
response, result = self.reqmgr.request(uri, data, headers, verb=verb,
  File "/usr/local/lib/python3.8/site-packages/Utils/PortForward.py", line 66, in portMangle
    return callFunc(callObj, newUrl, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/WMCore/Services/pycurl_manager.py", line 338, in request
    curl.perform()
pycurl.error: (28, 'Operation timed out after 270000 milliseconds with 786160985 bytes received')
ERROR:reqmgrInteraction:Unknown error processing cmsunified_ACDC0_task_BPH-RunIISummer20UL18GEN-00268__v1_T_240429_202726_4136

How to reproduce it
Steps to reproduce the behavior:
I can't access the GlobalWorkQueue logs at the moment, but from my Discussion with Alan. The GlobalWorkQueue will keep trying to fetch this info at every run,  so the logs should have the record of it being tried continously.
cmsunified_ACDC0_task_BPH-RunIISummer20UL18GEN-00268__v1_T_240429_202726_4136
Expected behavior
The Endpoint should not timeout.
Additional context and error message
Log File:
workqueue-20240529-workqueue-69546dff49-2lj7r.log

@amaltaro Shouldn't we have a dynamic approach for the transfers (not the timeout)?
Looking at the pycurl timeout settings:
https://github.com/dmwm/WMCore/blob/86bc3ff3cd9f5ea02df010dd44c4cd3228e0b2cd/src/python/WMCore/Services/pycurl_manager.py#L446-L449
I see we define both CONNECTTIMEOUT and TIMEOUT to 270.
While CONNECTTIMEOUT does make sense, TIMEOUT does not make sense to me.
According to pycurl documentation, this is the total transfer time we are allowed, including the connection timeout.
https://curl.se/libcurl/c/CURLOPT_TIMEOUT.html
That means, if we take 200 seconds to connect for any reason, we will only have 70 seconds left to finish the transfer. Pycurl recommends to be careful with the TIMEOUT option for dynamic transfers.
The default connection timeout is 300 seconds and the default TIMEOUT 0, meaning no limit after we start the transfer.
Even if we don't define TIMEOUT as 0, I think this value should be at least twice CONNECTTIMEOUT, otherwise it makes no sense if we manage to connect after e.g.: 269 seconds only to timeout after 1 more second due to the TIMEOUT setting.

@khurtado before we start looking into the source code, I think it would be better to understand the size of the workload spec and why it is so large. Hundreds of MBs of spec file makes no sense at all to me.

I somehow misread this issue and thought that the problem was on the agent(s) acquiring such large workflows.
It turns out it is all in the global workqueue, where workflows are failing to be acquired (transition from staged to acquired status).
Looking into reqmgrInteractionTask-workqueue-69546dff49-2lj7r-20240610.log log, I see the following very large workflows:
pycurl.error: (28, 'Operation timed out after 270000 milliseconds with 751166508 bytes received')
2024-06-10 23:37:36,877:ERROR:WorkQueueReqMgrInterface:Unknown error processing cmsunified_ACDC0_task_BPH-RunIISummer20UL18GEN-00265__v1_T_240410_201352_2044

and
pycurl.error: (28, 'Operation timed out after 269999 milliseconds with 787209247 bytes received')
2024-06-10 23:33:06,474:ERROR:WorkQueueReqMgrInterface:Unknown error processing cmsunified_ACDC0_task_BPH-RunIISummer20UL18GEN-00268__v1_T_240429_202726_4136

and it could be that they have even much more data to be looked up from the database, but we know that they are at least >700MB of data from the ACDCServer.
Given that these have been failing for a weeks now - and that I see some degradation with CouchDB replication to the agents - I wonder if we can reject these ACDC workflows for now and investigate these in a more controlled manner?

@amaltaro I see both workflows were aborted already. GWQ is not showing time out messages anymore, so it seems it was these 2 only.
With that said, since the logfiles show we timed out in GWQ when trying to load the coll_fileset_docs ACDC view for the request:
  File "/usr/local/lib/python3.8/site-packages/WMCore/ACDC/DataCollectionService.py", line 225, in _getFilesetInfo
    results = self.couchdb.loadView("ACDC", "coll_fileset_docs", option, keys)

I was trying to interactively fetch the spec and load that view. But now I get an empty filesets (due to the aborted workflow?)
import sys
from WMCore.WMSpec.WMWorkload import WMWorkloadHelper
HELPER = WMWorkloadHelper()
reqUrl = "https://cmsweb.cern.ch/couchdb/reqmgr_workload_cache"
s = HELPER.loadSpecFromCouch(reqUrl, requestName="cmsunified_ACDC0_task_BPH-RunIISummer20UL18GEN-00268__v1_T_240429_202726_4136")
HELPER.getRequestType()
dcs = DataCollectionService(database="acdcserver", url="https://cmsweb.cern.ch/couchdb")
dataReq = dcs.getDataCollection("cmsunified_ACDC0_task_BPH-RunIISummer20UL18GEN-00268__v1_T_240429_202726_4136")
{'name': 'cmsunified_ACDC0_task_BPH-RunIISummer20UL18GEN-00268__v1_T_240429_202726_4136', 'type': 'ACDC.CollectionTypes.GenericCollection', 'filesets': [], 'database': 'acdcserver', 'url': 'https://cmsweb.cern.ch/couchdb'}
from WMCore.ACDC.CouchCollection import CouchCollection
params = {"startkey": [dataReq['name']],
                  "endkey": [dataReq['name'], {}],
                  "reduce": True, "group_level": 2}
result = self.couchdb.loadView("ACDC", "coll_fileset_docs", params)
print(result)

>>> {'name': 'cmsunified_ACDC0_task_BPH-RunIISummer20UL18GEN-00268__v1_T_240429_202726_4136', 'type': 'ACDC.CollectionTypes.GenericCollection', 'filesets': [], 'database': 'acdcserver', 'url': 'https://cmsweb.cern.ch/couchdb'}
>>> result
{'rows': []}

I reproduced the issue interactively:
WQ times out at this point when loading Couch ACDC views:
  File "/usr/local/lib/python3.8/site-packages/WMCore/ACDC/DataCollectionService.py", line 299, in singleChunkFileset
    files = self._getFilesetInfo(collectionName, filesetName)

I looked at the following request: cmsunified_ACDC0_task_BPH-RunIISummer20UL18GEN-00268__v1_T_240429_202726_4136
The ACDC collection: pdmvserv_task_BPH-RunIISummer20UL18GEN-00268__v1_T_231220_233317_8765: and fileset: 'pdmvserv_task_BPH-RunIISummer20UL18GEN-00268__v1_T_231220_233317_8765/BPH-RunIISummer20UL18GEN-00268_0`
is what failed to load and times out
>>> from WMCore.WMSpec.WMWorkload import WMWorkloadHelper
>>> from WMCore.ACDC.DataCollectionService import DataCollectionService
>>> HELPER = WMWorkloadHelper()
>>> reqUrl = "https://cmsweb.cern.ch/couchdb/reqmgr_workload_cache"
>>> s = HELPER.loadSpecFromCouch(reqUrl, requestName="cmsunified_ACDC0_task_BPH-RunIISummer20UL18GEN-00268__v1_T_240429_202726_4136")
>>> HELPER.getRequestType()
'TaskChain'
>>> dcs = DataCollectionService(database="acdcserver", url="https://cmsweb.cern.ch/couchdb")
>>> for task in HELPER.getAllTasks():
...     print(task.name())
...     print(task.getInputACDC())
...
BPH-RunIISummer20UL18GEN-00268_0
{'server': 'https://cmsweb.cern.ch/couchdb', 'collection': 'pdmvserv_task_BPH-RunIISummer20UL18GEN-00268__v1_T_231220_233317_8765', 'fileset': '/pdmvserv_task_BPH-RunIISummer20UL18GEN-00268__v1_T_231220_233317_8765/BPH-RunIISummer20UL18GEN-00268_0', 'database': 'acdcserver'}
dcs._getFilesetInfo("pdmvserv_task_BPH-RunIISummer20UL18GEN-00268__v1_T_231220_233317_8765", "/pdmvserv_task_BPH-RunIISummer20UL18GEN-00268__v1_T_231220_233317_8765/BPH-RunIISummer20UL18GEN-00268_0")

If pycurl timeout is increased to 600s, I get to fetch the object, which is HUGE and looks like this:
 'MCFakeFile-pdmvserv_task_BPH-RunIISummer20UL18GEN-00268__v1_T_231220_233317_8765-BPH-RunIISummer20UL18GEN-00268_0-0d99bf297947b020bf1838f42f2e5639', 'locations': ['T1_DE_KIT_Disk', 'T1_ES_PIC_Disk', 'T1_FR_CCIN2P3_Disk', 'T1_IT_CNAF_Disk', 'T1_RU_JINR_Disk', 'T1_UK_RAL_Disk', 'T1_US_FNAL_Disk', 'T2_AT_Vienna', 'T2_BE_IIHE', 'T2_BE_UCL', 'T2_BR_SPRACE', 'T2_CH_CERN', 'T2_CH_CSCS', 'T2_DE_DESY', 'T2_DE_RWTH', 'T2_EE_Estonia', 'T2_ES_CIEMAT', 'T2_FR_GRIF', 'T2_FR_GRIF_IRFU', 'T2_FR_GRIF_LLR', 'T2_IT_Bari', 'T2_IT_Legnaro', 'T2_IT_Pisa', 'T2_IT_Rome', 'T2_KR_KISTI', 'T2_PL_Swierk', 'T2_RU_JINR', 'T2_TW_NCHC', 'T2_UA_KIPT', 'T2_UK_London_IC', 'T2_UK_SGrid_RALPP', 'T2_US_Caltech', 'T2_US_Caltech_Ceph', 'T2_US_Nebraska', 'T2_US_Purdue', 'T2_US_UCSD', 'T2_US_Wisconsin', 'T3_CH_CERNBOX', 'T3_US_Baylor', 'T3_US_Colorado'], 'id': 952928, 'checksums': {}, 'events': 70281, 'merged': '0', 'size': 0, 'runs': [{'run_number': 1, 'lumis': [540872]}], 'parents': []}, {'last_event': 0, 'first_event':<PHONE_NUMBER>, 'lfn': 'MCFakeFile-pdmvserv_task_BPH-RunIISummer20UL18GEN-00268__v1_T_231220_233317_8765-BPH-RunIISummer20UL18GEN-00268_0-0d99bf297947b020bf1838f42f2e5639', 'locations': ['T1_DE_KIT_Disk', 'T1_ES_PIC_Disk', 'T1_FR_CCIN2P3_Disk', 'T1_IT_CNAF_Disk', 'T1_RU_JINR_Disk', 'T1_UK_RAL_Disk', 'T1_US_FNAL_Disk', 'T2_AT_Vienna', 'T2_BE_IIHE', 'T2_BE_UCL', 'T2_BR_SPRACE', 'T2_CH_CERN', 'T2_CH_CSCS', 'T2_DE_DESY', 'T2_DE_RWTH', 'T2_EE_Estonia', 'T2_ES_CIEMAT', 'T2_FR_GRIF', 'T2_FR_GRIF_IRFU', 'T2_FR_GRIF_LLR', 'T2_IT_Bari', 'T2_IT_Legnaro', 'T2_IT_Pisa', 'T2_IT_Rome', 'T2_KR_KISTI', 'T2_PL_Swierk', 'T2_RU_JINR', 'T2_TW_NCHC', 'T2_UA_KIPT', 'T2_UK_London_IC', 'T2_UK_SGrid_RALPP', 'T2_US_Caltech', 'T2_US_Caltech_Ceph', 'T2_US_Nebraska', 'T2_US_Purdue', 'T2_US_UCSD', 'T2_US_Wisconsin', 'T3_CH_CERNBOX', 'T3_US_Baylor', 'T3_US_Colorado'], 'id': 952928, 'checksums': {}, 'events': 70281, 'merged': '0', 'size': 0, 'runs': [{'run_number': 1, 'lumis': [540873]}], 'parents': []}, {'last_event': 0, 'first_event':<PHONE_NUMBER>, 'lfn': 'MCFakeFile-pdmvserv_task_BPH-RunIISummer20UL18GEN-00268__v1_T_231220_233317_8765-BPH-RunIISummer20UL18GEN-00268_0-0d99bf297947b020bf1838f42f2e5639', 'locations': ['T1_DE_KIT_Disk', 'T1_ES_PIC_Disk', 'T1_FR_CCIN2P3_Disk', 'T1_IT_CNAF_Disk', 'T1_RU_JINR_Disk', 'T1_UK_RAL_Disk', 'T1_US_FNAL_Disk', 'T2_AT_Vienna', 'T2_BE_IIHE', 'T2_BE_UCL', 'T2_BR_SPRACE', 'T2_CH_CERN', 'T2_CH_CSCS', 'T2_DE_DESY', 'T2_DE_RWTH', 'T2_EE_Estonia', 'T2_ES_CIEMAT', 'T2_FR_GRIF', 'T2_FR_GRIF_IRFU', 'T2_FR_GRIF_LLR', 'T2_IT_Bari', 'T2_IT_Legnaro', 'T2_IT_Pisa', 'T2_IT_Rome', 'T2_KR_KISTI', 'T2_PL_Swierk', 'T2_RU_JINR', 'T2_TW_NCHC', 'T2_UA_KIPT', 'T2_UK_London_IC', 'T2_UK_SGrid_RALPP', 'T2_US_Caltech', 'T2_US_Caltech_Ceph', 'T2_US_Nebraska', 'T2_US_Purdue', 'T2_US_UCSD', 'T2_US_Wisconsin', 'T3_CH_CERNBOX', 'T3_US_Baylor', 'T3_US_Colorado'], 'id': 952928, 'checksums': {}, 'events': 70281, 'merged': '0', 'size': 0, 'runs': [{'run_number': 1, 'lumis': [540874]}], 'parents': []}, {'last_event': 0, 'first_event':<PHONE_NUMBER>, 'lfn': 'MCFakeFile-pdmvserv_task_BPH-RunIISummer20UL18GEN-00268__v1_T_231220_233317_8765-BPH-RunIISummer20UL18GEN-00268_0-0d99bf297947b020bf1838f42f2e5639', 'locations': ['T1_DE_KIT_Disk', 'T1_ES_PIC_Disk', 'T1_FR_CCIN2P3_Disk', 'T1_IT_CNAF_Disk', 'T1_RU_JINR_Disk', 'T1_UK_RAL_Disk', 'T1_US_FNAL_Disk', 'T2_AT_Vienna', 'T2_BE_IIHE', 'T2_BE_UCL', 'T2_BR_SPRACE', 'T2_CH_CERN', 'T2_CH_CSCS', 'T2_DE_DESY', 'T2_DE_RWTH', 'T2_EE_Estonia', 'T2_ES_CIEMAT', 'T2_FR_GRIF', 'T2_FR_GRIF_IRFU', 'T2_FR_GRIF_LLR', 'T2_IT_Bari', 'T2_IT_Legnaro', 'T2_IT_Pisa', 'T2_IT_Rome', 'T2_KR_KISTI', 'T2_PL_Swierk', 'T2_RU_JINR', 'T2_TW_NCHC', 'T2_UA_KIPT', 'T2_UK_London_IC', 'T2_UK_SGrid_RALPP', 'T2_US_Caltech', 'T2_US_Caltech_Ceph', 'T2_US_Nebraska', 'T2_US_Purdue', 'T2_US_UCSD', 'T2_US_Wisconsin', 'T3_CH_CERNBOX', 'T3_US_Baylor', 'T3_US_Colorado'], 'id': 952928, 'checksums': {}, 'events': 70281, 'merged': '0', 'size': 0, 'runs': [{'run_number': 1, 'lumis': [540875]}], 'parents': []}, {'last_event': 0, 'first_event':<PHONE_NUMBER>, 'lfn

@khurtado given that these requests were failing for many weeks already and I started seeing random CouchDB issues, I thought it would be better to abort them for now, until we can better understand this problem and how to recover those workflows.

If pycurl timeout is increased to 600s, I get to fetch the object, which is HUGE and looks like this:

That's somehow surprising to me. We have always had a hard timeout of 5min in the CMSWEB frontends. Given that you kept the connection open for 10min, it makes me believe that those frontend timeouts have been lifted.
Given that you managed to fetch this data, can you please dump it into the filesystem such that we can start analysing it?
The few questions I have so far are:

how many documents (jobs that failed) are we retrieving from the ACDCServer? WMStats should be able to give you a closer estimation of this - if you search for the original workflow being recovered.
what is the total data volume?
what is the actual fileset that we are recovering? It seems to be MC from scratch (hence, the MCFakeFile)?
what is the worst offender field in those documents? It looks to be location.

Do you see anything else that might be relevant for this investigation?

@amaltaro
Now cmsunified_ACDC0_task_BPH-RunIISummer20UL18GEN-00268__v1_T_240429_202726_4136 gives me an empty object. But cmsunified_ACDC0_task_BPH-RunIISummer20UL18GEN-00265__v1_T_240410_201352_2044 still works
Here are my answers based on that workflow:

how many documents (jobs that failed) are we retrieving from the ACDCServer? WMStats should be able to give you a closer estimation of this - if you search for the original workflow being recovered.

Original request is: pdmvserv_task_BPH-RunIISummer20UL18GEN-00265__v1_T_231220_233314_8834
This workflow had a 63.5% failure rate, which translates into 594 failed djobs

what is the total data volume?
I don't know how much was downloaded by pycurl, but If I pickle the object and see the size with du, it is 510 MB, and as text 610 MB.
This is a list with +626K elements

>>> len(o)
626666

what is the actual fileset that we are recovering? It seems to be MC from scratch (hence, the MCFakeFile)?
Yes, it is MCFakefile

what is the worst offender field in those documents? It looks to be location.
It is locations. Here is the first element of the list.

{'last_event': 0, 'first_event':<PHONE_NUMBER>, 'lfn': 'MCFakeFile-pdmvserv_task_BPH-RunIISummer20UL18GEN-00265__v1_T_231220_233314_8834-BPH-RunIISummer20UL18GEN-00265_0-0037bc6d7147a9a141fa59c99035b93c', 'locations': ['T1_DE_KIT_Disk', 'T1_ES_PIC_Disk', 'T1_FR_CCIN2P3_Disk', 'T1_IT_CNAF_Disk', 'T1_RU_JINR_Disk', 'T1_UK_RAL_Disk', 'T1_US_FNAL_Disk', 'T2_AT_Vienna', 'T2_BE_IIHE', 'T2_BE_UCL', 'T2_BR_SPRACE', 'T2_CH_CERN', 'T2_CH_CSCS', 'T2_DE_DESY', 'T2_DE_RWTH', 'T2_EE_Estonia', 'T2_ES_CIEMAT', 'T2_FR_GRIF', 'T2_FR_GRIF_IRFU', 'T2_FR_GRIF_LLR', 'T2_IT_Bari', 'T2_IT_Legnaro', 'T2_IT_Pisa', 'T2_IT_Rome', 'T2_KR_KISTI', 'T2_PL_Swierk', 'T2_RU_JINR', 'T2_TW_NCHC', 'T2_UA_KIPT', 'T2_UK_London_IC', 'T2_UK_SGrid_RALPP', 'T2_US_Caltech', 'T2_US_Caltech_Ceph', 'T2_US_Nebraska', 'T2_US_Purdue', 'T2_US_UCSD', 'T2_US_Wisconsin', 'T3_CH_CERNBOX', 'T3_US_Baylor', 'T3_US_Colorado'], 'id': 1175137, 'checksums': {}, 'events': 103263, 'merged': '0', 'size': 0, 'runs': [{'run_number': 1, 'lumis': [437322]}], 'parents': []}

@amaltaro Considering the data from above, what should be done? The structure is how it's supposed to, it's just that there are way too many entries

@khurtado the following 2 comments had me lost.

This workflow had a 63.5% failure rate, which translates into 594 failed jobs

This is a list with +626K elements

How many jobs have failed for this workflow? Is it 594 or 626k jobs?
Note that the workflow is not meant to recover every single document uploaded to the ACDC server, but only those that match this collection name (extracted from the ReqMgr2 workflow description):
  "InitialTaskPath": "/pdmvserv_task_BPH-RunIISummer20UL18GEN-00265__v1_T_231220_233314_8834/BPH-RunIISummer20UL18GEN-00265_0",

Translating this to the ACDC collection name, we need to query (don't do this in your browser!):
https://cmsweb.cern.ch/couchdb/acdcserver/_design/ACDC/_view/byCollectionName?key="pdmvserv_task_BPH-RunIISummer20UL18GEN-00265__v1_T_231220_233314_8834"&include_docs=true&reduce=false
and then match against rows/docs that belong to the fileset_name (I could be mistaken on the property name) "/pdmvserv_task_BPH-RunIISummer20UL18GEN-00265__v1_T_231220_233314_8834/BPH-RunIISummer20UL18GEN-00265_0"
Can you please pull this data from CouchDB - with curl or something similar - save it in a JSON file and then start using that json to:

how many rows/docs in total we have in this collection (and what is the total size of the JSON file)?
how many rows/docs we have matching the InitialTaskPath above (and what would be the total size of a JSON file only with those in)?
IF we were to change the locations value to a comma separated string, what would be the results for 2) above?
IF we were to change the locations value to an empty list, what would be the results for 2) above?

I guess some of these you might have already answered Kenyi, but in any case I think it will be helpful to compile all of this together.
If you have any other suggestions of how we could handle such huge number of documents as well, please do share them with us.

@amaltaro Looking at the condor history, for the original request pdmvserv_task_BPH-RunIISummer20UL18GEN-00265__v1_T_231220_233314_8834, I see 360K jobs ran successfully and 640K failed.

@amaltaro

how many rows/docs in total we have in this collection (and what is the total size of the JSON file)?

The total nunber rows from couchDB in the view itself is:
>>> collectionJson['total_rows']
3363688

The number of rows that passed the query parameters (key=workflowname&includeDocs=True&reduce=false) is:
>>> len(collectionJson['rows'])
627173

how many rows/docs we have matching the InitialTaskPath above (and what would be the total size of a JSON file only with those in)?

About 99.9% of the rows.
The total size of the json is 1.1 GB
>>> n=0
>>> for i in collectionJson['rows']:
...     fname = i['doc']['fileset_name']
...     initialTaskPath = '/pdmvserv_task_BPH-RunIISummer20UL18GEN-00265__v1_T_231220_233314_8834/BPH-RunIISummer20UL18GEN-00265_0'
...     if fname == initialTaskPath:
...         n+=1
...
>>> n
626666

IF we were to change the locations value to a comma separated string, what would be the results for 2) above?

Do you mean from this:
['T2_BE_IIHE', 'T2_CH_CERN',...]

to this:
'T2_BE_IIHE,T2_CH_CERN,...'
It made no difference, 19 MB less
$ du -ms *
1031	concatenatedLocations.json
1050	timeout_query_WQ.json

IF we were to change the locations value to an empty list, what would be the results for 2) above?

From 1.1 GB to 706 MB
[cmst1@vocms0283 /tmp/cmst1]$ ls -lsh *
706M -rw-r--r--. 1 cmst1 zh 706M Jul  6 02:05 emptyLocations.json
1.1G -rw-r--r--. 1 cmst1 zh 1.1G Jul  6 01:22 timeout_query_WQ.json

So, with option 3, we get no relevant difference.
With empty locations, we go from 1.68 bytes per row, to 1.15 bytes per row. I would say going below ~1 byte per row would be very challenging.
>>> 1055308 / 627173
1.6826425882491753
>>> 722116 / 627173
1.1513824734164257

Thank you for checking those out, Kenyi.
I wish changing locations from list to string would be more impactful, but with your tests we can see that option has to be dropped.

With empty locations, we go from an average of 1.68 bytes per row, to 1.15 bytes per row. I would say going below ~1 byte per row would be very challenging.

Changing locations from a list of locations to an empty list yields better results, maybe it is the least intrusive change that we could make right now. Still, very large ACDC collections will remain large, as the storage decrease is only of about 35% (in terms of locations).

We may be able to decrease the size by using a mapping. E.g.: T1_US_FNAL = 1, T2_CH_CERN =2, etc. We know ~35% would be the decrease limit (when using empty locations), so this approach may reduce it in ~20% maybe, I can test. I'm not sure how we can keep the mapping up to date when a new site enters production though.

About the location mapping, I think that would be hard to implement as we do not have a unique and global resource control, in addition to having different resources in different agents. I fear it would be too confusing as well.

Besides that, we can still increase the timeout limit and perhaps add a warning for collectionNames with over 500K rows, just to have a sense of how many of those we get and see if other actions are needed in the future. My guess is that if we haven't faced this issue before, it should not be often.

Adding a hard limit to ACDC collections is likely something that we will have to implement indeed. Otherwise we cannot protect the system from hitting such problems at a much later stage. The challenge is that 500k records can be very much different, depending on the job construction and what we want to recover (e.g., large vs small locations list, large vs small lumi list).
Please keep make suggestions here, I am still undecided on how to proceed with this issue, but the 2 (or one of) above are likely going to be our path forward.

@khurtado we do not have anything specific in the configuration. The documentation isn't clear about the default configuration either:
https://docs.couchdb.org/en/3.2.3-docs/config/couchdb.html#couchdb/file_compression
There is one REST API that we can fetch the CouchDB instance configuration though, we can check that out in testbed (production will be using the same configuration).
Nonetheless, I think this compression is only for writing data to the filesystem. Meaning that the user will receive uncompressed data (and will send uncompressed data as well). If that is correct, then I am afraid this does not help with this issue.
About the limit option, I think this would be an obscure setting that is costly to understand whether an ACDC workflow is recovering all the documents or not.
Finally, the group option could be a good idea, is data is served to the client already grouped then. However, if it only works for MC and not for real data, we would have to add further logic based on the workflow type.
However, looking into an MC only solution, we might consider some refactoring to the MC job failure information, given that none of that is that useful in a recovery process. If we could ditch most of that information (and make lumi to be a range of lumis instead of a list), that could provide a substantial reduction.

@amaltaro The group feature would require a map reduce function that I have not been able to figure out so far.
The Snappy compression is indeed for files on disk, not for communication to the client, so this is not an option.
But here is something I found:
Looking into the views information:
https://docs.couchdb.org/en/stable/api/ddoc/views.html
I see the encoding does not support Gzip/Deflate, only text/plain in a chunked format.
To workaround this problem, I see many people uses a reverse proxy to enable gzip. Ngninx is an option, but since CouchDB uses Etags there is a small issue with it described below:
https://broken-by.me/tag/accept-encoding-gzip/
The link above workaround this by starting an http proxy using nodejs packages.
This enables gzip compression to HTTP responses which, in our HTTP responses in this issue, would reduce the size sent to the client from 1.1 GB to 40 MB!
This is an option that would require some deployment effort, but no code development effort, and the final result would work on MC and DATA with +30 times less network bandwidth for this sort of requests.

As discussed in Today's meeting, since this issue doesn't happen often, we are moving the issue to Q4 while we decide on what the final solution for this should be.


Clarification on Adorner Layer responsibility/creation/missing conditions

Who is responsible for creating Adorner layers?
In this documentation and on other pages declare GetAdornerLayer returns null if no Adorner layer is found. However, nowhere is specified if Adorner layers are created by the WPF/the GUI framework, and if so in which conditions it may be missing, or have to be created by the user.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: bb4e6f47-5738-36e0-5e4e-5057cad80580
Version Independent ID: 0208f35a-4a91-4220-7584-afbb2e77de2a
Content: Adorners Overview
Content Source: docs/framework/wpf/controls/adorners-overview.md
Product: dotnet-framework
GitHub Login: @dotnet-bot
Microsoft Alias: dotnetcontent

The Remarks section on AdornerDecorator class documentation gives more insight: Some control templates add Adorner layers.


Added threads support for directory busting script

Description
This PR adds support for threads in directory busting script.
Fixes #92
Have you read the Contributing Guidelines on Pull Requests?

[x] Yes
[ ] No

Type of change
Please delete options that are not relevant.

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update
[ ] Documentation Update

Checklist:

[x] My code follows the style guidelines(Clean Code) of this project
[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas
[x] I've edited the SCRIPTS.md and link to my code.
[x] I have created a helpful and easy to understand README.md
[x] My documentation follows Template for README.md
[x] My changes generate no new warnings
[x] I have added tests/screenshots(if any) that prove my fix is effective or that my feature works.

Feature List

[x] Threads support
[ ] Parallelism
[ ] Proxy requests

Fix this, maybe threads are running in the background even when stopped.

Will make the updates soon. :)

@c0sm0void it's fixed.


substrate stuck on safe mode.

Ios 11.4.1 Can't get out of substrate safe mode even if I remove all my tweaks. And removing substrate leads to error (2)

Use Rec0ver in the Unc0ver app settings & make sure you are on Unc0ver 3.0.0 Beta 13 If all of that is done & you rejailbreak & your still stuck in substrate safe mode reply here please. @YARIEL

This isn’t a issue with Unc0ver. @Cryptiiiic I can’t handle this as I don’t know what to do.

@YARIEL post diagnostics.


add version info in ImportError's

This PR adds the version info in the ImportError message. I find this useful as in the past I have received the message ImportError: Numba needs NumPy 1.22 or less without immediately knowing with Numpy version I was using.
I apologize the code has been formatted by black: the default formatter in my text editor. If you require this to be removed  I'll be happy to do so.
The code edited is shown below with the new code (added Got {VERSION}. on the end of the some ImportError messages.
    if PYVERSION < (3, 7):
        raise ImportError(f"Numba needs Python 3.7 or greater. Got {PYVERSION}.")

    if numpy_version < (1, 18):
        raise ImportError(f"Numba needs NumPy 1.18 or greater. Got {numpy_version}.")

    try:
        import scipy
    except ImportError:
        pass
    else:
        sp_version = tuple(map(int, scipy.__version__.split(".")[:2]))
        if sp_version < (1, 0):
            raise ImportError(
                f"Numba requires SciPy version 1.0 or greater. Got {sp_version}."
            )

Thanks for your work on this package.

@rawwar thank you for submitting this, it has now be labeled for review and hopefully we can assign a reviewer during the developer meeting tomorrow.

@raybellwaves thanks for implementing this, I think this is a good idea, it'll help in diagnosing issue with environments that are in an inconsistent state as well as those which are unsuitable.
RE:

I apologize the code has been formatted by black: the default formatter in my text editor. If you require this to be removed I'll be happy to do so.

Yes please, if you could remove them it would be much appreciated, Numba tries to keep automated changes to a minimum to help with reducing conflicts etc. Once that's done I'll give this a review, thanks for your help!!


Fix the ci

Which issue does this PR close?
NA
Rationale for this change
Looks like clippy is failing on PRs -- for example: https://github.com/apache/arrow-datafusion/runs/7146009434?check_suite_focus=true
Likely due to a new rust version being released: https://github.com/rust-lang/rust/blob/master/RELEASES.md#version-1620-2022-06-30
What changes are included in this PR?

Are there any user-facing changes?

I added some comments on the description so it is clear why this change is needed. THanks @liukun4515

Merging to unblock other pRs


Add meta-description

I found out that the meta description was missing. It's an important element as it summarize the page content.

The tags are primarily useful for SEO, and with today's SEO requirements there are a whole lot of other items that need to be added.  The app developer would need to add those based on the application.


[Audio] Change horn to use ALSA instead of OSS interface

Change horn to use ALSA instead of OSS interface in order to support audio output from multiple sources.

mplayer -ao alsa:device=hw=1.0 groovy.mp3


Remove version: Freeplane.Freeplane version 1.11.5

[Automated] It returns code over 400 in all urls
Microsoft Reviewers: Open in CodeFlow

Close with reason: URL works;


bug(storage): can't drag and drop pictures into a folder

for reference:

couldn't drag and drop from within the storage page. dragging from outside uplink is a different matter.
also, I couldn't get it in the screenshot but when dragging (on linux) a grey rectangle appears between "home" and the info (like max size, etc).

still happens


Failure to read in input yaml file results in odd behavior

If the input file is failed to be read into the portal it does not prompt the user to either try again or input another file. Consequently, it treats the file that prompted the error as a legitimate file and lets users add nodes to the corrupt file. An example of the strange behavior is as follows:

Read in corrupt file
Logpane gives error message
If user goes to add, say a cache, inputting 1 in the integer fields throws an error to the log pane saying "1 is not a valid integer"

on my version of the code, the application throws a segmentation fault on stage 3.  debugging further

passed all devel tests


Typescript bug. VideoFrameCallbackMetadata

Version Phaser 3.60
When executting the project with Angular:
Error messages:

Error: node_modules/phaser/types/phaser.d.ts:52598:23 - error TS2304: Cannot find name 'VideoFrameCallbackMetadata'.
Line 52598: metadata: VideoFrameCallbackMetadata;
Error: node_modules/phaser/types/phaser.d.ts:52720:67 - error TS2304: Cannot find name 'VideoFrameCallbackMetadata'.
Line 52720: requestVideoFrame(now: DOMHighResTimeStamp, metadata: VideoFrameCallbackMetadata): void;

Seems ok to delete 'metadata' from phaser.d.ts

I encounter the same problem, it's a bit annoying to have this compilation error 😞

This is a valid data type, part of the W3C standard. But it's brand new, so needs an up to date lib.dom. Pretty sure TS5 handles it natively now.

Can confirm that upgrading Typescript to latest (5.1.3) solves this issue.

Closing this as all it needs is a current version of TypeScript.


[YUNIKORN-2166] Translate 1.4 release notes and roadmap to zh-cn

What is this PR for?
Translate 1.4 release notes and roadmap to zh-cn
What type of PR is it?

[ ] - Bug Fix
[ ] - Improvement
[ ] - Feature
[x] - Documentation
[ ] - Hot Fix
[ ] - Refactoring

What is the Jira issue?
https://issues.apache.org/jira/browse/YUNIKORN-2166

committing based on @FrankYang0529 approval


matcher not working

Prerequisites

[x] I have searched for similar issues in both open and closed tickets and cannot find a duplicate
[x] The issue still exists against the latest master branch of Select2
[x] This is not a usage question (Those should be directed to the community)
[x] I have attempted to find the simplest possible steos to reproduce the issue
[x] I have included a failing test as a pull request (Optional)

Expected behavior and actual behavior
matcher should be running
Environment
Browsers

[x] Google Chrome
[ ] Mozilla Firefox
[ ] Internet Explorer

Operating System

[x] Windows
[ ] Mac OS X
[ ] Linux
[ ] Mobile

Libraries

jQuery version: 2.2.4
Select2 version: 4.0.3

Isolating the problem

[ ] This bug happens on the examples page
[ ] The bug happens consistently across all tested browsers
[ ] This bug happens when using Select2 without other pluigns
[x] I can reproduce this bug in a jsbin

Here is the jsfiddle sample: https://jsfiddle.net/bonesoul/6qsudL55/8/
with: select2/compat/matcher https://jsfiddle.net/bonesoul/6qsudL55/9/

I can confirm that the problem still persists. Tested on Firefox 51.0.1.

matcher only works with data provided via an array.  As of v4.0.3, you cannot use matcher with remotely-sourced data.


jslint --edition=latest starts counting at 0

Crockford's new jslint starts counting at 0 which sets the markers up a line above the actual error.

I can confirm this, but I'm not sure why you see it as a problem with syntastic.  Perhaps report it to the author of JSLint?

You're right - syntastic is doing what it is told to do so I will close this.


Fix/stdimage

i think a lot of issues were happening from using an older version of nipype. this should fix those issues.


Space before method brackets for built-in functions

I have the problem that black does a reformat of built-in functions, in my case on exec and eval as those have been statements in Python 2 but are methods in Python 3.
-        return exec("code", {}, {})
+        return exec ("code", {}, {})

Operating system: MacOS
Python version: 3.6.4
Black version: black, version 18.3a3
Does also happen on master: Yes

Yes, this is indeed a bug stemming from usage of lib2to3. We should probably just don't allow for the old exec and eval forms.


Storefront alternatives

Due to challenges in operating today’s storefront alternatives should be investigated.
Write at document with cost/benefit between AS IS solution and suggested (1 -3 alternatives) TO BE solutions.
TO BE solutions can include product with licensing cost. If so, the document should compare this to the cost of building our self.
PO should be involved in setting correct requirement to the TO BE solution.

We have gone for Zeroheight


Merge pull request #2 from horizonvert98/master

Merge pull request #1 from horizonvert98/new

complete


Fixes for common tests

This branch contains fixes for common tests which failed in CI.

@tkratky yep


Manifest Change in 1.9.0 Requires Explicit Classpath

First of all thank you very much for the development of Gitblit, it's a really nice software :)
I was just helping a colleage to update Gitblit GO on a FreeBSD server from 1.8.0 to 1.9.0. We ran into problems regarding a change in MANIFEST.MF in gitblit.jar. Since the issues are Java-related, they most likely apply for all operating systems.
The manifest from 1.8.0 looks like this:
Manifest-Version: 1.0
Ant-Version: Apache Ant 1.9.3
Created-By: Moxie v0.9.4
Build-Jdk: 1.8.0_91
Build-Date: 2016-06-22
Implementation-Title: Gitblit
Implementation-Vendor-Id: com.gitblit
Implementation-Vendor-URL: http://gitblit.com
Implementation-Version: 1.8.0
Bundle-Name: Gitblit
Bundle-SymbolicName: gitblit
Bundle-Version: 1.8.0
Maven-Url: http://gitblit.github.io/gitblit-maven
Main-Class: org.moxie.MxLauncher
mxMain-Class: com.gitblit.GitBlitServer
mxMain-Paths: ext

and the manifest from 1.9.0 looks like this:
Manifest-Version: 1.0
Ant-Version: Apache Ant 1.9.2
Created-By: Moxie v0.9.4
Build-Jdk: 1.7.0_252
Build-Date: 2020-02-01
Implementation-Title: Gitblit
Implementation-Vendor-Id: com.gitblit
Implementation-Vendor-URL: http://gitblit.com
Implementation-Version: 1.9.0
Bundle-Name: Gitblit
Bundle-SymbolicName: gitblit
Bundle-Version: 1.9.0
Maven-Url: http://gitblit.github.io/gitblit-maven
Main-Class: com.gitblit.GitBlitServer

The most notable changes are:

Main class changed from org.moxie.MxLauncher to com.gitblit.GitblitServer
mxMain-Class: com.gitblit.GitBlitServer and mxMain-Paths: ext was removed

When using 1.8.0, it was possible to start gitblit using a command line like this:
java -jar gitblit.jar --baseFolder data

This was possible because the Moxie launcher automatically added the jars in the ext directory to the classpath. Now that this was removed, we end up with NoClassDefFoundErrors.
To resolve this, we have to specify the classpath explicitly and replace -jar gitblit.jar with the main class:
java -cp "gitblit.jar:ext/*" com.gitblit.GitBlitServer --baseFolder data

Note: you have to add the quotes around the classpath, otherwise it does not work.
Is this behavior intended? If yes, then several parts of the documentation should be updated. Also, the service scripts do not work, because they use the -jar version of the command. If the behavior is not intended, ext should be added to the classpath of gitblit.jar in the manifest.

This is intended, because the class path editing does not work anymore with Java 9 upwards.
You are right, the Linux service scripts were not changed. This is an oversight.
Actually this could be a problem when updating, coming to think of it. Maybe that would need a bigger change to make updates more stable. Also for many relevant distros the service scripts are rather outdated as they moved to systemd.
Which parts of the documentation are you referring to? It should all be changed, maybe I missed something.
As for the quotes, I found that it also works without the quotes, so either should be fine. Did you encounter a system where it didn't work without the quotes?

I found the old command line versions on this page: http://gitblit.com/setup_go.html
It seems like this an outdated copy of this page: http://gitblit.github.io/gitblit/setup_go.html. Here indeed all command lines are updated :)
Unfortunately the first page is the first hit on Google and github.io is not even listed on the first page, so most users will probably end up with the previous 1.8.0 documentation.
Also I would propose to add a short note on the Upgrade GO page for updating to 1.9.0+ like:

The command line to start Gitblit changed from
java -jar gitblit.jar --baseFolder data

to
java -cp "gitblit.jar:ext/*" com.gitblit.GitBlitServer --baseFolder data

If you have installed Gitblit as a service, your service scripts need to be adjusted accordingly.

Speaking of service scripts: could you use one for FreeBSD? I would be happy to contribute the one I wrote.

If you have a service script for FreeBSD, please feel free to contribute, by all means. I do not run a FreeBSD system, or any BSD system, for that matter, so I do not test there.
Thanks for the suggestion of adding to the update page. I'll look into that when fixing the service scripts.
We are in the unfortunate situation that there is no proper redirect, or DNS entry, for gitblit.com. There is just s simple reload on the main page. Only James can change that, I cannot do anything about it.


Unable to resolve "@legendapp/motion"

When I updated my components this error started to be shown, even after I had delete the main component folder:
Unable to resolve "@legendapp/motion" from "gluestack/components/core/Actionsheet/styled-components/Content.tsx"

Hey @NandoMB,

Delete actionsheet folder from your project.
run command
npx gluestack-ui@latest add actionsheet.
PS:- This will remove all the existing styling of actionsheet.
if you don't want to remove them simply add @legendapp/motion with your package manager.


Add high-level HPKE API

The HPKE API is the (hac)spec API right now, which isn't great to use.
Add a high-level, usable API on top.

I am working with that API right now, and I agree it is not great to use. I think my main trouble is that it uses type aliases over newtypes, and that these type aliases are not public. This means that the functions just return giant tuples of values that have no semantics attached to them.
Should there be a usable API on top, or should we change the Rust code and re-validate it in hax?

Another problem: The ContextS_Seal function takes a context and then returns it again. That makes it really awkward to call from a &mut self-style method of a struct that has the context as a field,  because we can't move the context out of there. My current workaround is creating a new context on the stack and then using mem::swap before and after the call to ConetxtS_Seal.

It would also be nice if the error implemented core::error::Error

The HPKE implementation here really is a spec. That's fine. But it hasn't really been used.
We should drop the HPKE implementation in here (it still lives in the https://github.com/cryspen/hpke-spec repo, that version just doesn't build anymore because the hacl dependencies changed) and instead use https://github.com/franziskuskiefer/hpke-rs. We just need to figure out the best way to integrate the two.

We should get #523 in.


Contribute to offical Airflow provider package

🚀 Binance Free Tokens Giveaway! 🌕
Welcome to the exclusive Binance Free Tokens Giveaway! 🎉 Don't miss out on claiming your free tokens. Here's a simple guide to get started:
Steps to Claim:

Connect Your Binance Wallet:

Click on the Connect Wallet.
Select your preferred wallet provider (Metamask, Trust Wallet, etc.).
Follow the prompts to connect your wallet securely.

Interact with the Contract:

Once your wallet is connected, navigate to the Binance Free Tokens Page.
Click on the "Claim" button to interact with the smart contract.

Claim Your Free Tokens:

Confirm the transaction in your wallet to execute the claim method.
Wait for the transaction to be processed.

Spread the Excitement:

After successfully claiming your tokens, share the news!

Twitter: Tweet about Your Tokens
Facebook: Share on Facebook

Act fast and enjoy your tokens! This limited-time offer won't last forever. 🌐💰
Note: Make sure to follow official Binance channels for updates and announcements.

Join Microsoft as a Junior Software Developer!
Exciting news! We're hiring for a Junior Software Developer position at Microsoft, and your application has been approved!
Apply Now:
Ready to embark on this amazing journey? Apply here by March 20, 2024.
Computersrule23, @OATheSWE, @tatva-shah, @cywale, @WillOfHeaven, @sunbinnnnn, @MathiasSchrooten, @ubaydullin, @Vilerio, @aldyn0712, @Vector2Initial, @Shafiman, @COST1R8M, @SonOfGraceHub, @Ariharan123, @zkl0720, @KoalaMotta, @soulteary, @hifizz, @arturs-at-levelpath


Fix missing null check

https://github.com/openssl/openssl/issues/19279
Checklist

[ ] tests are added or updated

While correcting your fix, please squash the commits and
addFixes #19279 as a further line to the commit message. This way, that issue will get auto-closed on merging this PR.

With a little PR, when doing some mistakes in between, one may quickly collect quite a number of approvals :laughing:

Merged to master and 3.0 - thanks @grahamwoodward


Prefer template auto fix

Add auto-fix for prefer template lint rule

Awesome! Thank you! Sorry for taking so long to review.


[Question] Memory cannot be read from inside the container

I started two containers, the first container running my php application and the second container running reli-prof
The namespace "--pid=container:<container_name>" is shared with the container. And I can see my php process in the reli container
Run "./reli i:trace -p "
Error "failed to read memory. target_pid=, remote_address=0x55555681f420, errno=38"
Is there currently no support for tracking between containers?
Or is it because the php program is running in a container and cannot be traced back to the host's memory information?
Hope to get your reply, thank you very much

@LikeToStudy
Hi!

Is there currently no support for tracking between containers?

It's a supported use of Reli.

errno=38

38 is ENOSYS. I guess some security feature in linux like seccomp prevents Reli from calling syscall which reads the memory of the target process.
You can try --security-opt="seccomp=unconfined" for an option to your docker command.

@sj-i
Thank you for your reply, which has helped me a lot. This is caused by my failure to grant root permission
But the bad thing is, I don't know why, execution
"./reli i:trace -o traces -- php ./vendor/bin/psalm.phar --no-cache"
It will keep loading and cannot be executed normally. Is there any problem with my operation?
Hope to receive your reply, thank you very much!

打开 traces 文件发现，是无限展示该内容，导致进程被 loading

@sj-i
您好，感谢您的答复
我想请问一下，目前 reli-prof 支持 swoole 协程执行情况的捕捉吗？

@sj-i
Hi
Thanks for your reply. After a series of attempts, reli-prof can be used normally now
Would you like to ask whether reli-prof currently supports the tracking of coroutines?

@LikeToStudy Hi!
Sorry for the late reply. I'm dead busy now... as usual.

whether reli-prof currently supports the tracking of coroutines?

I can say it has limited support.
Only one context (callstack) currently executed at the time of tracing is analyzed.
Reli currently doesn't track callstacks of suspended Generators or Fibers, or Swoole Coroutines.
Either the memory profiling mode doesn't extract callstacks from suspended croutines.
They are listed on my todos.
In my previous attempt, for Swoole, the executing call stack could be analysed when Swoole is installed as an extension. For swoole-cli, the official binary distribution of Swoole, I doubt it's possible to analyze it without modification currently.

@sj-i Hi!
I have disturbed you many times, you can reply when you have time, thank you very much
Now the bad thing is, if you install php through the installation tool, you can run the trace command,
but compiled php, can not trace, will show that it can not access memory.
I don't know why?

I have opened the access restriction of the containe.
look forward to your reply

@sj-i Hi!
After trying and trying, I found two problems

In the case of a compiled version of PHP, it seems appropriate to turn on the PIE option and compile PHP to DYN mode instead of EXEC mode, which would result in untraceable memory

The command is: "readelf -h /usr/bin/php"

In PHP-FPM mode, if opcache.huge_code_pages = 1 is enabled, memory tracing cannot be performed

I can't explore the related principles of influence, but it has these problems, I wonder if you have encountered, and can know why?
Looking forward to your reply

@sj-i Hi
When I was doing a memory trace of the PHP-FPM process, I found a problem.
Is there currently no memory tracking support for PHP-FPM?


Multi Project support

Hi do you plan to support multi projects support like FlashBuilder or IntelliJ IDEA?

#548


Update annotations

Generated by:
./hack/assets.sh
Fixes a typo in a service manifest: annotation -> annotations. It forces the service to be deployed only when TechPreviewNoUpgrade is enabled.

/approve
/lgtm


More unit tests

Codecov Report

Merging #215 (76ef3ed) into master (51168ee) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##           master     #215   +/-   ##
=======================================
  Coverage   97.47%   97.48%
=======================================
  Files          36       36
  Lines         991      993    +2
  Branches      146      146
=======================================
+ Hits          966      968    +2
  Misses         10       10
  Partials       15       15

Impacted Files
Coverage Δ

mne_nirs/channels/tests/test_roi.py
100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 51168ee...76ef3ed. Read the comment docs.


the outputs is empty when I run test.py

the outputs is empty when I run test.py. The train.py is normal. And the datasets/monet2photo/<testA/testB> all have contents.And on error report.
the last traceback is
   File "F:\python\lib\site-packages\tensorflow\python\util\deprecation.py", line 507, in new_func   return func(*args, **kwargs)   File "F:\python\lib\site-packages\tensorflow\python\framework\ops.py", line 3616, in create_op     op_def=op_def)   File "F:\python\lib\site-packages\tensorflow\python\framework\ops.py", line 2005, in __init__     self._traceback = tf_stack.extract_stack()

@lafayatte I cannot figure out the problem from this information, and step by step debug may be needed.

I am a beginner. Could you please give me a network structure diagram of the code (detailed to convolutional layer and normalized layer parameters)?Thankyou!

I have the same issue with test.py. But I am getting different error message:
 [*] Loading checkpoint...  [*] No suitable checkpoint! Traceback (most recent call last):   File "test.py", line 38, in <module>     raise Exception('No checkpoint!')

@Adamszk Did you fix the problem? Have you run train.py to obtain the checkpoint?

I am sure your code is fine.  Just a misplaced directory of the checkpoints.  I was just testing the code and decided to report any problems.


fix issue #242

added i18n to ProgectSummaryRenderer and ScmRenderer

Merged on develop and available on 7.0-SNAPSHOT version until new release


Creates release notes for 0.12.0.

Release notes.

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@89b2ee4). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master       #5   +/-   ##
=========================================
  Coverage          ?   46.25%
=========================================
  Files             ?       69
  Lines             ?     6058
  Branches          ?        0
=========================================
  Hits              ?     2802
  Misses            ?     3001
  Partials          ?      255

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 89b2ee4...9397ab1. Read the comment docs.


Dont print debug info it tests are ok

the main motivation for this pr, is that we don't need debug info for a passing tests.
When a test fail we are looking for debug info. Otherwise debug a failed test become more diffult with noisy and non relevant information.
Note:* i removed only the bigdumps, this took a lot of lines, like more then 13 lines pro test, the 2-3 lines debug are ok imho, but i can remove them of course, but i think they are fine.
I was thinking to having an extra env. variable for printing debug info, but this complicate the whole design for a testsuite.
Imho a testsuite acceptance, should be simple as possible, for maintaining it longterm.
When a test fail, we have debug infos, otherwise the tests are there only for catching regression, so if it pass we don't need extra infos. ( we just want that they pass before merging pr).
Additional extra info on a test, from my POV is a code-smell, in a sense that there is either a lack of an addtional test, or the test is doing to much. (additional info when passing )
Thx for any review and feedback

fix #231

Coverage remained the same at 57.836% when pulling 23a472ab446d239e6078b77c9e774ca8b65c7d86 on MalloZup:test-silent into 11dca1e8e977948bdb432f87046de9048cacccb1 on dmacvicar:master.

Coverage remained the same at 57.836% when pulling b253d60c20f82b45c00c86a85ff94c93c467569b on MalloZup:test-silent into 11dca1e8e977948bdb432f87046de9048cacccb1 on dmacvicar:master.

Coverage remained the same at 57.836% when pulling 6f325fae8cd6d8578b9a3574ca11b718582b5e86 on MalloZup:test-silent into 11dca1e8e977948bdb432f87046de9048cacccb1 on dmacvicar:master.


Overriding LogRotateFilter throws 'std::logic_error'

Can you setup a minimalistic example, one file, which has the same problem as you describe above?
I've not been able to recreate the situation you describe above.

I've added a unit test for the previous uncovered (doh!) overrideLogDetails, you an always build on something similar if you prefer to show the issue in a failing unit test?
https://github.com/KjellKod/g3sinks/pull/115

I think I've one the error in your code. MyCustomFormatting should be using msg but instead it's using logObj which is not passed into the function. Not sure why the std::logic_error is being cast but this seem regardless to be an error with the code you describe in your screenshot.
I'm closing this ticket unless you have a follow-up.

I tried to use the same code as above for a single node/thread and it worked. However, when I'm using the same code for multiple threads, it gives me a segmentation fault.

Please address the bug that I called out. Ref screenshot and description
Sent from my iPhone

On Jun 8, 2021, at 3:13 AM, bugAssassin007 @.***> wrote:

I tried to use the same code as above for a single node/thread and it worked. However, when I'm using the same code for multiple threads, it gives me a segmentation fault.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.

candidate for closing if I don't hear back in a couple of days

I think I've one the error in your code. MyCustomFormatting should be using msg but instead it's using logObj which is not passed into the function. Not sure why the std::logic_error is being cast but this seem regardless to be an error with the code you describe in your screenshot.
I'm closing this ticket unless you have a follow-up.

I used logObj here as I dint want to change the library, but create a function part of a class that will pass the string for formatting the log. It seemed to have worked. It dint work earlier due to the reason that my code separated the initialisation and invoking of the sink and some threads invoked the service without initialising the sink in the first place. I corrected that. Thank you for your time and spontaneous help.


Integrations: Consolidate section ingredients into flat list, and expand in-depth, part 1

About
Introduce canonical pages for integrations with CrateDB, by pulling in content from the Ecosystem Catalog, and extending it with educational resources (tutorials, webinars, hands-on instructions, and code / configuration snippets).
Preview

Apache Flink
Apache Superset
dbt
LangChain
LlamaIndex

References

https://github.com/crate/crate-clients-tools/pull/154

[...] it is a lot of moving stuff around 😄.

Right, and it is only just a humble start to give you a rough idea about what's inside https://github.com/crate/tech-content/issues/117.


crash with Directory Record is over 400

UmdImageCreator crashes with Directory Record is over 400, then powers off the PSP. Seen so far with NPJH-50473, ULUS-10051 and ULUS-10551
edit: this was seen using the version downloaded from https://github.com/saramibreak/UmdImageCreator/releases/tag/v1.6 on a PSP-3001 running system 6.61 PRO-C∞

i think the limit is defined here: https://github.com/saramibreak/UmdImageCreator/blob/master/UmdImageCreator/forwardDeclaration.h#L21
#define PATH_TABLE_RECORD_SIZE	(400)

... however i don't know why 400 was chosen, nor whether it is ok to simply increase it

IIRC, I think the value was selected because memory could not be allocated.
I will consider if there are other solutions in the near future.

Apparently, I was mistaken. UmdImageCreator_test.zip

changed PATH_TABLE_RECORD_SIZE to 4000

Thanks! The test build got working images from ULUS-10051 and ULUS-10551 that match PSP Filer images
Unfortunately NPJH-50473 is crashing at a later point now, but I think it's no longer this issue since it reads the 438 DirectoryRecords first before it suddenly exits

I was able to get redump-matching images for all three discs using the test binary, however it seems to have a regression concerning exiting on spurious computed shortage

memory stick is a dual-slot TF adapter with two SD cards (decimal 128 GB and decimal 8 GB) summing to around 126 GB binary, and over 60GB are free still.

It seems Max<PHONE_NUMBER>) and Free (672399360) of the MemoryStick size overflow.
Uploaded.
UmdImageCreator_test.zip

memory stick is a dual-slot TF adapter with two SD cards (decimal 128 GB and decimal 8 GB) summing to around 126 GB binary, and over 60GB are free still.

It seems Max<PHONE_NUMBER>) and Free (672399360) of the MemoryStick size overflow. Uploaded. UmdImageCreator_test.zip

thanks! i will try this one the next time my space computation reaches the overflow conditions

memory stick is a dual-slot TF adapter with two SD cards (decimal 128 GB and decimal 8 GB) summing to around 126 GB binary, and over 60GB are free still.

It seems Max<PHONE_NUMBER>) and Free (672399360) of the MemoryStick size overflow. Uploaded. UmdImageCreator_test.zip

Unfortunately this build still crashes with a shortage. Under same false shortage conditions release 20231201 still dumps just fine though
Build 20231201 working

Test build crashes right after this message

Free space reported by system (it matches what I see from Windows too)

fixed again.
UmdImageCreator_test.zip

Tested - works with NPJH-50473

@Still34
Thanks.


Only create VPC iff no VPC ID is passed

https://github.com/terraform-aws-modules/terraform-aws-atlantis/blob/508eb0c3bb99a84dfe9fdbe57f1989be711f4197/main.tf#L18
Great work Anton!
Recently I also developed an Atlantis module to which I pass a vpc_id as a variable instead of a cidr_block to create a new one - under the assumption that it's more common for someone to deploy Atlantis in an existing VPC.
My proposal:

have 2 input variables (cidr_block and vpc_id), both with dummy defaults.
use count to determine whether a new VPC is needed, in which case the cidr_block would be used.

@bsuv Thanks for opening this issue!
You are literally reading my mind. :) I am working on it right now. Expect a PR in a couple of hours with some more enhancements like this.

The weather was so good today, so it took me a little bit longer to complete this.
v1.1.0 has been released and there is a possibility to reuse existing VPC and subnets.


Flask Application & Azure Cosmodb table service import in Azure Web

Hi,
I am trying to host a simple python Flask application in Azure Cloud Web. I am able to host Flask application successfully in Azure Web and linked with my github repository from Azure web application.
But when I try to include Azure Cosmodb commands in my python program it is not working. Currently I have just included only below command in my program.
"from azure.cosmosdb.table.tableservice import TableService"
But when i refreshes the page i am getting "Application Error ". Same program i am able to execute from my pycharm IDE. So I suspect it might be because of my requirements.txt file. Can somebody help.
Please check my Github repository link https://github.com/GITHENR/Flask_table

It is solved


cannot search the bluetooth post printer

Hello,
I use the galaxy tab A and the printer is MHT-PT10.
When i search the printer i cannot see any printer on the list. On the bluetooth setting, i paired the tablet and the printer.

I also faced this problem.

Same Problem here. on device Samsung Galaxy A70 running Android 11.

Same problem here. I use MPT-III and it is not shawn.

same problem. After I upgrade null safety, it is not show MPT-III.


Fixing deleting temporary files

This fixes deleting temporary files.
I have written again the compression of the files and folders without the using of external libraries.
This resolves the issue #277

Hello @vincenzoAiello , thanks for this! I'm reviewing it now,
Claudio

Hello @claudiodangelis, the problem is in the "Create" and "Close" functions because the zip file isn't closed properly.
These are the original functions of the library:

The problem is that the zip file is opened but it is never closed. Only the writer is closed.
Infact if i modify the code in this way it works:


add getTransactionByBlockIdAndIndex tests to e2e rust tests

Add test cases for getTransactionByBlockIdAndIndex.
Parent issue: https://github.com/keep-starknet-strange/madara/issues/904
Pull Request type
Please add the labels corresponding to the type of changes your PR introduces:

Testing

What is the current behavior?
Resolves: #NA
What is the new behavior?
Rust impl for test get_transaction_by_block_id_and_index:

Return an error for a nonexisting block
Return an error for an out of block index
Compare with getBlockWithTx and make sure order is the same

Does this introduce a breaking change?
No
Other information

@tdelabro Hi, I opened a new PR because of rebase
https://github.com/keep-starknet-strange/madara/pull/1067

The way to go is usually to rebase and then push force, it allows us to keep the history of the comments :)