HADOOP-19072. S3A: expand optimisations on stores with "fs.s3a.performance.flags" for mkdir

Description of PR
HADOOP-19072 S3A: expand optimisations on stores with "fs.s3a.create.performance"
How was this patch tested?
us-west-2
For code changes:

[X] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
[X] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 20s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+0 :ok:
markdownlint
0m  0s

markdownlint was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 6 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
14m  1s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
19m 19s

trunk passed

+1 :green_heart:
compile
8m 18s

trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
compile
7m 31s

trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08

+1 :green_heart:
checkstyle
1m 59s

trunk passed

+1 :green_heart:
mvnsite
1m 29s

trunk passed

+1 :green_heart:
javadoc
1m  6s

trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
1m  1s

trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08

-1 :x:
spotbugs
1m 24s
/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html
hadoop-common-project/hadoop-common in trunk has 1 extant spotbugs warnings.

+1 :green_heart:
shadedclient
19m 55s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 20s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
0m 45s

the patch passed

+1 :green_heart:
compile
7m 53s

the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javac
7m 53s

the patch passed

+1 :green_heart:
compile
7m 40s

the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08

+1 :green_heart:
javac
7m 40s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
1m 56s

the patch passed

+1 :green_heart:
mvnsite
1m 24s

the patch passed

+1 :green_heart:
javadoc
1m  1s

the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
1m  1s

the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08

+1 :green_heart:
spotbugs
2m 22s

the patch passed

+1 :green_heart:
shadedclient
19m 56s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
16m 34s

hadoop-common in the patch passed.

+1 :green_heart:
unit
2m 25s

hadoop-aws in the patch passed.

+1 :green_heart:
asflicense
0m 39s

The patch does not generate ASF License warnings.

144m 55s

Subsystem
Report/Notes

Docker
ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6543/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/6543

Optional Tests
dupname asflicense mvnsite codespell detsecrets markdownlint compile javac javadoc mvninstall unit shadedclient spotbugs checkstyle

uname
Linux d9248b10a158 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 2728303b77240d3e7bfe38f0b33d3f325c34ac37

Default Java
Private Build-1.8.0_392-8u392-ga-1~20.04-b08

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6543/1/testReport/

Max. process+thread count
2153 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6543/1/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

Tested against us-west-2:
mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale -Dprefetch

@shameersss1 what do you think here?
actually, maybe under magic paths we skip trying to create dirs at all, at least on the in-memory mode. no files to look for after all so all that happens is a dir tree is needlessly created, and the HEAD requests I'm proposing wouldn't even find any conflict with files that don't exist

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 22s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+0 :ok:
markdownlint
0m  0s

markdownlint was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 6 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
14m  4s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
21m 16s

trunk passed

+1 :green_heart:
compile
8m 48s

trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
compile
7m 59s

trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08

+1 :green_heart:
checkstyle
2m  0s

trunk passed

+1 :green_heart:
mvnsite
1m 24s

trunk passed

+1 :green_heart:
javadoc
1m  2s

trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
1m  2s

trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08

-1 :x:
spotbugs
1m 26s
/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html
hadoop-common-project/hadoop-common in trunk has 1 extant spotbugs warnings.

+1 :green_heart:
shadedclient
19m 55s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 20s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
0m 49s

the patch passed

+1 :green_heart:
compile
8m 18s

the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javac
8m 18s

the patch passed

+1 :green_heart:
compile
7m 45s

the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08

+1 :green_heart:
javac
7m 45s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

-0 :warning:
checkstyle
2m  4s
/results-checkstyle-root.txt
root: The patch generated 2 new + 15 unchanged - 0 fixed = 17 total (was 15)

+1 :green_heart:
mvnsite
1m 22s

the patch passed

+1 :green_heart:
javadoc
0m 58s

the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
1m  2s

the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08

+1 :green_heart:
spotbugs
2m 25s

the patch passed

+1 :green_heart:
shadedclient
19m 56s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
17m 12s

hadoop-common in the patch passed.

+1 :green_heart:
unit
2m 16s

hadoop-aws in the patch passed.

+1 :green_heart:
asflicense
0m 34s

The patch does not generate ASF License warnings.

148m 44s

Subsystem
Report/Notes

Docker
ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6543/3/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/6543

Optional Tests
dupname asflicense mvnsite codespell detsecrets markdownlint compile javac javadoc mvninstall unit shadedclient spotbugs checkstyle

uname
Linux d3062a6e7ed1 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 12e6bff73832fd72a82ad43667aa531b4976d1ab

Default Java
Private Build-1.8.0_392-8u392-ga-1~20.04-b08

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6543/3/testReport/

Max. process+thread count
3151 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6543/3/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 21s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+0 :ok:
markdownlint
0m  0s

markdownlint was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 6 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
13m 59s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
20m 18s

trunk passed

+1 :green_heart:
compile
8m 18s

trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
compile
7m 32s

trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08

+1 :green_heart:
checkstyle
2m  1s

trunk passed

+1 :green_heart:
mvnsite
1m 24s

trunk passed

+1 :green_heart:
javadoc
1m  3s

trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
0m 56s

trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08

-1 :x:
spotbugs
1m 24s
/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html
hadoop-common-project/hadoop-common in trunk has 1 extant spotbugs warnings.

+1 :green_heart:
shadedclient
19m 24s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 20s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
0m 43s

the patch passed

+1 :green_heart:
compile
8m  3s

the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javac
8m  3s

the patch passed

+1 :green_heart:
compile
7m 35s

the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08

+1 :green_heart:
javac
7m 35s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
1m 56s

the patch passed

+1 :green_heart:
mvnsite
1m 19s

the patch passed

+1 :green_heart:
javadoc
0m 55s

the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
0m 59s

the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08

+1 :green_heart:
spotbugs
2m 19s

the patch passed

+1 :green_heart:
shadedclient
19m 37s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
16m 50s

hadoop-common in the patch passed.

+1 :green_heart:
unit
2m 20s

hadoop-aws in the patch passed.

+1 :green_heart:
asflicense
0m 33s

The patch does not generate ASF License warnings.

144m 26s

Subsystem
Report/Notes

Docker
ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6543/4/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/6543

Optional Tests
dupname asflicense mvnsite codespell detsecrets markdownlint compile javac javadoc mvninstall unit shadedclient spotbugs checkstyle

uname
Linux f3c9795ec06a 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / ff8a9d2e9dab36887a266a0eb00d6268c4331a1c

Default Java
Private Build-1.8.0_392-8u392-ga-1~20.04-b08

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6543/4/testReport/

Max. process+thread count
1282 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6543/4/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@shameersss1 what do you think here? actually, maybe under magic paths we skip trying to create dirs at all, at least on the in-memory mode. no files to look for after all so all that happens is a dir tree is needlessly created, and the HEAD requests I'm proposing wouldn't even find any conflict with files that don't exist

@steveloughran In the proposed solution (https://issues.apache.org/jira/browse/HADOOP-19047), Even in the in-memory mode, The taskAttempt will write a (.pendingset) file containing the metadata of multi-part-upload (MPU) inside the magic path which will be read by the driver process and Hence the directory creation is necessary.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 23s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+0 :ok:
markdownlint
0m  0s

markdownlint was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 6 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
14m 16s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
21m 45s

trunk passed

+1 :green_heart:
compile
9m 46s

trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1

+1 :green_heart:
compile
9m  3s

trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06

+1 :green_heart:
checkstyle
2m 10s

trunk passed

+1 :green_heart:
mvnsite
1m 35s

trunk passed

+1 :green_heart:
javadoc
1m  9s

trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1

+1 :green_heart:
javadoc
1m  7s

trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06

+1 :green_heart:
spotbugs
2m 17s

trunk passed

+1 :green_heart:
shadedclient
21m 49s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 21s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
0m 54s

the patch passed

+1 :green_heart:
compile
8m 23s

the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1

+1 :green_heart:
javac
8m 23s

the patch passed

+1 :green_heart:
compile
7m 47s

the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06

+1 :green_heart:
javac
7m 47s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
1m 58s

the patch passed

+1 :green_heart:
mvnsite
1m 25s

the patch passed

+1 :green_heart:
javadoc
0m 53s

the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1

+1 :green_heart:
javadoc
1m  2s

the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06

+1 :green_heart:
spotbugs
2m 25s

the patch passed

+1 :green_heart:
shadedclient
20m  0s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
16m 39s

hadoop-common in the patch passed.

+1 :green_heart:
unit
2m 14s

hadoop-aws in the patch passed.

+1 :green_heart:
asflicense
0m 35s

The patch does not generate ASF License warnings.

154m 16s

Subsystem
Report/Notes

Docker
ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6543/5/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/6543

Optional Tests
dupname asflicense mvnsite codespell detsecrets markdownlint compile javac javadoc mvninstall unit shadedclient spotbugs checkstyle

uname
Linux 106dcbd22ec9 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / ff8a9d2e9dab36887a266a0eb00d6268c4331a1c

Default Java
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6543/5/testReport/

Max. process+thread count
1273 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6543/5/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@steveloughran could you please take another look?

reviewed, i'm just wondering how to make the test the cleanest.
Going to invite reviews from @shameersss1 @ahmarsuhail @HarshitGupta11 @mukund-thakur as they've been looking around here.
Does anyone expect anything to break from this? I don't: we know code doesn't normally try these tricks, otherwise we'd have had complaints about other optimisations.

ok, this is getting over complex.
proposed: copy the superclass code but remove the expectation of failures, retaining only setup and validation.

sounds good, addressed in the latest revision.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 22s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  1s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+0 :ok:
detsecrets
0m  1s

detect-secrets was not available.

+0 :ok:
markdownlint
0m  1s

markdownlint was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 6 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
14m  9s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
23m 56s

trunk passed

+1 :green_heart:
compile
8m 47s

trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1

+1 :green_heart:
compile
8m  2s

trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06

+1 :green_heart:
checkstyle
2m  2s

trunk passed

+1 :green_heart:
mvnsite
1m 20s

trunk passed

+1 :green_heart:
javadoc
1m  3s

trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1

+1 :green_heart:
javadoc
0m 59s

trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06

+1 :green_heart:
spotbugs
2m 14s

trunk passed

+1 :green_heart:
shadedclient
20m 42s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 20s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
0m 50s

the patch passed

+1 :green_heart:
compile
8m 24s

the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1

+1 :green_heart:
javac
8m 24s

the patch passed

+1 :green_heart:
compile
8m  2s

the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06

+1 :green_heart:
javac
8m  2s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
1m 55s

the patch passed

+1 :green_heart:
mvnsite
1m 23s

the patch passed

+1 :green_heart:
javadoc
0m 56s

the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1

+1 :green_heart:
javadoc
0m 59s

the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06

+1 :green_heart:
spotbugs
2m 24s

the patch passed

+1 :green_heart:
shadedclient
20m 28s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
16m 36s

hadoop-common in the patch passed.

+1 :green_heart:
unit
2m 15s

hadoop-aws in the patch passed.

+1 :green_heart:
asflicense
0m 33s

The patch does not generate ASF License warnings.

152m 50s

Subsystem
Report/Notes

Docker
ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6543/6/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/6543

Optional Tests
dupname asflicense mvnsite codespell detsecrets markdownlint compile javac javadoc mvninstall unit shadedclient spotbugs checkstyle

uname
Linux 2e19c4bcca99 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 8d3012c3e40f3c0575951064ea43fa790debb974

Default Java
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6543/6/testReport/

Max. process+thread count
2153 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6543/6/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

But if we're doing this for a whole directory, for all applications, I think that is a bit too risky.

I see your point.
Let me run the whole suite with the latest revision.

Tested against us-west-2, looks good for this change.
Though found a separate issue with scale tests using noaa-cors-pds bucket for my local endpoint/region setup. It's minor issue, not a big deal, will create a Jira later.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 20s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+0 :ok:
markdownlint
0m  0s

markdownlint was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 6 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
18m 37s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
20m 51s

trunk passed

+1 :green_heart:
compile
9m 41s

trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1

+1 :green_heart:
compile
9m  4s

trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06

+1 :green_heart:
checkstyle
2m 14s

trunk passed

+1 :green_heart:
mvnsite
1m 19s

trunk passed

+1 :green_heart:
javadoc
1m  1s

trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1

+1 :green_heart:
javadoc
0m 55s

trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06

+1 :green_heart:
spotbugs
2m  3s

trunk passed

+1 :green_heart:
shadedclient
21m 27s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 21s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
0m 47s

the patch passed

+1 :green_heart:
compile
9m 50s

the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1

+1 :green_heart:
javac
9m 50s

the patch passed

+1 :green_heart:
compile
8m 54s

the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06

+1 :green_heart:
javac
8m 54s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
2m  0s

the patch passed

+1 :green_heart:
mvnsite
1m 18s

the patch passed

+1 :green_heart:
javadoc
0m 54s

the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1

+1 :green_heart:
javadoc
0m 52s

the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06

+1 :green_heart:
spotbugs
2m 22s

the patch passed

+1 :green_heart:
shadedclient
21m 44s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
16m 49s

hadoop-common in the patch passed.

+1 :green_heart:
unit
2m 12s

hadoop-aws in the patch passed.

+1 :green_heart:
asflicense
0m 36s

The patch does not generate ASF License warnings.

160m 27s

Subsystem
Report/Notes

Docker
ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6543/7/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/6543

Optional Tests
dupname asflicense mvnsite codespell detsecrets markdownlint compile javac javadoc mvninstall unit shadedclient spotbugs checkstyle

uname
Linux 84529fe21bf3 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / fc306d5e87f9ba64320b9b181f2d4d0f71e678b2

Default Java
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6543/7/testReport/

Max. process+thread count
2769 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6543/7/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 21s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+0 :ok:
markdownlint
0m  0s

markdownlint was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 6 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
17m 49s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
21m 18s

trunk passed

+1 :green_heart:
compile
9m 58s

trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1

+1 :green_heart:
compile
8m 45s

trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06

+1 :green_heart:
checkstyle
2m 24s

trunk passed

+1 :green_heart:
mvnsite
1m 24s

trunk passed

+1 :green_heart:
javadoc
0m 58s

trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1

+1 :green_heart:
javadoc
0m 54s

trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06

+1 :green_heart:
spotbugs
2m  4s

trunk passed

+1 :green_heart:
shadedclient
21m 45s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 22s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
0m 48s

the patch passed

+1 :green_heart:
compile
9m 34s

the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1

+1 :green_heart:
javac
9m 34s

the patch passed

+1 :green_heart:
compile
8m 41s

the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06

+1 :green_heart:
javac
8m 41s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
2m 11s

the patch passed

+1 :green_heart:
mvnsite
1m 20s

the patch passed

+1 :green_heart:
javadoc
0m 52s

the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1

+1 :green_heart:
javadoc
0m 55s

the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06

+1 :green_heart:
spotbugs
2m 16s

the patch passed

+1 :green_heart:
shadedclient
21m 51s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
16m 57s

hadoop-common in the patch passed.

+1 :green_heart:
unit
2m 10s

hadoop-aws in the patch passed.

+1 :green_heart:
asflicense
0m 32s

The patch does not generate ASF License warnings.

160m 31s

Subsystem
Report/Notes

Docker
ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6543/8/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/6543

Optional Tests
dupname asflicense mvnsite codespell detsecrets markdownlint compile javac javadoc mvninstall unit shadedclient spotbugs checkstyle

uname
Linux 055bbc67034d 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / b7e4ede34ba1fb2094ba5363c0ec07cd763e5336

Default Java
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6543/8/testReport/

Max. process+thread count
1274 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6543/8/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

Don't think I have forgotten about this -I have just been very distracted.
I'm wondering if we should provide a list of options to "optimise", e.g "create, mkdir, delete" and the specific optimisations those get turned on. Yes, it is suspiciously like those option sets we have for databases. Microsoft are looking at what they can do to speed up IO, such as skipping HEAD requests in open(), making that an option which can be enabled where "things don't break". Having a consistent configuration key and list of optimisations would make a lot of sense.
Thoughts?

I'm wondering if we should provide a list of options to "optimise", e.g "create, mkdir, delete" and the specific optimisations those get turned on. Yes, it is suspiciously like those option sets we have for databases.

I believe this would be good optimization tuning overall, and yes it does remind of database optimizations. In fact, HBase can use these optimizations for HFile creation, deletion etc too (as long as it knowns when and what it is doing). Since we already have fs.s3a.create.performance, how about we file separate jiras for fs.s3a.delete.performance etc new modes?
or you meant, we provide a generic config fs.s3a.optimizations with values including create and/or delete, accordingly we skip HEAD requests?

Don't think I have forgotten about this -I have just been very distracted.

me too, not a problem at all :)

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

_ Prechecks _

+1 :green_heart:
dupname
0m 02s

No case conflicting files found.

+0 :ok:
codespell
0m 02s

codespell was not available.

+0 :ok:
detsecrets
0m 02s

detect-secrets was not available.

+0 :ok:
markdownlint
0m 02s

markdownlint was not available.

+0 :ok:
spotbugs
0m 01s

spotbugs executables are not available.

+1 :green_heart:
@author
0m 00s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m 00s

The patch appears to include 6 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
2m 16s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
90m 13s

trunk passed

+1 :green_heart:
compile
39m 39s

trunk passed

+1 :green_heart:
checkstyle
5m 56s

trunk passed

-1 :x:
mvnsite
4m 24s
/branch-mvnsite-hadoop-common-project_hadoop-common.txt
hadoop-common in trunk failed.

+1 :green_heart:
javadoc
9m 25s

trunk passed

+1 :green_heart:
shadedclient
161m 37s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+0 :ok:
mvndep
2m 18s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
8m 37s

the patch passed

+1 :green_heart:
compile
37m 56s

the patch passed

+1 :green_heart:
javac
37m 56s

the patch passed

+1 :green_heart:
blanks
0m 00s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
6m 00s

the patch passed

-1 :x:
mvnsite
4m 30s
/patch-mvnsite-hadoop-common-project_hadoop-common.txt
hadoop-common in the patch failed.

+1 :green_heart:
javadoc
9m 22s

the patch passed

+1 :green_heart:
shadedclient
171m 17s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
asflicense
8m 21s

The patch does not generate ASF License warnings.

528m 37s

Subsystem
Report/Notes

GITHUB PR
https://github.com/apache/hadoop/pull/6543

Optional Tests
dupname asflicense mvnsite codespell detsecrets markdownlint compile javac javadoc mvninstall unit shadedclient spotbugs checkstyle

uname
MINGW64_NT-10.0-17763 48a1e573d3ea 3.4.10-87d57229.x86_64 2024-02-14 20:17 UTC x86_64 Msys

Build tool
maven

Personality
/c/hadoop/dev-support/bin/hadoop.sh

git revision
trunk / b7e4ede34ba1fb2094ba5363c0ec07cd763e5336

Default Java
Azul Systems, Inc.-1.8.0_332-b09

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6543/1/testReport/

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6543/1/console

versions
git=2.44.0.windows.1

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

I'm going to propose we change how the options are done, and do something similar for ABFS.
I think we need something like C/C++ optimisers where you pass in a -O list of things to optimise.
This lets you turn on everything you can but turn off those which breaks your code.
This avoids us having to deal with regressions where suddenly something breaks -and the only fix is to turn off all optimisation.
It also gives us the ability to add some very aggressive optimisations, such as disabling probing for and recreating parent directories on delete and rename. harshit has tried this and some things break.
If we make this one of the flags, those deployments with applications which know they are robust can turn it on.

we add specific optimisation flags for different behaviours, which we can explicitly turn on and off.
we add a single "fs.s3a.performance.options" which takes a list of these, parses to an S3APerformanceFlags object containing the flags. we can move this parsing out of the s3aFileSystem code into the S3APerformanceFlags class, which assists testing.
unknown flags are logged once at info.
the performance flags are available from StoreContext.

It would still be good to wire this up to hasPathCapability.
Proposed: S3APerformanceFlags adds a hasCapability(string) method, and if s3aFS.hasPathCapability is probed for a feature begininning "fs.s3a.performance.options." then the rest of the string passed in for the check.
The create.performance flag now complicates things here. Too bad we have already shipped it.
Proposed: S3APerformanceFlags also looks for that flag value, but also adds "create" as one of the options
that is
fs.s3a.performance.options=create,mkdirs

would cover both that and this new change.
I think we can and should lay down a policy here then which is "the semantics of a specific optimisation flag MUST NOT CHANGE" but that new flags MAY be added to tune that behaviour further.
the most radical would be if we copy the presto connector and declare that all paths which don't return a file or a non-empty listing are a directory.
they can downgrade mkdirs() to a no-op, delete dir, rm dir etc always report a parent dir existing, so their code is happy.
They can get away with this because they know the exact semantics of the code -and that it does not break with this change. We lack that luxury across the broad pool of applications using our library. That doesn't mean we can't allow a select few applications to take advantage of optimisations we have written and tested with them.

The above proposal of providing list of optimization flags sounds impressive.
Please let me know if this summary looks good:
As part of this Jira:

Add fs.s3a.performance.options as new config with only valid values as create and mkdir for now.
Create S3APerformanceFlags class (which can contain List of Enum values). Enum can be PerformanceFlag and it should be defined in StoreContext.
Mapping of the comma separated String value of fs.s3a.performance.options to S3APerformanceFlags object can be done as static utility of S3APerformanceFlags class.
Unknown flags are logged once at info
Provide PathCapability for fs.s3a.performance.options.${flag} where ${flag} value would be create/mkdir for now. When this is probed, pathCapability should call S3APerformanceFlags#hasCapability(${flag}).
Document the policy for fs.s3a.performance.options to indicate that the semantic of a particular optimization flag must not change but new optimization option could be provided in future to tune this behavior.

For future Jiras:

Add more optimization options for delete, rename operations.

One question: IIUC, we don't need to keep the current PR behavior in case fs.s3a.create.performance is enabled, since we are not introducing new fs.s3a.performance.options, correct? Also, will it be prudent to deprecate config fs.s3a.create.performance and perhaps log at once in s3afs if user is still using it? Probably we can do it in separate jira too.

As of today, fs.s3a.create.performance is mandatory option while creating file:
      builder
          .create()
          .overwrite(true)
          .must(FS_S3A_CREATE_PERFORMANCE, performanceCreation);

Shall we remove fs.s3a.create.performance as mandatory one and add fs.s3a.performance.options instead? Can we add value of this config as serialized string of S3APerformanceFlags object, and let CreateFileBuilder deserialize it to S3APerformanceFlags object? This way, CreateFileBuilder gets the object back, though this seems too complicated.
Instead, maybe we remove fs.s3a.create.performance as mandatory and provide S3APerformanceFlags object (non-nullable) as part of CreateFileBuilder constructor directly?

I'm doing a quick PR of the design; @HarshitGupta11 and I discussed it.

I'm doing a quick PR of the design; @HarshitGupta11 and I discussed it.

Got it, i was planning to embed the logic as part of this PR sometime early next week but separate PR also sounds good.

Got it, i was planning to embed the logic as part of this PR sometime early next week but separate PR sounds more manageable!

it's a bit blurred as there are now options in the code for features we haven't implemented. to be strictest, we would maybe want that base impl to only do create; your pr to add "mkdir", etc.

with #6789 in, there's space in fs.s3a.performance.flags for this. I think I even created the enum option for you...

Sounds good, let me get to this in a few days to sync with the latest trunk.

surprised to see no merge conflicts here

i need to make doc changes and re-run the test suite.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
6m 44s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+0 :ok:
markdownlint
0m  0s

markdownlint was not available.

+1 :green_heart:
@author
0m  1s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 6 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
15m  2s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
20m 13s

trunk passed

+1 :green_heart:
compile
8m 56s

trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2

+1 :green_heart:
compile
8m 16s

trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08

+1 :green_heart:
checkstyle
2m  4s

trunk passed

+1 :green_heart:
mvnsite
1m 40s

trunk passed

+1 :green_heart:
javadoc
1m 12s

trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2

+1 :green_heart:
javadoc
1m 10s

trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08

+1 :green_heart:
spotbugs
2m 22s

trunk passed

+1 :green_heart:
shadedclient
20m 34s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 22s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
0m 55s

the patch passed

+1 :green_heart:
compile
8m 39s

the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2

+1 :green_heart:
javac
8m 39s

the patch passed

+1 :green_heart:
compile
8m 13s

the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08

+1 :green_heart:
javac
8m 13s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
2m  4s

the patch passed

+1 :green_heart:
mvnsite
1m 32s

the patch passed

+1 :green_heart:
javadoc
1m 10s

the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2

+1 :green_heart:
javadoc
1m  8s

the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08

+1 :green_heart:
spotbugs
2m 37s

the patch passed

+1 :green_heart:
shadedclient
20m 45s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
17m  6s

hadoop-common in the patch passed.

+1 :green_heart:
unit
2m 22s

hadoop-aws in the patch passed.

+1 :green_heart:
asflicense
0m 43s

The patch does not generate ASF License warnings.

158m 30s

Subsystem
Report/Notes

Docker
ClientAPI=1.46 ServerAPI=1.46 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6543/9/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/6543

Optional Tests
dupname asflicense mvnsite codespell detsecrets markdownlint compile javac javadoc mvninstall unit shadedclient spotbugs checkstyle

uname
Linux 47344517a8e4 5.15.0-116-generic #126-Ubuntu SMP Mon Jul 1 10:14:24 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 142f0d4db75fa02576b95b819b36a6aab67f596c

Default Java
Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6543/9/testReport/

Max. process+thread count
2153 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6543/9/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 19s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+0 :ok:
markdownlint
0m  0s

markdownlint was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 6 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
14m 42s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
20m  6s

trunk passed

+1 :green_heart:
compile
9m  0s

trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2

+1 :green_heart:
compile
8m 16s

trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08

+1 :green_heart:
checkstyle
2m  4s

trunk passed

+1 :green_heart:
mvnsite
1m 35s

trunk passed

+1 :green_heart:
javadoc
1m 16s

trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2

+1 :green_heart:
javadoc
1m 10s

trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08

+1 :green_heart:
spotbugs
2m 26s

trunk passed

+1 :green_heart:
shadedclient
20m 43s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 21s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
0m 55s

the patch passed

+1 :green_heart:
compile
8m 32s

the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2

+1 :green_heart:
javac
8m 32s

the patch passed

+1 :green_heart:
compile
8m  5s

the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08

+1 :green_heart:
javac
8m  5s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
2m  6s

the patch passed

+1 :green_heart:
mvnsite
1m 33s

the patch passed

+1 :green_heart:
javadoc
1m  9s

the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2

+1 :green_heart:
javadoc
1m  9s

the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08

+1 :green_heart:
spotbugs
2m 30s

the patch passed

+1 :green_heart:
shadedclient
20m 37s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
17m  1s

hadoop-common in the patch passed.

+1 :green_heart:
unit
2m 10s

hadoop-aws in the patch passed.

+1 :green_heart:
asflicense
0m 43s

The patch does not generate ASF License warnings.

150m 59s

Subsystem
Report/Notes

Docker
ClientAPI=1.46 ServerAPI=1.46 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6543/10/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/6543

Optional Tests
dupname asflicense mvnsite codespell detsecrets markdownlint compile javac javadoc mvninstall unit shadedclient spotbugs checkstyle

uname
Linux 037b9f503514 5.15.0-116-generic #126-Ubuntu SMP Mon Jul 1 10:14:24 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / e7ef0d8eff7dccfa4a521a551a88905a98918258

Default Java
Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6543/10/testReport/

Max. process+thread count
3153 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6543/10/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

Tested against us-west-2:
$ mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale -Dprefetch

@steveloughran @mukund-thakur
The latest PR revision is ready for review.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
6m 51s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+0 :ok:
markdownlint
0m  0s

markdownlint was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 6 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
14m 54s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
20m  3s

trunk passed

+1 :green_heart:
compile
9m  2s

trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04

+1 :green_heart:
compile
8m 11s

trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05

+1 :green_heart:
checkstyle
2m  6s

trunk passed

+1 :green_heart:
mvnsite
1m 37s

trunk passed

+1 :green_heart:
javadoc
1m 14s

trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04

+1 :green_heart:
javadoc
1m 10s

trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05

+1 :green_heart:
spotbugs
2m 11s

trunk passed

+1 :green_heart:
shadedclient
20m 14s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 23s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
0m 50s

the patch passed

+1 :green_heart:
compile
8m 41s

the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04

+1 :green_heart:
javac
8m 41s

the patch passed

+1 :green_heart:
compile
8m  8s

the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05

+1 :green_heart:
javac
8m  8s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
2m  4s

the patch passed

+1 :green_heart:
mvnsite
1m 32s

the patch passed

+1 :green_heart:
javadoc
1m 10s

the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04

+1 :green_heart:
javadoc
1m 10s

the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05

+1 :green_heart:
spotbugs
2m 24s

the patch passed

+1 :green_heart:
shadedclient
20m 40s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
16m 50s

hadoop-common in the patch passed.

+1 :green_heart:
unit
2m 12s

hadoop-aws in the patch passed.

+1 :green_heart:
asflicense
0m 42s

The patch does not generate ASF License warnings.

157m 24s

Subsystem
Report/Notes

Docker
ClientAPI=1.46 ServerAPI=1.46 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6543/11/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/6543

Optional Tests
dupname asflicense mvnsite codespell detsecrets markdownlint compile javac javadoc mvninstall unit shadedclient spotbugs checkstyle

uname
Linux 43900f0f3b1f 5.15.0-116-generic #126-Ubuntu SMP Mon Jul 1 10:14:24 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / c5652a084093d92051925cef3d54fb6040d506dd

Default Java
Private Build-1.8.0_422-8u422-b05-1~20.04-b05

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6543/11/testReport/

Max. process+thread count
3153 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6543/11/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

tried to pull to branch-3.4; tests failed
reran on trunk,. tests failed
@virajjasani you need to do an urgent followup; use same JIRA
[INFO] Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractMkdir
[ERROR] Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 12.921 s <<< FAILURE! - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractMkdir
[ERROR] testMkdirOverParentFile(org.apache.hadoop.fs.contract.s3a.ITestS3AContractMkdir)  Time elapsed: 0.682 s  <<< FAILURE!
java.lang.AssertionError:
mkdirs did not fail over a file but returned true; ls s3a://stevel-london/test/testMkdirOverParentFile [00] S3AFileStatus{path=s3a://stevel-london/test/testMkdirOverParentFile/child-to-mkdir; isDirectory=true; modification_time=0; access_time=0; owner=stevel; group=stevel; permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=true; isErasureCoded=false} isEmptyDirectory=FALSE eTag=null versionId=null

        at org.junit.Assert.fail(Assert.java:89)
        at org.apache.hadoop.fs.contract.AbstractContractMkdirTest.testMkdirOverParentFile(AbstractContractMkdirTest.java:99)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
        at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
        at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
        at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
        at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
        at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
        at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
        at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
        at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.lang.Thread.run(Thread.java:750)

[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]   ITestS3AContractMkdir>AbstractContractMkdirTest.testMkdirOverParentFile:99->Assert.fail:89 mkdirs did not fail over a file but returned true; ls s3a://stevel-london/test/testMkdirOverParentFile [00] S3AFileStatus{path=s3a://stevel-london/test/testMkdirOverParentFile/child-to-mkdir; isDirectory=true; modification_time=0; access_time=0; owner=stevel; group=stevel; permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=true; isErasureCoded=false} isEmptyDirectory=FALSE eTag=null versionId=null

The test is consistently passing for me, using oregon, let me try london.
But the above failure can happen only if fs.s3a.performance.flags contain mkdir. I checked my local configs too, nothing suspicious.

The only way i am able to reproduce the exact failure is by applying this patch:
diff --git a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractMkdir.java b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractMkdir.java
index bace0a79f24..e01048b5c3b 100644
--- a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractMkdir.java
+++ b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractMkdir.java
@@ -23,6 +23,7 @@
 import org.apache.hadoop.fs.contract.AbstractFSContract;

 import static org.apache.hadoop.fs.s3a.Constants.FS_S3A_CREATE_PERFORMANCE;
+import static org.apache.hadoop.fs.s3a.Constants.FS_S3A_PERFORMANCE_FLAGS;
 import static org.apache.hadoop.fs.s3a.S3ATestUtils.removeBaseAndBucketOverrides;

 /**
@@ -33,8 +34,12 @@ public class ITestS3AContractMkdir extends AbstractContractMkdirTest {
   @Override
   protected Configuration createConfiguration() {
     Configuration conf = super.createConfiguration();
-    removeBaseAndBucketOverrides(conf,
-        FS_S3A_CREATE_PERFORMANCE);
+    removeBaseAndBucketOverrides(
+        conf,
+        FS_S3A_CREATE_PERFORMANCE,
+        FS_S3A_PERFORMANCE_FLAGS);
+    conf.setStrings(FS_S3A_PERFORMANCE_FLAGS,
+        "create,mkdir");
     return conf;
   }

or i can reproduce the exact failure with this patch without any code changes:
diff --git a/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml b/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
index 4104e304314..97a7fcb3290 100644
--- a/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
+++ b/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
@@ -1428,6 +1428,11 @@
   </description>
 </property>

+  <property>
+    <name>fs.s3a.performance.flags</name>
+    <value>mkdir</value>
+  </property>
+
 <property>
   <name>fs.s3a.security.credential.provider.path</name>
   <value />

Addendum PR to override fs.s3a.performance.flags: https://github.com/apache/hadoop/pull/6985

Thanks for validating @mukund-thakur!
This failure can only happen if the perf flags are set explicitly, i am certain. This addendum will ensure that even if the perf flags are set, the test passes: https://github.com/apache/hadoop/pull/6985

ITestS3AFileContextCreateMkdir would also fail if the perf flags are set explicitly, so the addendum overrides perf flags for both tests (ITestS3AContractMkdir and ITestS3AFileContextCreateMkdir) https://github.com/apache/hadoop/pull/6985

my setup is the most aggressive you can get: all new options automatically. new pr fixes it
  <property>
    <name>fs.s3a.performance.flags</name>
    <value>*</value>
  </property>

aah, even after the addendum i"m getting failures.
Viraj. copy my setting then rerun everything with -Dscale. Then also a run with -Dprefix
[INFO]
[ERROR] Failures:
[ERROR]   ITestS3AFSMainOperations>FSMainOperationsBaseTest.testMkdirsFailsForSubdirectoryOfExistingFile:241 Should throw IOException.
[ERROR]   ITestS3AFileSystemContract>FileSystemContractBaseTest.testMkdirsFailsForSubdirectoryOfExistingFile:224 Should throw IOException.
[ERROR]   ITestS3AFileContextMainOperations>FileContextMainOperationsBaseTest.testMkdirsFailsForSubdirectoryOfExistingFile:276 Should throw IOException.
[ERROR]   ITestS3AFileContextURI>FileContextURIBase.testMkdirsFailsForSubdirectoryOfExistingFile:258 Should throw IOException.
[INFO]

[ERROR] testMkdirsFailsForSubdirectoryOfExistingFile(org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextURI)  Time elapsed: 1.217 s  <<< FAILURE!
java.lang.AssertionError: Should throw IOException.
at org.junit.Assert.fail(Assert.java:89)
at org.apache.hadoop.fs.FileContextURIBase.testMkdirsFailsForSubdirectoryOfExistingFile(FileContextURIBase.java:258)
[ERROR] testMkdirsFailsForSubdirectoryOfExistingFile(org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextMainOperations)  Time elapsed: 1.264 s  <<< FAILURE!
java.lang.AssertionError: Should throw IOException.
at org.junit.Assert.fail(Assert.java:89)
at org.apache.hadoop.fs.FileContextMainOperationsBaseTest.testMkdirsFailsForSubdirectoryOfExistingFile(FileContextMainOperationsBaseTest.java:276)
[ERROR] testMkdirsFailsForSubdirectoryOfExistingFile(org.apache.hadoop.fs.s3a.ITestS3AFileSystemContract)  Time elapsed: 0.856 s  <<< FAILURE!
java.lang.AssertionError: Should throw IOException.
at org.junit.Assert.fail(Assert.java:89)
at org.apache.hadoop.fs.FileSystemContractBaseTest.testMkdirsFailsForSubdirectoryOfExistingFile(FileSystemContractBaseTest.java:224)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
[ERROR] testMkdirsFailsForSubdirectoryOfExistingFile(org.apache.hadoop.fs.s3a.ITestS3AFSMainOperations)  Time elapsed: 0.844 s  <<< FAILURE!
java.lang.AssertionError: Should throw IOException.
at org.junit.Assert.fail(Assert.java:89)
at org.apache.hadoop.fs.FSMainOperationsBaseTest.testMkdirsFailsForSubdirectoryOfExistingFile(FSMainOperationsBaseTest.java:241)

suspect I didn't do a full 3.4 test run before pushing it up on that branch, instead just the modified files. my mistake. there'll be another patch to backport

my setup is the most aggressive you can get: all new options automatically. new pr fixes it
  <property>
    <name>fs.s3a.performance.flags</name>
    <value>*</value>
  </property>

Sure i will run all tests with this setup. Now i think we can even introduce new mvn option to run all tests will all perf flgas? Something like -DrunAllPerfFlags (as a follow up Jira)?

no, maybe too many profiles. best to have that variety of users and their own non-standard configs

Done with the full run, these are the failures:
[ERROR] Failures:
[ERROR]   ITestS3AFSMainOperations>FSMainOperationsBaseTest.testMkdirsFailsForSubdirectoryOfExistingFile:241 Should throw IOException.
[ERROR]   ITestS3AFileSystemContract>FileSystemContractBaseTest.testMkdirsFailsForSubdirectoryOfExistingFile:224 Should throw IOException.
[ERROR]   ITestS3AFileContextMainOperations>FileContextMainOperationsBaseTest.testMkdirsFailsForSubdirectoryOfExistingFile:276 Should throw IOException.
[ERROR]   ITestS3AFileContextURI>FileContextURIBase.testMkdirsFailsForSubdirectoryOfExistingFile:258 Should throw IOException.
[INFO]

Seems to be matching with Steve's above comment listing failed tests. I will prepare the addendum soon.

Here is another addendum https://github.com/apache/hadoop/pull/6993
I am re-running the whole suite again to ensure everything passes with above settings.


⚠ Bofrost.at has degraded performance

In d20b312, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 3338 ms

Resolved: Bofrost.at performance has improved in 0461da7.


Zero halo_axisA values in halo catalog

When generating a CachedHaloCatalog using the following:
halocat = CachedHaloCatalog(simname='bolshoi', halo_finder='rockstar', redshift=0, version_name='halotools_v0p4')
there are 39 entries that have a value of 0 for halo_axisA_x, halo_axisA_y, and halo_axisA_z. This introduces NaN values when trying to use these halo orientations to align galaxies within them.
It's a relatively small number in comparison to all the halos available, and I can mask them out when generating a mock, but it seemed major enough to bring to someone's attention in case it has slipped under the radar so far.

This is useful to know. Halotools just directly uses the publicly available Rockstar subhalo catalogs, so these outlier halos are inherited from the upstream data. I'm not sure what failure mode this would signify. One innocuous explanation would be that these halos are perfect spheres. What are their values of b/a and c/a?
Regardless, probably the simplest treatment is as you say: to just mask out the outlier halos before proceeding with any further preprocessing.

For these 39 halos, the 'halo_b_to_a' and 'halo_c_to_a' values are all zero.

Hmm, ok, thanks for checking. I would have guessed that if these were perfect spheres then c/a and b/a would be unity, not zero. But of course it's possible that when Rockstar encounters such a rare edge case, it just uses zero for the fill value instead of unity.  It's good that you raised this issue in case other people encounter it and go poking around for whether it has been noticed before. But since this is such a tiny fraction of objects, presumably throwing these subhalos out will not impact a cosmological analysis, so I think we should close this issue for now until we find that this is creating some downstream problem.


Pull pricing from more consistent invoice data rather than order details page

In #7 I noticed that transactions before 2023 were not mapping to Monarch (with the exception of refunds). The extension was not getting price data for the orders since the "Transactions" dropdown on the order details page was not present for older orders.
This pull request uses the Invoice page as an alternative data source. It is nasty, truly abysmal HTML markup to work with and the selectors used are not the kind of thing that I enjoy writing or supporting. However, it seems that all orders in my account history produce the same type of invoice. This page also clearly identifies the portion of a transaction paid with gift cards which is important for people like me who were buying discounted gift cards to use as a primary payment method. Similar to the order details page, multiple transactions are identified by date and amount at the bottom of the invoice, when applicable.
This one needs more work and testing before merging, I just wanted to share what I have so far in case you decided to work on the same thing.

[x] Single transactions
[x] Partial gift card payment (as a separate Monarch transaction in addition to other payment)
[x] Full gift card payment
[x] Multi-transaction orders
[ ] Refunds
[ ] Multi-transaction orders with partial gift card payment

Let me know if you find any other edge cases!

Jumped the gun here, the invoice does not acknowledge refunds at all and the gift card data that I thought was only on the invoice is also on the order details. Redoing!


[en] updated /node-pressure-eviction.md

Fix some typos and add - in the blank cells of table.
Preview: https://deploy-preview-35089--kubernetes-io-main-staging.netlify.app/docs/concepts/scheduling-eviction/node-pressure-eviction/

/approve


Rationalise pkgs and add qrexec lib and dom0 tools (take two)

This changeset clean up support for pkgs, cleaning up the tag finding mechanism.
Also makes it possible to create dependencies between packages, using tags.
Also add packages for building qrexec tools, unused in any image right now but useful for future API testing.

One quick aside -- it would be great to have some example code laying somewhere around so one can quickly get going with this.

Thank you for enumerating the PKGS instead of doing a runtime search. We should do that more, including for uname, etc.


Remove case sensitivity in discover

If I enter "modflow" in discover, I find nothing, but if I type "MODFLOW" I find a lot of MODFLOW models.

Closing.  Not sure what was going on, but I tried earlier today and "modflow" found nothing.  Now it is working properly.


api key for embedding models

After successfully adding our azure deployed gpt4o_mini and text-embedding-ada-002 instances (many thanks for this plugin!, btw, I've made a small PR to update the README)
I tried to test the embedding model:
llm embed -c 'This is some content' -m <my_azure_embedding_model_id>
and got the error:

openai.OpenAIError: Missing credentials. Please pass one of api_key, azure_ad_token, azure_ad_token_provider, or the AZURE_OPENAI_API_KEY or AZURE_OPENAI_AD_TOKEN environment variables.
If I prefix the command by AZURE_OPENAI_API_KEY=<content_of_the_azure_key> it works.

I have an azure key in keys.json
its content is correct (llm -m <my_azure_chat_model_id> "What is the capital of France ?" works without setting a key)
I tried to add api_key_name: azure in the embedding model config (in azure/config.yaml), but no luck

When using azure models, sometimes you may have some models under different deployments, and thus can have different API_KEYS. Therefore, supporting the api_key_name option in azure/config.yml may be useful.


[FR] More conversion functions for calculating distances

It would be nice to be able to also convert not just from meters to anything else, but also from lets say, miles to meters.
Currently I'm in a situation where we know the radius has to be 10 miles for the points, but I still need to calculate how much meters that is, manually or through another custom written function.
It wouldn't harm to have a function like:
convertUnits( value, fromUnit, toUnit)

I'm pretty sure there's already anything between one and ~1700 libraries to do unit conversion on npm. convertUnit is only a convenience method to display results in other units than meter. Alternatively, I'm exporting all the values necessary to do the conversion manually:
https://github.com/manuelbieh/geolib/blob/master/src/constants.ts#L24
You could e.g. do:
import { distanceConversion } from 'geolib/es/constants';

const tenMilesInMeters = Math.floor(10000 / distanceConversion.mi) * distanceConversion.km

Maybe I should export them in the CommonJS build as well 🤔

3.3.0 now exports distanceConversion from constants.ts

I agree to the request of @michtio - I'm in the same situation.
As the philosophy of geolib is to have zero dependencies, I'd like to keep that. Now I'm forced to add another. :(
Nice lib anyway, thx @manuelbieh

https://www.npmjs.com/package/convert-units
npm install convert-units --save

const convert = require('convert-units');
const thousandThreehundredMilesInMeters = convert(1300).from('mi').to('m');
geolib.doWhatever(thousandThreehundredMilesInMeters);


Add SCSS support.

node-sass already support SCSS but this plugin only allows SASS file extension.

Awesome thanks :+1: Published as 1.1.0.


cannot add edge records via GUI

OrientDB Version: 2.2.9
Java Version: 1.8
OS: ubuntu 14.04

Expected behavior
Should be able to add new edge records, as per available option on GUI.
Actual behavior
No way to add From Vertex - To Vertex (as per screenshot)
Steps to reproduce
select  new record option for a given edge on schema view

Hi @blogmaniak
Thanks for reporting this. It's a known issue. I will mark this as duplicate
https://github.com/orientechnologies/orientdb-studio/issues/439


Import ECC, Hash, Schnorr, Types from Agora

Those modules can live in their own library and do not need to be in Agora.
I will do a pass over the README (which is clearly incomplete) and the comments in general.

And we need a CI.

Got the CI to work: https://github.com/Geod24/crypto/actions/runs/546601928
Only need a pass on comments now.

Using agora. as the top package can become tricky if we ever end up creating a module / package with the same name. E.g. agora.utils.* in both Agora and Crypto repos. But I guess we can just be very careful about it.

Indeed. I was trying to think of a good name for this library, because "crypto" is just too generic, but decided to just go the easy way for now.

I think we can just go ahead.


Add on DSA: Binary Trees

Description:  In the dsa section i would like to add binary tree topics under Intermediate topics
Use Case: There is already a beginner section which include basic dsa topic. Now for Intermediate section we could start with Binary tree.

I would like to start the work on  Binary tree topic @Ajay-Dhangar , under gssoc24.

Resources for Guidance
Here are some resources that may be helpful as you contribute to CodeHarborHub:

https://www.codeharborhub.live/community/features
Docusaurus Documentation
Markdown Guide
MDX Documentation
Mermaid Documentation

Resources for DSA features on our site is supported

Tabs: switch languages
line-numbering
Math Equations


date fields formats

When I work with databases that have dates in YYYY-MM-DD format the calculations and queries are performed correctly, but when the dataframe is displayed or rendered the date format is displayed in a UNIX format, how can I display the dates in the normal format that is present in the database?

thanks

Check new changes committed to https://github.com/vanna-ai/vanna/tree/app-upgrades. this solved my issue.

This is now updated in v0.3.0


How Can I Block Lato Font?

Hi, im using semantic.ui minified css on my page. im not using lato font but it's loading because of semantic ui css. How can i block this on wordpress functions.php?
CDN URL: https://cdnjs.cloudflare.com/ajax/libs/semantic-ui/2.4.1/semantic.min.css?ver=2.4.1
@import url(https://fonts.googleapis.com/css?family=Lato:400,700,400italic,700italic&subset=latin);/*!
Isn't there any way to block without editing the file (leaving cdn)? It loads too much version of the font too as unnecessary. Thanks.

Hi I have see this issue, https://www.bitdefender.com/blog/hotforsecurity/german-website-fined-100-euros-after-court-says-googles-font-library-violates-gdpr/
So I want to download en serve this file from my server, how can do that?


[release/1.3 backport] bump runc v1.0.0-rc9

backport of:

https://github.com/containerd/containerd/pull/3713 Bump runc to 1b8a1eeec3f337ab5d94f28980
https://github.com/containerd/containerd/pull/3721 bump runc v1.0.0-rc9

Codecov Report

Merging #3724 into release/1.3 will increase coverage by 3.51%.
The diff coverage is n/a.

@@               Coverage Diff               @@
##           release/1.3    #3724      +/-   ##
===============================================
+ Coverage         42.1%   45.61%   +3.51%
===============================================
  Files              129      116      -13
  Lines            14307    11454    -2853
===============================================
- Hits              6024     5225     -799
+ Misses            7383     5330    -2053
+ Partials           900      899       -1

Flag
Coverage Δ

#linux
45.61% <ø> (ø)
:arrow_up:

#windows
?

Impacted Files
Coverage Δ

remotes/docker/fetcher.go
42.5% <0%> (-7.5%)
:arrow_down:

remotes/docker/auth.go
63.82% <0%> (-3.97%)
:arrow_down:

remotes/docker/status.go
21.42% <0%> (-3.58%)
:arrow_down:

platforms/cpuinfo.go
3.77% <0%> (-0.85%)
:arrow_down:

log/context.go
36.84% <0%> (-0.66%)
:arrow_down:

remotes/docker/handler.go
22.38% <0%> (-0.09%)
:arrow_down:

errdefs/grpc.go
77.46% <0%> (-0.04%)
:arrow_down:

remotes/docker/pusher.go
0% <0%> (ø)
:arrow_up:

content/local/locks.go
100% <0%> (ø)
:arrow_up:

runtime/v2/binary.go
0% <0%> (ø)
:arrow_up:

... and 73 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 36cf5b6...6cbad87. Read the comment docs.

LGTM


fix: pass parameters as object

#62

Looks good to me. Let's test the changes for both storybook v5 and v6


Implement port monitoring for Containerd

Signed-off-by: Nino Kodabande<EMAIL_ADDRESS>

https://github.com/rancher-sandbox/rancher-desktop/issues/2677


docs: basic_stream::expires_after is confusing

basic_stream::expires_after states:

Set the timeout for the next logical operation.

The use of the definite article implies the timeout applies to a single operation.
But then it goes on to state:

The timer applies collectively to any asynchronous reads or writes...

So, which is it? singular or plural?

Should be plural. I'll review the details and update the docs.


Purpur 1.20.1 Build #2015 does not save the book if the player is not op

Spark link

Expected behavior
Purpur 1.20.1 Build #2015 does not save the book if the player is not op. checked with the assembly Paper 1.20.1 Build # 81 they save
Observed/Actual behavior
test: I first without OP then with OP https://www.youtube.com/watch?v=2Fw--WkeUMc
Steps/models to reproduce
this is Paper 1.20.1 Build #81 without OP  https://www.youtube.com/watch?v=CUcpMOsj8Jc
Purpur version
Purpur 1.20.1 Build #2015
Agreements

[X] I am running the latest version of Purpur available from https://purpurmc.org/downloads.
[X] I have searched for and ensured there isn't already an open issue regarding this.
[ ] I ticked all the boxes without actually reading them
[X] My version of Minecraft is supported by Purpur.

Other
I'm sorry if I didn't format the ticket correctly. but the problem really exists only on purpur

the books were disabled  pufferfish config ) sorry
enable-books: true


Adding the last_page flag.

Adding the last page flag for pagination feature

The flag added to optional_attributes.
The optional_attributes will be used for adding extra communication between front end and back end.
Fixing some pagination bug.

Merging this branch to develop. Please check and review on new branch (Youtube API)


pretrained model on Visual Genome

Hi, I use the code from another repository "pytorch-faster-rcnn" of you and employ the pretrained model on Mscoco dataset. The results seems not so good. I wanna if there is a pytorch-version pretrained model of faster-rcnn on Visual Genome? Thank you!

Search fast-rcnn. Pytorch

Are you looking at http://data.lip6.fr/cadene/faster-rcnn.pytorch/faster_rcnn_1_19_48611.pth ?? If this is not, then I don't know.

Thank you very much!!


Changed grey to gray in ui.js

Changed the word "grey" to "gray" in ui.js in order to comply with the docs.

Automated message from Dropbox CLA bot
@Jeffthebest1, it looks like you've already signed the Dropbox CLA. Thanks!

@Jeffthebest1 merged, thanks!  If you're doing a lot of these typo searches, I'd encourage you to do several commits fixing different groups of typos in a single PR, rather than making a new PR for each group of them; it'll save a bit of overhead :).

Thanks @timabbott ! :+1:


Update policy assignment resource provider version

Description
To ensure that policyAssignments is operational across all clouds
Issue reference
The issue this PR will close: #636
Checklist
Please make sure you've completed the relevant tasks for this PR out of the following list:

[x] All acceptance criteria in the backlog item are met
[x] The documentation is updated to cover any new or changed features
[x] Manual tests have passed
[x] Relevant issues are linked to this PR

/azp ru

/azp run


Deployment failure: Operation interrupted.

When I tried to deploy function.zip I got the following error message on Google Cloud Functions:

Deployment failure: Operation interrupted.

This is what I did:
At first, clone and build:
$ git clone https://github.com/GoogleCloudPlatform/cloud-functions-go.git && cd cloud-functions-go
$ make

Then uploaded to Google Cloud Functions using the GCP web UI.
The following error entry is in the logs:
{
 insertId:  "-uie6zkd1h36"
 logName:  "projects/my-project-id/logs/cloudaudit.googleapis.com%2Factivity"
 operation: {
  id:  "operations/aXBpbmZvLTE4MTYyMC91cy1jZW50cmFsMS9mdW5jdGlvbi0xL1YyWERCd1Z4ZGdr"
  last:  true
  producer:  "cloudfunctions.googleapis.com"
 }
 protoPayload: {
  @type:  "type.googleapis.com/google.cloud.audit.AuditLog"
  methodName:  "google.cloud.functions.v1.CloudFunctionsService.CreateFunction"
  resourceName:  "projects/my-project-id/locations/us-central1/functions/function-1"
  serviceName:  "cloudfunctions.googleapis.com"
  status: {
   code:  10
   message:  "Operation interrupted."
  }
 }
 receiveTimestamp:  "2018-02-19T12:27:44.296923939Z"
 resource: {
  labels: {
   function_name:  "function-1"
   project_id:  "my-project-id"
   region:  "us-central1"
  }
  type:  "cloud_function"
 }
 severity:  "ERROR"
 timestamp:  "2018-02-19T12:27:43.703Z"
}

Maybe try again? I just tested the steps you listed and it worked for me.

Yes, indeed, it worked. Probably it was some temporary GCP issue…

I've run into the same problem. It spins for 5+minutes and then I get the same error. Ive tried three times, always the same problem.
I use go1.10.1 darwin/amd64 and when I created the function in the console I picked execute as the function to execute. I guess that is the default from main.go in this repo?

And now it worked. Fourth time...

Do you have the most recent version? We did fix a flakiness issue. Post here if you experience it again.

yes, i pulled the repo just minutes before i tried.

I looked into it and I think that there is a reasonably high probability that it was a GCF issue. When the code in this project doesn't work correctly, typically deployment fails with an error about your function failing the heath check. I believe that the error described in this issue is a GCF internal error.
Next time this happens, try deploying a hello world JS function over the Go one and see if that works. If it works, try deploying the Go one again.

For the last couple of days this is happening consistently again. Ive tried the following:

using gcloud command with gs:// source url. Didn't work.
web console with zip upload. Didn't work.
web console with inline js (always works), then replace with my function.zip. Didn't work.

Always with the ERROR: (gcloud.beta.functions.deploy) OperationError: code=10, message=Operation interrupted. error.
Any ideas?

facing same error using node.js. Even with the sample google (default) functions, throwing same error since a few hours now.

I'm currently facing the same issue with a slightly different environment.
I get this error after creating a Firebase project, going to the Google Cloud Platform console and uploading a function generated using cloud-functions-go. I also attempted to deploy a Firebase function via the GCP console by creating a Firebase Cloud Function using firebase init and copying over files from cloud-functions-go. Attempts to (re-)deploy either functions fail.
@iangudger Given how @MrBhatt has coincidentally experienced this an hour ago, perhaps there may have been a backend configuration change leading to this?

Perhaps there may have been an outage; I just reattempted a deploy and both functions worked!
It's good to see that Firebase functions also work.

If it happens with pure JavaScript functions as well, it seems most likely that it is a problem with GCF.

Today I am trying this and experiencing this issue as well.
I tried many times, over 10 times, and the link here https://status.firebase.google.com/ shows all services running properly

Are you implying they made changes that are causing those endless "Operation interrupted" errors when trying to import a zip package ?

I tried to give a look in the #go and the #functions channels there, didn't see anything exciting,  but I don't have access to the full history (yet).
@kz  Care to tell a bit more what you've read there ? :)

Hey I am still having the problem. Any solution ?

Was experiencing this issue as well trying to deploy a Python 3.7 app.

First I made sure I had the most up-to-date SDK ($ gcloud components update).

Then I ran: $ gcloud functions deploy FUNC_NAME --runtime python37 --trigger-http --entry-point handle

Spun for about 10 minutes, then returned ERROR: (gcloud.functions.deploy) OperationError: code=10, message=Operation interrupted.

Thought maybe it was a temporary bug last night, so re-ran this morning and still not working.

I could see that it had tried to deploy by both (1) looking in the GCP console (the function was there but with an error notice) and (2) running:
$ gcloud functions list, which returned:
NAME       STATUS   TRIGGER       REGION
FUNC_NAME  UNKNOWN  HTTP Trigger  us-central1

I tried $ gcloud functions deploy FUNC_NAME ... --region us-east1 and it worked immediately.
Try changing the region?

@GopikrishnanSasikumar looks like there the same issues have cropped up with GCF, as I've just started being unable to deploy some functions for the operation interrupted reason too

Same problem

Please try again now.

yup better now, thanks 👍

Works now, thanks

FYI, you have to have a Google Cloud Storage (GCS) bucket created. Cloud Functions require a GCS bucket for staging. Without at least one available bucket, you will get that "unknown" error message

Same Problem in one of the GCP projects (have 2 projects, deployment works on one proj and fails on any CF deployment on another):
Function's deployment in GCP console stops after 5 minutes with
Deployment failure:
Operation interrupted.

error message. Stackdriver log says "ABORTED".


Eliah Kagan: Watch portal4fitness

Eliah Kagan requests the watch of the watch_keyword portal4fitness. See the MS search here and the Stack Exchange search here.

portal4fitness has been seen in 10 true positives, 0 false positives, and 0 NAAs.

This keyword appears in contexts where I don't think the watches on URLs and email addresses will always catch it, such as social media URLs.

[I also posted this in chat, so you may have seen it there first.]
For something like this, where you want to detect "portal4fitness" when "portal4fitness.com" is being separately detected, it needs to be done in a way such that what you add doesn't also detect "portal4fitness.com". This is because we try not to have the same exact thing detected by two reasons. Although there are a number of situations where that does happen, even intentionally, it's something we usually strive to avoid.
If we are wanting to detect both the domain and the name without the TLD, then we will normally want to use two separate entries, even if they are both on the watchlist. The reason for that is because domains/websites, eventually, get blacklisted on the blacklisted-websites.txt list, whereas keywords get blacklisted on the bad_keywords.txt list. Having them separate when they are watched makes it easier to blacklist them, when they qualify, which could be at separate times.
In the current case, portal4fitness\.com is already blacklisted (MS search), so we definitely don't want to add another detection that finds the domain name. However, as you noticed, the spammer has begun to use "portal4fitness" in some URLs not as the domain name. That's actually fairly common for spammers to do, and we do desire to catch them, just in case they are used in spam posts which don't have the domain.
To do so, you need to use a regex that detects "portal4fitness" but not "portal4fitness.com". You can do that with a negative lookahead. For example, portal4fitness(?!\.com) (MS search) would work in this case.
As I've mentioned, it's moderately common for spammers to start using their domain names in this way. As it turns out, they also tend to start mashing additional text on at the end of their domain name in those uses (commonly subdomains and URL paths). The most common additional text is numbers, either in decimal or hexadecimal. Because the watchlist and keyword blacklist bookend the pattern with \b, adding portal4fitness(?!\.com) won't catch "example.com/portal4fitness1/", which is a relatively common thing for spammers to use, if they start using their domain name in URL paths or subdomains. If it looks like a spammer is heading in that direction (most spammers don't, but the more prolific drug spammers tend to do so), I'll often use a bit more complex regex to catch such things. What I commonly do is add (?:[\W_]*+(?:\d++|[\da-f]{4,}+))* to the end of the regex. So, for this one, I'd use portal4fitness(?!\.com)(?:[\W_]*+(?:\d++|[\da-f]{4,}+))*


update initialize_mth5

[ ] simplify initialize_mth5
[ ]

Codecov Report
Base: 55.47% // Head: 55.31% // Decreases project coverage by -0.16% :warning:

Coverage data is based on head (46b03c4) compared to base (54f17ed).
Patch coverage: 75.00% of modified lines in pull request are covered.

:exclamation: Current head 46b03c4 differs from pull request most recent head 85b9a58. Consider uploading reports for the commit 85b9a58 to get more accurate results

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #126      +/-   ##
==========================================
- Coverage   55.47%   55.31%   -0.17%
==========================================
  Files         110      104       -6
  Lines       10523    10370     -153
==========================================
- Hits         5838     5736     -102
+ Misses       4685     4634      -51

Impacted Files
Coverage Δ

mth5/utils/helpers.py
61.53% <75.00%> (-4.09%)
:arrow_down:

mth5/__init__.py

mth5/io/lemi/__init__.py

mth5/io/phoenix/__init__.py

mth5/io/zen/__init__.py

mth5/io/__init__.py

mth5/helpers.py

mth5/io/zen/zen.py
20.78% <0.00%> (+0.04%)
:arrow_up:

...ix/readers/segmented/decimated_segmented_reader.py
35.89% <0.00%> (+0.30%)
:arrow_up:

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


WIP: Upgrade nodejs versions

This was first attempted in #1652 but had to be reverted in #1688, as the Node version was incompatible with the Terraform version.
This PR is a second attempt at upgrading the NodeJS versions.

Turns out bump to TF 1.1 is not needed. Superseded by #1698.


Sidebar webview view sometimes doesn't show the html that is set

I was investigating https://github.com/microsoft/vscode-pull-request-github/issues/3199, and it looks like a VS Code issue, not an extension issue. The steps in that issue repro ~30% of the time for me.
Code wise, here's what's happening in the extension:

The extension's resolveWebviewView gets called
We set the html of the Webview
At this point, the webview is somehow empty.

I added a log point in $setHtml in MainThreadWebviews, and I've confirmed that the html is at least making it that far. When I inspect the element using the devtools, the html is not there:

Any ideas about what might be going on here? It looks like the iframe is just missing entirely.

Are there any other logs/errors in the dev tools when this happens?

No, nothing in the dev tools.

Are you still seeing this issue in current insiders builds?

No, it seems to be fixed. Thank you!


update to frontend list item markers

change stripped-marker instead of paragraph-marker

Starting to look really nice!
Looks like the collapsed paragraph isn't quite working yet, though


unable to execute /start-minecraft: Text file busy?

When running the latest itzg/minecraft-server image (Image ID: 61f744a5eaa6) on VirtualBox, I get the following error.
$ docker images itzg/minecraft-server
REPOSITORY              TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
itzg/minecraft-server   latest              61f744a5eaa6        2 hours ago         399.3 MB

$ docker run --detach --interactive --tty \
>   --name minecraft-whitelist \
>   -e VERSION=LATEST \
>   -e OPS=user1 \
>   -e WHITELIST=user1,user2,user3,user4 \
>   -e EULA=TRUE \
>   -e MOTD="VirtualBox World" \
>   -p 25565:25565 \
>   itzg/minecraft-server
e997fef2bb9b684e1220eeff716d1277255082f56363170fdefdf110da843b05

$ docker ps -a
CONTAINER ID        IMAGE                   COMMAND             CREATED             STATUS                      PORTS               NAMES
e997fef2bb9b        itzg/minecraft-server   "/start"            14 seconds ago      Exited (1) 14 seconds ago                       minecraft-whitelist

$ docker logs minecraft-whitelist
usermod: no changes
sudo: unable to execute /start-minecraft: Text file busy

However if I run it in another environment, such as Carina. It works.
$ docker images itzg/minecraft-server
REPOSITORY              TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
itzg/minecraft-server   latest              61f744a5eaa6        2 hours ago         399.3 MB

$ docker run --detach --interactive --tty \
>   --name minecraft-whitelist \
>   -e VERSION=LATEST \
>   -e OPS=user1 \
>   -e WHITELIST=user1,user2,user3,user4 \
>   -e EULA=TRUE \
>   -e MOTD="Carina World" \
>   -p 25565:25565 \
>   itzg/minecraft-server
8aaf17d6bd83c101c3e8ac6989de95247df59049df02e244e5e92041643ce64d

$ docker ps
CONTAINER ID        IMAGE                   COMMAND             CREATED             STATUS              PORTS                           NAMES
8aaf17d6bd83        itzg/minecraft-server   "/start"            28 seconds ago      Up 27 seconds       xxx.xxx.xxx.xxx:25565->25565/tcp   0dc4a907-960c-4c1b-9f07-cd9071fb87b1-n1/minecraft-whitelist

$ docker attach minecraft-whitelist
/whitelist list
[17:44:59] [Server thread/INFO]: There are 4 (out of 0 seen) whitelisted players:
[17:44:59] [Server thread/INFO]: user1, user2, user3 and user4

@itzg Can you please try it locally/VirtualBox?
I'd hate to think I borked your image with https://github.com/itzg/dockerfiles/pull/46. If it works for you, feel free to just close this.

Yep, I'll try in Linux and Windows (VirtualBox) and see what it does for me. It's very unlikely to be our last round of changes unless I did something to line endings on commit.

...and thanks for the pointer to Carina. I've been on the lookout for good container hosting.

Test 1, success, Ubuntu 15.10, local
 $ docker version                                                                                [12:53:50]
Client:
 Version:      1.8.3
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   f4bf5c7
 Built:        Mon Oct 12 05:39:44 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.8.3
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   f4bf5c7
 Built:        Mon Oct 12 05:39:44 UTC 2015
 OS/Arch:      linux/amd64

 $ docker images itzg/minecraft-server                                                                       [12:53:44]
REPOSITORY              TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
itzg/minecraft-server   latest              61f744a5eaa6        3 hours ago         399.3 MB

...and success with tests 2, Windows/VirtualBox, and 3, Carina. Carina is super nice BTW. My son and I got up and running in just a few minutes.
Try exploring a "temp/disposable" instance using:
docker run --rm -it -p 25565:25565 itzg/minecraft-server bash

and then confirm the presence and access to /start and /start-minecraft. From there you could try running starting from there with
EULA=TRUE /start-minecraft

Hold the phone, I just now hit this once during several runs on Windows/VirtualBox. I'm thinking about introducing a re-try loop in /start...

Alright. Let me know how it goes. I'm happy to do some testing.

I introduced an lsof check in the branch https://github.com/itzg/dockerfiles/tree/mc-issue-47 and so far I haven't seen the issue. Can you test that out too?

Just tried it.
Somehow the whitelist stuff disappeared from https://github.com/itzg/dockerfiles/blob/mc-issue-47/minecraft-server/start-minecraft.sh

Oops, apparently I hadn't pulled the latest master. Now I have merged and pushed the branch with the latest.

:+1:
I tried the new image and it worked both locally and on Carina. Thanks!

Thanks. It's now merged and pushed https://hub.docker.com/r/itzg/minecraft-server/builds/bq4ub7hnwtkptmbrvgadk5/


[DOC] Load balancer ports

I find the comments contradicting regarding - 80 and - 9999/tcp.
When using a single port number, do you get a random public port or not?
Or worse, only when you do not use the /tcp suffix.
http://docs.rancher.com/rancher/rancher-compose/rancher-services/#load-balancer-example-l4

It's a doc discrepancy, which I have now fixed.
The 9999/tcp part wasn't updated for random ports.


Istio control plane performance problems

Describe the bug
We do some performance tests on istio-1.0.0 control plane recently. The result indicate some of the problems:

When we have a large scale of services and pods, the pilot will consume a very large Memory (2500 pods need at least 30GB).
Mixer CPU will cost over 40 cores when under 100,000 QPS. With the rapid increase of the cost, the process will soon use up the VM Memory, and the pod will be killed by kubelet.
When we use multiple mixer pods. The share of CPU is not balanced.

Any one meet with the same issue or have any ideas on solving those problems?
Steps to reproduce the bug
All the resources of data and control plane are enough, the test application is tomcat.
Version
kubernetes-1.11.3, istio-1.0.0

@HWPeiyao I think you'll likely get more feedback on the discussion forum.  Can you open a thread on discuss.istio.io. The perf WG is the right place to raise these questions.
I'm going to close this issue for now.


Update main.js

I can see that you changed main.js and tried different colors. Keep experimenting and learning! The HW is incomplete however, because the task was to draw a rectangle. Score 2/5 for learning how to submit pull requests.

I didn't submit anything yet sir? I did try to make the rectangle multiple
times but with absolutely no success unfortunately.
On Thu, Feb 1, 2024 at 1:58 PM YArazov @.***> wrote:

I can see that you changed main.js and tried different colors. Keep
experimenting and learning! The HW is incomplete however, because the task
was to draw a rectangle. Score 2/5 for learning how to submit pull requests.
—
Reply to this email directly, view it on GitHub
https://github.com/YArazov/JavaScript-Engine-01/pull/3#issuecomment-1920944247,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/BFTLSU64ZAW2BLVNAHFFXJTYRNRNZAVCNFSM6AAAAABCK4FYW6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMRQHE2DIMRUG4
.
You are receiving this because you authored the thread.Message ID:
@.***>


Update MD file on new branch 2017-18-08_15-11-25

:x: Validation status: errors

File
Status
Preview URL
Details

:x:Error

Details

Line 16: [Error] Restore nuget package W:\mnq0-s.optemp\packages\opbuild.scripts.1.41.0-alpha-778-g9855cfb\tools\packages.config in folder:W:\mnq0-s.optemp\packages failed, possibly due to transient errors from nuget servers. Please retry building your content. If the issue still happens, open a ticket in http://SiteHelp. Detailed error: Unable to read package from path 'opbuild.filemappreprocessor.1.41.0-alpha-778-g9855cfb.nupkg'.

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Can we have more styling options?

Hi! I really like the idea of this plugin and I think I will use it often. :)
Could you add (in the future, not necessary now) more configuration options for styling the minimap window? You know, some basic staff like highlight groups, borders etc. ? Basically, things that go into nvim_win_set_option and nvim_open_win.
Thanks in advance!

Thanks for the suggestion, I'll look into it tomorrow.

Hi @rafcamlet , I've implemented it in 0886799fe9fd5e881a0c35435d9b2a5fba83c99e. Please tell me your experience using it. Thank you.

Thanks! Works for me. 👍
But you could consider preparing more user-friendly configuration settings. I wouldn't assume that all neovim users are familiar with its api.

Hi. Which default settings do you think we should implement?

Hmm... We're currently using winhl = "Normal:", is it good enough? :thinking:

Toggle feature for removing leading spaces
I thought about that feature, however I didn't implement it due to the "loss of indentation structure", I mean it'll be harder for the user to recognize the structure and context of the line.

Gotcha. I'll probably add option to disable indentation soon.

Hi @rafcamlet , you can now set disable_indentation option to true if you want to remove those white spaces or tab.
Implemented in bcd353e29d13745dd9c1e834a32bc42f4945db6b


Dear Sir ,we need System.ComponentModel.DataAnnotations It's too important

Dear Sir ,we need System.ComponentModel.DataAnnotations It's too important

please vote for it

	Thank you for your feedback. This is in the pipeline and I cannot give a timeline of when this will be delivered at the moment.

Please feel free to fork the code and send up a pull request if possible to speed things up.
	Get Outlook for iOS

On Sat, Jun 3, 2017 at 2:56 AM +0200, "564064202"<EMAIL_ADDRESS>wrote:
please vote for it
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.


Update asset_management chaincode to new interface.

Signed-off-by: Stanislav Liberman<EMAIL_ADDRESS>

LGTM. Thanks!

thanks, @stanliberman