HADOOP-13230. S3A to optionally retain directory markers

HADOOP-13230
successor to #1861

testing against s3 ireland; not run anything other than the (now expanded) cost tests

tested s3 ireland with params -Dparallel-tests -DtestsThreadCount=8 -Dscale -Dmarkers=keep -Ds3guard -Ddynamo
one failure
[ERROR] Tests run: 24, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 369.68 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3GuardOutOfBandOperations
[ERROR] testListingDelete[auth=false](org.apache.hadoop.fs.s3a.ITestS3GuardOutOfBandOperations)  Time elapsed: 19.103 s  <<< ERROR!
java.util.concurrent.ExecutionException: java.io.FileNotFoundException: No such file or directory: s3a://stevel-ireland/fork-0001/test/dir-e34a122f-a04d-48c3-90c1-9d35427fa939/file-1-e34a122f-a04d-48c3-90c1-9d35427fa939
	at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
	at org.apache.hadoop.fs.s3a.ITestS3GuardOutOfBandOperations.expectExceptionWhenReadingOpenFileAPI(ITestS3GuardOutOfBandOperations.java:1069)
	at org.apache.hadoop.fs.s3a.ITestS3GuardOutOfBandOperations.expectExceptionWhenReadingOpenFileAPI(ITestS3GuardOutOfBandOperations.java:1046)
	at org.apache.hadoop.fs.s3a.ITestS3GuardOutOfBandOperations.testListingDelete(ITestS3GuardOutOfBandOperations.java:1007)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.FileNotFoundException: No such file or directory: s3a://stevel-ireland/fork-0001/test/dir-e34a122f-a04d-48c3-90c1-9d35427fa939/file-1-e34a122f-a04d-48c3-90c1-9d35427fa939
	at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3131)
	at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:2971)
	at org.apache.hadoop.fs.s3a.S3AFileSystem.extractOrFetchSimpleFileStatus(S3AFileSystem.java:4764)
	at org.apache.hadoop.fs.s3a.S3AFileSystem.open(S3AFileSystem.java:1111)
	at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$null$22(S3AFileSystem.java:4839)
	at org.apache.hadoop.util.LambdaUtils.eval(LambdaUtils.java:52)
	at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$openFileWithOptions$23(S3AFileSystem.java:4838)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	... 1 more

cause is when status == null is passed in the failure happens is raised in the .get() on the future, but the operation expects it to fail in the first read(), which is only true with a file status is supplied)
this test path is only followed on an unversioned bucket, presumably that's what my test setup now is, and why it always worked before.

...though it worked from the IDE

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
37m 26s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  1s
No case conflicting files found.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s
The patch appears to include 17 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
1m 14s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
25m 36s
trunk passed

+1 :green_heart:
compile
24m 18s
trunk passed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04

+1 :green_heart:
compile
20m 59s
trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09

+1 :green_heart:
checkstyle
3m 21s
trunk passed

+1 :green_heart:
mvnsite
2m 27s
trunk passed

+1 :green_heart:
shadedclient
23m  5s
branch has no errors when building and testing our client artifacts.

-1 :x:
javadoc
0m 39s
hadoop-common in trunk failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.

-1 :x:
javadoc
0m 41s
hadoop-aws in trunk failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.

+1 :green_heart:
javadoc
1m 33s
trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09

+0 :ok:
spotbugs
1m 19s
Used deprecated FindBugs config; considering switching to SpotBugs.

+1 :green_heart:
findbugs
3m 43s
trunk passed

-0 :warning:
patch
1m 40s
Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 27s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
1m 37s
the patch passed

+1 :green_heart:
compile
24m  7s
the patch passed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04

-1 :x:
javac
24m  7s
root-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04 with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04 generated 4 new + 2047 unchanged - 4 fixed = 2051 total (was 2051)

+1 :green_heart:
compile
20m 21s
the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09

-1 :x:
javac
20m 21s
root-jdkPrivateBuild-1.8.0_252-8u252-b09-1~18.04-b09 with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 generated 4 new + 1939 unchanged - 4 fixed = 1943 total (was 1943)

-0 :warning:
checkstyle
3m 19s
root: The patch generated 53 new + 63 unchanged - 1 fixed = 116 total (was 64)

+1 :green_heart:
mvnsite
2m 38s
the patch passed

+1 :green_heart:
whitespace
0m  0s
The patch has no whitespace issues.

+1 :green_heart:
xml
0m  1s
The patch has no ill-formed XML file.

+1 :green_heart:
shadedclient
17m 14s
patch has no errors when building and testing our client artifacts.

-1 :x:
javadoc
0m 40s
hadoop-common in the patch failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.

-1 :x:
javadoc
0m 39s
hadoop-aws in the patch failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.

+1 :green_heart:
javadoc
1m 36s
the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09

-1 :x:
findbugs
1m 32s
hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)

_ Other Tests _

+1 :green_heart:
unit
10m 37s
hadoop-common in the patch passed.

+1 :green_heart:
unit
1m 28s
hadoop-aws in the patch passed.

+1 :green_heart:
asflicense
0m 48s
The patch does not generate ASF License warnings.

233m 41s

Reason
Tests

FindBugs
module:hadoop-tools/hadoop-aws

Dead store to leafMarkers in org.apache.hadoop.fs.s3a.tools.MarkerTool.scan(Path, boolean, int, boolean, StoreContext, OperationCallbacks)  At MarkerTool.java:org.apache.hadoop.fs.s3a.tools.MarkerTool.scan(Path, boolean, int, boolean, StoreContext, OperationCallbacks)  At MarkerTool.java:[line 187]

Subsystem
Report/Notes

Docker
ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/2149

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml

uname
Linux f140ee59064a 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 1f71c4ae714

Default Java
Private Build-1.8.0_252-8u252-b09-1~18.04-b09

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_252-8u252-b09-1~18.04-b09

javadoc
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/1/artifact/out/branch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt

javadoc
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/1/artifact/out/branch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt

javac
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/1/artifact/out/diff-compile-javac-root-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt

javac
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/1/artifact/out/diff-compile-javac-root-jdkPrivateBuild-1.8.0_252-8u252-b09-1~18.04-b09.txt

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/1/artifact/out/diff-checkstyle-root.txt

javadoc
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/1/artifact/out/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt

javadoc
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/1/artifact/out/patch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt

findbugs
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/1/artifact/out/new-findbugs-hadoop-tools_hadoop-aws.html

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/1/testReport/

Max. process+thread count
1465 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/1/console

versions
git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

latest update has marker tool with tests, no docs. We are going to need a full page on directory markers and compatibility, aren't we?
Tests in progress, failure on
[ERROR] testProxyConnection(org.apache.hadoop.fs.s3a.ITestS3AConfiguration)  Time elapsed: 1.761 s  <<< FAILURE!
java.lang.AssertionError: Expected a org.apache.hadoop.fs.s3a.AWSClientIOException to be thrown, but got the result: : "expected failure creating FS when using proxy <IP_ADDRESS>:1 got S3AFileSystem{uri=s3a://stevel-ireland, workingDir=s3a://stevel-ireland/user/stevel,...
	at org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:499)
	at org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:384)
	at org.apache.hadoop.fs.s3a.ITestS3AConfiguration.expectFSCreateFailure(ITestS3AConfiguration.java:159)
	at org.apache.hadoop.fs.s3a.ITestS3AConfiguration.testProxyConnection(ITestS3AConfiguration.java:134)

Seen this before -assume cause is that I've turned off bucket checks. Really test should combine a create and a list / and expect a failure in either place

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
1m 22s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  1s
No case conflicting files found.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s
The patch appears to include 18 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
0m 30s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
21m 40s
trunk passed

+1 :green_heart:
compile
21m  8s
trunk passed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04

+1 :green_heart:
compile
17m 47s
trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09

+1 :green_heart:
checkstyle
2m 52s
trunk passed

+1 :green_heart:
mvnsite
2m 10s
trunk passed

+1 :green_heart:
shadedclient
21m  4s
branch has no errors when building and testing our client artifacts.

-1 :x:
javadoc
0m 35s
hadoop-common in trunk failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.

-1 :x:
javadoc
0m 37s
hadoop-aws in trunk failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.

+1 :green_heart:
javadoc
1m 27s
trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09

+0 :ok:
spotbugs
1m 11s
Used deprecated FindBugs config; considering switching to SpotBugs.

+1 :green_heart:
findbugs
3m 27s
trunk passed

-0 :warning:
patch
1m 31s
Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 25s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
1m 28s
the patch passed

+1 :green_heart:
compile
26m 24s
the patch passed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04

-1 :x:
javac
26m 24s
root-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04 with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04 generated 4 new + 2047 unchanged - 4 fixed = 2051 total (was 2051)

+1 :green_heart:
compile
27m 25s
the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09

-1 :x:
javac
27m 25s
root-jdkPrivateBuild-1.8.0_252-8u252-b09-1~18.04-b09 with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 generated 4 new + 1939 unchanged - 4 fixed = 1943 total (was 1943)

-0 :warning:
checkstyle
3m 52s
root: The patch generated 71 new + 63 unchanged - 1 fixed = 134 total (was 64)

+1 :green_heart:
mvnsite
2m 54s
the patch passed

+1 :green_heart:
whitespace
0m  0s
The patch has no whitespace issues.

+1 :green_heart:
xml
0m  2s
The patch has no ill-formed XML file.

+1 :green_heart:
shadedclient
19m  6s
patch has no errors when building and testing our client artifacts.

-1 :x:
javadoc
0m 56s
hadoop-common in the patch failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.

-1 :x:
javadoc
0m 52s
hadoop-aws in the patch failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.

+1 :green_heart:
javadoc
2m  1s
the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09

+1 :green_heart:
findbugs
5m  3s
the patch passed

_ Other Tests _

-1 :x:
unit
11m 45s
hadoop-common in the patch passed.

+1 :green_heart:
unit
1m 49s
hadoop-aws in the patch passed.

+1 :green_heart:
asflicense
1m 18s
The patch does not generate ASF License warnings.

200m 25s

Reason
Tests

Failed junit tests
hadoop.metrics2.source.TestJvmMetrics

Subsystem
Report/Notes

Docker
ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/2/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/2149

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml

uname
Linux 6f8e550091a0 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / d5b47661582

Default Java
Private Build-1.8.0_252-8u252-b09-1~18.04-b09

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_252-8u252-b09-1~18.04-b09

javadoc
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/2/artifact/out/branch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt

javadoc
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/2/artifact/out/branch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt

javac
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/2/artifact/out/diff-compile-javac-root-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt

javac
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/2/artifact/out/diff-compile-javac-root-jdkPrivateBuild-1.8.0_252-8u252-b09-1~18.04-b09.txt

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/2/artifact/out/diff-checkstyle-root.txt

javadoc
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/2/artifact/out/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt

javadoc
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/2/artifact/out/patch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/2/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/2/testReport/

Max. process+thread count
1482 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/2/console

versions
git=2.17.1 maven=3.6.0 findbugs=4.0.6

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

see #2170 for the <= 3.2 subset of this patch, needed to not mistake paths with dir markers as empty.
As well as not deleting markers on file creation, I'm going to add the option of not doing any re-creation of markers when files are deleted.
This reduces IO on file deletion at the cost of a common assumption: "after you delete all files in a directory -the directory still exists".
I don't know what that's going to break, so it is going to be explicitly optional, with the recovery policy to be true/false/authoritative

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
37m 10s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s
No case conflicting files found.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s
The patch appears to include 19 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
1m 23s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
24m 42s
trunk passed

+1 :green_heart:
compile
25m 36s
trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1

+1 :green_heart:
compile
21m 12s
trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09

+1 :green_heart:
checkstyle
3m 18s
trunk passed

+1 :green_heart:
mvnsite
2m 22s
trunk passed

+1 :green_heart:
shadedclient
22m 41s
branch has no errors when building and testing our client artifacts.

-1 :x:
javadoc
0m 38s
hadoop-common in trunk failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.

-1 :x:
javadoc
0m 38s
hadoop-aws in trunk failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.

+1 :green_heart:
javadoc
1m 33s
trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09

+0 :ok:
spotbugs
1m 21s
Used deprecated FindBugs config; considering switching to SpotBugs.

+1 :green_heart:
findbugs
3m 55s
trunk passed

-0 :warning:
patch
1m 41s
Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 27s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
1m 43s
the patch passed

+1 :green_heart:
compile
24m 19s
the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1

-1 :x:
javac
24m 19s
root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 generated 4 new + 2047 unchanged - 4 fixed = 2051 total (was 2051)

+1 :green_heart:
compile
20m 22s
the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09

-1 :x:
javac
20m 22s
root-jdkPrivateBuild-1.8.0_252-8u252-b09-1~18.04-b09 with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 generated 4 new + 1939 unchanged - 4 fixed = 1943 total (was 1943)

-0 :warning:
checkstyle
3m 14s
root: The patch generated 86 new + 63 unchanged - 1 fixed = 149 total (was 64)

+1 :green_heart:
mvnsite
2m 27s
the patch passed

+1 :green_heart:
whitespace
0m  0s
The patch has no whitespace issues.

+1 :green_heart:
xml
0m  3s
The patch has no ill-formed XML file.

+1 :green_heart:
shadedclient
17m  1s
patch has no errors when building and testing our client artifacts.

-1 :x:
javadoc
0m 38s
hadoop-common in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.

-1 :x:
javadoc
0m 44s
hadoop-aws in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.

+1 :green_heart:
javadoc
1m 40s
the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09

+1 :green_heart:
findbugs
4m 24s
the patch passed

_ Other Tests _

+1 :green_heart:
unit
10m 54s
hadoop-common in the patch passed.

+1 :green_heart:
unit
1m 27s
hadoop-aws in the patch passed.

+1 :green_heart:
asflicense
0m 48s
The patch does not generate ASF License warnings.

234m 40s

Subsystem
Report/Notes

Docker
ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/3/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/2149

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml

uname
Linux a96c49475549 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 60a254621a3

Default Java
Private Build-1.8.0_252-8u252-b09-1~18.04-b09

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_252-8u252-b09-1~18.04-b09

javadoc
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/3/artifact/out/branch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt

javadoc
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/3/artifact/out/branch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt

javac
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/3/artifact/out/diff-compile-javac-root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt

javac
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/3/artifact/out/diff-compile-javac-root-jdkPrivateBuild-1.8.0_252-8u252-b09-1~18.04-b09.txt

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/3/artifact/out/diff-checkstyle-root.txt

javadoc
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/3/artifact/out/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt

javadoc
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/3/artifact/out/patch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/3/testReport/

Max. process+thread count
3296 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/3/console

versions
git=2.17.1 maven=3.6.0 findbugs=4.0.6

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
1m 25s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  1s
No case conflicting files found.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s
The patch appears to include 19 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
0m 21s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
21m 39s
trunk passed

+1 :green_heart:
compile
20m 53s
trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1

+1 :green_heart:
compile
17m 51s
trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09

+1 :green_heart:
checkstyle
2m 51s
trunk passed

+1 :green_heart:
mvnsite
2m 12s
trunk passed

+1 :green_heart:
shadedclient
21m  9s
branch has no errors when building and testing our client artifacts.

-1 :x:
javadoc
0m 36s
hadoop-common in trunk failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.

-1 :x:
javadoc
0m 37s
hadoop-aws in trunk failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.

+1 :green_heart:
javadoc
1m 28s
trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09

+0 :ok:
spotbugs
1m  9s
Used deprecated FindBugs config; considering switching to SpotBugs.

+1 :green_heart:
findbugs
3m 21s
trunk passed

-0 :warning:
patch
1m 29s
Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 22s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
1m 24s
the patch passed

+1 :green_heart:
compile
20m 10s
the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1

-1 :x:
javac
20m 10s
root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 generated 4 new + 2047 unchanged - 4 fixed = 2051 total (was 2051)

+1 :green_heart:
compile
19m 38s
the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09

-1 :x:
javac
19m 38s
root-jdkPrivateBuild-1.8.0_252-8u252-b09-1~18.04-b09 with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 generated 4 new + 1939 unchanged - 4 fixed = 1943 total (was 1943)

-0 :warning:
checkstyle
3m 23s
root: The patch generated 90 new + 63 unchanged - 1 fixed = 153 total (was 64)

+1 :green_heart:
mvnsite
2m 37s
the patch passed

+1 :green_heart:
whitespace
0m  0s
The patch has no whitespace issues.

+1 :green_heart:
xml
0m  2s
The patch has no ill-formed XML file.

+1 :green_heart:
shadedclient
17m 36s
patch has no errors when building and testing our client artifacts.

-1 :x:
javadoc
0m 48s
hadoop-common in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.

-1 :x:
javadoc
1m  2s
hadoop-aws in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.

+1 :green_heart:
javadoc
1m 50s
the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09

+1 :green_heart:
findbugs
4m 27s
the patch passed

_ Other Tests _

-1 :x:
unit
10m 51s
hadoop-common in the patch passed.

+1 :green_heart:
unit
1m 48s
hadoop-aws in the patch passed.

+1 :green_heart:
asflicense
1m 36s
The patch does not generate ASF License warnings.

182m 13s

Reason
Tests

Failed junit tests
hadoop.metrics2.source.TestJvmMetrics

Subsystem
Report/Notes

Docker
ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/4/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/2149

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml

uname
Linux 9e9d553d40d8 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 026dce5334b

Default Java
Private Build-1.8.0_252-8u252-b09-1~18.04-b09

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_252-8u252-b09-1~18.04-b09

javadoc
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/4/artifact/out/branch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt

javadoc
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/4/artifact/out/branch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt

javac
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/4/artifact/out/diff-compile-javac-root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt

javac
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/4/artifact/out/diff-compile-javac-root-jdkPrivateBuild-1.8.0_252-8u252-b09-1~18.04-b09.txt

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/4/artifact/out/diff-checkstyle-root.txt

javadoc
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/4/artifact/out/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt

javadoc
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/4/artifact/out/patch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/4/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/4/testReport/

Max. process+thread count
1474 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/4/console

versions
git=2.17.1 maven=3.6.0 findbugs=4.0.6

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
1m 11s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s
No case conflicting files found.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s
The patch appears to include 19 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
0m 25s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
22m  2s
trunk passed

+1 :green_heart:
compile
21m  3s
trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1

+1 :green_heart:
compile
17m 42s
trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09

+1 :green_heart:
checkstyle
2m 54s
trunk passed

+1 :green_heart:
mvnsite
2m 11s
trunk passed

+1 :green_heart:
shadedclient
20m 54s
branch has no errors when building and testing our client artifacts.

-1 :x:
javadoc
0m 36s
hadoop-common in trunk failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.

-1 :x:
javadoc
0m 40s
hadoop-aws in trunk failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.

+1 :green_heart:
javadoc
1m 30s
trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09

+0 :ok:
spotbugs
1m  9s
Used deprecated FindBugs config; considering switching to SpotBugs.

+1 :green_heart:
findbugs
3m 18s
trunk passed

-0 :warning:
patch
1m 29s
Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 24s
Maven dependency ordering for patch

-1 :x:
mvninstall
0m 26s
hadoop-aws in the patch failed.

-1 :x:
compile
19m 16s
root in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.

-1 :x:
javac
19m 16s
root in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.

-1 :x:
compile
16m 58s
root in the patch failed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09.

-1 :x:
javac
16m 58s
root in the patch failed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09.

-0 :warning:
checkstyle
2m 51s
root: The patch generated 24 new + 63 unchanged - 1 fixed = 87 total (was 64)

-1 :x:
mvnsite
0m 42s
hadoop-aws in the patch failed.

+1 :green_heart:
whitespace
0m  1s
The patch has no whitespace issues.

+1 :green_heart:
xml
0m  1s
The patch has no ill-formed XML file.

+1 :green_heart:
shadedclient
15m 33s
patch has no errors when building and testing our client artifacts.

-1 :x:
javadoc
0m 37s
hadoop-common in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.

-1 :x:
javadoc
0m 36s
hadoop-aws in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.

+1 :green_heart:
javadoc
1m 27s
the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09

-1 :x:
findbugs
0m 40s
hadoop-aws in the patch failed.

_ Other Tests _

+1 :green_heart:
unit
9m 38s
hadoop-common in the patch passed.

-1 :x:
unit
0m 41s
hadoop-aws in the patch failed.

+1 :green_heart:
asflicense
0m 50s
The patch does not generate ASF License warnings.

169m 23s

Subsystem
Report/Notes

Docker
ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/5/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/2149

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml

uname
Linux 2381fca69ed5 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 5dadf963d36

Default Java
Private Build-1.8.0_252-8u252-b09-1~18.04-b09

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_252-8u252-b09-1~18.04-b09

javadoc
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/5/artifact/out/branch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt

javadoc
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/5/artifact/out/branch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt

mvninstall
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/5/artifact/out/patch-mvninstall-hadoop-tools_hadoop-aws.txt

compile
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/5/artifact/out/patch-compile-root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt

javac
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/5/artifact/out/patch-compile-root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt

compile
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/5/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_252-8u252-b09-1~18.04-b09.txt

javac
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/5/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_252-8u252-b09-1~18.04-b09.txt

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/5/artifact/out/diff-checkstyle-root.txt

mvnsite
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/5/artifact/out/patch-mvnsite-hadoop-tools_hadoop-aws.txt

javadoc
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/5/artifact/out/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt

javadoc
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/5/artifact/out/patch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt

findbugs
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/5/artifact/out/patch-findbugs-hadoop-tools_hadoop-aws.txt

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/5/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/5/testReport/

Max. process+thread count
1664 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/5/console

versions
git=2.17.1 maven=3.6.0 findbugs=4.0.6

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
2m 19s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s
No case conflicting files found.

+0 :ok:
markdownlint
0m  3s
markdownlint was not available.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s
The patch appears to include 19 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
1m 25s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
22m 45s
trunk passed

+1 :green_heart:
compile
21m  4s
trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1

+1 :green_heart:
compile
17m 42s
trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09

+1 :green_heart:
checkstyle
2m 50s
trunk passed

+1 :green_heart:
mvnsite
2m 10s
trunk passed

+1 :green_heart:
shadedclient
21m 20s
branch has no errors when building and testing our client artifacts.

-1 :x:
javadoc
0m 38s
hadoop-common in trunk failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.

-1 :x:
javadoc
0m 36s
hadoop-aws in trunk failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.

+1 :green_heart:
javadoc
1m 30s
trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09

+0 :ok:
spotbugs
1m 12s
Used deprecated FindBugs config; considering switching to SpotBugs.

+1 :green_heart:
findbugs
3m 23s
trunk passed

-0 :warning:
patch
1m 31s
Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 21s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
1m 26s
the patch passed

+1 :green_heart:
compile
20m  9s
the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1

-1 :x:
javac
20m  9s
root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 generated 4 new + 2047 unchanged - 4 fixed = 2051 total (was 2051)

+1 :green_heart:
compile
17m 46s
the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09

-1 :x:
javac
17m 46s
root-jdkPrivateBuild-1.8.0_252-8u252-b09-1~18.04-b09 with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 generated 4 new + 1939 unchanged - 4 fixed = 1943 total (was 1943)

-0 :warning:
checkstyle
2m 55s
root: The patch generated 26 new + 63 unchanged - 1 fixed = 89 total (was 64)

+1 :green_heart:
mvnsite
2m 11s
the patch passed

-1 :x:
whitespace
0m  0s
The patch has 7 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply

+1 :green_heart:
xml
0m  1s
The patch has no ill-formed XML file.

+1 :green_heart:
shadedclient
15m 52s
patch has no errors when building and testing our client artifacts.

-1 :x:
javadoc
0m 38s
hadoop-common in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.

-1 :x:
javadoc
0m 37s
hadoop-aws in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.

+1 :green_heart:
javadoc
1m 28s
the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09

+1 :green_heart:
findbugs
3m 42s
the patch passed

_ Other Tests _

+1 :green_heart:
unit
10m  3s
hadoop-common in the patch passed.

+1 :green_heart:
unit
1m 23s
hadoop-aws in the patch passed.

+1 :green_heart:
asflicense
0m 55s
The patch does not generate ASF License warnings.

177m  9s

Subsystem
Report/Notes

Docker
ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/7/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/2149

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml markdownlint

uname
Linux d2501009b522 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 5dadf963d36

Default Java
Private Build-1.8.0_252-8u252-b09-1~18.04-b09

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_252-8u252-b09-1~18.04-b09

javadoc
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/7/artifact/out/branch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt

javadoc
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/7/artifact/out/branch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt

javac
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/7/artifact/out/diff-compile-javac-root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt

javac
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/7/artifact/out/diff-compile-javac-root-jdkPrivateBuild-1.8.0_252-8u252-b09-1~18.04-b09.txt

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/7/artifact/out/diff-checkstyle-root.txt

whitespace
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/7/artifact/out/whitespace-eol.txt

javadoc
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/7/artifact/out/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt

javadoc
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/7/artifact/out/patch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/7/testReport/

Max. process+thread count
3294 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/7/console

versions
git=2.17.1 maven=3.6.0 findbugs=4.0.6

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

ITestMarkerTool#testAuthPathIsMixed reproducibly fails. on Tokyo region.
$ mvn verify -Dtest=x -Dit.test=ITestMarkerTool
[INFO] Running org.apache.hadoop.fs.s3a.tools.ITestMarkerTool
[ERROR] Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 25.962 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.tools.ITestMarkerTool
[ERROR] testAuthPathIsMixed(org.apache.hadoop.fs.s3a.tools.ITestMarkerTool)  Time elapsed: 11.139 s  <<< ERROR!
org.apache.hadoop.fs.s3a.RemoteFileChangedException: copyFile(test/testAuthPathIsMixed/source/dir2/empty/, test/testAuthPathIsMixed/dest/dir2/empty/) `s3a://iwasakims-test/test/testAuthPathIsMixed/source/dir2/empty': File to rename not found on unguarded S3 store: copyFile(test/testAuthPathIsMixed/source/dir2/empty/, test/testAuthPathIsMixed/dest/dir2/empty/) on test/testAuthPathIsMixed/source/dir2/empty/: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: FCCC5CB9EB1CE3D5; S3 Extended Request ID: ykAUAu7E18BsSSfBSmzWVD7tOQ0SGFhCwYKS6lSLPg3mk2DWkJkIeV8PjEYRD1i10QWcqryfCkU=), S3 Extended Request ID: ykAUAu7E18BsSSfBSmzWVD7tOQ0SGFhCwYKS6lSLPg3mk2DWkJkIeV8PjEYRD1i10QWcqryfCkU=:404 Not Found
Caused by: java.io.FileNotFoundException: copyFile(test/testAuthPathIsMixed/source/dir2/empty/, test/testAuthPathIsMixed/dest/dir2/empty/) on test/testAuthPathIsMixed/source/dir2/empty/: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: FCCC5CB9EB1CE3D5; S3 Extended Request ID: ykAUAu7E18BsSSfBSmzWVD7tOQ0SGFhCwYKS6lSLPg3mk2DWkJkIeV8PjEYRD1i10QWcqryfCkU=), S3 Extended Request ID: ykAUAu7E18BsSSfBSmzWVD7tOQ0SGFhCwYKS6lSLPg3mk2DWkJkIeV8PjEYRD1i10QWcqryfCkU=:404 Not Found
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: FCCC5CB9EB1CE3D5; S3 Extended Request ID: ykAUAu7E18BsSSfBSmzWVD7tOQ0SGFhCwYKS6lSLPg3mk2DWkJkIeV8PjEYRD1i10QWcqryfCkU=)

@iwasakims yeah, been working on that new test. Here's my latest test run.

Test run; s3 london, bucket KMS encrypted, build params: ` -Dparallel-tests -DtestsThreadCount=6 -Dmarkers=keep -Dscale` *s3guard turned off*

Two failures are related to empty dir markers; encryption over rename could be unrelated (only just turned KMS on yesterday, but...)

ERROR] Tests run: 3, Failures: 1, Errors: 0, Skipped: 2, Time elapsed: 9.589 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3GuardEmptyDirs
[ERROR] testRenameEmptyDir(org.apache.hadoop.fs.s3a.ITestS3GuardEmptyDirs)  Time elapsed: 2.81 s  <<< FAILURE!
java.lang.AssertionError: Expected a java.io.FileNotFoundException to be thrown, but got the result: : S3AFileStatus{path=s3a://stevel-london/fork-0004/test/testRenameEmptyDir/AAA-source; isDirectory=true; modification_time=0; access_time=0; owner=stevel; group=stevel; permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=true; isErasureCoded=false} isEmptyDirectory=TRUE eTag=null versionId=null
at org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:499)
at org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:384)
at org.apache.hadoop.fs.s3a.ITestS3GuardEmptyDirs.testRenameEmptyDir(ITestS3GuardEmptyDirs.java:81)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:748)
[ERROR] Tests run: 4, Failures: 1, Errors: 0, Skipped: 2, Time elapsed: 10.359 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3AEncryptionWithDefaultS3Settings
[ERROR] testEncryptionOverRename(org.apache.hadoop.fs.s3a.ITestS3AEncryptionWithDefaultS3Settings)  Time elapsed: 2.256 s  <<< FAILURE!
org.junit.ComparisonFailure: Wrong algorithm in file s3a://stevel-london/fork-0005/test/testEncryptionOverRename-0400 with encryption algorithm AES256 and key null expected:<[aws:kms]> but was:<[AES256]>
at org.junit.Assert.assertEquals(Assert.java:115)
at org.apache.hadoop.fs.s3a.EncryptionTestUtils.assertEncrypted(EncryptionTestUtils.java:91)
at org.apache.hadoop.fs.s3a.ITestS3AEncryptionWithDefaultS3Settings.assertEncrypted(ITestS3AEncryptionWithDefaultS3Settings.java:95)
at org.apache.hadoop.fs.s3a.AbstractTestS3AEncryption.testEncryptionOverRename(AbstractTestS3AEncryption.java:116)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:748)
[INFO] Running org.apache.hadoop.fs.s3a.performance.ITestS3ADeleteCost
[ERROR] Tests run: 16, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 68.498 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.performance.ITestDirectoryMarkerListing
[ERROR] testRenameUnderMarker(org.apache.hadoop.fs.s3a.performance.ITestDirectoryMarkerListing)  Time elapsed: 4.903 s  <<< FAILURE!
java.lang.AssertionError: Expected 404 of fork-0003/test/testRenameUnderMarker/base/marker/: com.amazonaws.services.s3.model.ObjectMetadata@6aac4d33
at org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:499)
at org.apache.hadoop.fs.s3a.performance.ITestDirectoryMarkerListing.head404(ITestDirectoryMarkerListing.java:464)
at org.apache.hadoop.fs.s3a.performance.ITestDirectoryMarkerListing.testRenameUnderMarker(ITestDirectoryMarkerListing.java:392)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:748)

filed [https://issues.apache.org/jira/browse/HADOOP-17167](https://issues.apache.org/jira/browse/HADOOP-171670 over the ITestS3AEncryptionWithDefaultS3Settings failure; unrelated, happens on trunk with my s3 config

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
1m 18s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s
No case conflicting files found.

+0 :ok:
markdownlint
0m  1s
markdownlint was not available.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s
The patch appears to include 21 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
1m 10s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
22m 21s
trunk passed

+1 :green_heart:
compile
21m 11s
trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1

+1 :green_heart:
compile
17m 52s
trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09

+1 :green_heart:
checkstyle
2m 56s
trunk passed

+1 :green_heart:
mvnsite
2m 10s
trunk passed

+1 :green_heart:
shadedclient
21m 16s
branch has no errors when building and testing our client artifacts.

-1 :x:
javadoc
0m 36s
hadoop-common in trunk failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.

-1 :x:
javadoc
0m 36s
hadoop-aws in trunk failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.

+1 :green_heart:
javadoc
1m 28s
trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09

+0 :ok:
spotbugs
1m  8s
Used deprecated FindBugs config; considering switching to SpotBugs.

+1 :green_heart:
findbugs
3m 22s
trunk passed

-0 :warning:
patch
1m 27s
Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 23s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
1m 25s
the patch passed

+1 :green_heart:
compile
20m 15s
the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1

-1 :x:
javac
20m 15s
root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 generated 4 new + 2048 unchanged - 4 fixed = 2052 total (was 2052)

+1 :green_heart:
compile
17m 50s
the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09

-1 :x:
javac
17m 50s
root-jdkPrivateBuild-1.8.0_252-8u252-b09-1~18.04-b09 with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 generated 4 new + 1940 unchanged - 4 fixed = 1944 total (was 1944)

-0 :warning:
checkstyle
2m 52s
root: The patch generated 20 new + 65 unchanged - 2 fixed = 85 total (was 67)

+1 :green_heart:
mvnsite
2m  8s
the patch passed

-1 :x:
whitespace
0m  0s
The patch has 7 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply

+1 :green_heart:
xml
0m  2s
The patch has no ill-formed XML file.

+1 :green_heart:
shadedclient
15m 35s
patch has no errors when building and testing our client artifacts.

-1 :x:
javadoc
0m 40s
hadoop-common in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.

-1 :x:
javadoc
0m 50s
hadoop-aws in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.

+1 :green_heart:
javadoc
1m 29s
the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09

+1 :green_heart:
findbugs
3m 41s
the patch passed

_ Other Tests _

+1 :green_heart:
unit
9m 37s
hadoop-common in the patch passed.

+1 :green_heart:
unit
1m 25s
hadoop-aws in the patch passed.

+1 :green_heart:
asflicense
0m 45s
The patch does not generate ASF License warnings.

174m 59s

Subsystem
Report/Notes

Docker
ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/8/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/2149

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml markdownlint

uname
Linux e8ee681edc16 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 05b3337a460

Default Java
Private Build-1.8.0_252-8u252-b09-1~18.04-b09

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_252-8u252-b09-1~18.04-b09

javadoc
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/8/artifact/out/branch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt

javadoc
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/8/artifact/out/branch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt

javac
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/8/artifact/out/diff-compile-javac-root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt

javac
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/8/artifact/out/diff-compile-javac-root-jdkPrivateBuild-1.8.0_252-8u252-b09-1~18.04-b09.txt

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/8/artifact/out/diff-checkstyle-root.txt

whitespace
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/8/artifact/out/whitespace-eol.txt

javadoc
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/8/artifact/out/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt

javadoc
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/8/artifact/out/patch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/8/testReport/

Max. process+thread count
1615 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-2149/8/console

versions
git=2.17.1 maven=3.6.0 findbugs=4.0.6

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
34m 42s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s
No case conflicting files found.

+0 :ok:
markdownlint
0m  1s
markdownlint was not available.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s
The patch appears to include 21 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
0m 57s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
20m 11s
trunk passed

+1 :green_heart:
compile
20m 23s
trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1

+1 :green_heart:
compile
17m 35s
trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09

+1 :green_heart:
checkstyle
2m 42s
trunk passed

+1 :green_heart:
mvnsite
2m 19s
trunk passed

+1 :green_heart:
shadedclient
19m 48s
branch has no errors when building and testing our client artifacts.

-1 :x:
javadoc
0m 41s
hadoop-common in trunk failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.

-1 :x:
javadoc
0m 38s
hadoop-aws in trunk failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.

+1 :green_heart:
javadoc
1m 34s
trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09

+0 :ok:
spotbugs
1m 15s
Used deprecated FindBugs config; considering switching to SpotBugs.

+1 :green_heart:
findbugs
3m 29s
trunk passed

-0 :warning:
patch
1m 35s
Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 26s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
1m 38s
the patch passed

+1 :green_heart:
compile
20m 58s
the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1

-1 :x:
javac
20m 58s
root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 generated 4 new + 2045 unchanged - 4 fixed = 2049 total (was 2049)

+1 :green_heart:
compile
18m  8s
the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09

-1 :x:
javac
18m  8s
root-jdkPrivateBuild-1.8.0_252-8u252-b09-1~18.04-b09 with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 generated 4 new + 1939 unchanged - 4 fixed = 1943 total (was 1943)

-0 :warning:
checkstyle
2m 44s
root: The patch generated 20 new + 65 unchanged - 2 fixed = 85 total (was 67)

+1 :green_heart:
mvnsite
2m 18s
the patch passed

-1 :x:
whitespace
0m  0s
The patch has 7 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply

+1 :green_heart:
xml
0m  2s
The patch has no ill-formed XML file.

+1 :green_heart:
shadedclient
14m 39s
patch has no errors when building and testing our client artifacts.

-1 :x:
javadoc
0m 41s
hadoop-common in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.

-1 :x:
javadoc
0m 41s
hadoop-aws in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.

+1 :green_heart:
javadoc
1m 39s
the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09

+1 :green_heart:
findbugs
3m 41s
the patch passed

_ Other Tests _

+1 :green_heart:
unit
10m 46s
hadoop-common in the patch passed.

+1 :green_heart:
unit
1m 39s
hadoop-aws in the patch passed.

+1 :green_heart:
asflicense
0m 51s
The patch does not generate ASF License warnings.

206m  3s

Subsystem
Report/Notes

Docker
ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/3/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/2149

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml markdownlint

uname
Linux 26539ab9a0e4 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 05b3337a460

Default Java
Private Build-1.8.0_252-8u252-b09-1~18.04-b09

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_252-8u252-b09-1~18.04-b09

javadoc
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/3/artifact/out/branch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt

javadoc
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/3/artifact/out/branch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt

javac
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/3/artifact/out/diff-compile-javac-root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt

javac
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/3/artifact/out/diff-compile-javac-root-jdkPrivateBuild-1.8.0_252-8u252-b09-1~18.04-b09.txt

checkstyle
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/3/artifact/out/diff-checkstyle-root.txt

whitespace
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/3/artifact/out/whitespace-eol.txt

javadoc
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/3/artifact/out/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt

javadoc
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/3/artifact/out/patch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/3/testReport/

Max. process+thread count
3208 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/3/console

versions
git=2.17.1 maven=3.6.0 findbugs=4.0.6

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Overnight test run failures. This is with a change not pushed up, where during teardown if FS != keep we do an audit of the store. Aim is to make sure that 100% of the FS operations don't leave markers in delete mode, so as to be fully compatible with existing releases.
[ERROR] testRunVerboseAudit(org.apache.hadoop.fs.s3a.tools.ITestMarkerTool)  Time elapsed: 3.298 s  <<< FAILURE!
org.junit.ComparisonFailure: [Exit code of marker(s3a://stevel-london/fork-0001/test/testRunVerboseAudit, true, 0) -> ScanResult{exitCode=46, tracker=DirMarkerTracker{leafMarkers=2, surplusMarkers=4, lastDirChecked=s3a://stevel-london/fork-0001/test/testRunVerboseAudit, filesFound=3, scanCount=28}, purgeSummary=null}] expected:<[0]> but was:<[46]>
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at org.apache.hadoop.fs.s3a.tools.ITestMarkerTool.markerTool(ITestMarkerTool.java:528)
	at org.apache.hadoop.fs.s3a.AbstractS3ATestBase.teardown(AbstractS3ATestBase.java:73)
	at org.apache.hadoop.fs.s3a.tools.ITestMarkerTool.teardown(ITestMarkerTool.java:109)

One unexpected seek failure (!)
[ERROR] testReadSmallFile[0](org.apache.hadoop.fs.contract.s3a.ITestS3AContractSeek)  Time elapsed: 1.191 s  <<< FAILURE!
java.lang.AssertionError: expected:<1024> but was:<433>
	at org.junit.Assert.fail(Assert.java:88)
	at org.junit.Assert.failNotEquals(Assert.java:834)
	at org.junit.Assert.assertEquals(Assert.java:645)
	at org.junit.Assert.assertEquals(Assert.java:631)
	at org.apache.hadoop.fs.contract.AbstractContractSeekTest.testReadSmallFile(AbstractContractSeekTest.java:575)

and the landsat select test is timing out in parallel (6) test runs, but not standalone. This is consistent for me, new this week.
[ERROR] testSelectSeekFullLandsat(org.apache.hadoop.fs.s3a.select.ITestS3SelectLandsat)  Time elapsed: 599.989 s  <<< ERROR!
org.junit.runners.model.TestTimedOutException: test timed out after 600000 milliseconds
	at java.lang.Throwable.fillInStackTrace(Native Method)
	at java.lang.Throwable.fillInStackTrace(Throwable.java:784)
	at java.lang.Throwable.<init>(Throwable.java:251)
	at org.apache.log4j.spi.LoggingEvent.getLocationInformation(LoggingEvent.java:253)
	at org.apache.log4j.helpers.PatternParser$LocationPatternConverter.convert(PatternParser.java:500)
	at org.apache.log4j.helpers.PatternConverter.format(PatternConverter.java:65)
	at org.apache.log4j.PatternLayout.format(PatternLayout.java:506)
	at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:310)
	at org.apache.log4j.WriterAppender.append(WriterAppender.java:162)
	at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
	at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
	at org.apache.log4j.Category.callAppenders(Category.java:206)
	at org.apache.log4j.Category.forcedLog(Category.java:391)
	at org.apache.log4j.Category.log(Category.java:856)
	at org.slf4j.impl.Log4jLoggerAdapter.debug(Log4jLoggerAdapter.java:230)
	at org.apache.hadoop.util.DurationInfo.close(DurationInfo.java:101)
	at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:111)
	at org.apache.hadoop.fs.s3a.select.SelectInputStream.read(SelectInputStream.java:246)
	at org.apache.hadoop.fs.s3a.select.SelectInputStream.seek(SelectInputStream.java:324)
	at org.apache.hadoop.fs.FSDataInputStream.seek(FSDataInputStream.java:67)
	at org.apache.hadoop.fs.s3a.select.AbstractS3SelectTest.seek(AbstractS3SelectTest.java:701)
	at org.apache.hadoop.fs.s3a.select.ITestS3SelectLandsat.testSelectSeekFullLandsat(ITestS3SelectLandsat.java:427)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

really weird where it times out though -log4j. Deadlock?

followup thought: I'm running with log @ debug...this log statement will only be hit in that mode. So until this week I wouldn't have been running in quite the same config. Maybe we've just found a log4j problem?

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 34s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  1s
No case conflicting files found.

+0 :ok:
markdownlint
0m  1s
markdownlint was not available.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s
The patch appears to include 23 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
0m 50s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
20m 16s
trunk passed

+1 :green_heart:
compile
22m 33s
trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1

+1 :green_heart:
compile
18m 59s
trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09

+1 :green_heart:
checkstyle
2m 37s
trunk passed

+1 :green_heart:
mvnsite
2m 33s
trunk passed

+1 :green_heart:
shadedclient
19m 37s
branch has no errors when building and testing our client artifacts.

+1 :green_heart:
javadoc
1m  7s
trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1

+1 :green_heart:
javadoc
2m 19s
trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09

+0 :ok:
spotbugs
1m 20s
Used deprecated FindBugs config; considering switching to SpotBugs.

+1 :green_heart:
findbugs
3m 42s
trunk passed

-0 :warning:
patch
1m 41s
Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 25s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
1m 24s
the patch passed

+1 :green_heart:
compile
21m 58s
the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1

-1 :x:
javac
21m 58s
root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 generated 4 new + 2045 unchanged - 4 fixed = 2049 total (was 2049)

+1 :green_heart:
compile
18m 56s
the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09

-1 :x:
javac
18m 56s
root-jdkPrivateBuild-1.8.0_252-8u252-b09-1~18.04-b09 with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 generated 4 new + 1939 unchanged - 4 fixed = 1943 total (was 1943)

-0 :warning:
checkstyle
2m 48s
root: The patch generated 10 new + 66 unchanged - 2 fixed = 76 total (was 68)

-1 :x:
mvnsite
0m 51s
hadoop-aws in the patch failed.

-1 :x:
whitespace
0m  0s
The patch has 6 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply

+1 :green_heart:
xml
0m  1s
The patch has no ill-formed XML file.

+1 :green_heart:
shadedclient
14m 23s
patch has no errors when building and testing our client artifacts.

+1 :green_heart:
javadoc
1m  7s
the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1

+1 :green_heart:
javadoc
2m  9s
the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09

-1 :x:
findbugs
0m 45s
hadoop-aws in the patch failed.

_ Other Tests _

+1 :green_heart:
unit
9m 37s
hadoop-common in the patch passed.

-1 :x:
unit
0m 48s
hadoop-aws in the patch failed.

+1 :green_heart:
asflicense
0m 50s
The patch does not generate ASF License warnings.

175m 28s

Subsystem
Report/Notes

Docker
ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/6/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/2149

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml markdownlint

uname
Linux ff8c28055ac1 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / ab2b3df2de1

Default Java
Private Build-1.8.0_252-8u252-b09-1~18.04-b09

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_252-8u252-b09-1~18.04-b09

javac
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/6/artifact/out/diff-compile-javac-root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt

javac
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/6/artifact/out/diff-compile-javac-root-jdkPrivateBuild-1.8.0_252-8u252-b09-1~18.04-b09.txt

checkstyle
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/6/artifact/out/diff-checkstyle-root.txt

mvnsite
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/6/artifact/out/patch-mvnsite-hadoop-tools_hadoop-aws.txt

whitespace
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/6/artifact/out/whitespace-eol.txt

findbugs
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/6/artifact/out/patch-findbugs-hadoop-tools_hadoop-aws.txt

unit
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/6/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/6/testReport/

Max. process+thread count
3249 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/6/console

versions
git=2.17.1 maven=3.6.0 findbugs=4.0.6

Powered by
Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

last local batch run;

mistake in one of the rename cost assumptions
audit code overreacts if test suite FS is closed

fail with  mvit -Dparallel-tests -DtestsThreadCount=4 -Dmarkers=delete -Ds3guard -Ddynamo -Dscale -Dtest=moo -Dfs.s3a.directory.marker.audit=true -Dit.test=ITestDirectoryMarkerListing
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.hadoop.fs.s3a.performance.ITestDirectoryMarkerListing
[ERROR] Tests run: 17, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 32.636 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.performance.ITestDirectoryMarkerListing
[ERROR] testRenameUnderMarker(org.apache.hadoop.fs.s3a.performance.ITestDirectoryMarkerListing)  Time elapsed: 2.496 s  <<< FAILURE!
java.lang.AssertionError: Expected 404 of test/testRenameUnderMarker/base/marker/: "Object test/testRenameUnderMarker/base/marker/ of length 0"
   at org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:499)
   at org.apache.hadoop.fs.s3a.performance.ITestDirectoryMarkerListing.head404(ITestDirectoryMarkerListing.java:520)
   at org.apache.hadoop.fs.s3a.performance.ITestDirectoryMarkerListing.testRenameUnderMarker(ITestDirectoryMarkerListing.java:434)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m  0s
Docker mode activated.

-1 :x:
patch
0m  5s
https://github.com/apache/hadoop/pull/2149 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.

Subsystem
Report/Notes

GITHUB PR
https://github.com/apache/hadoop/pull/2149

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/7/console

versions
git=2.17.1

Powered by
Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

+the marker tool audit operation should add a -nonauth option which doesn't worry about markers present in authoritative paths. That way you can do a bucket audit without it failing on /hive or other specific paths. Without that, to audit an entire bucket, you'd need to list all root directories which weren't authoritative and explicitly audit them
straightforward to do -simply leave out markers in auth paths from the audit as they come in from the listObjects call

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m  0s
Docker mode activated.

-1 :x:
docker
8m 48s
Docker failed to build yetus/hadoop:e277d338da2.

Subsystem
Report/Notes

GITHUB PR
https://github.com/apache/hadoop/pull/2149

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/8/console

versions
git=2.17.1

Powered by
Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 31s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  1s
No case conflicting files found.

+0 :ok:
markdownlint
0m  1s
markdownlint was not available.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s
The patch appears to include 23 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
3m 19s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
25m 53s
trunk passed

+1 :green_heart:
compile
19m 24s
trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1

+1 :green_heart:
compile
16m 45s
trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09

+1 :green_heart:
checkstyle
2m 39s
trunk passed

+1 :green_heart:
mvnsite
2m 22s
trunk passed

+1 :green_heart:
shadedclient
19m 42s
branch has no errors when building and testing our client artifacts.

+1 :green_heart:
javadoc
1m 14s
trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1

+1 :green_heart:
javadoc
2m 17s
trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09

+0 :ok:
spotbugs
1m 15s
Used deprecated FindBugs config; considering switching to SpotBugs.

+1 :green_heart:
findbugs
3m 26s
trunk passed

-0 :warning:
patch
1m 35s
Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 25s
Maven dependency ordering for patch

-1 :x:
mvninstall
0m 27s
hadoop-aws in the patch failed.

-1 :x:
compile
17m 42s
root in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.

-1 :x:
javac
17m 42s
root in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.

-1 :x:
compile
15m 47s
root in the patch failed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09.

-1 :x:
javac
15m 47s
root in the patch failed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09.

-0 :warning:
checkstyle
2m 42s
root: The patch generated 11 new + 67 unchanged - 2 fixed = 78 total (was 69)

-1 :x:
mvnsite
0m 49s
hadoop-aws in the patch failed.

-1 :x:
whitespace
0m  0s
The patch has 8 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply

-1 :x:
whitespace
0m  0s
The patch 3 line(s) with tabs.

+1 :green_heart:
xml
0m  1s
The patch has no ill-formed XML file.

+1 :green_heart:
shadedclient
13m 59s
patch has no errors when building and testing our client artifacts.

+1 :green_heart:
javadoc
1m 12s
the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1

+1 :green_heart:
javadoc
2m 17s
the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09

-1 :x:
findbugs
0m 48s
hadoop-aws in the patch failed.

_ Other Tests _

+1 :green_heart:
unit
9m 16s
hadoop-common in the patch passed.

-1 :x:
unit
0m 48s
hadoop-aws in the patch failed.

+1 :green_heart:
asflicense
0m 53s
The patch does not generate ASF License warnings.

169m 49s

Subsystem
Report/Notes

Docker
ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/9/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/2149

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml markdownlint

uname
Linux 51ef73c05764 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 3f73facd7b3

Default Java
Private Build-1.8.0_252-8u252-b09-1~18.04-b09

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_252-8u252-b09-1~18.04-b09

mvninstall
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/9/artifact/out/patch-mvninstall-hadoop-tools_hadoop-aws.txt

compile
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/9/artifact/out/patch-compile-root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt

javac
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/9/artifact/out/patch-compile-root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt

compile
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/9/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_252-8u252-b09-1~18.04-b09.txt

javac
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/9/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_252-8u252-b09-1~18.04-b09.txt

checkstyle
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/9/artifact/out/diff-checkstyle-root.txt

mvnsite
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/9/artifact/out/patch-mvnsite-hadoop-tools_hadoop-aws.txt

whitespace
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/9/artifact/out/whitespace-eol.txt

whitespace
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/9/artifact/out/whitespace-tabs.txt

findbugs
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/9/artifact/out/patch-findbugs-hadoop-tools_hadoop-aws.txt

unit
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/9/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/9/testReport/

Max. process+thread count
1481 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/9/console

versions
git=2.17.1 maven=3.6.0 findbugs=4.0.6

Powered by
Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 32s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  1s
No case conflicting files found.

+0 :ok:
markdownlint
0m  1s
markdownlint was not available.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s
The patch appears to include 24 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
3m 19s
Maven dependency ordering for branch

-1 :x:
mvninstall
27m 13s
root in trunk failed.

-1 :x:
compile
14m 34s
root in trunk failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.

+1 :green_heart:
compile
17m 46s
trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01

-0 :warning:
checkstyle
2m 24s
The patch fails to run checkstyle in root

+1 :green_heart:
mvnsite
2m 14s
trunk passed

+1 :green_heart:
shadedclient
19m 30s
branch has no errors when building and testing our client artifacts.

+1 :green_heart:
javadoc
1m  6s
trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1

+1 :green_heart:
javadoc
2m 19s
trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01

+0 :ok:
spotbugs
1m 19s
Used deprecated FindBugs config; considering switching to SpotBugs.

+1 :green_heart:
findbugs
3m 53s
trunk passed

-0 :warning:
patch
1m 39s
Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 30s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
1m 34s
the patch passed

+1 :green_heart:
compile
23m 41s
the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1

-1 :x:
javac
23m 41s
root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 generated 210 new + 1839 unchanged - 0 fixed = 2049 total (was 1839)

+1 :green_heart:
compile
20m 50s
the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01

-1 :x:
javac
20m 50s
root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 4 new + 1939 unchanged - 4 fixed = 1943 total (was 1943)

-0 :warning:
checkstyle
3m  5s
The patch fails to run checkstyle in root

+1 :green_heart:
mvnsite
2m 25s
the patch passed

-1 :x:
whitespace
0m  0s
The patch has 8 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply

-1 :x:
whitespace
0m  0s
The patch 3 line(s) with tabs.

+1 :green_heart:
xml
0m  2s
The patch has no ill-formed XML file.

+1 :green_heart:
shadedclient
16m 48s
patch has no errors when building and testing our client artifacts.

+1 :green_heart:
javadoc
1m  7s
the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1

+1 :green_heart:
javadoc
2m 23s
the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01

+1 :green_heart:
findbugs
4m  3s
the patch passed

_ Other Tests _

+1 :green_heart:
unit
10m  3s
hadoop-common in the patch passed.

+1 :green_heart:
unit
1m 42s
hadoop-aws in the patch passed.

+1 :green_heart:
asflicense
0m 57s
The patch does not generate ASF License warnings.

184m 43s

Subsystem
Report/Notes

Docker
ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/10/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/2149

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml markdownlint

uname
Linux 3834dc4fafee 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / c7e71a6c0be

Default Java
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01

mvninstall
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/10/artifact/out/branch-mvninstall-root.txt

compile
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/10/artifact/out/branch-compile-root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt

checkstyle
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/10/artifact/out/buildtool-branch-checkstyle-root.txt

javac
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/10/artifact/out/diff-compile-javac-root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt

javac
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/10/artifact/out/diff-compile-javac-root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt

checkstyle
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/10/artifact/out/buildtool-patch-checkstyle-root.txt

whitespace
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/10/artifact/out/whitespace-eol.txt

whitespace
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/10/artifact/out/whitespace-tabs.txt

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/10/testReport/

Max. process+thread count
2732 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/10/console

versions
git=2.17.1 maven=3.6.0 findbugs=4.0.6

Powered by
Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

latest patch tweaks markertool, but also adds pathcapabilities probes to the s3a store so you can see if an instance is (a) markeraware and (b) whether markers are being kept or deleted on a given path. Look at the docs for details.
s bin/hadoop jar $CLOUDSTORE pathcapability fs.s3a.capability.directory.marker.keep s3a://stevel-london/tables
Probing s3a://stevel-london/tables for capability fs.s3a.capability.directory.marker.keep
2020-08-11 22:15:15,501 [main] INFO  s3a.S3AFileSystem (S3Guard.java:logS3GuardDisabled(1152)) - S3Guard is disabled on this bucket: stevel-london
2020-08-11 22:15:15,506 [main] INFO  impl.DirectoryPolicyImpl (DirectoryPolicyImpl.java:getDirectoryPolicy(143)) - Directory markers will be kept on authoritative paths
Using filesystem s3a://stevel-london
Path s3a://stevel-london/tables has capability fs.s3a.capability.directory.marker.keep

branch-3.2 doesn't support Path capabilities, but it does do it for stream capabilities; I'll extend cloudstore to have a streamcapabilities command too

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
33m  5s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s
No case conflicting files found.

+0 :ok:
markdownlint
0m  0s
markdownlint was not available.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  1s
The patch appears to include 24 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
3m 22s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
29m 43s
trunk passed

+1 :green_heart:
compile
19m 28s
trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1

+1 :green_heart:
compile
16m 50s
trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01

+1 :green_heart:
checkstyle
2m 44s
trunk passed

+1 :green_heart:
mvnsite
2m 25s
trunk passed

+1 :green_heart:
shadedclient
19m 46s
branch has no errors when building and testing our client artifacts.

+1 :green_heart:
javadoc
1m 15s
trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1

+1 :green_heart:
javadoc
2m 15s
trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01

+0 :ok:
spotbugs
1m 15s
Used deprecated FindBugs config; considering switching to SpotBugs.

+1 :green_heart:
findbugs
3m 27s
trunk passed

-0 :warning:
patch
1m 35s
Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 26s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
1m 23s
the patch passed

+1 :green_heart:
compile
18m 46s
the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1

-1 :x:
javac
18m 46s
root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 generated 4 new + 2045 unchanged - 4 fixed = 2049 total (was 2049)

+1 :green_heart:
compile
16m 44s
the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01

-1 :x:
javac
16m 44s
root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 4 new + 1939 unchanged - 4 fixed = 1943 total (was 1943)

-0 :warning:
checkstyle
3m 23s
root: The patch generated 16 new + 72 unchanged - 2 fixed = 88 total (was 74)

+1 :green_heart:
mvnsite
2m 23s
the patch passed

-1 :x:
whitespace
0m  0s
The patch has 12 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply

-1 :x:
whitespace
0m  0s
The patch 3 line(s) with tabs.

+1 :green_heart:
xml
0m  2s
The patch has no ill-formed XML file.

+1 :green_heart:
shadedclient
13m 54s
patch has no errors when building and testing our client artifacts.

+1 :green_heart:
javadoc
1m 14s
the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1

+1 :green_heart:
javadoc
2m 14s
the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01

-1 :x:
findbugs
1m 23s
hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)

_ Other Tests _

+1 :green_heart:
unit
9m 17s
hadoop-common in the patch passed.

+1 :green_heart:
unit
1m 39s
hadoop-aws in the patch passed.

+1 :green_heart:
asflicense
0m 55s
The patch does not generate ASF License warnings.

211m 16s

Reason
Tests

FindBugs
module:hadoop-tools/hadoop-aws

Found reliance on default encoding in org.apache.hadoop.fs.s3a.tools.MarkerTool.run(String[], PrintStream):in org.apache.hadoop.fs.s3a.tools.MarkerTool.run(String[], PrintStream): new java.io.FileWriter(String)  At MarkerTool.java:[line 284]

Subsystem
Report/Notes

Docker
ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/11/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/2149

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml markdownlint

uname
Linux cc5f0b23c1ae 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 3fd3aeb621e

Default Java
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01

javac
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/11/artifact/out/diff-compile-javac-root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt

javac
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/11/artifact/out/diff-compile-javac-root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt

checkstyle
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/11/artifact/out/diff-checkstyle-root.txt

whitespace
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/11/artifact/out/whitespace-eol.txt

whitespace
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/11/artifact/out/whitespace-tabs.txt

findbugs
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/11/artifact/out/new-findbugs-hadoop-tools_hadoop-aws.html

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/11/testReport/

Max. process+thread count
3250 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/11/console

versions
git=2.17.1 maven=3.6.0 findbugs=4.0.6

Powered by
Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

It looks good to me overall as I did not see relevant integration tests failures with the patch applied. It is a bit confusing that there are Parameterized tests covering both keep and delete in addition to -Dmarkers option while we need both.
It would help reviewing to split s3guard related part and performance tests to follow-up JIRAs. We can commit essential part first if it retains present default (DELETE) behaviour.
mvn verify -Dtest=x on trunk:
[ERROR] Errors:
[ERROR]   ITestS3AConfiguration.testAutomaticProxyPortSelection:190->expectFSCreateFailure:159->lambda$expectFSCreateFailure$0:161 » AccessDenied
[ERROR]   ITestS3ATemporaryCredentials.testInvalidSTSBinding:257 » SdkClient Unable to f...
[ERROR]   ITestS3ATemporaryCredentials.testSTS:130 » SdkClient Unable to find a region v...
[ERROR]   ITestS3ATemporaryCredentials.testSessionRequestExceptionTranslation:441->lambda$testSessionRequestExceptionTranslation$5:442 » SdkClient
[ERROR]   ITestS3ATemporaryCredentials.testSessionTokenExpiry:222 » SdkClient Unable to ...
[ERROR]   ITestS3ATemporaryCredentials.testSessionTokenPropagation:193 » SdkClient Unabl...
[ERROR]   ITestDirectoryCommitProtocol>ITestStagingCommitProtocol.setup:67->AbstractITCommitProtocol.setup:160->AbstractITCommitProtocol.cleanupDestDir:120->AbstractCommitITest.rmdir:164 » AccessDenied
[INFO]
[ERROR] Tests run: 700, Failures: 0, Errors: 7, Skipped: 251

mvn verify -Dtest=x -Dmarkers=keep with the patch applied:
[ERROR] Failures:
[ERROR]   ITestSessionDelegationInFileystem.testDTUtilShell:719->dtutil:708->Assert.assertEquals:631->Assert.assertEquals:645->Assert.failNotEquals:834->Assert.fail:88 expected:<0> but was:<1>
[ERROR] Errors:
[ERROR]   ITestS3AConfiguration.testAutomaticProxyPortSelection:190->expectFSCreateFailure:159->lambda$expectFSCreateFailure$0:161 » AccessDenied
[ERROR]   ITestS3ATemporaryCredentials.testInvalidSTSBinding:257 » SdkClient Unable to f...
[ERROR]   ITestS3ATemporaryCredentials.testSTS:130 » SdkClient Unable to find a region v...
[ERROR]   ITestS3ATemporaryCredentials.testSessionRequestExceptionTranslation:441->lambda$testSessionRequestExceptionTranslation$5:442 » SdkClient
[ERROR]   ITestS3ATemporaryCredentials.testSessionTokenExpiry:222 » SdkClient Unable to ...
[ERROR]   ITestS3ATemporaryCredentials.testSessionTokenPropagation:193 » SdkClient Unabl...
[ERROR]   ITestDelegatedMRJob.testJobSubmissionCollectsTokens:286 » SdkClient Unable to ...
[ERROR]   ITestSessionDelegationInFileystem.testAddTokensFromFileSystem:239 » SdkClient ...
[ERROR]   ITestSessionDelegationInFileystem.testCanRetrieveTokenFromCurrentUserCreds:264->createDelegationTokens:296->AbstractDelegationIT.mkTokens:88 » SdkClient
[ERROR]   ITestSessionDelegationInFileystem.testDTCredentialProviderFromCurrentUserCreds:282->createDelegationTokens:296->AbstractDelegationIT.mkTokens:88 » SdkClient
[ERROR]   ITestSessionDelegationInFileystem.testDelegatedFileSystem:312->createDelegationTokens:296->AbstractDelegationIT.mkTokens:88 » SdkClient
[ERROR]   ITestSessionDelegationInFileystem.testDelegationBindingMismatch1:441->createDelegationTokens:296->AbstractDelegationIT.mkTokens:88 » SdkClient
[ERROR]   ITestSessionDelegationInFileystem.testFileSystemBoundToCreator:690 » SdkClient
[ERROR]   ITestSessionDelegationInFileystem.testGetDTfromFileSystem:216 » SdkClient Unab...
[ERROR]   ITestSessionDelegationInFileystem.testHDFSFetchDTCommand:615->lambda$testHDFSFetchDTCommand$3:616 » SdkClient
[ERROR]   ITestSessionDelegationInFileystem.testYarnCredentialPickup:585 » SdkClient Una...
[ERROR]   ITestSessionDelegationTokens.testCreateAndUseDT:176 » SdkClient Unable to find...
[ERROR]   ITestSessionDelegationTokens.testCreateWithRenewer:230 » SdkClient Unable to f...
[ERROR]   ITestSessionDelegationTokens.testSaveLoadTokens:121 » SdkClient Unable to find...
[INFO]
[ERROR] Tests run: 1349, Failures: 1, Errors: 19, Skipped: 521

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 34s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  1s
No case conflicting files found.

+0 :ok:
markdownlint
0m  1s
markdownlint was not available.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s
The patch appears to include 28 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
3m 21s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
25m 56s
trunk passed

+1 :green_heart:
compile
19m 28s
trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1

+1 :green_heart:
compile
16m 45s
trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01

+1 :green_heart:
checkstyle
2m 42s
trunk passed

+1 :green_heart:
mvnsite
2m 22s
trunk passed

+1 :green_heart:
shadedclient
19m 39s
branch has no errors when building and testing our client artifacts.

+1 :green_heart:
javadoc
1m 15s
trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1

+1 :green_heart:
javadoc
2m 15s
trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01

+0 :ok:
spotbugs
1m 15s
Used deprecated FindBugs config; considering switching to SpotBugs.

+1 :green_heart:
findbugs
3m 25s
trunk passed

-0 :warning:
patch
1m 36s
Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 28s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
1m 23s
the patch passed

+1 :green_heart:
compile
18m 47s
the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1

-1 :x:
javac
18m 47s
root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 generated 4 new + 2045 unchanged - 4 fixed = 2049 total (was 2049)

+1 :green_heart:
compile
16m 51s
the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01

-1 :x:
javac
16m 51s
root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 4 new + 1939 unchanged - 4 fixed = 1943 total (was 1943)

-0 :warning:
checkstyle
2m 40s
root: The patch generated 20 new + 75 unchanged - 2 fixed = 95 total (was 77)

+1 :green_heart:
mvnsite
2m 17s
the patch passed

-1 :x:
whitespace
0m  0s
The patch has 20 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply

-1 :x:
whitespace
0m  0s
The patch 3 line(s) with tabs.

+1 :green_heart:
xml
0m  2s
The patch has no ill-formed XML file.

+1 :green_heart:
shadedclient
13m 52s
patch has no errors when building and testing our client artifacts.

+1 :green_heart:
javadoc
1m 13s
the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1

+1 :green_heart:
javadoc
2m 17s
the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01

-1 :x:
findbugs
1m 24s
hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)

_ Other Tests _

+1 :green_heart:
unit
9m 19s
hadoop-common in the patch passed.

+1 :green_heart:
unit
1m 41s
hadoop-aws in the patch passed.

+1 :green_heart:
asflicense
0m 56s
The patch does not generate ASF License warnings.

173m 52s

Reason
Tests

FindBugs
module:hadoop-tools/hadoop-aws

Found reliance on default encoding in org.apache.hadoop.fs.s3a.tools.MarkerTool.run(String[], PrintStream):in org.apache.hadoop.fs.s3a.tools.MarkerTool.run(String[], PrintStream): new java.io.FileWriter(String)  At MarkerTool.java:[line 287]

Subsystem
Report/Notes

Docker
ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/12/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/2149

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml markdownlint

uname
Linux 496f50095ac4 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 10716040a85

Default Java
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01

javac
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/12/artifact/out/diff-compile-javac-root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt

javac
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/12/artifact/out/diff-compile-javac-root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt

checkstyle
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/12/artifact/out/diff-checkstyle-root.txt

whitespace
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/12/artifact/out/whitespace-eol.txt

whitespace
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/12/artifact/out/whitespace-tabs.txt

findbugs
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/12/artifact/out/new-findbugs-hadoop-tools_hadoop-aws.html

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/12/testReport/

Max. process+thread count
1443 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/12/console

versions
git=2.17.1 maven=3.6.0 findbugs=4.0.6

Powered by
Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

Overall looks good to me apart from some minor nits and doc changes. Really nice documentation.
Ran tests with  two configs
mvn clean verify  -Ds3guard -Ddynamo  -Dauth -Dmarkers=delete -Dparallel-tests -DtestsThreadCount=8
mvn clean verify  -Ds3guard -Ddynamo  -Dauth -Dscale -Dparallel-tests -DtestsThreadCount=8
I don't see any related failure.

The condition and message doesn't seem consistent. Please review.

it is. I'm adding a comment. I'm going to put in a unit test to verify the condition. Key point: if needEmptyDirFlag is false: we don't care, if it is true then the probe must contain a list, so
!needEmptyDirectoryFlag ||
 (needEmptyDirectoryFlag  && probes.contains(StatusProbeEnum.List))

And obviously, the second test of the flag state is superflouous, which is why IDEs and things complain about it

It is a bit confusing that there are Parameterized tests covering both keep and delete in addition to -Dmarkers option while we need both.

the parameterized ones lets us explicitly explore behaviours when we turn features on and off, so verify that they do what we expect in terms of keep vs delete.
the -Dmarkers= lets us switch the entire test run to keep/delete, so verify that the whole API works with keep=true, and, with auditing enabled, that the delete option isn't somehow retaining markers. It's a PITA to have another option to run with, but making it explicit lets us show when pasting in our test results which option we used

It would help reviewing to split s3guard related part and performance tests to follow-up JIRAs. We can commit essential part first if it retains present default (DELETE) behaviour.

the s3guard bit which gabor was talking about? -if everyone wants it, then I shall reluctantly do this
the performance tests, I'd like them in. They are looking at the cost of delete/rename and listings. All the existing cost tests broke as fewer head calls were being made, more list etc, and trying to distinguish regressions from legit changes of the in-code-constants was hard. The OperationCost type with simple aggregation makes it easy to determine what the cost are.
oh, they aren't really performance BTW -cost. Should I put them in a cost/ package instead?

@iwasakims thanks for that review.
checkstyles in, and feedback from Gabor: we now have static path capabilities probes for bucket policy as well as the dynamic "what will happen on this path". All in DirectoryPolicyImpl, so it can be unit tested too!
Testing: s3 london: -Dparallel-tests -DtestsThreadCount=4 -Dmarkers=keep  -Dscale.
and
-Dparallel-tests -DtestsThreadCount=4 -Dmarkers=delete -Ds3guard -Ddynamo -Dscale -Dfs.s3a.directory.marker.audit=true

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
2m 20s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  1s
No case conflicting files found.

+0 :ok:
markdownlint
0m  1s
markdownlint was not available.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s
The patch appears to include 28 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
3m 23s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
30m 43s
trunk passed

+1 :green_heart:
compile
23m 58s
trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1

+1 :green_heart:
compile
22m 46s
trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01

+1 :green_heart:
checkstyle
2m 56s
trunk passed

+1 :green_heart:
mvnsite
2m 11s
trunk passed

+1 :green_heart:
shadedclient
21m  8s
branch has no errors when building and testing our client artifacts.

+1 :green_heart:
javadoc
0m 57s
trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1

+1 :green_heart:
javadoc
2m  2s
trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01

+0 :ok:
spotbugs
1m 10s
Used deprecated FindBugs config; considering switching to SpotBugs.

+1 :green_heart:
findbugs
3m 20s
trunk passed

-0 :warning:
patch
1m 28s
Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 23s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
1m 26s
the patch passed

+1 :green_heart:
compile
20m  8s
the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1

-1 :x:
javac
20m  8s
root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 generated 4 new + 2048 unchanged - 4 fixed = 2052 total (was 2052)

+1 :green_heart:
compile
17m 26s
the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01

-1 :x:
javac
17m 26s
root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 4 new + 1941 unchanged - 4 fixed = 1945 total (was 1945)

-0 :warning:
checkstyle
2m 54s
root: The patch generated 3 new + 75 unchanged - 2 fixed = 78 total (was 77)

+1 :green_heart:
mvnsite
2m  7s
the patch passed

-1 :x:
whitespace
0m  0s
The patch has 21 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply

-1 :x:
whitespace
0m  0s
The patch 3 line(s) with tabs.

+1 :green_heart:
xml
0m  1s
The patch has no ill-formed XML file.

+1 :green_heart:
shadedclient
15m 40s
patch has no errors when building and testing our client artifacts.

+1 :green_heart:
javadoc
0m 55s
the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1

+1 :green_heart:
javadoc
1m 59s
the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01

+1 :green_heart:
findbugs
3m 37s
the patch passed

_ Other Tests _

+1 :green_heart:
unit
10m 26s
hadoop-common in the patch passed.

+1 :green_heart:
unit
1m 25s
hadoop-aws in the patch passed.

+1 :green_heart:
asflicense
0m 48s
The patch does not generate ASF License warnings.

194m 41s

Subsystem
Report/Notes

Docker
ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/13/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/2149

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml markdownlint

uname
Linux dda80bdd291e 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 86bbd38c8dc

Default Java
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01

javac
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/13/artifact/out/diff-compile-javac-root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt

javac
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/13/artifact/out/diff-compile-javac-root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt

checkstyle
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/13/artifact/out/diff-checkstyle-root.txt

whitespace
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/13/artifact/out/whitespace-eol.txt

whitespace
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/13/artifact/out/whitespace-tabs.txt

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/13/testReport/

Max. process+thread count
1913 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/13/console

versions
git=2.17.1 maven=3.6.0 findbugs=4.0.6

Powered by
Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

checkstyles are either unfixable (initialize()) or deliberate (test case naming)
./hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:337:  public void initialize(URI name, Configuration originalConf):3: Method length is 151 lines (max allowed is 150). [MethodLength]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/tools/ITestMarkerToolRootOperations.java:49:  public void test_100_audit_root_noauth() throws Throwable {:15: Name 'test_100_audit_root_noauth' must match pattern '^[a-z][a-zA-Z0-9]*$'. [MethodName]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/tools/ITestMarkerToolRootOperations.java:60:  public void test_200_clean_root() throws Throwable {:15: Name 'test_200_clean_root' must match pattern '^[a-z][a-zA-Z0-9]*$'. [MethodName]

I'm going to merge this in now -just do one final review of the docs. I think I'll add a compatibility statement on the index.md page too, something to repeat for the other releases as I backport the getFileStatus change

and its merged in. Thanks to everyone for their reviews and feedback!
Next steps

3.3.x cherrypick and test rerun (in progress)
3.2.x listing patch
patch from 3.2.x to 3.1, 3.0.
I'll see about a branch-2 fix, because I'll inevitably end up having to do that...

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 31s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  1s
No case conflicting files found.

+0 :ok:
markdownlint
0m  1s
markdownlint was not available.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s
The patch appears to include 28 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
3m 23s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
26m 20s
trunk passed

+1 :green_heart:
compile
19m 48s
trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1

+1 :green_heart:
compile
16m 47s
trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01

+1 :green_heart:
checkstyle
2m 41s
trunk passed

+1 :green_heart:
mvnsite
2m 21s
trunk passed

+1 :green_heart:
shadedclient
19m 44s
branch has no errors when building and testing our client artifacts.

+1 :green_heart:
javadoc
1m 16s
trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1

+1 :green_heart:
javadoc
2m 20s
trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01

+0 :ok:
spotbugs
1m 14s
Used deprecated FindBugs config; considering switching to SpotBugs.

+1 :green_heart:
findbugs
3m 25s
trunk passed

-0 :warning:
patch
1m 35s
Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 26s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
1m 26s
the patch passed

+1 :green_heart:
compile
18m 44s
the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1

-1 :x:
javac
18m 44s
root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 generated 4 new + 2045 unchanged - 4 fixed = 2049 total (was 2049)

+1 :green_heart:
compile
16m 44s
the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01

-1 :x:
javac
16m 44s
root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 4 new + 1939 unchanged - 4 fixed = 1943 total (was 1943)

-0 :warning:
checkstyle
2m 41s
root: The patch generated 3 new + 75 unchanged - 2 fixed = 78 total (was 77)

+1 :green_heart:
mvnsite
2m 21s
the patch passed

-1 :x:
whitespace
0m  0s
The patch has 19 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply

-1 :x:
whitespace
0m  0s
The patch 3 line(s) with tabs.

+1 :green_heart:
xml
0m  1s
The patch has no ill-formed XML file.

+1 :green_heart:
shadedclient
14m 10s
patch has no errors when building and testing our client artifacts.

+1 :green_heart:
javadoc
1m 14s
the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1

+1 :green_heart:
javadoc
2m 21s
the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01

+1 :green_heart:
findbugs
3m 43s
the patch passed

_ Other Tests _

+1 :green_heart:
unit
9m 20s
hadoop-common in the patch passed.

+1 :green_heart:
unit
1m 44s
hadoop-aws in the patch passed.

+1 :green_heart:
asflicense
0m 56s
The patch does not generate ASF License warnings.

175m 16s

Subsystem
Report/Notes

Docker
ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/14/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/2149

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml markdownlint

uname
Linux 8bf5646cbc90 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / e3d1966f58a

Default Java
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01

javac
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/14/artifact/out/diff-compile-javac-root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt

javac
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/14/artifact/out/diff-compile-javac-root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt

checkstyle
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/14/artifact/out/diff-checkstyle-root.txt

whitespace
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/14/artifact/out/whitespace-eol.txt

whitespace
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/14/artifact/out/whitespace-tabs.txt

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/14/testReport/

Max. process+thread count
2515 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2149/14/console

versions
git=2.17.1 maven=3.6.0 findbugs=4.0.6

Powered by
Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

doing some followup tuning of the CLI tooling in #2254 ; nothing major. Better for integration test and general message formatting...the things which only surface after playing with the tool for a few days

Thanks Steve. Looking forward to a Hadoop 2 backport!

@liuml07 Looking forward to a Hadoop 2 backport!
I'll do a hadoop-2 fixup of list time changes, but if you look #2310 delete gets complex, especially handling partial failures of deletes. I'm going to have to say that any branch without that (major) change will be able to read data in a path which keeps markers, but it must not delete things -otherwise S3Guard can get out of sync.