HADOOP-15183 S3Guard store becomes inconsistent after partial failure of rename

This patch carefully pulls out the changes into a new class, s3a.impl.MultiObjectDeleteSupport which isolates this from everything else, and is designed for testing in isolation.
Tests are still WiP; I've pulled the partial rename failure ones out ITestAssumeRole into isolation here, because the fact that they use assumed roles is a detail to set up the failures.
It currently only does the delete failure, but I'll probably include the copy failures in here as they are so related.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
28
Docker mode activated.

_ Prechecks _

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 8 new or modified test files.

_ trunk Compile Tests _

0
mvndep
64
Maven dependency ordering for branch

+1
mvninstall
1058
trunk passed

+1
compile
1029
trunk passed

+1
checkstyle
179
trunk passed

+1
mvnsite
126
trunk passed

+1
shadedclient
992
branch has no errors when building and testing our client artifacts.

+1
findbugs
150
trunk passed

+1
javadoc
91
trunk passed

_ Patch Compile Tests _

0
mvndep
21
Maven dependency ordering for patch

-1
mvninstall
27
hadoop-aws in the patch failed.

-1
compile
875
root in the patch failed.

-1
javac
875
root in the patch failed.

-0
checkstyle
190
root: The patch generated 14 new + 40 unchanged - 0 fixed = 54 total (was 40)

-1
mvnsite
39
hadoop-aws in the patch failed.

+1
whitespace
0
The patch has no whitespace issues.

+1
shadedclient
630
patch has no errors when building and testing our client artifacts.

-1
findbugs
28
hadoop-aws in the patch failed.

+1
javadoc
81
the patch passed

_ Other Tests _

+1
unit
521
hadoop-common in the patch passed.

-1
unit
35
hadoop-aws in the patch failed.

+1
asflicense
36
The patch does not generate ASF License warnings.

6369

Subsystem
Report/Notes

Docker
Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-654/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/654

Optional Tests
dupname  asflicense  compile  javac  javadoc  mvninstall  mvnsite  unit  shadedclient  findbugs  checkstyle

uname
Linux f1be4a6ffe3d 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 15d38b1

maven
version: Apache Maven 3.3.9

Default Java
1.8.0_191

findbugs
v3.1.0-RC1

mvninstall
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/1/artifact/out/patch-mvninstall-hadoop-tools_hadoop-aws.txt

compile
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/1/artifact/out/patch-compile-root.txt

javac
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/1/artifact/out/patch-compile-root.txt

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/1/artifact/out/diff-checkstyle-root.txt

mvnsite
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/1/artifact/out/patch-mvnsite-hadoop-tools_hadoop-aws.txt

findbugs
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/1/artifact/out/patch-findbugs-hadoop-tools_hadoop-aws.txt

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/1/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/1/testReport/

Max. process+thread count
1434 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/1/console

Powered by
Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
28
Docker mode activated.

_ Prechecks _

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 8 new or modified test files.

_ trunk Compile Tests _

0
mvndep
60
Maven dependency ordering for branch

+1
mvninstall
994
trunk passed

+1
compile
944
trunk passed

+1
checkstyle
196
trunk passed

+1
mvnsite
118
trunk passed

+1
shadedclient
1032
branch has no errors when building and testing our client artifacts.

+1
findbugs
136
trunk passed

+1
javadoc
81
trunk passed

_ Patch Compile Tests _

0
mvndep
20
Maven dependency ordering for patch

-1
mvninstall
24
hadoop-aws in the patch failed.

-1
compile
838
root in the patch failed.

-1
javac
838
root in the patch failed.

-0
checkstyle
185
root: The patch generated 14 new + 40 unchanged - 0 fixed = 54 total (was 40)

-1
mvnsite
38
hadoop-aws in the patch failed.

+1
whitespace
0
The patch has no whitespace issues.

+1
shadedclient
614
patch has no errors when building and testing our client artifacts.

-1
findbugs
32
hadoop-aws in the patch failed.

+1
javadoc
86
the patch passed

_ Other Tests _

-1
unit
519
hadoop-common in the patch failed.

-1
unit
39
hadoop-aws in the patch failed.

+1
asflicense
36
The patch does not generate ASF License warnings.

6197

Reason
Tests

Failed junit tests
hadoop.security.token.delegation.TestZKDelegationTokenSecretManager

Subsystem
Report/Notes

Docker
Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-654/2/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/654

Optional Tests
dupname  asflicense  compile  javac  javadoc  mvninstall  mvnsite  unit  shadedclient  findbugs  checkstyle

uname
Linux 11a912ddd3a7 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / f3f5128

maven
version: Apache Maven 3.3.9

Default Java
1.8.0_191

findbugs
v3.1.0-RC1

mvninstall
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/2/artifact/out/patch-mvninstall-hadoop-tools_hadoop-aws.txt

compile
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/2/artifact/out/patch-compile-root.txt

javac
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/2/artifact/out/patch-compile-root.txt

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/2/artifact/out/diff-checkstyle-root.txt

mvnsite
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/2/artifact/out/patch-mvnsite-hadoop-tools_hadoop-aws.txt

findbugs
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/2/artifact/out/patch-findbugs-hadoop-tools_hadoop-aws.txt

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/2/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/2/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/2/testReport/

Max. process+thread count
1688 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/2/console

Powered by
Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
0
Docker mode activated.

-1
patch
6
https://github.com/apache/hadoop/pull/654 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.

Subsystem
Report/Notes

GITHUB PR
https://github.com/apache/hadoop/pull/654

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/3/console

Powered by
Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

Note, layout of packages & aspects of arch are based on https://github.com/steveloughran/engineering-proposals/blob/master/refactoring-s3a.md , primarily

moving multidelete support into its own module in .impl & testing in isolation

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
0
Docker mode activated.

-1
patch
8
https://github.com/apache/hadoop/pull/654 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.

Subsystem
Report/Notes

GITHUB PR
https://github.com/apache/hadoop/pull/654

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/4/console

Powered by
Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
0
Docker mode activated.

-1
patch
8
https://github.com/apache/hadoop/pull/654 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.

Subsystem
Report/Notes

GITHUB PR
https://github.com/apache/hadoop/pull/654

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/5/console

Powered by
Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
0
Docker mode activated.

-1
patch
6
https://github.com/apache/hadoop/pull/654 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.

Subsystem
Report/Notes

GITHUB PR
https://github.com/apache/hadoop/pull/654

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/6/console

Powered by
Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
0
Docker mode activated.

-1
patch
7
https://github.com/apache/hadoop/pull/654 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.

Subsystem
Report/Notes

GITHUB PR
https://github.com/apache/hadoop/pull/654

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/7/console

Powered by
Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

not applying to trunk...don't understand this. Rebased

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
0
Docker mode activated.

-1
patch
7
https://github.com/apache/hadoop/pull/654 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.

Subsystem
Report/Notes

GITHUB PR
https://github.com/apache/hadoop/pull/654

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/8/console

Powered by
Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

Having weirdness here as yetus can't apply patch. Local attempt (over SFO airport wifi)
> dev-support/bin/smart-apply-patch --project=hadoop GH:654
Processing: GH:654
GITHUB PR #654 is being downloaded at Fri 19 Apr 2019 18:14:35 PDT from
https://github.com/apache/hadoop/pull/654
Patch from GITHUB PR #654 is being downloaded at Fri 19 Apr 2019 18:14:36 PDT from
https://github.com/apache/hadoop/pull/654.patch
ERROR: Aborting! GH:654 cannot be verified.

OK, some bit of the patch history is causing confusion

Checking patch hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/ContractTestUtils.java...
Checking patch hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java...
Checking patch hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInstrumentation.java...
Checking patch hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AUtils.java...
Checking patch hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Statistic.java...
Checking patch hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AFailureHandling.java...
Checking patch hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestUtils.java...
Checking patch hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AExceptionTranslation.java...
Checking patch hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/ITestAssumeRole.java...
Checking patch hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java...
Checking patch hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AUtils.java...
Checking patch hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/MultiObjectDeleteSupport.java...
Checking patch hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/S3Guard.java...
Checking patch hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ExtraAssertions.java...
Checking patch hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AFailureHandling.java...
Checking patch hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestUtils.java...
Checking patch hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AExceptionTranslation.java...
Checking patch hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/ITestAssumeRole.java...
Checking patch hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/ITestPartialRenamesDeletes.java...
Checking patch hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/TestPartialDeleteFailures.java...
Checking patch hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java...
Checking patch hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/MultiObjectDeleteSupport.java...
Checking patch hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AFailureHandling.java...
Checking patch hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/ITestPartialRenamesDeletes.java...
Checking patch hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/TestPartialDeleteFailures.java...
Checking patch hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ExtraAssertions.java => hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/test/ExtraAssertions.java...
error: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ExtraAssertions.java: does not exist in index
Checking patch hadoop-tools/hadoop-aws/pom.xml...
Checking patch hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/RoleTestUtils.java...
Checking patch hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/ITestPartialRenamesDeletes.java...
Checking patch hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/MultiObjectDeleteSupport.java...
Checking patch hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/ITestAssumeRole.java...
Checking patch hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/RoleTestUtils.java...
Checking patch hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/ITestPartialRenamesDeletes.java...
Checking patch hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/TestPartialDeleteFailures.java...
Checking patch hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java...
Checking patch hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/MultiObjectDeleteSupport.java...
Checking patch hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/StoreContext.java...
Checking patch hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AFailureHandling.java...
Checking patch hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/TestPartialDeleteFailures.java...

squashed entire patch into one

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
25
Docker mode activated.

_ Prechecks _

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 8 new or modified test files.

_ trunk Compile Tests _

0
mvndep
61
Maven dependency ordering for branch

+1
mvninstall
1015
trunk passed

+1
compile
972
trunk passed

+1
checkstyle
140
trunk passed

+1
mvnsite
121
trunk passed

+1
shadedclient
988
branch has no errors when building and testing our client artifacts.

+1
findbugs
157
trunk passed

+1
javadoc
106
trunk passed

_ Patch Compile Tests _

0
mvndep
22
Maven dependency ordering for patch

+1
mvninstall
75
the patch passed

+1
compile
913
the patch passed

+1
javac
913
the patch passed

-0
checkstyle
140
root: The patch generated 19 new + 40 unchanged - 0 fixed = 59 total (was 40)

+1
mvnsite
127
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
xml
2
The patch has no ill-formed XML file.

+1
shadedclient
673
patch has no errors when building and testing our client artifacts.

-1
findbugs
67
hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)

-1
javadoc
36
hadoop-tools_hadoop-aws generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1)

_ Other Tests _

+1
unit
509
hadoop-common in the patch passed.

+1
unit
283
hadoop-aws in the patch passed.

+1
asflicense
47
The patch does not generate ASF License warnings.

6680

Reason
Tests

FindBugs
module:hadoop-tools/hadoop-aws

Inconsistent synchronization of org.apache.hadoop.fs.s3a.S3AFileSystem.directoryAllocator; locked 75% of time  Unsynchronized access at S3AFileSystem.java:75% of time  Unsynchronized access at S3AFileSystem.java:[line 2343]

Subsystem
Report/Notes

Docker
Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-654/9/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/654

Optional Tests
dupname  asflicense  compile  javac  javadoc  mvninstall  mvnsite  unit  shadedclient  findbugs  checkstyle  xml

uname
Linux 7577e1cc2fef 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 317fcba

maven
version: Apache Maven 3.3.9

Default Java
1.8.0_191

findbugs
v3.1.0-RC1

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/9/artifact/out/diff-checkstyle-root.txt

findbugs
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/9/artifact/out/new-findbugs-hadoop-tools_hadoop-aws.html

javadoc
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/9/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-aws.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/9/testReport/

Max. process+thread count
1381 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/9/console

Powered by
Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
54
Docker mode activated.

_ Prechecks _

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 8 new or modified test files.

_ trunk Compile Tests _

0
mvndep
318
Maven dependency ordering for branch

+1
mvninstall
1228
trunk passed

+1
compile
986
trunk passed

+1
checkstyle
151
trunk passed

+1
mvnsite
123
trunk passed

+1
shadedclient
1059
branch has no errors when building and testing our client artifacts.

+1
findbugs
156
trunk passed

+1
javadoc
95
trunk passed

_ Patch Compile Tests _

0
mvndep
20
Maven dependency ordering for patch

+1
mvninstall
76
the patch passed

+1
compile
931
the patch passed

-1
javac
931
root generated 2 new + 1481 unchanged - 0 fixed = 1483 total (was 1481)

-0
checkstyle
183
root: The patch generated 25 new + 40 unchanged - 0 fixed = 65 total (was 40)

+1
mvnsite
123
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
xml
1
The patch has no ill-formed XML file.

+1
shadedclient
742
patch has no errors when building and testing our client artifacts.

-1
findbugs
60
hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)

-1
javadoc
31
hadoop-tools_hadoop-aws generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1)

_ Other Tests _

+1
unit
548
hadoop-common in the patch passed.

+1
unit
294
hadoop-aws in the patch passed.

+1
asflicense
42
The patch does not generate ASF License warnings.

7398

Reason
Tests

FindBugs
module:hadoop-tools/hadoop-aws

Inconsistent synchronization of org.apache.hadoop.fs.s3a.S3AFileSystem.directoryAllocator; locked 75% of time  Unsynchronized access at S3AFileSystem.java:75% of time  Unsynchronized access at S3AFileSystem.java:[line 2373]

Subsystem
Report/Notes

Docker
Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-654/10/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/654

Optional Tests
dupname  asflicense  compile  javac  javadoc  mvninstall  mvnsite  unit  shadedclient  findbugs  checkstyle  xml

uname
Linux 457afd609ae2 4.4.0-141-generic #167~14.04.1-Ubuntu SMP Mon Dec 10 13:20:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / a703dae

maven
version: Apache Maven 3.3.9

Default Java
1.8.0_191

findbugs
v3.1.0-RC1

javac
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/10/artifact/out/diff-compile-javac-root.txt

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/10/artifact/out/diff-checkstyle-root.txt

findbugs
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/10/artifact/out/new-findbugs-hadoop-tools_hadoop-aws.html

javadoc
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/10/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-aws.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/10/testReport/

Max. process+thread count
1471 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/10/console

Powered by
Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
55
Docker mode activated.

_ Prechecks _

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 8 new or modified test files.

_ trunk Compile Tests _

0
mvndep
107
Maven dependency ordering for branch

+1
mvninstall
1350
trunk passed

+1
compile
1552
trunk passed

+1
checkstyle
147
trunk passed

+1
mvnsite
125
trunk passed

+1
shadedclient
1022
branch has no errors when building and testing our client artifacts.

+1
findbugs
153
trunk passed

+1
javadoc
97
trunk passed

_ Patch Compile Tests _

0
mvndep
22
Maven dependency ordering for patch

+1
mvninstall
78
the patch passed

+1
compile
962
the patch passed

-1
javac
962
root generated 2 new + 1481 unchanged - 0 fixed = 1483 total (was 1481)

-0
checkstyle
136
root: The patch generated 27 new + 46 unchanged - 0 fixed = 73 total (was 46)

+1
mvnsite
119
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
xml
3
The patch has no ill-formed XML file.

+1
shadedclient
713
patch has no errors when building and testing our client artifacts.

-1
findbugs
60
hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)

-1
javadoc
35
hadoop-tools_hadoop-aws generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1)

_ Other Tests _

+1
unit
549
hadoop-common in the patch passed.

+1
unit
285
hadoop-aws in the patch passed.

+1
asflicense
45
The patch does not generate ASF License warnings.

7799

Reason
Tests

FindBugs
module:hadoop-tools/hadoop-aws

Inconsistent synchronization of org.apache.hadoop.fs.s3a.S3AFileSystem.directoryAllocator; locked 75% of time  Unsynchronized access at S3AFileSystem.java:75% of time  Unsynchronized access at S3AFileSystem.java:[line 2375]

Subsystem
Report/Notes

Docker
Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-654/11/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/654

Optional Tests
dupname  asflicense  compile  javac  javadoc  mvninstall  mvnsite  unit  shadedclient  xml  findbugs  checkstyle

uname
Linux 1f3e91622723 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14 08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / b5dcf64

maven
version: Apache Maven 3.3.9

Default Java
1.8.0_191

findbugs
v3.1.0-RC1

javac
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/11/artifact/out/diff-compile-javac-root.txt

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/11/artifact/out/diff-checkstyle-root.txt

findbugs
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/11/artifact/out/new-findbugs-hadoop-tools_hadoop-aws.html

javadoc
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/11/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-aws.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/11/testReport/

Max. process+thread count
1345 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/11/console

Powered by
Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
43
Docker mode activated.

_ Prechecks _

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 9 new or modified test files.

_ trunk Compile Tests _

0
mvndep
30
Maven dependency ordering for branch

+1
mvninstall
1156
trunk passed

+1
compile
1124
trunk passed

+1
checkstyle
151
trunk passed

+1
mvnsite
123
trunk passed

+1
shadedclient
1036
branch has no errors when building and testing our client artifacts.

+1
findbugs
155
trunk passed

+1
javadoc
93
trunk passed

_ Patch Compile Tests _

0
mvndep
22
Maven dependency ordering for patch

+1
mvninstall
81
the patch passed

+1
compile
1215
the patch passed

-1
javac
1215
root generated 2 new + 1481 unchanged - 0 fixed = 1483 total (was 1481)

-0
checkstyle
165
root: The patch generated 29 new + 50 unchanged - 0 fixed = 79 total (was 50)

+1
mvnsite
159
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
xml
3
The patch has no ill-formed XML file.

+1
shadedclient
807
patch has no errors when building and testing our client artifacts.

-1
findbugs
77
hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)

-1
javadoc
40
hadoop-tools_hadoop-aws generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1)

_ Other Tests _

+1
unit
609
hadoop-common in the patch passed.

+1
unit
302
hadoop-aws in the patch passed.

+1
asflicense
56
The patch does not generate ASF License warnings.

7660

Reason
Tests

FindBugs
module:hadoop-tools/hadoop-aws

Inconsistent synchronization of org.apache.hadoop.fs.s3a.S3AFileSystem.directoryAllocator; locked 75% of time  Unsynchronized access at S3AFileSystem.java:75% of time  Unsynchronized access at S3AFileSystem.java:[line 2375]

Subsystem
Report/Notes

Docker
Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-654/12/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/654

Optional Tests
dupname  asflicense  compile  javac  javadoc  mvninstall  mvnsite  unit  shadedclient  xml  findbugs  checkstyle

uname
Linux 73f5d8472364 4.4.0-141-generic #167~14.04.1-Ubuntu SMP Mon Dec 10 13:20:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / b5dcf64

maven
version: Apache Maven 3.3.9

Default Java
1.8.0_191

findbugs
v3.1.0-RC1

javac
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/12/artifact/out/diff-compile-javac-root.txt

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/12/artifact/out/diff-checkstyle-root.txt

findbugs
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/12/artifact/out/new-findbugs-hadoop-tools_hadoop-aws.html

javadoc
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/12/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-aws.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/12/testReport/

Max. process+thread count
1495 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/12/console

Powered by
Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
56
Docker mode activated.

_ Prechecks _

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 10 new or modified test files.

_ trunk Compile Tests _

0
mvndep
110
Maven dependency ordering for branch

+1
mvninstall
1245
trunk passed

+1
compile
1016
trunk passed

+1
checkstyle
144
trunk passed

+1
mvnsite
121
trunk passed

+1
shadedclient
1047
branch has no errors when building and testing our client artifacts.

+1
findbugs
152
trunk passed

+1
javadoc
94
trunk passed

_ Patch Compile Tests _

0
mvndep
21
Maven dependency ordering for patch

+1
mvninstall
75
the patch passed

+1
compile
990
the patch passed

-1
javac
990
root generated 2 new + 1481 unchanged - 0 fixed = 1483 total (was 1481)

-0
checkstyle
148
root: The patch generated 30 new + 54 unchanged - 0 fixed = 84 total (was 54)

+1
mvnsite
128
the patch passed

-1
whitespace
0
The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply

+1
xml
3
The patch has no ill-formed XML file.

+1
shadedclient
724
patch has no errors when building and testing our client artifacts.

-1
findbugs
71
hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)

-1
javadoc
28
hadoop-tools_hadoop-aws generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1)

_ Other Tests _

+1
unit
603
hadoop-common in the patch passed.

+1
unit
304
hadoop-aws in the patch passed.

+1
asflicense
43
The patch does not generate ASF License warnings.

7295

Reason
Tests

FindBugs
module:hadoop-tools/hadoop-aws

Inconsistent synchronization of org.apache.hadoop.fs.s3a.S3AFileSystem.directoryAllocator; locked 75% of time  Unsynchronized access at S3AFileSystem.java:75% of time  Unsynchronized access at S3AFileSystem.java:[line 2375]

Subsystem
Report/Notes

Docker
Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-654/13/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/654

Optional Tests
dupname  asflicense  compile  javac  javadoc  mvninstall  mvnsite  unit  shadedclient  findbugs  checkstyle  xml

uname
Linux 958688d5bfd6 4.4.0-144-generic #170~14.04.1-Ubuntu SMP Mon Mar 18 15:02:05 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / b5dcf64

maven
version: Apache Maven 3.3.9

Default Java
1.8.0_191

findbugs
v3.1.0-RC1

javac
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/13/artifact/out/diff-compile-javac-root.txt

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/13/artifact/out/diff-checkstyle-root.txt

whitespace
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/13/artifact/out/whitespace-eol.txt

findbugs
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/13/artifact/out/new-findbugs-hadoop-tools_hadoop-aws.html

javadoc
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/13/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-aws.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/13/testReport/

Max. process+thread count
1699 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/13/console

Powered by
Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
25
Docker mode activated.

_ Prechecks _

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 10 new or modified test files.

_ trunk Compile Tests _

0
mvndep
69
Maven dependency ordering for branch

+1
mvninstall
1071
trunk passed

+1
compile
1085
trunk passed

+1
checkstyle
143
trunk passed

+1
mvnsite
120
trunk passed

+1
shadedclient
945
branch has no errors when building and testing our client artifacts.

+1
findbugs
156
trunk passed

+1
javadoc
97
trunk passed

_ Patch Compile Tests _

0
mvndep
19
Maven dependency ordering for patch

+1
mvninstall
74
the patch passed

+1
compile
956
the patch passed

-1
javac
956
root generated 2 new + 1481 unchanged - 0 fixed = 1483 total (was 1481)

-0
checkstyle
140
root: The patch generated 31 new + 54 unchanged - 0 fixed = 85 total (was 54)

+1
mvnsite
127
the patch passed

-1
whitespace
0
The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply

+1
xml
3
The patch has no ill-formed XML file.

+1
shadedclient
648
patch has no errors when building and testing our client artifacts.

-1
findbugs
66
hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)

+1
javadoc
105
the patch passed

_ Other Tests _

+1
unit
505
hadoop-common in the patch passed.

+1
unit
285
hadoop-aws in the patch passed.

+1
asflicense
44
The patch does not generate ASF License warnings.

6803

Reason
Tests

FindBugs
module:hadoop-tools/hadoop-aws

Inconsistent synchronization of org.apache.hadoop.fs.s3a.S3AFileSystem.directoryAllocator; locked 75% of time  Unsynchronized access at S3AFileSystem.java:75% of time  Unsynchronized access at S3AFileSystem.java:[line 2375]

Subsystem
Report/Notes

Docker
Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-654/14/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/654

Optional Tests
dupname  asflicense  compile  javac  javadoc  mvninstall  mvnsite  unit  shadedclient  findbugs  checkstyle  xml

uname
Linux 35c3e9a3b1c2 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / b5dcf64

maven
version: Apache Maven 3.3.9

Default Java
1.8.0_191

findbugs
v3.1.0-RC1

javac
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/14/artifact/out/diff-compile-javac-root.txt

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/14/artifact/out/diff-checkstyle-root.txt

whitespace
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/14/artifact/out/whitespace-eol.txt

findbugs
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/14/artifact/out/new-findbugs-hadoop-tools_hadoop-aws.html

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/14/testReport/

Max. process+thread count
1382 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/14/console

Powered by
Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
42
Docker mode activated.

_ Prechecks _

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 10 new or modified test files.

_ trunk Compile Tests _

0
mvndep
66
Maven dependency ordering for branch

+1
mvninstall
1121
trunk passed

+1
compile
979
trunk passed

+1
checkstyle
143
trunk passed

+1
mvnsite
123
trunk passed

+1
shadedclient
1071
branch has no errors when building and testing our client artifacts.

+1
findbugs
164
trunk passed

+1
javadoc
94
trunk passed

_ Patch Compile Tests _

0
mvndep
23
Maven dependency ordering for patch

-1
mvninstall
22
hadoop-aws in the patch failed.

-1
compile
963
root in the patch failed.

-1
javac
963
root in the patch failed.

-0
checkstyle
144
root: The patch generated 31 new + 54 unchanged - 0 fixed = 85 total (was 54)

-1
mvnsite
35
hadoop-aws in the patch failed.

-1
whitespace
0
The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply

+1
xml
4
The patch has no ill-formed XML file.

+1
shadedclient
768
patch has no errors when building and testing our client artifacts.

-1
findbugs
32
hadoop-aws in the patch failed.

+1
javadoc
91
the patch passed

_ Other Tests _

+1
unit
526
hadoop-common in the patch passed.

-1
unit
33
hadoop-aws in the patch failed.

+1
asflicense
40
The patch does not generate ASF License warnings.

6720

Subsystem
Report/Notes

Docker
Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-654/15/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/654

Optional Tests
dupname  asflicense  compile  javac  javadoc  mvninstall  mvnsite  unit  shadedclient  findbugs  checkstyle  xml

uname
Linux 49d281d33d9a 4.4.0-141-generic #167~14.04.1-Ubuntu SMP Mon Dec 10 13:20:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 556eafd

maven
version: Apache Maven 3.3.9

Default Java
1.8.0_191

findbugs
v3.1.0-RC1

mvninstall
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/15/artifact/out/patch-mvninstall-hadoop-tools_hadoop-aws.txt

compile
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/15/artifact/out/patch-compile-root.txt

javac
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/15/artifact/out/patch-compile-root.txt

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/15/artifact/out/diff-checkstyle-root.txt

mvnsite
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/15/artifact/out/patch-mvnsite-hadoop-tools_hadoop-aws.txt

whitespace
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/15/artifact/out/whitespace-eol.txt

findbugs
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/15/artifact/out/patch-findbugs-hadoop-tools_hadoop-aws.txt

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/15/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/15/testReport/

Max. process+thread count
1503 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/15/console

Powered by
Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
28
Docker mode activated.

_ Prechecks _

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 11 new or modified test files.

_ trunk Compile Tests _

0
mvndep
65
Maven dependency ordering for branch

+1
mvninstall
1040
trunk passed

+1
compile
994
trunk passed

+1
checkstyle
142
trunk passed

+1
mvnsite
130
trunk passed

+1
shadedclient
1003
branch has no errors when building and testing our client artifacts.

+1
findbugs
156
trunk passed

+1
javadoc
105
trunk passed

_ Patch Compile Tests _

0
mvndep
23
Maven dependency ordering for patch

+1
mvninstall
77
the patch passed

+1
compile
949
the patch passed

-1
javac
949
root generated 2 new + 1481 unchanged - 0 fixed = 1483 total (was 1481)

-0
checkstyle
174
root: The patch generated 31 new + 55 unchanged - 0 fixed = 86 total (was 55)

+1
mvnsite
119
the patch passed

-1
whitespace
0
The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply

+1
xml
3
The patch has no ill-formed XML file.

+1
shadedclient
631
patch has no errors when building and testing our client artifacts.

+1
findbugs
174
the patch passed

+1
javadoc
96
the patch passed

_ Other Tests _

+1
unit
501
hadoop-common in the patch passed.

+1
unit
280
hadoop-aws in the patch passed.

+1
asflicense
48
The patch does not generate ASF License warnings.

6729

Subsystem
Report/Notes

Docker
Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-654/16/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/654

Optional Tests
dupname  asflicense  compile  javac  javadoc  mvninstall  mvnsite  unit  shadedclient  findbugs  checkstyle  xml

uname
Linux 4bb401b3b051 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 556eafd

maven
version: Apache Maven 3.3.9

Default Java
1.8.0_191

findbugs
v3.1.0-RC1

javac
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/16/artifact/out/diff-compile-javac-root.txt

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/16/artifact/out/diff-checkstyle-root.txt

whitespace
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/16/artifact/out/whitespace-eol.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/16/testReport/

Max. process+thread count
1381 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/16/console

Powered by
Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
46
Docker mode activated.

_ Prechecks _

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 11 new or modified test files.

_ trunk Compile Tests _

0
mvndep
22
Maven dependency ordering for branch

+1
mvninstall
1263
trunk passed

+1
compile
1025
trunk passed

+1
checkstyle
145
trunk passed

+1
mvnsite
127
trunk passed

+1
shadedclient
1053
branch has no errors when building and testing our client artifacts.

+1
findbugs
154
trunk passed

+1
javadoc
94
trunk passed

_ Patch Compile Tests _

0
mvndep
22
Maven dependency ordering for patch

-1
mvninstall
17
hadoop-aws in the patch failed.

-1
compile
873
root in the patch failed.

-1
javac
873
root in the patch failed.

-0
checkstyle
136
The patch fails to run checkstyle in root

-1
mvnsite
31
hadoop-aws in the patch failed.

-1
whitespace
0
The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply

+1
xml
2
The patch has no ill-formed XML file.

+1
shadedclient
736
patch has no errors when building and testing our client artifacts.

-1
findbugs
29
hadoop-aws in the patch failed.

-1
javadoc
25
hadoop-aws in the patch failed.

_ Other Tests _

+1
unit
535
hadoop-common in the patch passed.

-1
unit
27
hadoop-aws in the patch failed.

+1
asflicense
41
The patch does not generate ASF License warnings.

6660

Subsystem
Report/Notes

Docker
Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-654/17/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/654

Optional Tests
dupname  asflicense  compile  javac  javadoc  mvninstall  mvnsite  unit  shadedclient  findbugs  checkstyle  xml

uname
Linux a870475580a0 4.4.0-141-generic #167~14.04.1-Ubuntu SMP Mon Dec 10 13:20:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 556eafd

maven
version: Apache Maven 3.3.9

Default Java
1.8.0_191

findbugs
v3.1.0-RC1

mvninstall
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/17/artifact/out/patch-mvninstall-hadoop-tools_hadoop-aws.txt

compile
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/17/artifact/out/patch-compile-root.txt

javac
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/17/artifact/out/patch-compile-root.txt

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/17/artifact/out//testptch/patchprocess/maven-patch-checkstyle-root.txt

mvnsite
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/17/artifact/out/patch-mvnsite-hadoop-tools_hadoop-aws.txt

whitespace
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/17/artifact/out/whitespace-eol.txt

findbugs
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/17/artifact/out/patch-findbugs-hadoop-tools_hadoop-aws.txt

javadoc
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/17/artifact/out/patch-javadoc-hadoop-tools_hadoop-aws.txt

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/17/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/17/testReport/

Max. process+thread count
1492 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/17/console

Powered by
Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

looks like the last version doesn't compile. oops

@steveloughran how much performance improvement is expected after this change is taken in?

@harshavardhana I'm not making any promises of performance. Already on large files the AWS SDK transfer manager breaks things up into 128MB units, so what we gain here is the copying files in parallel above that, which is probably best for smaller files.
I'm looking at resilience of failures and S3Guard consistency —that's the key driver and takes priority over the speedups.
Note that time to rename on AWS will always been O(data), even if we do more files in parallel; it won't be atomic. For high performance commit algorithms you need something like the S3A committers, or S3-first data structures like Apache Iceberg

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
42
Docker mode activated.

_ Prechecks _

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 11 new or modified test files.

_ trunk Compile Tests _

0
mvndep
29
Maven dependency ordering for branch

+1
mvninstall
1130
trunk passed

+1
compile
977
trunk passed

+1
checkstyle
146
trunk passed

+1
mvnsite
121
trunk passed

+1
shadedclient
1031
branch has no errors when building and testing our client artifacts.

+1
findbugs
151
trunk passed

+1
javadoc
93
trunk passed

_ Patch Compile Tests _

0
mvndep
21
Maven dependency ordering for patch

+1
mvninstall
78
the patch passed

+1
compile
928
the patch passed

+1
javac
928
the patch passed

-0
checkstyle
145
root: The patch generated 43 new + 55 unchanged - 0 fixed = 98 total (was 55)

+1
mvnsite
120
the patch passed

-1
whitespace
0
The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply

+1
xml
2
The patch has no ill-formed XML file.

+1
shadedclient
739
patch has no errors when building and testing our client artifacts.

-1
findbugs
67
hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)

+1
javadoc
93
the patch passed

_ Other Tests _

+1
unit
619
hadoop-common in the patch passed.

+1
unit
292
hadoop-aws in the patch passed.

+1
asflicense
48
The patch does not generate ASF License warnings.

6979

Reason
Tests

FindBugs
module:hadoop-tools/hadoop-aws

Return value of RenameOperation.renameFailed(Exception) ignored, but method has no side effect  At DelayedUpdateRenameOperation.java:but method has no side effect  At DelayedUpdateRenameOperation.java:[line 129]

Subsystem
Report/Notes

Docker
Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-654/18/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/654

Optional Tests
dupname  asflicense  compile  javac  javadoc  mvninstall  mvnsite  unit  shadedclient  findbugs  checkstyle  xml

uname
Linux 018c629fe40d 4.4.0-141-generic #167~14.04.1-Ubuntu SMP Mon Dec 10 13:20:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 4877f0a

maven
version: Apache Maven 3.3.9

Default Java
1.8.0_191

findbugs
v3.1.0-RC1

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/18/artifact/out/diff-checkstyle-root.txt

whitespace
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/18/artifact/out/whitespace-eol.txt

findbugs
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/18/artifact/out/new-findbugs-hadoop-tools_hadoop-aws.html

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/18/testReport/

Max. process+thread count
1501 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/18/console

Powered by
Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
43
Docker mode activated.

_ Prechecks _

+1
dupname
1
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 11 new or modified test files.

_ trunk Compile Tests _

0
mvndep
21
Maven dependency ordering for branch

+1
mvninstall
1175
trunk passed

+1
compile
2384
trunk passed

+1
checkstyle
351
trunk passed

+1
mvnsite
409
trunk passed

+1
shadedclient
1852
branch has no errors when building and testing our client artifacts.

+1
javadoc
114
trunk passed

0
spotbugs
76
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
198
trunk passed

_ Patch Compile Tests _

0
mvndep
47
Maven dependency ordering for patch

+1
mvninstall
86
the patch passed

+1
compile
1003
the patch passed

+1
javac
1003
the patch passed

-0
checkstyle
145
root: The patch generated 44 new + 55 unchanged - 0 fixed = 99 total (was 55)

+1
mvnsite
117
the patch passed

-1
whitespace
0
The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply

+1
xml
2
The patch has no ill-formed XML file.

+1
shadedclient
729
patch has no errors when building and testing our client artifacts.

-1
javadoc
30
hadoop-tools_hadoop-aws generated 2 new + 1 unchanged - 0 fixed = 3 total (was 1)

+1
findbugs
199
the patch passed

_ Other Tests _

+1
unit
511
hadoop-common in the patch passed.

+1
unit
289
hadoop-aws in the patch passed.

+1
asflicense
46
The patch does not generate ASF License warnings.

9328

Subsystem
Report/Notes

Docker
Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-654/19/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/654

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml

uname
Linux abbf9992c7a6 4.4.0-141-generic #167~14.04.1-Ubuntu SMP Mon Dec 10 13:20:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / d6b7609

Default Java
1.8.0_191

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/19/artifact/out/diff-checkstyle-root.txt

whitespace
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/19/artifact/out/whitespace-eol.txt

javadoc
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/19/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-aws.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/19/testReport/

Max. process+thread count
1461 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/19/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
32
Docker mode activated.

_ Prechecks _

+1
dupname
1
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 11 new or modified test files.

_ trunk Compile Tests _

0
mvndep
23
Maven dependency ordering for branch

+1
mvninstall
1164
trunk passed

+1
compile
1230
trunk passed

+1
checkstyle
140
trunk passed

+1
mvnsite
127
trunk passed

+1
shadedclient
961
branch has no errors when building and testing our client artifacts.

+1
javadoc
78
trunk passed

0
spotbugs
56
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
168
trunk passed

-0
patch
91
Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.

_ Patch Compile Tests _

0
mvndep
25
Maven dependency ordering for patch

+1
mvninstall
73
the patch passed

+1
compile
958
the patch passed

+1
javac
958
the patch passed

-0
checkstyle
145
root: The patch generated 48 new + 55 unchanged - 0 fixed = 103 total (was 55)

+1
mvnsite
127
the patch passed

-1
whitespace
0
The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply

+1
xml
2
The patch has no ill-formed XML file.

+1
shadedclient
688
patch has no errors when building and testing our client artifacts.

-1
javadoc
38
hadoop-tools_hadoop-aws generated 2 new + 1 unchanged - 0 fixed = 3 total (was 1)

+1
findbugs
203
the patch passed

_ Other Tests _

+1
unit
521
hadoop-common in the patch passed.

+1
unit
271
hadoop-aws in the patch passed.

+1
asflicense
45
The patch does not generate ASF License warnings.

7215

Subsystem
Report/Notes

Docker
Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-654/20/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/654

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml

uname
Linux 9371b4f89817 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / f194540

Default Java
1.8.0_191

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/20/artifact/out/diff-checkstyle-root.txt

whitespace
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/20/artifact/out/whitespace-eol.txt

javadoc
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/20/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-aws.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/20/testReport/

Max. process+thread count
1380 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/20/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
45
Docker mode activated.

_ Prechecks _

+1
dupname
1
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 12 new or modified test files.

_ trunk Compile Tests _

0
mvndep
68
Maven dependency ordering for branch

+1
mvninstall
1155
trunk passed

+1
compile
1069
trunk passed

+1
checkstyle
152
trunk passed

+1
mvnsite
121
trunk passed

+1
shadedclient
1061
branch has no errors when building and testing our client artifacts.

+1
javadoc
94
trunk passed

0
spotbugs
63
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
185
trunk passed

-0
patch
98
Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.

_ Patch Compile Tests _

0
mvndep
21
Maven dependency ordering for patch

+1
mvninstall
78
the patch passed

+1
compile
966
the patch passed

+1
javac
966
the patch passed

-0
checkstyle
148
root: The patch generated 54 new + 54 unchanged - 1 fixed = 108 total (was 55)

+1
mvnsite
118
the patch passed

-1
whitespace
0
The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply

+1
xml
2
The patch has no ill-formed XML file.

+1
shadedclient
734
patch has no errors when building and testing our client artifacts.

-1
javadoc
31
hadoop-tools_hadoop-aws generated 2 new + 1 unchanged - 0 fixed = 3 total (was 1)

-1
findbugs
73
hadoop-tools/hadoop-aws generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)

_ Other Tests _

+1
unit
513
hadoop-common in the patch passed.

+1
unit
277
hadoop-aws in the patch passed.

+1
asflicense
45
The patch does not generate ASF License warnings.

7531

Reason
Tests

FindBugs
module:hadoop-tools/hadoop-aws

org.apache.hadoop.fs.s3a.s3guard.PathOrderComparators$TopmostFirst implements Comparator but not Serializable  At PathOrderComparators.java:Serializable  At PathOrderComparators.java:[lines 52-72]

org.apache.hadoop.fs.s3a.s3guard.PathOrderComparators$TopmostLast implements Comparator but not Serializable  At PathOrderComparators.java:Serializable  At PathOrderComparators.java:[lines 76-87]

Subsystem
Report/Notes

Docker
Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-654/21/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/654

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml

uname
Linux f565774759ab 4.4.0-141-generic #167~14.04.1-Ubuntu SMP Mon Dec 10 13:20:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / fb7c1ca

Default Java
1.8.0_191

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/21/artifact/out/diff-checkstyle-root.txt

whitespace
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/21/artifact/out/whitespace-eol.txt

javadoc
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/21/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-aws.txt

findbugs
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/21/artifact/out/new-findbugs-hadoop-tools_hadoop-aws.html

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/21/testReport/

Max. process+thread count
1430 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/21/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

This patch is now of a state where it is ready for review

it's going to have to be changed to keep up with the S3Guard versioning patches so I'm hoping to nurture those in, but the incompatibilities are related to the type of FileStatus passed around & general git merge problems, rather than functional conflict.

There's one production side improvement I'd like to add.
This new patch does the move incrementally: whenever you add a file we call s3guard.move(null, dest-file-status) to add the destination (and ancestors), on a bulk delete we update the deletes,
But: that move(List, List) call creates all the parent paths, relying on a hash table to avoid duplicates,. Once you move to single-file additions then both that and metastore.put() are creating too many entries due to their need to meet the goal of "no duplicates". I want to restore the original behavior by passing in to the metastore the map being built up in the rename tracker, so it knows what already exists. (Note: this all needs to be done thread safely, so that when > 1 copy completes...I don't want the locks for that to also block other updates to the metastore)
This isn't a functionality change, it's a performance and cost improvement, one designed to keep those DDB write IOPs down.
Please take a look at the code as it stands.
The architecture is based on my refactoring S3A doc -the new classes are designed to work with the new StoreContext class; the metastore moves with this.

+testing. S3 ireland (versioned store), with/without auth. Also: local. I've expressed interest in removing the local mode as its a distraction in tests (it doesn't match production, what does it prove?): the changes here have the potential to amplify that mismatch.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
47
Docker mode activated.

_ Prechecks _

+1
dupname
1
No case conflicting files found.

+1
@author
1
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 15 new or modified test files.

_ trunk Compile Tests _

0
mvndep
27
Maven dependency ordering for branch

+1
mvninstall
1137
trunk passed

+1
compile
1293
trunk passed

+1
checkstyle
148
trunk passed

+1
mvnsite
127
trunk passed

+1
shadedclient
1060
branch has no errors when building and testing our client artifacts.

+1
javadoc
94
trunk passed

0
spotbugs
65
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
184
trunk passed

-0
patch
100
Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.

_ Patch Compile Tests _

0
mvndep
21
Maven dependency ordering for patch

+1
mvninstall
76
the patch passed

+1
compile
1010
the patch passed

+1
javac
1010
the patch passed

-0
checkstyle
151
root: The patch generated 51 new + 64 unchanged - 0 fixed = 115 total (was 64)

+1
mvnsite
119
the patch passed

-1
whitespace
0
The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply

+1
xml
2
The patch has no ill-formed XML file.

+1
shadedclient
735
patch has no errors when building and testing our client artifacts.

-1
javadoc
32
hadoop-tools_hadoop-aws generated 2 new + 1 unchanged - 0 fixed = 3 total (was 1)

-1
findbugs
73
hadoop-tools/hadoop-aws generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)

_ Other Tests _

+1
unit
556
hadoop-common in the patch passed.

+1
unit
301
hadoop-aws in the patch passed.

-1
asflicense
56
The patch generated 1 ASF License warnings.

7451

Reason
Tests

FindBugs
module:hadoop-tools/hadoop-aws

org.apache.hadoop.fs.s3a.s3guard.PathOrderComparators$TopmostFirst implements Comparator but not Serializable  At PathOrderComparators.java:Serializable  At PathOrderComparators.java:[lines 69-89]

org.apache.hadoop.fs.s3a.s3guard.PathOrderComparators$TopmostLast implements Comparator but not Serializable  At PathOrderComparators.java:Serializable  At PathOrderComparators.java:[lines 98-109]

Subsystem
Report/Notes

Docker
Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-654/22/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/654

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml

uname
Linux d6fa208c0967 4.4.0-144-generic #170~14.04.1-Ubuntu SMP Mon Mar 18 15:02:05 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 8ecbf61

Default Java
1.8.0_191

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/22/artifact/out/diff-checkstyle-root.txt

whitespace
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/22/artifact/out/whitespace-eol.txt

javadoc
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/22/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-aws.txt

findbugs
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/22/artifact/out/new-findbugs-hadoop-tools_hadoop-aws.html

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/22/testReport/

asflicense
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/22/artifact/out/patch-asflicense-problems.txt

Max. process+thread count
1446 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/22/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
36
Docker mode activated.

_ Prechecks _

+1
dupname
1
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 18 new or modified test files.

_ trunk Compile Tests _

0
mvndep
65
Maven dependency ordering for branch

+1
mvninstall
1140
trunk passed

+1
compile
1121
trunk passed

+1
checkstyle
143
trunk passed

+1
mvnsite
120
trunk passed

+1
shadedclient
978
branch has no errors when building and testing our client artifacts.

+1
javadoc
91
trunk passed

0
spotbugs
64
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
191
trunk passed

-0
patch
95
Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.

_ Patch Compile Tests _

0
mvndep
26
Maven dependency ordering for patch

+1
mvninstall
80
the patch passed

+1
compile
1048
the patch passed

+1
javac
1048
the patch passed

-0
checkstyle
141
root: The patch generated 54 new + 69 unchanged - 1 fixed = 123 total (was 70)

+1
mvnsite
120
the patch passed

-1
whitespace
0
The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply

+1
xml
3
The patch has no ill-formed XML file.

+1
shadedclient
666
patch has no errors when building and testing our client artifacts.

-1
javadoc
30
hadoop-tools_hadoop-aws generated 3 new + 1 unchanged - 0 fixed = 4 total (was 1)

-1
findbugs
73
hadoop-tools/hadoop-aws generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)

_ Other Tests _

+1
unit
511
hadoop-common in the patch passed.

-1
unit
3651
hadoop-aws in the patch failed.

+1
asflicense
41
The patch does not generate ASF License warnings.

10463

Reason
Tests

FindBugs
module:hadoop-tools/hadoop-aws

org.apache.hadoop.fs.s3a.s3guard.PathOrderComparators$TopmostFirst implements Comparator but not Serializable  At PathOrderComparators.java:Serializable  At PathOrderComparators.java:[lines 69-89]

org.apache.hadoop.fs.s3a.s3guard.PathOrderComparators$TopmostLast implements Comparator but not Serializable  At PathOrderComparators.java:Serializable  At PathOrderComparators.java:[lines 98-109]

Failed junit tests
hadoop.fs.s3a.commit.staging.TestStagingCommitter

Subsystem
Report/Notes

Docker
Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-654/23/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/654

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml

uname
Linux 7a7d8342fcbb 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 9b0aace

Default Java
1.8.0_191

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/23/artifact/out/diff-checkstyle-root.txt

whitespace
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/23/artifact/out/whitespace-eol.txt

javadoc
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/23/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-aws.txt

findbugs
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/23/artifact/out/new-findbugs-hadoop-tools_hadoop-aws.html

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/23/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/23/testReport/

Max. process+thread count
1463 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/23/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

Test failure: interesting. Will fix.
For people watching this, rename seems happy, but I don't like the way every put() is creating the entire list of parent entries and pushing them out, even for bulk operations like rename into a directory tree or commits into a tree. Both of these generate needless write load, one of O(files + depth(files)
plan: the notion of a rename operation is expanded to cover a "bulk add operation" which is also initiated for the commits" -it'll track which entries have already been created and not reissue them. There's a separate JIRA for commit load; there's enough commonality to cover it here too.
Not going to touch this until May 14; doing review of other people's work and some backporting until then
reviews of this PR as is welcome!

just pushed out my addition of a bulk update context all the way through the commit operation; it's not yet wired up for the commit phase and too many needless PUTs are being generated.
This does represent my model of how we can have an ongoing rename/commit operation where we know not to create duplicate parents
Not incorporated andrew's comments...will do that separately.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
46
Docker mode activated.

_ Prechecks _

+1
dupname
1
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 22 new or modified test files.

_ trunk Compile Tests _

0
mvndep
66
Maven dependency ordering for branch

+1
mvninstall
1144
trunk passed

+1
compile
1064
trunk passed

+1
checkstyle
148
trunk passed

+1
mvnsite
123
trunk passed

+1
shadedclient
1056
branch has no errors when building and testing our client artifacts.

+1
javadoc
93
trunk passed

0
spotbugs
63
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
185
trunk passed

-0
patch
99
Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.

_ Patch Compile Tests _

0
mvndep
20
Maven dependency ordering for patch

+1
mvninstall
76
the patch passed

+1
compile
978
the patch passed

+1
javac
978
the patch passed

-0
checkstyle
149
root: The patch generated 62 new + 85 unchanged - 2 fixed = 147 total (was 87)

+1
mvnsite
121
the patch passed

-1
whitespace
0
The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply

+1
xml
2
The patch has no ill-formed XML file.

+1
shadedclient
735
patch has no errors when building and testing our client artifacts.

-1
javadoc
32
hadoop-tools_hadoop-aws generated 4 new + 1 unchanged - 0 fixed = 5 total (was 1)

-1
findbugs
71
hadoop-tools/hadoop-aws generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)

_ Other Tests _

+1
unit
537
hadoop-common in the patch passed.

+1
unit
287
hadoop-aws in the patch passed.

+1
asflicense
44
The patch does not generate ASF License warnings.

8782

Reason
Tests

FindBugs
module:hadoop-tools/hadoop-aws

org.apache.hadoop.fs.s3a.s3guard.PathOrderComparators$TopmostFirst implements Comparator but not Serializable  At PathOrderComparators.java:Serializable  At PathOrderComparators.java:[lines 69-89]

org.apache.hadoop.fs.s3a.s3guard.PathOrderComparators$TopmostLast implements Comparator but not Serializable  At PathOrderComparators.java:Serializable  At PathOrderComparators.java:[lines 98-109]

Subsystem
Report/Notes

Docker
Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-654/24/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/654

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml

uname
Linux 212b4c9f4b9d 4.4.0-141-generic #167~14.04.1-Ubuntu SMP Mon Dec 10 13:20:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 9569015

Default Java
1.8.0_191

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/24/artifact/out/diff-checkstyle-root.txt

whitespace
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/24/artifact/out/whitespace-eol.txt

javadoc
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/24/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-aws.txt

findbugs
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/24/artifact/out/new-findbugs-hadoop-tools_hadoop-aws.html

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/24/testReport/

Max. process+thread count
1489 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/24/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
47
Docker mode activated.

_ Prechecks _

+1
dupname
1
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 22 new or modified test files.

_ trunk Compile Tests _

0
mvndep
68
Maven dependency ordering for branch

+1
mvninstall
1159
trunk passed

+1
compile
1075
trunk passed

+1
checkstyle
147
trunk passed

+1
mvnsite
124
trunk passed

+1
shadedclient
1037
branch has no errors when building and testing our client artifacts.

+1
javadoc
96
trunk passed

0
spotbugs
66
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
184
trunk passed

-0
patch
101
Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.

_ Patch Compile Tests _

0
mvndep
21
Maven dependency ordering for patch

+1
mvninstall
78
the patch passed

+1
compile
979
the patch passed

+1
javac
979
the patch passed

-0
checkstyle
152
root: The patch generated 61 new + 85 unchanged - 2 fixed = 146 total (was 87)

+1
mvnsite
120
the patch passed

-1
whitespace
0
The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply

+1
xml
3
The patch has no ill-formed XML file.

+1
shadedclient
752
patch has no errors when building and testing our client artifacts.

-1
javadoc
31
hadoop-tools_hadoop-aws generated 4 new + 1 unchanged - 0 fixed = 5 total (was 1)

-1
findbugs
74
hadoop-tools/hadoop-aws generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)

_ Other Tests _

+1
unit
517
hadoop-common in the patch passed.

+1
unit
292
hadoop-aws in the patch passed.

+1
asflicense
47
The patch does not generate ASF License warnings.

7204

Reason
Tests

FindBugs
module:hadoop-tools/hadoop-aws

org.apache.hadoop.fs.s3a.s3guard.PathOrderComparators$TopmostFirst implements Comparator but not Serializable  At PathOrderComparators.java:Serializable  At PathOrderComparators.java:[lines 69-89]

org.apache.hadoop.fs.s3a.s3guard.PathOrderComparators$TopmostLast implements Comparator but not Serializable  At PathOrderComparators.java:Serializable  At PathOrderComparators.java:[lines 98-109]

Subsystem
Report/Notes

Docker
Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-654/25/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/654

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml

uname
Linux 6d224096b2df 4.4.0-141-generic #167~14.04.1-Ubuntu SMP Mon Dec 10 13:20:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / c15b3bc

Default Java
1.8.0_212

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/25/artifact/out/diff-checkstyle-root.txt

whitespace
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/25/artifact/out/whitespace-eol.txt

javadoc
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/25/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-aws.txt

findbugs
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/25/artifact/out/new-findbugs-hadoop-tools_hadoop-aws.html

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/25/testReport/

Max. process+thread count
1498 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/25/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
33
Docker mode activated.

_ Prechecks _

+1
dupname
1
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 22 new or modified test files.

_ trunk Compile Tests _

0
mvndep
63
Maven dependency ordering for branch

+1
mvninstall
1060
trunk passed

+1
compile
1077
trunk passed

+1
checkstyle
130
trunk passed

+1
mvnsite
117
trunk passed

+1
shadedclient
960
branch has no errors when building and testing our client artifacts.

+1
javadoc
80
trunk passed

0
spotbugs
58
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
172
trunk passed

-0
patch
90
Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.

_ Patch Compile Tests _

0
mvndep
20
Maven dependency ordering for patch

+1
mvninstall
72
the patch passed

+1
compile
986
the patch passed

+1
javac
986
the patch passed

-0
checkstyle
141
root: The patch generated 61 new + 85 unchanged - 2 fixed = 146 total (was 87)

+1
mvnsite
116
the patch passed

-1
whitespace
0
The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply

+1
xml
3
The patch has no ill-formed XML file.

+1
shadedclient
674
patch has no errors when building and testing our client artifacts.

-1
javadoc
28
hadoop-tools_hadoop-aws generated 4 new + 1 unchanged - 0 fixed = 5 total (was 1)

-1
findbugs
75
hadoop-tools/hadoop-aws generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)

_ Other Tests _

+1
unit
505
hadoop-common in the patch passed.

+1
unit
286
hadoop-aws in the patch passed.

+1
asflicense
47
The patch does not generate ASF License warnings.

8679

Reason
Tests

FindBugs
module:hadoop-tools/hadoop-aws

org.apache.hadoop.fs.s3a.s3guard.PathOrderComparators$TopmostFirst implements Comparator but not Serializable  At PathOrderComparators.java:Serializable  At PathOrderComparators.java:[lines 69-89]

org.apache.hadoop.fs.s3a.s3guard.PathOrderComparators$TopmostLast implements Comparator but not Serializable  At PathOrderComparators.java:Serializable  At PathOrderComparators.java:[lines 98-109]

Subsystem
Report/Notes

Docker
Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-654/26/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/654

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml

uname
Linux 49e2b56470fa 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 4cb3da6

Default Java
1.8.0_212

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/26/artifact/out/diff-checkstyle-root.txt

whitespace
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/26/artifact/out/whitespace-eol.txt

javadoc
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/26/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-aws.txt

findbugs
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/26/artifact/out/new-findbugs-hadoop-tools_hadoop-aws.html

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/26/testReport/

Max. process+thread count
1471 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/26/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
43
Docker mode activated.

_ Prechecks _

+1
dupname
2
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 22 new or modified test files.

_ trunk Compile Tests _

0
mvndep
20
Maven dependency ordering for branch

+1
mvninstall
1131
trunk passed

+1
compile
1062
trunk passed

+1
checkstyle
189
trunk passed

+1
mvnsite
137
trunk passed

+1
shadedclient
1101
branch has no errors when building and testing our client artifacts.

+1
javadoc
95
trunk passed

0
spotbugs
79
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
217
trunk passed

-0
patch
113
Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.

_ Patch Compile Tests _

0
mvndep
22
Maven dependency ordering for patch

+1
mvninstall
91
the patch passed

+1
compile
1100
the patch passed

+1
javac
1100
the patch passed

-0
checkstyle
144
root: The patch generated 61 new + 85 unchanged - 2 fixed = 146 total (was 87)

+1
mvnsite
119
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
xml
2
The patch has no ill-formed XML file.

+1
shadedclient
725
patch has no errors when building and testing our client artifacts.

-1
javadoc
31
hadoop-tools_hadoop-aws generated 3 new + 1 unchanged - 0 fixed = 4 total (was 1)

-1
findbugs
73
hadoop-tools/hadoop-aws generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)

_ Other Tests _

+1
unit
520
hadoop-common in the patch passed.

+1
unit
288
hadoop-aws in the patch passed.

+1
asflicense
46
The patch does not generate ASF License warnings.

9589

Reason
Tests

FindBugs
module:hadoop-tools/hadoop-aws

org.apache.hadoop.fs.s3a.s3guard.PathOrderComparators$TopmostFirst implements Comparator but not Serializable  At PathOrderComparators.java:Serializable  At PathOrderComparators.java:[lines 69-89]

org.apache.hadoop.fs.s3a.s3guard.PathOrderComparators$TopmostLast implements Comparator but not Serializable  At PathOrderComparators.java:Serializable  At PathOrderComparators.java:[lines 98-109]

Subsystem
Report/Notes

Docker
Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-654/27/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/654

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml

uname
Linux 5298d3f9a220 4.4.0-141-generic #167~14.04.1-Ubuntu SMP Mon Dec 10 13:20:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 12c8161

Default Java
1.8.0_212

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/27/artifact/out/diff-checkstyle-root.txt

javadoc
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/27/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-aws.txt

findbugs
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/27/artifact/out/new-findbugs-hadoop-tools_hadoop-aws.html

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/27/testReport/

Max. process+thread count
1501 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/27/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
41
Docker mode activated.

_ Prechecks _

+1
dupname
1
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 21 new or modified test files.

_ trunk Compile Tests _

0
mvndep
39
Maven dependency ordering for branch

+1
mvninstall
1158
trunk passed

+1
compile
1060
trunk passed

+1
checkstyle
147
trunk passed

+1
mvnsite
122
trunk passed

+1
shadedclient
1057
branch has no errors when building and testing our client artifacts.

+1
javadoc
94
trunk passed

0
spotbugs
62
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
182
trunk passed

-0
patch
97
Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.

_ Patch Compile Tests _

0
mvndep
21
Maven dependency ordering for patch

+1
mvninstall
77
the patch passed

+1
compile
972
the patch passed

+1
javac
972
the patch passed

-0
checkstyle
147
root: The patch generated 29 new + 79 unchanged - 2 fixed = 108 total (was 81)

+1
mvnsite
116
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
xml
3
The patch has no ill-formed XML file.

+1
shadedclient
751
patch has no errors when building and testing our client artifacts.

+1
javadoc
93
the patch passed

+1
findbugs
202
the patch passed

_ Other Tests _

+1
unit
518
hadoop-common in the patch passed.

+1
unit
289
hadoop-aws in the patch passed.

+1
asflicense
47
The patch does not generate ASF License warnings.

7134

Subsystem
Report/Notes

Docker
Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-654/28/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/654

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml

uname
Linux f0756a85f48a 4.4.0-141-generic #167~14.04.1-Ubuntu SMP Mon Dec 10 13:20:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 732133c

Default Java
1.8.0_212

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/28/artifact/out/diff-checkstyle-root.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/28/testReport/

Max. process+thread count
1491 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/28/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
48
Docker mode activated.

_ Prechecks _

+1
dupname
1
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 21 new or modified test files.

_ trunk Compile Tests _

0
mvndep
69
Maven dependency ordering for branch

+1
mvninstall
1156
trunk passed

+1
compile
1086
trunk passed

+1
checkstyle
149
trunk passed

+1
mvnsite
121
trunk passed

+1
shadedclient
1058
branch has no errors when building and testing our client artifacts.

+1
javadoc
93
trunk passed

0
spotbugs
63
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
185
trunk passed

-0
patch
98
Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.

_ Patch Compile Tests _

0
mvndep
21
Maven dependency ordering for patch

+1
mvninstall
76
the patch passed

+1
compile
977
the patch passed

+1
javac
977
the patch passed

-0
checkstyle
148
root: The patch generated 12 new + 79 unchanged - 2 fixed = 91 total (was 81)

+1
mvnsite
118
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
xml
2
The patch has no ill-formed XML file.

+1
shadedclient
735
patch has no errors when building and testing our client artifacts.

+1
javadoc
93
the patch passed

+1
findbugs
200
the patch passed

_ Other Tests _

+1
unit
517
hadoop-common in the patch passed.

+1
unit
288
hadoop-aws in the patch passed.

+1
asflicense
46
The patch does not generate ASF License warnings.

7180

Subsystem
Report/Notes

Docker
Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-654/29/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/654

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml

uname
Linux 9d8443beb58b 4.4.0-141-generic #167~14.04.1-Ubuntu SMP Mon Dec 10 13:20:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 729ccb2

Default Java
1.8.0_212

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/29/artifact/out/diff-checkstyle-root.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/29/testReport/

Max. process+thread count
1378 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-654/29/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

Closing this PR and kicking off a new one


[WIP - Do not merge] Integrate nrfx

This pull request integrates the nrfx standalone driver package in mynewt-core.
This allows us to bring in all the drivers from mynewt_nordic such as PWM and ADC and to stop depending on it.
To-Do:

[x] Port and test the pwm_nrf52 driver;
[x] Port and test the adc_nrf52 driver;
[x] Port the adc_nrf51 driver;
[ ] Test the adc_nrf51 driver (I am unable to test it);
[x] Test HAL GPIO driver;
[x] Test HAL SPI driver;
[x] Test HAL I2C;
[x] Cleanup;

Trying to build this on an nrf52 thingy I get a compilation error
Error: repos/apache-mynewt-core/hw/mcu/nordic/nrf52xxx/src/hal_gpio.c: In function 'hal_gpio_read':
repos/apache-mynewt-core/hw/mcu/nordic/nrf52xxx/src/hal_gpio.c:175:42: error: expected expression before ')' token
         (port->OUT >> HAL_GPIO_INDEX(pin)) & 1UL :
                                          ^
repos/apache-mynewt-core/hw/mcu/nordic/nrf52xxx/src/hal_gpio.c:177:1: error: control reaches end of non-void function [-Werror=return-type]
 }

Looks like HAL_GPIO_INDEX is blank
#ifdef NRF52
#define HAL_GPIO_INDEX(pin)
#define HAL_GPIO_PORT(pin)      (NRF_P0)
#define HAL_GPIO_MASK(pin)      (1 << pin)
#define HAL_GPIOTE_PIN_MASK     GPIOTE_CONFIG_PSEL_Msk
#endif

Looks like HAL_GPIO_INDEX is blank
@jacobrosenthal fixed, in this case(nRF52832) the macro should do nothing, my bad.

Are we trying to bring this in at the hal level or the driver level?

Im not sure what you need tested for Test HAL I2C but I ran my drv2605 driver successfully on an nrf52 thingy

@mkiiskila @jacobrosenthal I'm not sure our current HAL uses the nrfx header files or files like that?  I thought nrfx was a replacement for the existing, non-BSD licensed driver pack.  the hal relies on NRF bitfields and a few other definitions, but shouldn't overlap with nrfx?

I thought nrfx was a replacement for the existing, non-BSD licensed driver pack.

@sterlinghughes It is.

the hal relies on NRF bitfields and a few other definitions, but shouldn't overlap with nrfx?

IMHO nrfx also should replace both mynewt_nordic and all the files used under hw/mcu/nordic/src/ext/nRF5_SDK_11.0.0_89a8197 namely files containing the NRF bitfields definitions.

Of note, the old nordic adc and pwm implementations are using the driver(drv) not hal implementations which can get pretty tied into nordics use of timers and other hardware we need to be careful of that and probably document it better.
For instance if Im reading this correctly, simply including nrfx_config.h which both pwm and adc do, will enable everything in that file ( pwm and adc, plus nrfx_systick whatever that is) IE turning on pwm turns on adc. I think well have to replace the hardcoded enables in nrfx_config.h in with checks on MYNEWT_VAL()
Also this PR creates a new sysflag ADC that needs to be enabled in addition, plus ADC_0 for the 0 channel. Its more newty to simply check of the ADC_0 etc, for instance like the STM adc driver currently does
#if MYNEWT_VAL(ADC_1)||MYNEWT_VAL(ADC_2)||MYNEWT_VAL(ADC_3)

I think I have a lot of comments on the way those drivers implemented. Mainly ADC especially is hugely compile time configured for some reason, and I think they need to be more runtime configuration based. But Im not sure that should hold up the SDK refresh.
Perhaps the sdk refresh PR needs to be separated from the driver merge and refactor, or at least it would help to rebase this pr into fewer scoped commits to better understand what were changing from the upstream nrfx package, and

@mlaz Seems like you deleted the adc_test you had in here? What did you run on nrf52 that I can test on nrf51?

Also this PR creates a new sysflag ADC that needs to be enabled in addition, plus ADC_0 for the 0 channel. Its more newty to simply check of the ADC_0 etc, for instance like the STM adc driver currently does

Agreed, I'll make it work like STM  ADC driver. Probably the PWM macros should work this way as well for pwm_nrf52.

@mlaz Seems like you deleted the adc_test you had in here? What did you run on nrf52 that I can test on nrf51?

I did because this app will probably only work with adc_nrf52, since the driver makes you use nordic structures namely nrf_saadc_channel_config_t. These test apps, imho, should be generic, i.e. the code should be able to run with whatever driver according to whatever syscfg macros the user sets. Just like pwm_test works with pwm_nrf52 and soft_pwm.

@mlaz and I worked tonight on the port. Were pulling out the problematic nrfx_config, as its pretty clear the way this was handled in the past was within the bsp pkg.cflags
Those are all out of date now so were fixing them, and also, a bunch of them are enabled even though newt doesnt use any nordic driver code. That must be from a very old version of mynewt where nordic stuff was included in the repo and has been copy and pasted over and over since.
On the flip side, the adc and pwm which there are drivers for we are enabling, but behind the pkg.cflags.ADC_0 conditional which is more recent development I believe
However, I would argue that this shouldnt be done in the bsp but rather in the driver package so that theyre only enabled if you bring the driver in and dont have to be edited out of the bsp if someone wants to write a new newt driver not based on any of the nordic 'driver' code. However when I do that theyre not included in the correct order to keep the nrfx headers from bitching theyre not defined. Not sure if thats an easy fix under the newt hood?

ADC is now in with minimal changes (for better or worse)
Were getting close to squashing this.
But mlaz and I are going back and forth on why you might use ign.files like
https://github.com/runtimeco/mynewt_nordic/blob/master/hw/mcu/nordic_sdk/pkg.yml#L35
There are one or two files in the sdk that dont compile and were not using them so those obviously go on the list, but should literally all the files were not using be on that list, does that speed up newt compiles at all?

Looks pretty good. Here's one target that seems to be failing:
[marko@IsMyLaptop:~/src/incubator-mynewt-blinky]$ newt target show bleprph_nrf52
targets/bleprph_nrf52
app=@apache-mynewt-core/apps/bleprph
bsp=@apache-mynewt-core/hw/bsp/nrf52840pdk
build_profile=debug
syscfg=BLE_EXT_ADV=1:BLE_LL_CFG_FEAT_LE_2M_PHY=1:BLE_LL_CFG_FEAT_LE_CODED_PHY=1
[marko@IsMyLaptop:~/src/incubator-mynewt-blinky]$ newt build -j 1 bleprph_nrf52
Building target targets/bleprph_nrf52
Compiling repos/apache-mynewt-core/hw/drivers/nimble/nrf52/src/ble_phy.c
Error: repos/apache-mynewt-core/hw/drivers/nimble/nrf52/src/ble_phy.c: In function 'ble_phy_get_cur_rx_phy_mode':
repos/apache-mynewt-core/hw/drivers/nimble/nrf52/src/ble_phy.c:796:35: error: subscripted value is neither array nor pointer nor vector
phy = NRF_RADIO->RESERVED7[0] & 0x06 ?
^
exit status 1

LGTM.


LAB_02

English:
Exercise 1: Implement Azure Policy.
Current Translation:
Exercice 1 : Implémentez Azure Policy.
Suggested Translation:
Exercice 1 : Implémenter Azure Policy.
Description:
Exercice title. Please use infinitive form

Hello contributor,
Thank you for your contribution! We are processing your suggestions and we will update you shortly.
Kind regards,
Microsoft Worldwide Learning Team

Run the bugfix tool and waiting for result. ICMS QUERY

Manually checked and modified


(427) Apply navigation fixes and non-statutory sentence type changes

We have previously had a high number of applications where the sentence type was 'Non statutory'.
This should be a very rare occurence and after post-hoc inspection each of these cases was not a valid non-statutory case.
In order to better guide the user to the correct sentence type we add some guidance about what is a valid non-statutory case and if non-statutory is selected we take the user to a page where they're asked to confirmed if the user is managed by MAPPA (as they must be if it is a valid non-stat case).
With the current integration test setup we have this is quite hard to test as it would require adding a fair amount of logic to the 'completeBasicInformation' method in Apply helpers so intead I suggest we test this behavio
As well as this addition I have correct the behaviour of the 'Back' button on a couple of pages of the Apply journey.
Trello card
Screenshots of UI changes
Before

After

LGTM thanks!