HBASE-23347 Allowable custom authentication methods for RPCs

This is a big one. Sorry ;)
Start here with a one-page writeup: https://github.com/joshelser/hbase/blob/23347-pluggable-authentication/dev-support/design-docs/HBASE-23347-pluggable-authentication.md
Next, look at the client side interfaces: https://github.com/joshelser/hbase/tree/23347-pluggable-authentication/hbase-client/src/main/java/org/apache/hadoop/hbase/security/provider
Then, the server side interfaces: https://github.com/joshelser/hbase/tree/23347-pluggable-authentication/hbase-server/src/main/java/org/apache/hadoop/hbase/security/provider
Finally, an end-to-end example of how you can use this: https://github.com/joshelser/hbase/blob/23347-pluggable-authentication/hbase-server/src/test/java/org/apache/hadoop/hbase/security/provider/TestCustomSaslAuthenticationProvider.java
This is very-much a first draft. This is "new art" for HBase and I expect some discussion on how we want to safely do this. I'm fully expecting lots of revisions.
Relevant tests should be passing, but this is also partially for me to let QA-bot run. Everything I did was in attempts to retain backwards compatibility with older clients, as well as RPC-impl compatibility. This should all be implementation-details inside of HBase.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
1m 57s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  1s
No case conflicting files found.

+0 :ok:
markdownlint
0m  1s
markdownlint was not available.

+1 :green_heart:
hbaseanti
0m  0s
Patch does not have any anti-patterns.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s
The patch appears to include 6 new or modified test files.

_ master Compile Tests _

+0 :ok:
mvndep
0m 35s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
5m 16s
master passed

+1 :green_heart:
compile
3m  1s
master passed

+1 :green_heart:
checkstyle
2m 32s
master passed

+1 :green_heart:
shadedjars
4m 36s
branch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
4m  4s
master passed

+0 :ok:
spotbugs
3m 45s
Used deprecated FindBugs config; considering switching to SpotBugs.

+1 :green_heart:
findbugs
19m 27s
master passed

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 17s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
5m 46s
the patch passed

+1 :green_heart:
compile
3m 24s
the patch passed

+1 :green_heart:
javac
3m 24s
the patch passed

-1 :x:
checkstyle
2m 33s
root: The patch generated 71 new + 49 unchanged - 12 fixed = 120 total (was 61)

-1 :x:
whitespace
0m  0s
The patch has 9 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply

-1 :x:
shadedjars
0m 13s
patch has 7 errors when building our shaded downstream artifacts.

+1 :green_heart:
hadoopcheck
15m 55s
Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.

-1 :x:
javadoc
0m 24s
hbase-client generated 4 new + 2 unchanged - 0 fixed = 6 total (was 2)

-1 :x:
javadoc
2m 49s
root generated 4 new + 3 unchanged - 0 fixed = 7 total (was 3)

-1 :x:
findbugs
4m 20s
hbase-server generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)

-1 :x:
findbugs
13m 32s
root generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)

_ Other Tests _

-1 :x:
unit
10m 11s
root in the patch failed.

-1 :x:
asflicense
0m 58s
The patch generated 4 ASF License warnings.

112m 20s

Reason
Tests

FindBugs
module:hbase-server

Should org.apache.hadoop.hbase.security.provider.DigestSaslServerAuthenticationProvider$SaslDigestCallbackHandler be a static inner class?  At DigestSaslServerAuthenticationProvider.java:inner class?  At DigestSaslServerAuthenticationProvider.java:[lines 66-119]

FindBugs
module:root

Should org.apache.hadoop.hbase.security.provider.DigestSaslServerAuthenticationProvider$SaslDigestCallbackHandler be a static inner class?  At DigestSaslServerAuthenticationProvider.java:inner class?  At DigestSaslServerAuthenticationProvider.java:[lines 66-119]

Failed junit tests
hadoop.hbase.security.provider.TestSaslClientAuthenticationProviders

Subsystem
Report/Notes

Docker
Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/884

Optional Tests
dupname asflicense markdownlint javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile

uname
Linux a4005708683f 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 GNU/Linux

Build tool
maven

Personality
/home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-884/out/precommit/personality/provided.sh

git revision
master / 636fa2c6b0

Default Java
1.8.0_181

checkstyle
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/1/artifact/out/diff-checkstyle-root.txt

whitespace
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/1/artifact/out/whitespace-eol.txt

shadedjars
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/1/artifact/out/patch-shadedjars.txt

javadoc
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/1/artifact/out/diff-javadoc-javadoc-hbase-client.txt

javadoc
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/1/artifact/out/diff-javadoc-javadoc-root.txt

findbugs
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/1/artifact/out/new-findbugs-hbase-server.html

findbugs
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/1/artifact/out/new-findbugs-root.html

unit
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/1/artifact/out/patch-unit-root.txt

Test Results
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/1/testReport/

asflicense
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/1/artifact/out/patch-asflicense-problems.txt

Max. process+thread count
298 (vs. ulimit of 10000)

modules
C: hbase-common hbase-client hbase-server . U: .

Console output
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/1/console

versions
git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 36s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  1s
No case conflicting files found.

+0 :ok:
markdownlint
0m  1s
markdownlint was not available.

+1 :green_heart:
hbaseanti
0m  0s
Patch does not have any anti-patterns.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s
The patch appears to include 6 new or modified test files.

_ master Compile Tests _

+0 :ok:
mvndep
0m 35s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
5m 30s
master passed

+1 :green_heart:
compile
3m  4s
master passed

+1 :green_heart:
checkstyle
2m 38s
master passed

+1 :green_heart:
shadedjars
4m 47s
branch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
4m 21s
master passed

+0 :ok:
spotbugs
3m 52s
Used deprecated FindBugs config; considering switching to SpotBugs.

+1 :green_heart:
findbugs
20m 56s
master passed

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 15s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
4m 58s
the patch passed

+1 :green_heart:
compile
3m  3s
the patch passed

+1 :green_heart:
javac
3m  3s
the patch passed

-1 :x:
checkstyle
2m 32s
root: The patch generated 71 new + 49 unchanged - 12 fixed = 120 total (was 61)

-1 :x:
whitespace
0m  0s
The patch has 9 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply

-1 :x:
shadedjars
0m 12s
patch has 7 errors when building our shaded downstream artifacts.

+1 :green_heart:
hadoopcheck
15m 40s
Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.

-1 :x:
javadoc
0m 24s
hbase-client generated 4 new + 2 unchanged - 0 fixed = 6 total (was 2)

-1 :x:
javadoc
2m 48s
root generated 4 new + 3 unchanged - 0 fixed = 7 total (was 3)

-1 :x:
findbugs
4m 16s
hbase-server generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)

-1 :x:
findbugs
13m 39s
root generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)

_ Other Tests _

-1 :x:
unit
10m 23s
root in the patch failed.

-1 :x:
asflicense
0m 58s
The patch generated 4 ASF License warnings.

112m  4s

Reason
Tests

FindBugs
module:hbase-server

Should org.apache.hadoop.hbase.security.provider.DigestSaslServerAuthenticationProvider$SaslDigestCallbackHandler be a static inner class?  At DigestSaslServerAuthenticationProvider.java:inner class?  At DigestSaslServerAuthenticationProvider.java:[lines 66-119]

FindBugs
module:root

Should org.apache.hadoop.hbase.security.provider.DigestSaslServerAuthenticationProvider$SaslDigestCallbackHandler be a static inner class?  At DigestSaslServerAuthenticationProvider.java:inner class?  At DigestSaslServerAuthenticationProvider.java:[lines 66-119]

Failed junit tests
hadoop.hbase.security.provider.TestSaslClientAuthenticationProviders

Subsystem
Report/Notes

Docker
Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/2/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/884

Optional Tests
dupname asflicense markdownlint javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile

uname
Linux 25a259dae9fa 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 GNU/Linux

Build tool
maven

Personality
/home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-884/out/precommit/personality/provided.sh

git revision
master / 0f166edc3d

Default Java
1.8.0_181

checkstyle
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/2/artifact/out/diff-checkstyle-root.txt

whitespace
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/2/artifact/out/whitespace-eol.txt

shadedjars
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/2/artifact/out/patch-shadedjars.txt

javadoc
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/2/artifact/out/diff-javadoc-javadoc-hbase-client.txt

javadoc
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/2/artifact/out/diff-javadoc-javadoc-root.txt

findbugs
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/2/artifact/out/new-findbugs-hbase-server.html

findbugs
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/2/artifact/out/new-findbugs-root.html

unit
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/2/artifact/out/patch-unit-root.txt

Test Results
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/2/testReport/

asflicense
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/2/artifact/out/patch-asflicense-problems.txt

Max. process+thread count
313 (vs. ulimit of 10000)

modules
C: hbase-common hbase-client hbase-server . U: .

Console output
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/2/console

versions
git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

Slowly working through review comments. Keep 'em coming.

Client fallback
It looks like we had copied an idea from Hadoop where the client which was configured to auth'n over Kerberos, could fall back to trying to authenticate via SIMPLE. This seems silly to me and is off by default.  I see one instance of someone poking at this https://issues.apache.org/jira/browse/HBASE-20993 but I'm missing the "why". I can't think of a reason that we would want to encourage this (not using the correct configuration to talk to HBase).
Am I missing a use-case?
How will compat be managed?
Great question. I specifically avoided being prescriptive as to how compatibility should be maintained within some AuthenticationProvider. We have a couple ways that compatibility rears its head.
RPC client/server <-> AuthenticationProvider interface
I think this shouldn't be too bad to maintain since the interfaces are presently not tied to any RPC-related classes. Meaning, the RPC impl can do whatever processing it needs to do and just needs to extract the info to make the calls into the AuthProvider methods.
ClientAuthnProvider v1 to ServerAuthnProvider v2 (etc)
Clearly, each client/server AuthenticationProvider impl needs to have compatibility across versions to ensure that clients can continue to authenticate properly. I specifically avoided being prescriptive as to how implementations should account for this. The current interfaces don't leave much to "extrapolate" in the current interface -- I don't see much being able to change on the surface.
Presently, I've only done stuff with Auth'nProviders which still rely on Hadoop Tokens. In theory, we could set up TLS in some other way, but that's a can of worms I didn't want to open. The other related piece here is that the LimitedPrivate annotation I marked this code with comes with a "you bet your butt it's gonna evolve" message. I think we have a good start, but expect that there's a world of other kinds of auth'n where these interfaces fall short (but I just don't know enough to plan for them all).
You have a use-case?
Wellington and I have been playing around with an implementation where we hook up HBase to Linux PAM. Client creates a new Token (ala TokenUtil) in which they put in their username/password, this is sent across the wire to the Master/RegionServer who asks PAM if the credentials are valid, allowing/disallowing the RPC per PAM's reply. This has been pretty slick in practice -- suddenly, HBase has username/password auth'n and we aren't in the business of storing/managing those passwords.
The problem with this approach is that we have to handle a plaintext password going over HBase RPC. Best as I know, the RPC encryption we get from SASL only comes with the GSSAPI mechanism, not as a function of any SASL mechanism. PAM is (usually for real installations) going against some corporate auth'n database (e.g. active directory), and not just the local password db on each host. When we scale up&out, there's also going to be a worry about how well the backend auth'n database works (and is that going to kill HBase perf).
So, it's a win to me, but it does bring some baggage with it. Hope that helps.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 37s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s
No case conflicting files found.

+0 :ok:
markdownlint
0m  0s
markdownlint was not available.

+1 :green_heart:
hbaseanti
0m  0s
Patch does not have any anti-patterns.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s
The patch appears to include 6 new or modified test files.

_ master Compile Tests _

+0 :ok:
mvndep
0m 35s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
5m 19s
master passed

+1 :green_heart:
compile
1m 55s
master passed

+1 :green_heart:
checkstyle
2m 13s
master passed

+1 :green_heart:
shadedjars
4m 42s
branch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
1m 23s
master passed

+0 :ok:
spotbugs
4m 29s
Used deprecated FindBugs config; considering switching to SpotBugs.

+1 :green_heart:
findbugs
6m 25s
master passed

-0 :warning:
patch
4m 51s
Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 15s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
5m  2s
the patch passed

+1 :green_heart:
compile
1m 48s
the patch passed

+1 :green_heart:
javac
1m 48s
the patch passed

-1 :x:
checkstyle
0m 32s
hbase-client: The patch generated 4 new + 22 unchanged - 10 fixed = 26 total (was 32)

-1 :x:
checkstyle
1m 18s
hbase-server: The patch generated 1 new + 26 unchanged - 3 fixed = 27 total (was 29)

+1 :green_heart:
whitespace
0m  0s
The patch has no whitespace issues.

-1 :x:
shadedjars
0m 13s
patch has 7 errors when building our shaded downstream artifacts.

+1 :green_heart:
hadoopcheck
15m 52s
Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.

+1 :green_heart:
javadoc
1m 27s
the patch passed

+1 :green_heart:
findbugs
6m 31s
the patch passed

_ Other Tests _

+1 :green_heart:
unit
3m 11s
hbase-common in the patch passed.

+1 :green_heart:
unit
1m 56s
hbase-client in the patch passed.

-1 :x:
unit
49m  3s
hbase-server in the patch failed.

+1 :green_heart:
asflicense
0m 52s
The patch does not generate ASF License warnings.

119m  3s

Reason
Tests

Failed junit tests
hadoop.hbase.security.access.TestUnloadAccessController

hadoop.hbase.client.TestAsyncAccessControlAdminApi

hadoop.hbase.quotas.TestSpaceQuotaOnNonExistingTables

hadoop.hbase.ipc.TestNettyIPC

hadoop.hbase.ipc.TestRpcHandlerException

hadoop.hbase.client.TestMultiActionMetricsFromClient

hadoop.hbase.TestClientClusterMetrics

hadoop.hbase.ipc.TestBlockingIPC

Subsystem
Report/Notes

Docker
Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/3/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/884

Optional Tests
dupname asflicense markdownlint javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile

uname
Linux 3b9b4c9b87d4 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 GNU/Linux

Build tool
maven

Personality
/home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-884/out/precommit/personality/provided.sh

git revision
master / 5e34a59d90

Default Java
1.8.0_181

checkstyle
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/3/artifact/out/diff-checkstyle-hbase-client.txt

checkstyle
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/3/artifact/out/diff-checkstyle-hbase-server.txt

shadedjars
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/3/artifact/out/patch-shadedjars.txt

unit
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/3/artifact/out/patch-unit-hbase-server.txt

Test Results
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/3/testReport/

Max. process+thread count
716 (vs. ulimit of 10000)

modules
C: hbase-common hbase-client hbase-server U: .

Console output
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/3/console

versions
git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 34s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  1s
No case conflicting files found.

+0 :ok:
markdownlint
0m  1s
markdownlint was not available.

+1 :green_heart:
hbaseanti
0m  0s
Patch does not have any anti-patterns.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s
The patch appears to include 6 new or modified test files.

_ master Compile Tests _

+0 :ok:
mvndep
0m 37s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
5m 21s
master passed

+1 :green_heart:
compile
1m 47s
master passed

+1 :green_heart:
checkstyle
2m 20s
master passed

+1 :green_heart:
shadedjars
4m 40s
branch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
1m 23s
master passed

+0 :ok:
spotbugs
4m 10s
Used deprecated FindBugs config; considering switching to SpotBugs.

+1 :green_heart:
findbugs
6m  0s
master passed

-0 :warning:
patch
4m 34s
Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 15s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
4m 58s
the patch passed

+1 :green_heart:
compile
1m 46s
the patch passed

+1 :green_heart:
javac
1m 46s
the patch passed

+1 :green_heart:
checkstyle
0m 26s
The patch passed checkstyle in hbase-common

+1 :green_heart:
checkstyle
0m 32s
hbase-client: The patch generated 0 new + 22 unchanged - 10 fixed = 22 total (was 32)

+1 :green_heart:
checkstyle
1m 18s
hbase-server: The patch generated 0 new + 26 unchanged - 3 fixed = 26 total (was 29)

+1 :green_heart:
whitespace
0m  1s
The patch has no whitespace issues.

+1 :green_heart:
shadedjars
4m 34s
patch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
hadoopcheck
15m 39s
Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.

+1 :green_heart:
javadoc
1m 21s
the patch passed

+1 :green_heart:
findbugs
6m 28s
the patch passed

_ Other Tests _

+1 :green_heart:
unit
3m  9s
hbase-common in the patch passed.

+1 :green_heart:
unit
1m 54s
hbase-client in the patch passed.

-1 :x:
unit
49m  6s
hbase-server in the patch failed.

+1 :green_heart:
asflicense
0m 51s
The patch does not generate ASF License warnings.

121m 59s

Reason
Tests

Failed junit tests
hadoop.hbase.security.access.TestUnloadAccessController

hadoop.hbase.quotas.TestSpaceQuotaOnNonExistingTables

hadoop.hbase.ipc.TestNettyIPC

hadoop.hbase.client.TestMultiActionMetricsFromClient

hadoop.hbase.ipc.TestRpcHandlerException

hadoop.hbase.TestClientClusterMetrics

hadoop.hbase.client.TestAsyncAccessControlAdminApi

hadoop.hbase.ipc.TestBlockingIPC

Subsystem
Report/Notes

Docker
Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/4/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/884

Optional Tests
dupname asflicense markdownlint javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile

uname
Linux cd7051bad42a 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 GNU/Linux

Build tool
maven

Personality
/home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-884/out/precommit/personality/provided.sh

git revision
master / 5e34a59d90

Default Java
1.8.0_181

unit
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/4/artifact/out/patch-unit-hbase-server.txt

Test Results
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/4/testReport/

Max. process+thread count
746 (vs. ulimit of 10000)

modules
C: hbase-common hbase-client hbase-server U: .

Console output
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/4/console

versions
git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 35s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s
No case conflicting files found.

+0 :ok:
markdownlint
0m  0s
markdownlint was not available.

+1 :green_heart:
hbaseanti
0m  0s
Patch does not have any anti-patterns.

+1 :green_heart:
@author
0m  1s
The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s
The patch appears to include 6 new or modified test files.

_ master Compile Tests _

+0 :ok:
mvndep
0m 36s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
5m 19s
master passed

+1 :green_heart:
compile
1m 49s
master passed

+1 :green_heart:
checkstyle
2m 19s
master passed

+1 :green_heart:
shadedjars
4m 35s
branch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
1m 23s
master passed

+0 :ok:
spotbugs
4m 13s
Used deprecated FindBugs config; considering switching to SpotBugs.

+1 :green_heart:
findbugs
6m  2s
master passed

-0 :warning:
patch
4m 36s
Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 15s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
4m 54s
the patch passed

+1 :green_heart:
compile
1m 48s
the patch passed

+1 :green_heart:
javac
1m 48s
the patch passed

+1 :green_heart:
checkstyle
0m 27s
The patch passed checkstyle in hbase-common

-1 :x:
checkstyle
0m 32s
hbase-client: The patch generated 1 new + 22 unchanged - 10 fixed = 23 total (was 32)

+1 :green_heart:
checkstyle
1m 19s
hbase-server: The patch generated 0 new + 26 unchanged - 3 fixed = 26 total (was 29)

+1 :green_heart:
whitespace
0m  0s
The patch has no whitespace issues.

+1 :green_heart:
xml
0m  2s
The patch has no ill-formed XML file.

+1 :green_heart:
shadedjars
4m 49s
patch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
hadoopcheck
16m 33s
Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.

-1 :x:
javadoc
0m 22s
hbase-client generated 2 new + 2 unchanged - 0 fixed = 4 total (was 2)

+1 :green_heart:
findbugs
6m 47s
the patch passed

_ Other Tests _

+1 :green_heart:
unit
3m  9s
hbase-common in the patch passed.

+1 :green_heart:
unit
1m 56s
hbase-client in the patch passed.

+1 :green_heart:
unit
166m 35s
hbase-server in the patch passed.

+1 :green_heart:
asflicense
1m 26s
The patch does not generate ASF License warnings.

241m 39s

Subsystem
Report/Notes

Docker
Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/5/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/884

Optional Tests
dupname asflicense markdownlint javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile xml

uname
Linux b382b4d045fa 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 GNU/Linux

Build tool
maven

Personality
/home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-884/out/precommit/personality/provided.sh

git revision
master / 6d251efa04

Default Java
1.8.0_181

checkstyle
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/5/artifact/out/diff-checkstyle-hbase-client.txt

javadoc
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/5/artifact/out/diff-javadoc-javadoc-hbase-client.txt

Test Results
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/5/testReport/

Max. process+thread count
4833 (vs. ulimit of 10000)

modules
C: hbase-common hbase-client hbase-server U: .

Console output
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/5/console

versions
git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 32s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s
No case conflicting files found.

+0 :ok:
markdownlint
0m  0s
markdownlint was not available.

+1 :green_heart:
hbaseanti
0m  0s
Patch does not have any anti-patterns.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s
The patch appears to include 6 new or modified test files.

_ master Compile Tests _

+0 :ok:
mvndep
0m 37s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
5m 43s
master passed

+1 :green_heart:
compile
1m 44s
master passed

+1 :green_heart:
checkstyle
2m 28s
master passed

+1 :green_heart:
shadedjars
4m 58s
branch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
1m 17s
master passed

+0 :ok:
spotbugs
4m 27s
Used deprecated FindBugs config; considering switching to SpotBugs.

+1 :green_heart:
findbugs
6m 18s
master passed

-0 :warning:
patch
4m 47s
Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 14s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
5m 26s
the patch passed

+1 :green_heart:
compile
1m 46s
the patch passed

+1 :green_heart:
javac
1m 46s
the patch passed

+1 :green_heart:
checkstyle
0m 25s
The patch passed checkstyle in hbase-common

-1 :x:
checkstyle
0m 32s
hbase-client: The patch generated 1 new + 22 unchanged - 10 fixed = 23 total (was 32)

+1 :green_heart:
checkstyle
1m 28s
hbase-server: The patch generated 0 new + 26 unchanged - 3 fixed = 26 total (was 29)

+1 :green_heart:
whitespace
0m  0s
The patch has no whitespace issues.

+1 :green_heart:
xml
0m  1s
The patch has no ill-formed XML file.

+1 :green_heart:
shadedjars
4m 59s
patch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
hadoopcheck
17m  6s
Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.

-1 :x:
javadoc
0m 21s
hbase-client generated 2 new + 2 unchanged - 0 fixed = 4 total (was 2)

+1 :green_heart:
findbugs
6m 56s
the patch passed

_ Other Tests _

+1 :green_heart:
unit
2m 57s
hbase-common in the patch passed.

+1 :green_heart:
unit
1m 50s
hbase-client in the patch passed.

-1 :x:
unit
156m 30s
hbase-server in the patch failed.

+1 :green_heart:
asflicense
1m 13s
The patch does not generate ASF License warnings.

233m 37s

Reason
Tests

Failed junit tests
hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures

Subsystem
Report/Notes

Docker
Client=19.03.4 Server=19.03.4 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/6/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/884

Optional Tests
dupname asflicense markdownlint javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile xml

uname
Linux 5c95dd1d06db 4.15.0-70-generic #79-Ubuntu SMP Tue Nov 12 10:36:11 UTC 2019 x86_64 GNU/Linux

Build tool
maven

Personality
/home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-884/out/precommit/personality/provided.sh

git revision
master / 6d251efa04

Default Java
1.8.0_181

checkstyle
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/6/artifact/out/diff-checkstyle-hbase-client.txt

javadoc
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/6/artifact/out/diff-javadoc-javadoc-hbase-client.txt

unit
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/6/artifact/out/patch-unit-hbase-server.txt

Test Results
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/6/testReport/

Max. process+thread count
4427 (vs. ulimit of 10000)

modules
C: hbase-common hbase-client hbase-server U: .

Console output
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/6/console

versions
git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
1m 56s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s
No case conflicting files found.

+0 :ok:
markdownlint
0m  0s
markdownlint was not available.

+1 :green_heart:
hbaseanti
0m  0s
Patch does not have any anti-patterns.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s
The patch appears to include 6 new or modified test files.

_ master Compile Tests _

+0 :ok:
mvndep
0m 35s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
5m 37s
master passed

+1 :green_heart:
compile
1m 50s
master passed

+1 :green_heart:
checkstyle
2m 16s
master passed

+1 :green_heart:
shadedjars
4m 36s
branch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
1m 21s
master passed

+0 :ok:
spotbugs
4m  9s
Used deprecated FindBugs config; considering switching to SpotBugs.

+1 :green_heart:
findbugs
5m 58s
master passed

-0 :warning:
patch
4m 32s
Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 16s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
5m  1s
the patch passed

+1 :green_heart:
compile
1m 59s
the patch passed

+1 :green_heart:
javac
1m 59s
the patch passed

+1 :green_heart:
checkstyle
0m 29s
The patch passed checkstyle in hbase-common

-1 :x:
checkstyle
0m 34s
hbase-client: The patch generated 1 new + 22 unchanged - 10 fixed = 23 total (was 32)

+1 :green_heart:
checkstyle
1m 21s
hbase-server: The patch generated 0 new + 26 unchanged - 3 fixed = 26 total (was 29)

-1 :x:
whitespace
0m  0s
The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply

+1 :green_heart:
xml
0m  2s
The patch has no ill-formed XML file.

+1 :green_heart:
shadedjars
4m 41s
patch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
hadoopcheck
16m  8s
Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.

+1 :green_heart:
javadoc
1m 22s
the patch passed

+1 :green_heart:
findbugs
6m 34s
the patch passed

_ Other Tests _

+1 :green_heart:
unit
3m 10s
hbase-common in the patch passed.

+1 :green_heart:
unit
1m 57s
hbase-client in the patch passed.

+1 :green_heart:
unit
147m 31s
hbase-server in the patch passed.

+1 :green_heart:
asflicense
1m 39s
The patch does not generate ASF License warnings.

224m 13s

Subsystem
Report/Notes

Docker
Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/7/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/884

Optional Tests
dupname asflicense markdownlint javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile xml

uname
Linux 08ada1d62fcd 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 GNU/Linux

Build tool
maven

Personality
/home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-884/out/precommit/personality/provided.sh

git revision
master / 6d251efa04

Default Java
1.8.0_181

checkstyle
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/7/artifact/out/diff-checkstyle-hbase-client.txt

whitespace
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/7/artifact/out/whitespace-eol.txt

Test Results
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/7/testReport/

Max. process+thread count
4434 (vs. ulimit of 10000)

modules
C: hbase-common hbase-client hbase-server U: .

Console output
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/7/console

versions
git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

270f5ba makes the parsing on DefaultProviderSelector a bit more stringent
c07ae33 breaks apart the SaslAuthenticationProvider interface(s)

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
2m 31s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  1s
No case conflicting files found.

+0 :ok:
markdownlint
0m  1s
markdownlint was not available.

+1 :green_heart:
hbaseanti
0m  0s
Patch does not have any anti-patterns.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s
The patch appears to include 7 new or modified test files.

_ master Compile Tests _

+0 :ok:
mvndep
0m 41s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
7m  8s
master passed

+1 :green_heart:
compile
2m  1s
master passed

+1 :green_heart:
checkstyle
2m 38s
master passed

+1 :green_heart:
shadedjars
5m 38s
branch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
1m 32s
master passed

+0 :ok:
spotbugs
4m 51s
Used deprecated FindBugs config; considering switching to SpotBugs.

+1 :green_heart:
findbugs
6m 58s
master passed

-0 :warning:
patch
5m 13s
Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 15s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
6m  8s
the patch passed

+1 :green_heart:
compile
2m  1s
the patch passed

+1 :green_heart:
javac
2m  1s
the patch passed

+1 :green_heart:
checkstyle
0m 29s
The patch passed checkstyle in hbase-common

-1 :x:
checkstyle
0m 36s
hbase-client: The patch generated 1 new + 22 unchanged - 10 fixed = 23 total (was 32)

+1 :green_heart:
checkstyle
1m 34s
hbase-server: The patch generated 0 new + 26 unchanged - 3 fixed = 26 total (was 29)

-1 :x:
whitespace
0m  0s
The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply

+1 :green_heart:
xml
0m  1s
The patch has no ill-formed XML file.

-1 :x:
shadedjars
2m 57s
patch has 10 errors when building our shaded downstream artifacts.

+1 :green_heart:
hadoopcheck
19m 37s
Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.

+1 :green_heart:
javadoc
1m 34s
the patch passed

-1 :x:
findbugs
1m 31s
hbase-client generated 4 new + 0 unchanged - 0 fixed = 4 total (was 0)

_ Other Tests _

+1 :green_heart:
unit
3m 24s
hbase-common in the patch passed.

+1 :green_heart:
unit
2m  6s
hbase-client in the patch passed.

-1 :x:
unit
260m 38s
hbase-server in the patch failed.

-1 :x:
asflicense
1m 40s
The patch generated 1 ASF License warnings.

348m 13s

Reason
Tests

FindBugs
module:hbase-client

Inconsistent synchronization of org.apache.hadoop.hbase.security.provider.DefaultProviderSelector.conf; locked 66% of time  Unsynchronized access at DefaultProviderSelector.java:66% of time  Unsynchronized access at DefaultProviderSelector.java:[line 86]

Inconsistent synchronization of org.apache.hadoop.hbase.security.provider.DefaultProviderSelector.digestAuth; locked 60% of time  Unsynchronized access at DefaultProviderSelector.java:60% of time  Unsynchronized access at DefaultProviderSelector.java:[line 96]

Inconsistent synchronization of org.apache.hadoop.hbase.security.provider.DefaultProviderSelector.krbAuth; locked 75% of time  Unsynchronized access at DefaultProviderSelector.java:75% of time  Unsynchronized access at DefaultProviderSelector.java:[line 102]

Inconsistent synchronization of org.apache.hadoop.hbase.security.provider.DefaultProviderSelector.simpleAuth; locked 75% of time  Unsynchronized access at DefaultProviderSelector.java:75% of time  Unsynchronized access at DefaultProviderSelector.java:[line 87]

Subsystem
Report/Notes

Docker
Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/8/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/884

Optional Tests
dupname asflicense markdownlint javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile xml

uname
Linux bc5673d3157d 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 GNU/Linux

Build tool
maven

Personality
/home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-884/out/precommit/personality/provided.sh

git revision
master / 9c82a65b29

Default Java
1.8.0_181

checkstyle
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/8/artifact/out/diff-checkstyle-hbase-client.txt

whitespace
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/8/artifact/out/whitespace-eol.txt

shadedjars
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/8/artifact/out/patch-shadedjars.txt

findbugs
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/8/artifact/out/new-findbugs-hbase-client.html

unit
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/8/artifact/out/patch-unit-hbase-server.txt

Test Results
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/8/testReport/

asflicense
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/8/artifact/out/patch-asflicense-problems.txt

Max. process+thread count
4676 (vs. ulimit of 10000)

modules
C: hbase-common hbase-client hbase-server U: .

Console output
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/8/console

versions
git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 34s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  1s
No case conflicting files found.

+0 :ok:
markdownlint
0m  1s
markdownlint was not available.

+1 :green_heart:
hbaseanti
0m  0s
Patch does not have any anti-patterns.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s
The patch appears to include 7 new or modified test files.

_ master Compile Tests _

+0 :ok:
mvndep
0m 37s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
5m 14s
master passed

+1 :green_heart:
compile
1m 49s
master passed

+1 :green_heart:
checkstyle
2m 14s
master passed

+1 :green_heart:
shadedjars
4m 40s
branch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
1m 23s
master passed

+0 :ok:
spotbugs
4m 29s
Used deprecated FindBugs config; considering switching to SpotBugs.

+1 :green_heart:
findbugs
6m 24s
master passed

-0 :warning:
patch
4m 52s
Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 15s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
5m  2s
the patch passed

+1 :green_heart:
compile
1m 46s
the patch passed

+1 :green_heart:
javac
1m 46s
the patch passed

+1 :green_heart:
checkstyle
0m 26s
The patch passed checkstyle in hbase-common

+1 :green_heart:
checkstyle
0m 31s
hbase-client: The patch generated 0 new + 22 unchanged - 10 fixed = 22 total (was 32)

+1 :green_heart:
checkstyle
1m 18s
hbase-server: The patch generated 0 new + 26 unchanged - 3 fixed = 26 total (was 29)

-1 :x:
whitespace
0m  0s
The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply

+1 :green_heart:
xml
0m  1s
The patch has no ill-formed XML file.

-1 :x:
shadedjars
2m 30s
patch has 10 errors when building our shaded downstream artifacts.

+1 :green_heart:
hadoopcheck
15m 56s
Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.

+1 :green_heart:
javadoc
1m 23s
the patch passed

-1 :x:
findbugs
1m 17s
hbase-client generated 4 new + 0 unchanged - 0 fixed = 4 total (was 0)

_ Other Tests _

+1 :green_heart:
unit
3m  7s
hbase-common in the patch passed.

+1 :green_heart:
unit
1m 55s
hbase-client in the patch passed.

+1 :green_heart:
unit
152m 31s
hbase-server in the patch passed.

-1 :x:
asflicense
1m 26s
The patch generated 1 ASF License warnings.

225m  2s

Reason
Tests

FindBugs
module:hbase-client

Inconsistent synchronization of org.apache.hadoop.hbase.security.provider.DefaultProviderSelector.conf; locked 66% of time  Unsynchronized access at DefaultProviderSelector.java:66% of time  Unsynchronized access at DefaultProviderSelector.java:[line 86]

Inconsistent synchronization of org.apache.hadoop.hbase.security.provider.DefaultProviderSelector.digestAuth; locked 60% of time  Unsynchronized access at DefaultProviderSelector.java:60% of time  Unsynchronized access at DefaultProviderSelector.java:[line 96]

Inconsistent synchronization of org.apache.hadoop.hbase.security.provider.DefaultProviderSelector.krbAuth; locked 75% of time  Unsynchronized access at DefaultProviderSelector.java:75% of time  Unsynchronized access at DefaultProviderSelector.java:[line 102]

Inconsistent synchronization of org.apache.hadoop.hbase.security.provider.DefaultProviderSelector.simpleAuth; locked 75% of time  Unsynchronized access at DefaultProviderSelector.java:75% of time  Unsynchronized access at DefaultProviderSelector.java:[line 87]

Subsystem
Report/Notes

Docker
Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/9/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/884

Optional Tests
dupname asflicense markdownlint javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile xml

uname
Linux fa75fc3bc3af 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 GNU/Linux

Build tool
maven

Personality
/home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-884/out/precommit/personality/provided.sh

git revision
master / 60d9430195

Default Java
1.8.0_181

whitespace
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/9/artifact/out/whitespace-eol.txt

shadedjars
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/9/artifact/out/patch-shadedjars.txt

findbugs
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/9/artifact/out/new-findbugs-hbase-client.html

Test Results
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/9/testReport/

asflicense
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/9/artifact/out/patch-asflicense-problems.txt

Max. process+thread count
4500 (vs. ulimit of 10000)

modules
C: hbase-common hbase-client hbase-server U: .

Console output
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/9/console

versions
git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 33s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  1s
No case conflicting files found.

+0 :ok:
markdownlint
0m  1s
markdownlint was not available.

+1 :green_heart:
hbaseanti
0m  0s
Patch does not have any anti-patterns.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s
The patch appears to include 7 new or modified test files.

_ master Compile Tests _

+0 :ok:
mvndep
0m 35s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
5m  8s
master passed

+1 :green_heart:
compile
1m 45s
master passed

+1 :green_heart:
checkstyle
2m 18s
master passed

+1 :green_heart:
shadedjars
4m 35s
branch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
1m 22s
master passed

+0 :ok:
spotbugs
4m  5s
Used deprecated FindBugs config; considering switching to SpotBugs.

+1 :green_heart:
findbugs
5m 56s
master passed

-0 :warning:
patch
4m 28s
Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 15s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
4m 58s
the patch passed

+1 :green_heart:
compile
1m 44s
the patch passed

+1 :green_heart:
javac
1m 44s
the patch passed

+1 :green_heart:
checkstyle
0m 26s
The patch passed checkstyle in hbase-common

+1 :green_heart:
checkstyle
0m 34s
hbase-client: The patch generated 0 new + 22 unchanged - 10 fixed = 22 total (was 32)

+1 :green_heart:
checkstyle
1m 19s
hbase-server: The patch generated 0 new + 26 unchanged - 3 fixed = 26 total (was 29)

-1 :x:
whitespace
0m  0s
The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply

+1 :green_heart:
xml
0m  1s
The patch has no ill-formed XML file.

+1 :green_heart:
shadedjars
4m 33s
patch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
hadoopcheck
15m 41s
Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.

+1 :green_heart:
javadoc
1m 19s
the patch passed

+1 :green_heart:
findbugs
6m 19s
the patch passed

_ Other Tests _

+1 :green_heart:
unit
3m  8s
hbase-common in the patch passed.

+1 :green_heart:
unit
1m 55s
hbase-client in the patch passed.

-1 :x:
unit
154m 32s
hbase-server in the patch failed.

+1 :green_heart:
asflicense
1m 39s
The patch does not generate ASF License warnings.

227m 47s

Subsystem
Report/Notes

Docker
Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/10/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/884

Optional Tests
dupname asflicense markdownlint javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile xml

uname
Linux 46e92cfb39f0 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 GNU/Linux

Build tool
maven

Personality
/home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-884/out/precommit/personality/provided.sh

git revision
master / 82e155eb26

Default Java
1.8.0_181

whitespace
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/10/artifact/out/whitespace-eol.txt

unit
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/10/artifact/out/patch-unit-hbase-server.txt

Test Results
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/10/testReport/

Max. process+thread count
4830 (vs. ulimit of 10000)

modules
C: hbase-common hbase-client hbase-server U: .

Console output
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/10/console

versions
git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 34s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  1s
No case conflicting files found.

+0 :ok:
markdownlint
0m  1s
markdownlint was not available.

+1 :green_heart:
hbaseanti
0m  0s
Patch does not have any anti-patterns.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s
The patch appears to include 7 new or modified test files.

_ master Compile Tests _

+0 :ok:
mvndep
0m 36s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
5m 15s
master passed

+1 :green_heart:
compile
1m 47s
master passed

+1 :green_heart:
checkstyle
2m 21s
master passed

+1 :green_heart:
shadedjars
4m 36s
branch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
1m 29s
master passed

+0 :ok:
spotbugs
4m 10s
Used deprecated FindBugs config; considering switching to SpotBugs.

+1 :green_heart:
findbugs
6m  6s
master passed

-0 :warning:
patch
4m 34s
Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 21s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
4m 58s
the patch passed

+1 :green_heart:
compile
1m 46s
the patch passed

+1 :green_heart:
javac
1m 46s
the patch passed

+1 :green_heart:
checkstyle
0m 26s
The patch passed checkstyle in hbase-common

+1 :green_heart:
checkstyle
0m 32s
hbase-client: The patch generated 0 new + 22 unchanged - 10 fixed = 22 total (was 32)

-1 :x:
checkstyle
1m 18s
hbase-server: The patch generated 2 new + 26 unchanged - 3 fixed = 28 total (was 29)

-1 :x:
whitespace
0m  0s
The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply

+1 :green_heart:
xml
0m  1s
The patch has no ill-formed XML file.

+1 :green_heart:
shadedjars
4m 39s
patch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
hadoopcheck
15m 40s
Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.

+1 :green_heart:
javadoc
1m 20s
the patch passed

+1 :green_heart:
findbugs
6m 20s
the patch passed

_ Other Tests _

+1 :green_heart:
unit
3m  8s
hbase-common in the patch passed.

+1 :green_heart:
unit
1m 57s
hbase-client in the patch passed.

+1 :green_heart:
unit
154m 36s
hbase-server in the patch passed.

+1 :green_heart:
asflicense
1m 41s
The patch does not generate ASF License warnings.

228m 42s

Subsystem
Report/Notes

Docker
Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/11/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/884

Optional Tests
dupname asflicense markdownlint javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile xml

uname
Linux 164128e29848 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 GNU/Linux

Build tool
maven

Personality
/home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-884/out/precommit/personality/provided.sh

git revision
master / 82e155eb26

Default Java
1.8.0_181

checkstyle
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/11/artifact/out/diff-checkstyle-hbase-server.txt

whitespace
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/11/artifact/out/whitespace-eol.txt

Test Results
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/11/testReport/

Max. process+thread count
4944 (vs. ulimit of 10000)

modules
C: hbase-common hbase-client hbase-server U: .

Console output
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/11/console

versions
git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

@belugabehr thanks for taking a look. Can you do me a solid?

Batch up requests into a review rather than adding a single comment (keeps my inbox from blowing up)
Try to focus on the changes this PR introduces, rather than verbatim copy-pastes from before.

@joshelser My apologies. I'm pretty clumsy (as you can already tell) with Github review.  Still trying to figure out how to use the tool effectively.

No problem, sir. Thanks for taking the time to come here and leave some feedback.
When leaving a comment, you can click "Start a review" and that will start batching them all together. When you're done, you can click "Finish your review" at the top of the page while will post them all at once.

Client fallback
It looks like we had copied an idea from Hadoop where the client which was configured to auth'n over Kerberos, could fall back to trying to authenticate via SIMPLE. This seems silly to me and is off by default. I see one instance of someone poking at this https://issues.apache.org/jira/browse/HBASE-20993 but I'm missing the "why". I can't think of a reason that we would want to encourage this (not using the correct configuration to talk to HBase).
Am I missing a use-case?

I would suggest asking specifically about this on the mailing list. IIRC the main use case was the ability to enable security on an existing cluster without downtime HBASE-14700 Support a "permissive" mode for secure clusters to allow "simple" auth clients

I haven't seen much difference comparing to the old token based authentication, so I'm a bit nervous that we doing a lot of work and then, no one will actually use it...

Yeah, it's specifically on the roadmap at Cloudera. I think the unit test I provided gives that impression that we're aren't doing much different since we're not really doing anything fancy server-side.

Can we add a more reasonable example in the hbase-example module, to say that, we do have different authentication methods, comparing to the old provided methods?

I've been chatting with Busbey and Wellington about what would be a non-contrived and semi-representative example. It's hard to come up with some single implementation because it

relies on lots of infrastructure that is dependent on the company/organization (e.g. ActiveDirectory or PKI)
has business/data dependent security policies that have to be applied (e.g. encryption strength)

That said, I'm happy to try to put an example together to demonstrate this. The best thing I've been able to come up with is making a userdatabase from a file in HDFS (either a flat file or a JKS), and wiring up HBase to check against that. How does that strike you?
Obviously not ready to be deployed in some organization, but sufficiently decoupled that we can keep maintaining it and (hopefully) representative of what you can do.

I have a question concerning compatibility though it is supposed to be non-compat in 3.0 and it can be set aside as an upcoming if we want: have you tested 1.x or 2.x client talks to 3.x server?

Nothing on the wire has actually changed. We're still using the auth byte and parsing things exactly how we did before. But, no, I've not explicitly tested that.

IIRC the main use case was the ability to enable security on an existing cluster without downtime HBASE-14700 Support a "permissive" mode for secure clusters to allow "simple" auth clients

Thanks! I think this is less of an issue now. I had punted on this originally, but Wellington's reviews lead me back here and fixed it up (at least, according to the unit tests).

I think I have done my first round review.
Nice feature!:+1:

I think I have all outstanding reviews answered/implemented except for:

unwrapUgi() from Duo
Implementing an example in hbase-examples
Stack's package move request

I'll work on the hbase-examples addition next, but need some feedback for the other two.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 32s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  1s
No case conflicting files found.

+0 :ok:
markdownlint
0m  1s
markdownlint was not available.

+1 :green_heart:
hbaseanti
0m  0s
Patch does not have any anti-patterns.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s
The patch appears to include 6 new or modified test files.

_ master Compile Tests _

+0 :ok:
mvndep
0m 35s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
5m 10s
master passed

+1 :green_heart:
compile
1m 46s
master passed

+1 :green_heart:
checkstyle
2m 19s
master passed

+1 :green_heart:
shadedjars
4m 35s
branch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
1m 21s
master passed

+0 :ok:
spotbugs
4m 12s
Used deprecated FindBugs config; considering switching to SpotBugs.

+1 :green_heart:
findbugs
6m  0s
master passed

-0 :warning:
patch
4m 34s
Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 15s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
4m 57s
the patch passed

+1 :green_heart:
compile
1m 45s
the patch passed

+1 :green_heart:
javac
1m 45s
the patch passed

-1 :x:
checkstyle
0m 32s
hbase-client: The patch generated 2 new + 22 unchanged - 10 fixed = 24 total (was 32)

-1 :x:
checkstyle
1m 16s
hbase-server: The patch generated 2 new + 21 unchanged - 3 fixed = 23 total (was 24)

-1 :x:
whitespace
0m  0s
The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply

+1 :green_heart:
xml
0m  1s
The patch has no ill-formed XML file.

+1 :green_heart:
shadedjars
4m 38s
patch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
hadoopcheck
15m 30s
Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.

-1 :x:
javadoc
0m 23s
hbase-client generated 2 new + 2 unchanged - 0 fixed = 4 total (was 2)

+1 :green_heart:
findbugs
6m 35s
the patch passed

_ Other Tests _

+1 :green_heart:
unit
3m 10s
hbase-common in the patch passed.

+1 :green_heart:
unit
1m 54s
hbase-client in the patch passed.

+1 :green_heart:
unit
159m 39s
hbase-server in the patch passed.

+1 :green_heart:
asflicense
1m 38s
The patch does not generate ASF License warnings.

232m 50s

Subsystem
Report/Notes

Docker
Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/13/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/884

Optional Tests
dupname asflicense markdownlint javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile xml

uname
Linux b1a2c1f44e10 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 GNU/Linux

Build tool
maven

Personality
/home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-884/out/precommit/personality/provided.sh

git revision
master / 17e180e4ee

Default Java
1.8.0_181

checkstyle
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/13/artifact/out/diff-checkstyle-hbase-client.txt

checkstyle
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/13/artifact/out/diff-checkstyle-hbase-server.txt

whitespace
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/13/artifact/out/whitespace-eol.txt

javadoc
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/13/artifact/out/diff-javadoc-javadoc-hbase-client.txt

Test Results
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/13/testReport/

Max. process+thread count
4834 (vs. ulimit of 10000)

modules
C: hbase-common hbase-client hbase-server U: .

Console output
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/13/console

versions
git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 33s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  1s
No case conflicting files found.

+0 :ok:
markdownlint
0m  1s
markdownlint was not available.

+1 :green_heart:
hbaseanti
0m  0s
Patch does not have any anti-patterns.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s
The patch appears to include 6 new or modified test files.

_ master Compile Tests _

+0 :ok:
mvndep
0m 36s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
5m 19s
master passed

+1 :green_heart:
compile
1m 45s
master passed

+1 :green_heart:
checkstyle
2m 18s
master passed

+1 :green_heart:
shadedjars
4m 40s
branch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
1m 22s
master passed

+0 :ok:
spotbugs
4m 14s
Used deprecated FindBugs config; considering switching to SpotBugs.

+1 :green_heart:
findbugs
6m  4s
master passed

-0 :warning:
patch
4m 38s
Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 16s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
4m 55s
the patch passed

+1 :green_heart:
compile
1m 46s
the patch passed

+1 :green_heart:
javac
1m 46s
the patch passed

-1 :x:
checkstyle
0m 31s
hbase-client: The patch generated 4 new + 22 unchanged - 10 fixed = 26 total (was 32)

-1 :x:
checkstyle
1m 20s
hbase-server: The patch generated 2 new + 21 unchanged - 3 fixed = 23 total (was 24)

-1 :x:
whitespace
0m  0s
The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply

+1 :green_heart:
xml
0m  1s
The patch has no ill-formed XML file.

+1 :green_heart:
shadedjars
4m 38s
patch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
hadoopcheck
15m 50s
Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.

-1 :x:
javadoc
0m 23s
hbase-client generated 2 new + 2 unchanged - 0 fixed = 4 total (was 2)

+1 :green_heart:
findbugs
6m 31s
the patch passed

_ Other Tests _

+1 :green_heart:
unit
3m 10s
hbase-common in the patch passed.

+1 :green_heart:
unit
1m 55s
hbase-client in the patch passed.

+1 :green_heart:
unit
160m 45s
hbase-server in the patch passed.

+1 :green_heart:
asflicense
1m 40s
The patch does not generate ASF License warnings.

234m 32s

Subsystem
Report/Notes

Docker
Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/14/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/884

Optional Tests
dupname asflicense markdownlint javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile xml

uname
Linux 10689cb7998c 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 GNU/Linux

Build tool
maven

Personality
/home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-884/out/precommit/personality/provided.sh

git revision
master / 17e180e4ee

Default Java
1.8.0_181

checkstyle
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/14/artifact/out/diff-checkstyle-hbase-client.txt

checkstyle
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/14/artifact/out/diff-checkstyle-hbase-server.txt

whitespace
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/14/artifact/out/whitespace-eol.txt

javadoc
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/14/artifact/out/diff-javadoc-javadoc-hbase-client.txt

Test Results
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/14/testReport/

Max. process+thread count
4967 (vs. ulimit of 10000)

modules
C: hbase-common hbase-client hbase-server U: .

Console output
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/14/console

versions
git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

unwrapUgi() from Duo

Thinking about this with a fresh mind: we have this interface method specifically for one purpose -- if a user is authenticating via Kerberos, we must execute an RPC to HBase as the "real" user, not a proxy user.
I see two approaches of equal "correctness" in my mind:

Always get the "real" user from the UGI we have ConnectionID > User > UserGroupInformation and don't push this into the AuthenticationProvider. This makes sense as this is the only "reason" we have in the HBase codebase now to do this -- we don't have any authentication method in which we wouldn't use the user with real UGI credentials to execute an RPC.
Change the method to be UGI getRealUser(User). This leaves our indirection intact, hopefully future-proofing this. It's a little more semantically correct in that sense.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 38s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s
No case conflicting files found.

+0 :ok:
markdownlint
0m  0s
markdownlint was not available.

+1 :green_heart:
hbaseanti
0m  0s
Patch does not have any anti-patterns.

+1 :green_heart:
@author
0m  1s
The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s
The patch appears to include 6 new or modified test files.

_ master Compile Tests _

+0 :ok:
mvndep
0m 36s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
5m 27s
master passed

+1 :green_heart:
compile
1m 51s
master passed

+1 :green_heart:
checkstyle
2m 18s
master passed

+1 :green_heart:
shadedjars
4m 35s
branch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
1m 24s
master passed

+0 :ok:
spotbugs
4m 20s
Used deprecated FindBugs config; considering switching to SpotBugs.

+1 :green_heart:
findbugs
6m 11s
master passed

-0 :warning:
patch
4m 43s
Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 15s
Maven dependency ordering for patch

-1 :x:
mvninstall
1m 29s
root in the patch failed.

-1 :x:
compile
0m 22s
hbase-client in the patch failed.

-1 :x:
compile
0m 36s
hbase-server in the patch failed.

-1 :x:
javac
0m 22s
hbase-client in the patch failed.

-1 :x:
javac
0m 36s
hbase-server in the patch failed.

+1 :green_heart:
checkstyle
0m 23s
The patch passed checkstyle in hbase-common

-1 :x:
checkstyle
0m 30s
hbase-client: The patch generated 1 new + 22 unchanged - 10 fixed = 23 total (was 32)

+1 :green_heart:
checkstyle
1m 17s
hbase-server: The patch generated 0 new + 21 unchanged - 3 fixed = 21 total (was 24)

+1 :green_heart:
whitespace
0m  0s
The patch has no whitespace issues.

+1 :green_heart:
xml
0m  2s
The patch has no ill-formed XML file.

-1 :x:
shadedjars
2m 24s
patch has 14 errors when building our shaded downstream artifacts.

-1 :x:
hadoopcheck
1m 16s
The patch causes 14 errors with Hadoop v2.8.5.

-1 :x:
hadoopcheck
2m 36s
The patch causes 14 errors with Hadoop v2.9.2.

-1 :x:
hadoopcheck
4m  2s
The patch causes 14 errors with Hadoop v3.1.2.

-1 :x:
javadoc
0m 33s
hbase-server generated 4 new + 0 unchanged - 0 fixed = 4 total (was 0)

-1 :x:
findbugs
0m 22s
hbase-client in the patch failed.

-1 :x:
findbugs
0m 36s
hbase-server in the patch failed.

_ Other Tests _

+1 :green_heart:
unit
3m  8s
hbase-common in the patch passed.

-1 :x:
unit
0m 22s
hbase-client in the patch failed.

-1 :x:
unit
0m 37s
hbase-server in the patch failed.

+1 :green_heart:
asflicense
0m 32s
The patch does not generate ASF License warnings.

44m  8s

Subsystem
Report/Notes

Docker
Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/15/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/884

Optional Tests
dupname asflicense markdownlint javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile xml

uname
Linux d4aa6322b8f6 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 GNU/Linux

Build tool
maven

Personality
/home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-884/out/precommit/personality/provided.sh

git revision
master / 17e180e4ee

Default Java
1.8.0_181

mvninstall
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/15/artifact/out/patch-mvninstall-root.txt

compile
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/15/artifact/out/patch-compile-hbase-client.txt

compile
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/15/artifact/out/patch-compile-hbase-server.txt

javac
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/15/artifact/out/patch-compile-hbase-client.txt

javac
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/15/artifact/out/patch-compile-hbase-server.txt

checkstyle
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/15/artifact/out/diff-checkstyle-hbase-client.txt

shadedjars
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/15/artifact/out/patch-shadedjars.txt

hadoopcheck
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/15/artifact/out/patch-javac-2.8.5.txt

hadoopcheck
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/15/artifact/out/patch-javac-2.9.2.txt

hadoopcheck
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/15/artifact/out/patch-javac-3.1.2.txt

javadoc
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/15/artifact/out/diff-javadoc-javadoc-hbase-server.txt

findbugs
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/15/artifact/out/patch-findbugs-hbase-client.txt

findbugs
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/15/artifact/out/patch-findbugs-hbase-server.txt

unit
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/15/artifact/out/patch-unit-hbase-client.txt

unit
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/15/artifact/out/patch-unit-hbase-server.txt

Test Results
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/15/testReport/

Max. process+thread count
288 (vs. ulimit of 10000)

modules
C: hbase-common hbase-client hbase-server U: .

Console output
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/15/console

versions
git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 32s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  1s
No case conflicting files found.

+0 :ok:
markdownlint
0m  1s
markdownlint was not available.

+1 :green_heart:
hbaseanti
0m  0s
Patch does not have any anti-patterns.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s
The patch appears to include 6 new or modified test files.

_ master Compile Tests _

+0 :ok:
mvndep
0m 36s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
5m 16s
master passed

+1 :green_heart:
compile
1m 44s
master passed

+1 :green_heart:
checkstyle
2m 20s
master passed

+1 :green_heart:
shadedjars
4m 38s
branch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
1m 24s
master passed

+0 :ok:
spotbugs
4m  7s
Used deprecated FindBugs config; considering switching to SpotBugs.

+1 :green_heart:
findbugs
5m 57s
master passed

-0 :warning:
patch
4m 30s
Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 15s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
4m 58s
the patch passed

+1 :green_heart:
compile
1m 53s
the patch passed

+1 :green_heart:
javac
1m 53s
the patch passed

+1 :green_heart:
checkstyle
0m 27s
The patch passed checkstyle in hbase-common

-1 :x:
checkstyle
0m 33s
hbase-client: The patch generated 1 new + 22 unchanged - 10 fixed = 23 total (was 32)

+1 :green_heart:
checkstyle
1m 19s
hbase-server: The patch generated 0 new + 21 unchanged - 3 fixed = 21 total (was 24)

+1 :green_heart:
whitespace
0m  0s
The patch has no whitespace issues.

+1 :green_heart:
xml
0m  2s
The patch has no ill-formed XML file.

+1 :green_heart:
shadedjars
4m 35s
patch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
hadoopcheck
15m 46s
Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.

+1 :green_heart:
javadoc
1m 20s
the patch passed

+1 :green_heart:
findbugs
6m 35s
the patch passed

_ Other Tests _

+1 :green_heart:
unit
3m  9s
hbase-common in the patch passed.

+1 :green_heart:
unit
1m 55s
hbase-client in the patch passed.

+1 :green_heart:
unit
165m 27s
hbase-server in the patch passed.

+1 :green_heart:
asflicense
1m 40s
The patch does not generate ASF License warnings.

239m 15s

Subsystem
Report/Notes

Docker
Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/16/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/884

Optional Tests
dupname asflicense markdownlint javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile xml

uname
Linux e6d0e124182b 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 GNU/Linux

Build tool
maven

Personality
/home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-884/out/precommit/personality/provided.sh

git revision
master / 17e180e4ee

Default Java
1.8.0_181

checkstyle
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/16/artifact/out/diff-checkstyle-hbase-client.txt

Test Results
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/16/testReport/

Max. process+thread count
4865 (vs. ulimit of 10000)

modules
C: hbase-common hbase-client hbase-server U: .

Console output
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-884/16/console

versions
git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

I have some Text nits from Andrew, Wellington's suggestion for an init(Configuration), and some removal of SASL from name/structure suggestions from Andrew to (maybe now?) implement. I'll knock these out tmrw, along with finishing up the hbase-example addition that I've been sitting on since the holidays :)
Thanks Reid for also dropping by! Anyone else who is still in the process of reviewing, please shout before I move on without you :)


misleading function name

This function https://github.com/Schoyen/quantum-systems/blob/3499369a0cf42cdc42157126081118b75a4f66b8/quantum_systems/spatial_orbital_system.py#L52
does not change the system itself. It just creates a copy, which can be a bit misleading. Maybe it should be renamed? Either that, or be made static?

Well, it doesn't quite create a copy as the function "destroys" the current SpatialOrbitalSystem. However, I can agree that the name doesn't quite fit; SpatialOrbitalSystem doesn't get so much changed as it gets replaced. Any suggestions? 😀

See suggested fix based on our discussion in #31 :)


Share Extension: dialog may sometimes render empty

Using riot-ios 0.6.12.
The "Share" feature would work most of the time.
In fact, it's only once that I've seen it render empty (as seen on the screenshot below).
This is while trying to share a URL from Firefox.
The first time, it worked and I managed to Share with some "Person". The 2nd time, it rendered empty like this.

I haven't been able to reproduce it since then.

Just had it happen again in 0.7.3, so it seems like it's still a thing.
I shared a link (from Firefox) with a group.. and then tried to share again, and the dialog was empty (same as on the screenshot above).
Maybe it has something to do with the connection being bad and the first send not actually completing..? And me trying to start the share dialog once more while it's in such a state..?


Use recommended constraint

Problem 1
- phpdocumentor/graphviz 2.0.0 requires php ^7.2 || 8.0.* -> your php version (8.1.0) does not satisfy that requirement.
Blocks several apps and libraries trying to use this lib
Note: Config locking of PHP versions is not recommended for libs, only for projects
As it also prevents proper testing of higher versions of libs against your code.
Use prefer lowest instead on top.

@jaapio Are we able to make a patch release here? This would unblock a few libraries and projects of ours to get tested on 8.1+

The unrelated CS issues in CI should be done in a separate PR.

+1

Release done :-)


Error of the server

Hi, all the time that I try to start my server doesn´t go
This is my log: http://pastebin.com/pdu5aitK

java.net.MalformedURLException

You need to specify a valid URL in the fr.xml file.

Ok resolved but appeared this warns http://pastebin.com/s8Wi0TvW

Cardinal is not correctly compiled. Please use official downloads from http://ci.twizmwaz.in/job/Cardinal/


Add prefix/suffix for scope sorts

Sorting allows named scopes to be used. For example, if there is a status attribute that is derived, then the query parameters could include sort=status. However, that means there would need to be a scope on the model called status, which is a little vague.
It would make more sense in the model for the scope to be named sort_by_status or status_sort. However, that makes the required query parameters a little redundant: sort=sort_by_status.
If the scope look up checked using a suffix or prefix (as well as the name as is), then the query parameter could be just the attribute name while the scope could be named with the clarifying suffix/prefix.
Here's the check in NextPage::Sorter#build_sort that would need to be udpated:
        if @model.respond_to?(key)
          ->(query) { query.send(key) }

Scope Prefix / Suffix
In order to keep the peace between frontend and backend developers, scope names can include a prefix or suffix that the front end can ignore. For example, given a scope that sorts on a derived attribute (such as status in the Direction Scope Sorts example), the backend developer might prefer to name the scope status_sort or sort_by_status, as a class method that shares the same name as an attribute might be unclear. However, the frontend developer does not want a query parameter that says sort=sort_by_status; it doesn't match the name of the attribute and it's not pretty.
The configuration allows a prefix and or suffix to be specified. If either is specified, then in addition to looking for a scope that matches the parameter name, it will also look for a scope that matches the prefixed and/or suffixed name. Prefixes are defined by configuration option sort_scope_prefix and suffixes are defined by sort_scope_suffix.
For example, if the backend developer prefers sort_by_status then the following configuration can be used:
NextPage.configure do |config|
  config.sort_scope_prefix = 'sort_by_'
end

This allows the query parameter to be the following:
sort=status

Added in 0.1.7:
Enhanced Sorting


Move binary to data-files in setup.py

Fixes #1

Thanks! I will try to update the package this week.
Cheers, Eneko

Two things:

Could you squash the commits?
Why is data-files a better choice than scripts to save the wkhtmltopdf binary?

@eillarra Sure thing. I'll get this cleaned up

@eillarra The commit is clean now. (And sorry about the delay getting back to you about this!)


Closes: #1

Updated validations.py python script.
Fixed the behavior of validate_user function in validations.py by adding an additional conditional check for special characters.
Fixes #<issue_number_goes_here>

It's a good idea to open an issue first for discussion.

[ ] Tests pass
[ ] Appropriate changes to README are included in PR

@googlebot I signed it!


Fixed NoConstantConditionRuleTest

Test didn't do what the description said and I noticed this was thee same tests as the one on line 50

Thanks for noticing this.


Enable Inlining during compilation.

When profiling a compilation of http4s/core, I noticed that there were several Mb of objects of type Some, Tuple2, or Tuple3, that were created from the better-monadic-for plugin.
In the build of the Scala compiler, the inliner is used to remove and optimise away these small objects. See https://github.com/scala/scala/blob/2.13.x/project/ScriptCommands.scala#L118

Does this actually change anything?  My reading of scalac -opt-inline-from:help leads me to believe that -opt-inline-from:scala/** would only make a difference when you have sources in a directory called scala.
Probably best approach this with a driving JMH benchmark.

~/scala_floss/scalac (optimise_intersect) $ scalac -opt-inline-from:help
Patterns for classfile names from which the inliner is allowed to pull in code.
  a.**           Classes in a and in sub-packages of a

My understanding of this text is that this optionlets the inliner to pull (that is copy and expand) the code from any .class  that belongs to the scala. package. This should include the constructors for these classes Some and Tuple, and the methods to access them.
Not sure how to benchmark this, though.

So is it a typo that you're using -opt-inline-from:scala/** instead of -opt-inline-from:scala.**?


Inroduce kubenetes deployment yamls for dev and prod environments.

Start the artman conductor as part of the deployment file instead of Dockerfile.

Codecov Report

:exclamation: No coverage uploaded for pull request head (jenkins@5aaa089).

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 00d9f15...5aaa089. Read the comment docs.


aft-customizations-pipeline bucket should use S3 block public access

As with the aft-backend buckets, the aft-customizations-pipeline bucket should also enable S3 block all public access to be inline with the AWS Foundational Security Best Practices controls: https://docs.aws.amazon.com/securityhub/latest/userguide/securityhub-standards-fsbp-controls.html#fsbp-s3-8

Thank you for the call out. We have created an internal backlog to track this feature request.
I will go ahead and close this issue for now.


Feature request: Mark unread threads or threads with unread comments

I tip almost every post I read, just so that it's easy to see next time which threads I haven't read yet (it changes the color of the bolt icon).
It would be great if there was a way to see which threads got new comments since I read them last time.

Gmd also asked for this awhile ago. I'd like this too.


Event when player connects/disconnects

posix: https://github.com/decentraland/posix/pull/5
kernel: https://github.com/decentraland/kernel/pull/108
sdk: https://github.com/decentraland/js-sdk-toolchain/pull/47


Support YaRN models (RoFormer implementation in rotary_embedding kernel)

Feature request
@OlivierDehaene
Nous Research and EleutherAI have recently released the YaRN model, which comes in two versions with context sizes of 64k and 128k. This model utilizes RoFormer-style embeddings, distinguishing it from GPT-NeoX and GPT-J. It is built upon the foundation of the LLaMa 2 model, making it largely compatible with some minor adjustments required for optimal support.
Motivation
The YaRN model's longer context length (up to 128k) is highly valuable for tasks involving extensive context, compared to the limited 4096 context length of the llama2 base model.
Your contribution
YaRN paper: YaRN: Efficient Context Window Extension of Large Language Models
YaRN Code: YaRN Github

+1 that would be awesome
there are models The Bloke hub:
For 7b Llama 2: https://huggingface.co/TheBloke/Yarn-Llama-2-7B-128K-GPTQ
For 13b Llama 2: https://huggingface.co/TheBloke/Yarn-Llama-2-13B-128K-GPTQ
Both originating from:
https://huggingface.co/NousResearch/Yarn-Llama-2-7b-128k
https://huggingface.co/NousResearch/Yarn-Llama-2-13b-128k

+1

+1

+1

+1

Just created a PR for it: https://github.com/huggingface/text-generation-inference/pull/1099
However the GPTQ models don't  seem to be working correctly (NousResearch ones seem ok).
If anyone can take this implementation for a spin that'll help figuring out the probable bug.


Clean up manual steps on Android

We have a bunch of manual steps needed to use our cordova plugin on Android. We need to fix that so that everything is automated.

This is covered by #14


Add ability to drop metrics from cAdvisor

@gianrubio
What this PR does / why we need it:
Lots of cAdvisor metrics are simply 0 and can be dropped safely. This reduces a significant number of series in large Kubernetes clusters.
Checklist
[Place an '[x]' (no spaces) in all applicable fields. Please remove unrelated fields.]

[x] DCO signed
[x] Chart Version bumped
[x] Variables are documented in the README.md

Wait, while testing we're seeing that source_labels are not ending up in the configuration for some reason. Investigating. Will update soon.

@gianrubio Can you PTAL now? It's verified to be working now :)

Gentle ping @gianrubio

Hi @gianrubio

@gouthamve Can you please add the value to the values file and the file in the ci/ directory?

@vsliouniaev Done!

/ok-to-test
/lgtm


Update feature tests to use examples from govuk-content-schemas

Trello: https://trello.com/c/uR7v1Rri
Related to:
https://github.com/alphagov/govuk-content-schemas/pull/892
https://github.com/alphagov/govuk-content-schemas/pull/891
Content item types, facet_groups, facets and facet_values were changed to be "base_path-less" in https://github.com/alphagov/govuk-content-schemas/pull/891.
We did this because, that's how they were already being treated in publishing-api and content-store, but we couldn't add re-usable test examples until this was explicitly stated in govuk-content-schemas.
However, this started causing the feature tests in this app to fail with errors like:
 Failure/Error:
       GovukSchemas::RandomExample.for_schema(frontend_schema: schema_name) do |random_item|
         random_item.merge(merge_with)
       end

     GovukSchemas::InvalidContentGenerated:
       The content item you are trying to generate is invalid against the schema.
       The item was valid before being customised.

and
         {
           "schema": "046a2c23-3e82-578d-86d3-8d23a5e24eb2#",
           "fragment": "#/links/taxons/1/links/facet_groups/2",
           "message": "The property '#/links/taxons/1/links/facet_groups/2' did not contain a required property of 'base_path' in schema 046a2c23-3e82-578d-86d3-8d23a5e24eb2#",
           "failed_attribute": "Required"
         },

The feature tests rely on govuk_schemas to create a random content item based on a schema, and then modify a part of it, e.g. the content item title. govuk_schemas validates the content item it creates twice. Once when the base content item is created, and again when the modifications are added, i.e. when the content item is "customised".
It's not entirely clear why the validation was failing. The content item was being created "correctly", but the links for facet_groups and facet_values were being validated against the wrong schema definition.
Also, the links being created were unnecessary. taxons and parent links etc, don't need to be tagged to a facet_group and it's extremely unlikely that they ever will be.
Rather than spending a lot of time trying to fix random content item generation here, the feature tests have been updated to read in already validated examples to generate the links hash.
Note: The tests won't pass until https://github.com/alphagov/govuk-content-schemas/pull/892 is deployed to production.

I suspect the problem arises from the facets relating to their facet group via the parent link https://github.com/alphagov/govuk-content-schemas/blob/master/formats/facet.jsonnet#L91 which is a recursive link type.


Invalid file upload. ENOENT: no such file or directory

This is the continue on issue https://github.com/Mintplex-Labs/anything-llm/issues/1652
Following the suggestion to start collector for production, I can click the big upload button and open a windows to select files to upload. However after the selected files confirmed,  an error in red shows up in the area of big upload button :
Invalid file upload. NOENT: no such file or directory, open '/home/collector/hotdir/mydocument.pdf'
I doubt it is caused by '/home' as the path to collector.
Is there anything lost in the scripts? Please let me paste them again.
echo "Killing node processes"
pkill node

echo "Installing collector dependencies"
cd /home/ubuntu/anything-llm/collector && yarn

echo "Installing server dependencies & running migrations"
cd /home/ubuntu/anything-llm/server && yarn
cd /home/ubuntu/anything-llm/server && npx prisma migrate deploy --schema=./prisma/schema.prisma
cd /home/ubuntu/anything-llm/server && npx prisma generate

echo "Booting up services."
truncate -s 0 /logs/server.log
truncate -s 0 /logs/collector.log

cd /home/ubuntu/anything-llm/server
(NODE_ENV=production node index.js) &> /logs/server.log &

cd /home/ubuntu/anything-llm/collector
(NODE_ENV=production node index.js) &> /logs/collector.log &

BTW everything works fine in development.

I doubt it is caused by '/home' as the path to collector.

Are you sure that is the case? I think writing to that folder is causing the issue. The root /home in Ubuntu is usually write-protected by default and only the user home directory is freely writable /home/<user>

Closing this as its not a bug but it is a setup issue, but changing into discussion

@yongshengma I had the same issue and resolved it by ensuring that the "STORAGE_DIR" parameter in .env file in ./server matches the path whereby the Collector server is actually launched from.
This is necessary as, currently, the Collector defines the document cache "hotdir" to be a relative path (../../collector/hotdir) from where "STORAGE_DIR" is.
In other words, if you have kept the original structure intact, and do not mind the storage to be where the code is, this will work:

Frontend server launched from ./anything-llm/server
Collector server launched from ./anything-llm/collector
STORAGE_DIR in ./server.env defined as 'FULL_PATH/anything-llm/server/storage'

Hope this helps!

@yongshengma I had the same issue and resolved it by ensuring that the "STORAGE_DIR" parameter in .env file in ./server matches the path whereby the Collector server is actually launched from.
This is necessary as, currently, the Collector defines the document cache "hotdir" to be a relative path (../../collector/hotdir) from where "STORAGE_DIR" is.
In other words, if you have kept the original structure intact, and do not mind the storage to be where the code is, this will work:

Frontend server launched from ./anything-llm/server
Collector server launched from ./anything-llm/collector
STORAGE_DIR in ./server.env defined as 'FULL_PATH/anything-llm/server/storage'

Hope this helps!

Exactly like what you said. I changed STORAGE_DIR and the uploading works now. Really helps a lot. Thanks!


Move databases to WiredTiger

Older versions of MongoDB use a storage engine called MMAP. As of MongoDB 3.2, they switched to a new storage engine called WiredTiger. WiredTiger is faster and uses less memory and disk space for collections. Given our relatively tight ram resource limit, we should switch to WiredTiger. Doing so requires dumping and restoring databases and changing some config stuff but will not affect data or code.
Creating an issue to discuss this.

Adam took a pass at this. The DB shrunk from about 3GB to about 0.6GB, which is pretty cool and should bode well for memory usage. The districts DB also shrunk a lot. Indices are way small.
Still todo:

[ ] MongoD is not setup to start using WiredTiger
[ ] There's a weird permission warning, which I think is connected to the above


Question about testing the test2017.

Hi, thanks for your great work. I try to test the test2017 and want to get the test_dev_results, but failed. Could you give me some advice? Thank you.

Please use the Error Template. We do not know which failure you met so we are not able to give any advice here.

Thans for your reply. I forget changing the annotations. And it's ok now.


add error handling for aiohttp clienterror

This is a suggestion for a fix for https://github.com/ansible/event-driven-ansible/issues/46
As it is now, if a site in unavailable, let's see, the web server is simply down, then url_check will error out as we are not catching
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host rhel9apache.sudo.net:80 ssl:default [Connect call failed ('<IP_ADDRESS>', 80)]

It seems like a common usecase to want to be able to fix a broken web server process on a server, so I made an attempt and added error handling. From https://github.com/aio-libs/aiohttp/issues/4064 I read that ClientError catches all types of client side errors, such as no being able to connect at all, SSL errors and etc. So seems like a good fit, if this is a good idea at all :-)

Here's what a run looks like with this patch. The scenario is simply that the webserver process is not running on the target system.
mglantz@darkred event-driven-ansible]$ ansible-rulebook --rulebook website-automation.yml -i inventory --verbose
INFO:ansible_rulebook.app:Starting sources
INFO:ansible_rulebook.app:Starting rules
INFO:ansible_rulebook.engine:run_ruleset
INFO:ansible_rulebook.engine:ruleset define: {"name": "Listen for events on a webhook", "hosts": ["all"], "sources": [{"EventSource": {"name": "ansible.eda.url_check", "source_name": "ansible.eda.url_check", "source_args": {"urls": ["http://rhel9apache.sudo.net"], "delay": 10}, "source_filters": []}}], "rules": [{"Rule": {"name": "Web site is up", "condition": {"AllCondition": [{"EqualsExpression": {"lhs": {"Event": "url_check.status"}, "rhs": {"String": "up"}}}]}, "action": {"Action": {"action": "run_playbook", "action_args": {"name": "site_up.yml"}}}, "enabled": true}}, {"Rule": {"name": "Web site is down", "condition": {"AllCondition": [{"EqualsExpression": {"lhs": {"Event": "url_check.status"}, "rhs": {"String": "down"}}}]}, "action": {"Action": {"action": "run_playbook", "action_args": {"name": "site_down.yml"}}}, "enabled": true}}]}
INFO:ansible_rulebook.engine:load source
INFO:ansible_rulebook.engine:load source filters
INFO:ansible_rulebook.engine:Calling main in ansible.eda.url_check
INFO:ansible_rulebook.engine:Waiting for event from Listen for events on a webhook
INFO:ansible_rulebook.rule_generator:calling Web site is down
INFO:ansible_rulebook.engine:call_action run_playbook
INFO:ansible_rulebook.engine:substitute_variables [{'name': 'site_down.yml'}] [{'event': {'url_check': {'status_code': 500, 'url': 'http://rhel9apache.sudo.net', 'status': 'down'}}, 'fact': {'url_check': {'status_code': 500, 'url': 'http://rhel9apache.sudo.net', 'status': 'down'}}}]
INFO:ansible_rulebook.engine:action args: {'name': 'site_down.yml'}
INFO:ansible_rulebook.builtin:running Ansible playbook: site_down.yml
INFO:ansible_rulebook.builtin:ruleset: Listen for events on a webhook, rule: Web site is down
INFO:ansible_rulebook.builtin:Calling Ansible runner

PLAY [Site Down] ***************************************************************

TASK [debug] *******************************************************************
ok: [rhel9apache.sudo.net] => {
    "msg": "Website is down!"
}

TASK [Restart apache webserver] ************************************************
changed: [rhel9apache.sudo.net]

PLAY RECAP *********************************************************************
rhel9apache.sudo.net       : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
200
INFO:ansible_rulebook.rule_generator:calling Web site is up
INFO:ansible_rulebook.engine:call_action run_playbook
INFO:ansible_rulebook.engine:substitute_variables [{'name': 'site_up.yml'}] [{'event': {'url_check': {'status_code': 200, 'url': 'http://rhel9apache.sudo.net', 'status': 'up'}}, 'fact': {'url_check': {'status_code': 200, 'url': 'http://rhel9apache.sudo.net', 'status': 'up'}}}]
INFO:ansible_rulebook.engine:action args: {'name': 'site_up.yml'}
INFO:ansible_rulebook.builtin:running Ansible playbook: site_up.yml
INFO:ansible_rulebook.builtin:ruleset: Listen for events on a webhook, rule: Web site is up
INFO:ansible_rulebook.builtin:Calling Ansible runner

PLAY [Site is up] **************************************************************

TASK [debug] *******************************************************************
ok: [rhel9apache.sudo.net] => {
    "msg": "All is up and well"
}
ok: [...] => {
    "msg": "All is up and well"
}

PLAY RECAP *********************************************************************
...                        : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
rhel9apache.sudo.net       : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0


Electron 113

Fixes code duplication (and hence regression) issues raised by Navya.
@VikasShashidhar @lneir Please review and merge.

I'll close this as PR https://github.com/symphonyoss/SymphonyElectron/pull/185 contains all these changes.