HADOOP-19052.Hadoop use Shell command to get the count of the hard link which takes a lot of time

Description of PR
As described in HADOOP_19052. When we try to append a file, we will execute method getHardLinkCount twice. When we execute method getHardLinkCount java will start a new process to execute a shell command and wait for it to fork. When the QPS of append execution is very high, method  getHardLinkCount will take a long time to finish which will cause a long-time wait to acquire lock.
I used another method to get the linkCount of a file whose file store supports the file attributes identified by the given file attribute view. This method does not start a new process and will finish in very short time even if the QPS of append execution is high.
How was this patch tested?
I add a new UT testGetLinkCountFromFileAttribute and a public method supportsHardLink to get whether or not this file store supports the file attributes identified by the given file attribute view.
For code changes:

[x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
[ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 22s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  1s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 1 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
31m 18s

trunk passed

+1 :green_heart:
compile
8m 11s

trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
compile
7m 20s

trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08

+1 :green_heart:
checkstyle
0m 40s

trunk passed

+1 :green_heart:
mvnsite
0m 51s

trunk passed

+1 :green_heart:
javadoc
0m 42s

trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
0m 27s

trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08

+1 :green_heart:
spotbugs
1m 25s

trunk passed

+1 :green_heart:
shadedclient
21m  0s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
0m 29s

the patch passed

+1 :green_heart:
compile
7m 45s

the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javac
7m 45s

the patch passed

+1 :green_heart:
compile
7m 23s

the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08

+1 :green_heart:
javac
7m 23s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

-0 :warning:
checkstyle
0m 34s
/results-checkstyle-hadoop-common-project_hadoop-common.txt
hadoop-common-project/hadoop-common: The patch generated 4 new + 42 unchanged - 0 fixed = 46 total (was 42)

+1 :green_heart:
mvnsite
0m 49s

the patch passed

+1 :green_heart:
javadoc
0m 36s

the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
0m 30s

the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08

+1 :green_heart:
spotbugs
1m 24s

the patch passed

+1 :green_heart:
shadedclient
20m 11s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
16m 15s

hadoop-common in the patch passed.

+1 :green_heart:
asflicense
0m 35s

The patch does not generate ASF License warnings.

130m 47s

Subsystem
Report/Notes

Docker
ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6527/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/6527

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets

uname
Linux d611acab831f 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / f810715d570523fead07564f3fb3dd20db5afcd1

Default Java
Private Build-1.8.0_392-8u392-ga-1~20.04-b08

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6527/1/testReport/

Max. process+thread count
1273 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6527/1/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 21s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 1 new or modified test files.

_ trunk Compile Tests _

-1 :x:
mvninstall
0m 23s
/branch-mvninstall-root.txt
root in trunk failed.

-1 :x:
compile
0m 21s
/branch-compile-root-jdkUbuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.txt
root in trunk failed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.

-1 :x:
compile
0m 23s
/branch-compile-root-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt
root in trunk failed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08.

-0 :warning:
checkstyle
0m 20s
/buildtool-branch-checkstyle-hadoop-common-project_hadoop-common.txt
The patch fails to run checkstyle in hadoop-common

-1 :x:
mvnsite
0m 21s
/branch-mvnsite-hadoop-common-project_hadoop-common.txt
hadoop-common in trunk failed.

-1 :x:
javadoc
0m 22s
/branch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.txt
hadoop-common in trunk failed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.

-1 :x:
javadoc
0m 21s
/branch-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt
hadoop-common in trunk failed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08.

-1 :x:
spotbugs
0m 22s
/branch-spotbugs-hadoop-common-project_hadoop-common.txt
hadoop-common in trunk failed.

+1 :green_heart:
shadedclient
2m 32s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

-1 :x:
mvninstall
0m 23s
/patch-mvninstall-hadoop-common-project_hadoop-common.txt
hadoop-common in the patch failed.

-1 :x:
compile
0m 22s
/patch-compile-root-jdkUbuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.txt
root in the patch failed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.

-1 :x:
javac
0m 22s
/patch-compile-root-jdkUbuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.txt
root in the patch failed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.

-1 :x:
compile
0m 22s
/patch-compile-root-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt
root in the patch failed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08.

-1 :x:
javac
0m 22s
/patch-compile-root-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt
root in the patch failed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08.

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

-0 :warning:
checkstyle
0m 20s
/buildtool-patch-checkstyle-hadoop-common-project_hadoop-common.txt
The patch fails to run checkstyle in hadoop-common

-1 :x:
mvnsite
0m 22s
/patch-mvnsite-hadoop-common-project_hadoop-common.txt
hadoop-common in the patch failed.

-1 :x:
javadoc
0m 22s
/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.txt
hadoop-common in the patch failed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.

-1 :x:
javadoc
0m 22s
/patch-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt
hadoop-common in the patch failed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08.

-1 :x:
spotbugs
0m 22s
/patch-spotbugs-hadoop-common-project_hadoop-common.txt
hadoop-common in the patch failed.

+1 :green_heart:
shadedclient
4m 14s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

-1 :x:
unit
0m 22s
/patch-unit-hadoop-common-project_hadoop-common.txt
hadoop-common in the patch failed.

+0 :ok:
asflicense
0m 22s

ASF License check generated no output?

12m 45s

Subsystem
Report/Notes

Docker
ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6527/2/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/6527

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets

uname
Linux e2238e84e519 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / fc72fe8e72b7a87c05b02e0ff8b5009bddd44a71

Default Java
Private Build-1.8.0_392-8u392-ga-1~20.04-b08

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6527/2/testReport/

Max. process+thread count
9 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6527/2/console

versions
git=2.25.1 maven=3.6.3

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 21s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 1 new or modified test files.

_ trunk Compile Tests _

-1 :x:
mvninstall
0m 22s
/branch-mvninstall-root.txt
root in trunk failed.

-1 :x:
compile
0m 22s
/branch-compile-root-jdkUbuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.txt
root in trunk failed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.

-1 :x:
compile
0m 22s
/branch-compile-root-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt
root in trunk failed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08.

-0 :warning:
checkstyle
0m 20s
/buildtool-branch-checkstyle-hadoop-common-project_hadoop-common.txt
The patch fails to run checkstyle in hadoop-common

-1 :x:
mvnsite
0m 22s
/branch-mvnsite-hadoop-common-project_hadoop-common.txt
hadoop-common in trunk failed.

-1 :x:
javadoc
0m 22s
/branch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.txt
hadoop-common in trunk failed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.

-1 :x:
javadoc
0m 22s
/branch-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt
hadoop-common in trunk failed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08.

-1 :x:
spotbugs
0m 22s
/branch-spotbugs-hadoop-common-project_hadoop-common.txt
hadoop-common in trunk failed.

+1 :green_heart:
shadedclient
2m 33s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

-1 :x:
mvninstall
0m 22s
/patch-mvninstall-hadoop-common-project_hadoop-common.txt
hadoop-common in the patch failed.

-1 :x:
compile
0m 22s
/patch-compile-root-jdkUbuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.txt
root in the patch failed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.

-1 :x:
javac
0m 22s
/patch-compile-root-jdkUbuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.txt
root in the patch failed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.

-1 :x:
compile
0m 22s
/patch-compile-root-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt
root in the patch failed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08.

-1 :x:
javac
0m 22s
/patch-compile-root-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt
root in the patch failed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08.

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

-0 :warning:
checkstyle
0m 20s
/buildtool-patch-checkstyle-hadoop-common-project_hadoop-common.txt
The patch fails to run checkstyle in hadoop-common

-1 :x:
mvnsite
0m 22s
/patch-mvnsite-hadoop-common-project_hadoop-common.txt
hadoop-common in the patch failed.

-1 :x:
javadoc
0m 22s
/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.txt
hadoop-common in the patch failed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.

-1 :x:
javadoc
0m 22s
/patch-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt
hadoop-common in the patch failed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08.

-1 :x:
spotbugs
0m 23s
/patch-spotbugs-hadoop-common-project_hadoop-common.txt
hadoop-common in the patch failed.

+1 :green_heart:
shadedclient
4m 12s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

-1 :x:
unit
0m 22s
/patch-unit-hadoop-common-project_hadoop-common.txt
hadoop-common in the patch failed.

+0 :ok:
asflicense
0m 22s

ASF License check generated no output?

12m 43s

Subsystem
Report/Notes

Docker
ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6527/3/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/6527

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets

uname
Linux 83c945d0e036 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / f0b7a765d971d6c56d0e6395d80ce2c221609ac0

Default Java
Private Build-1.8.0_392-8u392-ga-1~20.04-b08

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6527/3/testReport/

Max. process+thread count
29 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6527/3/console

versions
git=2.25.1 maven=3.6.3

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.


Webhooks broken and SSL cert invalid in eu-west-2

Hi,
The SSL certificate used for https://webhooks.amplify.eu-west-2.amazonaws.com appears to be invalid meaning I have to pass a --insecure flag to curl in order to post to the webhook. This obviously isn't ideal and seems to be an error since the webhook url for us-east-1 looks to be correct.

In fact, POSTing to the webhook url displayed in the amplify console doesn't seem to work at all and returns this content:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<TITLE>ERROR: The request could not be satisfied</TITLE>

403 ERROR

The request could not be satisfied.

Bad request.
We can't connect to the server for this app or website at this time. There might be too much traffic or a configuration error. Try again later, or contact the app or website owner.

If you provide content to customers through CloudFront, you can find steps to troubleshoot and help prevent this error by reviewing the CloudFront documentation.

Generated by cloudfront (CloudFront)
Request ID: Bp4JlvjiAEK1VZD_nL4-Cabli2XTGUvrwvMp0WkQvV60lt5c-scvTw==

<ADDRESS>
</ADDRESS>

Hi, thanks for reporting - we've pushed a fix and this should work now. Please re-open if you're still seeing issues.

Thanks - will do.
Tom Roberts
Head of Digital Studio
t.roberts<EMAIL_ADDRESS>On 18 Nov 2020, at 21:23, Don J. Kahn<EMAIL_ADDRESS>wrote:

Hi, thanks for reporting - we've pushed a fix and this should work now. Please re-open if you're still seeing issues.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHubhttps://github.com/aws-amplify/amplify-console/issues/1241#issuecomment-729963835, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABEYFZY2BATTHPP7MQGNPETSQQ3MTANCNFSM4TZ5LNGQ.

Great, thanks!