HADOOP-17990. Fix failing concurrent FS.initialize commands when fs.azure.createRemoteFileSystemDuringInitialization is enabled.

Description of PR
When fs.azure.createRemoteFileSystemDuringInitialization is enabled, the filesystem will create a container if it does not already exist inside the initialize method. The current flow of creating the container will fail in the case of concurrent initialize methods being executed simultaneously (only one request can create the container, the rest will fail instead of moving on). This PR is fixing this issue by also catching org.apache.Hadoop.fs.FileAlreadyExistsException generated by the createFilesystem command.
How was this patch tested?
A new test in ITestAzureBlobFileSystemInitAndCreate is introduced which was breaking before the fox.
For code changes:

[x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
[ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 57s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 1 new or modified test files.

_ trunk Compile Tests _

-1 :x:
mvninstall
51m  8s
/branch-mvninstall-root.txt
root in trunk failed.

-1 :x:
compile
0m 23s
/branch-compile-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt
hadoop-azure in trunk failed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.

-1 :x:
compile
0m 23s
/branch-compile-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt
hadoop-azure in trunk failed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.

+1 :green_heart:
checkstyle
0m 29s

trunk passed

-1 :x:
mvnsite
0m 24s
/branch-mvnsite-hadoop-tools_hadoop-azure.txt
hadoop-azure in trunk failed.

-1 :x:
javadoc
0m 24s
/branch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt
hadoop-azure in trunk failed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.

-1 :x:
javadoc
0m 24s
/branch-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt
hadoop-azure in trunk failed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.

-1 :x:
spotbugs
0m 23s
/branch-spotbugs-hadoop-tools_hadoop-azure.txt
hadoop-azure in trunk failed.

+1 :green_heart:
shadedclient
25m 51s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

-1 :x:
mvninstall
0m 16s
/patch-mvninstall-hadoop-tools_hadoop-azure.txt
hadoop-azure in the patch failed.

-1 :x:
compile
0m 16s
/patch-compile-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt
hadoop-azure in the patch failed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.

-1 :x:
javac
0m 16s
/patch-compile-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt
hadoop-azure in the patch failed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.

-1 :x:
compile
0m 16s
/patch-compile-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt
hadoop-azure in the patch failed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.

-1 :x:
javac
0m 16s
/patch-compile-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt
hadoop-azure in the patch failed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

-0 :warning:
checkstyle
0m 21s
/results-checkstyle-hadoop-tools_hadoop-azure.txt
hadoop-tools/hadoop-azure: The patch generated 2 new + 2 unchanged - 0 fixed = 4 total (was 2)

-1 :x:
mvnsite
0m 18s
/patch-mvnsite-hadoop-tools_hadoop-azure.txt
hadoop-azure in the patch failed.

-1 :x:
javadoc
0m 16s
/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt
hadoop-azure in the patch failed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.

-1 :x:
javadoc
0m 16s
/patch-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt
hadoop-azure in the patch failed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.

-1 :x:
spotbugs
0m 16s
/patch-spotbugs-hadoop-tools_hadoop-azure.txt
hadoop-azure in the patch failed.

+1 :green_heart:
shadedclient
26m 44s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

-1 :x:
unit
0m 18s
/patch-unit-hadoop-tools_hadoop-azure.txt
hadoop-azure in the patch failed.

+1 :green_heart:
asflicense
0m 35s

The patch does not generate ASF License warnings.

109m  5s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3620/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/3620

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell

uname
Linux 34101780a730 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / a53b8fe46288fdbacea025af63cb7d50a7aecbbc

Default Java
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3620/1/testReport/

Max. process+thread count
524 (vs. ulimit of 5500)

modules
C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3620/1/console

versions
git=2.25.1 maven=3.6.3

Powered by
Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

had a quick look at the code

hadoop trunk has its own VisibleForTesting as we are trying do cut guava out of a lot of source (at least isolate it)
see the testing abfs requirements & declare which azure endpoint you ran the test suite, ideally including options

example https://github.com/apache/hadoop/pull/3611#issuecomment-961180315

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 52s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 1 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
35m  8s

trunk passed

+1 :green_heart:
compile
0m 38s

trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
compile
0m 33s

trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
checkstyle
0m 24s

trunk passed

+1 :green_heart:
mvnsite
0m 38s

trunk passed

+1 :green_heart:
javadoc
0m 30s

trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javadoc
0m 27s

trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
spotbugs
1m  0s

trunk passed

+1 :green_heart:
shadedclient
22m 50s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

-1 :x:
mvninstall
0m 26s
/patch-mvninstall-hadoop-tools_hadoop-azure.txt
hadoop-azure in the patch failed.

-1 :x:
compile
0m 30s
/patch-compile-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt
hadoop-azure in the patch failed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.

-1 :x:
javac
0m 30s
/patch-compile-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt
hadoop-azure in the patch failed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.

-1 :x:
compile
0m 26s
/patch-compile-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt
hadoop-azure in the patch failed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.

-1 :x:
javac
0m 26s
/patch-compile-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt
hadoop-azure in the patch failed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

-0 :warning:
checkstyle
0m 17s
/results-checkstyle-hadoop-tools_hadoop-azure.txt
hadoop-tools/hadoop-azure: The patch generated 2 new + 2 unchanged - 0 fixed = 4 total (was 2)

-1 :x:
mvnsite
0m 27s
/patch-mvnsite-hadoop-tools_hadoop-azure.txt
hadoop-azure in the patch failed.

+1 :green_heart:
javadoc
0m 22s

the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javadoc
0m 20s

the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

-1 :x:
spotbugs
0m 26s
/patch-spotbugs-hadoop-tools_hadoop-azure.txt
hadoop-azure in the patch failed.

+1 :green_heart:
shadedclient
24m 44s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

-1 :x:
unit
0m 29s
/patch-unit-hadoop-tools_hadoop-azure.txt
hadoop-azure in the patch failed.

+1 :green_heart:
asflicense
0m 30s

The patch does not generate ASF License warnings.

90m 36s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3620/2/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/3620

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell

uname
Linux 918868c68e5e 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 2fa8e9f4c9ad9178111ef44ea39cac77621714c3

Default Java
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3620/2/testReport/

Max. process+thread count
594 (vs. ulimit of 5500)

modules
C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3620/2/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

Thanks for the review, I have addressed the commets.

The annotation used in AzureBlobFileSystem is org.apache.hadoop.thirdparty.com.google.common.annotations.VisibleForTesting
I ran the dev-support/testrun-scripts/runtests.sh using the own HNS enabled storage.

Test result output
AppendBlob-HNS-OAuth
========================
[INFO] Results:
[INFO]
[WARNING] Tests run: 106, Failures: 0, Errors: 0, Skipped: 26
[INFO] Results:
[INFO]
[WARNING] Tests run: 559, Failures: 0, Errors: 0, Skipped: 559
[INFO] Results:
[INFO]
[WARNING] Tests run: 257, Failures: 0, Errors: 0, Skipped: 257

HNS-OAuth
========================
[INFO] Results:
[INFO]
[WARNING] Tests run: 106, Failures: 0, Errors: 0, Skipped: 26
[INFO] Results:
[INFO]
[WARNING] Tests run: 559, Failures: 0, Errors: 0, Skipped: 559
[INFO] Results:
[INFO]
[WARNING] Tests run: 257, Failures: 0, Errors: 0, Skipped: 257

HNS-SharedKey
========================
[INFO] Results:
[INFO]
[WARNING] Tests run: 106, Failures: 0, Errors: 0, Skipped: 26
[INFO] Results:
[INFO]
[WARNING] Tests run: 559, Failures: 0, Errors: 0, Skipped: 559
[INFO] Results:
[INFO]
[WARNING] Tests run: 257, Failures: 0, Errors: 0, Skipped: 257

NonHNS-SharedKey
========================
[INFO] Results:
[INFO]
[WARNING] Tests run: 106, Failures: 0, Errors: 0, Skipped: 26
[INFO] Results:
[INFO]
[WARNING] Tests run: 559, Failures: 0, Errors: 0, Skipped: 559
[INFO] Results:
[INFO]
[WARNING] Tests run: 257, Failures: 0, Errors: 0, Skipped: 257

I saw this compilation error coming from Yetus, is this expected?

yetus has been a bit flaky recently (I managed to break trunk & 3.3; some docker image/openssl issues, but these seem new)
[WARNING] The requested profile "yarn-ui" could not be activated because it does not exist.
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile (default-testCompile) on project hadoop-azure: Compilation failure
[ERROR] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-3620/ubuntu-focal/src/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemInitAndCreate.java:[83,6] error: constructor AzureBlobFileSystemStore in class AzureBlobFileSystemStore cannot be applied to given types;
[ERROR] -> [Help 1]
[ERROR]

style
./hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemInitAndCreate.java:80:    boolean isCreateFileSystemCalled = false;:13: Variable 'isCreateFileSystemCalled' must be private and have accessor methods. [VisibilityModifier]
./hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemInitAndCreate.java:82:    public MockAzureBlobFileSystemStore(Configuration config) throws IOException {:5: Redundant 'public' modifier. [RedundantModifier]
```

this is CI doing its work, afraid you do get to fix them.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 54s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 1 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
36m 29s

trunk passed

+1 :green_heart:
compile
0m 45s

trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
compile
0m 37s

trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
checkstyle
0m 26s

trunk passed

+1 :green_heart:
mvnsite
0m 44s

trunk passed

+1 :green_heart:
javadoc
0m 34s

trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javadoc
0m 30s

trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
spotbugs
1m 10s

trunk passed

+1 :green_heart:
shadedclient
24m 21s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

-1 :x:
mvninstall
0m 30s
/patch-mvninstall-hadoop-tools_hadoop-azure.txt
hadoop-azure in the patch failed.

-1 :x:
compile
0m 37s
/patch-compile-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt
hadoop-azure in the patch failed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.

-1 :x:
javac
0m 37s
/patch-compile-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt
hadoop-azure in the patch failed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.

-1 :x:
compile
0m 30s
/patch-compile-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt
hadoop-azure in the patch failed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.

-1 :x:
javac
0m 30s
/patch-compile-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt
hadoop-azure in the patch failed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
0m 19s

the patch passed

-1 :x:
mvnsite
0m 33s
/patch-mvnsite-hadoop-tools_hadoop-azure.txt
hadoop-azure in the patch failed.

+1 :green_heart:
javadoc
0m 25s

the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javadoc
0m 23s

the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

-1 :x:
spotbugs
0m 31s
/patch-spotbugs-hadoop-tools_hadoop-azure.txt
hadoop-azure in the patch failed.

+1 :green_heart:
shadedclient
26m  3s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

-1 :x:
unit
0m 34s
/patch-unit-hadoop-tools_hadoop-azure.txt
hadoop-azure in the patch failed.

+1 :green_heart:
asflicense
0m 33s

The patch does not generate ASF License warnings.

96m  7s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3620/3/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/3620

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell

uname
Linux 3073ecc6d04e 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 09264770f5adfcba272d4664e11f6fd1da178974

Default Java
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3620/3/testReport/

Max. process+thread count
588 (vs. ulimit of 5500)

modules
C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3620/3/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.


Installer: API & Types Definition Update (Subchain)

Technical Story

Description

Requirements

Blocked by

Acceptance criteria

Important things we need to watch out for:

Since we are saving at every step, it is important that we cater for the case when a subchain name can be modified. This should happen without having to create a new configuration
What will happen in the case that the base-chain is modified?


[bug]: Lnd doesn't seem to gracefully shutdown on 0.17.0-beta.rc1

Background
I am running lnd 0.17.0 rc1 plus the fix of Elle that was merged recently. When trying to restart or shutting down LND my docker container never stops, I have 15min timeout in the compose file before a kill signal is sent by docker. I just get these logs and the timeout eventually hits and the container gets killed so LND doesn't shut down gracefully.
2023-08-26 20:26:08.468 [ERR] DISC: Update edge for short_chan_id(878811056959193089) got: router shutting down
2023-08-26 20:26:08.468 [ERR] DISC: Update edge for short_chan_id(818216971034558465) got: router shutting down
2023-08-26 20:26:08.468 [ERR] DISC: Update edge for short_chan_id(878297584961191937) got: router shutting down
2023-08-26 20:26:08.469 [ERR] DISC: Update edge for short_chan_id(771274421704720385) got: router shutting down
2023-08-26 20:26:13.261 [ERR] DISC: Update edge for short_chan_id(865996249021546496) got: router shutting down
2023-08-26 20:26:13.374 [ERR] DISC: Update edge for short_chan_id(875772006717980672) got: router shutting down
Your environment

version of 0.17.0-beta.rc1
ubuntu
bitcoind 24

Could you please pull a goroutine dump while it's attempting to shut down? That would help a lot.

What extra patch are you running? W/e that was should be already folded into the latest rc.

What extra patch are you running? W/e that was should be already folded into the latest rc.

The extra patch that is being referred here is this fix: https://github.com/lightningnetwork/lnd/pull/7922
@niteshbalusu11 did you compile the master, or applied this pr to a branch of yours?

I compiled master.

Could you please pull a goroutine dump while it's attempting to shut down? That would help a lot.

How do I do this?

Enable profiling
profile=4000
curl http://localhost:4000/debug/pprof/goroutine?debug=1
curl: (56) Recv failure: Connection reset by peer
I enabled profiling in conf file, restarted lnd and then after it started back up, i just get this when attempting to curl

curl: (56) Recv failure: Connection reset by peer

Ah, so I guess that part is already shut down...
Can you try sending SIGQUIT (kill -3 <pid>) and capturing stdout (it should output a goroutine dump).

curl: (56) Recv failure: Connection reset by peer

Ah, so I guess that part is already shut down... Can you try sending SIGQUIT (kill -3 <pid>) and capturing stdout (it should output a goroutine dump).

logs.zip

Thanks a lot!
Took me a while to comb through ~19k lines... Might be I missed something but this specific area looks suspicious to me:
There are 218 goroutines waiting for MarkEdgeLive:
goroutine 49695 [sync.Mutex.Lock]:
runtime.gopark(0x20977310290c469?, 0x14a898b01b63e7c?, 0x2a?, 0xd2?, 0x331e34001f682a8?)
                runtime/proc.go:381 +0xd6 fp=0xc00661d460 sp=0xc00661d440 pc=0x43a236
runtime.goparkunlock(...)
                runtime/proc.go:387
runtime.semacquire1(0xc0009fe5b4, 0x0?, 0x3, 0x1, 0xce?)
                runtime/sema.go:160 +0x20f fp=0xc00661d4c8 sp=0xc00661d460 pc=0x44b5af
sync.runtime_SemacquireMutex(0x29d42370128852f?, 0xc?, 0x3e7f388006cd0bf?)
                runtime/sema.go:77 +0x26 fp=0xc00661d500 sp=0xc00661d4c8 pc=0x468e86
sync.(*Mutex).lockSlow(0xc0009fe5b0)
                sync/mutex.go:171 +0x165 fp=0xc00661d550 sp=0xc00661d500 pc=0x489b65
sync.(*Mutex).Lock(...)
                sync/mutex.go:90
sync.(*RWMutex).Lock(0xc00661d300?)
                sync/rwmutex.go:147 +0x36 fp=0xc00661d580 sp=0xc00661d550 pc=0x48b1b6
github.com/lightningnetwork/lnd/channeldb.(*ChannelGraph).MarkEdgeLive(0xc0009fe5a0, 0x0)
                github.com/lightningnetwork/lnd/channeldb/graph.go:3896 +0x65 fp=0xc00661d610 sp=0xc00661d580 pc=0x1514105
github.com/lightningnetwork/lnd/routing.(*ChannelRouter).MarkEdgeLive(0xc005be2000?, {0x4644600?, 0xc0?, 0x0?})
                github.com/lightningnetwork/lnd/routing/router.go:2766 +0x35 fp=0xc00661d630 sp=0xc00661d610 pc=0x166a1f5

And 43 goroutines waiting for HasChannelEdge:
goroutine 50286 [sync.Mutex.Lock]:
runtime.gopark(0x7f73e6c5df18?, 0xa78706?, 0x0?, 0x0?, 0x155?)
                runtime/proc.go:381 +0xd6 fp=0xc006722438 sp=0xc006722418 pc=0x43a236
runtime.goparkunlock(...)
                runtime/proc.go:387
runtime.semacquire1(0xc0009fe5b4, 0x0?, 0x3, 0x1, 0x26?)
                runtime/sema.go:160 +0x20f fp=0xc0067224a0 sp=0xc006722438 pc=0x44b5af
sync.runtime_SemacquireMutex(0x16000018?, 0x0?, 0x1940?)
                runtime/sema.go:77 +0x26 fp=0xc0067224d8 sp=0xc0067224a0 pc=0x468e86
sync.(*Mutex).lockSlow(0xc0009fe5b0)
                sync/mutex.go:171 +0x165 fp=0xc006722528 sp=0xc0067224d8 pc=0x489b65
sync.(*Mutex).Lock(...)
                sync/mutex.go:90
sync.(*RWMutex).Lock(0xc0009fe5b0?)
                sync/rwmutex.go:147 +0x36 fp=0xc006722558 sp=0xc006722528 pc=0x48b1b6
github.com/lightningnetwork/lnd/channeldb.(*ChannelGraph).HasChannelEdge(0xc0009fe5a0, 0x9241f000c630000)
                github.com/lightningnetwork/lnd/channeldb/graph.go:1138 +0x2f4 fp=0xc0067226a0 sp=0xc006722558 pc=0x1505e14
github.com/lightningnetwork/lnd/routing.(*ChannelRouter).IsKnownEdge(0x9241f000c630000?, {0x1f7b5fd?, 0x0?, 0x2a0c?})
                github.com/lightningnetwork/lnd/routing/router.go:2700 +0x35 fp=0xc0067226d0 sp=0xc0067226a0 pc=0x1669f35

Not sure if we changed anything in that area or if that is a red herring (and it's just normal graph sync and zombie resurrection going on).
@niteshbalusu11 for how long has the node been running now without shutting down?
The other mutex waiting for a lock is this one, which sounds more like something we recently worked on (cc @yyforyongyu):
goroutine 43625 [sync.Mutex.Lock]:
runtime.gopark(0x10?, 0xc0006dae38?, 0xb0?, 0x5b?, 0x40dbe8?)
                runtime/proc.go:381 +0xd6 fp=0xc0029f5b38 sp=0xc0029f5b18 pc=0x43a236
runtime.goparkunlock(...)
                runtime/proc.go:387
runtime.semacquire1(0xc0017161cc, 0x0?, 0x3, 0x1, 0x31?)
                runtime/sema.go:160 +0x20f fp=0xc0029f5ba0 sp=0xc0029f5b38 pc=0x44b5af
sync.runtime_SemacquireMutex(0xc0029f5c18?, 0x63?, 0xc004fe17c0?)
                runtime/sema.go:77 +0x26 fp=0xc0029f5bd8 sp=0xc0029f5ba0 pc=0x468e86
sync.(*Mutex).lockSlow(0xc0017161c8)
                sync/mutex.go:171 +0x165 fp=0xc0029f5c28 sp=0xc0029f5bd8 pc=0x489b65
sync.(*Mutex).Lock(...)
                sync/mutex.go:90
github.com/lightningnetwork/lnd/multimutex.(*Mutex[...]).Lock(0x22e06a0, {0x73, 0xa, 0x2d, 0x16, 0xcc, 0x88, 0x19, 0x79, 0xc9, ...})
                github.com/lightningnetwork/lnd/multimutex/multimutex.go:58 +0x11a fp=0xc0029f5c60 sp=0xc0029f5c28 pc=0x9f4eda
github.com/lightningnetwork/lnd/blockcache.(*BlockCache).GetBlock(0xc0006780a0, 0xc001802d60, 0xc0029f5e38)
                github.com/lightningnetwork/lnd/blockcache/blockcache.go:38 +0xa5 fp=0xc0029f5da0 sp=0xc0029f5c60 pc=0x1630ee5
github.com/lightningnetwork/lnd/chainntnfs/bitcoindnotify.(*BitcoindNotifier).GetBlock(...)
                github.com/lightningnetwork/lnd/chainntnfs/bitcoindnotify/bitcoind.go:1037
github.com/lightningnetwork/lnd/chainntnfs/bitcoindnotify.(*BitcoindNotifier).historicalSpendDetails(0xc0000f9290, {{{0x89, 0x62, 0xc7, 0x6f, 0x40, 0x1c, 0x95, 0x0, 0x5c, ...}, ...}, ...}, ...)
                github.com/lightningnetwork/lnd/chainntnfs/bitcoindnotify/bitcoind.go:853 +0xe7 fp=0xc0029f5e78 sp=0xc0029f5da0 pc=0x1681fa7
github.com/lightningnetwork/lnd/chainntnfs/bitcoindnotify.(*BitcoindNotifier).notificationDispatcher.func2(0xc0018007d0)
                github.com/lightningnetwork/lnd/chainntnfs/bitcoindnotify/bitcoind.go:300 +0x118 fp=0xc0029f5fc8 sp=0xc0029f5e78 pc=0x167fff8
github.com/lightningnetwork/lnd/chainntnfs/bitcoindnotify.(*BitcoindNotifier).notificationDispatcher.func5()
                github.com/lightningnetwork/lnd/chainntnfs/bitcoindnotify/bitcoind.go:339 +0x2a fp=0xc0029f5fe0 sp=0xc0029f5fc8 pc=0x167feaa
runtime.goexit()
                runtime/asm_amd64.s:1598 +0x1 fp=0xc0029f5fe8 sp=0xc0029f5fe0 pc=0x46d261
created by github.com/lightningnetwork/lnd/chainntnfs/bitcoindnotify.(*BitcoindNotifier).notificationDispatcher
                github.com/lightningnetwork/lnd/chainntnfs/bitcoindnotify/bitcoind.go:297 +0x94b

for how long has the node been running now without shutting down?

I had to restart a few times to get these logs so between the last two restarts it's like 5 mins apart.

Okay, this seems to be related to some of the other reported issues, think this is the most relevant one:
goroutine 47339 [semacquire]:
runtime.gopark(0x0?, 0x0?, 0x20?, 0x36?, 0x10?)
                runtime/proc.go:381 +0xd6 fp=0xc002ca1a30 sp=0xc002ca1a10 pc=0x43a236
runtime.goparkunlock(...)
                runtime/proc.go:387
runtime.semacquire1(0xc00af2a6c8, 0x88?, 0x1, 0x0, 0x0?)
                runtime/sema.go:160 +0x20f fp=0xc002ca1a98 sp=0xc002ca1a30 pc=0x44b5af
sync.runtime_Semacquire(0x46b1ce?)
                runtime/sema.go:62 +0x27 fp=0xc002ca1ad0 sp=0xc002ca1a98 pc=0x468d67
sync.(*WaitGroup).Wait(0xc002ca1b58?)
                sync/waitgroup.go:116 +0x4b fp=0xc002ca1af8 sp=0xc002ca1ad0 pc=0x48b66b
github.com/lightningnetwork/lnd/peer.(*Brontide).WaitForDisconnect(0xc00af2a380, 0xc00076ca20)
                github.com/lightningnetwork/lnd/peer/brontide.go:1102 +0xeb fp=0xc002ca1b88 sp=0xc002ca1af8 pc=0x19021ab
github.com/lightningnetwork/lnd.(*server).peerTerminationWatcher(0xc000004300, 0xc00af2a380, 0xc00e0b8360?)
                github.com/lightningnetwork/lnd/server.go:4017 +0x90 fp=0xc002ca1fb8 sp=0xc002ca1b88 pc=0x19cc250
github.com/lightningnetwork/lnd.(*server).peerInitializer.func2()
                github.com/lightningnetwork/lnd/server.go:3973 +0x2e fp=0xc002ca1fe0 sp=0xc002ca1fb8 pc=0x19cc12e
runtime.goexit()
                runtime/asm_amd64.s:1598 +0x1 fp=0xc002ca1fe8 sp=0xc002ca1fe0 pc=0x46d261
created by github.com/lightningnetwork/lnd.(*server).peerInitializer
                github.com/lightningnetwork/lnd/server.go:3973 +0x14a

We have a candidate fix here: https://github.com/lightningnetwork/lnd/issues/7928

We have a candidate fix here: https://github.com/lightningnetwork/lnd/issues/7928

Please merge 🙏

@niteshbalusu11 does that seem to do the trick?

I have not tested it yet.

Just tested it, seems to be working well.

I am experiencing this exact behavior and logging patterns in 0.16.4-beta
As we speak, I am at nearly an hour of waiting for LND to shut down, with a system that is at 70% IO wait and 8% CPU consumption.

Is anyone using 0.17.0-rc2 with this patch in production?

Is anyone using 0.17.0-rc2 with this patch in production?

I am, but I still think the bug exists

@Roasbeef Here's another goroutine dump. I think the issue still persists.
godump.zip

Are you able to repro on the current master? We fixed another related instance with #8041

Are you able to repro on the current master? We fixed another related instance with #8041

I updated to rc6, need to wait for sometime to see if its still happening.

I am on LND v0.17.0-beta and I same problem after lncli stop:
2023-10-04 19:26:36.369 [INF] HSWC: ChannelLink(7219175edcc8d63018d094e6c3e45c5edfb0bb889207ede2306514878cb2e968:1): exited
2023-10-04 19:26:36.369 [INF] HSWC: ChannelLink(45291b63fb8cff3803b74f3bf153c33cc781079352727103fdd94dcd5c68fc14:0): exited
2023-10-04 19:26:36.370 [INF] HSWC: ChannelLink(6f33a2407297e173a0c9d134fa23cd0bc2e500ecb05be8abedfaba9783e31fed:2): exited
2023-10-04 19:26:36.371 [INF] HSWC: ChannelLink(952e8ca933cdde292f358f9af4db3989a632efa92157ce23199a748cb266fc0d:5): exited
2023-10-04 19:26:36.371 [WRN] HSWC: ChannelLink(d6593789cb15d55188e7ad6036609c1be4c62d136f54dfc6e74a7754ad956783:0): error when syncing channel states: link shutting down
2023-10-04 19:26:36.373 [INF] HSWC: ChannelLink(326b7165f8a8bf4cdfc713c6b52e295642a9d28bd1424a2d1675ff9ac4590dad:5): exited
2023-10-04 19:26:36.374 [INF] HSWC: ChannelLink(dadc1d1d10d58fb00429a0b141c2652bc68c99766c130e2dd22ebd7c45266e97:1): exited
2023-10-04 19:26:36.374 [INF] HSWC: ChannelLink(bc8d8238b231b325e69a4c83a2ae50e2f1dce4d2a1a7a17e68bb17d149c74048:4): exited
2023-10-04 19:26:36.374 [INF] HSWC: ChannelLink(bff99bbfedef64aff601e485c90ac7f9505d9eadd79b0e1da6510ba51b85d4bf:0): exited
2023-10-04 19:26:36.375 [INF] HSWC: ChannelLink(d6593789cb15d55188e7ad6036609c1be4c62d136f54dfc6e74a7754ad956783:0): exited
2023-10-04 19:26:38.193 [INF] HSWC: Onion processor shutting down
2023-10-04 19:26:38.193 [INF] HSWC: Decaying hash log received shutdown request
2023-10-04 19:26:38.193 [INF] NTFN: Cancelling epoch notification, epoch_id=43
2023-10-04 19:26:38.194 [INF] INVC: InvoiceRegistry shutting down
2023-10-04 19:26:38.194 [INF] NTFN: Cancelling epoch notification, epoch_id=42
2023-10-04 19:26:38.194 [INF] CRTR: Channel Router shutting down
2023-10-04 19:26:38.194 [INF] CRTR: FilteredChainView stopping
2023-10-04 19:26:38.194 [INF] CNCT: ChainArbitrator shutting down
2023-10-04 19:26:38.194 [INF] NTFN: Cancelling epoch notification, epoch_id=40
...
2023-10-04 19:26:38.203 [WRN] FNDG: fundingManager shutting down, stopping funding flow for ChannelPoint(e0c6c3e1aa2615e92e4e1c3c6f7a94d5b28002820c57fee0958967fe9b14a99c:1)
2023-10-04 19:26:38.203 [ERR] FNDG: Unable to advance pending state of ChannelPoint(e0c6c3e1aa2615e92e4e1c3c6f7a94d5b28002820c57fee0958967fe9b14a99c:1): error waiting for funding confirmation for ChannelPoint(e0c6c3e1aa2615e92e4e1c3c6f7a94d5b28002820c57fee0958967fe9b14a99c:1): funding manager shutting down
2023-10-04 19:26:38.203 [WRN] FNDG: fundingManager shutting down, stopping funding flow for ChannelPoint(8964bb6a11497f6f60c061ec8f9591b0608ce68df01e536cef20f5fd9c8f8bad:1)
2023-10-04 19:26:38.203 [WRN] FNDG: fundingManager shutting down, stopping funding flow for ChannelPoint(b6f6a07daa2f5c409debdbbaf693c450e6d75be5d385ab39083476155006ba4d:0)
2023-10-04 19:26:38.203 [ERR] FNDG: Unable to advance pending state of ChannelPoint(422672827139067ff61811db084019d39249ad38a0426842eb8f38f77982b634:1): error waiting for funding confirmation for ChannelPoint(422672827139067ff61811db084019d39249ad38a0426842eb8f38f77982b634:1): funding manager shutting down
2023-10-04 19:26:38.203 [ERR] FNDG: Unable to advance pending state of ChannelPoint(a1e24f85b97d9b4abb29829f9ec4b7129cf3d3cebb5ec37b0b16bc1e3c21ef19:0): error waiting for funding confirmation for ChannelPoint(a1e24f85b97d9b4abb29829f9ec4b7129cf3d3cebb5ec37b0b16bc1e3c21ef19:0): funding manager shutting down
2023-10-04 19:26:38.203 [WRN] FNDG: fundingManager shutting down, stopping funding flow for ChannelPoint(327d82ecd4d22662c1bfc9afcf4fcf5907dea985a4d9b558ee445d781f66c1ba:1)
2023-10-04 19:26:38.203 [ERR] FNDG: Unable to advance pending state of ChannelPoint(babefd57730d52014e38f727d49243e9bae4a44a807e9b07fd080a8495c095c1:0): error waiting for funding confirmation for ChannelPoint(babefd57730d52014e38f727d49243e9bae4a44a807e9b07fd080a8495c095c1:0): funding manager shutting down
2023-10-04 19:26:38.203 [WRN] FNDG: fundingManager shutting down, stopping funding flow for ChannelPoint(babefd57730d52014e38f727d49243e9bae4a44a807e9b07fd080a8495c095c1:0)
2023-10-04 19:26:38.203 [ERR] FNDG: Unable to advance pending state of ChannelPoint(a3699b626b5cf53477a4a1a31d1279a1bc722706ffd2c3bc6c94537f3317ade9:1): error waiting for funding confirmation for ChannelPoint(a3699b626b5cf53477a4a1a31d1279a1bc722706ffd2c3bc6c94537f3317ade9:1): funding manager shutting down
2023-10-04 19:26:38.203 [ERR] FNDG: Unable to advance pending state of ChannelPoint(8964bb6a11497f6f60c061ec8f9591b0608ce68df01e536cef20f5fd9c8f8bad:1): error waiting for funding confirmation for ChannelPoint(8964bb6a11497f6f60c061ec8f9591b0608ce68df01e536cef20f5fd9c8f8bad:1): funding manager shutting down
2023-10-04 19:26:38.203 [ERR] FNDG: Unable to advance pending state of ChannelPoint(327d82ecd4d22662c1bfc9afcf4fcf5907dea985a4d9b558ee445d781f66c1ba:1): error waiting for funding confirmation for ChannelPoint(327d82ecd4d22662c1bfc9afcf4fcf5907dea985a4d9b558ee445d781f66c1ba:1): funding manager shutting down
2023-10-04 19:26:38.203 [WRN] FNDG: fundingManager shutting down, stopping funding flow for ChannelPoint(b4ab8679f5777353ef24b07de8fcca4fc01b0c65e6a7aba2f0848e032c3f5215:0)
2023-10-04 19:26:38.203 [ERR] FNDG: Unable to advance pending state of ChannelPoint(86a6066a510ab0321ac244e5c58bef283d96e7b8f1e244d03cf2169339371282:1): error waiting for funding confirmation for ChannelPoint(86a6066a510ab0321ac244e5c58bef283d96e7b8f1e244d03cf2169339371282:1): funding manager shutting down
2023-10-04 19:26:38.203 [WRN] FNDG: fundingManager shutting down, stopping funding flow for ChannelPoint(f9844eeae391decd8c667cc2698fc9f776085523df3b2bd914013115605061a5:0)
2023-10-04 19:26:39.951 [INF] DISC: GossipSyncer(031c0d8356994448e6859d7d767c792f8af4b86fe19d62e3431aba0488c0522931): filtering chan range: start_height=88002, num_blocks=293184
2023-10-04 19:26:39.952 [INF] DISC: GossipSyncer(031c0d8356994448e6859d7d767c792f8af4b86fe19d62e3431aba0488c0522931): sending final chan range chunk, size=0
2023-10-04 19:26:41.284 [INF] DISC: GossipSyncer(028fdddd8450b196f7948c11ffb55d78c0531197ecedcef69f24fa707c7671b1c0): filtering chan range: start_height=730811, num_blocks=1024
2023-10-04 19:26:41.285 [INF] DISC: GossipSyncer(028fdddd8450b196f7948c11ffb55d78c0531197ecedcef69f24fa707c7671b1c0): sending final chan range chunk, size=136
2023-10-04 19:26:42.136 [ERR] DISC: Update edge for short_chan_id(847391412616757249) got: router shutting down
2023-10-04 19:26:42.136 [ERR] DISC: Update edge for short_chan_id(825817894890110978) got: router shutting down
2023-10-04 19:26:42.614 [ERR] DISC: Update edge for short_chan_id(837667331771858944) got: router shutting down
2023-10-04 19:26:43.086 [ERR] DISC: Update edge for short_chan_id(888167900823420929) got: router shutting down
2023-10-04 19:26:46.456 [ERR] DISC: Update edge for short_chan_id(890958461360144384) got: router shutting down
2023-10-04 19:26:47.137 [ERR] DISC: Update edge for short_chan_id(805028329151070208) got: router shutting down
...
2023-10-04 19:29:10.610 [ERR] DISC: Update edge for short_chan_id(843911458225848321) got: router shutting down
2023-10-04 19:29:10.610 [ERR] DISC: Update edge for short_chan_id(847412303278899200) got: router shutting down
2023-10-04 19:29:10.699 [ERR] DISC: Update edge for short_chan_id(853691614143971329) got: router shutting down
2023-10-04 19:29:10.778 [ERR] DISC: Update edge for short_chan_id(855785084338307073) got: router shutting down
2023-10-04 19:29:10.778 [ERR] DISC: Update edge for short_chan_id(865424502887088128) got: router shutting down
2023-10-04 19:29:10.779 [ERR] DISC: Update edge for short_chan_id(869552069430214657) got: router shutting down
2023-10-04 19:29:10.779 [ERR] DISC: Update edge for short_chan_id(872554835805995009) got: router shutting down
2023-10-04 19:29:10.780 [ERR] DISC: Update edge for short_chan_id(872438287593177089) got: router shutting down
2023-10-04 19:29:10.843 [ERR] DISC: Update edge for short_chan_id(874825327292448769) got: router shutting down
2023-10-04 19:29:10.843 [ERR] DISC: Update edge for short_chan_id(885184925726015489) got: router shutting down
2023-10-04 19:29:10.843 [ERR] DISC: Update edge for short_chan_id(801764978598281216) got: router shutting down
2023-10-04 19:29:10.844 [ERR] DISC: Update edge for short_chan_id(877040843189452801) got: router shutting down
2023-10-04 19:29:10.844 [ERR] DISC: Update edge for short_chan_id(888545033256828929) got: router shutting down
2023-10-04 19:29:10.844 [ERR] DISC: Update edge for short_chan_id(890791335537672193) got: router shutting down
2023-10-04 19:29:12.649 [INF] PEER: Peer(0324ba2392e25bff76abd0b1f7e4b53b5f82aa53fddc3419b051b6c801db9e2247): unable to read message from peer: read next header: EOF
2023-10-04 19:29:12.649 [INF] PEER: Peer(0324ba2392e25bff76abd0b1f7e4b53b5f82aa53fddc3419b051b6c801db9e2247): disconnecting 0324ba2392e25bff76abd0b1f7e4b53b5f82aa53fddc3419b051b6c801db9e2247@<IP_ADDRESS>:20667, reason: read handler closed
2023-10-04 19:29:12.649 [INF] NTFN: Cancelling epoch notification, epoch_id=320
2023-10-04 19:29:12.857 [ERR] DISC: Update edge for short_chan_id(874060067071393793) got: router shutting down
2023-10-04 19:29:12.857 [ERR] DISC: Update edge for short_chan_id(764533315779756033) got: router shutting down
2023-10-04 19:29:12.858 [ERR] DISC: Update edge for short_chan_id(868625181244063749) got: router shutting down
2023-10-04 19:29:12.858 [ERR] DISC: Update edge for short_chan_id(850021444418732033) got: router shutting down
2023-10-04 19:29:12.858 [ERR] DISC: Update edge for short_chan_id(874715376068984832) got: router shutting down
2023-10-04 19:29:12.877 [ERR] DISC: Update edge for short_chan_id(878850639249997825) got: router shutting down
2023-10-04 19:29:12.877 [ERR] DISC: Update edge for short_chan_id(814069613186449409) got: router shutting down
2023-10-04 19:29:12.877 [ERR] DISC: Update edge for short_chan_id(845852096344555521) got: router shutting down
2023-10-04 19:29:12.878 [ERR] DISC: Update edge for short_chan_id(819448424021164032) got: router shutting down
2023-10-04 19:29:12.878 [ERR] DISC: Update edge for short_chan_id(825349502997561344) got: router shutting down
2023-10-04 19:29:12.878 [ERR] DISC: Update edge for short_chan_id(789998005232336897) got: router shutting down
2023-10-04 19:29:12.878 [ERR] DISC: Update edge for short_chan_id(840615122495602688) got: router shutting down
2023-10-04 19:29:13.494 [ERR] DISC: Update edge for short_chan_id(811133917205168129) got: router shutting down
2023-10-04 19:29:13.579 [ERR] DISC: Update edge for short_chan_id(842686602342498304) got: router shutting down
2023-10-04 19:29:14.710 [INF] PEER: Peer(0390b5d4492dc2f5318e5233ab2cebf6d48914881a33ef6a9c6bcdbb433ad986d0): unable to read message from peer: read next header: EOF
2023-10-04 19:29:14.710 [INF] PEER: Peer(0390b5d4492dc2f5318e5233ab2cebf6d48914881a33ef6a9c6bcdbb433ad986d0): disconnecting 0390b5d4492dc2f5318e5233ab2cebf6d48914881a33ef6a9c6bcdbb433ad986d0@<IP_ADDRESS>:9735, reason: read handler closed
2023-10-04 19:29:14.710 [INF] NTFN: Cancelling epoch notification, epoch_id=48
2023-10-04 19:29:14.734 [ERR] DISC: Update edge for short_chan_id(878258002747654144) got: router shutting down
2023-10-04 19:29:14.734 [ERR] DISC: Update edge for short_chan_id(887477407519735809) got: router shutting down
2023-10-04 19:29:14.734 [ERR] DISC: Update edge for short_chan_id(827887175824441344) got: router shutting down
2023-10-04 19:29:14.734 [ERR] DISC: Update edge for short_chan_id(878239310831419393) got: router shutting down
2023-10-04 19:29:14.734 [ERR] DISC: Update edge for short_chan_id(885650019272753153) got: router shutting down
2023-10-04 19:29:14.734 [ERR] DISC: Update edge for short_chan_id(885996365332086784) got: router shutting down
2023-10-04 19:29:16.183 [ERR] DISC: Update edge for short_chan_id(805520910380040192) got: router shutting down
2023-10-04 19:29:17.859 [ERR] DISC: Update edge for short_chan_id(864147969807745025) got: router shutting down
2023-10-04 19:29:17.860 [ERR] DISC: Update edge for short_chan_id(877070529918664704) got: router shutting down
2023-10-04 19:29:17.860 [ERR] DISC: Update edge for short_chan_id(831722272314490881) got: router shutting down
2023-10-04 19:29:17.879 [ERR] DISC: Update edge for short_chan_id(886608793449398273) got: router shutting down
2023-10-04 19:29:17.879 [ERR] DISC: Update edge for short_chan_id(879194786391719936) got: router shutting down
2023-10-04 19:29:17.879 [ERR] DISC: Update edge for short_chan_id(838843809186643969) got: router shutting down
2023-10-04 19:29:17.879 [ERR] DISC: Update edge for short_chan_id(888071143737786368) got: router shutting down
2023-10-04 19:29:17.879 [ERR] DISC: Update edge for short_chan_id(821746403319152641) got: router shutting down
2023-10-04 19:29:17.880 [ERR] DISC: Update edge for short_chan_id(889942512594124800) got: router shutting down
2023-10-04 19:29:18.931 [ERR] DISC: Update edge for short_chan_id(882430649155649537) got: router shutting down
2023-10-04 19:29:19.742 [ERR] DISC: Update edge for short_chan_id(891147577337577472) got: router shutting down
2023-10-04 19:29:19.742 [ERR] DISC: Update edge for short_chan_id(888591212859424771) got: router shutting down
2023-10-04 19:29:19.742 [ERR] DISC: Update edge for short_chan_id(878807758431977473) got: router shutting down
2023-10-04 19:29:19.742 [ERR] DISC: Update edge for short_chan_id(888048054068576257) got: router shutting down
2023-10-04 19:29:19.743 [ERR] DISC: Update edge for short_chan_id(890531850769530881) got: router shutting down
2023-10-04 19:29:19.743 [ERR] DISC: Update edge for short_chan_id(870296438837673984) got: router shutting down
2023-10-04 19:29:19.743 [ERR] DISC: Update edge for short_chan_id(890908983314546688) got: router shutting down
...
2023-10-04 19:30:48.353 [ERR] DISC: Update edge for short_chan_id(885831438618722305) got: router shutting down
2023-10-04 19:30:48.353 [ERR] DISC: Update edge for short_chan_id(881577428139704321) got: router shutting down
2023-10-04 19:30:49.763 [ERR] DISC: Update edge for short_chan_id(890516457590751233) got: router shutting down
2023-10-04 19:30:49.763 [ERR] DISC: Update edge for short_chan_id(890648399108636672) got: router shutting down
2023-10-04 19:30:49.763 [ERR] DISC: router shutting down
2023-10-04 19:30:49.764 [ERR] DISC: router shutting down
2023-10-04 19:30:49.764 [ERR] DISC: router shutting down
2023-10-04 19:30:49.764 [ERR] DISC: router shutting down
2023-10-04 19:30:49.764 [ERR] DISC: router shutting down
2023-10-04 19:30:49.764 [ERR] DISC: router shutting down
2023-10-04 19:30:52.107 [ERR] DISC: Update edge for short_chan_id(856051166132633600) got: router shutting down
2023-10-04 19:30:52.222 [ERR] DISC: Update edge for short_chan_id(849041779581517825) got: router shutting down
2023-10-04 19:30:52.862 [ERR] DISC: Update edge for short_chan_id(842389734186483713) got: router shutting down
2023-10-04 19:30:52.863 [ERR] DISC: Update edge for short_chan_id(856342536794537984) got: router shutting down
2023-10-04 19:30:52.863 [ERR] DISC: Update edge for short_chan_id(799701195284021248) got: router shutting down
2023-10-04 19:30:52.863 [ERR] DISC: Update edge for short_chan_id(863190295111270400) got: router shutting down
2023-10-04 19:30:52.882 [ERR] DISC: Update edge for short_chan_id(862646036845953024) got: router shutting down

This bug is so hard to reproduce. It just seems to be happening once in a while.

Reopening based on the recent comments.

@LNBIG-COM if that happens again, can you get a goroutine profile? Then we should be able to see what's holding stuff up.

I was advised to submit a bug report on this issue. I am experiencing the same issues as niteshbalasu11 and dskvr.
Huge I/O consumption and minimal CPU being used. I have a feeling it's trigerred by an outgoing HTLC.
I was experiencing issues so I figured an upgrade would solve them. I THINK they were similar issues (random shut downs) I didn't pay much attention because it had been so long since I upgraded.
I upgraded from an old version of LND (can't remember which version but the old binaries show they were last modified Dec 31st).
Sharing some intermittent logs that look very similar to some logs mentioned earlier. Please let me know how to further troubleshoot.
2023-10-14 19:18:57.504 [DBG] CRTR: Received ChannelEdgePolicy for channel<PHONE_NUMBER>59432192
2023-10-14 19:18:57.728 [INF] CRTR: Processed channels=0 updates=92 nodes=3 in last 59.999901577s
2023-10-14 19:18:58.697 [DBG] CRTR: Received ChannelEdgePolicy for channel<PHONE_NUMBER>84734210
2023-10-14 19:19:03.761 [DBG] CRTR: Received ChannelEdgePolicy for channel<PHONE_NUMBER>00390401
2023-10-14 19:19:03.767 [DBG] CRTR: Received ChannelEdgePolicy for channel<PHONE_NUMBER>32904705
2023-10-14 19:19:03.827 [DBG] CRTR: Received ChannelEdgePolicy for channel<PHONE_NUMBER>02823170
2023-10-14 19:19:03.828 [DBG] CRTR: Received ChannelEdgePolicy for channel<PHONE_NUMBER>44290049
2023-10-14 19:19:03.878 [DBG] CRTR: Received ChannelEdgePolicy for channel<PHONE_NUMBER>60371456
2023-10-14 19:19:03.879 [DBG] CRTR: Received ChannelEdgePolicy for channel<PHONE_NUMBER>82612737
2023-10-14 19:19:03.898 [DBG] CRTR: Received ChannelEdgePolicy for channel<PHONE_NUMBER>60884992
2023-10-14 19:19:03.899 [DBG] CRTR: Received ChannelEdgePolicy for channel<PHONE_NUMBER>43451649
2023-10-14 19:19:03.904 [DBG] CRTR: Received ChannelEdgePolicy for channel<PHONE_NUMBER>52124419
2023-10-14 19:19:03.960 [DBG] CRTR: Received ChannelEdgePolicy for channel<PHONE_NUMBER>42102273
2023-10-14 19:19:05.958 [DBG] CRTR: Received ChannelEdgePolicy for channel<PHONE_NUMBER>62049281
2023-10-14 19:19:05.959 [DBG] CRTR: Received ChannelEdgePolicy for channel<PHONE_NUMBER>42366464
2023-10-14 19:19:05.959 [DBG] CRTR: Received ChannelEdgePolicy for channel<PHONE_NUMBER>42263296
2023-10-14 19:19:05.959 [DBG] CRTR: Received ChannelEdgePolicy for channel<PHONE_NUMBER>48297984
2023-10-14 19:19:05.959 [DBG] CRTR: Received ChannelEdgePolicy for channel<PHONE_NUMBER>03981824
2023-10-14 19:19:05.959 [DBG] CRTR: Received ChannelEdgePolicy for channel<PHONE_NUMBER>22562817
2023-10-14 19:19:06.250 [DBG] CRTR: Received ChannelEdgePolicy for channel<PHONE_NUMBER>04817409
2023-10-14 19:19:06.251 [DBG] CRTR: Received ChannelEdgePolicy for channel<PHONE_NUMBER>18087169
2023-10-14 19:19:37.364 [DBG] CRTR: Received ChannelEdgePolicy for channel<PHONE_NUMBER>47518212
2023-10-14 19:19:37.393 [DBG] CRTR: Received ChannelEdgePolicy for channel<PHONE_NUMBER>41176833
2023-10-14 19:19:37.468 [DBG] CRTR: Received ChannelEdgePolicy for channel<PHONE_NUMBER>54954497
2023-10-14 19:19:37.505 [DBG] CRTR: Received ChannelEdgePolicy for channel<PHONE_NUMBER>46223361
2023-10-14 19:19:39.519 [ERR] RPCS: [/routerrpc.Router/SendPaymentV2]: context canceled
2023-10-14 19:19:43.095 [DBG] CRTR: Received ChannelEdgePolicy for channel<PHONE_NUMBER>97721369
2023-10-14 19:19:43.108 [DBG] CRTR: Received ChannelEdgePolicy for channel<PHONE_NUMBER>39250690
2023-10-14 19:19:43.108 [DBG] CRTR: Received ChannelEdgePolicy for channel<PHONE_NUMBER>91692032
2023-10-14 19:19:43.127 [DBG] CRTR: Received ChannelEdgePolicy for channel<PHONE_NUMBER>86194688
2023-10-14 19:19:43.152 [DBG] CRTR: Received ChannelEdgePolicy for channel<PHONE_NUMBER>66405377
2023-10-14 19:19:43.154 [DBG] CRTR: Received ChannelEdgePolicy for channel<PHONE_NUMBER>39250697
2023-10-14 19:19:47.408 [DBG] CRTR: Received ChannelEdgePolicy for channel<PHONE_NUMBER>14498331
2023-10-14 19:19:47.408 [DBG] CRTR: Received ChannelEdgePolicy for channel<PHONE_NUMBER>04466433
2023-10-14 19:19:47.408 [DBG] CRTR: Received ChannelEdgePolicy for channel<PHONE_NUMBER>14498331
2023-10-14 19:19:47.408 [DBG] CRTR: Received ChannelEdgePolicy for channel<PHONE_NUMBER>04466433
2023-10-14 19:19:47.446 [DBG] CRTR: Received ChannelEdgePolicy for channel<PHONE_NUMBER>64435201
2023-10-14 19:19:47.459 [DBG] CRTR: Received ChannelEdgePolicy for channel<PHONE_NUMBER>19743489
2023-10-14 19:19:47.470 [DBG] CRTR: Received ChannelEdgePolicy for channel<PHONE_NUMBER>78324482
2023-10-14 19:19:50.485 [INF] LTND: Received interrupt
2023-10-14 19:19:50.500 [INF] LTND: Shutting down...
2023-10-14 19:19:50.505 [INF] LTND: Gracefully shutting down.
2023-10-14 19:19:50.553 [INF] NANN: Channel Status Manager shutting down
2023-10-14 19:19:50.553 [INF] HSWC: HTLC Switch shutting down
2023-10-14 19:19:50.569 [INF] NTFN: Cancelling epoch notification, epoch_id=4
2023-10-14 19:19:50.570 [INF] HSWC: Removing channel link with ChannelID(b2d323fcfdee29f692ec769f59975cd895777f66a078f2cc1494a60c698df493)
2023-10-14 19:19:50.571 [ERR] CRTR: Payment 9db00b6354e5410f5db7d8f676496bbade3b7750dd907c531d7adeee12d69fde failed: htlcswitch shutting down
2023-10-14 19:19:50.580 [INF] HSWC: Removing channel link with ChannelID(da4dacd3e4897c06d503cd2b4fa381a819cbf817b97d0c935e824d331e66c2d3)
2023-10-14 19:19:50.580 [INF] HSWC: Removing channel link with ChannelID(9a57bc547b3841d59fc187d559ff0309d813b348c4ef2840f13e129b27edbf87)
2023-10-14 19:19:50.580 [INF] HSWC: Removing channel link with ChannelID(c190af25515c23b32e8821b74c29f998829fd78b5a509fe10916e1855ff30c44)
2023-10-14 19:19:50.580 [INF] HSWC: ChannelLink(450cf35f85e11609e19f505a8bd79f8298f9294cb721882eb3235c5125af90c1:1): stopping
2023-10-14 19:19:50.580 [INF] HSWC: ChannelLink(86bfed279b123ef14028efc448b313d80903ff59d587c19fd541387b54bc579a:1): stopping
2023-10-14 19:19:50.580 [INF] HSWC: ChannelLink(d2c2661e334d825e930c7db917f8cb19a881a34f2bcd03d5067c89e4d3ac4dda:1): stopping
2023-10-14 19:19:50.580 [INF] HSWC: ChannelLink(92f48d690ca69414ccf278a0667f7795d85c97599f76ec92f629eefdfc23d3b2:1): stopping
2023-10-14 19:19:50.581 [INF] HSWC: ChannelLink(d2c2661e334d825e930c7db917f8cb19a881a34f2bcd03d5067c89e4d3ac4dda:1): exited
2023-10-14 19:19:50.582 [INF] HSWC: ChannelLink(86bfed279b123ef14028efc448b313d80903ff59d587c19fd541387b54bc579a:1): exited
2023-10-14 19:19:50.582 [INF] HSWC: ChannelLink(450cf35f85e11609e19f505a8bd79f8298f9294cb721882eb3235c5125af90c1:1): exited
2023-10-14 19:19:50.582 [INF] HSWC: ChannelLink(92f48d690ca69414ccf278a0667f7795d85c97599f76ec92f629eefdfc23d3b2:1): exited
2023-10-14 19:19:50.591 [INF] HSWC: Onion processor shutting down
2023-10-14 19:19:50.591 [INF] HSWC: Decaying hash log received shutdown request
2023-10-14 19:19:50.591 [INF] NTFN: Cancelling epoch notification, epoch_id=9
2023-10-14 19:19:50.591 [INF] INVC: InvoiceRegistry shutting down
2023-10-14 19:19:50.591 [INF] NTFN: Cancelling epoch notification, epoch_id=8
2023-10-14 19:19:50.592 [INF] INVC: Cancelling invoice subscription for client=1
2023-10-14 19:19:50.601 [INF] CRTR: Channel Router shutting down
2023-10-14 19:19:50.601 [INF] CRTR: FilteredChainView stopping
2023-10-14 19:19:50.601 [INF] CNCT: ChainArbitrator shutting down
2023-10-14 19:19:50.601 [INF] NTFN: Cancelling epoch notification, epoch_id=6
2023-10-14 19:19:50.624 [INF] FNDG: Funding manager shutting down
2023-10-14 19:19:50.630 [INF] BRAR: Breach arbiter shutting down
2023-10-14 19:19:50.631 [INF] UTXN: UTXO nursery shutting down
2023-10-14 19:19:50.631 [INF] NTFN: Cancelling epoch notification, epoch_id=3
2023-10-14 19:19:50.631 [INF] DISC: Authenticated gossiper shutting down
2023-10-14 19:19:50.631 [INF] DISC: Authenticated Gossiper is stopping
2023-10-14 19:19:50.631 [INF] NTFN: Cancelling epoch notification, epoch_id=7
2023-10-14 19:19:50.638 [INF] DISC: Authenticated Gossiper stopped
2023-10-14 19:19:50.639 [INF] SWPR: Sweeper shutting down
2023-10-14 19:19:50.639 [INF] NTFN: Cancelling epoch notification, epoch_id=2
2023-10-14 19:19:50.639 [INF] CHNF: ChannelNotifier shutting down
2023-10-14 19:19:50.639 [INF] PRNF: PeerNotifier shutting down
2023-10-14 19:19:50.639 [INF] HSWC: HtlcNotifier shutting down
2023-10-14 19:19:50.639 [ERR] RPCS: [/routerrpc.Router/SubscribeHtlcEvents]: htlc event subscription terminated
2023-10-14 19:19:50.639 [INF] CHBU: Stopping chanbackup.SubSwapper
2023-10-14 19:19:50.640 [INF] NTFN: bitcoind notifier shutting down
2023-10-14 19:19:50.689 [INF] NTFN: Stopping mempool notifier
2023-10-14 19:19:50.689 [INF] CHFT: Stopping event store
2023-10-14 19:19:50.690 [ERR] HSWC: InterceptableSwitch stopped: block epoch stream stopped
2023-10-14 19:19:51.138 [INF] SRVR: Disconnecting from 027833211bdc28059b323492f2e51453559a2fcbaf947c16f716d060e560128eb2@7fuuhlo2jxvoqixai43cu67iyvlndrzdajicmupwhhoccvz3h7z4thid.onion:9735
2023-10-14 19:19:51.138 [INF] PEER: Peer(027833211bdc28059b323492f2e51453559a2fcbaf947c16f716d060e560128eb2): disconnecting 027833211bdc28059b323492f2e51453559a2fcbaf947c16f716d060e560128eb2@7fuuhlo2jxvoqixai43cu67iyvlndrzdajicmupwhhoccvz3h7z4thid.onion:9735, reason: server: DisconnectPeer called
2023-10-14 19:19:51.139 [INF] PEER: Peer(027833211bdc28059b323492f2e51453559a2fcbaf947c16f716d060e560128eb2): unable to read message from peer: read next header: read tcp <IP_ADDRESS>:54530-><IP_ADDRESS>:9050: use of closed network connection
2023-10-14 19:19:51.139 [INF] SRVR: Disconnecting from 02658b39acdaa98e1847f612e42c38302ce24e99ac8ed69ac63e52730ff26d27b8@f4xa4sqjvjfdky5dipzpfevcugjgogpzok4nokbc7ohgr5zxofwslxqd.onion:9735
2023-10-14 19:19:51.139 [INF] PEER: Peer(02658b39acdaa98e1847f612e42c38302ce24e99ac8ed69ac63e52730ff26d27b8): disconnecting 02658b39acdaa98e1847f612e42c38302ce24e99ac8ed69ac63e52730ff26d27b8@f4xa4sqjvjfdky5dipzpfevcugjgogpzok4nokbc7ohgr5zxofwslxqd.onion:9735, reason: server: DisconnectPeer called
2023-10-14 19:19:51.140 [INF] PEER: Peer(02658b39acdaa98e1847f612e42c38302ce24e99ac8ed69ac63e52730ff26d27b8): unable to read message from peer: read next header: read tcp <IP_ADDRESS>:54532-><IP_ADDRESS>:9050: use of closed network connection
2023-10-14 19:19:51.143 [INF] HLCK: Health monitor shutting down
2023-10-14 19:19:51.248 [INF] RPCS: Stopping RPC Server
2023-10-14 19:19:51.249 [INF] RPCS: Stopping ChainRPC Sub-RPC Server
2023-10-14 19:19:51.281 [INF] RPCS: Stopping SignRPC Sub-RPC Server
2023-10-14 19:19:51.281 [INF] RPCS: Stopping AutopilotRPC Sub-RPC Server
2023-10-14 19:19:51.282 [INF] RPCS: Stopping InvoicesRPC Sub-RPC Server
2023-10-14 19:19:51.282 [INF] RPCS: Stopping RouterRPC Sub-RPC Server
2023-10-14 19:19:51.283 [INF] RPCS: Stopping WatchtowerClientRPC Sub-RPC Server
2023-10-14 19:19:51.283 [INF] RPCS: Stopping NeutrinoKitRPC Sub-RPC Server
2023-10-14 19:19:51.283 [INF] RPCS: Stopping VersionRPC Sub-RPC Server
2023-10-14 19:19:51.283 [INF] RPCS: Stopping WalletKitRPC Sub-RPC Server
2023-10-14 19:19:51.283 [INF] RPCS: Stopping WatchtowerRPC Sub-RPC Server
2023-10-14 19:19:51.319 [INF] RPCS: Stopping PeersRPC Sub-RPC Server
2023-10-14 19:19:51.356 [INF] TORC: Stopping tor controller
2023-10-14 19:19:51.783 [INF] LTND: Shutdown complete
2023-10-14 23:45:25.245 [INF] CRTR: Processed channels=0 updates=37 nodes=3 in last 59.999830481s
2023-10-14 23:45:29.125 [DBG] CRTR: Received ChannelEdgePolicy for channel<PHONE_NUMBER>72009984
2023-10-14 23:45:29.202 [DBG] CRTR: Received ChannelEdgePolicy for channel<PHONE_NUMBER>00777729
2023-10-14 23:45:29.349 [DBG] CRTR: Received ChannelEdgePolicy for channel<PHONE_NUMBER>83131397
2023-10-14 23:45:29.377 [DBG] CRTR: Received ChannelEdgePolicy for channel<PHONE_NUMBER>29586432
2023-10-14 23:45:29.425 [DBG] CRTR: Received ChannelEdgePolicy for channel<PHONE_NUMBER>88137728
2023-10-14 23:45:29.430 [DBG] CRTR: Received ChannelEdgePolicy for channel<PHONE_NUMBER>48122625
2023-10-14 23:45:32.623 [DBG] CRTR: Received ChannelEdgePolicy for channel<PHONE_NUMBER>57899008
2023-10-14 23:45:32.709 [DBG] CRTR: Received ChannelEdgePolicy for channel<PHONE_NUMBER>64965889
2023-10-14 23:45:37.343 [DBG] CRTR: Received ChannelEdgePolicy for channel<PHONE_NUMBER>01722881
2023-10-14 23:45:37.630 [DBG] CRTR: Received ChannelEdgePolicy for channel<PHONE_NUMBER>61497601
2023-10-14 23:45:37.690 [DBG] CRTR: Received ChannelEdgePolicy for channel<PHONE_NUMBER>10817025
2023-10-14 23:45:37.897 [DBG] CRTR: Received ChannelEdgePolicy for channel<PHONE_NUMBER>05662721
2023-10-14 23:45:37.975 [DBG] CRTR: Received ChannelEdgePolicy for channel<PHONE_NUMBER>87230209
2023-10-14 23:45:42.436 [DBG] CRTR: Received ChannelEdgePolicy for channel<PHONE_NUMBER>59138817
2023-10-14 23:45:42.648 [DBG] CRTR: Received ChannelEdgePolicy for channel<PHONE_NUMBER>83648769
2023-10-14 23:45:42.662 [DBG] CRTR: Received ChannelEdgePolicy for channel<PHONE_NUMBER>01457409
2023-10-14 23:45:42.701 [DBG] CRTR: Received ChannelEdgePolicy for channel<PHONE_NUMBER>55728906
2023-10-14 23:45:47.416 [DBG] CRTR: Received ChannelEdgePolicy for channel<PHONE_NUMBER>98966272
2023-10-14 23:45:47.418 [DBG] CRTR: Received ChannelEdgePolicy for channel<PHONE_NUMBER>35218179
2023-10-14 23:45:47.465 [DBG] CRTR: Received ChannelEdgePolicy for channel<PHONE_NUMBER>45595649
2023-10-14 23:45:47.474 [DBG] CRTR: Received ChannelEdgePolicy for channel<PHONE_NUMBER>03656449
2023-10-14 23:45:47.503 [DBG] CRTR: Received ChannelEdgePolicy for channel<PHONE_NUMBER>90996992
2023-10-14 23:45:47.565 [DBG] CRTR: Received ChannelEdgePolicy for channel<PHONE_NUMBER>27136001
2023-10-14 23:45:50.430 [CRT] SRVR: Health check: chain backend failed after 3 calls
2023-10-14 23:45:50.431 [INF] SRVR: Sending request for shutdown
2023-10-14 23:45:50.437 [INF] LTND: Received shutdown request.
2023-10-14 23:45:50.437 [INF] LTND: Shutting down...
2023-10-14 23:45:50.507 [INF] LTND: Gracefully shutting down.
2023-10-14 23:45:50.569 [INF] NANN: Channel Status Manager shutting down
2023-10-14 23:45:50.625 [INF] HSWC: HTLC Switch shutting down
2023-10-14 23:45:50.625 [INF] NTFN: Cancelling epoch notification, epoch_id=4
2023-10-14 23:45:50.693 [INF] HSWC: Removing channel link with ChannelID(c190af25515c23b32e8821b74c29f998829fd78b5a509fe10916e1855ff30c44)
2023-10-14 23:45:50.715 [INF] HSWC: Removing channel link with ChannelID(9a57bc547b3841d59fc187d559ff0309d813b348c4ef2840f13e129b27edbf87)
2023-10-14 23:45:50.715 [INF] HSWC: Removing channel link with ChannelID(da4dacd3e4897c06d503cd2b4fa381a819cbf817b97d0c935e824d331e66c2d3)
2023-10-14 23:45:50.750 [INF] HSWC: Removing channel link with ChannelID(b2d323fcfdee29f692ec769f59975cd895777f66a078f2cc1494a60c698df493)
2023-10-14 23:45:50.818 [INF] HSWC: ChannelLink(92f48d690ca69414ccf278a0667f7795d85c97599f76ec92f629eefdfc23d3b2:1): stopping

@mycelia1 are you able to obtain a goroutine dump when this happens?

@mycelia1 if you run into that state again, you can run CTRL-\, that'll dump all the goroutines, then you can put that in a paste bin and send it to us here.

Still waiting on a lnd to randomly crash, but can provide the goroutines logs for my frozen payment issue (may be related)
Steps I took:
Paid an invoice to another locally-hosted node I control.
Payment stays 'In Flight' indefinitely.
I cancel the payment manually (debug log shows [ERR] RPCS: [/routerrpc.Router/SendPaymentV2]: context canceled)
I manually kill lnd and restart.
Payment settles once lnd has restarted.
Debug logs go back to a constant state of "Received ChannelEdgePolicy, Checking stale node,  Waiting for dependent on job=lnwire.NodeAnnouncement)
goroutine profile total 451.txt

Here are my goroutine logs captured during the repeated "Received ChannelEdgePolicy" sequence.
goroutine profile total 429.txt

Captured goroutine logs right when everything was freezing up. Rather than LND crashing, my btcpay server instance (including NBXplorer) disconnected and threw connection errors instead (when LND crashes, it's because of communication errors with the backend so I'm assuming it's related).
goroutine profile total 928.txt
Hope this file is more helpful in pinpointing the bug and won't share any more of these logs unless requested.

@mycelia1 thanks for the goroutine profiles. This looks very similar to something we've already fixed. Can you please confirm what exact version of lnd you are running?

@guggero
"version":  "0.17.0-beta commit=v0.17.0-beta"

Hmm, okay, thanks.
So I see this goroutine waiting for a wait group
7 @ 0x43df8e 0x44f438 0x44f40f 0x46cf25 0x48f1a8 0x181c312 0x18e0545 0x471161
#       0x46cf24        sync.runtime_Semacquire+0x24                                            runtime/sema.go:62
#       0x48f1a7        sync.(*WaitGroup).Wait+0x47                                             sync/waitgroup.go:116
#       0x181c311       github.com/lightningnetwork/lnd/peer.(*Brontide).WaitForDisconnect+0xd1 github.com/lightningnetwork/lnd/peer/brontide.go:1102
#       0x18e0544       github.com/lightningnetwork/lnd.(*server).peerTerminationWatcher+0x84   github.com/lightningnetwork/lnd/server.go:4017

Looks like the readHeader goroutine didn't shut down though:
7 @ 0x43df8e 0x436677 0x46b625 0x4deae7 0x4dfdda 0x4dfdc8 0x5526e5 0x564285 0x4b81d0 0x16fcd88 0x16fcd62 0x16f7a25 0x181c690 0x181e18d 0x471161
#       0x46b624        internal/poll.runtime_pollWait+0x84                                     runtime/netpoll.go:343
#       0x4deae6        internal/poll.(*pollDesc).wait+0x26                                     internal/poll/fd_poll_runtime.go:84
#       0x4dfdd9        internal/poll.(*pollDesc).waitRead+0x279                                internal/poll/fd_poll_runtime.go:89
#       0x4dfdc7        internal/poll.(*FD).Read+0x267                                          internal/poll/fd_unix.go:164
#       0x5526e4        net.(*netFD).Read+0x24                                                  net/fd_posix.go:55
#       0x564284        net.(*conn).Read+0x44                                                   net/net.go:179
#       0x4b81cf        io.ReadAtLeast+0x8f                                                     io/io.go:335
#       0x16fcd87       io.ReadFull+0x47                                                        io/io.go:354
#       0x16fcd61       github.com/lightningnetwork/lnd/brontide.(*Machine).ReadHeader+0x21     github.com/lightningnetwork/lnd/brontide/noise.go:852
#       0x16f7a24       github.com/lightningnetwork/lnd/brontide.(*Conn).ReadNextHeader+0x44    github.com/lightningnetwork/lnd/brontide/conn.go:127
#       0x181c68f       github.com/lightningnetwork/lnd/peer.(*Brontide).readNextMessage+0xaf   github.com/lightningnetwork/lnd/peer/brontide.go:1146
#       0x181e18c       github.com/lightningnetwork/lnd/peer.(*Brontide).readHandler+0x2ec      github.com/lightningnetwork/lnd/peer/brontide.go:1519

It's stuck at noiseConn.ReadNextHeader():
https://github.com/lightningnetwork/lnd/blob/eae9dd07f0579cac94d71e8d8b4c7e3408ce85d0/peer/brontide.go#L1146
Which looks like it doesn't have a timeout set at all? But could be misinterpreting things...

Going off of this:

logs.zip

The ChannelRouter is waiting for its goroutines to stop. There are two calls to UpdateEdgePolicy which are hanging, they are waiting for the cacheMu lock (this is obfuscated somewhat - the cacheMu mutex is the locker in the chanScheduler). All of the calls to HasChannelEdge and MarkEdgeLive are also waiting for the cacheMu lock. There is one call to MarkEdgeLive that is not waiting for the cacheMu lock:
goroutine 48194 [runnable]:\
runtime.asyncPreempt2()\
        runtime/preempt.go:307 +0x3f fp=0xc002a5cd38 sp=0xc002a5cd18 pc=0x43893f\
runtime.asyncPreempt()\
        runtime/preempt_amd64.s:53 +0xdb fp=0xc002a5cec0 sp=0xc002a5cd38 pc=0x46e8db\
go.etcd.io/bbolt.(*pgids).Less(0x9, 0x459b2, 0x45862)\
        <autogenerated>:1 +0x9 fp=0xc002a5cee0 sp=0xc002a5cec0 pc=0xa49849\
sort.partition(\{0x22dfca0, 0xc0069ceb70\}, 0x455c2, 0x45b05, 0xc0069ceb70?)\
        sort/zsortinterface.go:154 +0x182 fp=0xc002a5cf18 sp=0xc002a5cee0 pc=0x478102\
sort.pdqsort(\{0x22dfca0, 0xc0069ceb70\}, 0x449b1?, 0x4ab9c?, 0xc0069ceb70?)\
        sort/zsortinterface.go:114 +0x225 fp=0xc002a5cf98 sp=0xc002a5cf18 pc=0x477e65\
sort.pdqsort(\{0x22dfca0, 0xc0069ceb70\}, 0xf57c?, 0x4f890?, 0x40dc1d?)\
        sort/zsortinterface.go:121 +0x276 fp=0xc002a5d018 sp=0xc002a5cf98 pc=0x477eb6\
sort.pdqsort(\{0x22dfca0, 0xc0069ceb70\}, 0x18?, 0x1b64280?, 0xc001278501?)\
        sort/zsortinterface.go:125 +0x2b5 fp=0xc002a5d098 sp=0xc002a5d018 pc=0x477ef5\
sort.Sort(\{0x22dfca0, 0xc0069ceb70\})\
        sort/sort.go:48 +0x5d fp=0xc002a5d0d0 sp=0xc002a5d098 pc=0x4761bd\
go.etcd.io/bbolt.(*freelist).hashmapGetFreePageIDs(0xc000664380)\
        go.etcd.io/bbolt@v1.3.7/freelist_hmap.go:84 +0x165 fp=0xc002a5d1e0 sp=0xc002a5d0d0 pc=0xa3f825\
go.etcd.io/bbolt.(*freelist).hashmapGetFreePageIDs-fm()\
        <autogenerated>:1 +0x26 fp=0xc002a5d1f8 sp=0xc002a5d1e0 pc=0xa4a206\
go.etcd.io/bbolt.(*freelist).copyall(0xc000664380, \{0xc014c4c018, 0xae16c, 0xae16c\})\
        go.etcd.io/bbolt@v1.3.7/freelist.go:104 +0x21d fp=0xc002a5d2f8 sp=0xc002a5d1f8 pc=0xa3d6dd\
go.etcd.io/bbolt.(*freelist).write(0xc0006d6ee0?, 0xc014c4c000)\
        go.etcd.io/bbolt@v1.3.7/freelist.go:335 +0xcb fp=0xc002a5d360 sp=0xc002a5d2f8 pc=0xa3e84b\
go.etcd.io/bbolt.(*Tx).commitFreelist(0xc0006d6ee0)\
        go.etcd.io/bbolt@v1.3.7/tx.go:243 +0xaa fp=0xc002a5d3a8 sp=0xc002a5d360 pc=0xa44f0a\
go.etcd.io/bbolt.(*Tx).Commit(0xc0006d6ee0)\
        go.etcd.io/bbolt@v1.3.7/tx.go:178 +0x1cd fp=0xc002a5d4d0 sp=0xc002a5d3a8 pc=0xa44a2d\
github.com/btcsuite/btcwallet/walletdb/bdb.(*transaction).Commit(0x22ebec0?)\
        github.com/btcsuite/btcwallet/walletdb@v1.4.0/bdb/db.go:101 +0x25 fp=0xc002a5d508 sp=0xc002a5d4d0 pc=0xa4b1a5\
github.com/btcsuite/btcwallet/walletdb/bdb.(*db).Update(0xc002a5d300?, 0xc002a04ad0, 0xc002a5d2e0?)\
        github.com/btcsuite/btcwallet/walletdb@v1.4.0/bdb/db.go:437 +0x11c fp=0xc002a5d580 sp=0xc002a5d508 pc=0xa4e13c\
github.com/lightningnetwork/lnd/kvdb.Update(...)\
        github.com/lightningnetwork/lnd/kvdb@v1.4.2/interface.go:16\
github.com/lightningnetwork/lnd/channeldb.(*ChannelGraph).MarkEdgeLive(0xc0009fe5a0, 0x0)\
        github.com/lightningnetwork/lnd/channeldb/graph.go:3899 +0xe5 fp=0xc002a5d610 sp=0xc002a5d580 pc=0x1514185\
github.com/lightningnetwork/lnd/routing.(*ChannelRouter).MarkEdgeLive(0xc0069d1550?, \{0x2409040?, 0xc0?, 0x0?\})\
        github.com/lightningnetwork/lnd/routing/router.go:2766 +0x35 fp=0xc002a5d630 sp=0xc002a5d610 pc=0x166a1f5\
github.com/lightningnetwork/lnd/discovery.(*AuthenticatedGossiper).processZombieUpdate(0xc0065ad600, 0xc0030d4580, 0xc0069d1550)\
        github.com/lightningnetwork/lnd/discovery/gossiper.go:2056 +0x1b7 fp=0xc002a5d6c0 sp=0xc002a5d630 pc=0x176bfb7\
github.com/lightningnetwork/lnd/discovery.(*AuthenticatedGossiper).handleChanUpdate(0xc0065ad600, 0xc00453d700, 0xc0069d1550, \{0xc0065830a0, 0x1, 0x1\})\
        github.com/lightningnetwork/lnd/discovery/gossiper.go:2731 +0xb1e fp=0xc002a5de90 sp=0xc002a5d6c0 pc=0x1770ede\
github.com/lightningnetwork/lnd/discovery.(*AuthenticatedGossiper).processNetworkAnnouncement(0xc007c0d1d0?, 0xc00453d700)\
        github.com/lightningnetwork/lnd/discovery/gossiper.go:2005 +0xfe fp=0xc002a5def8 sp=0xc002a5de90 pc=0x176bd5e\
github.com/lightningnetwork/lnd/discovery.(*AuthenticatedGossiper).handleNetworkMessages(0xc0065ad600, 0xc00453d700, 0x18ff846?, 0xc007c0d1d0)\
        github.com/lightningnetwork/lnd/discovery/gossiper.go:1506 +0x22d fp=0xc002a5dfb0 sp=0xc002a5def8 pc=0x1768f4d\
github.com/lightningnetwork/lnd/discovery.(*AuthenticatedGossiper).networkHandler.func4()\
        github.com/lightningnetwork/lnd/discovery/gossiper.go:1421 +0x32 fp=0xc002a5dfe0 sp=0xc002a5dfb0 pc=0x1768bd2\
runtime.goexit()\
        runtime/asm_amd64.s:1598 +0x1 fp=0xc002a5dfe8 sp=0xc002a5dfe0 pc=0x46d261\
created by github.com/lightningnetwork/lnd/discovery.(*AuthenticatedGossiper).networkHandler\
        github.com/lightningnetwork/lnd/discovery/gossiper.go:1421 +0x913\

This goroutine that was trying to call MarkEdgeLive and write the freelist was preempted (see: runtime.asyncPreempt2). In this specific scenario, I don't think there's a deadlock. But I don't know why that database transaction isn't happening. We did update bbolt in 17, but I'm not sure if that's related. How big is your database @niteshbalusu11 ?

How big is your database @niteshbalusu11 ?

@Crypt-iQ 429mb

Noticed something in this log dump that is separate from my comment above:

@Roasbeef Here's another goroutine dump. I think the issue still persists.
godump.zip

The ChannelRouter is trying to Stop and is waiting for the wait group counter to go to zero. However, there is an outstanding GetBlock call via fetchFundingTx:
goroutine 15327295 [select, 43 minutes]:\
runtime.gopark(0xc00d085878?, 0x3?, 0x20?, 0x70?, 0xc00d08583a?)\
        runtime/proc.go:398 +0xce fp=0xc00d0856d0 sp=0xc00d0856b0 pc=0x43df0e\
runtime.selectgo(0xc00d085878, 0xc00d085834, 0x6238e8?, 0x0, 0x0?, 0x1)\
        runtime/select.go:327 +0x725 fp=0xc00d0857f0 sp=0xc00d0856d0 pc=0x44e385\
github.com/btcsuite/btcwallet/chain.(*BitcoindConn).GetBlock(0xc00064c580, 0xc01d6d3da0)\
        github.com/btcsuite/btcwallet@v0.16.10-0.20230804184612-07be54bc22cf/chain/bitcoind_conn.go:410 +0x1d1 fp=0xc00d0858b8 sp=0xc00d0857f0 pc=0x947fd1\
github.com/btcsuite/btcwallet/chain.(*BitcoindClient).GetBlock(0xc008e2b770?, 0x0?)\
        github.com/btcsuite/btcwallet@v0.16.10-0.20230804184612-07be54bc22cf/chain/bitcoind_client.go:144 +0x17 fp=0xc00d0858d8 sp=0xc00d0858b8 pc=0x93f5f7\
github.com/btcsuite/btcwallet/chain.Interface.GetBlock-fm(0x96826bc740000002?)\
        <autogenerated>:1 +0x2b fp=0xc00d0858f8 sp=0xc00d0858d8 pc=0x1580d8b\
github.com/lightningnetwork/lnd/blockcache.(*BlockCache).GetBlock(0xc0014af8c0, 0xc01d6d3da0, 0xc00d085ac8)\
        github.com/lightningnetwork/lnd/blockcache/blockcache.go:55 +0x22a fp=0xc00d085a38 sp=0xc00d0858f8 pc=0x156e10a\
github.com/lightningnetwork/lnd/lnwallet/btcwallet.(*BtcWallet).GetBlock(0x1000001e00bab?, 0x7fdcf6b86c78?)\
        github.com/lightningnetwork/lnd/lnwallet/btcwallet/blockchain.go:139 +0x1de fp=0xc00d085b30 sp=0xc00d085a38 pc=0x1575d9e\
github.com/lightningnetwork/lnd/routing.(*ChannelRouter).fetchFundingTx(0xc000149040, 0xc008e2ade0)\
        github.com/lightningnetwork/lnd/routing/router.go:1780 +0x5c fp=0xc00d085ba0 sp=0xc00d085b30 pc=0x15a0bfc\
github.com/lightningnetwork/lnd/routing.(*ChannelRouter).processUpdate(0xc000149040, \{0x1c82220?, 0xc018812580?\}, \{0xc001a0b0d8, 0x1, 0x1\})\
        github.com/lightningnetwork/lnd/routing/router.go:1555 +0xea7 fp=0xc00d085f10 sp=0xc00d085ba0 pc=0x159fca7\
github.com/lightningnetwork/lnd/routing.(*ChannelRouter).handleNetworkUpdate(0xc000149040, 0xc009855bf0, 0xc0066a67e0)\
        github.com/lightningnetwork/lnd/routing/router.go:1031 +0x10d fp=0xc00d085fb8 sp=0xc00d085f10 pc=0x159c72d\
github.com/lightningnetwork/lnd/routing.(*ChannelRouter).networkHandler.func4()\
        github.com/lightningnetwork/lnd/routing/router.go:1129 +0x28 fp=0xc00d085fe0 sp=0xc00d085fb8 pc=0x159db08\
runtime.goexit()\
        runtime/asm_amd64.s:1650 +0x1 fp=0xc00d085fe8 sp=0xc00d085fe0 pc=0x4710e1\
created by github.com/lightningnetwork/lnd/routing.(*ChannelRouter).networkHandler in goroutine 61461\
        github.com/lightningnetwork/lnd/routing/router.go:1129 +0xbbf\

Since you're running a pruned node, it will wait here for the block: https://github.com/btcsuite/btcwallet/blob/9c135429d5b7c5c84b19eaffce9e6a32e2d9d5d0/chain/bitcoind_conn.go#L479
The logs also show that your node didn't have any peers at the time and was waiting here: https://github.com/lightninglabs/neutrino/blob/42a196facefac36282e68ddf4a02c9ce4601a0b0/query/workmanager.go#L263. This prevents shutdown until your node can connect to a peer that sends over the block.

goroutine 33483 [semacquire, 4282 minutes]:\
runtime.gopark(0xc000132530?, 0xc002271c20?, 0xc0?, 0x20?, 0xc006c0d0c0?)\
        runtime/proc.go:398 +0xce fp=0xc006c0d048 sp=0xc006c0d028 pc=0x43df0e\
runtime.goparkunlock(...)\
        runtime/proc.go:404\
runtime.semacquire1(0xc002e06d88, 0x1e?, 0x1, 0x0, 0x60?)\
        runtime/sema.go:160 +0x218 fp=0xc006c0d0b0 sp=0xc006c0d048 pc=0x44f3b8\
sync.runtime_Semacquire(0xc006c0d100?)\
        runtime/sema.go:62 +0x25 fp=0xc006c0d0e8 sp=0xc006c0d0b0 pc=0x46cea5\
sync.(*WaitGroup).Wait(0xc002b9dce0?)\
        sync/waitgroup.go:116 +0x48 fp=0xc006c0d110 sp=0xc006c0d0e8 pc=0x48f128\
github.com/lightningnetwork/lnd/contractcourt.(*ChannelArbitrator).Stop(0x1ac59e0?)\
        github.com/lightningnetwork/lnd/contractcourt/channel_arbitrator.go:826 +0x19c fp=0xc006c0d178 sp=0xc006c0d110 pc=0x150c8fc\
github.com/lightningnetwork/lnd/contractcourt.(*ChainArbitrator).ResolveContract(0xc0057ca4e0, \{\{0x6a, 0x25, 0xc7, 0x8, 0x2, 0x4e, 0xce, 0x95, 0x27, ...\}, ...\})\
        github.com/lightningnetwork/lnd/contractcourt/chain_arbitrator.go:468 +0x21c fp=0xc006c0d228 sp=0xc006c0d178 pc=0x15010bc\
github.com/lightningnetwork/lnd/contractcourt.newActiveChannelArbitrator.func4()\
        github.com/lightningnetwork/lnd/contractcourt/chain_arbitrator.go:404 +0x78 fp=0xc006c0d278 sp=0xc006c0d228 pc=0x1500cb8\
github.com/lightningnetwork/lnd/contractcourt.(*ChannelArbitrator).stateStep(0xc002e06b00, 0x1e93776?, 0x0, 0x1?)\
        github.com/lightningnetwork/lnd/contractcourt/channel_arbitrator.go:1265 +0x107c fp=0xc006c0dbb0 sp=0xc006c0d278 pc=0x150da1c\
github.com/lightningnetwork/lnd/contractcourt.(*ChannelArbitrator).advanceState(0xc002e06b00, 0x1e80b45?, 0x0, 0xc001a40640?)\
        github.com/lightningnetwork/lnd/contractcourt/channel_arbitrator.go:1512 +0x17b fp=0xc006c0dc50 sp=0xc006c0dbb0 pc=0x150fadb\
github.com/lightningnetwork/lnd/contractcourt.(*ChannelArbitrator).channelAttendant(0xc002e06b00, 0xc5454)\
        github.com/lightningnetwork/lnd/contractcourt/channel_arbitrator.go:2930 +0x10b2 fp=0xc006c0dfc0 sp=0xc006c0dc50 pc=0x1518352\
github.com/lightningnetwork/lnd/contractcourt.(*ChannelArbitrator).Start.func2()\
        github.com/lightningnetwork/lnd/contractcourt/channel_arbitrator.go:567 +0x25 fp=0xc006c0dfe0 sp=0xc006c0dfc0 pc=0x150b205\
runtime.goexit()\
        runtime/asm_amd64.s:1650 +0x1 fp=0xc006c0dfe8 sp=0xc006c0dfe0 pc=0x4710e1\
created by github.com/lightningnetwork/lnd/contractcourt.(*ChannelArbitrator).Start in goroutine 1\
        github.com/lightningnetwork/lnd/contractcourt/channel_arbitrator.go:567 +0x68e\
\

Isn't that also strange, seems to be circular as channelAttendant calls the ChannelArbitrator's Stop method, which is waiting on the work group channelAttendant it is part of?

Isn't that also strange, seems to be circular as channelAttendant calls the ChannelArbitrator's Stop method, which is waiting on the work group channelAttendant it is part of?

Good catch. This doesn't prevent shutdown, but is certainly not the right behavior. Apparently, that Stop function when called from channelAttendant just waits indefinitely

0.17.2-beta
still getting getting logs flooded with lines like
2023-12-04 17:11:18.158 [ERR] DISC: Update edge for short_chan_id(879313533626023936) got: router shutting down

This is preventing a shutdown of the node. Is this issue actually resolved?

@guggero @Crypt-iQ To be clear, the original text of this issue

I am running lnd 0.17.0 rc1 plus the fix of Elle that was merged recently. When trying to restart or shutting down LND my docker container never stops, I have 15min timeout in the compose file before a kill signal is sent by docker. I just get these logs and the timeout eventually hits and the container gets killed so LND doesn't shut down gracefully.

What I have described is the exact same issue described by OP. I posted about here here, this issue was echoed shortly after by @mycelia1 here, followed up with a goroutine that was reviewed by @guggero here.

That just sounds like a logging issue. Shutting down does involve quite a bit of work, so it might take some seconds.

I'm not talking seconds, I'm talking 1-2 hours.
The only difference between what I've described and the OP that I am not forcing a SIGTERM after 15m. It takes about 1-2 hours to shut down a node for a layer-2 solution that demands ~100% uptime from its nodes. So between the stated purpose and risks of said software, the layer-2 problem it aims to solve and the contents of this thread, it would be difficult to infer anything other than the following:
Blocking a restart while updating non-essential data in a graph in a circumstance that can result in monetary loss, is not the intended behavior.

The linked PR that closed this issue did fix a shutdown bug, but it seems that another issue is lock contention with the graph's cacheMu mutex.

The router tries to Stop and waits for the wait group to go to zero.

The handleNetworkUpdate goroutine is trying to call UpdateEdgePolicy which waits for cacheMu.

Many ChannelUpdate are received in the gossiper and call IsKnownEdge or MarkEdgeLive in handleChanUpdate. These functions try to acquire cacheMu.

These calls prevent the UpdateEdgePolicy calls from running until they are done.

I'm not sure how (or if) lock contention is handled in the go runtime (i.e. whether it's FIFO on mutex acquisition or something). I also wonder if we're receiving ChannelUpdate fast enough that it prevents shutdown.

Okay, thanks for the clarification. It did sound like you were mostly complaining about the log spam. But anything more than a minute for shutting down is unacceptable, I agree.

Maybe this issue is resolved? See #8250 for a description of why lnd refused to stop (using a pruned bitcoind).

Hi @niteshbalusu11 can you please confirm if this is still an issue with lnd 0.17.4? thanks


support "set time zone to 'some-timezone'"

close #303
sqlparser-rs original only supports SET TIMEZONE = 'some-timezone' now.
there're many dbs support both TIMEZONE and TIME ZONE in the SET statement (e.g. postgresql, pyspark, ...)
this pr is to enable SET TIMEZONE = 'some-timezone'

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

25 of 25 (100.0%)  changed or added relevant lines in 2 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.03%) to 85.589%

Totals

Change from base Build<PHONE_NUMBER>:
0.03%

Covered Lines:
9752

Relevant Lines:
11394

💛  - Coveralls

@avantgardnerio @alamb this could enable set timezone and set time zone


[v2.10] Add support for docker 27.4.0 and 27.4.1

Add support for docker v27.4.0 and v27.4.1
https://docs.docker.com/engine/release-notes/27/#274

Tested Docker v27.4.1 install script on rancher v2.10-head and v2.9-head.
Docker install script worked correctly and cluster provisions successfully on following OSes.

Ubuntu 22.04 & 24.04
RHEL, Rocky and OL (9.4 and 8.10)

Verified on rancher v2.10-head with k8s v1.30 :

Tested docker v27.4.0 install script, by successfully provisioning node driver and custom cluster on below OSes:

RHEL 9.4 and 8.10
Rocky 9.4 and 8.10
OL 9.4 and 8.10
Ubuntu 24 and 22


Centos and epel packages fail to build on COPR

Which jobs are failing?
The recent builds fail on each main commit: https://copr.fedorainfracloud.org/coprs/rhcontainerbot/podman-next/build/4788630/
Which tests are failing?
Every copr build on main
Since when has it been failing?
I don't think they ever worked.
Reason for failure (if possible)
No response
Anything else we need to know?
No response
cc @rphillips @lsm5

I removed the dependency on external copr lsm5/wasmtime and added wasmtime into the podman-next copr itself.
Latest build at: https://copr.fedorainfracloud.org/coprs/rhcontainerbot/podman-next/build/4803413/
c9s and c8s are successful.
epel 9 uses cargo: cargo-1.58.1-1.el9 and currently it's failing with:
cargo build --release
    Updating crates.io index
error: failed to select a version for the requirement `tzdb = "^0.4.4"`
candidate versions found which didn't match: 0.3.6, 0.3.5, 0.3.4, ...
location searched: crates.io index
required by package `shadow-rs v0.16.3`
    ... which satisfies dependency `shadow-rs = "^0.16.3"` (locked to 0.16.3) of package `conmonrs v0.1.0 (/builddir/build/BUILD/conmon-rs/conmon-rs/server)`
make: *** [Makefile:18: release] Error 101
error: Bad exit status from /var/tmp/rpm-tmp.4cFbMb (%build)
    Bad exit status from /var/tmp/rpm-tmp.4cFbMb (%build)

Similar for epel-8: cargo-1.58.1-1.module+el8.6.0+14021+586eff1a
cargo build --release
    Updating crates.io index

RPM build errors:
error: failed to select a version for the requirement `tzdb = "^0.4.4"`
candidate versions found which didn't match: 0.3.6, 0.3.5, 0.3.4, ...
location searched: crates.io index
required by package `shadow-rs v0.16.3`
    ... which satisfies dependency `shadow-rs = "^0.16.3"` (locked to 0.16.3) of package `conmonrs v0.1.0 (/builddir/build/BUILD/conmon-rs/conmon-rs/server)`
make: *** [Makefile:18: release] Error 101
error: Bad exit status from /var/tmp/rpm-tmp.CyesTZ (%build)
    Bad exit status from /var/tmp/rpm-tmp.CyesTZ (%build)

@saschagrunert @haircommander @rphillips If we're happy with c9s and c8s, I could just disable the epel builds, let me know.

@lsm5 thank you! I assume that we need the epel builds for RHCOS later on, so I guess we have to fix the build on our end.

@saschagrunert so c8s uses cargo v1.60 which is the least version (AFAICT) for a successful build, while epel8 and epel9 are currently 1.58. Possible options are:

Make conmon-rs work with cargo 1.58
Wait for epel buildroot to get cargo 1.60 (I gotta check ETA )
Do some gymnastics for epel repos to use cargo from stream repos.

I think 1 would be ideal if doable, but if not, I can check on 2 and 3.

Make conmon-rs work with cargo 1.58

I played around with 1.58 and we could disable the tzdb feature, but this leads into a new issue:
error: package `time v0.3.14` cannot be built because it requires rustc 1.59.0 or newer, while the currently active rustc version is 1.58.1

Make conmon-rs work with cargo 1.58

I played around with 1.58 and we could disable the tzdb feature, but this leads into a new issue:
error: package `time v0.3.14` cannot be built because it requires rustc 1.59.0 or newer, while the currently active rustc version is 1.58.1

ack, let me check on the other options and get back. Thanks!

I'm building cargo 1.59 at https://copr.fedorainfracloud.org/coprs/rhcontainerbot/podman-next/build/4805564/ after which I'll use it to bootstrap cargo 1.60 on that copr and then get conmon-rs built for epel repos. The rustc builds take a while, so this should complete sometime next week (hopefully).

I'm building cargo 1.59 at https://copr.fedorainfracloud.org/coprs/rhcontainerbot/podman-next/build/4805564/ after which I'll use it to bootstrap cargo 1.60 on that copr and then get conmon-rs built for epel repos. The rustc builds take a while, so this should complete sometime next week (hopefully).

Cool, looks like it succeeded! Thank you so much!

rust 1.60 building
epel9: https://copr.fedorainfracloud.org/coprs/rhcontainerbot/podman-next/build/4809364/
epel8: https://copr.fedorainfracloud.org/coprs/rhcontainerbot/podman-next/build/4809442/
once these are in place, i can go ahead with conmon-rs rebuild. I'll also try to keep rust for epel on copr in-sync with the stream versions going forward.

mostly successful build at https://copr.fedorainfracloud.org/coprs/rhcontainerbot/podman-next/build/4811256/
@saschagrunert @haircommander is conmon-rs known to not work on i386? I can disable i386 for it in that case. Just that i686 is still one of the default fedora arches, so we might wanna ExclusiveArch it for official packaging.

btw, I also wanna make sure everyone's aware before consuming this repo (especially coreos people). The podman-next copr also has other packages which are built from their respective upstream main branches. So, if coreos doesn't want the latest podman, crun etc., those should be excluded from the get go in their yum configs.

@saschagrunert @haircommander is conmon-rs known to not work on i386?

No I don't think so but I'm working on a fix in https://github.com/containers/conmon-rs/pull/688


Change linter to pycodestyle

I want to recommend that we switch from flake8 as our linter to pycodestyle as it supports automatic correction through autopep8. Automatic correction reduces linting overhead as if the style can be applied automatically then the developer doesn't have to worry about it. So far, only pycodestyle/autopep8 and black support auto-correction of python linting problems.

Nice, thanks so much for testing! I checked out both and I definitely am a fan of the black-produced syntax, and like the idea of it having stronger opinions. If you found them both equally easy to work with, I'd say let's go with black.
Do we add it as a check and require people to run it locally, or is there an easy way to integrate w/ Github such that it auto-commits the changes for each PR commit?

I don't have much experience here, so very little to add.  However, I did notice that black is still in Beta.  Not that that is a deterrent, but I just wanted to make sure that that's clear.

Good reminder! I'm fine with that, I worked in another project with it and enjoyed it :) @jaron-l , if you have any time to send over a Black PR I'd love to review!

I'll get to this soon.
On Thu, Nov 4, 2021 at 7:17 PM Michael Rooney 'notifications at github.com'
<jaron-github+reply+ADGNKUGVF33VBZRMH3VPS3F7SBTTVEVBNHHD3RK4EE=
@.***> wrote:

Good reminder! I'm fine with that, I worked in another project with it and
enjoyed it :) @jaron-l https://github.com/jaron-l , if you have any
time to send over a Black PR I'd love to review!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/mintapi/mintapi/issues/324#issuecomment-961530995,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADGNKUEGPTOVAH6XYSNJFDTUKMWDVANCNFSM5GX42BGA
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

In my attempt to create a pull request for this, I noticed that black did indeed break our code. The only example that was caught by tests was it removed the return statement from Mint::get_transactions.
I can easily fix it but perhaps this shows black is still in beta and maybe we should perhaps reconsider using autopep8? I can see both sides:
Pro for autopep8 is that because it is less aggressive, it is more stable and less likely to break code
Pro for black is that in nominal conditions, the only code it will be formatting in PRs in the PR specific code, which if we have a test standard for, will unlikely get broken due to black.