HDFS-17218. NameNode should remove its excess blocks from the ExcessRedundancyMap When a DN registers

Description of PR
https://issues.apache.org/jira/browse/HDFS-17218
Currently found that DN will lose all pending DNA_INVALIDATE blocks if it restarts.
DN enables asynchronously deletion, it have many pending deletion blocks in memory.
when DN restarts, these cached blocks may be lost. it causes some blocks in the excess map in the namenode to be leaked and this will result in many blocks having more replicas then expected.
Root case
1.block1 of dn1 is chosen as excess, added to excessRedundancyMap and add To Invalidates.
2.dn1 heartbeat gets Invalidates command.
3.dn1 will execute async deletion when receive commands, but before it is actually deleted, the service stop, so the block1 still exsit.
4.at this time, nn's excessRedundancyMap will still have the block of dn1
5. restart the dn, at this time nn has not determined that the dn is in a dead state.
6. dn restarts will FBR is executed (processFirstBlockReport will not be executed here, processReport will be executed). since block1 is not a new block, the processExtraRedundancy logic will not be executed.
In HeartbeatManager#register(final DatanodeDescriptor d)
https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HeartbeatManager.java#L230-L238
//here current dn still is alive(expired heartbeat time has not been exceeded), dn register will not call d.updateHeartbeatState, so torageInfo.hasReceivedBlockReport() still is true
synchronized void register(final DatanodeDescriptor d) {
  if (!d.isAlive()) {
    addDatanode(d);
    //update its timestamp
    d.updateHeartbeatState(StorageReport.EMPTY_ARRAY, 0L, 0L, 0, 0, null);
    stats.add(d);
  }
}

In BlockManager#processReport, the dn restart run FBR, here current dn still is alive,storageInfo.hasReceivedBlockReport() is true, so will call method processReport
https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java#L2916-L2946
if (!storageInfo.hasReceivedBlockReport()) {
        // The first block report can be processed a lot more efficiently than
        // ordinary block reports.  This shortens restart times.
        blockLog.info("BLOCK* processReport 0x{} with lease ID 0x{}: Processing first "
            + "storage report for {} from datanode {}",
            strBlockReportId, fullBrLeaseId,
            storageInfo.getStorageID(),
            nodeID);
        processFirstBlockReport(storageInfo, newReport);
      } else {
        // Block reports for provided storage are not
        // maintained by DN heartbeats
        if (!StorageType.PROVIDED.equals(storageInfo.getStorageType())) {
          invalidatedBlocks = processReport(storageInfo, newReport);
        }
      }
      storageInfo.receivedBlockReport();
    } finally {
      endTime = Time.monotonicNow();
      namesystem.writeUnlock("processReport");
    }

    if (blockLog.isDebugEnabled()) {
      for (Block b : invalidatedBlocks) {
        blockLog.debug("BLOCK* processReport 0x{} with lease ID 0x{}: {} on node {} size {} " +
                "does not belong to any file.", strBlockReportId, fullBrLeaseId, b,
            node, b.getNumBytes());
      }
    }

In BlockManager#processReport run FBR, since the current DatanodeStorageInfo exists in the triplets in the BlockInfo corresponding to the reported block, so will not add toAdd list, addStoredBlock and processExtraRedundancy logic will not be executed.
https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java#L3044-L3085
Collection<Block> processReport(
      final DatanodeStorageInfo storageInfo,
      final BlockListAsLongs report) throws IOException {
    // Normal case:
    // Modify the (block-->datanode) map, according to the difference
    // between the old and new block report.
    //
    Collection<BlockInfoToAdd> toAdd = new ArrayList<>();
    Collection<BlockInfo> toRemove = new HashSet<>();
    Collection<Block> toInvalidate = new ArrayList<>();
    Collection<BlockToMarkCorrupt> toCorrupt = new ArrayList<>();
    Collection<StatefulBlockInfo> toUC = new ArrayList<>();
    reportDiff(storageInfo, report,
                 toAdd, toRemove, toInvalidate, toCorrupt, toUC);

    DatanodeDescriptor node = storageInfo.getDatanodeDescriptor();
    // Process the blocks on each queue
    for (StatefulBlockInfo b : toUC) {
      addStoredBlockUnderConstruction(b, storageInfo);
    }
    for (BlockInfo b : toRemove) {
      removeStoredBlock(b, node);
    }
    int numBlocksLogged = 0;
    for (BlockInfoToAdd b : toAdd) {
      addStoredBlock(b.stored, b.reported, storageInfo, null,
          numBlocksLogged < maxNumBlocksToLog);
      numBlocksLogged++;
    }
    if (numBlocksLogged > maxNumBlocksToLog) {
      blockLog.info("BLOCK* processReport: logged info for {} of {} " +
          "reported.", maxNumBlocksToLog, numBlocksLogged);
    }
    for (Block b : toInvalidate) {
      addToInvalidates(b, node);
    }
    for (BlockToMarkCorrupt b : toCorrupt) {
      markBlockAsCorrupt(b, storageInfo, node);
    }

    return toInvalidate;
  }

so the block of dn1 will always exist in excessRedundancyMap (until HA switch is performed).

In BlockManager#processChosenExcessRedundancy will add the redundancy of the given block stored in the given datanode to the excess map.
https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java#L4267-L4285
private void processChosenExcessRedundancy(
      final Collection<DatanodeStorageInfo> nonExcess,
      final DatanodeStorageInfo chosen, BlockInfo storedBlock) {
    nonExcess.remove(chosen);
    excessRedundancyMap.add(chosen.getDatanodeDescriptor(), storedBlock);
    //
    // The 'excessblocks' tracks blocks until we get confirmation
    // that the datanode has deleted them; the only way we remove them
    // is when we get a "removeBlock" message.
    //
    // The 'invalidate' list is used to inform the datanode the block
    // should be deleted.  Items are removed from the invalidate list
    // upon giving instructions to the datanodes.
    //
    final Block blockToInvalidate = getBlockOnStorage(storedBlock, chosen);
    addToInvalidates(blockToInvalidate, chosen.getDatanodeDescriptor());
    blockLog.debug("BLOCK* chooseExcessRedundancies: ({}, {}) is added to invalidated blocks set",
        chosen, storedBlock);
  }

but because the dn side has not deleted the block, it will not call processIncrementalBlockReport, so the block of dn can not remove from excessRedundancyMap.
Solution
Current consider doing some cleanup operations when the dn is re-registered (the status on the NN is still isAlive), such as :

remove blocks from excessRedundancyMap by datanode.
remove blocks to be invalidated by datanode.
remove blocks from invalidateBlocks by datanode.
add blocks into postponed queue only when active NN is out of safe mode, the block can process extra redundancy block

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 52s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 2 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
48m 13s

trunk passed

+1 :green_heart:
compile
1m 23s

trunk passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
compile
1m 14s

trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
checkstyle
1m 11s

trunk passed

+1 :green_heart:
mvnsite
1m 24s

trunk passed

+1 :green_heart:
javadoc
1m  8s

trunk passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
javadoc
1m 35s

trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
spotbugs
3m 24s

trunk passed

+1 :green_heart:
shadedclient
40m  6s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
1m 12s

the patch passed

+1 :green_heart:
compile
1m 15s

the patch passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
javac
1m 15s

the patch passed

+1 :green_heart:
compile
1m  7s

the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
javac
1m  7s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
1m  3s

the patch passed

+1 :green_heart:
mvnsite
1m 15s

the patch passed

+1 :green_heart:
javadoc
0m 57s

the patch passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
javadoc
1m 29s

the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
spotbugs
3m 28s

the patch passed

+1 :green_heart:
shadedclient
40m 23s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

-1 :x:
unit
120m  5s
/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs in the patch passed.

+0 :ok:
asflicense
0m 44s

ASF License check generated no output?

274m  1s

Reason
Tests

Failed junit tests
hadoop.hdfs.TestFileChecksum

hadoop.hdfs.TestMaintenanceWithStriped

hadoop.hdfs.TestErasureCodingExerciseAPIs

hadoop.hdfs.TestFileAppend2

hadoop.hdfs.TestDFSStripedOutputStreamWithRandomECPolicy

hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy

hadoop.hdfs.TestWriteReadStripedFile

hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy

Subsystem
Report/Notes

Docker
ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6176/2/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/6176

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets

uname
Linux 50977c3b1dae 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19 13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 6024caf2a7e4965dc2103b83a0c32b67058860d2

Default Java
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6176/2/testReport/

Max. process+thread count
2213 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6176/2/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
23m 18s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+0 :ok:
detsecrets
0m  1s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 2 new or modified test files.

_ trunk Compile Tests _

-1 :x:
mvninstall
0m 26s
/branch-mvninstall-root.txt
root in trunk failed.

-1 :x:
compile
0m 35s
/branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04.txt
hadoop-hdfs in trunk failed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04.

-1 :x:
compile
0m 25s
/branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt
hadoop-hdfs in trunk failed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05.

+1 :green_heart:
checkstyle
5m 27s

trunk passed

-1 :x:
mvnsite
0m 17s
/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs in trunk failed.

-1 :x:
javadoc
0m 38s
/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04.txt
hadoop-hdfs in trunk failed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04.

-1 :x:
javadoc
0m 24s
/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt
hadoop-hdfs in trunk failed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05.

-1 :x:
spotbugs
0m 17s
/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs in trunk failed.

-1 :x:
shadedclient
10m 14s

branch has errors when building and testing our client artifacts.

_ Patch Compile Tests _

-1 :x:
mvninstall
0m 25s
/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs in the patch failed.

-1 :x:
compile
0m 21s
/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04.txt
hadoop-hdfs in the patch failed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04.

-1 :x:
javac
0m 21s
/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04.txt
hadoop-hdfs in the patch failed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04.

-1 :x:
compile
0m 25s
/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt
hadoop-hdfs in the patch failed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05.

-1 :x:
javac
0m 25s
/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt
hadoop-hdfs in the patch failed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05.

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

-0 :warning:
checkstyle
0m 23s
/buildtool-patch-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
The patch fails to run checkstyle in hadoop-hdfs

-1 :x:
mvnsite
0m 23s
/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs in the patch failed.

-1 :x:
javadoc
1m 20s
/results-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04.txt
hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04 with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04 generated 99 new + 0 unchanged - 0 fixed = 99 total (was 0)

-1 :x:
javadoc
0m 35s
/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt
hadoop-hdfs in the patch failed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05.

-1 :x:
spotbugs
0m 26s
/patch-spotbugs-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs in the patch failed.

+1 :green_heart:
shadedclient
10m 41s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

-1 :x:
unit
0m 25s
/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs in the patch failed.

+1 :green_heart:
asflicense
0m 42s

The patch does not generate ASF License warnings.

48m 56s

Subsystem
Report/Notes

Docker
ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6176/3/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/6176

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets

uname
Linux 5a634e698c7e 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19 13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / a63b1a086641814de796404d04a5d761435a3aad

Default Java
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6176/3/testReport/

Max. process+thread count
80 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6176/3/console

versions
git=2.25.1 maven=3.6.3

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 51s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 2 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
48m 44s

trunk passed

+1 :green_heart:
compile
1m 23s

trunk passed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
compile
1m 13s

trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
checkstyle
1m 12s

trunk passed

+1 :green_heart:
mvnsite
1m 25s

trunk passed

+1 :green_heart:
javadoc
1m  9s

trunk passed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
1m 36s

trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
spotbugs
3m 23s

trunk passed

+1 :green_heart:
shadedclient
41m 35s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
1m 11s

the patch passed

+1 :green_heart:
compile
1m 17s

the patch passed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javac
1m 17s

the patch passed

+1 :green_heart:
compile
1m  9s

the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
javac
1m  9s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
1m  3s

the patch passed

+1 :green_heart:
mvnsite
1m 15s

the patch passed

+1 :green_heart:
javadoc
0m 56s

the patch passed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
1m 29s

the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
spotbugs
3m 20s

the patch passed

+1 :green_heart:
shadedclient
40m 27s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

-1 :x:
unit
242m  5s
/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs in the patch passed.

+1 :green_heart:
asflicense
0m 42s

The patch does not generate ASF License warnings.

398m  3s

Reason
Tests

Failed junit tests
hadoop.hdfs.server.datanode.TestDirectoryScanner

hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy

Subsystem
Report/Notes

Docker
ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6176/4/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/6176

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets

uname
Linux 36a7fabcf599 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19 13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 9bcc4d81da0aa93e74dd9a893d86554ccefb47d6

Default Java
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6176/4/testReport/

Max. process+thread count
2292 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6176/4/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

The failed unit test seems unrelated to the change.

Hi @Hexiaoqiao @zhangshuyan0  do you have any comments or suggestions about this PR? Thanks.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 53s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+0 :ok:
detsecrets
0m  1s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 2 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
48m 58s

trunk passed

+1 :green_heart:
compile
1m 25s

trunk passed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
compile
1m 14s

trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
checkstyle
1m 11s

trunk passed

+1 :green_heart:
mvnsite
1m 23s

trunk passed

+1 :green_heart:
javadoc
1m  8s

trunk passed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
1m 40s

trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
spotbugs
3m 22s

trunk passed

+1 :green_heart:
shadedclient
40m 38s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
1m 13s

the patch passed

+1 :green_heart:
compile
1m 18s

the patch passed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javac
1m 18s

the patch passed

+1 :green_heart:
compile
1m  9s

the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
javac
1m  9s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
1m  2s

the patch passed

+1 :green_heart:
mvnsite
1m 13s

the patch passed

+1 :green_heart:
javadoc
0m 56s

the patch passed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
1m 29s

the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
spotbugs
3m 27s

the patch passed

+1 :green_heart:
shadedclient
40m 59s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

-1 :x:
unit
244m 17s
/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs in the patch passed.

+1 :green_heart:
asflicense
1m  0s

The patch does not generate ASF License warnings.

400m 26s

Reason
Tests

Failed junit tests
hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes

Subsystem
Report/Notes

Docker
ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6176/5/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/6176

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets

uname
Linux a943d1e8742e 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19 13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / c6cea4228e85dbd0b6c6a01650e69f65e3e3ad44

Default Java
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6176/5/testReport/

Max. process+thread count
2423 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6176/5/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 26s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+0 :ok:
detsecrets
0m  1s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 2 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
33m 21s

trunk passed

+1 :green_heart:
compile
0m 53s

trunk passed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
compile
0m 47s

trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
checkstyle
0m 46s

trunk passed

+1 :green_heart:
mvnsite
0m 56s

trunk passed

+1 :green_heart:
javadoc
0m 50s

trunk passed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
1m 12s

trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
spotbugs
1m 55s

trunk passed

+1 :green_heart:
shadedclient
21m 22s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
0m 40s

the patch passed

+1 :green_heart:
compile
0m 47s

the patch passed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javac
0m 47s

the patch passed

+1 :green_heart:
compile
0m 41s

the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
javac
0m 41s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
0m 38s

the patch passed

+1 :green_heart:
mvnsite
0m 45s

the patch passed

+1 :green_heart:
javadoc
0m 37s

the patch passed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
0m 59s

the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
spotbugs
1m 57s

the patch passed

+1 :green_heart:
shadedclient
21m 35s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

-1 :x:
unit
138m 35s
/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs in the patch passed.

+0 :ok:
asflicense
0m 29s

ASF License check generated no output?

231m 20s

Reason
Tests

Failed junit tests
hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion

hadoop.hdfs.server.namenode.TestMetaSave

hadoop.hdfs.server.namenode.snapshot.TestRandomOpsWithSnapshots

hadoop.hdfs.server.namenode.TestCacheDirectivesWithViewDFS

hadoop.hdfs.server.namenode.TestListOpenFiles

hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength

hadoop.hdfs.server.namenode.TestNameNodeReconfigure

hadoop.hdfs.server.namenode.TestHDFSConcat

hadoop.hdfs.server.namenode.TestNamenodeRetryCache

hadoop.hdfs.server.namenode.snapshot.TestFileContextSnapshot

hadoop.hdfs.server.namenode.snapshot.TestINodeFileUnderConstructionWithSnapshot

hadoop.hdfs.server.namenode.snapshot.TestSnapshotNameWithInvalidCharacters

Subsystem
Report/Notes

Docker
ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6176/7/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/6176

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets

uname
Linux 6c018a447c49 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19 13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 64dc3196d60242dcd089d524c499a0abbde4ade9

Default Java
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6176/7/testReport/

Max. process+thread count
3160 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6176/7/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
1m  9s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+0 :ok:
detsecrets
0m  1s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 2 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
51m  6s

trunk passed

+1 :green_heart:
compile
1m 27s

trunk passed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
compile
1m 26s

trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
checkstyle
1m 17s

trunk passed

+1 :green_heart:
mvnsite
1m 25s

trunk passed

+1 :green_heart:
javadoc
1m 16s

trunk passed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
1m 44s

trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
spotbugs
3m 46s

trunk passed

+1 :green_heart:
shadedclient
44m 28s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
1m 20s

the patch passed

+1 :green_heart:
compile
1m 30s

the patch passed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javac
1m 30s

the patch passed

+1 :green_heart:
compile
1m 14s

the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
javac
1m 14s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
1m  8s

the patch passed

+1 :green_heart:
mvnsite
1m 23s

the patch passed

+1 :green_heart:
javadoc
1m  4s

the patch passed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
1m 34s

the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
spotbugs
3m 42s

the patch passed

+1 :green_heart:
shadedclient
43m 54s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

-1 :x:
unit
239m 25s
/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs in the patch passed.

+1 :green_heart:
asflicense
0m 42s

The patch does not generate ASF License warnings.

407m  9s

Reason
Tests

Failed junit tests
hadoop.hdfs.server.datanode.TestDirectoryScanner

Subsystem
Report/Notes

Docker
ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6176/6/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/6176

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets

uname
Linux ff37c399625c 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19 13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 2cd0509ae1f9547558be8a4e404cf1aa4c035923

Default Java
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6176/6/testReport/

Max. process+thread count
2300 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6176/6/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

Back to this PR.

IMO, adding a timeout mechanism may not add much pressure on NameNode. However, it seems that the implementation of that solution is more complex than the current patch and requires more comprehensive design and consideration. The good aspect is that the timeout mechanism can completely solve the problem of excess replica leakage, after all, the situation where datanodes fail to successfully delete replicas according to commands may not be limited to the scenario described in this JIRA.

I totally support the solution @zhangshuyan0 mentioned here. https://issues.apache.org/jira/browse/HDFS-17218?focusedCommentId=17774766&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17774766

Back to this PR.

IMO, adding a timeout mechanism may not add much pressure on NameNode. However, it seems that the implementation of that solution is more complex than the current patch and requires more comprehensive design and consideration. The good aspect is that the timeout mechanism can completely solve the problem of excess replica leakage, after all, the situation where datanodes fail to successfully delete replicas according to commands may not be limited to the scenario described in this JIRA.

I totally support the solution @zhangshuyan0 mentioned here. https://issues.apache.org/jira/browse/HDFS-17218?focusedCommentId=17774766&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17774766

Thanks @Hexiaoqiao  for your comment.
Indeed this timeout mechanism solution can completely solve the problem of redundant copy leakage, however the implementation cost of this solution maybe relatively high. for a large cluster, such as balance and reduce replication etc occur frequently, and ExcessRedundancyMap stores a lot of information.
During each iteration, DataNodes and their corresponding blocks are fully traversed for processing. this approach might potentially increase the load on holding the NameNode write lock, sure we can make good designs to avoid holding the write lock time as much as possible.
The current pr to slove problem of ExcessRedundancyMap leakage on a case-by-case basis, and  the implementation cost is relatively low.
I think if there will be problems like ExcessRedundancyMap leakage in the future again, we should probably solve it case by case and find the root cause of the leakage to solve it.
Of course, if we decide to adopt the timeout mechanism solution, I will submit a new PR. look forward to your feedback. Thanks.

I try to implement this based on the timeout mechanism solution. However, there is a case where I have some questions, such as:

t1 time: Block1 on DN1 is choosed to be added to ExcessRedundancyMap.
t2 time: DN1 heartbeat gets Invalidates command.
t3 time: Due to a  serious accumulationin DN1 async deletion queue, the replica might not be deleted for a prolonged period.

The question here  is how the current NN can define a reasonable timeframe to determine whether Block1 corresponding to DN1 in ExcessRedundancyMap has timed out.
Currently, I haven't  think of a particularly good way to define this.
Hi @Hexiaoqiao @ZanderXu @zhangshuyan0 excuse me, do you have any suggestions for this case?
look forward to your feedback, Thanks~

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
1m 28s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  3s

codespell was not available.

+0 :ok:
detsecrets
0m  3s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 2 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
53m  8s

trunk passed

+1 :green_heart:
compile
1m 31s

trunk passed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
compile
1m 14s

trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
checkstyle
1m 13s

trunk passed

+1 :green_heart:
mvnsite
1m 26s

trunk passed

-1 :x:
javadoc
1m 11s
/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04.txt
hadoop-hdfs in trunk failed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04.

+1 :green_heart:
javadoc
1m 41s

trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
spotbugs
3m 30s

trunk passed

+1 :green_heart:
shadedclient
41m 55s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
1m 16s

the patch passed

+1 :green_heart:
compile
1m 18s

the patch passed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javac
1m 18s

the patch passed

+1 :green_heart:
compile
1m 19s

the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
javac
1m 19s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
1m  4s

the patch passed

+1 :green_heart:
mvnsite
1m 21s

the patch passed

-1 :x:
javadoc
0m 58s
/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04.txt
hadoop-hdfs in the patch failed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04.

+1 :green_heart:
javadoc
2m  0s

the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
spotbugs
4m 19s

the patch passed

+1 :green_heart:
shadedclient
42m 57s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
255m 25s

hadoop-hdfs in the patch passed.

+1 :green_heart:
asflicense
1m 16s

The patch does not generate ASF License warnings.

422m  1s

Subsystem
Report/Notes

Docker
ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6176/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/6176

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets

uname
Linux a97889964d53 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19 13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 64dc3196d60242dcd089d524c499a0abbde4ade9

Default Java
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6176/1/testReport/

Max. process+thread count
2223 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6176/1/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

I think we can determine whether the replica in ExcessRedundancyMap has timed out based on the configured timeout paprameter. As for the scenario you mentioned, I think this can be done directly: NN determines that DN1 has timed out and sends it another delete command. Will this have any adverse effects?

I think we can determine whether the replica in ExcessRedundancyMap has timed out based on the configured timeout paprameter. As for the scenario you mentioned, I think this can be done directly: NN determines that DN1 has timed out and sends it another delete command. Will this have any adverse effects?

Thanks @zhangshuyan0 for  your detailed suggestions.
I think this should work, I will submit the MR as soon as possible, thanks again.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 53s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+0 :ok:
xmllint
0m  0s

xmllint was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 2 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
49m 28s

trunk passed

+1 :green_heart:
compile
1m 23s

trunk passed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
compile
1m 14s

trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
checkstyle
1m 11s

trunk passed

+1 :green_heart:
mvnsite
1m 22s

trunk passed

+1 :green_heart:
javadoc
1m 10s

trunk passed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
1m 37s

trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
spotbugs
3m 20s

trunk passed

+1 :green_heart:
shadedclient
40m 44s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
1m 10s

the patch passed

+1 :green_heart:
compile
1m 17s

the patch passed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javac
1m 17s

the patch passed

+1 :green_heart:
compile
1m  8s

the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
javac
1m  8s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
1m  3s

the patch passed

+1 :green_heart:
mvnsite
1m 16s

the patch passed

+1 :green_heart:
javadoc
0m 57s

the patch passed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
1m 33s

the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
spotbugs
3m 23s

the patch passed

+1 :green_heart:
shadedclient
42m 23s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

-1 :x:
unit
242m 34s
/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs in the patch passed.

+1 :green_heart:
asflicense
0m 50s

The patch does not generate ASF License warnings.

400m 55s

Reason
Tests

Failed junit tests
hadoop.hdfs.server.datanode.TestDirectoryScanner

Subsystem
Report/Notes

Docker
ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6176/8/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/6176

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint

uname
Linux 5600f898960d 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19 13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 7fa80cc5de2c2b929d1c950e397f6a4d12f0449d

Default Java
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6176/8/testReport/

Max. process+thread count
2521 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6176/8/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

Hi @Hexiaoqiao @ayushtkn @ZanderXu @zhangshuyan0
Could you please help review this pr again when you have free time. Thanks~

Test failures seems unrelated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 57s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+0 :ok:
xmllint
0m  0s

xmllint was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 2 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
50m  4s

trunk passed

+1 :green_heart:
compile
1m 25s

trunk passed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
compile
1m 19s

trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
checkstyle
1m 14s

trunk passed

+1 :green_heart:
mvnsite
1m 26s

trunk passed

+1 :green_heart:
javadoc
1m 11s

trunk passed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
1m 38s

trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
spotbugs
3m 30s

trunk passed

+1 :green_heart:
shadedclient
41m 56s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
1m 21s

the patch passed

+1 :green_heart:
compile
1m 21s

the patch passed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javac
1m 21s

the patch passed

+1 :green_heart:
compile
1m 10s

the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
javac
1m 10s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
1m  5s

the patch passed

+1 :green_heart:
mvnsite
1m 19s

the patch passed

+1 :green_heart:
javadoc
1m  0s

the patch passed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
1m 33s

the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
spotbugs
3m 35s

the patch passed

+1 :green_heart:
shadedclient
41m 33s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

-1 :x:
unit
246m 10s
/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs in the patch passed.

+1 :green_heart:
asflicense
0m 50s

The patch does not generate ASF License warnings.

406m 37s

Reason
Tests

Failed junit tests
hadoop.hdfs.TestDFSUtil

Subsystem
Report/Notes

Docker
ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6176/9/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/6176

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint

uname
Linux 93d1c35b9681 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19 13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / cafe3c666a53c04f84a6eccaa4d3a2d0423987db

Default Java
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6176/9/testReport/

Max. process+thread count
1978 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6176/9/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

Thanks @Hexiaoqiao for your  comment, i will fix it later.

Update PR.
Hi @Hexiaoqiao @zhangshuyan0 please help me review it again when you have free time. Thanks~

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
1m  1s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+0 :ok:
xmllint
0m  0s

xmllint was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 2 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
48m  6s

trunk passed

+1 :green_heart:
compile
1m 23s

trunk passed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
compile
1m 14s

trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
checkstyle
1m 15s

trunk passed

+1 :green_heart:
mvnsite
1m 24s

trunk passed

+1 :green_heart:
javadoc
1m  9s

trunk passed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
1m 37s

trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
spotbugs
3m 19s

trunk passed

+1 :green_heart:
shadedclient
40m 27s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
1m 13s

the patch passed

+1 :green_heart:
compile
1m 16s

the patch passed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javac
1m 16s

the patch passed

+1 :green_heart:
compile
1m  7s

the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
javac
1m  7s

the patch passed

+1 :green_heart:
blanks
0m  1s

The patch has no blanks issues.

-0 :warning:
checkstyle
1m  4s
/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 454 unchanged - 0 fixed = 455 total (was 454)

+1 :green_heart:
mvnsite
1m 16s

the patch passed

+1 :green_heart:
javadoc
0m 56s

the patch passed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
1m 30s

the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

-1 :x:
spotbugs
3m 25s
/new-spotbugs-hadoop-hdfs-project_hadoop-hdfs.html
hadoop-hdfs-project/hadoop-hdfs generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0)

+1 :green_heart:
shadedclient
40m 32s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

-1 :x:
unit
238m  5s
/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs in the patch passed.

+1 :green_heart:
asflicense
0m 42s

The patch does not generate ASF License warnings.

393m  2s

Reason
Tests

SpotBugs
module:hadoop-hdfs-project/hadoop-hdfs

BlockInfo is incompatible with expected argument type ExcessRedundancyMap$ExcessBlockInfo in org.apache.hadoop.hdfs.server.blockmanagement.ExcessRedundancyMap.contains(DatanodeDescriptor, BlockInfo)  At ExcessRedundancyMap.java:argument type ExcessRedundancyMap$ExcessBlockInfo in org.apache.hadoop.hdfs.server.blockmanagement.ExcessRedundancyMap.contains(DatanodeDescriptor, BlockInfo)  At ExcessRedundancyMap.java:[line 70]

BlockInfo is incompatible with expected argument type ExcessRedundancyMap$ExcessBlockInfo in org.apache.hadoop.hdfs.server.blockmanagement.ExcessRedundancyMap.remove(DatanodeDescriptor, BlockInfo)  At ExcessRedundancyMap.java:argument type ExcessRedundancyMap$ExcessBlockInfo in org.apache.hadoop.hdfs.server.blockmanagement.ExcessRedundancyMap.remove(DatanodeDescriptor, BlockInfo)  At ExcessRedundancyMap.java:[line 105]

org.apache.hadoop.hdfs.server.blockmanagement.ExcessRedundancyMap$ExcessBlockInfo.equals(Object) checks for operand being a BlockInfo  At ExcessRedundancyMap.java:a BlockInfo  At ExcessRedundancyMap.java:[line 162]

Failed junit tests
hadoop.hdfs.server.datanode.TestDirectoryScanner

hadoop.hdfs.TestDFSUtil

Subsystem
Report/Notes

Docker
ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6176/10/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/6176

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint

uname
Linux b9af0ea38fc7 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19 13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 77c13421c0fd86b3007c6a1dc828862f090b3acf

Default Java
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6176/10/testReport/

Max. process+thread count
2218 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6176/10/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

Test failures seems unrelated.
However changes in the contains method and remove method input blockInfo and the equals method in ExcessBlockInfo will cause some SpotBugs problems.

Hi @ayushtkn @tomscut @xinglin Would you mind to also take a review this pr when you have free time? thank you very much~

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
17m 45s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+0 :ok:
detsecrets
0m  1s

detect-secrets was not available.

+0 :ok:
xmllint
0m  1s

xmllint was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 2 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
48m 14s

trunk passed

+1 :green_heart:
compile
1m 23s

trunk passed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
compile
1m 14s

trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
checkstyle
1m 14s

trunk passed

+1 :green_heart:
mvnsite
1m 24s

trunk passed

+1 :green_heart:
javadoc
1m  8s

trunk passed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
1m 38s

trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
spotbugs
3m 19s

trunk passed

+1 :green_heart:
shadedclient
40m 36s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
1m 11s

the patch passed

+1 :green_heart:
compile
1m 15s

the patch passed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javac
1m 15s

the patch passed

+1 :green_heart:
compile
1m  7s

the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
javac
1m  7s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
1m  5s

the patch passed

+1 :green_heart:
mvnsite
1m 15s

the patch passed

+1 :green_heart:
javadoc
0m 55s

the patch passed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
1m 29s

the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
spotbugs
3m 21s

the patch passed

+1 :green_heart:
shadedclient
40m  0s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

-1 :x:
unit
253m 43s
/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs in the patch passed.

+1 :green_heart:
asflicense
0m 46s

The patch does not generate ASF License warnings.

423m 55s

Reason
Tests

Failed junit tests
hadoop.hdfs.server.datanode.TestDirectoryScanner

hadoop.hdfs.TestRollingUpgrade

Subsystem
Report/Notes

Docker
ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6176/12/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/6176

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint

uname
Linux f4711f33a855 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 09eed20fc0df5c71731d279a2b516a4b8db4e6b6

Default Java
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6176/12/testReport/

Max. process+thread count
2835 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6176/12/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

The failed unit test seems unrelated to the change, local run test is normal.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 51s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+0 :ok:
xmllint
0m  1s

xmllint was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 2 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
47m 53s

trunk passed

+1 :green_heart:
compile
1m 24s

trunk passed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
compile
1m 15s

trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
checkstyle
1m 15s

trunk passed

+1 :green_heart:
mvnsite
1m 24s

trunk passed

+1 :green_heart:
javadoc
1m  8s

trunk passed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
1m 36s

trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
spotbugs
3m 22s

trunk passed

+1 :green_heart:
shadedclient
39m 54s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
1m 10s

the patch passed

+1 :green_heart:
compile
1m 15s

the patch passed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javac
1m 15s

the patch passed

+1 :green_heart:
compile
1m  7s

the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
javac
1m  7s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
1m  4s

the patch passed

+1 :green_heart:
mvnsite
1m 15s

the patch passed

+1 :green_heart:
javadoc
0m 55s

the patch passed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
1m 32s

the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
spotbugs
3m 20s

the patch passed

+1 :green_heart:
shadedclient
39m 29s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

-1 :x:
unit
253m 16s
/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs in the patch passed.

+1 :green_heart:
asflicense
0m 41s

The patch does not generate ASF License warnings.

404m 59s

Reason
Tests

Failed junit tests
hadoop.hdfs.server.datanode.TestDirectoryScanner

Subsystem
Report/Notes

Docker
ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6176/13/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/6176

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint

uname
Linux 67b6b0c7dd15 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / b7f9b236cd9e38657ac2b64d215ec688320c8f6b

Default Java
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6176/13/testReport/

Max. process+thread count
2471 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6176/13/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

Let's try to checkin while wait for two workdays if no more comments. Thanks.

Thanks @Hexiaoqiao for your review!

Thanks @Hexiaoqiao @zhangshuyan0 @ZanderXu for your review and merge.


keyboard focus is not visible on the settings dropdown and checkboxes present in the setting dropdown.

Search terms

Expected Behavior
keyboard focus should be visible on the 'settings' dropdown and on the checkboxes present in the setting dropdown.
Actual Behavior
While navigate to settings dropdown, keyboard focus is not visible on the 'settings' dropdown and on the checkboxes present in the setting dropdown.
https://github.com/TypeStrong/typedoc/assets/93735775/4e988358-a364-4190-9dc4-871fcbf1ddad
Steps to reproduce the bug

Open TypeDoc - v0.25.13 in edge browser.
Navigate to settings dropdown and observe the keyboard on it, and after invoking it observe the keyboard focus on the checkboxes.

Environment

Typedoc version:  0.25.13
TypeScript version:
Node.js version:
OS: 23H2 Build 22631.3527

Fixed with TypeDoc 0.26, which is releasing 2024/06/21


Not working

Hi, ive gotten the bot working on my discord server, but i cant interact with it in the server itself

Hi! Have you added the bot to your server?

Yes, i have. the bot was up and running but nothing

Also, did you run register.js first? You need to do that the first time to create the commands, and then index.js associates those previously-created commands with its code. If you didn't run register.js, the command won't show up.

I've gotten it working now. Was something on my end.
Now i do need to know how to create a timer :p

Good luck with that 😅


Can this be changed to debug?

We've been tuning our logging lately and noticed this was set to info but wondering if it could be changed to debug?
https://github.com/zendesk/ruby-kafka/blob/fff91eff27665b0650231432d48055a13f56c84e/lib/kafka/produce_operation.rb#L107

Wouldn't you want to be able to see that info when looking into past incidents?

Wouldn't you want to be able to see that info when looking into past incidents?

For us that message doesn't include enough information to be all that helpful in debugging. We tend to lean more on metrics (datadog) and error reporting (sentry) to discover and debug an incident.


Can't install via elm-package

I'm not able to install using elm-package install styled-components/elm-styled.
If I try that I'm getting this error:
Error: Could not find any packages named styled-components/elm-styled.

Here are some packages that have similar names:

    elm-community/elm-time
    seanhess/elm-style
    elm-community/elm-test
    poying/elm-style

Maybe you want one of those?

The package seems not to be listed in http://package.elm-lang.org/
Am I missing something?
I'm very new to Elm, so I apologise if I'm making some silly mistake.
I'm using elm version 0.18.0

I have not released v1 yet. I want to fix some minor issues first.

Ok, thanks ;)

So how we can install it?

It is not possible to install it via elm-package because of Native code. Not sure why there is elm-package command in README.
You can install it with elm-github-install or elm-grove. But you should know that Native code will be not allowed in upcoming Elm 0.19

Any update on this?


Choose index based on stats

After histogram implemented, support CBO index selection.

Index selection by CBO is supported since #244


Electric Dispositions Table

what's in here?
Added the disposition side of the electric energy account table! (the source side was in #2094). This entailed:

adding/editing the metadata to add a new table (resources & fields).
updating the settings files so this table is actually loaded 😎
making a new transformer class (ElectricEnergyAccountDispositionsFerc1TableTransformer)
only thing that needed to be special cased for the disposition table was the normalize_metadata_xbrl method. this is basically just to rename a column. I want to move this into the generic normalize_metadata_xbrl.... but the plant_in_service_ferc1 table kind needs the other name.
Added parameters. I mirrored the sources table for the types of transforms I needed. Used dict comprehension to generate the rename dictionary that adds the "value_types" to the end of all of the columns we want to end up behind energy_disposition_type's. Felt cleaner/easier.
The table transformed just fine with no other additions except it was failing on the enforce_schema step because of a PK failure. There was one energy_disposition_type that had data being reported with a start/end date in 2020. So I added a new transformer called select_date_range_duration_xbrl and added this into process_duration_xbrl. This is a type of transform which gets run on all of the tables, but if there are no parameters, no transformation will be applied.
Add the table & transformer into the main transform function.

questions....

are electric_energy_account_dispositions_ferc1 and electric_energy_account_sources_ferc1 ridiculous table names?!? its just way long. wtf does "disposition" mean in this context... it is where the energy goes. sources and outflows? it's not really end-use. idk.
(two others in comments)

PR Checklist
Before requesting a review of your pull request, please make sure you've done the
following:

[x] Merge the most recent version of dev (or the appropriate upstream branch) into
your branch and resolved any merge conflicts. You may need to do this several
times over the course of a PR as dev changes frequently.
[ ] Verify that all of the CI checks on your PR are passing. See
Running Tests with Tox
for details on how to run the full test suite locally if you need to debug a
particular failure.
[x] Ensure that the docstrings for any new modules, classes, functions, or methods are
descriptive enough for developers and users to understand your code.
[ ] If you expanded data coverage or changed the outputs, ensure that the full
data validation tests
pass locally on a fresh DB.
[x] If you've added new functions or classes, ensure that they have at least basic
unit tests.
[ ] If you've added new analyses, make sure they include defensive sanity checks that
will catch unexpected data issues.
[ ] Update the
release notes
to reflect your changes. Make sure to reference the PR and any related issues.
[x] Do your own review of the PR. Add comments highlighting areas where you have
questions you'd like reviewers to answer, known issues, solutions you're
unsatisfied with, or other things that deserve special attention from the
reviewer.

Also "sources" and "dispositions" seems to be a common language used to describe how assets (securities, commodities, etc) are acquired and gotten rid of by companies: https://www.investopedia.com/terms/d/disposition.asp
The EIA 176 is another "sources & dispositions" data source, but talking about natural gas.


Can't activate vector

I'm using an out of the box 2.0 vector. Never activated with ddl.
wire-pod is running on rpi 4.
I followed the instructions but my vector can't seem to connect to wire-pod.
Using a sniffer I can see no connection attempts at all from vector to wire-pod. It does connect to the Internet and pulls data.
It looks like an issue with mDNS. I pulled the logs from vector and see these:
vic-cloud: Token queue error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup escapepod.local: no such host"
avahi is running on the rpi4. I can resolve and ping escapepod.local fine from a client on the same wifi as Vector. I can also see using a sniffer that Vector sends mDNS requests for escapepod.local and I can see a reply from the rpi with an answer.
Vector's firmware is v<IP_ADDRESS>76
Any ideas?
Thanks.

Is wire-pod fully set up? It will only accept connections if it shows wire-pod started successfully! in its logs. May you send over all logs from wire-pod and from the bot?

Closing due to inactivity.


Add pre-commit rule to format .proto files.

protos are added in https://github.com/RespiraWorks/VentilatorSoftware/pull/125.

aha. so it falls back to file extensions.

It's not technically file extensions -- or at least, there's a level of indirection.
https://pre-commit.com/#filtering-files-with-types -> see link "tags by extension / naming convention"

Thanks for the review!


Only modify wp-config.php when WP_SITEURL is set

Do you know if it is possible to use wp cli to only do search ands replace on wp-config.php?
Actually this error saved me because I was testing pointing to a external staging db, but, using a custom non-existient hostname, if that error had not ocurred I would had messed up the database.
Some times is just enough to only set WP_SITEURL and WP_HOME to be able to access the site, so it would be a better approach to handle both cases separately (wp-config.php and database) and avoid potential mistakes.
WDYT ?

I think WPCLI goes directly only to the database. I think splitting it out could be useful. Something like a variable URL_UPDATE_MODE=none, db, file, both with some sort of default sounds like it would do the trick. I'm away for a week or so but can put in when I return!

New Environment variable: SITE_URL_UPDATE_MODE: DB FILE ALL NONE in place for version 5.2.0 for all branches now

Excellent, I will test them ASAP.
Thanks.

I found a nasty omission in the php-fpm initialization routines not properly filling in the post_max_size and upload_max_size - A new nginx-php-fpm image has been pushed and it won't have been reflected in the available wordpress image yet. It's lickity split on my own build systems but Docker Hub is very slow to queue up changes with descendent images. This issue should work without issues - just know that you'll want to pull again in a few days when Docker Hub finishes it's thing. I'm pretty much done with all my mass updates to images for the foreseeable future so I thank you for the volatile past week with changes.


Fix phone validation

Resolves #115 by making that case out of scope for StudMap
Resolves #123 by defining what makes a student unique

Codecov Report
Base: 57.58% // Head: 57.58% // No change to project coverage :thumbsup:

Coverage data is based on head (3cf917b) compared to base (af001fb).
Patch has no changes to coverable lines.

Additional details and impacted files
@@            Coverage Diff            @@
##             master     #200   +/-   ##
=========================================
  Coverage     57.58%   57.58%
  Complexity      655      655
=========================================
  Files           118      118
  Lines          2697     2697
  Branches        313      313
=========================================
  Hits           1553     1553
  Misses         1010     1010
  Partials        134      134

Impacted Files
Coverage Δ

.../main/java/seedu/studmap/model/student/Module.java
53.84% <ø> (ø)

...c/main/java/seedu/studmap/model/student/Phone.java
72.22% <ø> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


Loosen the find revision ID regex a bit to support more languages

If arc's language was set to a language other than English, it will fail to get the correct revision ID from commit message. For example, the Chinese version of arc will append something like Differential 修订: http://phabricator.x.com/D438 instead of Differential Revision: http://phabricator.x.com/D438.
This PR loosen the regex a little bit to make arc can work perfectly on more languages, if not all

https://secure.phabricator.com/book/phabcontrib/article/contributing_code/#no-pull-requests

@epriestley But I cannot login onto secure.phabricator.com
The external account ("Google") you just authenticated with is not configured to allow registration on this Phabricator install. An administrator may have recently disabled it., Github account doesnt work either

https://secure.phabricator.com/book/phabcontrib/article/contributing_code/#level-requirements

@epriestley does it mean we can't even have free basic access to track tasks on https://secure.phabricator.com/ as community members?


Shortcut Ctrl+W doesn't work in chrome

In Firefox I can set a keybind to CTRL+W to close a tab with '-M ctrl w -m ctrl`.
In Chrome, this doesn't seem to work.
Based on https://github.com/atx/wtype/issues/31 I had change the w to KEY_W.
However, now it seems to close the first tab, instead of the tab that is selected.
Not sure how to fix this and get the desired behavior.

wtype -M ctrl -k w -m ctrl works fine for me in chromium and brave


Adding a search input to scope tab

In scope Tab
1- Add a search input to developer can easily search for a specific controller.
2 -Add a search input to Models tab so developer can search for specific model.
I an angular app that a have a lot of controllers in the page and a lot of models in the scope of each
controller it is very hard to search for a specific model or scope
so i suggest to add a search input.

dupe of #46


Filename is too long while cloning repository in windows

go get -u github.com/microsoftgraph/msgraph-sdk-go-core
error: unable to create file msgraph-mail/java/utilities/src/main/java/graphjavav4/utilities/users/item/mailFolders/item/messages/item/multiValueExtendedProperties/item/MultiValueLegacyExtendedPropertyRequestBuilder.java: Filename too long
error: unable to create file msgraph-mail/java/utilities/src/main/java/graphjavav4/utilities/users/item/mailFolders/item/messages/item/singleValueExtendedProperties/item/SingleValueLegacyExtendedPropertyRequestBuilder.java: Filename too long
Unable to checkout 'b42e1a879cf62da7e666dcf38d4ab69357abba1d' in submodule path 'samples'****
Please provide alternative way to do this in windows platform.

Hi @nagnagendra
Thanks for trying out the SDK and for reaching out.
This is caused because the SDK depends on libraries that live in kiota, the generator used for this SDK, and this repo has a submodule linked to the samples repo. During the dependency resolution process, the Go package manager which depends on Git, also clones that submodule.
What's the path of your repo on disk? using a shorter root path should work around this issue.
Alternatively, you can enable long paths support on your system, see this (option 2, registry)
As we move towards the GA of the SDK we have a plan to move those dependencies onto their own dedicated repository which should solve this issue, see https://github.com/microsoft/kiota/issues/370
I've also issued a pull request to remove the submodule and make the update of the packages faster/easier.

Closing since https://github.com/microsoft/kiota/pull/1199 has been merged and the reference has been updated.


in pycharm, envfiles not loaded when running tests

Hi,
In pycharm 2022.2, envfile works as expected when running/debugging. However, when executing a test wiht pycharm's pytest runner it does not - no environment variables are loaded.

Idem here

idem here
Intellij 2022.2.4 (Ultimate edition)

PyCharm 2022.3 (Professional Edition)

I have the same issue. PyCharm 2022.3 Professional.
The documentation mentions something about items in italic working only in the paid version. The Django and Django test run targets are in italic. However, I could not find anything about how to buy the paid version of the plugin. @ashald I'd consider buying, but please share more instructions about that.

PyCharm  2022.2.3 (Professional Edition).

Same here. PyCharm 2022.3.1 (Professional Edition).

Same issue, PyCharm 2022.3.3 (Professional Edition).

Same issue, PyCharm 2022.3.3 (Professional Edition), but only when using "autodetect" for Python test configurations. Explicitly setting a test runner in Settings>Tools>Python Integrated Tools fixes the problem for me.

This solved the issue for me. Thanks!

Facing this. Does anyone know of a maintained alternative to this plugin? I wish pycharm could support this basic functionality that VSCode has had basically since it launched.

It seems specifying the .env file again in the Paths to ".env" files field for the autodetect template also helps. Still, would be nice if this could be fixed properly in EnvFile


Correct typos in config strings

Thanks for this very useful package! I happened to notice a couple of typos in the config page.
Luke

That's great! Thanks for that!


How to debug my job?

There are many modules in Saturn, an executor, job, console, the whole logic flow is that the executor read the jar file from its path and execut them.
What if I find some bug in my job and I want to set some breakpoint and debug it?  I read https://github.com/vipshop/Saturn/wiki/saturn开发指引之java but it does not make sense for debugging?

[Debug As] cannot work?

What do you mean debug as. I hope the flow can be begin in the very beginning.
But saturn needs a jar, my job is still in java format, how can I put break point and debug it? Would you please complete the guide?


About word alignment information

Hi, @yl4579 !
Thanks for the exciting work!
I have a question regarding inputs for your model.
In PnG BERT, the authors used the word position embedding as word alignment information. Also, they found that the information helped to improve the model's performance on MLM, G2P, and P2G tasks.
Predicting grapheme without considering word position could be challenging for the model (of course, you already proved that the model is stable and effective in your paper), but, in my understanding, your model only uses position embedding for phoneme tokens.
So, my question is below;

Why has word position embedding not been used for your model?
Did you have some experiments to find the difference whether using word position embedding?

Thanks!

For the first question, I do not see any reason to use word position embedding in the first place. The reason to use position embeddings in the transformer is that the input has no inherent position information. For example, in the sentence "A dog eats a hot dog," there are two "dogs", but they have different meanings, so you need to specify which dog is which, as the transformer has no inherent representation of the word positions (neither attention nor fully connection layers distinguish these two "dogs"). Since in PnG BERT, the grapheme (word) is an input, word positions are important for the model to deal with cases where some words might occur more than one time. On the other hand, in PL-BERT, the only input is the phoneme, so there is no need to specify the exact position of the word.
I haven't tried to experiment with using word position embedding, and I think the only information it actually provides is the word boundary (i.e., when a word ends in a sequence of phonemes), which is already provided in the input tokens of phonemes, so this information wouldn't be helpful.


contains-string-matcher.eo:31-33: Enable the matcher when...

The puzzle 218-bb16def5 from #218 has to be resolved:
https://github.com/objectionary/eo-hamcrest/blob/298bd4884878fbdcd1bac2a2aa35b22603bf18e4/src/main/eo/org/eolang/hamcrest/matchers/contains-string-matcher.eo#L31-L33
The puzzle was created by @levBagryansky on 17-Oct-23.
Estimate: 90 minutes,  role: DEV.
If you have any technical questions, don't ask me, submit new tickets instead. The task will be "done" when the problem is fixed and the text of the puzzle is removed from the source code. Here is more about PDD and about me.

@graur please pay attention to this new issue.


Convert remaining java classes to kotlin in shared module

Towards #532

@JSunde  PTAL?

Too many code analysis checks are failing due to bulk refactor. Closing this in favor of doing one at a time


Update banner style variants

Banner style variants include success (green), warning (red) and information (blue).

Add moj-banner--error and associate with govuk-colour("red")
Update moj-banner--warning to be govuk-colour("bright-red")

This will bring consistency with badges, tags and GOV.UK naming convention for "red".

We wouldn't have an moj-banner--error as this is catered for in the GOV.UK Design System.


Create Cloud Platform encoder module

Encoder SPI should provide pluggable interface which can be sued by service to decode incoming message regardless of the data format used. Also the same API can be used by Java clients to encode message before sending it to the protocol adapters or IoT connector.

Done - https://rhiot.gitbooks.io/rhiotdocumentation/content/cloudplatform/encoding/encoding.html .


Use hash join p2p

Symmetric change to https://github.com/dask/distributed/pull/7514 that will use a specialized HashJoin layer.
Not sure how to test this, yet. Suggestions welcome
TODO:

[x] This is not respecting the dask.config("shuffle") value

I'm using get_default_shuffle_algorithm now as well for the merge. with https://github.com/dask/dask/pull/9991 this would be enabled as well, be default

Since this is a very low impact change that has only an effect if one provides the keyword p2p and this is blocking https://github.com/dask/distributed/pull/7514 I will go ahead and merge this.
If there is anything wrong with it, I'll take care of it of course.


[Snyk] Upgrade @trustwallet/wallet-core from 3.0.1 to 3.1.9

This PR was automatically created by Snyk using the credentials of a real user.Snyk has created this PR to upgrade @trustwallet/wallet-core from 3.0.1 to 3.1.9.
:information_source: Keep your dependencies up-to-date. This makes it easier to fix existing vulnerabilities and to more quickly identify and fix newly disclosed vulnerabilities when they affect your project.

The recommended version is 17 versions ahead of your current version.
The recommended version was released a month ago, on 2023-01-19.

Release notes

Package name: @trustwallet/wallet-core

3.1.9 - 2023-01-19What's Changed
Breaking Change
ThorSwap EVM transactions are now using EIP-1559 tx mode by default. Use the gas parameters accordingly.
New features

⛓ TON: TON support by @ stspbu
⛓ WAX: Add WAX chain by @ miloserdow and @ Milerius
🖊 EIP1014: eip1014 support by @ Milerius

Improvements

⚡ ThorSwap: fee optimization for EVM chains by @ Milerius

Misc

✏ Misc: Rename Elrond to MultiversX (configuration change) by @ andreibancioiu
🔒 Misc: add SECURITY.MD by @ Milerius
🍎 Swift: Update Package.swift by @ Milerius

New Contributors

@ stspbu made their first contribution in #2813

Full Changelog: 3.1.8...3.1.9

3.1.8 - 2023-01-11What's Changed
New features

⛓ Immutable X: Complete Implementation by @ Milerius
⛓ Agoric: Add Agoric support by @ AyushBherwani1998
⛓ Native Injective: Add Native Injective support by @ AyushBherwani1998
⚡ ThorSwap: Add full Atom support by @ Milerius
⚡ ThorSwap: Add full Avax support by @ Milerius

Misc

💻 Docker Build: Update dockerfile by @ jlcordeiro
✏ Code smell: Change several classes to structs by @ phuang
✏ Code Improvements: Add extra hrp and coin check to lockScriptForAddress() by @ catenocrypt
🔖 Misc: update CODEOWNERS by @ Milerius

New Contributors

@ jlcordeiro made their first contribution in #2809

Full Changelog: 3.1.7...3.1.8

3.1.7 - 2022-12-16What's Changed
New features

⚡ ThorSwap: full DOGE/BCH/LTC support by @ Milerius

Misc

🧪 CI: Read GITHUB_USER without WC_ prefix by @ hewigovens
🧹 Refactoring: Unify Entry::deriveAddress() methods by @ catenocrypt
✏ Script: New ‘newevmchain’ script, subset of ‘newcoin’ by @ catenocrypt

Full Changelog: 3.1.6...3.1.7

3.1.6 - 2022-12-09What's Changed
Breaking change
Bitcode has been deprecated by apple: https://developer.apple.com/documentation/Xcode-Release-Notes/xcode-14-release-notes
"Starting with Xcode 14, bitcode is no longer required for watchOS and tvOS applications, and the App Store no longer accepts bitcode submissions from Xcode 14.
Xcode no longer builds bitcode by default and generates a warning message if a project explicitly enables bitcode: “Building with bitcode is deprecated. Please update your project and/or target settings to disable bitcode.” The capability to build with bitcode will be removed in a future Xcode release. IPAs that contain bitcode will have the bitcode stripped before being submitted to the App Store. Debug symbols can only be downloaded from App Store Connect / TestFlight for existing bitcode submissions and are no longer available for submissions made with Xcode 14. (86118779)"
You need to disable bitcode on all your targets if u use wallet-core >= 3.1.6.
New features

⚡ ThorSwap: ERC20 token transfer by @ Milerius
🤐 SecretNetwork: Support for Secret Network by @ toml01
🤝 Bindgen: Add rust wallet core rust library + Aptos blind signing by @ Milerius
⛓ Aptos: Add Staking/Unstaking through tortuga finance by @ Milerius

Fixes

📱 Swift: fix warning on public key declaration by @ Milerius
⚒ Fix: Fix error when uploading kotlin docs by @ miloserdow

Misc

☁ SonarCloud: run only on org branch by @ Milerius
🏗 Build: update bootstrap.sh by @ Milerius

Full Changelog: 3.1.4...3.1.6

3.1.5 - 2022-11-29

3.1.4 - 2022-11-25

3.1.3 - 2022-11-22

3.1.1 - 2022-11-21

3.1.0 - 2022-11-14

3.0.9 - 2022-11-04

3.0.8 - 2022-10-19

3.0.7 - 2022-10-13

3.0.6 - 2022-10-10

3.0.5 - 2022-10-03

3.0.4 - 2022-09-20

3.0.3 - 2022-09-15

3.0.2 - 2022-09-01

3.0.1 - 2022-08-04

    from @trustwallet/wallet-core GitHub release notes

Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open upgrade PRs.
For more information:
🧐 View latest project report
🛠 Adjust upgrade PR settings
🔕 Ignore this dependency or unsubscribe from future upgrade PRs

We'd be happy to look into this for you
kindly file a support ticket below
https://forms.gle/7KbQyuY5NoxyrhNz5


Audit the need for WANT_HAVEs and cut them off in case its not

More information in #3630

That's unrelated to #3630

Bitswap sends WANT_BLOCK for each sample to one peer and WANT_HAVE to other peers in the session as a back-up in case WANT_BLOCK is unsuccefull. The original worry behind this issue was that WANT_HAVE adds a roundtrip, which is not the case, so we can close this issue.


Rough implementation of GridSpec

Implements a GridSpec layout which allows declaring an NxM grid and assigning objects to the grid cells. Here is an example of a 3x3 grid:

[ ] Invert indexing coordinates (rows then columns)
[ ] Improve width/height setting logic
[ ] Fix vertical margin issues
[ ] Add unit tests
[ ] Add documentation

Wow, you are on a roll!

What happens when a grid cell is empty?

Would it be possible for the resulting grid to span the entire page area available (in a server context) or the entire width available (in a notebook context), and to responsively resize, with each item in a grid cell centered by default?  That would be an immediate decent-looking and usable dashboard with very little work for users.

What happens when a grid cell is empty?

Would it be possible for the resulting grid to span the entire page area available (in a server context) or the entire width available (in a notebook context), and to responsively resize, with each item in a grid cell centered by default?

I don't think this will work right now, but the new widget work at least theoretically makes that possible if all the other issues can be resolved.

Sounds good, thanks.

Thanks, I tried the bokeh GridSpec initially but it didn't do quite what we needed. That said I can't recall why, so I should go back before this is merged and compare/contrast the two implementations.

GridSpec was added for a very particular task in one of bokeh's examples (removed at this point), so refining it is an option, especially there are no tests for it and it wasn't every publicized properly.

Okay, I remember now. One of the main usecases for a GridSpec is to assign individual plots to multiple grid cells, the implementation currently in bokeh does not seem to allow for that. When assigning to multiple cells at once you have to provide a list of plots, so really it's just a different way of building the grid yourself and works nothing like matplotlib's GridSpec.

Sounds like once panel's GridSpec is solid, Bokeh's GridSpec should either be removed with a pointer to use Panel instead, or else panel's GridSpec should be split into a Bokeh-only implementation with a panel-specific extension on top (i.e. one that understands Panel objects, not just Bokeh models).

We should definitely consider that. One reason why it might be weird to do that is that this implementation of GridSpec combines both layout and sizing, i.e. inserting an object into the grid defines the size of the plot. This is what makes it possible and what it means to assign a plot/model to multiple grid cells. It may be weird to mess with the sizes of bokeh models directly like that if this were added to bokeh itself, but I'd be happy to file an issue on bokeh to start that discussion.


Where can I get surface normal images of blender dataset.

Hi, where can I get surface normal images of blender dataset.
The value of the surface normal image in the data accessible from the official github page seems to be an inaccurate value.
Thank you so much!

The predicted normal from the paper.

The GT normal from the data.

I believe these need to be mapped from [-1, 1] to [0, 1] by dividing by 2 and adding 0.5.

The predicted normal image is already mapped from [-1, 1] to [0, 1].

Check the ground truth.

The ground truth is a png file, so it ranges from 0 to 255. Even if I change the range from [0, 255] to [-1, 1], it does not work. Am I missing something? Thank you for your reply!

+) From this line, it means that the suffix of the normal image file is "_normal.png". However the suffix of the normal image from the official github page  is "_normal_0000.png".

Did you get this working?

No, it doesn't work... The GT normal image that I have seems to be different from the one you have.

This can happen if the normals are clipped before being exported, so you don't have the correct range on the normals. In this case, the ground truth normals have the wrong range.


prepping v1.9.0.dev0

Issue #, if available:
Description of changes:
prepping v1.9.0.dev0
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

AWS CodeBuild CI Report

CodeBuild project: IDFModuleTestsInfrastructur-59JoFoVH64CI
Commit ID: 25666c009cd785a85c6274e9a6b1f9bf585945dc
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository


Auth0 user delete script

Toggl: Sletning af inaktive brugere
Trello kort
Changes
This add's a script which allows us to delete users who hasn't been active in 5 years.
It should be fairly easy to use in a CRON job.
All the users to be deleted will be listed out.
They are not actually deleted until the --delete flag has been set on the command.
The call's to delete a user is throtlled as I've read that there's a request limit. I haven't had the time or the setup to test where the limit is, but this should be acceptable.
The script is now stand  alone
Testing the changes
The changes can be tested by running the following command in the root of the project:
node cleanUpUsers --domain  --clientId  --clientSecret
Don't test the deletion. I've done that, and it requires some fidling in the code.

Pretty sure the script does not work as intended. It seems the limit in getUsers is 50. All the users I get is with the last login in 2018, but there is also users with the last login in 2017.

we now find a 1000 users to delete. It's quite possible that there's more, but we simply don't know as that is as many users that Auth0 will let us get in one sitting

@ITViking har du ikke lavet nogle opdateringer til dette script under vores møde med Signe? Ville være fint at få det rettet og merged.

dem revertede jeg igen, fordi vi kun havde behov for dem da

rettelsen var at vi ikke to 5 år fra den dag, men fra da Signe havde trukket tallene for at få tallene til at stemme


error: implicit instantiation of undefined template 'std::__1::array<VkImageView_T *, 3>'

clang-10 complains:
                std::array<VkImageView, 3> attachments = {
                                           ^
/usr/include/c++/v1/__tuple:219:64: note: template is declared here
template <class _Tp, size_t _Size> struct _LIBCPP_TEMPLATE_VIS array;
                                                               ^
1 error generated.

Did you forget to include <array>?
Same in src/simulator/graphics/vulkan/descriptorSetLayout.cpp
Same in src/simulator/graphics/vulkan/renderPass.cpp

The current master branch code has been refactored and no longer generates this error when compiling with clang (tested with clang-12).
Thanks.