HDFS-17453. IncrementalBlockReport can have race condition with Edit Log Tailer

Description of PR
Summary
There is a race condition between IncrementalBlockReports (IBR) and EditLogTailer in Standby NameNode (SNN) which can lead to leaked IBRs and false corrupt blocks after HA Failover. The race condition occurs when the SNN loads the edit logs before it receives the block reports from DataNode (DN).
Example
In the following example there is a block (b1) with 3 generation stamps (gs1, gs2, gs3).

SNN1 loads edit logs for b1gs1 and b1gs2.
DN1 sends the IBR for b1gs1 to SNN1.
SNN1 will determine that the reported block b1gs1 from DN1 is corrupt and it will be queued for later. BlockManager.java

    BlockToMarkCorrupt c = checkReplicaCorrupt(
        block, reportedState, storedBlock, ucState, dn);
    if (c != null) {
      if (shouldPostponeBlocksFromFuture) {
        // If the block is an out-of-date generation stamp or state,
        // but we're the standby, we shouldn't treat it as corrupt,
        // but instead just queue it for later processing.
        // Storing the reported block for later processing, as that is what
        // comes from the IBR / FBR and hence what we should use to compare
        // against the memory state.
        // See HDFS-6289 and HDFS-15422 for more context.
        queueReportedBlock(storageInfo, block, reportedState,
            QUEUE_REASON_CORRUPT_STATE);
      } else {
        toCorrupt.add(c);
      }
      return storedBlock;
    }

DN1 sends IBR for b1gs2 and b1gs3 to SNN1.
SNN1 processes b1sg2 and updates the blocks map.
SNN1 queues b1gs3 for later because it determines that b1gs3 is a future genstamp.
SNN1 loads b1gs3 edit logs and processes the queued reports for b1.
SNN1 processes b1gs1 first and puts it back in the queue.
SNN1 processes b1gs3 next and updates the blocks map.
Later, SNN1 becomes the Active NameNode (ANN) during an HA Failover.
SNN1 will catch to the latest edit logs, then process all queued block reports to become the ANN.
ANN1 will process b1gs1 and mark it as corrupt.

If the example above happens for every DN which stores b1, then when the HA failover happens, b1 will be incorrectly marked as corrupt. This will be fixed when the first DN sends a FullBlockReport or an IBR for b1.
Logs from Active Cluster
I added the following logs to confirm this issue in an active cluster:
BlockToMarkCorrupt c = checkReplicaCorrupt(
    block, reportedState, storedBlock, ucState, dn);
if (c != null) {
  DatanodeStorageInfo storedStorageInfo = storedBlock.findStorageInfo(dn);
  LOG.info("Found corrupt block {} [{}, {}] from DN {}. Stored block {} from DN {}",
      block, reportedState.name(), ucState.name(), storageInfo, storedBlock, storedStorageInfo);
  if (storageInfo.equals(storedStorageInfo) &&
        storedBlock.getGenerationStamp() > block.getGenerationStamp()) {
    LOG.info("Stored Block {} from the same DN {} has a newer GenStamp." +
        storedBlock, storedStorageInfo);
  }
  if (shouldPostponeBlocksFromFuture) {
    // If the block is an out-of-date generation stamp or state,
    // but we're the standby, we shouldn't treat it as corrupt,
    // but instead just queue it for later processing.
    // Storing the reported block for later processing, as that is what
    // comes from the IBR / FBR and hence what we should use to compare
    // against the memory state.
    // See HDFS-6289 and HDFS-15422 for more context.
    queueReportedBlock(storageInfo, block, reportedState,
        QUEUE_REASON_CORRUPT_STATE);
    LOG.info("Queueing the block {} for later processing", block);
  } else {
    toCorrupt.add(c);
    LOG.info("Marking the block {} as corrupt", block);
  }
  return storedBlock;
}

Logs from nn1 (Active):
2024-04-03T03:00:52.524-0700,INFO,[IPC Server handler 6 on default port 443],org.apache.hadoop.hdfs.server.namenode.FSNamesystem,"updatePipeline(blk_66092666802_65700910634, newGS=65700925027, newLength=10485760, newNodes=[[DN1]:10010, [DN2]:10010, [DN3]:10010, client=client1)"
2024-04-03T03:00:52.539-0700,INFO,[IPC Server handler 6 on default port 443],org.apache.hadoop.hdfs.server.namenode.FSNamesystem,"updatePipeline(blk_66092666802_65700910634 => blk_66092666802_65700925027) success"
2024-04-03T03:01:07.413-0700,INFO,[IPC Server handler 6 on default port 443],org.apache.hadoop.hdfs.server.namenode.FSNamesystem,"updatePipeline(blk_66092666802_65700925027, newGS=65700933553, newLength=20971520, newNodes=[[DN1]:10010, [DN2]:10010, [DN3]:10010, client=client1)"
2024-04-03T03:01:07.413-0700,INFO,[IPC Server handler 6 on default port 443],org.apache.hadoop.hdfs.server.namenode.FSNamesystem,"updatePipeline(blk_66092666802_65700925027 => blk_66092666802_65700933553) success"

Logs from nn2 (Standby):
2024-04-03T03:01:23.067-0700,INFO,[Block report processor],org.apache.hadoop.hdfs.server.blockmanagement.BlockManager,"Found corrupt block blk_66092666802_65700925027 [FINALIZED, COMPLETE] from DN [DISK]DS-1:NORMAL:[DN1]:10010. Stored block blk_66092666802_65700933553 from DN null"
2024-04-03T03:01:23.067-0700,INFO,[Block report processor],org.apache.hadoop.hdfs.server.blockmanagement.BlockManager,"Queueing the block blk_66092666802_65700925027 for later processing"
2024-04-03T03:01:24.159-0700,INFO,[Block report processor],org.apache.hadoop.hdfs.server.blockmanagement.BlockManager,"Found corrupt block blk_66092666802_65700925027 [FINALIZED, COMPLETE] from DN [DISK]DS-3:NORMAL:[DN3]:10010. Stored block blk_66092666802_65700933553 from DN null"
2024-04-03T03:01:24.159-0700,INFO,[Block report processor],org.apache.hadoop.hdfs.server.blockmanagement.BlockManager,"Queueing the block blk_66092666802_65700925027 for later processing"
2024-04-03T03:01:24.159-0700,INFO,[Block report processor],org.apache.hadoop.hdfs.server.blockmanagement.BlockManager,"Found corrupt block blk_66092666802_65700925027 [FINALIZED, COMPLETE] from DN [DISK]DS-2:NORMAL:[DN2]:10010. Stored block blk_66092666802_65700933553 from DN null"
2024-04-03T03:01:24.159-0700,INFO,[Block report processor],org.apache.hadoop.hdfs.server.blockmanagement.BlockManager,"Queueing the block blk_66092666802_65700925027 for later processing"

Logs from nn2 when it transitions to Active:
2024-04-03T15:39:09.050-0700,INFO,[IPC Server handler 8 on default port 8020],org.apache.hadoop.hdfs.server.blockmanagement.BlockManager,"Found corrupt block blk_66092666802_65700925027 [FINALIZED, COMPLETE] from DN [DISK]DS-1:NORMAL:[DN1]:10010. Stored block blk_66092666802_65700933553 from DN [DISK]DS-1:NORMAL:[DN1]:10010"
2024-04-03T15:39:09.050-0700,INFO,[IPC Server handler 8 on default port 8020],org.apache.hadoop.hdfs.server.blockmanagement.BlockManager,"Stored Block blk_66092666802_65700933553 from the same DN [DISK]DS-1:NORMAL:[DN1]:10010 has a newer GenStamp."
2024-04-03T15:39:09.050-0700,INFO,[IPC Server handler 8 on default port 8020],org.apache.hadoop.hdfs.server.blockmanagement.BlockManager,"Marking the block blk_66092666802_65700925027 as corrupt"
2024-04-03T15:39:09.050-0700,INFO,[IPC Server handler 8 on default port 8020],org.apache.hadoop.hdfs.server.blockmanagement.BlockManager,"Found corrupt block blk_66092666802_65700925027 [FINALIZED, COMPLETE] from DN [DISK]DS-2:NORMAL:[DN2]:10010. Stored block blk_66092666802_65700933553 from DN [DISK]DS-2:NORMAL:[DN2]:10010"
2024-04-03T15:39:09.050-0700,INFO,[IPC Server handler 8 on default port 8020],org.apache.hadoop.hdfs.server.blockmanagement.BlockManager,"Stored Block blk_66092666802_65700933553 from the same DN [DISK]DS-2:NORMAL:[DN2]:10010 has a newer GenStamp."
2024-04-03T15:39:09.050-0700,INFO,[IPC Server handler 8 on default port 8020],org.apache.hadoop.hdfs.server.blockmanagement.BlockManager,"Marking the block blk_66092666802_65700925027 as corrupt"
2024-04-03T15:39:09.050-0700,INFO,[IPC Server handler 8 on default port 8020],org.apache.hadoop.hdfs.server.blockmanagement.BlockManager,"Found corrupt block blk_66092666802_65700925027 [FINALIZED, COMPLETE] from DN [DISK]DS-3:NORMAL:[DN3]:10010. Stored block blk_66092666802_65700933553 from DN [DISK]DS-3:NORMAL:[DN3]:10010"
2024-04-03T15:39:09.050-0700,INFO,[IPC Server handler 8 on default port 8020],org.apache.hadoop.hdfs.server.blockmanagement.BlockManager,"Stored Block blk_66092666802_65700933553 from the same DN [DISK]DS-3:NORMAL:[DN3]:10010 has a newer GenStamp."
2024-04-03T15:39:09.050-0700,INFO,[IPC Server handler 8 on default port 8020],org.apache.hadoop.hdfs.server.blockmanagement.BlockManager,"Marking the block blk_66092666802_65700925027 as corrupt"

How was this patch tested?
For code changes:

[X] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
[ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

I am planning to add another unit test that simulates this race condition.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 32s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

-1 :x:
test4tests
0m  0s

The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
45m 15s

trunk passed

+1 :green_heart:
compile
1m 20s

trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1

+1 :green_heart:
compile
1m 16s

trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06

+1 :green_heart:
checkstyle
1m  9s

trunk passed

+1 :green_heart:
mvnsite
1m 23s

trunk passed

+1 :green_heart:
javadoc
1m  5s

trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1

+1 :green_heart:
javadoc
1m 40s

trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06

+1 :green_heart:
spotbugs
3m 28s

trunk passed

+1 :green_heart:
shadedclient
36m 24s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
1m  9s

the patch passed

+1 :green_heart:
compile
1m 13s

the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1

+1 :green_heart:
javac
1m 13s

the patch passed

+1 :green_heart:
compile
1m  9s

the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06

+1 :green_heart:
javac
1m  9s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
1m  5s

the patch passed

+1 :green_heart:
mvnsite
1m 17s

the patch passed

+1 :green_heart:
javadoc
0m 52s

the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1

+1 :green_heart:
javadoc
1m 35s

the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06

+1 :green_heart:
spotbugs
3m 33s

the patch passed

+1 :green_heart:
shadedclient
37m 20s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

-1 :x:
unit
232m 19s
/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs in the patch passed.

+1 :green_heart:
asflicense
0m 47s

The patch does not generate ASF License warnings.

376m 37s

Reason
Tests

Failed junit tests
hadoop.hdfs.server.blockmanagement.TestPendingDataNodeMessages

Subsystem
Report/Notes

Docker
ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6708/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/6708

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets

uname
Linux b3e715c91d4a 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 7556dbe46d7e3e7a555110b58756785755e55b32

Default Java
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6708/1/testReport/

Max. process+thread count
4659 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6708/1/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 28s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 1 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
45m 30s

trunk passed

+1 :green_heart:
compile
1m 22s

trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1

+1 :green_heart:
compile
1m 15s

trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06

+1 :green_heart:
checkstyle
1m 17s

trunk passed

+1 :green_heart:
mvnsite
1m 21s

trunk passed

+1 :green_heart:
javadoc
1m  7s

trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1

+1 :green_heart:
javadoc
1m 41s

trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06

+1 :green_heart:
spotbugs
3m 16s

trunk passed

+1 :green_heart:
shadedclient
36m 52s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
1m 15s

the patch passed

+1 :green_heart:
compile
1m 17s

the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1

+1 :green_heart:
javac
1m 17s

the patch passed

+1 :green_heart:
compile
1m 14s

the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06

+1 :green_heart:
javac
1m 14s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

-0 :warning:
checkstyle
1m  3s
/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 86 unchanged - 0 fixed = 87 total (was 86)

+1 :green_heart:
mvnsite
1m 17s

the patch passed

+1 :green_heart:
javadoc
0m 51s

the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1

+1 :green_heart:
javadoc
1m 34s

the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06

+1 :green_heart:
spotbugs
3m 18s

the patch passed

+1 :green_heart:
shadedclient
37m 52s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

-1 :x:
unit
231m 45s
/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs in the patch passed.

+1 :green_heart:
asflicense
0m 46s

The patch does not generate ASF License warnings.

377m 44s

Reason
Tests

Failed junit tests
hadoop.hdfs.server.blockmanagement.TestPendingDataNodeMessages

Subsystem
Report/Notes

Docker
ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6708/2/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/6708

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets

uname
Linux bf6537361823 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 96cbeddc700d96b57c1d818a173657a91efcc8cc

Default Java
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6708/2/testReport/

Max. process+thread count
4458 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6708/2/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@goiri I have added the unit test which reproduces the exact race condition. I confirmed this by testing the unit test against a branch without the fix and caught the false corrupt replicas.

@sodonnel Could you take a look at my PR. I think it addresses the issue you mentioned here https://issues.apache.org/jira/browse/HDFS-15422?focusedCommentId=17287194&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17287194

@kihwal Could you take a look at my PR? I think it addresses the issue you mentioned here https://issues.apache.org/jira/browse/HDFS-14941?focusedCommentId=17140156&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17140156

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 31s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 2 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
47m 20s

trunk passed

+1 :green_heart:
compile
1m 22s

trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1

+1 :green_heart:
compile
1m 16s

trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06

+1 :green_heart:
checkstyle
1m 10s

trunk passed

+1 :green_heart:
mvnsite
1m 29s

trunk passed

+1 :green_heart:
javadoc
1m 15s

trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1

+1 :green_heart:
javadoc
1m 42s

trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06

+1 :green_heart:
spotbugs
3m 21s

trunk passed

+1 :green_heart:
shadedclient
38m 30s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
1m 13s

the patch passed

+1 :green_heart:
compile
1m 15s

the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1

+1 :green_heart:
javac
1m 15s

the patch passed

+1 :green_heart:
compile
1m  8s

the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06

+1 :green_heart:
javac
1m  8s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

-0 :warning:
checkstyle
0m 59s
/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs-project/hadoop-hdfs: The patch generated 6 new + 86 unchanged - 0 fixed = 92 total (was 86)

+1 :green_heart:
mvnsite
1m 20s

the patch passed

+1 :green_heart:
javadoc
0m 55s

the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1

+1 :green_heart:
javadoc
1m 35s

the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06

+1 :green_heart:
spotbugs
3m 28s

the patch passed

+1 :green_heart:
shadedclient
39m  2s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

-1 :x:
unit
237m  9s
/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs in the patch passed.

+1 :green_heart:
asflicense
0m 42s

The patch does not generate ASF License warnings.

387m 32s

Reason
Tests

Failed junit tests
hadoop.hdfs.server.mover.TestMover

hadoop.hdfs.server.blockmanagement.TestPendingDataNodeMessages

hadoop.hdfs.server.common.blockaliasmap.impl.TestLevelDbMockAliasMapClient

hadoop.hdfs.server.diskbalancer.command.TestDiskBalancerCommand

hadoop.hdfs.server.diskbalancer.TestDiskBalancerWithMockMover

hadoop.hdfs.server.namenode.ha.TestDNFencing

Subsystem
Report/Notes

Docker
ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6708/3/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/6708

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets

uname
Linux 43af3362757d 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / d637a41658748bc01a7bf5f821b105449de82959

Default Java
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6708/3/testReport/

Max. process+thread count
4400 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6708/3/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 57s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+0 :ok:
detsecrets
0m  1s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 2 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
44m 58s

trunk passed

+1 :green_heart:
compile
1m 23s

trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1

+1 :green_heart:
compile
1m 15s

trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06

+1 :green_heart:
checkstyle
1m 11s

trunk passed

+1 :green_heart:
mvnsite
1m 26s

trunk passed

+1 :green_heart:
javadoc
1m  8s

trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1

+1 :green_heart:
javadoc
1m 45s

trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06

+1 :green_heart:
spotbugs
3m 21s

trunk passed

+1 :green_heart:
shadedclient
35m 54s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
1m 10s

the patch passed

+1 :green_heart:
compile
1m 14s

the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1

+1 :green_heart:
javac
1m 14s

the patch passed

+1 :green_heart:
compile
1m  7s

the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06

+1 :green_heart:
javac
1m  7s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

-0 :warning:
checkstyle
0m 58s
/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs-project/hadoop-hdfs: The patch generated 6 new + 86 unchanged - 0 fixed = 92 total (was 86)

+1 :green_heart:
mvnsite
1m 15s

the patch passed

+1 :green_heart:
javadoc
0m 53s

the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1

+1 :green_heart:
javadoc
1m 34s

the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06

+1 :green_heart:
spotbugs
3m 17s

the patch passed

+1 :green_heart:
shadedclient
36m  3s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

-1 :x:
unit
282m 26s
/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs in the patch passed.

+1 :green_heart:
asflicense
0m 48s

The patch does not generate ASF License warnings.

425m  8s

Reason
Tests

Failed junit tests
hadoop.hdfs.server.blockmanagement.TestPendingDataNodeMessages

hadoop.hdfs.server.datanode.TestBlockReplacement

hadoop.hdfs.server.namenode.ha.TestDNFencing

hadoop.hdfs.TestRollingUpgrade

Subsystem
Report/Notes

Docker
ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6708/4/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/6708

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets

uname
Linux a59a5c1df2b8 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 4ec0e6c6f1d16b4cbd6e0e4bc9203d73bb67613d

Default Java
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6708/4/testReport/

Max. process+thread count
3725 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6708/4/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 30s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+0 :ok:
detsecrets
0m  1s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 3 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
44m 33s

trunk passed

+1 :green_heart:
compile
1m 23s

trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1

+1 :green_heart:
compile
1m 18s

trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06

+1 :green_heart:
checkstyle
1m 26s

trunk passed

+1 :green_heart:
mvnsite
1m 19s

trunk passed

+1 :green_heart:
javadoc
1m  8s

trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1

+1 :green_heart:
javadoc
1m 40s

trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06

+1 :green_heart:
spotbugs
3m 19s

trunk passed

+1 :green_heart:
shadedclient
37m 31s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
1m  9s

the patch passed

+1 :green_heart:
compile
1m 10s

the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1

+1 :green_heart:
javac
1m 10s

the patch passed

+1 :green_heart:
compile
1m  3s

the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06

+1 :green_heart:
javac
1m  3s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
0m 57s

the patch passed

+1 :green_heart:
mvnsite
1m 12s

the patch passed

+1 :green_heart:
javadoc
0m 50s

the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1

+1 :green_heart:
javadoc
1m 30s

the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06

+1 :green_heart:
spotbugs
3m 11s

the patch passed

+1 :green_heart:
shadedclient
37m 18s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

-1 :x:
unit
231m 42s
/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs in the patch passed.

+1 :green_heart:
asflicense
0m 43s

The patch does not generate ASF License warnings.

375m 51s

Reason
Tests

Failed junit tests
hadoop.hdfs.server.datanode.TestBlockReplacement

hadoop.hdfs.server.blockmanagement.TestPendingDataNodeMessages

Subsystem
Report/Notes

Docker
ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6708/5/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/6708

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets

uname
Linux 292dbe764264 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 91a7973c8f68e79476deb1cb956bf8b9d663b972

Default Java
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6708/5/testReport/

Max. process+thread count
4446 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6708/5/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
11m 37s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+0 :ok:
detsecrets
0m  1s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 3 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
46m  2s

trunk passed

+1 :green_heart:
compile
1m 22s

trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1

+1 :green_heart:
compile
1m 14s

trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06

+1 :green_heart:
checkstyle
1m 12s

trunk passed

+1 :green_heart:
mvnsite
1m 26s

trunk passed

+1 :green_heart:
javadoc
1m  9s

trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1

+1 :green_heart:
javadoc
1m 41s

trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06

+1 :green_heart:
spotbugs
3m 18s

trunk passed

+1 :green_heart:
shadedclient
35m 44s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
1m 13s

the patch passed

+1 :green_heart:
compile
1m 15s

the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1

+1 :green_heart:
javac
1m 15s

the patch passed

+1 :green_heart:
compile
1m  7s

the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06

+1 :green_heart:
javac
1m  7s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
0m 58s

the patch passed

+1 :green_heart:
mvnsite
1m 15s

the patch passed

+1 :green_heart:
javadoc
0m 53s

the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1

+1 :green_heart:
javadoc
1m 37s

the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06

+1 :green_heart:
spotbugs
3m 17s

the patch passed

+1 :green_heart:
shadedclient
35m 59s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
230m  8s

hadoop-hdfs in the patch passed.

+1 :green_heart:
asflicense
0m 46s

The patch does not generate ASF License warnings.

384m 30s

Subsystem
Report/Notes

Docker
ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6708/8/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/6708

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets

uname
Linux 3d8c054a2c10 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 7e36db7affc66da101cfda141c9f35afc82aa606

Default Java
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6708/8/testReport/

Max. process+thread count
4658 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6708/8/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@dannytbecker  thanks for your work, I'm just wondering what's the case of three GS with one block


Amplitude analytics-browser 2.11.0 with autocapture: true freezes on click on dropdown menu

Expected Behavior
Not freeze and capture the interaction
Current Behavior
We recently upgraded from:
    "@amplitude/analytics-browser": "^2.9.3",
    "@amplitude/plugin-session-replay-browser": "^1.6.5",

to
    "@amplitude/analytics-browser": "^2.11.0",
    "@amplitude/plugin-session-replay-browser": "^1.6.21",

and enabled autocapture. But it broken one of our dropdown menu select and had to revert back to previous version. Essentially when a user clicked on a dropdown menu it would just freeze any interaction like JS thread was in an infinite loop. Does anyone else had this problem?
Possible Solution
downgrade to ^2.9.3
Steps to Reproduce
I haven't tried to reproduce it on a clean repo with just amplitude enabled as that's a lot of work. Just trying to understand if anyone else is having this issue. Can do further digging if needed.
Environment

SDK Version: mentioned above
Node version: v20.13.1
Chromium Engine Version 127.0.6533.120

@mjsilva - A similar issue was fixed in 2.11.1. Can you please try that version of the SDK to see if it resolves your problem?


Ingress - Update to CruResrouce and  Client Side Validation

Update ConfigMap to CruResource Flow
Labels and Annotations within Resources

Move to Vertical Tabs
1 tab name - "Labels & Annotations"
Inside the tab, use the LabelsAndAnnotations component, which should cause the labels to be stacked on top of the annotations and it should be full width.

Add in client side validation based on changes from adding ability to do it int this PR to have friendlier error messages.

Verified with Rancher server rancher\rancher:master-head version b5e1ce703
Ingress create has moved to vertical tabs:

Client side validation is added:


Adding parameters to memory cells

I am writing to ask about the possibility of using CiMLoop to simulate DRAM/eDRAM with specific parameters such as retention time and refresh period.
It appears that these parameters are not currently included in the memory cells YAML files.
Could you please advise me on how or where I might add such parameters to CiMLoop?
Thank you very much for your assistance.

My understanding is that for eDRAM with a set retention time & refresh period, the critical consideration would be the power spent on refreshing devices (please correct me if this understanding is incorrect).
To account for this in CiMLoop, you can create a primitive or compound component (tutorial here https://github.com/Accelergy-Project/timeloop-accelergy-exercises) that uses these parameters to calculate the power spent on refresh. I would put this power in the "leak" action for the component. If you'd like to power-gate unused cells (i.e., if you don't use a particular eDRAM row, don't refresh it), then you can use the has_power_gating attribute as described in https://timeloop.csail.mit.edu/v4/input-formats/design/architecture


[DOC] Update inference readme

📌 Checklist before creating the PR

[x] I have created an issue for this PR for traceability
[x] The title follows the standard format: [doc/gemini/tensor/...]: A concise description
[ ] I have added relevant tags if possible for us to better distinguish different PRs

🚨 Issue number

Link this PR to your issue with words like fixed to automatically close the linked issue upon merge
e.g. fixed #1234, closed #1234, resolved #1234

closes #5271
📝 What does this PR do?

Summarize your work here.
if you have any plots/diagrams/screenshots/tables, please attach them here.
Increment for design and usage

💥 Checklist before requesting a review

[x] I have linked my PR to an issue (instruction)
[x] My issue clearly describes the problem/feature/proposal, with diagrams/charts/table/code if possible
[ ] I have performed a self-review of my code
[ ] I have added thorough tests.
[ ] I have added docstrings for all the functions/methods I implemented

⭐ Do you enjoy contributing to Colossal-AI?

[x] 🌝 Yes, I do.
[ ] 🌚 No, I don't.

Tell us more if you don't enjoy contributing to Colossal-AI.

Figs are not ready, to be added

Figs are not ready, to be added

READY NOW ! 😄


ROX-12948 add port pool in DeploymentExposure tests

Description
Changes the tests to retry with a different port if applying the service fails due to the port being already allocated.
Checklist

[ ] Investigated and inspected CI test results
~[ ] Unit test and regression tests added~
~[ ] Evaluated and added CHANGELOG entry if required~
~[ ] Determined and documented upgrade steps~
~[ ] Documented user facing changes (create PR based on openshift/openshift-docs and merge into rhacs-docs)~

If any of these don't apply, please comment below.
Testing Performed

Tested with a local cluster and an infra cluster. Test passed even with all the services having a port collision.

/retest

/retest

/test gke-nongroovy-e2e-tests

/test gke-nongroovy-e2e-tests

This failure was unrelated to the changes; re-triggering to make sure the changed tests pass.

/test gke-nongroovy-e2e-tests

/test gke-nongroovy-e2e-tests


Resize-images doesn't work with gzip option

[REQUIRED] Step 2: Describe your configuration

Extension name: storage-resize-images
Extension version: 0.1.10
Extension configuration: default

[REQUIRED] Step 3: Describe the problem
When I tried to upload images with enabled gzip option they were getting resized but I couldn't load them in a browser. Link was broken and gave me domain network error.
Steps to reproduce:
Upload image trough the admin-sdk with gzip enabled. Generate link and open it in a browser.
Expected result
Actual result

Hey @ghert, thank you for bringing this to our attention. You're correct, it does indeed cause the resized images to error. I think the FE team will probably review this internally to figure out how/if it is something to be handled as part of the extension.

As the PR suggests, we're ignoring gzipped images.


MAINTAINERS: Add MaochenWang1 as a collaborator to Wi-Fi

Have contributed regularly to Wi-Fi, please see following: https://github.com/zephyrproject-rtos/zephyr/pulls?q=is%3Apr+is%3Aclosed+author%3AMaochenWang1

Thanks @MaochenWang1 , but you don't seem to be yet registered as contributor to the project? Please create a "Contributor Nomination" issue for that. https://docs.zephyrproject.org/latest/project/project_roles.html

Hi @aescolar  thanks for the remind.
Please see this nomination issue: https://github.com/zephyrproject-rtos/zephyr/issues/81759
Then how to handle this?


CodeQL Action not in the Marketplace?

I was surprised when I searched for this action it wasn't in the marketplace:
https://github.com/marketplace?type=actions&query=codeql-action+

Are the GitHub authored Actions special cased and included by default? I think it could make some of these Actions more discoverable if they were searchable in the marketplace.
Or did I use the incorrect search term?

Hi,
that's curious! I'll investigate this with the team, and report back. Surely there are some actions from us in the marketplace: https://github.com/marketplace?type=&verification=&query=publisher%3Agithub+

Okay, I got a quick response: The codeql-action technically consists of 4 actions, and these can't be bundled on the marketplace.
We see little value in offering 4 different actions "CodeQL: Init", "CodeQL: Analyze" etc, as they only make sense when used together.
Thus, as long as the marketplace doesn't support releasing groups of actions, codeql-action will not be in the marketplace.


Include shasum  checksums

Please include all shasum checksums hashes for every file on the latest releases page
so  there is more confidence from the community to install the files.

I started researching how to include PGP signatures on all the binaries to help provide this additional confidence. I'll close this one out as soon as that's implemented.

We also just realized that within the yaml files packaged for release checking, a sha512 checksum is included with every build. They can be found in the following files:

latest.yml (windows)
latest-linux.yml
latest-mac.yml

The binaries themselves do resolve to these versions:
$ shasum -b -a 512 eos-voter-0.1.5.dmg
99899ae0dbc64bf5d0392511949881883648ecc89169eb7220630ec9ac89890d18109006a55651cbd5895fe165efadefc79b8143acc61b764efc907b4509a136 *eos-voter-0.1.5.dmg


AWS S3 folder copy not working

Bug Report
The folder copy feature for AmazonS3-PUSH scenario, controlled via objectPrefix property is throwing an exception within our dataplane (running tractus-x version 0.7.2).
Describe the Bug
The upload into the consumer bucket is failing for folder copies. While at the same time single files (specified by objectName) are working fine for the same bucket.
The scenario is setup as the following and can be reproduced.
Asset:
{
  "@context" : {
    "@vocab" : "https://w3id.org/edc/v0.0.1/ns/"
  },
  "@type" : "https://w3id.org/edc/v0.0.1/ns/Asset",
  "@id" : "5db242a2-ee41-4c97-ae2d-f5944b123931",
  "https://w3id.org/edc/v0.0.1/ns/properties" : {
    "name" : "DSI-Demo-Asset",
    "description" : "Asset for demonstration purposes",
    "version" : "0.6.0",
    "contenttype" : "application/json",
    "additionalDescription" : "DSI Demo Asset"
  },
  "https://w3id.org/edc/v0.0.1/ns/privateProperties" : {
    "privateKey" : "privateValue - This field is used to simulate a private property"
  },
  "https://w3id.org/edc/v0.0.1/ns/dataAddress" : {
    "@type" : "DataAddress",
    "type" : "AmazonS3",
    "name" : "DataAddressS3",
    "region" : "eu-central-1",
    "bucketName" : "dsibucket-dev-provider-001",
    "accessKeyId" : "AKIA47CRXF32HAGFF35F",
    "secretAccessKey" : "<<masked>>",
    "objectPrefix" : "testFolder"
  }
}

Transferprocess:
{
    "@context": {
        "@vocab": "https://w3id.org/edc/v0.0.1/ns/"
    },
    "@type": "TransferRequest",
    "assetId": "5db242a2-ee41-4c97-ae2d-f5944b123931",
    "contractId": "388e8680-32a8-40c6-9c91-6d67401a62d8",
    "counterPartyAddress": "https://foss-edc-test.c-139b975.stage.kyma.ondemand.com/api/v1/dsp",
    "dataDestination": {
        "accessKeyId": "AKIA47CRXF32JPT5VTNI",
        "bucketName": "dsibucket-dev-consumer-001",
        "folderName": "testFolder",
        "privateKey": "privateValue - This field is used to simulate a private property",
        "region": "eu-central-1",
        "secretAccessKey": "<<masked>>",
        "type": "AmazonS3"
    },
    "privateProperties": {
        "privateKey": "privateValue - This field is used to simulate a private property"
    },
    "protocol": "dataspace-protocol-http",
    "transferType": "AmazonS3-PUSH"
}

Expected Behavior
On consumer bucket I would expect the contents of the provider folder testFolder to be copied into testFolder/testFolder
Observed Behavior
The upload does not happen at all. In the dataplane logs I can see the failure listed below.
Debugging the issue on provider side's dataplane within S3DataSink, I see that there is an individual part for the provider folder testFolder which has bytesChunk.length = 0.

The S3DataSink code generates an uploadId and without transferring a single chunk it tries to complete the MultipartUpload. This is failing with the following error message:
SEVERE 2024-07-24T14:17:52.37640665 Failed to upload the testFolder/testFolder/ object: The XML you provided was not well-formed or did not validate against our published schema (Service: S3, Status Code: 400, Request ID: YGYWJBB3Z951R41R, Extended Request ID: P2KQ+nsZtCSZFZ+2OYUGSRTGefMbWP6xQ5j7LiXYVU4fS0z0kfZK2IhnDUGjz1QsfZh8WA3M5RWrdLmLP4b0kA==)
software.amazon.awssdk.services.s3.model.S3Exception: The XML you provided was not well-formed or did not validate against our published schema (Service: S3, Status Code: 400, Request ID: YGYWJBB3Z951R41R, Extended Request ID: P2KQ+nsZtCSZFZ+2OYUGSRTGefMbWP6xQ5j7LiXYVU4fS0z0kfZK2IhnDUGjz1QsfZh8WA3M5RWrdLmLP4b0kA==)
	at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleErrorResponse(AwsXmlPredicatedResponseHandler.java:156)
	at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleResponse(AwsXmlPredicatedResponseHandler.java:108)
	at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:85)
	at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:43)
	at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler$Crc32ValidationResponseHandler.handle(AwsSyncClientHandler.java:93)
	at software.amazon.awssdk.core.internal.handler.BaseClientHandler.lambda$successTransformationResponseHandler$7(BaseClientHandler.java:279)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:50)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:38)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:72)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:78)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:40)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:55)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:39)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:81)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:36)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56)
	at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:50)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:32)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
	at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:224)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:103)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:173)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:80)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:182)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:74)
	at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
	at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:53)
	at software.amazon.awssdk.services.s3.DefaultS3Client.completeMultipartUpload(DefaultS3Client.java:731)
	at org.eclipse.edc.connector.dataplane.aws.s3.S3DataSink.transferParts(S3DataSink.java:75)
	at org.eclipse.edc.connector.dataplane.util.sink.ParallelSink.lambda$transfer$3(ParallelSink.java:82)
	at org.eclipse.edc.spi.telemetry.Telemetry.lambda$contextPropagationMiddleware$2(Telemetry.java:102)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
DEBUG 2024-07-24T14:17:52.377924963 [DataPlaneManagerImpl] DataFlow 1a3e9729-8d19-43ed-b9e3-5dd0c2917b11 is now in state FAILED

I guess this happens because completedParts  does not contain a single entry which is considered as invalid request by AWS (The XML you provided was not well-formed or did not validate against our published schema (Service: S3, Status Code: 400...). As the folder part always comes as the initial part, the full copy process is aborted.

This usually happens when someone creates an empty folder / directory in S3 and then uploads some file into same folder / directory. In that case list-objects-v2 API returns two object, one folder with size 0 and another file with its equal size.
When S3DataSink#transferParts() tries to upload an object with size 0, it fails with above exception.
How to reproduce via AWS CLI

Upload an empty folder inside a bucket.

aws s3api put-object --bucket dsibucket-dev-consumer-001 --key testfolder1/

Upload a file into same folder.

aws s3api put-object --bucket dsibucket-dev-consumer-001 --key testfolder1/10mb.txt --body ./10mb.txt

List Objects in the folder.

aws s3api list-objects-v2 --bucket dsibucket-dev-consumer-001 --prefix testfolder1/

Response:
{
    "Contents": [
        {
            "Key": "testfolder1/",
            "LastModified": "2024-07-25T06:14:42+00:00",
            "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
            "Size": 0,
            "StorageClass": "STANDARD"
        },
        {
            "Key": "testfolder1/10mb.txt",
            "LastModified": "2024-07-25T06:16:34+00:00",
            "ETag": "\"2d94c9f3cbfa5fbc410a7a8b72f8cee1\"",
            "Size": 10485772,
            "StorageClass": "STANDARD"
        }
    ],
    "RequestCharged": null
}

If file is directly uploaded without creating folder first, it doesn't return folder in the list of objects.
aws s3api put-object --bucket dsibucket-dev-consumer-001 --key testfolder2/10mb.txt --body ./10mb.txt
aws s3api list-objects-v2 --bucket dsibucket-dev-consumer-001 --prefix testfolder2/

Response:
{
    "Contents": [
        {
            "Key": "testfolder2/10mb.txt",
            "LastModified": "2024-07-25T06:17:37+00:00",
            "ETag": "\"2d94c9f3cbfa5fbc410a7a8b72f8cee1\"",
            "Size": 10485772,
            "StorageClass": "STANDARD"
        }
    ],
    "RequestCharged": null
}

Proposed Solution
In S3DataSource#openPartStream() method, filter any S3 objects based on any one of the below criteria.

FIlter out if object has size 0.
FIlter out if Key is same as prefix. i.e. skip the empty folder object which was created first before file being uploaded into it.
https://github.com/eclipse-edc/Technology-Aws/blob/562c56859089e6522c396acf6011d6197760768d/extensions/data-plane/data-plane-aws-s3/src/main/java/org/eclipse/edc/connector/dataplane/aws/s3/S3DataSource.java#L65-L76

Reference
https://stackoverflow.com/a/75620490

Even if the multipart upload worked for a 0 byte object, the key name for the folder part is wrongly resolved as testFolder/testFolder/ by the getDestinationObjectName() function (you can see the key var value in the debug print).
As for the proposed solution by @hemantxpatel, i do agree filtering the empty folder object part is a good solution to avoid it being wrongly resolved by the getDestinationObjectName().
However i do not agree we should filter out objects of size 0 as that limits transfering empty files which can be a real transfer scenario.
For empty files, the AWS documentation lacks to explain if having an empty completedParts list is valid to complete a multi part upload. It just says here a Part cannot be invalid, but says nothing about empty lists. Nevertheless, using multi part might not be the best approach in the empty file Part case. When bytesChunk comes empty and completedParts is also empty, the multi part should be aborted and a PutObject should be used instead. This way we can guarantee empty files are also a valid transfer scenario.
Also, I quickly glanced the tests and it seems no case exists for an empty file transfer. Something to be improved.
@hemantxpatel Since you have the initial solution proposal, would you like to come forward and bring in a PR for this?

hey everyone :) can I be assigned to this issue, please?

hey everyone :) can I be assigned to this issue, please?

for some reason GH doesn't let me assign this to you. I used @rafaelmag110 as stand-in, so we know someone's working on it.

Hi all,
@rafaelmag110 asked me to work on it, so I had started working on it. I verified my code via doing an S3 to S3 transfer and it works well.
@bmg13 Let me know if you haven't already started and I can open the PR, otherwise it's a small fix.
Just need to convert while loop to a do while loop, so that a part is uploaded even it has size zero.
https://github.com/eclipse-edc/Technology-Aws/blob/e6e78a3cb1dbcecd29ad6b7a1ea93e6ac609f9b0/extensions/data-plane/data-plane-aws-s3/src/main/java/org/eclipse/edc/connector/dataplane/aws/s3/S3DataSink.java#L63-L73

Thanks @hemantxpatel
I contacted you directly because in this case we wanted to move the fix a bit faster so we could have it in time for a downstream bugfix release.
We indeed started working on this and should have the PR ready today.
Sorry for the confusion.


fix(btrfs-assistant): Prevent snapper from saturating storage with snapshots

This PR disables snapper-timeline.timer whenever snapper config is setup.
It also disables snapshots on any currently affected devices. Users might need to reenable them afterwards.

Closed in favor of https://github.com/ublue-os/bazzite/pull/1939


Other haproxy rules that need to be placed appropriately

Similar to #313 there are other rules that need to be placed properly.
I'm gathering from the haproxy code that there is an order here:
For frontend & listen: (where applicable options are available)
tcp-request connection
tcp-request session
tcp-request content
monitor fail
block (deprecated)
http-request
reqxxx
reqadd
redirect
use_backend

I tested this locally and was able to get the configuration validation warnings for all scenarios except the ordering related to monitor fail. The block also complains about being deprecated.
The fix could be just adding some more checks in the template for these other options to output them in the correct order.

Is there somewhere in the Haproxy docs we can point to, and maybe crib from to get this sorted?
Or at least have the values we pass on, not break Haproxy by default?

I haven't found it any documentation as of yet. I just sent an email per their repo's docs on contacting them, requesting some info/docs on the ordering of the config.

Nothing documented beyond the code. The order in the initial comment there is the hierarchy then. Thoughts on handling that in the template when any of those extra_options are set?

I'm really not sure we can do anything smart, apart from not reload Haproxy if the config is wrong in that case.

I was thinking we could do similar to the change I made for use_backend and http-request and just apply the config values, if set in other options, in the correct order in the template.

I will be opening a new PR to add some documentation around these errors and the specific ordering the values need to be placed in using extra-options in lieu of making nasty erb template files.


Update FAPI-SIG's scope

Closes #321

@VinodAnandan Could you check this PR for updating README.md?

LGTM @tnorimat, Thank you !


Can you provide the whole code of this paper?

I found  that the code in this repo is only a part of your paper, which is the back propagation function. Can you provide the whole code so that I can reimplement the experiments.

Thank you for interest in our work.
We regarded the construction of the "differentiable solvers" as the key contribution and primarily released these torch modules as we found them to be most valuable for the community.
The tasks we constructed and solved are not recognized benchmarks and were not intended to become ones.
For these reasons, we will probably not maintain a repository supporting all of our experiments. Unless there is significant interest.
Having said that, the code for reproducing the shortest path experiment can be downloaded here (this link is also in the readme). We will shortly extend this code to cover more of our experiments but as I said, we probably will not invest time and energy into polishing the code and maintaining a large separate repository.

Dear @mrolinek ,
Thanks for such a nice repository and sharing additional resources for Shortest path experiment.
I was wondering whether the same thing could also be possible for traveling sales man problem? It would be great to have an example code showing how Gurobi optimizer can be used together with the NN model.


not able to install apache2 on ubutu 20.04 LTS

Description

Platforms affected

[ ] Azure DevOps
[ ] GitHub Actions - Standard Runners
[ ] GitHub Actions - Larger Runners

Runner images affected

[X] Ubuntu 20.04
[ ] Ubuntu 22.04
[ ] Ubuntu 24.04
[ ] macOS 12
[ ] macOS 13
[ ] macOS 13 Arm64
[ ] macOS 14
[ ] macOS 14 Arm64
[ ] Windows Server 2019
[ ] Windows Server 2022

Image version and build link
20_04-lts-gen2
Is it regression?
cause not found.
Expected behavior
it was expected to install apache2 using cli
Actual behavior
not able to fetch azure package page
Repro steps
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
apache2-bin apache2-data apache2-utils libapr1 libaprutil1 libaprutil1-dbd-sqlite3 libaprutil1-ldap libjansson4
liblua5.2-0 ssl-cert
Suggested packages:
apache2-doc apache2-suexec-pristine | apache2-suexec-custom www-browser openssl-blacklist
The following NEW packages will be installed:
apache2 apache2-bin apache2-data apache2-utils libapr1 libaprutil1 libaprutil1-dbd-sqlite3 libaprutil1-ldap
libjansson4 liblua5.2-0 ssl-cert
0 upgraded, 11 newly installed, 0 to remove and 0 not upgraded.
Need to get 1875 kB of archives.
After this operation, 8121 kB of additional disk space will be used.
Do you want to continue?

Hi @vkplays ,
We will investigate about this issue and update you soon. Thank you !

Still the issues continues

@vkplays - Try these below workarounds -

Try changing http://security.ubuntu.com/ubuntu to http://old-releases.ubuntu.com/ubuntu in /etc/apt/sources.list,
then run  sudo apt-get update
or Try this below one
Open /etc/apt/sources.list file.
Check the word after http://security.ubuntu.com/ubuntu/ in the sources.list file, lets say the word be xyz
Go to the address http://security.ubuntu.com/ubuntu/ and then click on dists.
Now check whether xyz is present at this path. If it is not present, then replace xyz with the element which is present in 'dists' such as:- saucy, lucid, trusty etc.

Now run sudo apt-get update command.
or try this below sed command workaround option

To make sudo apt-get update work - we simply need to edit /etc/apt/sources.list and update all URLs from security.ubuntu.com and archive.ubuntu.com to point to old-releases.ubuntu.com -> as per the following sed command:

sudo sed -i.bak -r 's/(archive|security).ubuntu.com/old-releases.ubuntu.com/g' /etc/apt/sources.list
After this, sudo apt-get update should work . And once you do this you should run the below command
sudo apt-get install update-manager-core; do-release-upgrade
Please let me know if it works for you.

Note, this file is written by cloud-init on first boot of an instance
modifications made here will not survive a re-bundle.
if you wish to make changes you can:
a.) add 'apt_preserve_sources_list: true' to /etc/cloud/cloud.cfg
or do the same in user-data
b.) add sources in /etc/apt/sources.list.d
c.) make changes to template file /etc/cloud/templates/sources.list.tmpl
See http://help.ubuntu.com/community/UpgradeNotes for how to upgrade to
newer versions of the distribution.
deb http://azure.archive.ubuntu.com/ubuntu/ jammy main restricted
deb-src http://azure.archive.ubuntu.com/ubuntu/ jammy main restricted
Major bug fix updates produced after the final release of the
distribution.
deb http://azure.archive.ubuntu.com/ubuntu/ jammy-updates main restricted
deb-src http://azure.archive.ubuntu.com/ubuntu/ jammy-updates main restricted
N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu
team. Also, please note that software in universe WILL NOT receive any
review or updates from the Ubuntu security team.
deb http://azure.archive.ubuntu.com/ubuntu/ jammy universe
deb-src http://azure.archive.ubuntu.com/ubuntu/ jammy universe
deb http://azure.archive.ubuntu.com/ubuntu/ jammy-updates universe
deb-src http://azure.archive.ubuntu.com/ubuntu/ jammy-updates universe
N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu
team, and may not be under a free licence. Please satisfy yourself as to
your rights to use the software. Also, please note that software in
multiverse WILL NOT receive any review or updates from the Ubuntu
security team.
deb http://azure.archive.ubuntu.com/ubuntu/ jammy multiverse
deb-src http://azure.archive.ubuntu.com/ubuntu/ jammy multiverse
deb http://azure.archive.ubuntu.com/ubuntu/ jammy-updates multiverse
deb-src http://azure.archive.ubuntu.com/ubuntu/ jammy-updates multiverse
N.B. software from this repository may not have been tested as
extensively as that contained in the main release, although it includes
newer versions of some applications which may provide useful features.
Also, please note that software in backports WILL NOT receive any review
or updates from the Ubuntu security team.
deb http://azure.archive.ubuntu.com/ubuntu/ jammy-backports main restricted universe multiverse
deb-src http://azure.archive.ubuntu.com/ubuntu/ jammy-backports main restricted universe multiverse
deb http://azure.archive.ubuntu.com/ubuntu/ jammy-security main restricted
deb-src http://azure.archive.ubuntu.com/ubuntu/ jammy-security main restricted
deb http://azure.archive.ubuntu.com/ubuntu/ jammy-security universe
deb-src http://azure.archive.ubuntu.com/ubuntu/ jammy-security universe
deb http://azure.archive.ubuntu.com/ubuntu/ jammy-security multiverse
deb-src http://azure.archive.ubuntu.com/ubuntu/ jammy-security multiverse
there is no link like http://security.ubuntu.com/ubuntu or http://security.ubuntu.com/ubuntu/
this issue has appeared today in microsoft azure yesterday everything was fine.

@vkplays - try to run this below command .
sudo apt-get update --fix-missing
and follow the below link - This could be caused by a number of factors, such as network connectivity issues or firewall rules that are blocking access to the Ubuntu mirrors.
Check your network connectivity: Make sure that your Azure VM has a stable internet connection and that you can access other websites.
Check your firewall rules: Make sure that your Azure VM's firewall is not blocking access to the Ubuntu mirrors. You can check this by running the following command:
sudo ufw status
Check your DNS settings: Make sure that your Azure VM is configured to use the correct DNS servers. You can check this by running the following command:
cat /etc/resolv.conf
Check your proxy settings: Make sure that your Azure VM is not configured to use a proxy server that is blocking access to the Ubuntu mirrors. You can check this by running the following command:
env | grep -i proxy
Check the Azure VM's network security group: Make sure that the Azure VM's network security group is configured to allow traffic to the Ubuntu mirrors.
Check the Azure VM's network interface: Make sure that the Azure VM's network interface is configured to allow traffic to the Ubuntu mirrors.
Check the Azure VM's virtual network: Make sure that the Azure VM's virtual network is configured to allow traffic to the Ubuntu mirrors.
Source - https://learn.microsoft.com/en-us/answers/questions/1168604/unable-to-connect-to-azure-archive-ubuntu-com

i had not issues installing nginx but the issues is with apache2

this is screenshot while installing nginx

this is screenshot while installing apache2

@vkplays - Can you try again running below command
sudo apt-get update
Please confirm if you can able to install apache2 on ubuntu 20.04 LTS version.

still not able to install apache2 on Ubuntu LTS 20.04

Can i Share Vps's credentials and i will change it later ?

@vkplays - No please don't share. Please try once with the below command again -
sudo apt-get update --fix-missing

Hi,
I had a similar problem from a Ubuntu 22.04 installation:
Ihave replaced changed archive.ubuntu.com to de.archive.ubuntu.com in sources.list and everything worked.
Regards Axel

@ijunaidm the issues is resolved now maybee the issues was from azure only.
Btw Thanks for the help

@vkplays - Okay. Thank you for confirming. I will close the ticket. If you face any such issues related to images in future please reach out to us. Thank you.


Set execute permissions on script

This sets execute permissions (0777) on the *.sh file. Hoping you're comfortable with this, 0777 is arguably a bit broad but the file exists for a very short period of time.
This fixes #1

Good find, was wondering why I was getting a permission error on the folder (but not the file).
I pushed a similar change on https://github.com/Laurae2/xgbdl .


[Java] NullVector field name can't be set

Currently, the Java NullVector has a default Field name fixed to DATA_VECTOR_NAME, which is "$data$". This should be able to be changed by the user, probably by having an alternate constructor that accepts a name.
Reporter: Bryan Cutler / @BryanCutler
Note: This issue was originally created as ARROW-11382. Please see the migration documentation for further details.

This is now already implemented, and subsequently verified in https://github.com/apache/arrow/pull/38973.


Add spring security to save

        add("implementation", "org.springframework.boot:spring-boot-starter-security:${Versions.springBoot}")

An old related issue is #34

Related tasks:

Describe is save-deploy how to configure firewall rules to allow external access only to nginx (on default 80/443 ports)
Describe in save-deploy how to secure hosted save-cloud with HTTPS


trailingSlash: 'always' not building file structure

Describe the bug
I'm trying to generate a static site to be hosted on DreamHost shared server. I spun up a blank skeleton project and added this adapter, but changed nothing else, just to make sure it wasn't some other code of mine messing with it. It's still not working in production.
This is my svelte.config.js file:
import adapter from '@sveltejs/adapter-static';
import preprocess from 'svelte-preprocess';

/** @type {import('@sveltejs/kit').Config} */
const config = {
	// Consult https://github.com/sveltejs/svelte-preprocess
	// for more information about preprocessors
	preprocess: preprocess(),

	kit: {
		adapter: adapter({
			out: 'build',
			trailingSlash: 'always'
		})
	}
};

export default config;

This is my +layout.svelte file:

Layout

    Home
    About

    <slot />

This is my +layout.js file:
export const prerender = true;

And my `+page.svelte' files are:

Index

and

About

It generates this files structure:
root
  - _app
  - about.html
  - faviceon.png
  - index.html

Everything works fine on dev; and, in the build, as long as you only use the navigation, the secondary pages load fine, but as soon as you refresh or try to navigate to a secondary page by URL, it returns not found.
Additionally, if you add .html to the secondary page URL, it will load, but if you add that to the navigation, it won't build. Any help would be greatly appreciated!
Reproduction
https://github.com/coffeepostal/sveltekit-static-test
Logs
No response
System Info
System:
    OS: macOS Mojave 10.14.6
    CPU: (12) x64 Intel(R) Xeon(R) CPU           W3680  @ 3.33GHz
    Memory: 22.54 MB / 16.00 GB
    Shell: 3.2.57 - /bin/bash
  Binaries:
    Node: 16.14.2 - /usr/local/bin/node
    npm: 8.18.0 - /usr/local/bin/npm
  Browsers:
    Firefox Developer Edition: 71.0
  npmPackages:
    @sveltejs/adapter-node: ^1.0.0-next.96 => 1.0.0-next.96
    @sveltejs/adapter-static: ^1.0.0-next.44 => 1.0.0-next.44
    @sveltejs/kit: next => 1.0.0-next.511
    svelte: ^3.44.0 => 3.50.1
    vite: ^3.1.0 => 3.1.6

Severity
blocking all usage of SvelteKit
Additional Information
No response

trailingSlash isn't an adapter-specific option. Your svelte.config.js should look like this:
import adapter from '@sveltejs/adapter-static';
import preprocess from 'svelte-preprocess';

/** @type {import('@sveltejs/kit').Config} */
const config = {
	// Consult https://github.com/sveltejs/svelte-preprocess
	// for more information about preprocessors
	preprocess: preprocess(),

	kit: {
		adapter: adapter({
			out: 'build',
-			trailingSlash: 'always'
		}),

+		trailingSlash: 'always'
	}
};

export default config;

OH MY GOD! That's it! Thank you @Rich-Harris!


Restructuring individuals: solver

Description of the issue
As described in https://github.com/OpenEnergyPlatform/ontology/issues/859 most of the individuals in the OEO are lacking a definition.
In addition the upper classes are evaluated again.
Ideas of solution

Type
Individual
Updated
Definition
License

solver
coin-or branch and cut (CBC)
yes
COIN-OR branch and cut (CBC or Cbc) is an open-source mixed integer programming solver written in C++. It can be used as both a stand-alone executable and as a callable library, General Algebraic Modeling System (GAMS), MPL, AIMMS, PuLP, CMPL, OpenSolver for Excel, JuMP, or MiniZinc).
EPL-2.0

solver
coin-or lp (CLP)
added
COIN-OR LP (CLP) is an open-source linear programming solver written in C++. It is published under the Common Public License so it can be used in proprietary software with none of the restrictions of the GNU General Public License. CLP is primarily meant to be used as a callable library, although a stand-alone executable version can be built.
EPL-2.0

solver
ibm ilog cplex optimizer (CPLEX)
yes
Popular solver with an API for several programming languages, and also has a modelling language and works with AIMMS, AMPL, GAMS, MPL, OpenOpt, OPL Development Studio, and TOMLAB. Free for academic use.
Proprietary

solver
gnu linear programming kit (GLPK)
yes
GNU Linear Programming Kit (GLPK), an LP/MILP solver with a native C API and numerous third-party wrappers for other languages. Specialist support for flow networks. Bundles the AMPL-like GNU MathProg modelling language and translator.
GPL-3.0-or-later

solver
gurobi
yes
Gurobi is a solver to solve numerical programming tasks. Supported are linear programming (LP), quadratic programming (QP), quadratic constrained programming (QCP), mixed integer linear programming (MILP)[3], mixed integer quadratic programming (MIQP), and mixed integer quadratic constrained programming (MIQCP).
Proprietary

solver
mosek
yes
MOSEK is a software package for the solution of linear, mixed-integer linear, quadratic, mixed-integer quadratic, quadratically constraint, conic and convex nonlinear mathematical optimization problems. The applicability of the solver varies widely and is commonly used for solving problems in areas such as engineering, finance and computer science. A solver for large scale optimization with API for several languages (C++,java,.net, Matlab and python).
Proprietary

solver
pips-ipmpp
added
PIPS-IPM++ (PIPS-IPMpp) is a (MPI + OpenMP) parallel interior-point method for doubly bordered block diagonal Linear Programs. There exist two general purpose interfaces to PIPS-IPM++, one via GAMS (thus a GAMS license is required) and one established with the METIS-project (on branch PyomoToPIPS).
LGPL-3.0

solver
highs
added
HiGHS is an open source serial and parallel solvers for large-scale sparse linear programming (LP), mixed-integer programming (MIP), and quadratic programming (QP) models.
MIT

l-emele commented yesterday:
Regarding the solvers:  I like the definition of gurobi: Gurobi is a solver to solve numerical programming tasks. ...
Others need some refinement:

coin-or lp: COIN-OR LP (CLP) is an open-source linear programming solver written in C++. ... What about: COIN-OR LP (CLP) is a solver that is written in C++ and has an open source license. This a) explicitly names the precise class solver and b) shows the relation to open source.
ibm ilog cplex optimiser: IBM ILOG CPLEX opimiser (CPLES) is a ~popular~ solver with an API for several programming languages, and also has a modelling language and works with AIMMS, AMPL, GAMS, MPL, OpenOpt, OPL Development Studio, and TOMLAB. Free for academic use.
Similar changes to the other solvers...

Workflow checklist

[x] I discussed the issue with someone else than me before working on a solution
[x] I already read the latest version of the workflow for this repository
[x] The goal of this ontology is clear to me

I am aware that

[x] every entry in the ontology should have a definition
[x] classes should arise from concepts rather than from words

I added a column with the licenses. This is a key parameter for the use in energy system modelling.

Okay, the licenses will then be included with 'has license' some 'software license'. If we want to have the exact licenses, we need something like 'has license' value 'EPL-2.0' or something like that. However, we have not yet included/imported license individuals.
In #31 we discussed about importing the SPDX ontology.

I wonder whether it makes sense at all to have specific solver individuals in the OEO at all.
And even if we have them in the OEO I wonder whether the OEO is the right place to catch these details like the specific licences. Maybe it is better to store such information in the OEKG or to have solver fact sheets on the OEP or something like that.
Anyway, I'll move this issue to a future milestone.


[Dialog] [React] Backdrop presence not being inherited from root prop

Description
I encountered an inconsistency while using Framer Motion's AnimatePresence with the Dialog component. I'm new to the library, so I'm unsure if this behavior is intentional. I'm reporting it just in case 🙂
The presence prop functions as expected for every Dialog component except Backdrop.
While Dialog.Positioner and Dialog.Content handle it like this:
const dialog = useDialogContext()
const presence = usePresenceContext()
const mergedProps = mergeProps(dialog.getContentProps(), presence.getPresenceProps(ref), props)

if (presence.unmounted) {
  return null
}

Dialog.Backdrop handles it differently:
const dialog = useDialogContext()
const renderStrategyProps = useRenderStrategyPropsContext()
const presence = usePresence({ ...renderStrategyProps, present: dialog.open })
const mergedProps = mergeProps(dialog.getBackdropProps(), presence.getPresenceProps(ref), props)

if (presence.unmounted) {
  return null
}

The issue here is that, while the first components read presence from the Dialog.Root PresenceProvider, Dialog.Backdrop doesn't.
When changing the Dialog.Backdrop code to behave like Dialog.Positioner and Dialog.Content, everything seems to work perfectly. I've opened https://github.com/chakra-ui/ark/pull/2556 with this change in case you want to check it out 👀
In the reproduction I'm wrapping the Dialog.Positioner with Dialog.Backdrop as that's how I do it with Radix. Note that the issue persist even when rendering the Dialog.Positioner and Dialog.Backdrop as siblings.
PS: I really love all of your projects! Kudos to the Chakra team!
Link to Reproduction (or Detailed Explanation)
See Reproduction
Steps to Reproduce

Open the Dialog without Backdrop
The opening animation should play
Close the Dialog without Backdrop
The closing animation should play
Open the dialog with Backdrop
The opening animation should play
Close the Dialog with Backdrop
The Dialog unmounts before Framer Motion gets to animate it

Ark UI Version
3.3.0
Framework

[X] React
[ ] Solid
[ ] Vue

Browser
Google Chrome, Version 125.0.6422.142 (Official Build) (arm64)
Additional Information
Here's a video of the issue 🙂
https://github.com/chakra-ui/ark/assets/48155434/d6e2deae-f665-4c2a-b164-32e93395b523

We designed it this way to allow for animating the backdrop with a different timeline compared to the content.
When composing with framer-motion, I'd recommend skipping the backdrop entirely and rendering your custom version. These components were built to be animated with CSS animations.
const DialogWithBackdrop = () => {
  const [open, setOpen] = useState(false);
  return (
    <Dialog.Root
      lazyMount
      present
      open={open}
      onOpenChange={(event) => setOpen(event.open)}
    >
      <Dialog.Trigger>Open dialog with backdrop (broken)</Dialog.Trigger>

      <AnimatePresence initial={false}>
        {open && (
          <motion.div
            initial={{ opacity: 0 }}
            animate={{ opacity: 1 }}
            exit={{ opacity: 0 }}
            transition={{ duration: 1.5 }}
          />
        )}
      </AnimatePresence>

      <AnimatePresence initial={false}>
        {open && (
          <Dialog.Positioner asChild>
            <motion.div
              initial={{ scale: 0 }}
              animate={{ scale: 1 }}
              exit={{ scale: 0 }}
              transition={{ duration: 2 }}
            >
              <Dialog.Content>
                <Dialog.Title>Dialog Title</Dialog.Title>
              </Dialog.Content>
            </motion.div>
          </Dialog.Positioner>
        )}
      </AnimatePresence>
    </Dialog.Root>
  );
};

https://codesandbox.io/p/devbox/ark-framer-composition-4c3mmd?file=%2Fsrc%2FApp.tsx%3A13%2C6


Operational Plugin: Metrics

One of the operational plugins (#41) .

Endpoint /operations/metrics
Must be possible to configure this plugin with additional metrics to report.

Supported operations:

Report metrics about the running application

Examples
GET /operations/metrics

To Do: Define the output format, but the info must include:
Virtual Machine Statistics
--------------------------
uptime			0h53m32s
memory			99,753,984 bytes
	old			90,801,952 bytes (91.0%)
	young		4,022,440 bytes (4.0%)
	used		67,200,720 bytes (67.4%)
	free		27,623,672 bytes (27.700000000000003%)
GCs				486 (6610ms between GCs)
	full			1 totalling 87ms (0.0% uptime), avg 87.0ms
	incr		485 totalling 448ms (0.0% uptime), avg 0.9ms
	tenures		45,864 (avg 0 GCs/tenure)
Since last view	1 (1002ms between GCs)
	uptime		1.0s
	full			0 totalling 0ms (0.0% uptime)
	incr		1 totalling 1ms (0.1% uptime), avg 1.0ms
	tenures		0

Format supported by Prometheus and OpenTSDB
https://docs.google.com/document/d/1ZjyKiKxZV83VI9ZKAXRGKaUKK2BIWCT7oiGBKDBpjEY/edit#heading=h.wnviarbnyxcj

https://openmetrics.io/


Add more stuff

Added basic argparser
Added options
Added automatic filename output generator
Added standard library (basically just a thing at the top)

Why did you close your PR? It's good. You should reopen it and fix the merge conflicts.

Woohoo!
@Abb1x


Notification did not show after being sent from FCM [ObjC]

So, i tried to copy paste all the codes inside the MessagingExample code and configure all the APN + Google.plist certificate for my project
I can receive the FCM Registration token and also the APNs device token for my app
However, after i tried to send a message from the Firebase Console, the notification never appear in my iphone for both foreground and background.
I JUST added this code during the configuration on my AppDelegate.m (applicationDidFinishLaunchingOptions) in order to able to display the message to the console log
[FIRMessaging messaging].shouldEstablishDirectChannel = TRUE;
If I did not add this code, it won't display the result of this delegate to the console
//  Copyright (c) 2016 Google Inc.
//
//  Licensed under the Apache License, Version 2.0 (the "License");
//  you may not use this file except in compliance with the License.
//  You may obtain a copy of the License at
//
//  http://www.apache.org/licenses/LICENSE-2.0
//
//  Unless required by applicable law or agreed to in writing, software
//  distributed under the License is distributed on an "AS IS" BASIS,
//  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
//  See the License for the specific language governing permissions and
//  limitations under the License.
//

#import "AppDelegate.h"

#if defined(__IPHONE_10_0) && __IPHONE_OS_VERSION_MAX_ALLOWED >= __IPHONE_10_0
@import UserNotifications;
#endif

@import Firebase;
@import FirebaseMessaging;

// Implement UNUserNotificationCenterDelegate to receive display notification via APNS for devices
// running iOS 10 and above.
#if defined(__IPHONE_10_0) && __IPHONE_OS_VERSION_MAX_ALLOWED >= __IPHONE_10_0
@interface AppDelegate () <UNUserNotificationCenterDelegate, FIRMessagingDelegate>
@end
#endif

// Copied from Apple's header in case it is missing in some cases (e.g. pre-Xcode 8 builds).
#ifndef NSFoundationVersionNumber_iOS_9_x_Max
#define NSFoundationVersionNumber_iOS_9_x_Max 1299
#endif

@implementation AppDelegate

    NSString *const kGCMMessageIDKey = @"gcm.message_id";

- (BOOL)application:(UIApplication *)application
didFinishLaunchingWithOptions:(NSDictionary *)launchOptions {
    // [START configure_firebase]
#ifdef DEBUG
    NSString *filePath = [[NSBundle bundleForClass:[self class]] pathForResource:@"GoogleService-Development" ofType:@"plist"];
#else
    NSString *filePath = [[NSBundle bundleForClass:[self class]] pathForResource:@"GoogleService-Info" ofType:@"plist"];
#endif
    FIROptions *options = [[FIROptions alloc] initWithContentsOfFile:filePath];
    [FIRApp configureWithOptions:options];

    // [START set_messaging_delegate]
    [FIRMessaging messaging].delegate = self;
    // [END set_messaging_delegate]

    // Register for remote notifications. This shows a permission dialog on first run, to
    // show the dialog at a more appropriate time move this registration accordingly.
    if (floor(NSFoundationVersionNumber) <= NSFoundationVersionNumber_iOS_7_1) {
        // iOS 7.1 or earlier. Disable the deprecation warnings.
#pragma clang diagnostic push
#pragma clang diagnostic ignored "-Wdeprecated-declarations"
        UIRemoteNotificationType allNotificationTypes =
        (UIRemoteNotificationTypeSound |
         UIRemoteNotificationTypeAlert |
         UIRemoteNotificationTypeBadge);
        [application registerForRemoteNotificationTypes:allNotificationTypes];
#pragma clang diagnostic pop
    } else {
        // iOS 8 or later
        // [START register_for_notifications]
        if (floor(NSFoundationVersionNumber) <= NSFoundationVersionNumber_iOS_9_x_Max) {
            UIUserNotificationType allNotificationTypes =
            (UIUserNotificationTypeSound | UIUserNotificationTypeAlert | UIUserNotificationTypeBadge);
            UIUserNotificationSettings *settings =
            [UIUserNotificationSettings settingsForTypes:allNotificationTypes categories:nil];
            [application registerUserNotificationSettings:settings];
        } else {
            // iOS 10 or later
#if defined(__IPHONE_10_0) && __IPHONE_OS_VERSION_MAX_ALLOWED >= __IPHONE_10_0
            // For iOS 10 display notification (sent via APNS)
            [UNUserNotificationCenter currentNotificationCenter].delegate = self;
            UNAuthorizationOptions authOptions =
            UNAuthorizationOptionAlert
            | UNAuthorizationOptionSound
            | UNAuthorizationOptionBadge;
            [[UNUserNotificationCenter currentNotificationCenter] requestAuthorizationWithOptions:authOptions completionHandler:^(BOOL granted, NSError * _Nullable error) {
            }];
#endif
        }

        [FIRMessaging messaging].shouldEstablishDirectChannel = TRUE;
        [application registerForRemoteNotifications];
        // [END register_for_notifications]
    }
    return YES;
}

    // [START receive_message]
- (void)application:(UIApplication *)application didReceiveRemoteNotification:(NSDictionary *)userInfo {
    // If you are receiving a notification message while your app is in the background,
    // this callback will not be fired till the user taps on the notification launching the application.
    // TODO: Handle data of notification

    // With swizzling disabled you must let Messaging know about the message, for Analytics
    // [[FIRMessaging messaging] appDidReceiveMessage:userInfo];

    // Print message ID.
    if (userInfo[kGCMMessageIDKey]) {
        NSLog(@"Message ID: %@", userInfo[kGCMMessageIDKey]);
    }

    // Print full message.
    NSLog(@"%@", userInfo);
}

- (void)application:(UIApplication *)application didReceiveRemoteNotification:(NSDictionary *)userInfo
fetchCompletionHandler:(void (^)(UIBackgroundFetchResult))completionHandler {
    // If you are receiving a notification message while your app is in the background,
    // this callback will not be fired till the user taps on the notification launching the application.
    // TODO: Handle data of notification

    // With swizzling disabled you must let Messaging know about the message, for Analytics
    // [[FIRMessaging messaging] appDidReceiveMessage:userInfo];

    // Print message ID.
    if (userInfo[kGCMMessageIDKey]) {
        NSLog(@"Message ID: %@", userInfo[kGCMMessageIDKey]);
    }

    // Print full message.
    NSLog(@"%@", userInfo);

    completionHandler(UIBackgroundFetchResultNewData);
}
    // [END receive_message]

// [START ios_10_message_handling]
// Receive displayed notifications for iOS 10 devices.
#if defined(__IPHONE_10_0) && __IPHONE_OS_VERSION_MAX_ALLOWED >= __IPHONE_10_0
// Handle incoming notification messages while app is in the foreground.
- (void)userNotificationCenter:(UNUserNotificationCenter *)center
       willPresentNotification:(UNNotification *)notification
         withCompletionHandler:(void (^)(UNNotificationPresentationOptions))completionHandler {
    NSDictionary *userInfo = notification.request.content.userInfo;

    // With swizzling disabled you must let Messaging know about the message, for Analytics
    // [[FIRMessaging messaging] appDidReceiveMessage:userInfo];

    // Print message ID.
    if (userInfo[kGCMMessageIDKey]) {
        NSLog(@"Message ID: %@", userInfo[kGCMMessageIDKey]);
    }

    // Print full message.
    NSLog(@"%@", userInfo);

    // Change this to your preferred presentation option
    completionHandler(UNNotificationPresentationOptionAlert);
}

// Handle notification messages after display notification is tapped by the user.
- (void)userNotificationCenter:(UNUserNotificationCenter *)center
didReceiveNotificationResponse:(UNNotificationResponse *)response
#if defined(__IPHONE_11_0)
         withCompletionHandler:(void(^)(void))completionHandler {
#else
withCompletionHandler:(void(^)())completionHandler {
#endif
    NSDictionary *userInfo = response.notification.request.content.userInfo;
    if (userInfo[kGCMMessageIDKey]) {
        NSLog(@"Message ID: %@", userInfo[kGCMMessageIDKey]);
    }

    // Print full message.
    NSLog(@"%@", userInfo);

    completionHandler();
}
#endif
    // [END ios_10_message_handling]

// [START refresh_token]
- (void)messaging:(FIRMessaging *)messaging didReceiveRegistrationToken:(NSString *)fcmToken {
    NSLog(@"FCM registration token: %@", fcmToken);

    // TODO: If necessary send token to application server.
    // Note: This callback is fired at each app startup and whenever a new token is generated.
}
// [END refresh_token]

// [START ios_10_data_message]
// Receive data messages on iOS 10+ directly from FCM (bypassing APNs) when the app is in the foreground.
// To enable direct data messages, you can set [Messaging messaging].shouldEstablishDirectChannel to YES.
- (void)messaging:(FIRMessaging *)messaging didReceiveMessage:(FIRMessagingRemoteMessage *)remoteMessage {
    NSLog(@"Received data message: %@", remoteMessage.appData);
}
// [END ios_10_data_message]

- (void)application:(UIApplication *)application didFailToRegisterForRemoteNotificationsWithError:(NSError *)error {
    NSLog(@"Unable to register for remote notifications: %@", error);
}

// This function is added here only for debugging purposes, and can be removed if swizzling is enabled.
// If swizzling is disabled then this function must be implemented so that the APNs device token can be paired to
// the FCM registration token.
- (void)application:(UIApplication *)application didRegisterForRemoteNotificationsWithDeviceToken:(NSData *)deviceToken {
    NSLog(@"APNs device token retrieved: %@", deviceToken);

    // With swizzling disabled you must set the APNs device token here.
    // [FIRMessaging messaging].APNSToken = deviceToken;
}
@end

Here is the result of the remoteMessage delegate function. Please take note i have set the priority high in the FCM
// I fill in both of the "Message Text" and "Message Label" in the FCM, but i receive only the Message Label
Received data message: {
    "collapse_key" = "com.testing.push";
    from =<PHONE_NUMBER>75;
    notification =     {
        body = asd;
        e = 1;
        sound = default;
        sound2 = default;
    };
}

I also have added the 'UNNotificationPresentationOptions' in this following function
-(void)userNotificationCenter:(UNUserNotificationCenter *)center
       willPresentNotification:(UNNotification *)notification
         withCompletionHandler:(void (^)(UNNotificationPresentationOptions))completionHandler {
       NSDictionary *userInfo = notification.request.content.userInfo;

    // With swizzling disabled you must let Messaging know about the message, for Analytics
    // [[FIRMessaging messaging] appDidReceiveMessage:userInfo];

    // Print message ID.
    if (userInfo[kGCMMessageIDKey]) {
        NSLog(@"Message ID: %@", userInfo[kGCMMessageIDKey]);
    }

    // Print full message.
    NSLog(@"%@", userInfo);

    // Change this to your preferred presentation option
    completionHandler(UNNotificationPresentationOptionAlert);

Here are some of the screenshots of my project set up in the "Capabilities" Menu

The version details:

Xcode 9
iOS 10.3.3
Firebase/Core (4.5.0):
FirebaseMessaging (2.0.6):
FirebaseInstanceID (2.0.5)

What am I missing here? It is really frustrating after few hours researching and debugging. Thanks!

I'm also have this problem. I suspect it might have something to do with the provisioning profile expecting to be set up with SSL certs but firebase is set up using the new p8 token. Can anybody shed some light on this topic?

@avjiang, when using the APNs authentication key make sure the App ID Prefix in console is set to the Team ID of the development team that the APNs auth key belongs to.
The code itself looks fine, can you share some of the logs when running it? If there are any errors produced, those will have to be addressed too.
@wvdk "me too" comments without any code or repro steps are not easy for us to debug, since there may be several causes of issues in FCM that manifest in similar manners.

@morganchen12 Gotcha.
My issue seems to be very similar to this one. I've uploaded the p8 token in the console and set the App ID Prefix to our Team ID. When I launch the app I receive the FCM token and am granted notification authorization. But when I try to send a notification to that FCM token nothing happens (app is in background).
I'm looking for further steps I can take to debug.
Here's my AppDelegate:
import UIKit
import Firebase
import UserNotifications

@UIApplicationMain
class AppDelegate: UIResponder, UIApplicationDelegate {

    static var shared: AppDelegate!

    let firebasePlist = "GoogleService-Info"

    func application(_: UIApplication, didFinishLaunchingWithOptions: [UIApplicationLaunchOptionsKey: Any]?) -> Bool {
        AppDelegate.shared = self

        // Push Notifications
        UNUserNotificationCenter.current().delegate = self
        let options: UNAuthorizationOptions = [.alert, .badge, .sound]
        UNUserNotificationCenter.current().requestAuthorization(options: options) { (granted, error) in
            if granted {
                Log.d("Successfully authorized to for notifications.")
            } else {
                Log.w(error)
            }
        }

        // Set up firebase
        guard let firebaseOptions = FirebaseOptions(contentsOfFile: Bundle.main.path(forResource: firebasePlist, ofType: "plist")!) else {
            Log.e("Could not set up firebase options")
            fatalError()
        }
        FirebaseApp.configure(options: firebaseOptions)
        notificationsReferance = Database.database().reference(withPath: "Notifications")
        Messaging.messaging().delegate = self

        return true
    }

}

extension AppDelegate: UNUserNotificationCenterDelegate {
    func application(_ application: UIApplication, didRegisterForRemoteNotificationsWithDeviceToken deviceToken: Data) {
        Messaging.messaging().apnsToken = deviceToken
    }
}

extension AppDelegate: MessagingDelegate {

    func messaging(_ messaging: Messaging, didReceiveRegistrationToken fcmToken: String) {
        Log.d("Received registration token from FCM: \(fcmToken)")
    }

}

@wvdk are you able to send notifications directly through APNs? If so, what payload are you trying to send through FCM? If not, you'll have to double-check your APNs setup.

@morganchen12 I don't know how to send directly through APNs. The payload is a simple "hello world":

See step 4 of this blog post.


Improve session message injection

Hi there!
I'm trying to build a scalable, persistent MQTT server around RabbitMQ, using the MQTTNet server as a frontend.
While doing this, I found that there is currently no proper way to correlate injected QoS 1 messages with the packets received by the ClientAcknowledgedPublishPacketAsync event. So, there is no way for me to pass the PUBACK further to RabbitMQ's message acknowledgment.
Additionally, I noticed that DeliverApplicationMessageAsync() method may hang indefinitely when MqttPendingMessagesOverflowStrategy.DropNewMessage is used, because in the case of an output buffer overflow, the packetBusItem being awaited is dropped from the queue without the Task's Result or Exception being set.
So, in this PR:

I added MqttPublishPacket as a result of message injection methods, to be able to correlate them by packet id later.
Fixed the potential infinite wait in the DeliverApplicationMessageAsync() method.
Added some tests.

Thanks for the great project, btw!

@dotnet-policy-service agree

@chkr1011 Thanks!
The CI-workflow is not approved yet. It this intentional?

@chkr1011  Sorry to bother you again, but the CI has failed on the "sign nugets step". :))


Integrate CBOR test vector with gson.

Refer: test-vectors

The reference o/p doesn't seem to be standard JSON, need some groking.

The reference o/p doesn't seem to be standard JSON, need some groking.


[NEW ICON] Skill Name

Skill Name
AdonisJS
Why?
With AdonisJS, we don't need to worry about configuration or project structure, but with business rules. As a result, we become more productive. AdonisJS is an alternative to NestJS. Also, it is quite similar to laravel.. And I'm really enjoying learning about it.
Reference Image
No response

You might consider changing your issue name to [NEW ICON] AdonisJS

Also, a duplicate of #189


Update readme to fix npm registry escaped HTML (fixes #259)

fixes #259

Thanks! First PR :smile:
It does suck that you can't update the readme without npm publish (afaik)

Yeah, I think you're right. Maybe I can do a real PR one of these days :smile:


Implement Linear Calls process (in progress)

Coverage remained the same at 100.0% when pulling 97ec3cd15f936e50dd46ec331ba976740b722556 on feature/implementCallsProcess into a0f2bc15cedd82f55ec2bdd66bab24cc34364a8d on master.

Coverage remained the same at 100.0% when pulling eca7b7272dac9789f27e4f41767c0157826fe3a8 on feature/implementCallsProcess into a0f2bc15cedd82f55ec2bdd66bab24cc34364a8d on master.

Coverage remained the same at 100.0% when pulling e94d8e185907ed3ae13e86e3732264a71db78d03 on feature/implementCallsProcess into a0f2bc15cedd82f55ec2bdd66bab24cc34364a8d on master.