Backward syncing issues when restarting Nimbus

While testing out an unrelated fix with the unstable branch of nimbus, I observed at least three separate backward sync issues on our sepolia validating node. Note, I've only seen this issue on Sepolia and it was probably triggered by nimbus not getting peers before falling > 64 blocks behind the beacon chain which switches it into "far wall" sync mode, rather than lock-step sync.
It's worth noting that when nimbus is restarted, it requests a lot of eth_getLogs until it refills its deposit cache, which may be having some impact. When I enabled DEBUG during the backward sync issues, the eth_getLogs calls were ongoing so it seems related.
There's a thread discussing it with tersec from nimbus, starting about here https://discord.com/channels/595666850260713488/1093963476948496455/1101258732190572676
Observations
(1) Node offline during upgrade so I'd expect a small backwards sync (or lock-step sync depending on client).  Peers: 14","throwable":""}
{"@timestamp":"2023-04-27T00:11:14,769","level":"INFO","thread":"nioEventLoopGroup-3-9","class":"BackwardSyncContext","message":"Backward sync phase 2 of 2, 1.45% completed, imported 1 blocks of at least 69 (current head 3368103, target head 3368171). Peers: 12","throwable":""}
{"@timestamp":"2023-04-27T00:11:17,887","level":"INFO","thread":"nioEventLoopGroup-3-9","class":"BackwardSyncContext","message":"Backward sync phase 2 of 2 completed, imported a total of 66 blocks. Peers: 14","throwable":""}
{"@timestamp":"2023-04-27T00:11:17,916","level":"INFO","thread":"nioEventLoopGroup-3-9","class":"BackwardSyncContext","message":"Backward sync phase 2 of 2 completed, imported a total of 67 blocks. Peers: 14","throwable":""}
{"@timestamp":"2023-04-27T00:11:17,953","level":"INFO","thread":"nioEventLoopGroup-3-9","class":"BackwardSyncContext","message":"Backward sync phase 2 of 2 completed, imported a total of 68 blocks. Peers: 12","throwable":""}

Need to confirm, but I believe teku will give us the latest hash they have once they are in sync and we get a consistent set of % complete logs for phase 2 of 2 with one completed message at the end. Something about the pattern of requests nimbus is sending is causing multiple completed logs.
(2)  Besu is "in sync" as evidenced by being able to propose but then is frequently being switched to backwards sync mode presumably by being given a hash that we don't have a parent for.
(3) The reorg logs, which suggests it's being thrashed between forks potentially...
(2) and (3) both could be same root cause, or maybe all 3 are same cause.
relevant logs for (2) and (3)...
{"@timestamp":"2023-04-27T00:47:24,089","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineGetPayload","message":"Fetch block proposal by identifier: 0x0068b3ffaea1c01c, hash: 0xd1fb6c37bf03b2ed56bfd07699e9097119f143b600f4a33e4010edee3261a75a, number: 3368337, coinbase: 0x3826539cbd8d68dcf119e80b994557b4278cec9f, transaction count: 1","throwable":""}
{"@timestamp":"2023-04-27T00:47:24,107","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #3,368,337 / 1 tx / 16 ws / base fee 7 wei / 82,582 (0.3%) gas / (0xd1fb6c37bf03b2ed56bfd07699e9097119f143b600f4a33e4010edee3261a75a) in 0.003s. Peers: 13","throwable":""}
{"@timestamp":"2023-04-27T00:47:24,241","level":"WARN","thread":"vert.x-worker-thread-0","class":"DefaultBlockchain","message":"Chain Reorganization +192 new / -0 old
       Old - hash: 0xbca29e861aabf1b975cde3ee6f147a86ad8de3bfab1a39bfb2d98594a72eb63d, height: 3368145
       New - hash: 0xd1fb6c37bf03b2ed56bfd07699e9097119f143b600f4a33e4010edee3261a75a, height: 3368337
  Ancestor - hash: 0xbca29e861aabf1b975cde3ee6f147a86ad8de3bfab1a39bfb2d98594a72eb63d, height: 3368145","throwable":""}
{"@timestamp":"2023-04-27T00:47:24,670","level":"WARN","thread":"nioEventLoopGroup-3-9","class":"DefaultBlockchain","message":"Chain Reorganization +0 new / -192 old
       Old - hash: 0xd1fb6c37bf03b2ed56bfd07699e9097119f143b600f4a33e4010edee3261a75a, height: 3368337
       New - hash: 0xbca29e861aabf1b975cde3ee6f147a86ad8de3bfab1a39bfb2d98594a72eb63d, height: 3368145
  Ancestor - hash: 0xbca29e861aabf1b975cde3ee6f147a86ad8de3bfab1a39bfb2d98594a72eb63d, height: 3368145","throwable":""}
{"@timestamp":"2023-04-27T00:47:26,555","level":"INFO","thread":"nioEventLoopGroup-3-9","class":"BackwardSyncContext","message":"Backward sync phase 2 of 2 completed, imported a total of 116 blocks. Peers: 12","throwable":""}
{"@timestamp":"2023-04-27T00:47:27,713","level":"INFO","thread":"nioEventLoopGroup-3-9","class":"BackwardSyncContext","message":"Backward sync phase 2 of 2, -53.45% completed, imported -62 blocks of at least 116 (current head 3368154, target head 3368332). Peers: 13","throwable":""}

Full logs
besu-23.1.2.log
nimbus-ea8ef5-2.log
Nimbus versions differing behaviour

The unstable branch I tested was what produced these logs
v23.4.0 seems to show similar behaviour
v23.3.2 didn't seem to have as many problems

Update 06/05/2023
Further testing with targeted besu debug logging enabled, using the following log RPC:
curl --retry-delay 50 --retry-connrefused --location \
     --request POST 'http://<IP_ADDRESS>:8545' \
     --header 'Content-Type: application/json' \
     --data-raw '{
    "jsonrpc": "2.0",
    "method": "admin_changeLogLevel",
    "params": [
        "DEBUG", [
          "org.hyperledger.besu.ethereum.eth.sync.backwardsync",
          "org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.engine.AbstractEngineNewPayload",
          "org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.engine.AbstractEngineForkchoiceUpdated"
        ]
    ],
    "id": 1
}'

Besu was version 23.4.0 for all three nimbus versions.
Ran the same test on the three different nimbus versions on non-validating sepolia node, the test was just to restart nimbus:

v23.4.0-ea8ef579 (from unstable branch)
v23.4.0
v23.3.2

Logs attached here: https://we.tl/t-RZMmscEG1A
In all cases, there was less noise than on the validating node which suggests validating (with lots of validators) may exacerbate the problems.

Some details:
If Nimbus is restarted while Besu is already in the process of syncing, Besu goes into backward sync, which never completes and returns negative values for percentage complete while Nimbus reports receiving blocks from an unviable fork.  Stopping both clients, starting Besu and allowing it to reach the point where it is waiting for a connection from the CL prior to starting Nimbus, allows sync to resume normally.
Here's what Besu returns when it is stopped in the middle of the failed backward sync:
2023-05-02 08:45:19.497-04:00 | nioEventLoopGroup-3-5 | INFO  | BackwardSyncContext | Backward sync phase 2 of 2, -60.40% completed, imported -90 blocks of at least 149 (current head 8928631, target head 8928870). Peers: 4
^C2023-05-02 08:45:21.952-04:00 | BesuCommand-Shutdown-Hook | INFO  | NetworkRunner | Stopping Network.
2023-05-02 08:45:21.955-04:00 | BesuCommand-Shutdown-Hook | INFO  | EthProtocolManager | Stopping eth Subprotocol.
2023-05-02 08:45:21.956-04:00 | BesuCommand-Shutdown-Hook | INFO  | EthProtocolManager | eth Subprotocol stopped.
2023-05-02 08:45:21.956-04:00 | BesuCommand-Shutdown-Hook | INFO  | NetworkRunner | Network stopped.
2023-05-02 08:45:21.956-04:00 | BesuCommand-Shutdown-Hook | INFO  | AutoTransactionLogBloomCachingService | Shutting down Auto transaction logs caching service.
2023-05-02 08:45:21.956-04:00 | BesuCommand-Shutdown-Hook | INFO  | NatService | No NAT environment detected so no service could be stopped
2023-05-02 08:45:21.956-04:00 | nioEventLoopGroup-3-5 | ERROR | RocksDBColumnarKeyValueStorage | Attempting to use a closed RocksDbKeyValueStorage
2023-05-02 08:45:21.956-04:00 | ForkJoinPool.commonPool-worker-25 | ERROR | RocksDBColumnarKeyValueStorage | Attempting to use a closed RocksDbKeyValueStorage
2023-05-02 08:45:21.957-04:00 | nioEventLoopGroup-3-5 | ERROR | MainnetTransactionProcessor | Critical Exception Processing Transaction
java.lang.IllegalStateException: Storage has been closed
        at org.hyperledger.besu.plugin.services.storage.rocksdb.segmented.RocksDBColumnarKeyValueStorage.throwIfClosed(RocksDBColumnarKeyValueStorage.java:336)
        at org.hyperledger.besu.plugin.services.storage.rocksdb.segmented.RocksDBColumnarKeyValueStorage.get(RocksDBColumnarKeyValueStorage.java:256)
        at org.hyperledger.besu.plugin.services.storage.rocksdb.segmented.RocksDBColumnarKeyValueStorage.get(RocksDBColumnarKeyValueStorage.java:69)
        at org.hyperledger.besu.services.kvstore.SegmentedKeyValueStorageAdapter.get(SegmentedKeyValueStorageAdapter.java:64)
        at org.hyperledger.besu.ethereum.bonsai.storage.BonsaiWorldStateKeyValueStorage.getStorageValueByStorageSlotKey(BonsaiWorldStateKeyValueStorage.java:257)
        at org.hyperledger.besu.ethereum.bonsai.worldview.BonsaiWorldState.getStorageValueByStorageSlotKey(BonsaiWorldState.java:522)
        at org.hyperledger.besu.ethereum.bonsai.worldview.BonsaiWorldStateUpdateAccumulator.getStorageValueByStorageSlotKey(BonsaiWorldStateUpdateAccumulator.java:422)
        at org.hyperledger.besu.ethereum.bonsai.worldview.BonsaiWorldStateUpdateAccumulator.getStorageValue(BonsaiWorldStateUpdateAccumulator.java:404)
        at org.hyperledger.besu.ethereum.bonsai.BonsaiAccount.getStorageValue(BonsaiAccount.java:211)
        at org.hyperledger.besu.evm.worldstate.UpdateTrackingAccount.getStorageValue(UpdateTrackingAccount.java:231)
        at org.hyperledger.besu.evm.worldstate.UpdateTrackingAccount.getStorageValue(UpdateTrackingAccount.java:231)
        at org.hyperledger.besu.evm.worldstate.UpdateTrackingAccount.getStorageValue(UpdateTrackingAccount.java:231)
        at org.hyperledger.besu.evm.worldstate.UpdateTrackingAccount.getStorageValue(UpdateTrackingAccount.java:231)
        at org.hyperledger.besu.evm.worldstate.UpdateTrackingAccount.getStorageValue(UpdateTrackingAccount.java:231)
        at org.hyperledger.besu.evm.operation.SLoadOperation.execute(SLoadOperation.java:64)
        at org.hyperledger.besu.evm.EVM.runToHalt(EVM.java:265)
        at org.hyperledger.besu.evm.processor.AbstractMessageProcessor.codeExecute(AbstractMessageProcessor.java:174)
        at org.hyperledger.besu.evm.processor.AbstractMessageProcessor.process(AbstractMessageProcessor.java:192)
        at org.hyperledger.besu.ethereum.mainnet.MainnetTransactionProcessor.process(MainnetTransactionProcessor.java:508)
        at org.hyperledger.besu.ethereum.mainnet.MainnetTransactionProcessor.processTransaction(MainnetTransactionProcessor.java:419)
        at org.hyperledger.besu.ethereum.mainnet.AbstractBlockProcessor.processBlock(AbstractBlockProcessor.java:125)
        at org.hyperledger.besu.ethereum.mainnet.BlockProcessor.processBlock(BlockProcessor.java:79)
        at org.hyperledger.besu.ethereum.MainnetBlockValidator.processBlock(MainnetBlockValidator.java:207)
        at org.hyperledger.besu.ethereum.MainnetBlockValidator.validateAndProcessBlock(MainnetBlockValidator.java:135)
        at org.hyperledger.besu.ethereum.MainnetBlockValidator.validateAndProcessBlock(MainnetBlockValidator.java:72)
        at org.hyperledger.besu.ethereum.eth.sync.backwardsync.BackwardSyncContext.saveBlock(BackwardSyncContext.java:303)
        at org.hyperledger.besu.ethereum.eth.sync.backwardsync.ForwardSyncStep.saveBlocks(ForwardSyncStep.java:124)
        at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:646)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
        at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
        at org.hyperledger.besu.ethereum.eth.manager.task.RetryingGetBlocksFromPeersTask.lambda$executeTaskOnCurrentPeer$0(RetryingGetBlocksFromPeersTask.java:86)
        at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:646)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
        at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
        at org.hyperledger.besu.ethereum.eth.manager.task.AbstractPeerRequestTask.lambda$executeTask$2(AbstractPeerRequestTask.java:83)
        at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
        at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
        at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
        at org.hyperledger.besu.ethereum.eth.manager.task.AbstractPeerRequestTask.lambda$handleMessage$3(AbstractPeerRequestTask.java:106)
        at java.base/java.util.Optional.ifPresent(Optional.java:178)
        at org.hyperledger.besu.ethereum.eth.manager.task.AbstractPeerRequestTask.handleMessage(AbstractPeerRequestTask.java:104)
        at org.hyperledger.besu.ethereum.eth.manager.task.AbstractPeerRequestTask.lambda$executeTask$0(AbstractPeerRequestTask.java:67)
        at org.hyperledger.besu.ethereum.eth.manager.RequestManager$ResponseStream.dispatchBufferedResponses(RequestManager.java:211)
        at org.hyperledger.besu.ethereum.eth.manager.RequestManager$ResponseStream.processMessage(RequestManager.java:202)
        at org.hyperledger.besu.ethereum.eth.manager.RequestManager.lambda$dispatchResponse$0(RequestManager.java:82)
        at java.base/java.util.Optional.ifPresentOrElse(Optional.java:196)
        at org.hyperledger.besu.ethereum.eth.manager.RequestManager.dispatchResponse(RequestManager.java:81)
        at org.hyperledger.besu.ethereum.eth.manager.EthPeer.lambda$dispatch$7(EthPeer.java:411)
        at java.base/java.util.Optional.ifPresentOrElse(Optional.java:196)
        at org.hyperledger.besu.ethereum.eth.manager.EthPeer.dispatch(EthPeer.java:410)
        at org.hyperledger.besu.ethereum.eth.manager.EthPeers.dispatchMessage(EthPeers.java:256)
        at org.hyperledger.besu.ethereum.eth.manager.EthProtocolManager.processMessage(EthProtocolManager.java:320)
        at org.hyperledger.besu.ethereum.p2p.network.NetworkRunner.lambda$setupHandlers$1(NetworkRunner.java:142)
        at org.hyperledger.besu.ethereum.p2p.rlpx.connections.PeerConnectionEvents.lambda$dispatchMessage$1(PeerConnectionEvents.java:64)
        at org.hyperledger.besu.util.Subscribers.lambda$forEach$0(Subscribers.java:131)
        at java.base/java.lang.Iterable.forEach(Iterable.java:75)
        at org.hyperledger.besu.util.Subscribers.forEach(Subscribers.java:128)
        at org.hyperledger.besu.ethereum.p2p.rlpx.connections.PeerConnectionEvents.dispatchMessage(PeerConnectionEvents.java:64)
        at org.hyperledger.besu.ethereum.p2p.rlpx.connections.netty.ApiHandler.channelRead0(ApiHandler.java:99)
        at org.hyperledger.besu.ethereum.p2p.rlpx.connections.netty.ApiHandler.channelRead0(ApiHandler.java:33)
        at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
        at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:833)
2023-05-02 08:45:21.958-04:00 | nioEventLoopGroup-3-5 | INFO  | AbstractBlockProcessor | Block processing error: transaction invalid Internal Error in Besu - java.lang.IllegalStateException: Storage has been closed. Block 0x93bbbafad4ec34467c662e1d3cd0371febcc9fc60de7498d41fa841c2acfbb5b Transaction 0x791c4a4e4b7154da487d4f3822a2cc834c103b9931667d931e7e118629f805e0
2023-05-02 08:45:21.959-04:00 | nioEventLoopGroup-3-5 | INFO  | MainnetBlockValidator | Invalid block 8928642 (0x93bbbafad4ec34467c662e1d3cd0371febcc9fc60de7498d41fa841c2acfbb5b): Optional[Block processing error: transaction invalid Internal Error in Besu - java.lang.IllegalStateException: Storage has been closed. Block 0x93bbbafad4ec34467c662e1d3cd0371febcc9fc60de7498d41fa841c2acfbb5b Transaction 0x791c4a4e4b7154da487d4f3822a2cc834c103b9931667d931e7e118629f805e0]
2023-05-02 08:45:21.959-04:00 | nioEventLoopGroup-3-5 | ERROR | RocksDBColumnarKeyValueStorage | Attempting to use a closed RocksDbKeyValueStorage
2023-05-02 08:45:21.959-04:00 | nioEventLoopGroup-3-5 | ERROR | RocksDBColumnarKeyValueStorage | Attempting to use a closed RocksDbKeyValueStorage
2023-05-02 08:45:21.979-04:00 | nioEventLoopGroup-3-5 | INFO  | BackwardSyncContext | Current backward sync session failed, it will be restarted

and here is an example of the unviable fork messages that repeat in Nimbus:
NOT 2023-05-02 08:45:19.236-04:00 Received blocks from an unviable fork      topics="syncman" request=5543552:32@350 peer=16U*W8RdP5 direction=forward blockRoot=5d0ae092 blockSlot=5543552 blocks_count=22 blocks_map=xxxx..xxxx.x.x..xx.xxxxxxxx...xx sync_ident=main

while Nimbus reports receiving blocks from an unviable fork

Generally, this is because of incorrect-INVALID results from Besu, such as in the logs you show: https://prater.beaconcha.in/block/8928642 is the block in question. It's perfectly fine and Besu should not have reported it invalid. Once this happens, Nimbus is entirely correct in treating everything past that as INVALID because https://github.com/ethereum/consensus-specs/blob/dev/sync/optimistic.md#transitioning-from-valid---invalidated-or-invalidated---valid states

Transitioning from VALID -> INVALIDATED or INVALIDATED -> VALID
These operations are purposefully omitted. It is outside of the scope of the specification since it's only possible with a faulty EE.
Such a scenario requires manual intervention.

No descendent of an INVALID block can ever be VALID. Besu appears to be treating a closed storage system as a reason to return
2023-05-02 08:45:21.956-04:00 | nioEventLoopGroup-3-5 | ERROR | RocksDBColumnarKeyValueStorage | Attempting to use a closed RocksDbKeyValueStorage
2023-05-02 08:45:21.956-04:00 | ForkJoinPool.commonPool-worker-25 | ERROR | RocksDBColumnarKeyValueStorage | Attempting to use a closed RocksDbKeyValueStorage
2023-05-02 08:45:21.957-04:00 | nioEventLoopGroup-3-5 | ERROR | MainnetTransactionProcessor | Critical Exception Processing Transaction
java.lang.IllegalStateException: Storage has been closed

INVALID.
Whether that's the actual cause, or it's something else, INVALID is simply never a valid default return value here. It's not "the EL doesn't know, maybe it's INVALID?". The EL has to be sure the block is invalid.
And, the fallout of that is that it just stalls Nimbus syncing, again, correctly, per spec.

Updated the description with some targeted debug logging from some test runs.

While testing out an unrelated fix with the unstable branch of nimbus, I observed at least three separate backward sync issues on our sepolia validating node. Note, I've only seen this issue on Sepolia and it was probably triggered by nimbus not getting peers before falling > 64 blocks behind the beacon chain which switches it into "far wall" sync mode, rather than lock-step sync.
...
besu-23.1.2.log nimbus-ea8ef5-2.log
Nimbus versions differing behaviour
* The [unstable](https://github.com/status-im/nimbus-eth2/tree/unstable) branch I tested was what produced these logs

* [v23.4.0](https://github.com/status-im/nimbus-eth2/releases/tag/v23.4.0) seems to show similar behaviour

* [v23.3.2](https://github.com/status-im/nimbus-eth2/releases/tag/v23.3.2) didn't seem to have as many problems

Examples of incorrect-INVALID in those logs:
#1:
Nimbus:

WRN 2023-04-27 00:12:17.548+00:00 execution payload forkChoiceUpdated status INVALID, but was previously VALID payloadExecutionStatus=INVALID headBlockHash=8cf50d9a safeBlockHash=0267e06f finalizedBlockHash=19588bc2 receivedBlock="(blck: (slot: 2235058, proposer_index: 15, parent_root: "69b483c3", state_root: "9d065bb6", eth1data: (deposit_root: 9df92d765b5aa041fd4bbe8d5878eb89290efa78e444c1a603eecfae2ea05fa4, deposit_count: 403, block_hash: 52f93be4726098aba5e18ea6282274182e5c2d902d6550852897409d3c201701), graffiti: "teku-geth", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 2, deposits_len: 0, voluntary_exits_len: 0, sync_committee_participants: 462, block_number: 3368175, fee_recipient: "0xf97e180c050e5ab072211ad2c213eb5aee4df134"), signature: "89d17655")"

Besu:

{"@timestamp":"2023-04-27T00:20:36,081","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineForkchoiceUpdated","message":"INVALID for fork-choice-update: head: 0x8cf50d9a8331ce7637977c0284a9fe83f6c526f9e7f15abc119afcebd8ffc342, finalized: 0x19588bc2f191f4debcf82c84fb1a7be67cc6ed128fa25369fa58e97c9c858f97, safeBlockHash: 0x0267e06f6118049d930361a6a3632b96f7abeb49a792f2167cbbabd131133595","throwable":""}

Block is valid: https://sepolia.beaconcha.in/slot/2235058
#2:
Nimbus:
WRN 2023-04-27 00:25:51.944+00:00 execution payload forkChoiceUpdated status INVALID, but was previously VALID payloadExecutionStatus=INVALID headBlockHash=bd28bfd1 safeBlockHash=0267e06f finalizedBlockHash=19588bc2 receivedBlock="(blck: (slot: 2235059, proposer_index: 173, parent_root: \"94ada9d0\", state_root: \"e797fb26\", eth1data: (deposit_root: 9df92d765b5aa041fd4bbe8d5878eb89290efa78e444c1a603eecfae2ea05fa4, deposit_count: 403, block_hash: ce76841f95090ea8bab2c02c352c96af743547e7a8022fd5dd1cfd665fe2a0ce), graffiti: \"Nethermind\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 1, deposits_len: 0, voluntary_exits_len: 0, sync_committee_participants: 462, block_number: 3368176, fee_recipient: \"0x670b24610df99b1685aeac0dfd5307b92e0cf4d7\"), signature: \"ab13d979\")"
WRN 2023-04-27 00:25:52.008+00:00 execution payload forkChoiceUpdated status INVALID, but was previously VALID payloadExecutionStatus=INVALID headBlockHash=8be964af safeBlockHash=0267e06f finalizedBlockHash=19588bc2 receivedBlock="(blck: (slot: 2235060, proposer_index: 1229, parent_root: \"058b662e\", state_root: \"a1f74c72\", eth1data: (deposit_root: 9df92d765b5aa041fd4bbe8d5878eb89290efa78e444c1a603eecfae2ea05fa4, deposit_count: 403, block_hash: 52f93be4726098aba5e18ea6282274182e5c2d902d6550852897409d3c201701), graffiti: \"0xf09fa7b1f09f94a5204b696c6e2e666920f09f94a5f09fa7b100000000000000\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 1, deposits_len: 0, voluntary_exits_len: 0, sync_committee_participants: 462, block_number: 3368177, fee_recipient: \"0x1268ad189526ac0b386faf06effc46779c340ee6\"), signature: \"82c6d608\")"
WRN 2023-04-27 00:25:52.068+00:00 execution payload forkChoiceUpdated status INVALID, but was previously VALID payloadExecutionStatus=INVALID headBlockHash=ec9cad4d safeBlockHash=0267e06f finalizedBlockHash=19588bc2 receivedBlock="(blck: (slot: 2235061, proposer_index: 1779, parent_root: \"1101b3a5\", state_root: \"56383a9d\", eth1data: (deposit_root: 9df92d765b5aa041fd4bbe8d5878eb89290efa78e444c1a603eecfae2ea05fa4, deposit_count: 403, block_hash: 52f93be4726098aba5e18ea6282274182e5c2d902d6550852897409d3c201701), graffiti: \"teku/v23.3.1\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 1, deposits_len: 0, voluntary_exits_len: 0, sync_committee_participants: 462, block_number: 3368178, fee_recipient: \"0x8b0c2c4c8eb078bc6c01f48523764c8942c0c6c4\"), signature: \"b053cf59\")"

Besu:

{"@timestamp":"2023-04-27T00:25:51,944","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineForkchoiceUpdated","message":"INVALID for fork-choice-update: head: 0xbd28bfd13d3ddf2792e42c98bc048b8fcc89c918a028af49d519f285a5c321eb, finalized: 0x19588bc2f191f4debcf82c84fb1a7be67cc6ed128fa25369fa58e97c9c858f97, safeBlockHash: 0x0267e06f6118049d930361a6a3632b96f7abeb49a792f2167cbbabd131133595","throwable":""}
{"@timestamp":"2023-04-27T00:25:52,145","level":"WARN","thread":"vert.x-worker-thread-0","class":"DefaultBlockchain","message":"Chain Reorganization +34 new / -0 old
Old - hash: 0xbca29e861aabf1b975cde3ee6f147a86ad8de3bfab1a39bfb2d98594a72eb63d, height: 3368145
New - hash: 0xc08142d51de91ea6b34bc92f6358b87f838ada7c2e3a7b9dd94253c7cfdb0acb, height: 3368179
Ancestor - hash: 0xbca29e861aabf1b975cde3ee6f147a86ad8de3bfab1a39bfb2d98594a72eb63d, height: 3368145","throwable":""}
{"@timestamp":"2023-04-27T00:25:52,229","level":"WARN","thread":"nioEventLoopGroup-3-9","class":"DefaultBlockchain","message":"Chain Reorganization +0 new / -34 old
Old - hash: 0xc08142d51de91ea6b34bc92f6358b87f838ada7c2e3a7b9dd94253c7cfdb0acb, height: 3368179
New - hash: 0xbca29e861aabf1b975cde3ee6f147a86ad8de3bfab1a39bfb2d98594a72eb63d, height: 3368145
Ancestor - hash: 0xbca29e861aabf1b975cde3ee6f147a86ad8de3bfab1a39bfb2d98594a72eb63d, height: 3368145","throwable":""}
{"@timestamp":"2023-04-27T00:25:52,371","level":"WARN","thread":"vert.x-worker-thread-0","class":"DefaultBlockchain","message":"Chain Reorganization +36 new / -0 old
Old - hash: 0xbca29e861aabf1b975cde3ee6f147a86ad8de3bfab1a39bfb2d98594a72eb63d, height: 3368145
New - hash: 0x2a0d21df2f93d6a3675fff83c3c605cf471ea6250c35286a941dac71af85d31b, height: 3368181
Ancestor - hash: 0xbca29e861aabf1b975cde3ee6f147a86ad8de3bfab1a39bfb2d98594a72eb63d, height: 3368145","throwable":""}
{"@timestamp":"2023-04-27T00:25:52,484","level":"WARN","thread":"nioEventLoopGroup-3-9","class":"DefaultBlockchain","message":"Chain Reorganization +0 new / -36 old
Old - hash: 0x2a0d21df2f93d6a3675fff83c3c605cf471ea6250c35286a941dac71af85d31b, height: 3368181
New - hash: 0xbca29e861aabf1b975cde3ee6f147a86ad8de3bfab1a39bfb2d98594a72eb63d, height: 3368145
Ancestor - hash: 0xbca29e861aabf1b975cde3ee6f147a86ad8de3bfab1a39bfb2d98594a72eb63d, height: 3368145","throwable":""}
{"@timestamp":"2023-04-27T00:25:52,611","level":"WARN","thread":"vert.x-worker-thread-0","class":"DefaultBlockchain","message":"Chain Reorganization +38 new / -0 old
Old - hash: 0xbca29e861aabf1b975cde3ee6f147a86ad8de3bfab1a39bfb2d98594a72eb63d, height: 3368145
New - hash: 0xfd4c23bcf746c6e566cf6ba6c2d8554f28808f7266dbed622b5a5d8ac5b3a38e, height: 3368183
Ancestor - hash: 0xbca29e861aabf1b975cde3ee6f147a86ad8de3bfab1a39bfb2d98594a72eb63d, height: 3368145","throwable":""}
{"@timestamp":"2023-04-27T00:25:52,690","level":"WARN","thread":"nioEventLoopGroup-3-9","class":"DefaultBlockchain","message":"Chain Reorganization +0 new / -38 old
Old - hash: 0xfd4c23bcf746c6e566cf6ba6c2d8554f28808f7266dbed622b5a5d8ac5b3a38e, height: 3368183
New - hash: 0xbca29e861aabf1b975cde3ee6f147a86ad8de3bfab1a39bfb2d98594a72eb63d, height: 3368145
Ancestor - hash: 0xbca29e861aabf1b975cde3ee6f147a86ad8de3bfab1a39bfb2d98594a72eb63d, height: 3368145","throwable":""}

All of these supposedly INVALID blocks aren't, and same point as before applies to Nimbus correctly stalling syncing as a consequence: https://sepolia.beaconcha.in/slot/2235059 https://sepolia.beaconcha.in/slot/2235060 https://sepolia.beaconcha.in/slot/2235061
#3:
WRN 2023-04-27 00:12:13.989+00:00 execution payload forkChoiceUpdated status INVALID, but was previously VALID payloadExecutionStatus=INVALID headBlockHash=c34ff7b5 safeBlockHash=0267e06f finalizedBlockHash=054894b7 receivedBlock="(blck: (slot: 2235032, proposer_index: 1423, parent_root: \"28efccce\", state_root: \"9336298e\", eth1data: (deposit_root: 9df92d765b5aa041fd4bbe8d5878eb89290efa78e444c1a603eecfae2ea05fa4, deposit_count: 403, block_hash: 52f93be4726098aba5e18ea6282274182e5c2d902d6550852897409d3c201701), graffiti: \"Simply Staking\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 1, deposits_len: 0, voluntary_exits_len: 0, sync_committee_participants: 462, block_number: 3368151, fee_recipient: \"0x0000000000000000000000000000000000000000\"), signature: \"ab3dc5a7\")"
WRN 2023-04-27 00:12:14.048+00:00 execution payload forkChoiceUpdated status INVALID, but was previously VALID payloadExecutionStatus=INVALID headBlockHash=db594049 safeBlockHash=0267e06f finalizedBlockHash=054894b7 receivedBlock="(blck: (slot: 2235033, proposer_index: 114, parent_root: \"660716ac\", state_root: \"8d7b362d\", eth1data: (deposit_root: 9df92d765b5aa041fd4bbe8d5878eb89290efa78e444c1a603eecfae2ea05fa4, deposit_count: 403, block_hash: ce76841f95090ea8bab2c02c352c96af743547e7a8022fd5dd1cfd665fe2a0ce), graffiti: \"Nethermind\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 1, deposits_len: 0, voluntary_exits_len: 0, sync_committee_participants: 462, block_number: 3368152, fee_recipient: \"0x670b24610df99b1685aeac0dfd5307b92e0cf4d7\"), signature: \"a2c9bcca\")"
WRN 2023-04-27 00:12:14.112+00:00 execution payload forkChoiceUpdated status INVALID, but was previously VALID payloadExecutionStatus=INVALID headBlockHash=43d30cc0 safeBlockHash=0267e06f finalizedBlockHash=054894b7 receivedBlock="(blck: (slot: 2235034, proposer_index: 1249, parent_root: \"811a3060\", state_root: \"0e04acaa\", eth1data: (deposit_root: 9df92d765b5aa041fd4bbe8d5878eb89290efa78e444c1a603eecfae2ea05fa4, deposit_count: 403, block_hash: ce76841f95090ea8bab2c02c352c96af743547e7a8022fd5dd1cfd665fe2a0ce), graffiti: \"0xf09fa7b1f09f94a5204b696c6e2e666920f09f94a5f09fa7b100000000000000\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 1, deposits_len: 0, voluntary_exits_len: 0, sync_committee_participants: 462, block_number: 3368153, fee_recipient: \"0x1268ad189526ac0b386faf06effc46779c340ee6\"), signature: \"8881eea9\")"
WRN 2023-04-27 00:12:14.176+00:00 execution payload forkChoiceUpdated status INVALID, but was previously VALID payloadExecutionStatus=INVALID headBlockHash=adb70910 safeBlockHash=0267e06f finalizedBlockHash=054894b7 receivedBlock="(blck: (slot: 2235035, proposer_index: 1012, parent_root: \"da45258b\", state_root: \"cfe9fb70\", eth1data: (deposit_root: 9df92d765b5aa041fd4bbe8d5878eb89290efa78e444c1a603eecfae2ea05fa4, deposit_count: 403, block_hash: 52f93be4726098aba5e18ea6282274182e5c2d902d6550852897409d3c201701), graffiti: \"Goerli Nunki @q9f Lodestar\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 2, deposits_len: 0, voluntary_exits_len: 0, sync_committee_participants: 462, block_number: 3368154, fee_recipient: \"0xff58d746a67c2e42bcc07d6b3f58406e8837e883\"), signature: \"8b435ae4\")"
WRN 2023-04-27 00:12:14.236+00:00 execution payload forkChoiceUpdated status INVALID, but was previously VALID payloadExecutionStatus=INVALID headBlockHash=23d7c1eb safeBlockHash=0267e06f finalizedBlockHash=054894b7 receivedBlock="(blck: (slot: 2235037, proposer_index: 1417, parent_root: \"900797c2\", state_root: \"fe655c16\", eth1data: (deposit_root: 9df92d765b5aa041fd4bbe8d5878eb89290efa78e444c1a603eecfae2ea05fa4, deposit_count: 403, block_hash: 52f93be4726098aba5e18ea6282274182e5c2d902d6550852897409d3c201701), graffiti: \"Simply Staking\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 2, deposits_len: 0, voluntary_exits_len: 0, sync_committee_participants: 462, block_number: 3368155, fee_recipient: \"0x0000000000000000000000000000000000000000\"), signature: \"875b36b8\")"
WRN 2023-04-27 00:12:14.300+00:00 execution payload forkChoiceUpdated status INVALID, but was previously VALID payloadExecutionStatus=INVALID headBlockHash=64fe5dd3 safeBlockHash=0267e06f finalizedBlockHash=054894b7 receivedBlock="(blck: (slot: 2235038, proposer_index: 1834, parent_root: \"6fa89010\", state_root: \"9f14fba1\", eth1data: (deposit_root: 9df92d765b5aa041fd4bbe8d5878eb89290efa78e444c1a603eecfae2ea05fa4, deposit_count: 403, block_hash: 52f93be4726098aba5e18ea6282274182e5c2d902d6550852897409d3c201701), graffiti: \"Obol Distributed Validator\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 1, deposits_len: 0, voluntary_exits_len: 0, sync_committee_participants: 462, block_number: 3368156, fee_recipient: \"0x9a6034c84cd431409ac1a35278c7da36ffda53e5\"), signature: \"81407816\")"
WRN 2023-04-27 00:12:14.360+00:00 execution payload forkChoiceUpdated status INVALID, but was previously VALID payloadExecutionStatus=INVALID headBlockHash=c247ecf0 safeBlockHash=0267e06f finalizedBlockHash=054894b7 receivedBlock="(blck: (slot: 2235039, proposer_index: 1181, parent_root: \"371db712\", state_root: \"8baee00b\", eth1data: (deposit_root: 9df92d765b5aa041fd4bbe8d5878eb89290efa78e444c1a603eecfae2ea05fa4, deposit_count: 403, block_hash: ce76841f95090ea8bab2c02c352c96af743547e7a8022fd5dd1cfd665fe2a0ce), graffiti: \"grand rising\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 1, deposits_len: 0, voluntary_exits_len: 0, sync_committee_participants: 462, block_number: 3368157, fee_recipient: \"0x0000006916a87b82333f4245046623b23794c65c\"), signature: \"981a29c4\")"
WRN 2023-04-27 00:12:14.432+00:00 execution payload forkChoiceUpdated status INVALID, but was previously VALID payloadExecutionStatus=INVALID headBlockHash=a155d43f safeBlockHash=0267e06f finalizedBlockHash=054894b7 receivedBlock="(blck: (slot: 2235040, proposer_index: 381, parent_root: \"e9975ac1\", state_root: \"69dabd40\", eth1data: (deposit_root: 9df92d765b5aa041fd4bbe8d5878eb89290efa78e444c1a603eecfae2ea05fa4, deposit_count: 403, block_hash: 52f93be4726098aba5e18ea6282274182e5c2d902d6550852897409d3c201701), graffiti: \"Lighthouse/v3.5.0-0fb58a6\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 1, deposits_len: 0, voluntary_exits_len: 0, sync_committee_participants: 450, block_number: 3368158, fee_recipient: \"0x008b3b2f992c0e14edaa6e2c662bec549caa8df1\"), signature: \"a7ddf133\")"
WRN 2023-04-27 00:12:14.503+00:00 execution payload forkChoiceUpdated status INVALID, but was previously VALID payloadExecutionStatus=INVALID headBlockHash=7905e3bf safeBlockHash=0267e06f finalizedBlockHash=19588bc2 receivedBlock="(blck: (slot: 2235041, proposer_index: 1423, parent_root: \"9083b0e1\", state_root: \"a2587192\", eth1data: (deposit_root: 9df92d765b5aa041fd4bbe8d5878eb89290efa78e444c1a603eecfae2ea05fa4, deposit_count: 403, block_hash: 52f93be4726098aba5e18ea6282274182e5c2d902d6550852897409d3c201701), graffiti: \"Simply Staking\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 1, deposits_len: 0, voluntary_exits_len: 0, sync_committee_participants: 450, block_number: 3368159, fee_recipient: \"0x0000000000000000000000000000000000000000\"), signature: \"b2b4742b\")"
WRN 2023-04-27 00:12:14.561+00:00 execution payload forkChoiceUpdated status INVALID, but was previously VALID payloadExecutionStatus=INVALID headBlockHash=f214c0cf safeBlockHash=0267e06f finalizedBlockHash=19588bc2 receivedBlock="(blck: (slot: 2235042, proposer_index: 806, parent_root: \"55c17d1f\", state_root: \"abacaa06\", eth1data: (deposit_root: d70a234731285c6804c2a4f56711ddb8c82c99740f207854891028af34e27e5e, deposit_count: 0, block_hash: 491ebac1b7f9c0eb426047a495dc577140cb3e09036cd3f7266eda86b635d9fa), graffiti: \"\\\"prysm-k8s\\\"\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 1, deposits_len: 0, voluntary_exits_len: 0, sync_committee_participants: 462, block_number: 3368160, fee_recipient: \"0xd9a5179f091d85051d3c982785efd1455cec8699\"), signature: \"8abe189e\")"
WRN 2023-04-27 00:12:14.616+00:00 execution payload forkChoiceUpdated status INVALID, but was previously VALID payloadExecutionStatus=INVALID headBlockHash=2a71cdcc safeBlockHash=0267e06f finalizedBlockHash=19588bc2 receivedBlock="(blck: (slot: 2235043, proposer_index: 841, parent_root: \"eaa479f0\", state_root: \"82b660cb\", eth1data: (deposit_root: d70a234731285c6804c2a4f56711ddb8c82c99740f207854891028af34e27e5e, deposit_count: 0, block_hash: 491ebac1b7f9c0eb426047a495dc577140cb3e09036cd3f7266eda86b635d9fa), graffiti: \"\\\"prysm-k8s\\\"\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 1, deposits_len: 0, voluntary_exits_len: 0, sync_committee_participants: 462, block_number: 3368161, fee_recipient: \"0xd9a5179f091d85051d3c982785efd1455cec8699\"), signature: \"b25cfffa\")"
WRN 2023-04-27 00:12:14.676+00:00 execution payload forkChoiceUpdated status INVALID, but was previously VALID payloadExecutionStatus=INVALID headBlockHash=12bcbdce safeBlockHash=0267e06f finalizedBlockHash=19588bc2 receivedBlock="(blck: (slot: 2235044, proposer_index: 948, parent_root: \"a584be5e\", state_root: \"ba3354b8\", eth1data: (deposit_root: 9df92d765b5aa041fd4bbe8d5878eb89290efa78e444c1a603eecfae2ea05fa4, deposit_count: 403, block_hash: 52f93be4726098aba5e18ea6282274182e5c2d902d6550852897409d3c201701), graffiti: \"Attestant/Vouch\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 1, deposits_len: 0, voluntary_exits_len: 0, sync_committee_participants: 462, block_number: 3368162, fee_recipient: \"0x320715b08bcf4cac1df2c55288a6bad79da1566b\"), signature: \"aeb42efd\")"
WRN 2023-04-27 00:12:14.732+00:00 execution payload forkChoiceUpdated status INVALID, but was previously VALID payloadExecutionStatus=INVALID headBlockHash=feb805ad safeBlockHash=0267e06f finalizedBlockHash=19588bc2 receivedBlock="(blck: (slot: 2235045, proposer_index: 1519, parent_root: \"d70e7956\", state_root: \"ad179ba8\", eth1data: (deposit_root: 9df92d765b5aa041fd4bbe8d5878eb89290efa78e444c1a603eecfae2ea05fa4, deposit_count: 403, block_hash: 52f93be4726098aba5e18ea6282274182e5c2d902d6550852897409d3c201701), graffiti: \"teku/v23.2.0+14-g596a9acf9\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 1, deposits_len: 0, voluntary_exits_len: 0, sync_committee_participants: 462, block_number: 3368163, fee_recipient: \"0x6a7aa9b882d50bb7bc5da1a244719c99f12f06a3\"), signature: \"a8847012\")"
WRN 2023-04-27 00:12:14.784+00:00 execution payload forkChoiceUpdated status INVALID, but was previously VALID payloadExecutionStatus=INVALID headBlockHash=5e574a2b safeBlockHash=0267e06f finalizedBlockHash=19588bc2 receivedBlock="(blck: (slot: 2235046, proposer_index: 1427, parent_root: \"89f2001c\", state_root: \"f6fb3a71\", eth1data: (deposit_root: 9df92d765b5aa041fd4bbe8d5878eb89290efa78e444c1a603eecfae2ea05fa4, deposit_count: 403, block_hash: 52f93be4726098aba5e18ea6282274182e5c2d902d6550852897409d3c201701), graffiti: \"Simply Staking\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 1, deposits_len: 0, voluntary_exits_len: 0, sync_committee_participants: 462, block_number: 3368164, fee_recipient: \"0x0000000000000000000000000000000000000000\"), signature: \"b508956a\")"
WRN 2023-04-27 00:12:14.844+00:00 execution payload forkChoiceUpdated status INVALID, but was previously VALID payloadExecutionStatus=INVALID headBlockHash=eb7267c6 safeBlockHash=0267e06f finalizedBlockHash=19588bc2 receivedBlock="(blck: (slot: 2235047, proposer_index: 1398, parent_root: \"1ed94cdf\", state_root: \"174fe7b9\", eth1data: (deposit_root: 9df92d765b5aa041fd4bbe8d5878eb89290efa78e444c1a603eecfae2ea05fa4, deposit_count: 403, block_hash: 52f93be4726098aba5e18ea6282274182e5c2d902d6550852897409d3c201701), graffiti: \"Lighthouse/v4.1.0\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 1, deposits_len: 0, voluntary_exits_len: 0, sync_committee_participants: 462, block_number: 3368165, fee_recipient: \"0xc4bfccb1668d6e464f33a76badd8c8d7d341e04a\"), signature: \"915552c3\")"
WRN 2023-04-27 00:12:14.900+00:00 execution payload forkChoiceUpdated status INVALID, but was previously VALID payloadExecutionStatus=INVALID headBlockHash=46e86e3b safeBlockHash=0267e06f finalizedBlockHash=19588bc2 receivedBlock="(blck: (slot: 2235048, proposer_index: 322, parent_root: \"9f79cd08\", state_root: \"48177e0f\", eth1data: (deposit_root: 9df92d765b5aa041fd4bbe8d5878eb89290efa78e444c1a603eecfae2ea05fa4, deposit_count: 403, block_hash: 52f93be4726098aba5e18ea6282274182e5c2d902d6550852897409d3c201701), graffiti: \"Lighthouse/v3.5.0-0fb58a6\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 1, deposits_len: 0, voluntary_exits_len: 0, sync_committee_participants: 462, block_number: 3368166, fee_recipient: \"0x008b3b2f992c0e14edaa6e2c662bec549caa8df1\"), signature: \"a731a376\")"
WRN 2023-04-27 00:12:14.960+00:00 execution payload forkChoiceUpdated status INVALID, but was previously VALID payloadExecutionStatus=INVALID headBlockHash=2208824f safeBlockHash=0267e06f finalizedBlockHash=19588bc2 receivedBlock="(blck: (slot: 2235049, proposer_index: 1077, parent_root: \"11cabd31\", state_root: \"d1dbd15e\", eth1data: (deposit_root: 9df92d765b5aa041fd4bbe8d5878eb89290efa78e444c1a603eecfae2ea05fa4, deposit_count: 403, block_hash: 52f93be4726098aba5e18ea6282274182e5c2d902d6550852897409d3c201701), graffiti: \"Goerli Nunki @q9f Lodestar\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 2, deposits_len: 0, voluntary_exits_len: 0, sync_committee_participants: 462, block_number: 3368167, fee_recipient: \"0xff58d746a67c2e42bcc07d6b3f58406e8837e883\"), signature: \"917fafb1\")"
WRN 2023-04-27 00:12:15.016+00:00 execution payload forkChoiceUpdated status INVALID, but was previously VALID payloadExecutionStatus=INVALID headBlockHash=b71e5d6e safeBlockHash=0267e06f finalizedBlockHash=19588bc2 receivedBlock="(blck: (slot: 2235050, proposer_index: 1275, parent_root: \"eff7f854\", state_root: \"af69154a\", eth1data: (deposit_root: 9df92d765b5aa041fd4bbe8d5878eb89290efa78e444c1a603eecfae2ea05fa4, deposit_count: 403, block_hash: ce76841f95090ea8bab2c02c352c96af743547e7a8022fd5dd1cfd665fe2a0ce), graffiti: \"0xf09fa7b1f09f94a5204b696c6e2e666920f09f94a5f09fa7b100000000000000\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 1, deposits_len: 0, voluntary_exits_len: 0, sync_committee_participants: 462, block_number: 3368168, fee_recipient: \"0x1268ad189526ac0b386faf06effc46779c340ee6\"), signature: \"82198e3c\")"
WRN 2023-04-27 00:12:15.076+00:00 execution payload forkChoiceUpdated status INVALID, but was previously VALID payloadExecutionStatus=INVALID headBlockHash=9c124f54 safeBlockHash=0267e06f finalizedBlockHash=19588bc2 receivedBlock="(blck: (slot: 2235051, proposer_index: 823, parent_root: \"c479a69e\", state_root: \"626e5b22\", eth1data: (deposit_root: d70a234731285c6804c2a4f56711ddb8c82c99740f207854891028af34e27e5e, deposit_count: 0, block_hash: 491ebac1b7f9c0eb426047a495dc577140cb3e09036cd3f7266eda86b635d9fa), graffiti: \"\\\"prysm-k8s\\\"\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 1, deposits_len: 0, voluntary_exits_len: 0, sync_committee_participants: 462, block_number: 3368169, fee_recipient: \"0xd9a5179f091d85051d3c982785efd1455cec8699\"), signature: \"8eefda0f\")"
WRN 2023-04-27 00:12:15.132+00:00 execution payload forkChoiceUpdated status INVALID, but was previously VALID payloadExecutionStatus=INVALID headBlockHash=5f9f7758 safeBlockHash=0267e06f finalizedBlockHash=19588bc2 receivedBlock="(blck: (slot: 2235052, proposer_index: 1158, parent_root: \"42b48afe\", state_root: \"352b8316\", eth1data: (deposit_root: 9df92d765b5aa041fd4bbe8d5878eb89290efa78e444c1a603eecfae2ea05fa4, deposit_count: 403, block_hash: ce76841f95090ea8bab2c02c352c96af743547e7a8022fd5dd1cfd665fe2a0ce), graffiti: \"grand rising\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 1, deposits_len: 0, voluntary_exits_len: 0, sync_committee_participants: 462, block_number: 3368170, fee_recipient: \"0x0000006916a87b82333f4245046623b23794c65c\"), signature: \"b5518fb2\")"
WRN 2023-04-27 00:12:15.188+00:00 execution payload forkChoiceUpdated status INVALID, but was previously VALID payloadExecutionStatus=INVALID headBlockHash=36d6ac37 safeBlockHash=0267e06f finalizedBlockHash=19588bc2 receivedBlock="(blck: (slot: 2235053, proposer_index: 1945, parent_root: \"53bdba3a\", state_root: \"4c014413\", eth1data: (deposit_root: 9df92d765b5aa041fd4bbe8d5878eb89290efa78e444c1a603eecfae2ea05fa4, deposit_count: 403, block_hash: 52f93be4726098aba5e18ea6282274182e5c2d902d6550852897409d3c201701), graffiti: \"\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 1, deposits_len: 0, voluntary_exits_len: 0, sync_committee_participants: 462, block_number: 3368171, fee_recipient: \"0x4ff58ed19a4a4eb0440804b89cb6a1ec12e05979\"), signature: \"9035a6ac\")"
WRN 2023-04-27 00:12:15.248+00:00 execution payload forkChoiceUpdated status INVALID, but was previously VALID payloadExecutionStatus=INVALID headBlockHash=22e56550 safeBlockHash=0267e06f finalizedBlockHash=19588bc2 receivedBlock="(blck: (slot: 2235054, proposer_index: 803, parent_root: \"8613c18c\", state_root: \"e52700ec\", eth1data: (deposit_root: d70a234731285c6804c2a4f56711ddb8c82c99740f207854891028af34e27e5e, deposit_count: 0, block_hash: 491ebac1b7f9c0eb426047a495dc577140cb3e09036cd3f7266eda86b635d9fa), graffiti: \"\\\"prysm-k8s\\\"\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 1, deposits_len: 0, voluntary_exits_len: 0, sync_committee_participants: 462, block_number: 3368172, fee_recipient: \"0xd9a5179f091d85051d3c982785efd1455cec8699\"), signature: \"b9724e84\")"
WRN 2023-04-27 00:12:15.304+00:00 execution payload forkChoiceUpdated status INVALID, but was previously VALID payloadExecutionStatus=INVALID headBlockHash=5aa9ba86 safeBlockHash=0267e06f finalizedBlockHash=19588bc2 receivedBlock="(blck: (slot: 2235055, proposer_index: 1866, parent_root: \"44dc04b6\", state_root: \"e4dc7890\", eth1data: (deposit_root: 9df92d765b5aa041fd4bbe8d5878eb89290efa78e444c1a603eecfae2ea05fa4, deposit_count: 403, block_hash: 52f93be4726098aba5e18ea6282274182e5c2d902d6550852897409d3c201701), graffiti: \"Obol Distributed Validator\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 1, deposits_len: 0, voluntary_exits_len: 0, sync_committee_participants: 462, block_number: 3368173, fee_recipient: \"0x9a6034c84cd431409ac1a35278c7da36ffda53e5\"), signature: \"81542417\")"
WRN 2023-04-27 00:12:15.364+00:00 execution payload forkChoiceUpdated status INVALID, but was previously VALID payloadExecutionStatus=INVALID headBlockHash=cac157b4 safeBlockHash=0267e06f finalizedBlockHash=19588bc2 receivedBlock="(blck: (slot: 2235056, proposer_index: 469, parent_root: \"46c1c5d4\", state_root: \"7a68e763\", eth1data: (deposit_root: 9df92d765b5aa041fd4bbe8d5878eb89290efa78e444c1a603eecfae2ea05fa4, deposit_count: 403, block_hash: 52f93be4726098aba5e18ea6282274182e5c2d902d6550852897409d3c201701), graffiti: \"Lighthouse/v4.1.0-be8231b\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 1, deposits_len: 0, voluntary_exits_len: 0, sync_committee_participants: 462, block_number: 3368174, fee_recipient: \"0xc6e2459991bfe27cca6d86722f35da23a1e4cb97\"), signature: \"8dbbb1c3\")"

Which are https://sepolia.beaconcha.in/slot/2235032 https://sepolia.beaconcha.in/slot/2235033 https://sepolia.beaconcha.in/slot/2235034 https://sepolia.beaconcha.in/slot/2235035 https://sepolia.beaconcha.in/slot/2235037 https://sepolia.beaconcha.in/slot/2235038 https://sepolia.beaconcha.in/slot/2235039 https://sepolia.beaconcha.in/slot/2235040 https://sepolia.beaconcha.in/slot/2235041 https://sepolia.beaconcha.in/slot/2235042 https://sepolia.beaconcha.in/slot/2235043 https://sepolia.beaconcha.in/slot/2235044 https://sepolia.beaconcha.in/slot/2235045 https://sepolia.beaconcha.in/slot/2235046 https://sepolia.beaconcha.in/slot/2235047 https://sepolia.beaconcha.in/slot/2235048 https://sepolia.beaconcha.in/slot/2235049 https://sepolia.beaconcha.in/slot/2235050 https://sepolia.beaconcha.in/slot/2235051 https://sepolia.beaconcha.in/slot/2235052 https://sepolia.beaconcha.in/slot/2235053 https://sepolia.beaconcha.in/slot/2235054 https://sepolia.beaconcha.in/slot/2235055 https://sepolia.beaconcha.in/slot/2235056
all of which are are valid, despite Besu claiming otherwise in these logs.
#4:
WRN 2023-04-27 00:25:51.944+00:00 execution payload forkChoiceUpdated status INVALID, but was previously VALID payloadExecutionStatus=INVALID headBlockHash=bd28bfd1 safeBlockHash=0267e06f finalizedBlockHash=19588bc2 receivedBlock="(blck: (slot: 2235059, proposer_index: 173, parent_root: \"94ada9d0\", state_root: \"e797fb26\", eth1data: (deposit_root: 9df92d765b5aa041fd4bbe8d5878eb89290efa78e444c1a603eecfae2ea05fa4, deposit_count: 403, block_hash: ce76841f95090ea8bab2c02c352c96af743547e7a8022fd5dd1cfd665fe2a0ce), graffiti: \"Nethermind\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 1, deposits_len: 0, voluntary_exits_len: 0, sync_committee_participants: 462, block_number: 3368176, fee_recipient: \"0x670b24610df99b1685aeac0dfd5307b92e0cf4d7\"), signature: \"ab13d979\")"
WRN 2023-04-27 00:25:52.008+00:00 execution payload forkChoiceUpdated status INVALID, but was previously VALID payloadExecutionStatus=INVALID headBlockHash=8be964af safeBlockHash=0267e06f finalizedBlockHash=19588bc2 receivedBlock="(blck: (slot: 2235060, proposer_index: 1229, parent_root: \"058b662e\", state_root: \"a1f74c72\", eth1data: (deposit_root: 9df92d765b5aa041fd4bbe8d5878eb89290efa78e444c1a603eecfae2ea05fa4, deposit_count: 403, block_hash: 52f93be4726098aba5e18ea6282274182e5c2d902d6550852897409d3c201701), graffiti: \"0xf09fa7b1f09f94a5204b696c6e2e666920f09f94a5f09fa7b100000000000000\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 1, deposits_len: 0, voluntary_exits_len: 0, sync_committee_participants: 462, block_number: 3368177, fee_recipient: \"0x1268ad189526ac0b386faf06effc46779c340ee6\"), signature: \"82c6d608\")"
WRN 2023-04-27 00:25:52.068+00:00 execution payload forkChoiceUpdated status INVALID, but was previously VALID payloadExecutionStatus=INVALID headBlockHash=ec9cad4d safeBlockHash=0267e06f finalizedBlockHash=19588bc2 receivedBlock="(blck: (slot: 2235061, proposer_index: 1779, parent_root: \"1101b3a5\", state_root: \"56383a9d\", eth1data: (deposit_root: 9df92d765b5aa041fd4bbe8d5878eb89290efa78e444c1a603eecfae2ea05fa4, deposit_count: 403, block_hash: 52f93be4726098aba5e18ea6282274182e5c2d902d6550852897409d3c201701), graffiti: \"teku/v23.3.1\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 1, deposits_len: 0, voluntary_exits_len: 0, sync_committee_participants: 462, block_number: 3368178, fee_recipient: \"0x8b0c2c4c8eb078bc6c01f48523764c8942c0c6c4\"), signature: \"b053cf59\")"

which are https://sepolia.beaconcha.in/slot/2235059 https://sepolia.beaconcha.in/slot/2235060 https://sepolia.beaconcha.in/slot/2235061 also all valid.
#5:
WRN 2023-04-27 00:26:29.832+00:00 execution payload forkChoiceUpdated status INVALID, but was previously VALID payloadExecutionStatus=INVALID headBlockHash=4da34cad safeBlockHash=a155d43f finalizedBlockHash=0267e06f receivedBlock="(blck: (slot: 2235090, proposer_index: 1552, parent_root: \"8197deb2\", state_root: \"655e2df8\", eth1data: (deposit_root: 9df92d765b5aa041fd4bbe8d5878eb89290efa78e444c1a603eecfae2ea05fa4, deposit_count: 403, block_hash: 52f93be4726098aba5e18ea6282274182e5c2d902d6550852897409d3c201701), graffiti: \"teku-geth\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 1, deposits_len: 0, voluntary_exits_len: 0, sync_committee_participants: 462, block_number: 3368205, fee_recipient: \"0xf97e180c050e5ab072211ad2c213eb5aee4df134\"), signature: \"8fe23b29\")"

which is valid: https://sepolia.beaconcha.in/slot/2235090
In all of these cases, Besu is still effectively closing off the rest of the actually-valid/finalized chain from Nimbus via https://github.com/ethereum/consensus-specs/blob/dev/sync/optimistic.md#transitioning-from-valid---invalidated-or-invalidated---valid and claiming that VALID blocks are INVALID.
Nimbus forgets these (not actually...) "unviable" forks every restart, as it does not persist this particular cache, so restarting Nimbus appears to help briefly, but soon enough, these logs show Besu finding some other block to falsely label as INVALID.

(2) Besu is "in sync" as evidenced by being able to propose but then is frequently being switched to backwards sync mode presumably by being given a hash that we don't have a parent for.
(3) The reorg logs, which suggests it's being thrashed between forks potentially...
(2) and (3) both could be same root cause, or maybe all 3 are same cause.

Separating replies somewhat for easier readability between the enumerating-of-examples one and this: when Nimbus is told, repeatedly, that the (CL) blocks the rest of the Sepolia chain is claiming are correct aren't by Besu by way of INVALID execution payloads, it's prone to do something which looks a bit from the EL side like a sequence of reorgs as it tries to find a non-INVALID chain to follow.

Hi @tersec, I agree with your analysis of besu's INVALID responses.
Besu marking blocks as INVALID due to World State Root does not match or invalid transaction nonce error is most likely a Besu data inconsistency error. In certain cases, it may even indicate that the database was corrupted and may not be recoverable. I think in most of my tests, the situation was recoverable but still caused some havoc in the syncing process. Essentially, once we start incorrectly marking blocks as INVALID all bets are off. We need to address the root cause in Besu for this class of errors.
For the tests linked in this issue, I can't explain the exact cause that led to INVALID, but after performing tests with my PR, I have not seen the "invalid transaction nonce" error and have only seen one instance of the "World State Root does not match" error so an improvement has been made with PR (and should be released this week).
My PR focused on fixing the "reorg" warnings during backwards sync (Observation (3) above), which indicated Besu was moving the head by a large number of blocks. This bug was due to some errant block hash that we were holding onto in our state and trying to move the head to it.
I have created an issue for the remaining Besu error: https://github.com/hyperledger/besu/issues/5622

@tersec While testing my PR, I have been collecting engine api logs via a proxy and have some analysis to share.
Summary
The most important finding shows that nimbus sometimes sends batches of out of order newPayloads before sending a FCU.
The batches are often in reverse order but with the FCU associated with the batch matching the final payload (with lowest blockNumber).
Some other clients send newPayload batches too but in sequential order, so I believe the newPayload ordering is responsible for a lot of the issues I'm still seeing. I can provide some detailed examples from logs if required.
Next steps for the original Observations raised:
❓  (1) I believe Nimbus sending newPayloads in sequential order should resolve this.
❓  (2) I believe not triggering overlapping backwards sync sessions would solve this (maybe besu could do optimisations to detect these scenarios and ignore them too)
✅  (3) I believe my PR fixes the large reorg logs
Besu Backwards Sync
For clarity I'll start with defining "backwards sync" from a besu point of view...
When besu receives an EL block hash from a newPayload or an FCU and it doesn't have that EL block's parent in its database, it starts a backwards sync session.
The backward sync session will attempt to fill the gap in the chain between the latest imported EL block and the hash it has just received from the CL.
It fills this gap by querying other EL peers first for the headers "Backwards sync phase 1 of 2" (the actual backwards part, using the parent hash to traverse up the chain).
Once it has filled the gap with headers, it queries the EL peers for the EL block bodies and executes them in a "ForwardSyncStep", logged as "Backwards sync phase 2 of 2".
Once Besu has received this first hash, Besu does not require intervention from the CL in order to fill this gap, however it may ignore subsequent newPayload/FCUs while it is syncing if it already has that block imported, or it has already started a backwards sync session due to it.
Observation (1) - backwards sync triggered instead of lock-step
As stated in the summary, we receive batches of newPayload (with 1 FCU per batch) in reverse order by block number.
For example, when besu's latest block is 7, nimbus sent in this order:

newPayload[blockHash_10],
newPayload[blockHash_9],
newPayload[blockHash_8],
FCU[blockHash_8]

We don't have the parent of block_10 in our chain so we start a backwards sync session upon receipt of that first newPayload.
I had previously stated on Discord that other clients send besu a single hash and wait for it to sync; this appears to only be the case during initial sync so I was mistaken w.r.t. the scenario being discussed here: client restarts/outages/recovery rather than initial sync behaviour.
In fact, when either EL or CL is restarted, other clients have a range of strategies that are some variant of "lock-step".
Sometimes these also cause besu to backwards sync, but newPayloads are sent in order and we don't get the overlapping backwards sync sessions.
Observation (2) - overlapping backwards sync sessions
Sometimes, before besu completes its current batch of backwards sync, nimbus gives it a different hash to sync. I observe that nimbus first sends hash_n then after some time sends hash_n-1, then hash_n-2, etc.
This may also be related to the reverse order newPayloads (although could happen with any out of order "future" hashes sent via newPayload or FCU).
Since besu's backward sync algorithm is multithreaded, we end up with multiple threads of work with overlapping batches of blocks. I think this explains the odd and out of order "percentage completed" logs.
For example, let's say besu is on block 5 out of 10, I've seen behaviour like this...

nimbus --- newPayload[blockHash_8] ---> besu starts a backwards sync session to fill the gap from block 5 - 8
nimbus --- newPayload[blockHash_7] ---> besu starts a concurrent backwards sync session to fill the gap from block 5 - 7

So nimbus isn't giving besu a chance to complete the backwards sync session before it requests a new one.
Couple of ways besu would behave better in this scenario:
(i) sending newPayloads in order may avoid the majority of these cases.
(ii) If nimbus resends newPayload[blockHash_8] then besu will disregard since it can detect it has already started a backwards sync session to cover all blocks up to this hash. Not sure there's any merit to doing that from a nimbus POV.
(iii) If nimbus checks which block besu is on by using eth_blockNumber/eth_getBlockByHash or similar then maybe it can send a more appropriate hash for besu to sync to. I know you'd mentioned safety issue with this before but if nimbus knows it has sent an FCU to besu then isn't it reasonable to wait until besu has at least reached that hash before sending another?
eth_getLogs
Finally, there's the adjacent issue of lots of eth_getLogs request, which appeared to coincide with the issues observed.
Out of all the CL clients, nimbus sends us the most of these, I think because it rebuilds them from scratch every time either besu or nimbus is restarted (which is maybe a knock-on effect of besu being in "syncing" mode when it receives out of order newPayload).
I believe other clients must be persisting this information across restarts.
I don't think it's a problem in itself, other than the potential to cause besu resource constraints if it's having to do lots of other work at the same time.

eth_getLogs
Finally, there's the adjacent issue of lots of eth_getLogs request, which appeared to coincide with the issues observed.
Out of all the CL clients, nimbus sends us the most of these, I think because it rebuilds them from scratch every time either besu or nimbus is restarted (which is maybe a knock-on effect of besu being in "syncing" mode when it receives out of order newPayload). I believe other clients must be persisting this information across restarts. I don't think it's a problem in itself, other than the potential to cause besu resource constraints if it's having to do lots of other work at the same time.

This is true for Nimbus, yeah. It's possible that https://eips.ethereum.org/EIPS/eip-4881 will change this at some point, but that's how it currently works.

Something else we could try in besu is not triggering backwards sync when we receive a newPayload (only when receiving FCU)

nimbus sometimes sends batches of out of order newPayloads before sending a FCU.
The batches are often in reverse order but with the FCU associated with the batch matching the final payload (with lowest blockNumber).

Hi @tersec what do you think about this ^?
Would it be difficult to send the batches newPayloads in ascending order?
I understand the order of newPayloads isn't mandated in the spec, but it seems like a bug to me to send a batch and then the FCU for the oldest rather than the latest one.

Is this issue causing what I experience here:
https://github.com/status-im/nimbus-eth2/issues/5145
?

Is this issue causing what I experience here: status-im/nimbus-eth2#5145 ?

@kiss81 IIUC your issue is with nimbus reporting corrupted deposit logs? I have not experienced that issue, so not sure it's related.
The "backward syncing" referred to in this issue is about nimbus helping besu get its blocks back into sync after an outage.
The deposit logs (eth_getLog) are relevant because nimbus requests them every time either client is restarted, but I don't believe they are the cause of any backing syncing issues.
Feel free to post more of your besu logs in a new issue and we can see if we can spot anything that might help.

Something else we could try in besu is not triggering backwards sync when we receive a newPayload (only when receiving FCU)

I tested this out in #5666 and while it did eventually get back into sync, it took 2 hours following a simple restart of besu, so I don't think this is a silver bullet, and arguably worse than before.
Going to close this for now as I think any further progress in the short term may need to come from the nimbus side. We have a medium term goal of improving/rewriting our backward sync procedure, so not syncing on newPayload is certainly something we could explore further as part of that.

There's also been some more recent testing https://github.com/hyperledger/besu/pull/5699#issuecomment-1658497632
which has shown some changes in how nimbus send newPayloads, but the outcome is the same.

https://github.com/status-im/nimbus-eth2/pull/5248 should help here, in response to https://github.com/status-im/nimbus-eth2/issues/5242

status-im/nimbus-eth2#5248 should help here, in response to status-im/nimbus-eth2#5242

cc @ahamlat

have we been able to reproduce on besu latest?

I have not noticed this issue for some time - e.g. testnet nodes seem to recover OK after a restart/update, so suspect it's resolved. Can reopen if necessary


Update invision-links.md

Removed Areas wireframe since it is a duplicate of Areas - Assign Area wireframe.
Removed Areas - Settings and renamed Areas - Create New Area to Areas - Manage since this functionality is all collapsed into one set of wireframes and screens.
Removed Areas - Assign Area visual design line since visual design is not required.

@mindreeper2420 - Brian and I reviewed and cleaned up the Areas listings based on where we are.  Please review and let us know if you agree with the changes.  We'll need to get the Areas - manage (contains add areas, remove areas, manage area settings in whatever other way) wireframes reviewed and agreed upon before we move onto visuals for that one story.


Celery docker instance started with invenio-cli containerize --pre keeps restarting for "no such option: -A"

Package version: invenio-cli 0.17.2
Describe the bug
Celery docker instance created on macOS with "invenio-cli containerize --pre" does not start and in the logs you have:
You are using `-A` as an option of the worker sub-command:
celery worker -A celeryapp <...>

The support for this usage was removed in Celery 5.0. Instead you should use `-A` as a global option:
celery -A celeryapp worker <...>
Usage: celery worker [OPTIONS]
Try 'celery worker --help' for help.

Error: no such option: -A

Steps to Reproduce

on macOS, init preview env with python 3.7 and postgres (other combinations untested)
start containers with "invenio-cli containerize --pre"
check docker ps and Celery logs

Expected behavior
Celery to correctly start (or scripts to pick correct version)

After reproducing these steps I got the same behavior as described.
Log from docker ps shows:
6c6a5485a5a3        my-site                                                   "bash -c 'celery wor…"   2 hours ago         Restarting (2) 21 seconds ago
Log from the container:
celery worker -A celeryapp <...>
The support for this usage was removed in Celery 5.0. Instead you should use -A as a global option:
celery -A celeryapp worker <...>
Usage: celery worker [OPTIONS]
Try 'celery worker --help' for help.
Error: no such option: -A
You are using -A as an option of the worker sub-command:
celery worker -A celeryapp <...>

So it's fixed in the master branch of cookiecutter-invenio-rdm. It will be available in the upcoming October release.


Dont test archetype to fix build

After investing several hours to find out why the latest builds are not working anymore - even in case of no changes: https://github.com/terrestris/shogun2/pull/261 - the only solution to fix the builds currently seems to NOT test the webapp archetype.
For some reason the maven on the travis machine is not able to access the archetype from the local repository (even though it has just been installed there one step before).

This will fix https://github.com/terrestris/shogun2/pull/260

Please merge. Thanks.


Can Azure Storage Resources (Blobs / Tables) be accessed from Containers?

I have a Windows nanoserver container running a spring-boot application that accesses azure table storage from within a spring-boot CommandLineRunner. The container runs fine locally (Docker / Win10 Pro). When I deploy to a container instance, a StorageException is thrown at the point that an attempt is made to retrieve a DynamicTableEntity from a table. The connection to the storage account and the table appear to succeed, but the retrieve operation fails. The Tomcat server is running and responsive (I catch the exception and prepare basic error information to be returned by servlets that manage requests). I have been unable to find any examples or definitive statements about using the Java Azure storage libraries in containers, so I would like to know if there are any restrictions.
Thanks.

Further to my earlier post, the storage exception is apparently being thrown due to an "unknown host". See logs below. The CommandLineRunner: entries reflect logging from my code - storage account credentials are parsed and a reference to the table obtained without error (both of course using HTTPS encapsulated by version 8 of Java storage libraries). When testing the container on a local machine, the table is accessible.
2019-05-13 00:57:44.632  INFO 3060 --- [           main] a.c.n.sbtest.demo.DemoApplication        : CommandLineRunner: Engine Intialization Ready (status: 0) 2019-05-13 00:57:44.670  INFO 3060 --- [           main] a.c.n.sbtest.demo.DemoApplication        : CommandLineRunner: Deployment table <Deployments> accessible 2019-05-13 00:57:44.687  INFO 3060 --- [           main] a.c.n.sbtest.demo.DemoApplication        : CommandLineRunner: User <e7c6163e-8c06-4146-b0e4-78fec93e6cce> Container accessible 2019-05-13 01:00:17.755  INFO 3060 --- [nio-8080-exec-7] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring DispatcherServlet 'dispatcherServlet' 2019-05-13 01:00:17.755  INFO 3060 --- [nio-8080-exec-7] o.s.web.servlet.DispatcherServlet        : Initializing Servlet 'dispatcherServlet' 2019-05-13 01:00:17.768  INFO 3060 --- [nio-8080-exec-7] o.s.web.servlet.DispatcherServlet        : Completed initialization in 13 ms com.microsoft.azure.storage.StorageException:         at com.microsoft.azure.storage.StorageException.translateException(StorageException.java:87)         at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:209)         at com.microsoft.azure.storage.table.QueryTableOperation.performRetrieve(QueryTableOperation.java:178)         at com.microsoft.azure.storage.table.TableOperation.execute(TableOperation.java:694)         at com.microsoft.azure.storage.table.CloudTable.execute(CloudTable.java:529)         at com.microsoft.azure.storage.table.CloudTable.execute(CloudTable.java:496)         at ai.com.neuralstudio.sbtest.demo.DemoApplication.lambda$commandLineRunner$0(DemoApplication.java:562)         at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:813)         at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:797)         at org.springframework.boot.SpringApplication.run(SpringApplication.java:324)         at org.springframework.boot.SpringApplication.run(SpringApplication.java:1260)         at org.springframework.boot.SpringApplication.run(SpringApplication.java:1248)         at ai.com.neuralstudio.sbtest.demo.DemoApplication.main(DemoApplication.java:397)         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)         at java.lang.reflect.Method.invoke(Method.java:498)         at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)         at org.springframework.boot.loader.Launcher.launch(Launcher.java:87)         at org.springframework.boot.loader.Launcher.launch(Launcher.java:50)         at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:51) Caused by: java.net.UnknownHostException: neuralstudiotest002.table.core.windows.net         at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)         at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)         at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)         at java.net.Socket.connect(Socket.java:589)         at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:673)         at sun.net.NetworkClient.doConnect(NetworkClient.java:175)         at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)         at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)         at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264)         at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367)         at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)         at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1156)         at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1050)         at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)         at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1564)         at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)         at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)         at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:347)         at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:115)         ... 19 more
This is extremely frustrating - it takes 4-5 hours to upload an image to the repository, and debugging options are limited in the cloud.
I would greatly appreciate any insights into why table access is not working.


Bring back renode emulation support

Removed in 636a9a2e8539a11c4ff216625e18c4078161e243
See the source from prior:
https://github.com/jonlamb-gh/air-gradient-pro-rs/tree/003fb2252eba75097c83b5dd327f52c816dbe6b3
Needs peripheral support for PMS and S8 LP.
Some reference stuff on this project I was hacking on: https://github.com/jonlamb-gh/stm32f4-rtic-renode

Added in fd9841db8a62e690c6bf0de1d6761d14dff091fc


Changed the Contributing Readme

Issue #58
As u told to be under Contributing Readme
Please tell if it is correct or I need to make further any changes

@Satyam1923 Please give level1 and gssoc here in PR also orelse my contribution won't be counted  and please ensure that l is small letter


Network: geneve tunnel ID fix

Description
This PR fixes an issue about the creation of the gateway geneve interfaces. The issue is that the controller is not able to get the correct genevetunnel resource.

/merge


Very new user: Calculate the cartesian atomic coordinates for a molecule

All,
I would like to calculate for a given molecule the cartesian atomic coordinates in Angstroms.  Would it be possible to have a working example to do this that would allow me to vary the input molecule and get as an output, ideally in a simple text file, the above?
I would very much appreciate any help here, I am new to the code.  I appreciate I will need to understand eventually aspects relating to optimisation options and the model used PM3/PM7.  But a working example with trusted defaults would be most welcome.
Many thanks,

What is the file format your molecule of interest is described?  Because there are a couple of them, the most elementary the .xyz, where atoms' positions are described in a Cartesian coordinate system with (an implicit) scale in Angstrom.  See e.g., openbabel.
If "calculate for a given molecule the cartesian atomic coordinates" refers to a conversion of the underlying file format, openbabel (with its optional gui) might suffice.  Examples of conversion are compiled here.
If you seek a GUI to set up a subsequent computation for instance with MOPAC, avogadro and its input generators is one example of a helpful assistant:

Hi,
Thank you for your reply.  I'm confused, I do not have a file format for the input molecule.  But that is because I'm very new to this and don't know what I'm doing.
In the example you provide above, I would want to start with the knowledge that I know I want the cartesian atomic coordinates of CH4, and perform a computation to calculate those.  In my case I only know the molecule that I wish to calculate, I do not have an upstream code. I do not know why I need an input file.
Said another way, is it possible to provide as input {"C": 1, "H": 4} and from this obtain the cartesian atomic coordinates entirely in MOPAC?  I'm sorry about this, I'm quite confused. Hope you can help further.


Unused functions abort compilation

If an unused (never called) function is defined inside a JIT decorated function, an UnsuportedError is thrown.
Even if the workaround (commenting the unused definition) is very simple, I believe this should be taken as a bug,
Example (with fictitious useless functions)
from numba import jit

@jit()
def NoFail(n):
    def minusone(p):
        return p-1
    return n*minusone(n) # minusone is called, no error thrown

@jit()
def Fail(n):
    def minusone(p): # this function is defined but never called
        return p-1
    return n*(n-1)

print NoFail(3)
print Fail(3) # error raised here

Thanks for the report. I'm fairly sure this has been reported before, I'll have a look and see if I can cross reference the issue. The reason for what you are seeing is as follows...

In the NoFail function a call is made to the inner function, this means that type information can propagate to it and the compiler pass that inlines functions will be able to inline the inner function at the call site.
In the Fail function, as no call is made to the inner function there is no type information available (what type is p? how should it be compiled?) and there is no way or need to inline the function as it is not called, as a result the inner function remains as a declared but unused function. The result of this is that a make_function op code still exists in the Fail function at the time Numba's legalization compiler pass runs, this picks it up as something that is not supported and consequently raises the exception reported.

I think the way to fix this would be to run a dead code elimination pass prior to legalization. This pass ends up running (for other reasons) in the case of decorating the function with @jit(parallel=True) and as a result the code works fine!
@jit(parallel=True)
def WorksDueToDCE(n):
    def minusone(p): # this function is defined but never called
        return p-1
    return n*(n-1)

Cross reference: https://github.com/numba/numba/issues/2866

This issue is now caught and trying to compile the above Failing function produces:
numba.errors.UnsupportedError: Failed in object mode pipeline (step: ensure IR is legal prior to lowering)
Numba encountered the use of a language feature it does not support in this context: minusone (op code: make_function not supported). If the feature is explicitly supported it is likely that the result of the expression (minusone) is being used in an unsupported manner.

Closing this issue as it seems to be resolved. If this is not the case please re-open with a comment about any item that appears to be unresolved. Many thanks.


Fix commands deprecations

Partially fixes #124 (imho SimplePreAuthenticatorInterface deprecation should be in a separate PR)
Increased symfony dependencies to 3.4 because 3.3 is EOL'ed and 3.4 is a LTS release

@norkunas Can we get these changes tagged please? I'm not sure if this causes any kind of B.C. breaks to require a new major version, but I don't like the idea going to production with my application with tons of deprecations.

Sorry, but it's not up to me :)


Support Stencil 2.0

Hello,
Stencil 2.0 was released yesterday, I was wondering if this plugin will later support it.
If yes, how could I help to make it happen. I will begin a project in the coming weeks and your plugin will be extremely important for us.
Thanks for this excellent plugin !

Hi,
The next update I'll publish will use 2.0 as default. The plugin uses the stencil package available, so there's no direct dependency. The schematic installs version 1 but version 2 comes without breaking changes. I did some tests with the version 2 beta, and it worked, so you can try v2 with just updating the package.json.
Thanks and great that the plugin helps. If you've any problems, let me know.

Super ! @DominikPieper  Thanks for the quick answer. I will try it soon.


add some german shopping sites

Especially "kleinanzeigen.de". Because that's a big one.

Thanks!


@angular/cli gives error if node_modules is placed in parent folder

Probably related to https://github.com/angular/angular-cli/issues/3864

Bug Report or Feature Request (mark with an x)
- [x] bug report -> please search issues before submitting
- [ ] feature request

Command (mark with an x)
- [ ] new
- [ ] build
- [x] serve
- [ ] test
- [ ] e2e
- [ ] generate
- [ ] add
- [ ] update
- [ ] lint
- [ ] xi18n
- [ ] run
- [ ] config
- [ ] help
- [ ] version
- [ ] doc

Versions

node version: v8.11.3
npm version: 5.5.1
     _                      _                 ____ _     ___
    / \   _ __   __ _ _   _| | __ _ _ __     / ___| |   |_ _|
   / △ \ | '_ \ / _` | | | | |/ _` | '__|   | |   | |    | |
  / ___ \| | | | (_| | |_| | | (_| | |      | |___| |___ | |
 /_/   \_\_| |_|\__, |\__,_|_|\__,_|_|       \____|_____|___|
                |___/

Angular CLI: 6.1.2
Node: 8.11.3
OS: win32 x64
Angular:
...

Package                      Version
------------------------------------------------------
@angular-devkit/architect    0.7.2
@angular-devkit/core         0.7.2
@angular-devkit/schematics   0.7.2
@schematics/angular          0.7.2
@schematics/update           0.7.2
rxjs                         6.2.2
typescript                   2.7.2

Repro steps

Clone the project (project is generated with angular-cli, no other modules are added)
npm install in parent folder (node-modules-parent-folder)
cd frontend for angular project folder
ng serve to build and run

The log given by the failure

You seem to not be depending on "@angular/core" and/or "rxjs". This is an error.
Desired functionality

It should automatically navigate to parent folder (as per node resolution) and find the @angular/core and rxjs.
Mention any other details that might be useful

Issue should've been fixed based on this commit https://github.com/angular/angular-cli/issues/3864#issuecomment-409318373 which is merged in 6.1.2
https://github.com/Magnetars/node-modules-parent-folder.git
The project is a base ng project, no other modules are added. (You could create a new project instead of cloning)
"$schema"<EMAIL_ADDRESS>is changed to point to "$schema"<EMAIL_ADDRESS>(../node_modules instead of ./node_modules)

Please note that while this could simply be solved by installing the node_modules in the Angular project folder, since my project is using server-side-rendering (using Angular Universal), it's more convenient to have the node_modules in the parent folder shared by both frontend and backend
My project structure is something like this:
+-- node-modules-parent-folder <-------- project folder
|     +-- package.json <-------------------- package.json moved to parent folder
|     +-- node_modules
|     +-- frontend <------------------------ Angular project goes here
|           +-- angular.json
|     +-- backend  <------------------------ Server code goes here
|          +-- server.js

You can try using version < 6.1.0 and it will work. I think this issue is the same as this https://github.com/angular/angular-cli/issues/11685

Indeed you are right, but instead I went with temporarily having node_modules in both the Angular project folder and the parent project folder, this way I can keep developing against the latest version (6.1.2 as of now) to avoid any sudden happy surprises if I migrate through multiple versions later.
But for people whom this is not a viable solution (I believe people using Docker can't install node_modules in the Angular project folder), then you should go with version < 6.1.0 as mentioned by @jSchez since that's where the problem first regressed

I hope that they fix this .. monorepos are really cool and I don’t like experimenting either because as you mention, they may change the way it works and then you will habe to redo everything again 😥

Thanks all, I agree it's a dupe of #11685 looks like we need a test case for this


PipEnv Development dependencies not included in requirements.txt

Behavior of pipenv lock -r has been changed.
See: https://github.com/pypa/pipenv/issues/942
We now require another pipenv lock -r -d to produce a requirements.txt that includes dev dependencies required for the build.
An example would be pipenv --dev Cython

After reading the discussion in here, https://github.com/pypa/pipenv/pull/972 , the API behavior might change sometime in the future. We need a future proof way of fixing this like in https://github.com/ONSdigital/eq-survey-runner/pull/1327

@dschep Should Cython be a dev dependency? Wished pipenv had an option to include it as a build dependency.
I would suggest that the correct build behavior going forward should be the following:
pipenv lock --requirements > requirements.txt
pipenv lock --dev > requirements-dev.txt
pip install -r requirements-dev.txt
pip install -r requirements.txt
pip uninstall -r requirements-dev.txt
echo 'Zip artifacts...'

Hmm. yeah, as you saw in #87 we don't want to include dev deps in the deployment. I haven't used Cython personally, if I understand right, it needs to installed when pip install is called, is that right?

Yup. Cython needs to be installed first before doing pip install of other packages that detect the presence of Cython (before compiling optimizations automatically).
Example:
pip install cython
pip install cymysql

Cool, thanks. This is pretty tricky to address afaict. dev reqs with pipenv don't seem to be the right fit, since i'm just using pipenv lock -r and you don't really want to deploy cython anyway. Also, we'd want build deps to work somehow with requirements.txt based projects too. Tho if you're fine with the dev reqs getting deployed, I think it'd be reasonable to add an option to this plugin to add the -d flag to the pipenv lock call.
As a work around, my suggestions would be:

if not using docker: create&activate a virtualenv and install cython
if using docker: create your own docker image that has cython installed

"create your own docker image that has cython installed" seems like a good workaround, since the docker file would only need three lines. I would say this could be added to the documentation to address build, dependencies.
Or even better, a config option could be added to specify additional commands to be run in the docker container before the actual build. For example to yum install some devel libraries. That would make it very user friendly:

the user does not have to learn docker and Dockerfile,
the user does not have to maintain the docker image, ensuring the image is up-to-date with the upstream over the long run.

lol.. how could I forget I started on that in #103.. but it needs to be adapted to work for docker correctly.

Hey 👋 I'm closing this ticket as it looks like it's heavily outdated, we can of course reopen it if needed 👍


⚠ ClamAV has degraded performance

In 14a5fdd, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 449 ms

Resolved: ClamAV performance has improved in 8a4008c.


Tei export

 Export TEI-XML

There is a lot of string catenation going on. Have you tested it with a very large and complex transliteration?
Alternatives would include using .map(...).join(...) or template literals.


Generalize HTTP server code from apps/ocsp.c to apps/lib/http_server.c

This is a further spin-off PR of #11470 as requested by @mattcaswell: https://github.com/openssl/openssl/pull/11470#discussion_r419925336.
This adds apps/lib/http_server.c and apps/include/http_server.h
simplifying apps/ocsp.c. This going to be used also for apps/cmp.c
and will be helpful as well for any future app acting as an HTTP server.
This PR also contains a couple of related fixes as separate commits:

Fix bio_wait() in crypto/bio/bio_lib.c in case OPENSSL_NO_SOCK
Constify 'req' parameter of OSSL_HTTP_post_asn1()
Fix bug in OSSL_CMP_SRV_process_request() on transaction renewal

Thanks @mattcaswell and @vdukhovni for your reviews and comments.
I've addressed all of them.

Approved assuming the minor nit is fixed.

Thanks for your swift reaction!
I've fixed the doc nit and partly squashed the commit structure for merging tomorrow.

It would have been cool to leverage a ubiquitous light-weight HTTP server library in C, with a suitable license instead of rolling our own, but the only that comes to mind (libmicrohttpd) is LGPL.  Mind you, we only use the HTTP server in "apps", it wouldn't be quite so bad to have it be runtime linked to an LGPL library, but we'd still need to something on systems where that's not available.  Anyone know of suitable alternatives?

It would have been cool to leverage a ubiquitous light-weight HTTP server library in C, with a suitable license instead of rolling our own, but the only one that comes to mind (libmicrohttpd) is LGPL. Mind you, we only use the HTTP server in "apps", it wouldn't be quite so bad to have it be runtime linked to an LGPL library, but we'd still need to something on systems where that's not available. Anyone know of suitable alternatives?

Anyone have any experience good/bad with libhttp? The license is MIT, so not a problem...

24 hours has passed since 'approval: done' was set, but as this PR has been updated in that time the label 'approval: ready to merge' is not being automatically set.  Please review the updates and set the label manually.

Merged - thanks @mattcaswell and @vdukhovni!

It would have been cool to leverage a ubiquitous light-weight HTTP server library in C, with a suitable license instead of rolling our own

I agree this would make sense at least instead of trying to further complete this very rudimentary HTTP server implementation, which would be a mayor endeavor.
The same I'd say holds for the HTTP client merged in #10667, which is somewhat more advanced but also far from being complete and already took quite some effort.


Set default occurrence for all active variants without a default

Set default occurrence for all active variants without a default

Done


Adding footer to learning page

There is no footer for the learnings page. Can u assign me this issue?

Hey @fykaa . I would like to work on this issue. Can you assign me this?

Hey! I made these changes. Can I make a PR now?
BEFORE:

AFTER:

hey can you  assign it to me , i will make it more beutifull

Sure @unovajnv

What do you think about this?

Hi, @komalioruganti Your PR has merge conflicts, Can you please look into it. Thanks :) #79

Yeah sure. I actually made a new PR. Can you review it and let me know if there are any more conflicts?  I will resolve them.


bugfix in float_edit for non-english locales

Hi,
The float property line edit doesn't work on systems with a decimal comma separator locale. For example on Ubuntu 16.04, with ros kinetic and a french locale, it is impossible to enter a decimal value.
This pull requested fixes the problem.
Another possible fix would be to force the QDoubleValidator to use the "C" locale :

QDoubleValidator *validator = new QDoubleValidator(this);
validator->setLocale(QLocale::c());
setValidator(validator);

I guess this would close https://github.com/ros-visualization/rviz/issues/1025?

@wjwwood yes this fixes the problem of #1025

Backport to Indigo: https://github.com/ros-visualization/rviz/pull/1058


Training from scratch [Data format query]

Hi,
Thank you for your wonderful work with GPVAD. I am looking at training the student model from scratch for my dataset(s). My dataset is in the form of audio_signal (wav) and the region has been tagged within the audio sample.
For example:
[{'type': 'BACKGROUND NOISE', 'time-range': [3.041, 3.169]}, {'type': 'SPEECH', 'time-range': [5.208, 5.544]}, {'type': 'BACKGROUND NOISE', 'time-range': [4.339, 5.069]}] is a tagged audio. Can your data pipeline support training for such data formats? If not, what do you suggest I should do to find a work around this?
Thanks a lot!

Hey sorry didn't get the notification :(.
No that pipeline does not support that format, since my goal in this work is to use weak-only labeled datasets and then just use (soft) predictions from a teacher to a student.
You need to write that by yourself I guess.

Alright. Maybe splitting the audio files (at their respective intervals) with their labels and generating the dataset in your format is the way to go.

So I guess you might misunderstand what this repo is for or what the goal is:
It solves the partial issue of requiring any frame-level alignment provided by some external source, such as an HMM.
The goal of this work is exactly not to use data such as yours, since this sort of data can be just trained by any other VAD system.
GPVAD supports to use any arbitrary clip-level training without explicit supervision, which is its selling point.
However I already ran joint GPVAD training with additional frame-level labels and have seen an improvement in performance ( usually only for clean scenarios ).
So feel free to try that out!

Sure. Thanks a lot for this!


Added some useful text into exception.

https://github.com/Azure/azure-sdk-for-java/issues/5662

Close the PR, as the exception will log secret out.


Add support for alternative chef-vault node_name and client_key_path

There is probably a better way for this.  I can't figure out how to change test-kitchen's client.rb with a node_name and client_key_path for my encrypted data bag items.  I've been using the authentication overwrite params for ChefVault::Item.load() but the chef-splunk uses the 'chef_vault_item' helper natively.
This fork allowed me to define the alternative authentication by just setting node properties.

Hi. I am an automated pull request bot named Curry. There are commits in this pull request whose authors are not yet authorized to contribute to Chef Software, Inc. projects or are using a non-GitHub verified email address. To become authorized to contribute, you will need to sign the Contributor License Agreement (CLA) as an individual or on behalf of your company. You can read more on Chef's blog.
GitHub Users Who Are Not Authorized To Contribute
The following GitHub users do not appear to have signed a CLA:

@pc-fmarin

Please sign the CLA here.


Replay speedup

This is presently a draft PR awaiting the following tasks to be completed:

[x] As a user, I want to use the replay tool to test new Allocators, so that I can evaluate performance of different AllocationStrategies UM-288
[x] Replay: MixedPool signature incorrect UM-395
[x] As a developer, I want replay to be able to diff its own output UM-366
[x] Merge with develop branch (many conflicts in replay)

👍

TODO:

[ ] Rename bld_X methods to makeX
[ ] Move templated methods definitions to .inl and non-templated methods to .cpp files.
[ ] Move argument parsing into main
[ ] (optional) add a test that runs replay (will memcheck it)

Meeting notes:
* [ ]  Why does `m_allocator_array` hold pointers to `umpire::Allocator`s?

* [ ]  Rename `bld_X` methods to `makeX`

* [ ]  Have `bld_X` methods return the string built with compare_ss to avoid code duplication

* [ ]  Add defaults to switch cases in `Replay.hpp` that throw an error

* [ ]  Move templated methods definitions to `.inl` and non-templated methods to `.cpp` files

* [ ]  Move argument parsing into main

* [ ]  (optional) Add a test that runs replay (will memcheck it)

@jdahm, as discussed in our meeting yesterday, the 'Meeting notes' were meant to be a set of action items for me.  Please remove your additional questions from the 'Meeting notes' and feel free to ask any questions you like about the implementation as a code comment.


Generator not downloading edited avatar

After i edit the bighead to my taste, and i click on download SVG, it doesn't download what I edited. Instead it download the avatar on the hero of the landing page

I think there's something wrong with the svg button

Hi @hillonyechekwa
This is correct. The SVG download button depends on the /svg api handler, which I disabled due to high server costs. The API now points to a static .svg file so people using the API won't have empty images on their apps/websites all of a sudden.
Tbh I didn't really think about the download button, but I will look into a client-side download method which shouldn't be too hard.
Thanks for reporting!

Done! Download button now works client side.


FOR TESTING | Replaced rustedhalo repo with aasemble one

This is to test breaking up singleton package. This will also test
switchover to aaSemble packages as this patch will only use aaSemble repo.

please stage this

please test this

Test FAILed

please test this

Test FAILed

please test this

Test FAILed


Adding Support to run tests with Security

Signed-off-by: Aditya Jindal<EMAIL_ADDRESS>Description
Runs the IT test suite against a secured cluster.
Check List

[x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

CI Passing in local fork: https://github.com/aditjind/opensearch-rs-1/pull/5