[Bug] Various reports that substream-powered-subgraphs silently stop processing blocks

Bug report
Reports from StakeSquid, Pinax, Data Nexus all confirm that substream-powered-subgraphs periodically stop processing blocks and silently fail. Reports that restarting graph-node fixes it.
Relevant log output
No response
IPFS hash
No response
Subgraph name or link to explorer
No response
Some information to help us out

[ ] Tick this box if this bug is caused by a regression found in the latest release.
[ ] Tick this box if this bug is specific to the hosted service.
[X] I have searched the issue tracker to make sure this issue is not a duplicate.

OS information
None

In my experience as an indexer operator:
We used to have an issue where the substreams-based-subgraphs would stop syncing.  The graph-node would seemingly fail to reconnect if there was a disconnection and blocks would stop flowing indefinitely. We would have to restart the graph-node.
This seems to fit the description of the bug reported here.
I have alerts set up to warn me whenever this happens, and they used to happen frequently - about 2 or 3 times a week.
However, ever since we upgraded our indexer graph-nodes to v0.32.0-rc.0 and now v0.32.0, we have not had this issue at all.
We are running our indexer with the StreamingFast endpoints for both firehose and substreams.

The StreamingFast team is ready to assist in any debugging for this issue.
If you are encountering this issue, can you please provide the following:

graph-node version
substreams version
substreams endpoint
any logs whatsoever around the time of the incident indicating any disconnection

Hi, I am adding this here although our bug is a bit different. To first explain, we have 2 indexers, same exact setup on both in terms of config for graph-node, and both have version 32.
When we added substreams to the graph-node config on our primary indexer, it stopped syncing about 1/3 of our ETHEREUM only subgraphs, while the substream subgraph was syncing normally. A restart of the graph-node solved the stalling for a short while (less than 5min), but then those same subgraphs got stuck again. When we removed the susbtreams from the graph-node config and just left ethereum there, all the ethereum subgraphs started to sync again normally. When i switched back again, they stalled again. Note that the config is exactly the same as our other indexer, where it all works normally. Not one thing is different.
Unfortunately i cant let you know about versions etc since we just tested it and used a friends endpoint. I was also scouring logs but did not find nothing at all that would indicate any warning, error or anything that would explain why the ethereum subgraphs stopped syncing.

@mindstyle85 Can you describe your friends infra a bit, like is it a single machine deployment running everything? What is the loose specs of the disks used to store Firehose files. Also, how many Subgraphs were served by a Firehose provider when you perform your tests.
What Firehose provider config in graph-node were you using when you tried it? (you can mask endpoint/keys)

This is my experience running substreams based subgraphs:
docker kill index-node-3 && docker restart index-node-3 && docker logs index-node-3 -f --tail 1000000 2>&1 | grep -i QmQJovmQLigEwkMWGjMT8GbeS2gjDytqWCGL58BEhLu9Ag
index-node-3
index-node-3
Sep 13 21:17:37.798 INFO Resolve subgraph files using IPFS, n_templates: 0, n_data_sources: 1, sgd: 1356, subgraph_id: QmQJovmQLigEwkMWGjMT8GbeS2gjDytqWCGL58BEhLu9Ag, component: SubgraphInstanceManager
Sep 13 21:17:40.991 INFO Successfully resolved subgraph files using IPFS, features: ø, n_templates: 0, n_data_sources: 1, sgd: 1356, subgraph_id: QmQJovmQLigEwkMWGjMT8GbeS2gjDytqWCGL58BEhLu9Ag, component: SubgraphInstanceManager
Sep 13 21:17:41.002 INFO Starting subgraph writer, queue_size: 5, sgd: 1356, subgraph_id: QmQJovmQLigEwkMWGjMT8GbeS2gjDytqWCGL58BEhLu9Ag, component: SubgraphInstanceManager
Sep 13 21:17:41.088 INFO Data source count at start: 1, sgd: 1356, subgraph_id: QmQJovmQLigEwkMWGjMT8GbeS2gjDytqWCGL58BEhLu9Ag, component: SubgraphInstanceManager
Sep 13 21:17:41.095 INFO Blockstreams disconnected, connecting, provider_err_count: 0, cursor: xeKhcurHMv_Ro6Ttwgb0naWwLpcyB1JuVwLiIxdL1IKn-SfF3sj0AGR1bhvTmarziRHsHVyu2dnOR3tz-pRWutTix7k36iBqFy54movs-LG-fKGjO14cce9gWrjfN9LaUz3Vaw36fLsJtdXnPPTQbkcwZ8R1fzKyiz5WpIEAJvJA6iE3y236cpqHgqrCp9QUrOd0RuSnxH_2UmR9KklSbJqGYPKduTwm, start_block: 18129797, subgraph: QmQJovmQLigEwkMWGjMT8GbeS2gjDytqWCGL58BEhLu9Ag, endpoint_uri: substreams, provider: substreams, deployment: QmQJovmQLigEwkMWGjMT8GbeS2gjDytqWCGL58BEhLu9Ag, sgd: 1356, subgraph_id: QmQJovmQLigEwkMWGjMT8GbeS2gjDytqWCGL58BEhLu9Ag, component: SubstreamsBlockStream
Sep 13 21:17:41.246 INFO Blockstreams connected, provider: substreams, deployment: QmQJovmQLigEwkMWGjMT8GbeS2gjDytqWCGL58BEhLu9Ag, sgd: 1356, subgraph_id: QmQJovmQLigEwkMWGjMT8GbeS2gjDytqWCGL58BEhLu9Ag, component: SubstreamsBlockStream

That's it! No more logs, no errors, not advancing blocks, nada!
Tested with my own endpoint and Pinax's endpoint.
Restating the index node a dozen times, sometimes it moves it forward.
Specs of the machines:
graph-node machine - runs 7 index nodes and 1 query node - 5950X (16c/32t) / 128gb ram ddr4 / 2 nvme raid0
fire-hose all-in-one machine - runs firehose + substreams stack altogether - 5950X (16c/32t) / 128gb ram ddr4 / 3 nvme raid0
postgres db machine - runs only postgres, one shard, one instance - EPYC 7502P (32c/64t) / 128gb ram ddr4 / 5 nvme raid0
Load average on the machines:
graph-node machine:

firehose machine:

postgres db machine:

To be noted that the same behavior occurs whether or not my firehose machine is loaded liek that. The load right now comes from me syncing a handful of eth mainnet subs.
And again, same behaviour seen on Pinax's endpoints.

@trader-payne if you activate debug logs on your index nodes, can you see any additional information?
can you confirm that you're running graph-node v0.32.0 ?

Stacks versions here:
INDEXER_AGENT_VERSION="ghcr.io/graphprotocol/indexer-agent:v0.20.22"
INDEXER_SERVICE_VERSION="ghcr.io/semiotic-ai/autoagora-indexer-service:v0.1.2-v0.20.22"
GRAPH_NODE_VERSION="graphprotocol/graph-node:v0.32.0"

Will activate debug and will post here if I get more logs. Gotta close some allos now that I have those subs up to chainhead for an hour lol

Issue occured here  today for two subgraphs, with slightly different behavior

QmQJovmQLigEwkMWGjMT8GbeS2gjDytqWCGL58BEhLu9Ag
Graph-node stalled on block 18128999. At least one restart of graph-node wasn't enough to resume the indexation, but it resumed immediately after restart the Substream Tier 1 processes.

Sep 14 23:44:48.015 INFO Blockstreams disconnected, connecting, provider_err_count: 5, cursor: p_z44DJJ5njlKnwsDAxFVaWwLpc_DFtnVAntIxhIj4r0o3rB2JXyVGB1bRnUmaCj1EPiGlypjYqYHX8t-pRZu4e7xrEw7iRtTn19m4Do-bfnLPT2bAwYc-piDO-MYdzRWjTWYALyc7gK4de0MvfcbxZmY8cjKGK6hmxZ9IwFdqZC7CY8wmr1J8bVhqqT-NZC-rctQrWpnC2hBD0pK0sPO8-ANvLMuj4jYQ==, start_block: 18128000, subgraph: QmQJovmQLigEwkMWGjMT8GbeS2gjDytqWCGL58BEhLu9Ag, endpoint_uri: substreams, provider: substreams, deployment: QmQJovmQLigEwkMWGjMT8GbeS2gjDytqWCGL58BEhLu9Ag, sgd: 54, subgraph_id: QmQJovmQLigEwkMWGjMT8GbeS2gjDytqWCGL58BEhLu9Ag, component: SubstreamsBlockStream
Sep 14 23:44:48.047 INFO Blockstreams connected, provider: substreams, deployment: QmQJovmQLigEwkMWGjMT8GbeS2gjDytqWCGL58BEhLu9Ag, sgd: 54, subgraph_id: QmQJovmQLigEwkMWGjMT8GbeS2gjDytqWCGL58BEhLu9Ag, component: SubstreamsBlockStream
Sep 14 23:44:55.454 DEBG Start processing block, triggers: 1, block_hash: 0xeb243b56b0b25a5fe87a80be5af6427834ea42e9b8d6d3705e4540be52662589, block_number: 18129000, sgd: 54, subgraph_id: QmQJovmQLigEwkMWGjMT8GbeS2gjDytqWCGL58BEhLu9Ag, component: SubgraphInstanceManager

Qmf4VVPWSejxoGLg3nZuoCgPBxuARuiayjsdt6oandDPdf
Stalled at a graph-node restart, i.e. didn't start indexing.  Restarting firehose tier1 didn't fix it. After, another graph-node restart got it to catch up to block 18136999 (the latest xxxx999 block), no errors in graph-node logs. A third graph-node restart finally reached chain-head.

graph-node:v0.32.0
Internal substream endpoint running firehose-ethereum:v1.4.10
I don't have logs from the moment the indexation stalled, as these were rotated. This is concurring on fairly regular basis, every 2-3 days, usually on a xxxxxxx999 block.
This may have to do with the transition between the handling of cached substream from tier2 and live processing on tier1.

@colindickson activated debug logs, there's nothing more in the logs than I've already provided.

thanks for the details here - hopefully #4866 can help with some debugging, is there anything else that might help investigation on the Graph Node side here? the xx999 block is an interesting clue.

Sep 15 10:08:25.127 DEBG Start processing block, triggers: 1, block_hash: 0xb828e52d1f89c75094c53c2f5b21891f8d5e320d41b9762826711b2790d0bc8d, block_number: 18140929, sgd: 1, subgraph_id: QmeWyu8pzV3zViL5nz813NxKSab7REh9qJBivjzkCzE9Vw, component: SubgraphInstanceManager
Sep 15 10:08:25.127 DEBG Entity cache statistics, evict_time_ms: 0, accesses: 0, hit_rate: 100%, stale_update: false, evicted_count: 0, count: 3814, evicted_weight: 0, weight: 1864947, sgd: 1, subgraph_id: QmeWyu8pzV3zViL5nz813NxKSab7REh9qJBivjzkCzE9Vw, component: SubgraphInstanceManager
Sep 15 10:08:25.127 INFO Applying 2 entity operation(s), block_hash: 0xb828e52d1f89c75094c53c2f5b21891f8d5e320d41b9762826711b2790d0bc8d, block_number: 18140929, sgd: 1, subgraph_id: QmeWyu8pzV3zViL5nz813NxKSab7REh9qJBivjzkCzE9Vw, component: SubgraphInstanceManager
Sep 15 10:08:32.785 INFO Committed write batch, time_ms: 17950, weight: 4535480, entities: 7618, block_count: 3809, block_number: 18140925, sgd: 1, subgraph_id: QmeWyu8pzV3zViL5nz813NxKSab7REh9qJBivjzkCzE9Vw, component: SubgraphInstanceManager

Sep 15 10:08:25.127 DEBG Start processing block, triggers: 1, block_hash: 0xb828e52d1f89c75094c53c2f5b21891f8d5e320d41b9762826711b2790d0bc8d, block_number: 18140929, sgd: 1, subgraph_id: QmeWyu8pzV3zViL5nz813NxKSab7REh9qJBivjzkCzE9Vw, component: SubgraphInstanceManager
Sep 15 10:08:25.127 DEBG Entity cache statistics, evict_time_ms: 0, accesses: 0, hit_rate: 100%, stale_update: false, evicted_count: 0, count: 3814, evicted_weight: 0, weight: 1864947, sgd: 1, subgraph_id: QmeWyu8pzV3zViL5nz813NxKSab7REh9qJBivjzkCzE9Vw, component: SubgraphInstanceManager
Sep 15 10:08:25.127 INFO Applying 2 entity operation(s), block_hash: 0xb828e52d1f89c75094c53c2f5b21891f8d5e320d41b9762826711b2790d0bc8d, block_number: 18140929, sgd: 1, subgraph_id: QmeWyu8pzV3zViL5nz813NxKSab7REh9qJBivjzkCzE9Vw, component: SubgraphInstanceManager
Sep 15 10:08:32.785 INFO Committed write batch, time_ms: 17950, weight: 4535480, entities: 7618, block_count: 3809, block_number: 18140925, sgd: 1, subgraph_id: QmeWyu8pzV3zViL5nz813NxKSab7REh9qJBivjzkCzE9Vw, component: SubgraphInstanceManager

@mindstyle85 Can you describe your friends infra a bit, like is it a single machine deployment running everything? What is the loose specs of the machine(s) and disks used to serve Firehose blocks. Also, how many Subgraphs were served by a Firehose provider when you perform your tests.
What Firehose provider config in graph-node were you using when you tried it? (you can mask endpoint/keys)

Hey, sorry for late reply. I still dont know the exact specs but afaik its all 1 really strong server (ax102 eq or better) with enough disk space.
Our setup is also 1 server with 32 cores 128GB ram and 12tb nvme disks. Usage of cpu and ram is about 10% steady.
Config is basically the same as payne posted but without the token part. Graph-node is 0.32.0.
I just did another test with yet another endpoint besides the first one, and in both cases same thing has happened. When i switch to a substream config, it starts syncing the substream subgraph, but about 70 (out of approx 200) ethereum subgraphs get stalled. After restarting, about 20 of those 70 stalled ones pick back up for a minute or so then stall again, while the other 50 stay stalled. I did this flow 3 times now and its the same result each time.
I also dont see anything in the logs at all about this and i also dont notice any particular block height at which this would happen.
Sorry i cant help more but there really isnt much to go on with. I will see if i can get 1 or 2 more endpoints to test, to see if its really the same with all of them.

Currently running on testnet chain (Eth Goerli)
.env
INDEXER_AGENT_VERSION="ghcr.io/graphprotocol/indexer-agent:v0.20.22"
INDEXER_SERVICE_VERSION="ghcr.io/semiotic-ai/autoagora-indexer-service:v0.1.2-v0.20.22"
GRAPH_NODE_VERSION="graphprotocol/graph-node:v0.32.0"

config.toml
[store]
[store.primary]
connection = "postgresql://hau:secret@:5432/graph-node-db0?host=/var/run/postgresql"
pool_size = 10

[chains]
ingestor = "index_node_0"

[chains.mainnet]
shard = "primary"
provider = [
             { label = "substreams", details = { type = "substreams", url = "http://eth-sf-substreams-tier1-v4.service.march.consul.eosn.io:9000",  features = ["compression", "filters"], conn_pool_size = 100 } },
             { label = "peering", url = "http://eth-erigon.service.march.consul.eosn.io:8545", features = ["archive", "traces"] } ]

[deployment]
[[deployment.rule]]
indexers = [ "index_node_0" ]

[general]
query = "query_node_0"

Logs
Sep 15 10:08:14.855 DEBG Start processing block, triggers: 1, block_hash: 0xa791f522bf96ffd256fe17ec432acd9f860237e5910de5ea4d839c16779dd56d, block_number: 18140927, sgd: 1, subgraph_id: QmeWyu8pzV3zViL5nz813NxKSab7REh9qJBivjzkCzE9Vw, component: SubgraphInstanceManager
Sep 15 10:08:14.856 DEBG Entity cache statistics, evict_time_ms: 0, accesses: 0, hit_rate: 100%, stale_update: false, evicted_count: 0, count: 3812, evicted_weight: 0, weight: 1863968, sgd: 1, subgraph_id: QmeWyu8pzV3zViL5nz813NxKSab7REh9qJBivjzkCzE9Vw, component: SubgraphInstanceManager
Sep 15 10:08:14.856 INFO Applying 2 entity operation(s), block_hash: 0xa791f522bf96ffd256fe17ec432acd9f860237e5910de5ea4d839c16779dd56d, block_number: 18140927, sgd: 1, subgraph_id: QmeWyu8pzV3zViL5nz813NxKSab7REh9qJBivjzkCzE9Vw, component: SubgraphInstanceManager
Sep 15 10:08:14.860 DEBG Start processing block, triggers: 1, block_hash: 0xb3b3cd1b3a4fbffb386caf8fb2213c4264229ae082b0210eb64c83f9d0aec616, block_number: 18140928, sgd: 1, subgraph_id: QmeWyu8pzV3zViL5nz813NxKSab7REh9qJBivjzkCzE9Vw, component: SubgraphInstanceManager
Sep 15 10:08:14.861 DEBG Entity cache statistics, evict_time_ms: 0, accesses: 0, hit_rate: 100%, stale_update: false, evicted_count: 0, count: 3813, evicted_weight: 0, weight: 1864457, sgd: 1, subgraph_id: QmeWyu8pzV3zViL5nz813NxKSab7REh9qJBivjzkCzE9Vw, component: SubgraphInstanceManager
Sep 15 10:08:14.861 INFO Applying 2 entity operation(s), block_hash: 0xb3b3cd1b3a4fbffb386caf8fb2213c4264229ae082b0210eb64c83f9d0aec616, block_number: 18140928, sgd: 1, subgraph_id: QmeWyu8pzV3zViL5nz813NxKSab7REh9qJBivjzkCzE9Vw, component: SubgraphInstanceManager
Sep 15 10:08:25.127 DEBG Start processing block, triggers: 1, block_hash: 0xb828e52d1f89c75094c53c2f5b21891f8d5e320d41b9762826711b2790d0bc8d, block_number: 18140929, sgd: 1, subgraph_id: QmeWyu8pzV3zViL5nz813NxKSab7REh9qJBivjzkCzE9Vw, component: SubgraphInstanceManager
Sep 15 10:08:25.127 DEBG Entity cache statistics, evict_time_ms: 0, accesses: 0, hit_rate: 100%, stale_update: false, evicted_count: 0, count: 3814, evicted_weight: 0, weight: 1864947, sgd: 1, subgraph_id: QmeWyu8pzV3zViL5nz813NxKSab7REh9qJBivjzkCzE9Vw, component: SubgraphInstanceManager
Sep 15 10:08:25.127 INFO Applying 2 entity operation(s), block_hash: 0xb828e52d1f89c75094c53c2f5b21891f8d5e320d41b9762826711b2790d0bc8d, block_number: 18140929, sgd: 1, subgraph_id: QmeWyu8pzV3zViL5nz813NxKSab7REh9qJBivjzkCzE9Vw, component: SubgraphInstanceManager
Sep 15 10:08:32.785 INFO Committed write batch, time_ms: 17950, weight: 4535480, entities: 7618, block_count: 3809, block_number: 18140925, sgd: 1, subgraph_id: QmeWyu8pzV3zViL5nz813NxKSab7REh9qJBivjzkCzE9Vw, component: SubgraphInstanceManager
Sep 15 10:08:32.834 INFO Committed write batch, time_ms: 43, weight: 2137, entities: 2, block_count: 1, block_number: 18140926, sgd: 1, subgraph_id: QmeWyu8pzV3zViL5nz813NxKSab7REh9qJBivjzkCzE9Vw, component: SubgraphInstanceManager
Sep 15 10:08:32.849 INFO Committed write batch, time_ms: 15, weight: 2137, entities: 2, block_count: 1, block_number: 18140927, sgd: 1, subgraph_id: QmeWyu8pzV3zViL5nz813NxKSab7REh9qJBivjzkCzE9Vw, component: SubgraphInstanceManager
Sep 15 10:08:32.865 INFO Committed write batch, time_ms: 15, weight: 2137, entities: 2, block_count: 1, block_number: 18140928, sgd: 1, subgraph_id: QmeWyu8pzV3zViL5nz813NxKSab7REh9qJBivjzkCzE9Vw, component: SubgraphInstanceManager
Sep 15 10:08:32.877 INFO Committed write batch, time_ms: 12, weight: 2138, entities: 2, block_count: 1, block_number: 18140929, sgd: 1, subgraph_id: QmeWyu8pzV3zViL5nz813NxKSab7REh9qJBivjzkCzE9Vw, component: SubgraphInstanceManager

Currently running on testnet chain (Eth Goerli)
.env
INDEXER_AGENT_VERSION="ghcr.io/graphprotocol/indexer-agent:v0.20.22"
INDEXER_SERVICE_VERSION="ghcr.io/semiotic-ai/autoagora-indexer-service:v0.1.2-v0.20.22"
GRAPH_NODE_VERSION="graphprotocol/graph-node:v0.32.0"

config.toml
[store]
[store.primary]
connection = "postgresql://${DB_USER}:${DB_PASS}/graph-node-db0?host=/var/run/postgresql"
pool_size = 10

[chains]
ingestor = "index_node_0"

[chains.mainnet]
shard = "primary"
provider = [
             { label = "substreams", details = { type = "substreams", url = "$SUBSTREAMS",  features = ["compression", "filters"], conn_pool_size = 100 } },
             { label = "peering", url = "$RPC", features = ["archive", "traces"] } ]

[deployment]
[[deployment.rule]]
indexers = [ "index_node_0" ]

[general]
query = "query_node_0"

Logs
Sep 15 10:08:14.855 DEBG Start processing block, triggers: 1, block_hash: 0xa791f522bf96ffd256fe17ec432acd9f860237e5910de5ea4d839c16779dd56d, block_number: 18140927, sgd: 1, subgraph_id: QmeWyu8pzV3zViL5nz813NxKSab7REh9qJBivjzkCzE9Vw, component: SubgraphInstanceManager
Sep 15 10:08:14.856 DEBG Entity cache statistics, evict_time_ms: 0, accesses: 0, hit_rate: 100%, stale_update: false, evicted_count: 0, count: 3812, evicted_weight: 0, weight: 1863968, sgd: 1, subgraph_id: QmeWyu8pzV3zViL5nz813NxKSab7REh9qJBivjzkCzE9Vw, component: SubgraphInstanceManager
Sep 15 10:08:14.856 INFO Applying 2 entity operation(s), block_hash: 0xa791f522bf96ffd256fe17ec432acd9f860237e5910de5ea4d839c16779dd56d, block_number: 18140927, sgd: 1, subgraph_id: QmeWyu8pzV3zViL5nz813NxKSab7REh9qJBivjzkCzE9Vw, component: SubgraphInstanceManager
Sep 15 10:08:14.860 DEBG Start processing block, triggers: 1, block_hash: 0xb3b3cd1b3a4fbffb386caf8fb2213c4264229ae082b0210eb64c83f9d0aec616, block_number: 18140928, sgd: 1, subgraph_id: QmeWyu8pzV3zViL5nz813NxKSab7REh9qJBivjzkCzE9Vw, component: SubgraphInstanceManager
Sep 15 10:08:14.861 DEBG Entity cache statistics, evict_time_ms: 0, accesses: 0, hit_rate: 100%, stale_update: false, evicted_count: 0, count: 3813, evicted_weight: 0, weight: 1864457, sgd: 1, subgraph_id: QmeWyu8pzV3zViL5nz813NxKSab7REh9qJBivjzkCzE9Vw, component: SubgraphInstanceManager
Sep 15 10:08:14.861 INFO Applying 2 entity operation(s), block_hash: 0xb3b3cd1b3a4fbffb386caf8fb2213c4264229ae082b0210eb64c83f9d0aec616, block_number: 18140928, sgd: 1, subgraph_id: QmeWyu8pzV3zViL5nz813NxKSab7REh9qJBivjzkCzE9Vw, component: SubgraphInstanceManager
Sep 15 10:08:25.127 DEBG Start processing block, triggers: 1, block_hash: 0xb828e52d1f89c75094c53c2f5b21891f8d5e320d41b9762826711b2790d0bc8d, block_number: 18140929, sgd: 1, subgraph_id: QmeWyu8pzV3zViL5nz813NxKSab7REh9qJBivjzkCzE9Vw, component: SubgraphInstanceManager
Sep 15 10:08:25.127 DEBG Entity cache statistics, evict_time_ms: 0, accesses: 0, hit_rate: 100%, stale_update: false, evicted_count: 0, count: 3814, evicted_weight: 0, weight: 1864947, sgd: 1, subgraph_id: QmeWyu8pzV3zViL5nz813NxKSab7REh9qJBivjzkCzE9Vw, component: SubgraphInstanceManager
Sep 15 10:08:25.127 INFO Applying 2 entity operation(s), block_hash: 0xb828e52d1f89c75094c53c2f5b21891f8d5e320d41b9762826711b2790d0bc8d, block_number: 18140929, sgd: 1, subgraph_id: QmeWyu8pzV3zViL5nz813NxKSab7REh9qJBivjzkCzE9Vw, component: SubgraphInstanceManager
Sep 15 10:08:32.785 INFO Committed write batch, time_ms: 17950, weight: 4535480, entities: 7618, block_count: 3809, block_number: 18140925, sgd: 1, subgraph_id: QmeWyu8pzV3zViL5nz813NxKSab7REh9qJBivjzkCzE9Vw, component: SubgraphInstanceManager
Sep 15 10:08:32.834 INFO Committed write batch, time_ms: 43, weight: 2137, entities: 2, block_count: 1, block_number: 18140926, sgd: 1, subgraph_id: QmeWyu8pzV3zViL5nz813NxKSab7REh9qJBivjzkCzE9Vw, component: SubgraphInstanceManager
Sep 15 10:08:32.849 INFO Committed write batch, time_ms: 15, weight: 2137, entities: 2, block_count: 1, block_number: 18140927, sgd: 1, subgraph_id: QmeWyu8pzV3zViL5nz813NxKSab7REh9qJBivjzkCzE9Vw, component: SubgraphInstanceManager
Sep 15 10:08:32.865 INFO Committed write batch, time_ms: 15, weight: 2137, entities: 2, block_count: 1, block_number: 18140928, sgd: 1, subgraph_id: QmeWyu8pzV3zViL5nz813NxKSab7REh9qJBivjzkCzE9Vw, component: SubgraphInstanceManager
Sep 15 10:08:32.877 INFO Committed write batch, time_ms: 12, weight: 2138, entities: 2, block_count: 1, block_number: 18140929, sgd: 1, subgraph_id: QmeWyu8pzV3zViL5nz813NxKSab7REh9qJBivjzkCzE9Vw, component: SubgraphInstanceManager

Subgraphs
[https://testnet.thegraph.com/explorer/subgraph?id=ApnvbDR5gExNyW8LvWopSTkd88DK78njG7CbeYu8pjH3&chain=goerli](url)
[https://testnet.thegraph.com/explorer/subgraph?id=HUZDsRpEVP2AvzDCyzDHtdc64dyDxx8FQjzsmqSg4H3B&chain=goerli](url)
[https://testnet.thegraph.com/explorer/subgraph?id=3e63xu6PWvSGphm2hPbKtvh4rziS2jYSqxhz5rmRmcnx&chain=goerli](url)

Sometimes restart works / doesn't work on syncing the subgraphs (non-substreams subgraph sometimes affected as well)
Few hours

^I'm having the exact same behavior described here. With either my own endpoint or theirs.

Thanks for sharing - those details. I think you can remove the following from the substreams config (though I would be surprised if that is the root case) - @trader-payne can you try as I see you have some of that info in your substreams provider too?
features = ["compression", "filters"], conn_pool_size = 100

I don't know why there would be an impact on non-substreams subgraphs, does block ingestion via the RPC continue as normal when those RPC subgraphs stop?

@mindstyle85 @trader-payne can you also confirm whether those Ethereum subgraphs are syncing with a Firehose or RPC (maybe it is both, @trader-payne's config seemed to have a firehose in there but @mindstyle85's did not)

@trader-payne @mindstyle85 @madumas @jhjhjh94 I would like that we separate tracking of the Subgraphs through Firehose and Substreams powered subgraphs.
Those are two completely different components, both in terms of Firehose code as well as within graph-node. I read two different problems in this issue and they could be related (a gRPC stack problem within Rust would be a shared bug) but there is a good chance there is different things going on.

can help with some debugging, is there anything else that might help investigation on the Graph Node side here? the xx999 block is an interesting clue.

So we are missing visibility into what is happening on the server's side serving Substreams because some occurrence of long delay could be "normal" behavior (I say normal but we could more say "actual" since improvements could be done here, see below for more details)
You will often see a Substreams stream be "stuck" on a x999 boundary, let me explain you why. In production mode, the processing and the streaming back of data is done in separate processes. The processing is done in chunk of 1000 blocks on at end of which final map outputs are saved to disk.
It's only when this bundle is finally available that data is streamed back. Because of this behavior, a Substreams could be seen stuck while it's still progressing normally, it's just that no visibility was offered to see this. If a segment of 1000 blocks take X minutes to process (because for example it makes at lot of eth_call or CPU is overloaded and has a hard time executing WASM code), then you will not receive blocks.
And because of dependencies graph of modules, you could need to execute 30 of such modules before having the data for 1000 blocks. It would be perceived that Substreams is stuck but it's actually processing normally.
Another things that can happen on the Substreams side is let's say the configured RPC provider (within Substreams stack) is broken. This make Substreams retries eth_call until it succeed which could lead also to visible stuckness on the graph-node side while it's really stuck. Again, it's a visibility problem here.
Now, to add to further more problem. Let's say you restart the graph-node process thinking it will help. Due to missing feature on Substreams side, it creates even more problem. Indeed, some partials computation within this 1000 blocks segments are actually discarded. This means on restart, we start back from the start block of the 1000 blocks boundary. Leading to the impression that it will never finish.
Having long inactivity period on gRPC often also kicks sometimes weird networking issue. Users reported stream restarting more often than it should recently when using the Rust sink example (see https://github.com/streamingfast/substreams/issues/277). This is again a problem as work is being re-done from scratch on stream restart, leading to never ending work.
All above, I'm talking about Substreams. Firehose is a different beast and on this particular issue, I think the stuckness is more in graph-node then in Golang (it's my gut feelings talking here, I could be wrong). Hence my desire to split the issues in two.

Also, please note that I'm not assessing there is no bugs/problems, just describing actual behavior so that everyone can better understand what is happening under the hood.
About observability, it will be the first step to start debugging Substreams based issue. We have revamped progress messages sent back from the endpoint in the stream in the last month and this has not been re-surfaced yet within graph-node and it will be a major help here to identify wether the endpoint is functioning properly or the stream is stuck.
We have now steady progress messages coming back to the client (2 message per second, something like this) and the progress messages. Those messages contains every pieces of details about what the endpoint is currently doing, what are its current processing jobs, last block processed, last available linear block and much more. It also contains timing information, for example around eth_call performed.
As well as the trace_id, we will need to bring those back into graph-node with a formatted log (we could probably print less often within graph-node, for example present the latest received progress each 15s or something like that).
In the long run about observability, we would also like that it's possible to "attach" to graph-node (or directly to the endpoint providing a trace_id) and you could get a nice UI about what is happening on the Substreams perspective. Could be well integrated within graph-node, but that is not a short term task, will see. For this see issue #4826.

Thanks for sharing - those details. I think you can remove the following from the substreams config (though I would be surprised if that is the root case) - @trader-payne can you try as I see you have some of that info in your substreams provider too?
features = ["compression", "filters"], conn_pool_size = 100

I don't know why there would be an impact on non-substreams subgraphs, does block ingestion via the RPC continue as normal when those RPC subgraphs stop?

My config has that set to 1, but I'll remove them completely to see what happens. I'll be back with another comment.

More details on what I'm seeing on my end (gonna keep adding to this):
Scenario 1 - my erigon + pinax substream, no firehose defined
[chains.${CHAIN_0_NAME}]
shard = "primary"
protocol = "ethereum"
provider = [
{ label = "substreams", details = { type = "substreams", url = "${CHAIN_0_RPC_SUBSTREAMS}", token = "${STREAMING_FAST_API_TOKEN}" } },
{ label = "archive", url = "${CHAIN_0_RPC_ARCHIVE}", features = ["archive", "traces"] }
]

About 50-100 subgraphs stop from syncing, substreams work.

Now I'm on my own Firehose + Pinax's substreams, and a handful of subgraphs stopped syncing, no logs to share:

Thanks so much @trader-payne - when you don't have the Firehose configured does the block ingestor seem to be progressing?

Worth noting even with Pinax's stack, the Substreams get stuck after an hour or two, requiring graphnode restarts.

This is in "Scenario 3 - Pinax's substreams + my Firehose" - i.e. there is no persistent "success" configuration for you right now?

Thanks so much @trader-payne - when you don't have the Firehose configured does the block ingestor seem to be progressing?

No, it does, not, even though you can find the logs saying
Sep 20 23:45:09.872 INFO Starting firehose block ingestor for network, network_name: mainnet
And no errors nor anything suggesting it won't work

@azf20 @maoueh changing to this graphnode configuration works! 👀
[chains.${CHAIN_0_NAME}]
shard = "primary"
protocol = "ethereum"
provider = [
{ label = "substreams", details = { type = "substreams", url = "${CHAIN_0_RPC_SUBSTREAMS}", token = "${STREAMING_FAST_API_TOKEN}" } },
#{ label = "firehose", details = { type = "firehose", url = "${CHAIN_0_RPC_FIREHOSE}", token = "${STREAMING_FAST_API_TOKEN}", features = ["compression", "filters"], conn_pool_size = 1 } },
{ label = "bootstrap", details = { type = "web3", url = "${CHAIN_0_RPC_ARCHIVE}", features = ["archive", "traces"] } }
]

Notice the different label and type added for the archive part. 👀
Block ingestor works, subgraphs are syncing.
I'll leave this overnight, hope I won't get to eat my words, and it will keep going 🤞🏻

Interesting - did it work overnight?
@mangas @lutter would be interested if you have a view here on some of the different behaviour described above

Yeah but I had my eth node crash due to memory constraints - it was running on the same machine as the index nodes 🔪 💀

Any luck with the investigation so far @azf20 @maoueh?
Is there anything I can do to help here?
I found a potential bottleneck - the IO on the database server is basically pegged at 100% all the time:

I'm gonna stop the substreams to see how/if that affects anything.

@trader-payne is this with the config which was initially working?

Any config actually, the db is choked. But like I said, subgraphs sync no problem.

Adding a data point to the investigation. Found the Uniswap v2 substream-powered subgraph stopped today, but too late for capturing logs (30k blocks behind).
The command substreams gui ipfs://QmQJovmQLigEwkMWGjMT8GbeS2gjDytqWCGL58BEhLu9Ag, showed that the substream had also been interrupted around the same block and had now just resumed processing blocks into the cache. Once the substream appeared to have reached chain head (according to the gui), a few graph-node restarts over next 3 hours were necessary to bring graph-node to chain head, as it would stop on an intermediary xx999 block as if the substream cache wasn't as far as indicated.
On this setup, firehose is disabled. Using an internal substream + rpc combo.

Any luck with the investigation so far @azf20 @maoueh? Is there anything I can do to help here?
I found a potential bottleneck - the IO on the database server is basically pegged at 100% all the time:
I'm gonna stop the substreams to see how/if that affects anything. EDIT:
No change after killing the substreams, the IO stays the same, it's mainly read IO that's causing the load - lots of autovacuum processes running.
Even though there's this bottleneck, all the other subgraphs are keeping up to chainhead, and all the subgraphs that I'm syncing from scratch are progressing. The only thing stopping are the substreams. And I doubt @jhjhjh94 is in the same situation with a database containing 800+ subgraphs

Confirmed the same from my side, IO bottleneck, with or without substreams still the same

@jhjhjh94

Confirmed the same from my side, IO bottleneck, with or without substreams still the same

I/O is the same with and without Substreams?

@trader-payne No work has been done yet, but I would like to build now a version with extra logging to check where the hanging is happening. I will branch it from v0.32.0 for now.

For Substreams powered Subgraph stuck, version graphprotocol/graph-node:canary-investigate-stuck-substreams-def370693 can be tested.
Logs I added are in INFO and they might be a bit verbose, it's for Substreams only also, so you can deploy only on the node that is going to index the Substreams powered Subgraph, it prints:

Session init message (contains trace id, I wasn't sure if this version has it)
Progress message received each ~30s
Log before trying to read a new message from the gRPC stream
Log after having read a message from the gRPC stream

@jhjhjh94

Confirmed the same from my side, IO bottleneck, with or without substreams still the same

I/O is the same with and without Substreams?

Yes, that's correct

From @trader-payne testing
graph-node.log

The subgraph doesn't move towards the chainhead during the logs period, at all
indexer-agent/t6oandDPdf (C) Qmf4VVPWSejxoGLg3nZuoCgPBxuARuiayjsdt6oandDPdf
1474
index_node_3
true
mainnet
275
55.0 min

Logs I shared are with Pinax's Substreams + My Firehose

From there, it seems the gRPC is not stuck as progresses is sent regularly so the connection is still up. Here some data extraction from the logs above:
[11:49:01.180] Cursor {
    value: "c1:1:18248084:ce8a9684a967ade283106876aa1aa0830889a2d32858ab3cadb79676e1866aa4:18248002:200e573a4b69f26eb58734d7f51ad7e350606b223c4a2ad5c19ab99f1cf0503f"
}
[11:49:13.910] SessionInit {
    trace_id: "e0c0afb9363755551c73bf970ec663a4",
    resolved_start_block: 18248085,
    linear_handoff_block: 18248100,
    max_parallel_workers: 10,
}
[11:49:31.538] ModulesProgress {
    running_jobs: [
        Job { stage: 5, start_block: 18248000, stop_block: 18248100, processed_blocks: 0, duration_ms: 8759 },
        Job { stage: 4, start_block: 18248000, stop_block: 18248100, processed_blocks: 0, duration_ms: 8739 },
        Job { stage: 3, start_block: 18248000, stop_block: 18248100, processed_blocks: 0, duration_ms: 8719 },
        Job { stage: 8, start_block: 18248000, stop_block: 18248100, processed_blocks: 0, duration_ms: 8817 },
        Job { stage: 7, start_block: 18248000, stop_block: 18248100, processed_blocks: 0, duration_ms: 8798 },
        Job { stage: 6, start_block: 18248000, stop_block: 18248100, processed_blocks: 0, duration_ms: 8778 },
    ],
    modules_stats: [
        ModuleStats {
            name: "map_pool_created",
            total_processed_block_count: 300,
            total_processing_time_ms: 7771,
            external_call_metrics: [ExternalCallMetric { name: "rpc:eth_call", count: 108, time_ms: 408 }],
            total_store_operation_time_ms: 0,
            total_store_read_count: 0,
            total_store_write_count: 0,
            total_store_deleteprefix_count: 0,
            store_size_bytes: 0,
            total_store_merging_time_ms: 0,
            store_currently_merging: false,
            highest_contiguous_block: 18248100
        },
        ModuleStats {
            name: "store_pools",
            total_processed_block_count: 300,
            total_processing_time_ms: 233,
            external_call_metrics: [],
            total_store_operation_time_ms: 0,
            total_store_read_count: 0,
            total_store_write_count: 6,
            total_store_deleteprefix_count: 0,
            store_size_bytes: 83976637,
            total_store_merging_time_ms: 603,
            store_currently_merging: false,
            highest_contiguous_block: 18248100
        },
        ModuleStats {
            name: "store_output_token_supply",
            total_processed_block_count: 200,
            total_processing_time_ms: 4859,
            external_call_metrics: [],
            total_store_operation_time_ms: 0,
            total_store_read_count: 60792,
            total_store_write_count: 48,
            total_store_deleteprefix_count: 0,
            store_size_bytes: 17065215,
            total_store_merging_time_ms: 0,
            store_currently_merging: true,
            highest_contiguous_block: 18248100
        },
        ModuleStats {
            name: "map_pool_events",
            total_processed_block_count: 100,
            total_processing_time_ms: 2480,
            external_call_metrics: [],
            total_store_operation_time_ms: 2,
            total_store_read_count: 123218,
            total_store_write_count: 0,
            total_store_deleteprefix_count: 0,
            store_size_bytes: 0,
            total_store_merging_time_ms: 0,
            store_currently_merging: false,
            highest_contiguous_block: 18248000
        },
        ModuleStats {
            name: "store_input_token_balances",
            total_processed_block_count: 100,
            total_processing_time_ms: 220,
            external_call_metrics: [],
            total_store_operation_time_ms: 0,
            total_store_read_count: 0,
            total_store_write_count: 8310,
            total_store_deleteprefix_count: 0,
            store_size_bytes: 99275,
            total_store_merging_time_ms: 0,
            store_currently_merging: false,
            highest_contiguous_block: 18248000
        },
        ModuleStats {
            name: "store_volume_by_token_amount",
            total_processed_block_count: 100,
            total_processing_time_ms: 312,
            external_call_metrics: [],
            total_store_operation_time_ms: 0,
            total_store_read_count: 0,
            total_store_write_count: 6536,
            total_store_deleteprefix_count: 6536,
            store_size_bytes: 106693,
            total_store_merging_time_ms: 0,
            store_currently_merging: false,
            highest_contiguous_block: 18248000
        }
    ],
    stages: [
        Stage { modules: ["map_pool_created", "store_pools"], completed_ranges: [BlockRange { start_block: 10008300, end_block: 18248100 }] },
        Stage { modules: ["store_output_token_supply"], completed_ranges: [BlockRange { start_block: 10008300, end_block: 18248100 }] },
        Stage { modules: ["map_pool_events", "store_input_token_balances", "store_volume_by_token_amount"], completed_ranges: [BlockRange { start_block: 10008300, end_block: 18248000 }] },
        Stage { modules: ["store_native_prices"], completed_ranges: [BlockRange { start_block: 10008300, end_block: 18248000 }] },
        Stage { modules: ["store_usd_prices"], completed_ranges: [BlockRange { start_block: 10008300, end_block: 18248000 }] },
        Stage { modules: ["map_events_entity", "map_token_entity", "store_volume", "store_pool_tvl"], completed_ranges: [BlockRange { start_block: 10008300, end_block: 18248000 }] },
        Stage { modules: ["store_cumulative_fields", "store_protocol_daily_fields", "store_protocol_tvl", "store_daily_and_hourly_fields"], completed_ranges: [BlockRange { start_block: 10008300, end_block: 18248000 }] },
        Stage { modules: ["map_liquidity_pool_snapshots_entity", "map_liquidity_pool_entity", "store_protocol_cumulative_fields"], completed_ranges: [BlockRange { start_block: 10008300, end_block: 18248000 }] },
        Stage { modules: ["map_financial_daily_snapshot_entity", "map_protocol_entity", "graph_out"], completed_ranges: [] }
    ],
    processed_bytes: Some(ProcessedBytes { total_bytes_read: 173213415, total_bytes_written: 0 })
}

...

[11:54:34.451] ModulesProgress {
    running_jobs: [],
    modules_stats: [
        ModuleStats {
            name: "store_output_token_supply",
            total_processed_block_count: 800,
            total_processing_time_ms: 27417,
            external_call_metrics: [],
            total_store_operation_time_ms: 2,
            total_store_read_count: 243168,
            total_store_write_count: 192,
            total_store_deleteprefix_count: 0,
            store_size_bytes: 17065215,
            total_store_merging_time_ms: 158,
            store_currently_merging: false,
            highest_contiguous_block: 18248100
        },
        ModuleStats {
            name: "store_input_token_balances",
            total_processed_block_count: 700,
            total_processing_time_ms: 2379,
            external_call_metrics: [],
            total_store_operation_time_ms: 0,
            total_store_read_count: 0,
            total_store_write_count: 58170,
            total_store_deleteprefix_count: 0,
            store_size_bytes: 277707832,
            total_store_merging_time_ms: 3235,
            store_currently_merging: false,
            highest_contiguous_block: 18248100
        },
        ModuleStats {
            name: "store_native_prices",
            total_processed_block_count: 600,
            total_processing_time_ms: 4731,
            external_call_metrics: [],
            total_store_operation_time_ms: 0,
            total_store_read_count: 29916,
            total_store_write_count: 39888,
            total_store_deleteprefix_count: 0,
            store_size_bytes: 150312696,
            total_store_merging_time_ms: 1062,
            store_currently_merging: false,
            highest_contiguous_block: 18248100
        }, ModuleStats {
            name: "map_events_entity",
            total_processed_block_count: 400,
            total_processing_time_ms: 2463,
            external_call_metrics: [],
            total_store_operation_time_ms: 0,
            total_store_read_count: 26556,
            total_store_write_count: 0,
            total_store_deleteprefix_count: 0,
            store_size_bytes: 0,
            total_store_merging_time_ms: 0,
            store_currently_merging: false,
            highest_contiguous_block: 18248100
        },
        ModuleStats {
            name: "store_protocol_tvl",
            total_processed_block_count: 300,
            total_processing_time_ms: 1572,
            external_call_metrics: [],
            total_store_operation_time_ms: 0,
            total_store_read_count: 0,
            total_store_write_count: 4986,
            total_store_deleteprefix_count: 0,
            store_size_bytes: 75,
            total_store_merging_time_ms: 50,
            store_currently_merging: false,
            highest_contiguous_block: 18248100
        },
        ModuleStats {
            name: "store_daily_and_hourly_fields",
            total_processed_block_count: 300,
            total_processing_time_ms: 32174,
            external_call_metrics: [],
            total_store_operation_time_ms: 5,
            total_store_read_count: 0,
            total_store_write_count: 58824,
            total_store_deleteprefix_count: 58824,
            store_size_bytes: 1790515,
            total_store_merging_time_ms: 28,
            store_currently_merging: false,
            highest_contiguous_block: 18248100
        },
        ModuleStats {
            name: "map_liquidity_pool_entity",
            total_processed_block_count: 200,
            total_processing_time_ms: 2528,
            external_call_metrics: [],
            total_store_operation_time_ms: 0,
            total_store_read_count: 29916,
            total_store_write_count: 0,
            total_store_deleteprefix_count: 0,
            store_size_bytes: 0,
            total_store_merging_time_ms: 0,
            store_currently_merging: false,
            highest_contiguous_block: 18248100
        },
        ModuleStats {
            name: "store_protocol_cumulative_fields",
            total_processed_block_count: 200,
            total_processing_time_ms: 603,
            external_call_metrics: [],
            total_store_operation_time_ms: 0,
            total_store_read_count: 0,
            total_store_write_count: 13072,
            total_store_deleteprefix_count: 0,
            store_size_bytes: 297,
            total_store_merging_time_ms: 51,
            store_currently_merging: false,
            highest_contiguous_block: 18248100
        },
        ModuleStats {
            name: "map_financial_daily_snapshot_entity",
            total_processed_block_count: 100,
            total_processing_time_ms: 2313,
            external_call_metrics: [],
            total_store_operation_time_ms: 0,
            total_store_read_count: 16620,
            total_store_write_count: 0,
            total_store_deleteprefix_count: 0,
            store_size_bytes: 0,
            total_store_merging_time_ms: 0,
            store_currently_merging: false,
            highest_contiguous_block: 18248100
        },
        ModuleStats {
            name: "map_pool_events",
            total_processed_block_count: 700,
            total_processing_time_ms: 23616,
            external_call_metrics: [],
            total_store_operation_time_ms: 24,
            total_store_read_count: 862526,
            total_store_write_count: 0,
            total_store_deleteprefix_count: 0,
            store_size_bytes: 0,
            total_store_merging_time_ms: 0,
            store_currently_merging: false,
            highest_contiguous_block: 18248100
        },
        ModuleStats {
            name: "map_token_entity",
            total_processed_block_count: 400,
            total_processing_time_ms: 688,
            external_call_metrics: [],
            total_store_operation_time_ms: 0,
            total_store_read_count: 0,
            total_store_write_count: 0,
            total_store_deleteprefix_count: 0,
            store_size_bytes: 0,
            total_store_merging_time_ms: 0,
            store_currently_merging: false,
            highest_contiguous_block: 18248100
        },
        ModuleStats {
            name: "store_cumulative_fields",
            total_processed_block_count: 300,
            total_processing_time_ms: 1490,
            external_call_metrics: [],
            total_store_operation_time_ms: 0,
            total_store_read_count: 0,
            total_store_write_count: 19608,
            total_store_deleteprefix_count: 0,
            store_size_bytes: 94410895,
            total_store_merging_time_ms: 1097,
            store_currently_merging: false,
            highest_contiguous_block: 18248100
        },
        ModuleStats {
            name: "graph_out",
            total_processed_block_count: 100,
            total_processing_time_ms: 433,
            external_call_metrics: [],
            total_store_operation_time_ms: 0,
            total_store_read_count: 0,
            total_store_write_count: 0,
            total_store_deleteprefix_count: 0,
            store_size_bytes: 0,
            total_store_merging_time_ms: 0,
            store_currently_merging: false,
            highest_contiguous_block: 18248100
        },
        ModuleStats {
            name: "store_pools",
            total_processed_block_count: 900,
            total_processing_time_ms: 708,
            external_call_metrics: [],
            total_store_operation_time_ms: 0,
            total_store_read_count: 0,
            total_store_write_count: 18,
            total_store_deleteprefix_count: 0,
            store_size_bytes: 83976637,
            total_store_merging_time_ms: 603,
            store_currently_merging: false,
            highest_contiguous_block: 18248100
        },
        ModuleStats {
            name: "store_usd_prices",
            total_processed_block_count: 500,
            total_processing_time_ms: 3454,
            external_call_metrics: [],
            total_store_operation_time_ms: 0,
            total_store_read_count: 16745,
            total_store_write_count: 8520,
            total_store_deleteprefix_count: 0,
            store_size_bytes: 14233271,
            total_store_merging_time_ms: 184,
            store_currently_merging: false,
            highest_contiguous_block: 18248100
        },
        ModuleStats {
            name: "store_protocol_daily_fields",
            total_processed_block_count: 300,
            total_processing_time_ms: 1697,
            external_call_metrics: [],
            total_store_operation_time_ms: 0,
            total_store_read_count: 0,
            total_store_write_count: 19608,
            total_store_deleteprefix_count: 19608,
            store_size_bytes: 451,
            total_store_merging_time_ms: 50,
            store_currently_merging: false,
            highest_contiguous_block: 18248100
        },
        ModuleStats {
            name: "map_pool_created",
            total_processed_block_count: 900,
            total_processing_time_ms: 23975,
            external_call_metrics: [
                ExternalCallMetric { name: "rpc: eth_call", count: 360, time_ms: 774 }
            ],
            total_store_operation_time_ms: 0,
            total_store_read_count: 0,
            total_store_write_count: 0,
            total_store_deleteprefix_count: 0,
            store_size_bytes 0,
            total_store_merging_time_ms: 0,
            store_currently_merging: false,
            highest_contiguous_block: 18248100,
        },
        ModuleStats {
            name: "store_volume_by_token_amount",
            total_processed_block_count: 700,
            total_processing_time_ms: 11667,
            external_call_metrics: [],
            total_store_operation_time_ms: 0,
            total_store_read_count: 0,
            total_store_write_count: 45752,
            total_store_deleteprefix_count: 45752,
            store_size_bytes: 674079,
            total_store_merging_time_ms: 15,
            store_currently_merging: false,
            highest_contiguous_block: 18248100
        },
        ModuleStats {
            name: "store_volume",
            total_processed_block_count: 400,
            total_processing_time_ms: 2379,
            external_call_metrics: [],
            total_store_operation_time_ms: 0,
            total_store_read_count: 13072,
            total_store_write_count: 19608,
            total_store_deleteprefix_count: 0,
            store_size_bytes: 88619831,
            total_store_merging_time_ms: 727,
            store_currently_merging: false,
            highest_contiguous_block: 18248100
        },
        ModuleStats {
            name: "store_pool_tvl",
            total_processed_block_count: 400,
            total_processing_time_ms: 2118,
            external_call_metrics: [],
            total_store_operation_time_ms: 0,
            total_store_read_count: 33240,
            total_store_write_count: 6648,
            total_store_deleteprefix_count: 0,
            store_size_bytes: 50919718,
            total_store_merging_time_ms: 573,
            store_currently_merging: false,
            highest_contiguous_block: 18248100
        },
        ModuleStats {
            name: "map_liquidity_pool_snapshots_entity",
            total_processed_block_count: 200,
            total_processing_time_ms: 6221,
            external_call_metrics: [],
            total_store_operation_time_ms: 6,
            total_store_read_count: 119664,
            total_store_write_count: 0,
            total_store_deleteprefix_count: 0,
            store_size_bytes: 0,
            total_store_merging_time_ms: 0,
            store_currently_merging: false,
            highest_contiguous_block: 18248100
        },
        ModuleStats {
            name: "map_protocol_entity",
            total_processed_block_count: 100,
            total_processing_time_ms: 438,
            external_call_metrics: [],
            total_store_operation_time_ms: 0,
            total_store_read_count: 8310,
            total_store_write_count: 0,
            total_store_deleteprefix_count: 0,
            store_size_bytes: 0,
            total_store_merging_time_ms: 0,
            store_currently_merging: false,
            highest_contiguous_block: 18248100
        }
    ],
    stages: [
        Stage {
            modules: ["map_pool_created", "store_pools"],
            completed_ranges: [BlockRange { start_block: 10008300, end_block: 18248100 }]
        },
        Stage {
            modules: ["store_output_token_supply"],
            completed_ranges: [BlockRange { start_block: 10008300, end_block: 18248100 }]
        },
        Stage {
            modules: ["map_pool_events", "store_input_token_balances", "store_volume_by_token_amount"],
            completed_ranges: [BlockRange { start_block: 10008300, end_block: 18248100 }]
        },
        Stage {
            modules: ["store_native_prices"],
            completed_ranges: [BlockRange { start_block: 10008300, end_block: 18248100 }]
        },
        Stage {
            modules: ["store_usd_prices"],
            completed_ranges: [BlockRange { start_block: 10008300, end_block: 18248100 }]
        },
        Stage {
            modules: ["map_events_entity", "map_token_entity", "store_volume", "store_pool_tvl"],
            completed_ranges: [BlockRange { start_block: 10008300, end_block: 18248100 }]
        },
        Stage {
            modules: ["store_cumulative_fields", "store_protocol_daily_fields", "store_protocol_tvl", "store_daily_and_hourly_fields"],
            completed_ranges: [BlockRange { start_block: 10008300, end_block: 18248100 }]
        },
        Stage {
            modules: ["map_liquidity_pool_snapshots_entity", "map_liquidity_pool_entity", "store_protocol_cumulative_fields"],
            completed_ranges: [BlockRange { start_block: 10008300, end_block: 18248100 }]
        },
        Stage {
            modules: ["map_financial_daily_snapshot_entity", "map_protocol_entity", "graph_out"],
            completed_ranges: [BlockRange { start_block: 18248000, end_block: 18248100 }]
        }
    ],
    processed_bytes: Some(ProcessedBytes { total_bytes_read:<PHONE_NUMBER>, total_bytes_written: 0 })
}

Not clear yet, further investigation will be required, I'll probably need to change a bit the log but most probably, we will need to get server logs to understand what is happening or add more info to the progress message.

trace.log
Server's trace log, I think we will need to reproduce again with the server's in debug mode to log more stuff. Will analyze the logs for now.

@trader-payne Pinax's log for your request ends at 13:00 but your ends as ~11:54, did you let it run until 13:00?

I don't know how long I let it run, but I can reproduce this again and talk to Pinax to grab logs together.
Shall I?

Yes, we will need more logs to understand what is happening, we will need the server to be in debug mode to collect deeper logs that would help the investigation. So if you can talk with Pinax to coordinate that, would be perfect.
I got my hands on Pinax's log for your query, but it's in INFO and it was not enough to better understand the problem, although we have some hypothesis.

Currently fetching logs for
QmQJovmQLigEwkMWGjMT8GbeS2gjDytqWCGL58BEhLu9Ag (Substreams Uniswap v3 Ethereum on Goerli)
so far it's synced, waiting for the next stall and will provide the logs

@maoueh
graph.log
It wasn't stalling for the past 24 hours...I had to restart the index-node then it stalled...Attached above are the logs

@jhjhjh94 At what timestamp would you pinpoint when the stall started. Also, do you have by chance more logs of that, ideally since the last Blockstreams disconnected, connecting.
Do you also have the logs of the backend server by chance?

@jhjhjh94 At what timestamp would you pinpoint when the stall started? Also, do you have by chance more logs of that, ideally since the last Blockstreams disconnected, connecting.
Do you also have the logs of the backend server by chance?

@maoueh The stall started the same time the log started
I just did another run (I had to manually restart the index node then it starts stalling, unable to reproduce the "sudden stall".
logs looks almost the same as the one I submitted above. (Waiting for next response....Next response received... once in a while will have "Received progress update" and repeats).

Next step will be to collect DEBUG logs on the Substreams server when the stall happen, tier1 logs filtered by trace ID of the request (my version and 0.33 of graph-node should both logs it on re-connection) will be enough as a starting point.
From the progress, we see that there is no running jobs anymore but data seems that nothing is then sent. We will need the Substreams server logs to correlate what is happening.
I've also slightly tweaked the Next response received to also print the message received, so will help with the stream visibility. The image graphprotocol/graph-node:canary-investigate-stuck-substreams-ccac28fd4 should be available soon (CI running right now).

@azf20 Printing the progress messages at regular interval will be something needed IMO to have. Essentially, it's having this commit https://github.com/graphprotocol/graph-node/commit/def370693f66915727028ddeb1695785e9877b78 re-worked so that progress messages are printed, we should also re-work the log presentation to be more concise.

Next step will be to collect DEBUG logs on the Substreams server when the stall happen, tier1 logs filtered by trace ID of the request (my version and 0.33 of graph-node should both logs it on re-connection) will be enough as a starting point.
From the progress, we see that there is no running jobs anymore but data seems that nothing is then sent. We will need the Substreams server logs to correlate what is happening.
I've also slightly tweaked the Next response received to also print the message received, so will help with the stream visibility. The image graphprotocol/graph-node:canary-investigate-stuck-substreams-ccac28fd4 should be available soon (CI running right now).

@maoueh Here are the logs (Trace ID : 6f1bf5a6f4f593b59fd9c3a35789fbd3) I have attached additional unfiltered logs just in case you need more data. The beginning of the logs corresponds to the time when the stall happened.
graphnode.log
substreams_stall.txt (Only trace_id : 6f1bf5a6f4f593b59fd9c3a35789fbd3)
substreams_tier1.txt (Only tier1)
substreams.log (All)

Next step will be to collect DEBUG logs on the Substreams server when the stall happen, tier1 logs filtered by trace ID of the request (my version and 0.33 of graph-node should both logs it on re-connection) will be enough as a starting point.
From the progress, we see that there is no running jobs anymore but data seems that nothing is then sent. We will need the Substreams server logs to correlate what is happening.
I've also slightly tweaked the Next response received to also print the message received, so will help with the stream visibility. The image graphprotocol/graph-node:canary-investigate-stuck-substreams-ccac28fd4 should be available soon (CI running right now).

@maoueh Attached below are the logs corresponds to the time when the stall happened. I have attached additional logs in case you need more data.
graphnode.log
substreams_stall.txt (Only trace_id 6f1bf5a6f4f593b59fd9c3a35789fbd3)
substreams_tier1.txt (Only tier1)
substreams.log (All)

@jhjhjh94 Thank you, this seems to be a bit different than your last graph node logs. Indeed, I see running_jobs here which means works is being done (but maybe slowly):
ModulesProgress { running_jobs: [
    Job { stage: 4, start_block: 18332000, stop_block: 18333000, processed_blocks: 517, duration_ms: 233705 },
    Job { stage: 3, start_block: 18332000, stop_block: 18333000, processed_blocks: 747, duration_ms: 233695 },
    Job { stage: 2, start_block: 18332000, stop_block: 18333000, processed_blocks: 877, duration_ms: 233686 },
    Job { stage: 1, start_block: 18332000, stop_block: 18333000, processed_blocks: 898, duration_ms: 233677 },
    Job { stage: 0, start_block: 18332000, stop_block: 18333000, processed_blocks: 894, duration_ms: 233668 },
    Job { stage: 0, start_block: 18333000, stop_block: 18333002, processed_blocks: 0, duration_ms: 233659 }
], modules_stats: [], stages: [
Stage {
    modules: ["map_pools_created", "store_pools_created", "store_tokens", "store_pool_count"],
    completed_ranges: [BlockRange { start_block: 12369621, end_block: 18332000 }]
},
Stage {
    modules: ["map_extract_data_types", "map_tokens_whitelist_pools", "store_pool_liquidities", "store_native_amounts", "store_tokens_whitelist_pools", "store_prices", "store_token_tvl", "store_total_tx_counts", "store_pool_sqrt_price", "store_positions", "store_ticks_liquidities"],
    completed_ranges: [BlockRange { start_block: 12369621, end_block: 18332000 }] },
Stage {
    modules: ["store_eth_prices"],
    completed_ranges: [BlockRange { start_block: 12369621, end_block: 18332000 }] },
Stage {
    modules: ["store_derived_tvl", "store_swaps_volume", "store_min_windows", "store_max_windows"],
    completed_ranges: [BlockRange { start_block: 12369621, end_block: 18332000 }] },
Stage {
    modules: ["store_derived_factory_tvl"],
    completed_ranges: [BlockRange { start_block: 12369621, end_block: 18332000 }] },
Stage {
    modules: ["graph_out"],
    completed_ranges: []
}],

You can see here the running jobs they are still crunching so work is in progress. While it "seems" stuck, it's not really in the case. Could be seen as "still working" more than "stuck".

For the presentation of the actual progress @zorancv  @azf20 message, I’m thinking maybe that we could have a succinct presentation in INFO and a deeper one in DEBUG/TRACE. We could define the important pieces on INFO to be stages (ID + completed range) and running_jobs (processed range Job (s0) 18,000,000 | +573/1000 maybe as a compact representation).
I've also rebase my debugging branch on top of v0.33.0, Docker image should is available with tag canary-investigate-stuck-substreams-v0.33.0-2dbfca2d7

@maoueh this PR was merged which should help with monitoring here https://github.com/graphprotocol/graph-node/pull/4935
Are there open threads on this issue?

Are there open threads on this issue?

I don't understand the question, which "thread"?

@maoueh I meant are there any open investigations on this issue / what are the next steps?

I sent some notes to @maoueh in a discussion thread in slack based. This shows substreams stall without any subgraph being involved, with debugging enabled on the server.
My testcase (referencing a pinax local tier1 node):
substreams-sink-noop wax-sfst85:9000 https://github.com/pinax-network/substreams-atomicmarket/releases/download/v0.3.0/atomicmarket-v0.3.0.spkg graph_out 32848800: --plaintext

@azf20 We were waiting on DEBUG logs from the production service that runs the Substreams but Matthew seems that the logs he gave us might contain the information we need.
Now we need someone internally to check that more thoroughly.

There is a bug pending BSC dev team to fix which might have caused some issues, however there is currently a workaround https://github.com/bnb-chain/bsc/issues/2212

I believe this issue issue is reproducible by simply using the substreams cli.  (ie not a graph-node issue).  I sent a DEBUG log.  From my reading of the log it does not seem to explain why things are stuck.

New substreams release today. Let's see if there is some improvement here.

Some improvements, but substreams still get stuck.
It feels to me the problem is the handoff between processing historical blocks to live blocks when the substream has no "store" module.

Github issue for above: https://github.com/streamingfast/substreams/issues/421

Suggest to close this issue, and use the substreams issue above to continue this discussion.


CreateOrUpdate Snapshot REST API - No mention of how to specify that the disk is a managed disk

This issue was created by @meetamitbhatia from MicrosoftDocs/feedback#273
The REST API documentation for creating disk snapshots can be found at https://docs.microsoft.com/en-us/rest/api/compute/snapshots/createorupdate. It makes no mention of how to specify that the disk is a managed disk.

@Powerhelmsman transferred this issue

@wnjenkin
Thanks for the feedback! We are currently investigating and will update you shortly.

@wnjenkin I was able to take a snapshot on a Managed Disk without specifying it was a Managed Disk via the REST API.  I directed a comment to the customer in the original issue.


Automation

Hello,
Is it suppose to automatically create a new certificate when a new path is add in the load balancer ?
(Almost the same way JrCs/docker-letsencrypt-nginx-proxy-companion do ? )
Or Am I wrong ? and domain must be add to environment variable in letsencrypt-nginx and restarted ? )
Regards,

Yes, the domains must be specified in the environment.


Add CI for wayland.zig

[x] Add a workflow to clone users and test the new release
[x] Add a cron workflow to fetch dependencies and make pull request

Available here:

https://github.com/tiawl/wayland.zig/blob/trunk/.github/workflows/bot.yml
https://github.com/tiawl/wayland.zig/blob/trunk/.github/workflows/ci.yml


This example doesn't work as-is

I cloned this repository and went through the quick-start guide, replacing all the relevant fields in config.js (clientId, clientSecret and identityMetadata). The profile object passed via callback from a successful authentication was just fields with undefined values - "oid" being one of them, causing the example to fail.
I fixed this by changing the scope to ['profile'].

@HSAR Thank you, I included the fix in master branch.


Tensorflow support for misc. pruning algorithms

Hi,
I see that almost all the pruning algorithms are supported for PyTorch.
When there will be support for TF?
Thanks,
Ilya

Currently we don't have enough devs working on TensorFlow support, so the progress will be relatively slow.
You can suggest the algorithm you most want to try, and we will put it on higher priority.


showYearPicker disabled years

Is there a way I could remove all the disabled years when using showYearPicker? Or just adjust them with the yearItemNumber?
<DatePicker
minDate={new Date()}
maxDate={new Date(new Date().getFullYear() + 40, 0)}
showYearPicke
dateFormat={"yyyy"}
/>
I have 2017-2022 years in disabled along with 2064, but I want to remove these disabled years.

@ekiankim92
Remove maxDate property from DatePicker and you can select any date.
<DatePicker
minDate={new Date()}
showYearPicke
dateFormat={"yyyy"}
/>

If you want to remove minDate disabled as well then you can remove that as well.
<DatePicker
showYearPicke
dateFormat={"yyyy"}
/>

@aziyatali
What if I was to set a specific year range? I wanted to show the year range from this year all the way through 2035. I have set to show the years with the yearItemNumber={6} but when I hit the last year to see the 2035, this will show the disabled 2036 through 2040.
<DatePicker minDate={new Date()} maxDate={new Date(new Date().getFullYear() + 12, 0)} showYearPicker yearItemNumber={6} dateFormat={"yyyy"} />

@ekiankim92
That's because you choose  yearItemNumber as 6, which means it will show you 6 years per tab.
You can't hide 2036-2040 for above reason.
What you can do is, either set yearItemNumber to 1, or below code:

() => {
  const [startDate, setStartDate] = useState(new Date());
  return (
    <DatePicker
    minDate={new Date()}
    maxDate={new Date(new Date().getFullYear() + 17, 0)}
    showYearPicker
    yearItemNumber={Math.max(6,1)}
    dateFormat={"yyyy"} />
  );
};

Please close your issue if resolved.

thank you for your assitance!

@ekiankim92  If one of the above answer your question, please mark as an answered. So, next time someone stuck they can see what to do.
Please refer to this: https://docs.github.com/en/discussions/collaborating-with-your-community-using-discussions/participating-in-a-discussion


https://v2.vuepress.vuejs.org/ stuck at beta.26

Just wanted to make you aware that the official documentation site https://v2.vuepress.vuejs.org/ seems to be stuck at beta.26

Noticed this while trying to update to latest beta and noticed that breaking changes mentioned in the release notes were not reflected on the documentation site yet.

Really sorry for that.
We removed PWA from our doc site, because the PWA update process took quite a long time, and we did not see much benifit of PWA.
So the version would get stuck if users had visited our site previously.
To fix that, press F12 to open devtools, nav to Application tab, and manually remove the service worker:

Apologize again

Good old hard refresh fixed it - thanks :-D


do not cleanup ongoing compact files using fabric:cleanup_index_files/1

Overview

This PR is not to cleanup ongoing compact files using fabric:cleanup_index_files/1
Testing recommendations

make exunit tests=src/couch/test/exunit/fabric_test.exs
17:55:56.045 [debug] Calling 'teardown/3' for 'Elixir.Couch.Test.Setup.Step.Config'
  * test Fabric miscellaneous API Get inactive_index_files (1146.4ms)

Finished in 1.2 seconds
1 test, 0 failures

Related Issues or Pull Requests

https://github.com/apache/couchdb/pull/2101
Checklist

[X] Code is written and works correctly
[X] Changes are covered by tests
[ ] Any new configurable parameters are documented in rel/overlay/etc/default.ini
[ ] A PR for documentation changes has been made in https://github.com/apache/couchdb-documentation

The description of the PR is misleading. It should be updated to "remove view compaction files on view cleanup".

Is there a reason you think couch_mrview_cleanup:run(DbName). shouldn't be used here?

The description of the PR is misleading. It should be updated to "remove view compaction files on view cleanup".

I was wrong you description is good.

Is there a reason you think couch_mrview_cleanup:run(DbName). shouldn't be used here?
It is couch_mrview_cleanup:run(db) instead of  couch_mrview_cleanup:run(DbName). So it is shard db level instead of db level.

Thank you. I also take time to get better understanding on logic :-)


Output to real hardware

Hi,
I have a roland sc-55 connected to my computer with a usb to midi cable, is there any way i can configure this plugin to output to the sc-55?

Short version: No
Longer version:
While foo_midi works like a sequencer by sending messages to a 'device', it does except the 'device' to respond with a rendered version of the messages in the form of sound samples. Your hardware device will render directly to the audio subsystem of Windows. foo_midi has nothing to pass on to foobar2000 to play.

I'm aware of some previous discussions on this, but now that the development has changed hands, maybe you'd consider the "passthru" approach? Feed silence to foobar while sending midi commands to the system device.

I'll read the discussion later but the question remains: why use an audio player instead of a dedicated sequencer application? And how would I test that?

Well the "system device" in question would be the Microsoft GS SoftSynth by default (on Windows). Since the build instructions are for Visual Studio, I assume you'd be able to test this quite easily.

I'm intrigued, maybe later. But first I want to get to a stable version of foo_vis_spectrum_analyzer.

I started implementing this feature but I ran into a design problem: currently players get fed MIDI events in bursts based on the duration of an audio chunk (typically about 10ms worth). The timestamps of the events have been converted to sample offsets during pre-processing because the 'clock' is based on sample buffer requests. Feeding this stream type to the SoftSynth using midiOutShortMsg()  works but there are a lot of hick-ups in the playback because the call takes a while to return and seems to miss events that are sent too fast. The alternative is to use the MIDI stream buffer API but that requires the delta timestamps to be intact. It's going to take a lot (too much?) of rework to shoehorn this feature in the current design.

For what it's worth, I tried to implement something similar for a hardware YMF262 over a serial link, and ran into the same issue.


checksum failure when downloading

Just wanted to include a link to this bug, https://github.com/ansible/ansible/issues/18894, I came across it and followed CorbanR/streisand@39ad603 , setting the temp dirs in ansible.cfg to fix the issue. Linking here as it may help someone in the future.
Feel free to close this.

thanks for the tip!


Merge Dev into Main

NEW FEATURES -
/docs - searches the python docs for matches to the search query, returns results.
/define - uses a dictionary API to define words.

 Cant seem to get that right so it works in python for the linter, as well as docker.

Fixes:
https://github.com/practical-python-org/ZorakBot/issues/103
https://github.com/practical-python-org/ZorakBot/issues/102


Make issue key parsing case insensitive

This modifies the issue key regex to match lowercase characters.
In addition it adds a transform to the moo.js issue key tokenizer to maintain the same all caps output in the event that anything Jira side is case sensitive (though I don't believe it is).
I also added a lowercase test.

Related issue: https://github.com/integrations/jira/issues/220

@justinoue thanks for this contribution! The jira project is currently transitioning ownership within GitHub so I can't merge this right away, but I'm going to pass this on to the new team to take care of and make sure there aren't any unintended consequences with this! I really appreciate this work!

@justinoue thanks for this contribution! The jira project is currently transitioning ownership within GitHub so I can't merge this right away, but I'm going to pass this on to the new team to take care of and make sure there aren't any unintended consequences with this! I really appreciate this work!

Cool, thanks for the update. Here's to hoping nothing breaks. 🤞

is there any update to this?

is there any update to this?

Hey @rachellerathbone , while I don't use jira anymore, this is still a little disappointing.
This does not require the Jira application to support mixed case or lowercase issue keys.
All this does is parse things in brackets case-insensitively. The URL it generates will still be all uppercase. (Although for what it's worth, Jira does redirect lower/mixed case issue keys to uppercase.)
[Test-1]
[test-1]
[TEST-1]
Would all then be parsed and link to https://myjira.atlassian.net/browse/TEST-1
Without this change, only [TEST-1] works, and require you to yell all the time about your issues.

Hey @justinoue. I completely understand your frustrations. Unfortunately, while it seems like a simple change, it's a little more complicated on our end.
This is actually a significant change that we would have to consider across all our providers. We could end up in situations where a customer's code is case insensitive, but their CI/CD integration is case sensitive. Then there is the issue with false positives. This is potentially a change that we could introduce in the future but it would require thorough investigation (i.e evaluating existing data to see what percentage of new hits are true positives vs false positives).
I've passed this onto the team and we're going to add a feature request for making this improvement into our backlog.


Behavior of rules in character sets is underspecified

Roast has many tests where a rule is combined into a character set.  For example, S05-metasyntax/charset.t.  However, all of the current tests use a rule that can match against a single character.  This means that the behavior of a token that requires multiple characters is unspecified.
I believe that the correct behavior is, to quote jnthn from a different issue:

<+foo>  if foo matches more than one char [means that]: it has to match, but we only advance the cursor by 1 position however much it matches.

Unless anyone disagrees with this behavior, it should be specced, probably in S05-metasyntax/charset.t.

Cross references:

Raku/problem-solving#97
rakudo/rakudo#5430
Raku/doc#4389


Double click string in file manager editor

When one double clicks on a string in the file manager editor what is the expected behavior? I thought it used to highlight the variable name but may be confusing editors, currently it shrinks the file editor window.

Hi,
Yes, right. I remembered about it but than forgot.
Thanks, I will add it to 18.49-7 in few moments.


Add bottom padding to iPhones with home bar

My best shot at addressing this issue. Uses media queries to target the specific iPhones that have a home bar instead of a physical home button, adding 34px padding to the bottom of the navigation panel to match dead bird site if media conditions are met. Media queries sourced from this SO answer.
Tested only on a friend's iPhone 12 and I couldn't take screenshots, but I'm 95% this works as follows:
Previous behavior:

Mastodon nav panel has correct (read: zero) padding above the Safari navigation bar in Safari browser upon load 👍
Mastodon nav panel overlaps the home bar in Safari browser upon scroll (when Safari navigation bar collapses) 👎
Mastodon nav panel overlaps the home bar in Safari PWA at all times 👎

New behavior:

Mastodon nav panel has unnecessary padding above the Safari navigation bar in Safari browser upon load 👎
Mastodon nav panel has correct padding above the home bar in Safari browser upon scroll (when Safari navigation bar collapses) 👍
Mastodon nav panel has correct padding above the home bar in Safari PWA at all times 👍

Don't know how to address the new unnecessary padding above the Safari navigation bar; setting min-height values to account for it don't seem to work. Even if that would work, it would only work for Safari as Chrome's navigation bar is a completely different height, and you cannot detect useragents/browsers using CSS media queries.
Also, I'm not sure that this code is necessary or even does anything in the multiple column layout. I tested it and can notice zero change in behavior whatsoever. At a minimum I don't think it contradicts any existing code, so I added it into that stylesheet also.

Well yeah, I'm going to revert :sweat_smile:


Feature: Detailed Output with Verbose Mode

Description: Implement a verbose mode that provides additional details about each image during cleanup. When the --verbose flag is passed, include information such as image size, creation date, and more.
Example:

./dockclean --verbose

Hi @mmuazam98, I would love to contribute to this project. Can you assign this issue to me.

Could you also clarify on the purpose of dockclean file. If its a binary can I .gitignore that one

@deexithparand thanks for pointing this out. Have removed the binary and added .gitignore


Pre-commit support

Would be nice with support and guide to use Harper with pre-commit tool just like typos CLI tool do

Not sure if this helps but harper provides harper-cli which you can use to write a script to check all files.
This is a bit quick and dirty but works.
find . -type f -not -iwholename '*.git*' -exec harper-cli lint {} \;

You can install harper-cli using the following command: cargo install --git https://github.com/elijah-potter/harper harper-cli. I will note that harper-cli doesn't follow any dictionaries currently

@lukasmwerner is absolutely right. I will caution that harper-cli currently exists primarily for testing purposes. If anyone is interested in making changes toward turning it into a more robust experience for tools like pre-commit, feel free to leave a comment on this issue and we can make it happen. I personally don't have time to expand scope in this way at the moment.


Requirements/Setup Issues

Just thought I'd let you know in order to get this working I had to downgrade from Python 3.8 to 3.7 so I could use a tensorflow version < 2.0 as it uses a bunch of removed features. I also had to downgrade numba to 0.43.0. It also is unable to find cudart64_100.dll but that's optional so it isn't a huge deal.
PS How do I format my transcript file for training. I was looking at some of the examples and it looked liike they formatted it differently and I couldn't find any clear documentation on how it should be formatted.
PPS I messaged you on reddit first, but I figured I'd make an issue as well.

I believe the cudart64_100.dll error is related to the installation of cuda. For tensorflow_gpu-1.13.1 (what I used), you need to install CUDA 10.0 and cuDNN 7.4. Note that different versions have compatibility issues with one another so you need to stick with the official recommended versions. If you are installing on windows, CUDA 10.0 also requires visual studio 2017 to be installed (see here).
In the dc_tts repo the author provided two methods of processing training data. I used the method for the LJ dataset, which means that the transcript.csv needs to follow the metadata convention in the LJ speech dataset. The format is filename|transcript|normalized_transcript, where normalized transcripts have numerals and symbols fully in words.
For example, this is a line from the LJ dataset:

LJ002-0009|On the 27th April, in the following year,|On the twenty-seventh April, in the following year,

Okay, cool. Thank you my dude. I got a little confused as to what I actually needed to do since the LJ dataset had different formatting than the Bible one. Those were the 2 I looked at.


Fix min build

Closes: https://github.com/graphql/graphiql/issues/528

One thing to note about switching from -g (undocumented, as far as I can tell, "global" transform) to -t is that the built size may be different. I'm seeing a 30KB difference. Given the kinds of devices/networks where GraphiQL is used, this is an acceptable regression (and I am not even sure how much of it is due to the switch), so let's move forward with this in order to correct the breakage. Can loop back later if we think it's worthwhile.


lsp won't start without lsp-java

What did you expect to happen?
I opened a python file expected lsp to work.
What actually happened?
lsp cannot started automatically and when I try to start it I get (file is missing "Cannot open load file" "No such file or directory" "lsp-java").  If i add (java +lsp) there's no longer an error.
Additional details:
Whenever I try to start lsp:
Debugger entered--Lisp error: (file-missing "Cannot open load file" "No such file or directory" "lsp-java")
  require(lsp-java)
  (lambda nil (require 'lsp-java))()
  eval-after-load-helper("/home/justin/.config/chemacs/doom-emacs/.local/straight/build/lsp-mode/lsp-mode.elc")
  run-hook-with-args(eval-after-load-helper "/home/justin/.config/chemacs/doom-emacs/.local/straight/build/lsp-mode/lsp-mode.elc")
  do-after-load-evaluation("/home/justin/.config/chemacs/doom-emacs/.local/straight/build/lsp-mode/lsp-mode.elc")
  (lsp-deferred)
  (if nil (eglot-ensure) (lsp-deferred))
  lsp!()

Steps to reproduce:
Have lsp but not (java +lsp) in your config.

emacs   version    27.0.91
        features   XPM JPEG TIFF GIF PNG RSVG CAIRO SOUND GPM DBUS GLIB NOTIFY INOTIFY ACL GNUTLS LIBXML2 FREETYPE HARFBUZZ LIBOTF ZLIB TOOLKIT_SCROLL_BARS GTK3 X11 XDBE XIM MODULES THREADS LIBSYSTEMD JSON PDUMPER LCMS2 GMP
        build      Jul 25, 2020
        buildopts  (--prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib --localstatedir=/var --mandir=/usr/share/man --with-gameuser=:games --with-sound=alsa --with-modules --without-gconf --without-gsettings --with-x-toolkit=gtk3 --without-xaw3d --without-m17n-flt --with-cairo --without-compress-install 'CFLAGS=-march=x86-64 -mtune=generic -O2 -pipe -fno-plt -g -flto -g -flto -s -fuse-ld=gold' CPPFLAGS=-D_FORTIFY_SOURCE=2 LDFLAGS=-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now)
        windowsys  x
        daemonp    server-running
doom    version    2.0.9
        build      HEAD -> develop, origin/develop, origin/HEAD afcf56a61 2020-07-24 19:27:50 -0400
        dir        ~/.doom.d/
system  type       gnu/linux
        config     x86_64-pc-linux-gnu
        shell      /bin/zsh
        uname      Linux 5.7.10-arch1-1 #1 SMP PREEMPT Wed, 22 Jul 2020 19:57:42 +0000 x86_64
        path       (~/.local/bin/ ~/.config/chemacs/doom-emacs/bin/ /usr/lib/ccache/bin/ ~/bin /usr/local/bin /usr/local/sbin /usr/bin /usr/lib/jvm/default/bin /usr/bin/site_perl /usr/bin/vendor_perl /usr/bin/core_perl /usr/lib/emacs/27.0.91/x86_64-pc-linux-gnu)
config  envfile    nil
        elc-files  0
        modules    (:completion (company +childframe) (ivy +fuzzy +childframe +icons +prescient) :ui doom doom-dashboard modeline ophints (popup +defaults) treemacs unicode vc-gutter vi-tilde-fringe workspaces :editor (evil +everywhere) file-templates :emacs dired electric undo vc :checkers syntax spell grammar :tools ansible editorconfig (eval +overlay) (lookup +dictionary +offline +docsets) (lsp +peek) magit make pdf rgb :lang (cc +lsp) csharp emacs-lisp (haskell +dante) javascript latex lua markdown org raku php (python +lsp) (ruby +rails) (rust +lsp) sh (web +lsp) (yaml +lsp) :email notmuch :config (default +bindings +smartparens))
        packages   ((evil-snipe :disable t) (pkgbuild-mode) (puppet-mode))
        unpin      (n/a)
        elpa       (n/a)

Does this occur with the latest version of doom?

Just ran doom upgrade and doom sync and this is still the case.

I downloaded your configuration and tried to reproduce by opening a python file, but it worked for me without errors from lsp-java. I see you are using chemacs, can you try installing doom without chemacs and check if the problem persists?


Password and encryption

Will make longer random/custom password as an option? Like 16-digit or more.
And I wonder know what encryption method it uses.

I'm working on a new version and will probably bump the password length to 8 characters. Not sure if I'll put in longer custom passwords, I'll think about it. A longer password shouldn't have too much security implication: once the transfer is started, only one connection will be made. If an attacker shoulder-surfed the password or QR code (as the new version will have for connections involving Android/iOS), they'd be able to hijack the transfer, but it would be clear on the other device that it was not sending or receiving.
The encryption used in transit is AES-GCM. The password key-derivation function is scrypt, though this will likely be removed in the new version in favor of just SHA-256. I can't find a good version of scrypt or Argon2 in Kotlin, Swift, and Rust, and the new version will support Android.
A computationally-expensive key derivation function isn't actually necessary because nothing is being stored in encrypted form, so there's no hash to crack later. Additionally, all ad hoc WiFi connections made by Flying Carpet are protected with WPA2, so the data transferred is already encrypted. Why the second layer of encryption then, you might ask? Because when I first started this project in 2016 or so, ad hoc WiFi connections hosted by macOS didn't necessarily have any encryption, and I haven't bothered to change how it works since. Now, Apple doesn't let you run hotspots programmatically on macOS or iOS, so it's a non-issue. I'll probably leave the AES-GCM in for now, though may remove later if it improves performance significantly.
Thank you for the suggestion!

The new versions are out and I changed the password length to 8. Still have this on the list for consideration but closing this issue for now.


fix(network): Updates for fork

:tada: This PR is included in version 3.8.10 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Add basic build information to readme, add build task to VSCode

This change set adds a bit of documentation about about the basic setup required to build the demo. It also adds a rebuild task for use inside of VSCode.

@super-triangle I've also copied the newly suggested build section to Doom 32X Resurrection's README.md on github: https://github.com/viciious/d32xr#building along with the suggested settings.json: would you mind reviewing the latter?

Sounds good! I can check it out after work.

Hey @viciious, had a chance to clone d32xr and try a clean setup in VSCode. One thing though, you mentioned a "suggested settings.json", but I only included a "tasks.json" in my PR. The settings.json you've added is from your local copy, I presume?
Anyway, the d32xr tasks.json looks good, and build task runs as expected. The output seems to be the same with or without the settings.json from the README though? I can't fully verify things though, as I can't figure out how to correctly dump doom32x.wad, so the build fails when it attempts to copy the missing wad.
Linux isn't my primary OS, and couldn't figure out how to get DD to extract it at the offset you mentioned. Would it be possible to also document an example of the extraction command one would run to pull the original wad?

Oh, perhaps I should have been more clear from the get go: the suggested settings.json for D32XR should also work well for YATSSD. At least the compiler'd pickup proper compilation flags from the Makefile and you'd be getting proper IntelliSense tips as the result.

Hi @viciious, thanks for clarifying. After creating the suggested settings.json in YATSSD folder and installing Makefile Tools the target and build definitions options to be picked up correctly in Makefile Tools.
Setting the configuration to use Release or Debug doesn't seem to work unless you update the Makefile to add them:
release: m68k.bin $(TARGET).32x
debug: m68k.bin $(TARGET).32x
all: release
Also re: intellisense, SH includes are working now and things like INT32_MAX are correctly resolved to /opt/toolchains/sega/sh-elf/sh-elf/include/stdint.h. On the other hand, the m68k code seems to also resolve to the sh-elf versions of the includes. Should the be finding the m68k versions?

On Sun, 8 Jan 2023 at 02:08, Norah S @.***> wrote:

Hi @viciious https://github.com/viciious, thanks for clarifying. After
creating the suggested settings.json in YATSSD folder and installing Makefile
Tools
https://marketplace.visualstudio.com/items?itemName=ms-vscode.makefile-tools
the target and build definitions options to be picked up correctly in
Makefile Tools.
Setting the configuration to use Release or Debug doesn't seem to work
unless you update the Makefile to add them:
release: m68k.bin $(TARGET).32x
debug: m68k.bin $(TARGET).32x
all: release

Oh yeah, I’ll be adding them to the makefile soon, thanks!
Also re: intellisense, SH includes are working now and things like

INT32_MAX are correctly resolved to
/opt/toolchains/sega/sh-elf/sh-elf/include/stdint.h. On the other hand,
the m68k code seems to also resolve to the sh-elf versions of the includes.
Should the be finding the m68k versions?

I don’t think that can actually work, unless it somehow picks up the
sub-makefile in the src-md directory, I’m not sure..
—

Reply to this email directly, view it on GitHub
https://github.com/viciious/yatssd/pull/2#issuecomment-1374630211, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAI6MQTNO2BCHBBAX46OBATWRHSM7ANCNFSM6AAAAAATRQJPKQ
.
You are receiving this because you were mentioned.Message ID:
@.***>

--
Best regards,
Victor Luchitz


[Event Request] - New event OnBeforeOKOnPush inside Page 12111 "Show Computed Withh. Contrib."

Hello, please consider adding the following event to Page 12111 "Show Computed Withh. Contrib."
    [IntegrationEvent(false, false)]
    local procedure OnBeforeOKOnPush(var TmpWithHoldContr: Record "Tmp Withholding Contribution"; var IsHandled: Boolean)
    begin
    end;

and call it inside the "OKOnPush" procedure :
    local procedure OKOnPush()
    var
        **IsHandled: Boolean;**
    begin

        **IsHandled := false;
        OnBeforeOKOnPush(Rec, IsHandled);
        if IsHandled then
            exit;**

        GenJnlLine2.Reset();
        GenJnlLine2.SetRange("Journal Template Name", "Journal Template Name");
        GenJnlLine2.SetRange("Journal Batch Name", "Journal Batch Name");
        GenJnlLine2.SetRange("Line No.", "Line No.");
...
...

Thanks!

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. Please do not reply to this, as we do not monitor closed issues. If you have follow-up questions or requests, please create a new issue where you reference this one.
Build ID: 53542.


Hour select broken in TimePicker

Environment

Tech
Version

material-ui-pickers
1.0.0-rc.5

material-ui
1.0.0-beta.41

React
16.2.0

Browser
Chrome 62.0.3202.94

Platform
Linux Mint  18

Steps to reproduce

Open TimePicker
Attempt to set hours

Expected behavior
Hour should be set when clicking on the clock.
Actual behavior
Can't select hour when popup appears. Only minutes are updated.
This doesn't appear to be a problem when using material-ui-picker 1.0.0-rc.4.
Live example
https://codesandbox.io/s/2vr91op1v0

Having same issue!

Already fixed in develop
Will appear as fix in the nearest release (this week)

selected hour is not getting updated only minutes are getting updated.


PKCS7_verify can add errors to stack even when successful - alpha17

Similar to #14996 it is possible to make a call to PKCS7_verify that succeeds in the 3.0.0 alphas, but adds errors to the stack. Reproducer:
#include <openssl/bio.h>
#include <openssl/pkcs7.h>
#include <openssl/pem.h>
#include <openssl/err.h>
#include <assert.h>
#include <string.h>

int main(void) {
    int res = 0;
    const char *sig = "MIME-Version: 1.0\nContent-Type: multipart/signed; protocol=\"application/x-pkcs7-signature\"; micalg=\"sha-256\"; boundary=\"----9B5319FF2E4428B17CD26B69294E7F31\"\n\nThis is an S/MIME signed message\n\n------9B5319FF2E4428B17CD26B69294E7F31\nContent-Type: text/plain\r\n\r\nhello world\n------9B5319FF2E4428B17CD26B69294E7F31\nContent-Type: application/x-pkcs7-signature; name=\"smime.p7s\"\nContent-Transfer-Encoding: base64\nContent-Disposition: attachment; filename=\"smime.p7s\"\n\nMIIDEgYJKoZIhvcNAQcCoIIDAzCCAv8CAQExDzANBglghkgBZQMEAgEFADALBgkq\nhkiG9w0BBwGgggFVMIIBUTCB96ADAgECAgIDCTAKBggqhkjOPQQDAjAnMQswCQYD\nVQQGEwJVUzEYMBYGA1UEAwwPY3J5cHRvZ3JhcGh5IENBMB4XDTE3MDEwMTEyMDEw\nMFoXDTM4MTIzMTA4MzAwMFowJzELMAkGA1UEBhMCVVMxGDAWBgNVBAMMD2NyeXB0\nb2dyYXBoeSBDQTBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABBj/z7v5Obj13cPu\nwECLBnUGq0/N2CxSJE4f4BBGZ7VfFblivTvPDG++Gve0oQ+0uctuhrNQ+WxRv8GC\n177F+QWjEzARMA8GA1UdEwEB/wQFMAMBAf8wCgYIKoZIzj0EAwIDSQAwRgIhANES\n742XWm64tkGnz8DnpG6u2lHkZFQr3oaVvPcemvlbAiEA0WGGzmYx5C9UvfXIK7NE\nziT4pQtyESE0uRVKXw4nMqkxggGBMIIBfQIBATAtMCcxCzAJBgNVBAYTAlVTMRgw\nFgYDVQQDDA9jcnlwdG9ncmFwaHkgQ0ECAgMJMA0GCWCGSAFlAwQCAQUAoIHkMBgG\nCSqGSIb3DQEJAzELBgkqhkiG9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTIxMDUyMDE4\nNTA0OVowLwYJKoZIhvcNAQkEMSIEIOdwMRgQrqcnmMYvag+BVvErcc6bwUXI94Ds\nQkiyIU9pMHkGCSqGSIb3DQEJDzFsMGowCwYJYIZIAWUDBAEqMAsGCWCGSAFlAwQB\nFjALBglghkgBZQMEAQIwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwICAgCAMA0GCCqG\nSIb3DQMCAgFAMAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMAoGCCqGSM49BAMCBEcw\nRQIhANYMJku1fW9T1MIEcAyREArz9kXCY4tWck5Pt0xzrYhaAiBDSP6e43zj4YtI\nuvQW+Lzv+dNF8EPuhgoPNe17RuUSLw==\n\n------9B5319FF2E4428B17CD26B69294E7F31--\n\n";
    const char *signed_data = "Content-Type: text/plain\r\n\r\nhello world";
    BIO *msg_bio = BIO_new_mem_buf(signed_data, strlen(signed_data));
    const char *cert_pem = "-----BEGIN CERTIFICATE-----\nMIIBUTCB96ADAgECAgIDCTAKBggqhkjOPQQDAjAnMQswCQYDVQQGEwJVUzEYMBYG\nA1UEAwwPY3J5cHRvZ3JhcGh5IENBMB4XDTE3MDEwMTEyMDEwMFoXDTM4MTIzMTA4\nMzAwMFowJzELMAkGA1UEBhMCVVMxGDAWBgNVBAMMD2NyeXB0b2dyYXBoeSBDQTBZ\nMBMGByqGSM49AgEGCCqGSM49AwEHA0IABBj/z7v5Obj13cPuwECLBnUGq0/N2CxS\nJE4f4BBGZ7VfFblivTvPDG++Gve0oQ+0uctuhrNQ+WxRv8GC177F+QWjEzARMA8G\nA1UdEwEB/wQFMAMBAf8wCgYIKoZIzj0EAwIDSQAwRgIhANES742XWm64tkGnz8Dn\npG6u2lHkZFQr3oaVvPcemvlbAiEA0WGGzmYx5C9UvfXIK7NEziT4pQtyESE0uRVK\nXw4nMqk=\n-----END CERTIFICATE-----";
    BIO *x509_bio = BIO_new_mem_buf(cert_pem, strlen(cert_pem));
    X509 *cert = PEM_read_bio_X509(x509_bio, NULL, NULL, NULL);
    assert(cert != NULL);
    assert(ERR_peek_error() == 0);

    X509_STORE *store = X509_STORE_new();
    assert(store != NULL);
    res = X509_STORE_add_cert(store, cert);
    assert(res == 1);

    BIO *bio = BIO_new_mem_buf(sig, strlen(sig));
    PKCS7 *p7 = SMIME_read_PKCS7(bio, NULL);
    assert(p7 != NULL);
    assert(ERR_peek_error() == 0);
    int flags = PKCS7_TEXT;
    res = PKCS7_verify(p7, NULL, store, msg_bio, NULL, flags);
    assert(res == 1);
    if (ERR_peek_error() != 0) {
        printf("error on stack but verify succeeded\n");
        return 1;
    }
}

Thanks for the test. It would of taken quite a while to find this without your very specific test case..

Back in mid January the following commit
https://github.com/openssl/openssl/commit/dc88a0390608ccba76df6a26358215d1c0592948#diff-310dbb5e976563a48efb10737adf53420287a20eed4d1d1b0750450f38bf6809
subtly changed the behavior of BIO_write to handle NULL differently..
i.e.
bio_write_intern(BIO * b, …) changed from

   If (b == NULL)
       return 0;

TO

    if (b == NULL) {
        ERR_raise(ERR_LIB_BIO, BIO_R_NULL_PARAMETER);
        return -1;
    }

There are quite a few functions that print and do not check the error code.
This potentially puts an error on the error stack and returns no error. This in not ideal.
So the options are:

Revert the change - but maybe this was done to fix some other issue?
Fix up all the places where BIO_write is not checked..
For diagnotistic or printing functions should they just do:
if (bio == NULL)
return 1;  /* ignore */

@openssl/otc and @ddvo - does anyone have strong opinions on this?

I'd just revert that part of the dc88a03

It could affect 3rd party applications as well.

It's a silent change to existing semantics that will most likely affect other users as @t8m points out.  Revert it.

Back in mid January the following commit commit dc88a0390608ccba76df6a26358215d1c0592948
subtly changed the behavior of BIO_write to handle NULL differently..
i.e.
bio_write_intern(BIO * b, …) changed from

   If (b == NULL)
       return 0;

TO

    if (b == NULL) {
        ERR_raise(ERR_LIB_BIO, BIO_R_NULL_PARAMETER);
        return -1;
    }

There are quite a few functions that print and do not check the error code.
This potentially puts an error on the error stack and returns no error. This in not ideal.
So the options are:
1. Revert the change - but maybe this was done to fix some other issue?

2. Fix up all the places where BIO_write is not checked..
   For diagnotistic or printing functions should they just do:
   if (bio == NULL)
   return 1;  /* ignore otherwise we break how it behaved before */

@openssl/otc and @DDvO - does anyone have strong opinions on this?

Thanks for bringing this up.
As far as I recall, the error reporting I added there was not to fix a specific issue but just to follow the general model of doing input validation.
Yet I was not aware that for BIO_write_{,ex}() and thus also for BIO_write_intern() it should be legitimate to call them with bio == NULL, also because this feature is not documented.
So I propose reverting the change quoted here and documenting the option of NULL BIO input to BIO_write_{,ex}().

#15493 says "Fixes #15389", but the merged version https://github.com/openssl/openssl/commit/f505161e62d558b3f8442b264ccbf8112ebd58ef omitted that line. I presume this issue can be closed.

Thanks @h-vetinari..