My node's just produced 2nd block witch is visible in logs but explorer misses it. Again! Guys it's unserious I get blocks (and it's random) my block may produced as orphan (and it's also random) and all those stuff is multiplied by bugs in your code when blocks are just missing. And you giving points based on random^3 strategy... imo it'd better to account points as a lottery at least it'd be obvious.
Or somehow account such cases in the leaderboard
2021-01-20 10:02:30 UTC [Info] Producing new block with parent $breadcrumb
breadcrumb: {
"validated_transition": {
"data": {
"protocol_state": {
"previous_state_hash":
"3NL7LkNB9Gqu6r6njos8eD17aweicJY7qA4FbRknfYvFcFxFJyoG",
"body": {
"genesis_state_hash":
"3NLxYrjb7zmHdoFgBrubCN8ijM8v7eT8kvLiPLc9DHt3M8XrDDEG",
"blockchain_state": {
"staged_ledger_hash": {
"non_snark": {
"ledger_hash":
"jwVtiZ1LBT5kXNjL37eqZaJ6Qx5gJp6bhsJRmZSZWzFn1aBjUMg",
"aux_hash":
"UY8Ukpfq2NfPAH6TBe528GgxuxuTGDkqLJH3ehag7UCDZUwSRJ",
"pending_coinbase_aux":
"XABTbgxR3xv2qxE1MCbdCgmFrC3wMeSVVy8VdMRfvDMmf7aR7k"
},
"pending_coinbase_hash":
"2n1MB1Ngt2PE3Wwxqpgb8AHABvUPpfw55iFBgs3GTP3kbeDsWJ2w"
},
"snarked_ledger_hash":
"jx67BnVRpqbBMhjaqptCySc2smusNrgorLyjjHgkWX3pmGS8GRF",
"genesis_ledger_hash":
"jxX6VJ84HaafrKozFRA4qjnni4aPXqXC2H5vQLKSryNpKTXuz1R",
"snarked_next_available_token": "2",
"timestamp": "1611136590000"
},
"consensus_state": {
"blockchain_length": "2688",
"epoch_count": "1",
"min_window_density": "1",
"sub_window_densities": [
"2", "1", "2", "2", "1", "3", "1", "2", "3", "1", "1"
],
"last_vrf_output": "60WOtZ2Vw9NneOfara1p5wS1f-9dP_CmNUWucVyZAgA=",
"total_currency": "166892200000001000",
"curr_global_slot": {
"slot_number": "9894",
"slots_per_epoch": "7140"
},
"global_slot_since_genesis": "9894",
"staking_epoch_data": {
"ledger": {
"hash": "jxX6VJ84HaafrKozFRA4qjnni4aPXqXC2H5vQLKSryNpKTXuz1R",
"total_currency": "165950000000001000"
},
"seed": "2vb1Mjvydod6sEwn7qpbejKCfRqugMgyG3MHXXRKcAkwQLRs9fj8",
"start_checkpoint":
"3NK2tkzqqK5spR2sZ7tujjqPksL45M3UUrcA4WhCkeiPtnugyE2x",
"lock_checkpoint":
"3NK5G8Xqn1Prh3XoTyZ2tqntJC6X2nVwruv5mEJCL3GaTk7jKUNo",
"epoch_length": "1769"
},
"next_epoch_data": {
"ledger": {
"hash": "jx7XXjRfJj2mGXmiHQmpm6ZgTxz14udpugyFtw4DefJFpie7apN",
"total_currency": "166537000000001000"
},
"seed": "2vatDqy849TtaViQWZSyi2xJ9M436o2QZuanJY578N15vzRQ9VQm",
"start_checkpoint":
"3NLdAqxtBRYxYbCWMXxGu6j1hGDrpQwGkBDF9QvGxmtpziXQDADu",
"lock_checkpoint":
"3NL7LkNB9Gqu6r6njos8eD17aweicJY7qA4FbRknfYvFcFxFJyoG",
"epoch_length": "920"
},
"has_ancestor_in_same_checkpoint_window": true,
"block_stake_winner":
"B62qoMtFUHwc14hUt3aUNeZwYNNAK7iKiWqkKP6cPp2xbFGxqPDabAQ",
"block_creator":
"B62qoMtFUHwc14hUt3aUNeZwYNNAK7iKiWqkKP6cPp2xbFGxqPDabAQ",
"coinbase_receiver":
"B62qoMtFUHwc14hUt3aUNeZwYNNAK7iKiWqkKP6cPp2xbFGxqPDabAQ",
"supercharge_coinbase": true
},
"constants": {
"k": "290",
"slots_per_epoch": "7140",
"slots_per_sub_window": "7",
"delta": "0",
"genesis_state_timestamp": "1609355670000"
}
}
},
"protocol_state_proof": "",
"staged_ledger_diff": "",
"delta_transition_chain_proof": "",
"current_protocol_version": "1.1.0",
"proposed_protocol_version": ""
},
"hash": "3NLT6UuxVFqbEVsEboP8Fkg13E6PNk2ogBjhs3yQAXntWFdLHC7x"
},
"staged_ledger": "",
"just_emitted_a_proof": true,
"transition_receipt_time": "2021-01-20T09:58:39.547445Z"
}
2021-01-20 10:02:30 UTC [Info] No locked tokens in the delegator/delegatee account, applying supercharged coinbase
2021-01-20 10:03:44 UTC [Info] Protocol_state_proof proving time(ms)
2021-01-20 10:03:59 UTC [Info] Build breadcrumb on produced block
2021-01-20 10:04:00 UTC [Info] Successfully produced a new block
2021-01-20 10:04:03 UTC [Info] Generated transition $state_hash was accepted into transition frontier
state_hash: "3NKKfLZinDCe5ECRhqKvQM8mUvyuRYxtTJo7YVUv1tLhjqtqdKQ1"
2021-01-20 10:04:03 UTC [Info] Updating new available work took 10.71929931640625 ms
2021-01-20 10:04:16 UTC [Info] Determining next slot to produce block
2021-01-20 10:04:16 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 2757

For all the time, I produced 6 blocks and yesterday for the first time I encountered the fact that the block that I produced was not displayed in the explorer. I looked through the logs, everything was successfully produced yesterday. Unfortunately, now, after a day, I cannot find information in the logs to show you.
The server is very powerful. EX41 from Hetzner.
Dedicated Root Server AX41-NVMe
--
AMD Ryzen 5 3600 Hexa-Core "Matisse" (Zen2)
64 GB DDR4 RAM
2 x 512 GB NVMe SSD (Software-RAID 1)

https://minaexplorer.com/wallet/B62qkWFkU9PDSzAxWWXVcxxHe1nJnfGqLeYbtxDLv5BxPiekGcxLTpj
I have the next block in 18 hours. I will definitely save the logs.

For all the time, I produced 6 blocks and yesterday for the first time I encountered the fact that the block that I produced was not displayed in the explorer. I looked through the logs, everything was successfully produced yesterday. Unfortunately, now, after a day, I cannot find information in the logs to show you.
The server is very powerful. EX41 from Hetzner.
Dedicated Root Server AX41-NVMe
--
AMD Ryzen 5 3600 Hexa-Core "Matisse" (Zen2)
64 GB DDR4 RAM
2 x 512 GB NVMe SSD (Software-RAID 1)

https://minaexplorer.com/wallet/B62qkWFkU9PDSzAxWWXVcxxHe1nJnfGqLeYbtxDLv5BxPiekGcxLTpj
I have the next block in 18 hours. I will definitely save the logs.

So I was overtaken by the fate of being in the circle of these nice guys, from whom luck turned away.
Perplexed, I wondered what could have led to an unfavorable outcome? Maybe it's not a powerful enough processor? But no, he alone can produce blocks for the entire network.
Maybe these are new rules that allow you to launch as many block producers as you like, in order to create an infinite number of forks, in which honestly produced blocks of other participants are lost?
But apparently the reason is something else. Perhaps this is the curse, the curse of the random!
This is an unpleasant situation, because due to the lost block, I dropped out of the competition. The next block is in 2 days, which is doubly upsetting.
Anyway, I am attaching a log in which you can find an event about the successful creation of block 2613.
logs.zip
I also want to draw your attention to a strange event discovered a few days ago.
After the transition to a new era, the time until the next block was 4 days. Two days later, after updating the node, the time changed from 2 days to 2 minutes. It would seem that one can rejoice. But no. After this time, the block did not appear in the explorer. I checked the status of the node, the time before the block was 2 days, either recovered, or it coincided.
But this is not the end of the story, because after the appointed two days, block 2419 was produced instead of 2446. Logs are in the archive.
https://minaexplorer.com/wallet/B62qjPiM2hPCv4qogPYxDuz3U1UidPp7RRijjb2e2NcnTxq8GHyWb1a

So I was overtaken by the fate of being in the circle of these nice guys, from whom luck turned away.
Perplexed, I wondered what could have led to an unfavorable outcome? Maybe it's not a powerful enough processor? But no, he alone can produce blocks for the entire network.
Maybe these are new rules that allow you to launch as many block producers as you like, in order to create an infinite number of forks, in which honestly produced blocks of other participants are lost?
But apparently the reason is something else. Perhaps this is the curse, the curse of the random!
This is an unpleasant situation, because due to the lost block, I dropped out of the competition. The next block is in 2 days, which is doubly upsetting.
Anyway, I am attaching a log in which you can find an event about the successful creation of block 2613.
logs.zip
I also want to draw your attention to a strange event discovered a few days ago.
After the transition to a new era, the time until the next block was 4 days. Two days later, after updating the node, the time changed from 2 days to 2 minutes. It would seem that one can rejoice. But no. After this time, the block did not appear in the explorer. I checked the status of the node, the time before the block was 2 days, either recovered, or it coincided.
But this is not the end of the story, because after the appointed two days, block 2419 was produced instead of 2446. Logs are in the archive.
https://minaexplorer.com/wallet/B62qjPiM2hPCv4qogPYxDuz3U1UidPp7RRijjb2e2NcnTxq8GHyWb1a

Looked at another one of these block (3NKbr67kzfWZsM1RhNiVQTn6qzq5o2viNAjNHkoxoZzHc5DvDcZV) was broadcast successfully (broadcasting new state over gossip net) and built on 3NKcdQ7ixbJkCnTTWRsMjkWpALLJxFkTuCCqvwNt44T9skPzUXqD.
Looking in the explorer logs there is no mention of the block hash but multiple others with the same parent state hash.
It seems there are forks that are resolved but the explorer node doesn't necessarily catch up with the missing blocks if it did not receive them and not on the canonical chain?

Looked at another one of these block (3NKbr67kzfWZsM1RhNiVQTn6qzq5o2viNAjNHkoxoZzHc5DvDcZV) was broadcast successfully (broadcasting new state over gossip net) and built on 3NKcdQ7ixbJkCnTTWRsMjkWpALLJxFkTuCCqvwNt44T9skPzUXqD.
Looking in the explorer logs there is no mention of the block hash but multiple others with the same parent state hash.
It seems there are forks that are resolved but the explorer node doesn't necessarily catch up with the missing blocks if it did not receive them and not on the canonical chain?

@rozum-dev the hash you list above 3NLT6UuxVFqbEVsEboP8Fkg13E6PNk2ogBjhs3yQAXntWFdLHC7x  is in the canonical chain and on the explorer. You didn't create it with that public key though, it was created with B62qoMtFUHwc14hUt3aUNeZwYNNAK7iKiWqkKP6cPp2xbFGxqPDabAQ as it clearly shows in the logs. Are you staking with multiple keys?

@rozum-dev the hash you list above 3NLT6UuxVFqbEVsEboP8Fkg13E6PNk2ogBjhs3yQAXntWFdLHC7x  is in the canonical chain and on the explorer. You didn't create it with that public key though, it was created with B62qoMtFUHwc14hUt3aUNeZwYNNAK7iKiWqkKP6cPp2xbFGxqPDabAQ as it clearly shows in the logs. Are you staking with multiple keys?

@rozum-dev the hash you list above 3NLT6UuxVFqbEVsEboP8Fkg13E6PNk2ogBjhs3yQAXntWFdLHC7x  is in the canonical chain and on the explorer here https://minaexplorer.com/block/3NLT6UuxVFqbEVsEboP8Fkg13E6PNk2ogBjhs3yQAXntWFdLHC7x. You didn't create it with that public key though, it was created with B62qoMtFUHwc14hUt3aUNeZwYNNAK7iKiWqkKP6cPp2xbFGxqPDabAQ as it clearly shows in the logs. Are you staking with multiple keys?

No, I don't think so. I have only one key and one block producer. So how's that happen?

@rozum-dev the hash you list above 3NLT6UuxVFqbEVsEboP8Fkg13E6PNk2ogBjhs3yQAXntWFdLHC7x  is in the canonical chain and on the explorer here https://minaexplorer.com/block/3NLT6UuxVFqbEVsEboP8Fkg13E6PNk2ogBjhs3yQAXntWFdLHC7x. You didn't create it with that public key though, it was created with B62qoMtFUHwc14hUt3aUNeZwYNNAK7iKiWqkKP6cPp2xbFGxqPDabAQ as it clearly shows in the logs. Are you staking with multiple keys?

No, I don't think so. I have only one key and one block producer. So how's that happen?

No, I don't think so. I have only one key and one block producer. So how's that happen?

It's my mistake, I misread your logs, sorry. That's the parent hash

No, I don't think so. I have only one key and one block producer. So how's that happen?

It's my mistake, I misread your logs, sorry. That's the parent hash

@taoros this is the error on the explorer for your hash:
{
  "timestamp": "2021-01-20 05:29:31.150686Z",
  "level": "Error",
  "source": {
    "module": "Transition_router__Initial_validator",
    "location": "File \"src/lib/transition_router/initial_validator.ml\", line 58, characters 2-14"
  },
  "message": "Validation error: external transition with state hash $state_hash was rejected for reason $reason",
  "metadata": {
    "reason": "invalid time",
    "slot_diff": "1",
    "state_hash": "3NKamHpJ9bdBFvZcfXgVM3y77yqsuvQvyvJXxg63fLf4T77ur9Tk",
    "time_error": "too late"
  }
}

@taoros this is the error on the explorer for your hash:
{
  "timestamp": "2021-01-20 05:29:31.150686Z",
  "level": "Error",
  "source": {
    "module": "Transition_router__Initial_validator",
    "location": "File \"src/lib/transition_router/initial_validator.ml\", line 58, characters 2-14"
  },
  "message": "Validation error: external transition with state hash $state_hash was rejected for reason $reason",
  "metadata": {
    "reason": "invalid time",
    "slot_diff": "1",
    "state_hash": "3NKamHpJ9bdBFvZcfXgVM3y77yqsuvQvyvJXxg63fLf4T77ur9Tk",
    "time_error": "too late"
  }
}

same problem again :
{
"timestamp": "2021-01-20 15:38:41.153692Z",
"level": "Info",
"source": {
"module": "Block_producer",
"location": "File "src/lib/block_producer/block_producer.ml", line 570, characters 20-35"
},
"message": "Successfully produced a new block",
"metadata": {
"breadcrumb": {
"validated_transition": {
"data": {
"protocol_state": {
"previous_state_hash": "3NK7SDKMFtVer1JhU3WL8y5qQ88bMSvDQakLyu9gokdqWMHPuyV3",
"body": {
"genesis_state_hash": "3NLxYrjb7zmHdoFgBrubCN8ijM8v7eT8kvLiPLc9DHt3M8XrDDEG",
"blockchain_state": {
"staged_ledger_hash": {
"non_snark": {
"ledger_hash": "jxMGm5iNjKYNuMR6FBPG8ttCroX7Sej4a2p1FbekHcxLtjKQQ2Q",
"aux_hash": "UXRmyd2wYLGDcipQWkRTW6x3PfCvT6gAGkpRQqRqZCANbA4wtc",
"pending_coinbase_aux": "XUEeguj2J8Bj3knNEQz2GSFPTsdphoXMVdxeAQU5ZMUvafvTFC"
},
"pending_coinbase_hash": "2n1kgZMPBEbwG4NB4cjhfbWafVgPkRfu8TQrVg7Jq6Wep6rtYXA3"
},
"snarked_ledger_hash": "jwyae99tBovrVdfHxpVHwM2R44W8zF1C4Ea6HzM1ydcYVFTRchf",
"genesis_ledger_hash": "jxX6VJ84HaafrKozFRA4qjnni4aPXqXC2H5vQLKSryNpKTXuz1R",
"snarked_next_available_token": "2",
"timestamp": "1611156930000"
},
"consensus_state": {
"blockchain_length": "2713",
"epoch_count": "1",
"min_window_density": "1",
"sub_window_densities": [
"2",
"2",
"3",
"0",
"2",
"2",
"1",
"0",
"1",
"1",
"1"
],
"last_vrf_output": "jVNdtsghN2yQDDNGU_D8rmfD1uPFC-ixwWSIc5wTBAA=",
"total_currency": "166902600000001000",
"curr_global_slot": {
"slot_number": "10007",
"slots_per_epoch": "7140"
},
"global_slot_since_genesis": "10007",
"staking_epoch_data": {
"ledger": {
"hash": "jxX6VJ84HaafrKozFRA4qjnni4aPXqXC2H5vQLKSryNpKTXuz1R",
"total_currency": "165950000000001000"
},
"seed": "2vb1Mjvydod6sEwn7qpbejKCfRqugMgyG3MHXXRKcAkwQLRs9fj8",
"start_checkpoint": "3NK2tkzqqK5spR2sZ7tujjqPksL45M3UUrcA4WhCkeiPtnugyE2x",
"lock_checkpoint": "3NK5G8Xqn1Prh3XoTyZ2tqntJC6X2nVwruv5mEJCL3GaTk7jKUNo",
"epoch_length": "1769"
},
"next_epoch_data": {
"ledger": {
"hash": "jx7XXjRfJj2mGXmiHQmpm6ZgTxz14udpugyFtw4DefJFpie7apN",
"total_currency": "166537000000001000"
},
"seed": "2vaQinqP8sDNgY4saCLoVobRK7yyznh1mNq7Nk1pByo17Qwnf7zR",
"start_checkpoint": "3NLdAqxtBRYxYbCWMXxGu6j1hGDrpQwGkBDF9QvGxmtpziXQDADu",
"lock_checkpoint": "3NK7SDKMFtVer1JhU3WL8y5qQ88bMSvDQakLyu9gokdqWMHPuyV3",
"epoch_length": "945"
},
"has_ancestor_in_same_checkpoint_window": true,
"block_stake_winner": "B62qpNmgVUZ9vCzVwHX7qw329yMnmMNnYKG8KmhRWdbbPzYEN4qEQoo",
"block_creator": "B62qpNmgVUZ9vCzVwHX7qw329yMnmMNnYKG8KmhRWdbbPzYEN4qEQoo",
"coinbase_receiver": "B62qpNmgVUZ9vCzVwHX7qw329yMnmMNnYKG8KmhRWdbbPzYEN4qEQoo",
"supercharge_coinbase": true
},
"constants": {
"k": "290",
"slots_per_epoch": "7140",
"slots_per_sub_window": "7",
"delta": "0",
"genesis_state_timestamp": "1609355670000"
}
}
},
"protocol_state_proof": "",
"staged_ledger_diff": "",
"delta_transition_chain_proof": "",
"current_protocol_version": "1.1.0",
"proposed_protocol_version": ""
},
"hash": "3NKFzrh7sa1oZgeNcx8UbVc5Xrm1WDMSFPNHwtZfQA1zxfDgR7cz"
},
"staged_ledger": "",
"just_emitted_a_proof": true,
"transition_receipt_time": "2021-01-20T15:38:13.797755Z"
},
"host": "<IP_ADDRESS>",
"peer_id": "12D3KooWRQjjWVu78nW5kVwaamkmjfXj8ofR7YXnCwaAWCqMFgoC",
"pid": 2918,
"port": 8302
},
"event_id": "64e2d3e86c37c09b15efdaf7470ce879"
}

same problem again :
{
"timestamp": "2021-01-20 15:38:41.153692Z",
"level": "Info",
"source": {
"module": "Block_producer",
"location": "File "src/lib/block_producer/block_producer.ml", line 570, characters 20-35"
},
"message": "Successfully produced a new block",
"metadata": {
"breadcrumb": {
"validated_transition": {
"data": {
"protocol_state": {
"previous_state_hash": "3NK7SDKMFtVer1JhU3WL8y5qQ88bMSvDQakLyu9gokdqWMHPuyV3",
"body": {
"genesis_state_hash": "3NLxYrjb7zmHdoFgBrubCN8ijM8v7eT8kvLiPLc9DHt3M8XrDDEG",
"blockchain_state": {
"staged_ledger_hash": {
"non_snark": {
"ledger_hash": "jxMGm5iNjKYNuMR6FBPG8ttCroX7Sej4a2p1FbekHcxLtjKQQ2Q",
"aux_hash": "UXRmyd2wYLGDcipQWkRTW6x3PfCvT6gAGkpRQqRqZCANbA4wtc",
"pending_coinbase_aux": "XUEeguj2J8Bj3knNEQz2GSFPTsdphoXMVdxeAQU5ZMUvafvTFC"
},
"pending_coinbase_hash": "2n1kgZMPBEbwG4NB4cjhfbWafVgPkRfu8TQrVg7Jq6Wep6rtYXA3"
},
"snarked_ledger_hash": "jwyae99tBovrVdfHxpVHwM2R44W8zF1C4Ea6HzM1ydcYVFTRchf",
"genesis_ledger_hash": "jxX6VJ84HaafrKozFRA4qjnni4aPXqXC2H5vQLKSryNpKTXuz1R",
"snarked_next_available_token": "2",
"timestamp": "1611156930000"
},
"consensus_state": {
"blockchain_length": "2713",
"epoch_count": "1",
"min_window_density": "1",
"sub_window_densities": [
"2",
"2",
"3",
"0",
"2",
"2",
"1",
"0",
"1",
"1",
"1"
],
"last_vrf_output": "jVNdtsghN2yQDDNGU_D8rmfD1uPFC-ixwWSIc5wTBAA=",
"total_currency": "166902600000001000",
"curr_global_slot": {
"slot_number": "10007",
"slots_per_epoch": "7140"
},
"global_slot_since_genesis": "10007",
"staking_epoch_data": {
"ledger": {
"hash": "jxX6VJ84HaafrKozFRA4qjnni4aPXqXC2H5vQLKSryNpKTXuz1R",
"total_currency": "165950000000001000"
},
"seed": "2vb1Mjvydod6sEwn7qpbejKCfRqugMgyG3MHXXRKcAkwQLRs9fj8",
"start_checkpoint": "3NK2tkzqqK5spR2sZ7tujjqPksL45M3UUrcA4WhCkeiPtnugyE2x",
"lock_checkpoint": "3NK5G8Xqn1Prh3XoTyZ2tqntJC6X2nVwruv5mEJCL3GaTk7jKUNo",
"epoch_length": "1769"
},
"next_epoch_data": {
"ledger": {
"hash": "jx7XXjRfJj2mGXmiHQmpm6ZgTxz14udpugyFtw4DefJFpie7apN",
"total_currency": "166537000000001000"
},
"seed": "2vaQinqP8sDNgY4saCLoVobRK7yyznh1mNq7Nk1pByo17Qwnf7zR",
"start_checkpoint": "3NLdAqxtBRYxYbCWMXxGu6j1hGDrpQwGkBDF9QvGxmtpziXQDADu",
"lock_checkpoint": "3NK7SDKMFtVer1JhU3WL8y5qQ88bMSvDQakLyu9gokdqWMHPuyV3",
"epoch_length": "945"
},
"has_ancestor_in_same_checkpoint_window": true,
"block_stake_winner": "B62qpNmgVUZ9vCzVwHX7qw329yMnmMNnYKG8KmhRWdbbPzYEN4qEQoo",
"block_creator": "B62qpNmgVUZ9vCzVwHX7qw329yMnmMNnYKG8KmhRWdbbPzYEN4qEQoo",
"coinbase_receiver": "B62qpNmgVUZ9vCzVwHX7qw329yMnmMNnYKG8KmhRWdbbPzYEN4qEQoo",
"supercharge_coinbase": true
},
"constants": {
"k": "290",
"slots_per_epoch": "7140",
"slots_per_sub_window": "7",
"delta": "0",
"genesis_state_timestamp": "1609355670000"
}
}
},
"protocol_state_proof": "",
"staged_ledger_diff": "",
"delta_transition_chain_proof": "",
"current_protocol_version": "1.1.0",
"proposed_protocol_version": ""
},
"hash": "3NKFzrh7sa1oZgeNcx8UbVc5Xrm1WDMSFPNHwtZfQA1zxfDgR7cz"
},
"staged_ledger": "",
"just_emitted_a_proof": true,
"transition_receipt_time": "2021-01-20T15:38:13.797755Z"
},
"host": "<IP_ADDRESS>",
"peer_id": "12D3KooWRQjjWVu78nW5kVwaamkmjfXj8ofR7YXnCwaAWCqMFgoC",
"pid": 2918,
"port": 8302
},
"event_id": "64e2d3e86c37c09b15efdaf7470ce879"
}

Produced block missing in explorer
2021-01-21 02:31:52 UTC [Warn] RPC call error for "get_transition_chain_proof"
2021-01-21 02:35:30 UTC [Info] Producing new block with parent $breadcrumb
breadcrumb: {
"validated_transition": {
"data": {
"protocol_state": {
"previous_state_hash":
"3NLyAPo6YCqijZpDgtsz1iHUr1D81mjef3PYWiDBWhfwpgESmM9f",
"body": {
"genesis_state_hash":
"3NLxYrjb7zmHdoFgBrubCN8ijM8v7eT8kvLiPLc9DHt3M8XrDDEG",
"blockchain_state": {
"staged_ledger_hash": {
"non_snark": {
"ledger_hash":
"jwanpiTv7Ta1KaMGyffzv3xiXaSciQqN7JaA482EydikeAF3Krd",
"aux_hash":
"UvwdVoTWePLqNAkWW5SdZuWcWtFdejVVfK542vNju1i9HmCBfi",
"pending_coinbase_aux":
"WLo8mDN6oBUTSyBkFCy7Fky7Na5fN4R6oGq4HMf3YoHCAj4cwY"
},
"pending_coinbase_hash":
"2n1xkADc6AVG2YYGKqug9sMLoLkbirMEFKCwHJWoect5PjXdKT1w"
},
"snarked_ledger_hash":
"jwhGEA6U163bGFrqnxLvx2gumTtv2GVMYEL2PFHHVeGF9uMpfUB",
"genesis_ledger_hash":
"jxX6VJ84HaafrKozFRA4qjnni4aPXqXC2H5vQLKSryNpKTXuz1R",
"snarked_next_available_token": "2",
"timestamp": "1611177810000"
},
"consensus_state": {
"blockchain_length": "2747",
"epoch_count": "1",
"min_window_density": "1",
"sub_window_densities": [
"3", "2", "3", "3", "1", "1", "1", "2", "2", "1", "1"
],
"last_vrf_output": "blRkc2X9eQBPL0grw_hGfCqDkTiW-oFzS3hSdkGLEAA=",
"total_currency": "166914600000001000",
"curr_global_slot": {
"slot_number": "10123",
"slots_per_epoch": "7140"
},
"global_slot_since_genesis": "10123",
"staking_epoch_data": {
"ledger": {
"hash": "jxX6VJ84HaafrKozFRA4qjnni4aPXqXC2H5vQLKSryNpKTXuz1R",
"total_currency": "165950000000001000"
},
"seed": "2vb1Mjvydod6sEwn7qpbejKCfRqugMgyG3MHXXRKcAkwQLRs9fj8",
"start_checkpoint":
"3NK2tkzqqK5spR2sZ7tujjqPksL45M3UUrcA4WhCkeiPtnugyE2x",
"lock_checkpoint":
"3NK5G8Xqn1Prh3XoTyZ2tqntJC6X2nVwruv5mEJCL3GaTk7jKUNo",
"epoch_length": "1769"
},
"next_epoch_data": {
"ledger": {
"hash": "jx7XXjRfJj2mGXmiHQmpm6ZgTxz14udpugyFtw4DefJFpie7apN",
"total_currency": "166537000000001000"
},
"seed": "2vbmZfpiFd34aGzNCkBaRwbz6Tnibe5JfdwCxjgJYapri2dhsrNf",
"start_checkpoint":
"3NLdAqxtBRYxYbCWMXxGu6j1hGDrpQwGkBDF9QvGxmtpziXQDADu",
"lock_checkpoint":
"3NLyAPo6YCqijZpDgtsz1iHUr1D81mjef3PYWiDBWhfwpgESmM9f",
"epoch_length": "979"
},
"has_ancestor_in_same_checkpoint_window": true,
"block_stake_winner":
"B62qpB1oBckU1wXkRFxcwwCheq3chiKtVhY6T3q8aQiwJUaoCjpqgXS",
"block_creator":
"B62qnyS9n4evSWTfHBpEYkpd7rNW2B2Tq5tEunQ6AB3vfdwjZtzs7Hh",
"coinbase_receiver":
"B62qnyS9n4evSWTfHBpEYkpd7rNW2B2Tq5tEunQ6AB3vfdwjZtzs7Hh",
"supercharge_coinbase": true
},
"constants": {
"k": "290",
"slots_per_epoch": "7140",
"slots_per_sub_window": "7",
"delta": "0",
"genesis_state_timestamp": "1609355670000"
}
}
},
"protocol_state_proof": "",
"staged_ledger_diff": "",
"delta_transition_chain_proof": "",
"current_protocol_version": "1.1.0",
"proposed_protocol_version": ""
},
"hash": "3NLsuS28AVWwMRwhfPaa2WA6Mkqse9TiH2aMuc18Aj5y65sawzP5"
},
"staged_ledger": "",
"just_emitted_a_proof": true,
"transition_receipt_time": "2021-01-20T22:00:40.131764Z"
}
2021-01-21 02:35:30 UTC [Info] No locked tokens in the delegator/delegatee account, applying supercharged coinbase
2021-01-21 02:36:30 UTC [Warn] validation callback timed out before we could respond
2021-01-21 02:36:26 UTC [Error] validation timed out :(
2021-01-21 02:36:31 UTC [Info] Protocol_state_proof proving time(ms)
2021-01-21 02:36:34 UTC [Info] Build breadcrumb on produced block
2021-01-21 02:36:34 UTC [Info] Successfully produced a new block
2021-01-21 02:36:36 UTC [Info] Generated transition $state_hash was accepted into transition frontier
state_hash: "3NK5orWoG2sugCLZ5YZGbyL8Z7VhVTZk94EpXm1KxA2LdLGT7GXv"
2021-01-21 02:36:36 UTC [Info] Updating new available work took 9.598731994628906 ms
2021-01-21 02:36:36 UTC [Info] Determining next slot to produce block

Produced block missing in explorer
2021-01-21 02:31:52 UTC [Warn] RPC call error for "get_transition_chain_proof"
2021-01-21 02:35:30 UTC [Info] Producing new block with parent $breadcrumb
breadcrumb: {
"validated_transition": {
"data": {
"protocol_state": {
"previous_state_hash":
"3NLyAPo6YCqijZpDgtsz1iHUr1D81mjef3PYWiDBWhfwpgESmM9f",
"body": {
"genesis_state_hash":
"3NLxYrjb7zmHdoFgBrubCN8ijM8v7eT8kvLiPLc9DHt3M8XrDDEG",
"blockchain_state": {
"staged_ledger_hash": {
"non_snark": {
"ledger_hash":
"jwanpiTv7Ta1KaMGyffzv3xiXaSciQqN7JaA482EydikeAF3Krd",
"aux_hash":
"UvwdVoTWePLqNAkWW5SdZuWcWtFdejVVfK542vNju1i9HmCBfi",
"pending_coinbase_aux":
"WLo8mDN6oBUTSyBkFCy7Fky7Na5fN4R6oGq4HMf3YoHCAj4cwY"
},
"pending_coinbase_hash":
"2n1xkADc6AVG2YYGKqug9sMLoLkbirMEFKCwHJWoect5PjXdKT1w"
},
"snarked_ledger_hash":
"jwhGEA6U163bGFrqnxLvx2gumTtv2GVMYEL2PFHHVeGF9uMpfUB",
"genesis_ledger_hash":
"jxX6VJ84HaafrKozFRA4qjnni4aPXqXC2H5vQLKSryNpKTXuz1R",
"snarked_next_available_token": "2",
"timestamp": "1611177810000"
},
"consensus_state": {
"blockchain_length": "2747",
"epoch_count": "1",
"min_window_density": "1",
"sub_window_densities": [
"3", "2", "3", "3", "1", "1", "1", "2", "2", "1", "1"
],
"last_vrf_output": "blRkc2X9eQBPL0grw_hGfCqDkTiW-oFzS3hSdkGLEAA=",
"total_currency": "166914600000001000",
"curr_global_slot": {
"slot_number": "10123",
"slots_per_epoch": "7140"
},
"global_slot_since_genesis": "10123",
"staking_epoch_data": {
"ledger": {
"hash": "jxX6VJ84HaafrKozFRA4qjnni4aPXqXC2H5vQLKSryNpKTXuz1R",
"total_currency": "165950000000001000"
},
"seed": "2vb1Mjvydod6sEwn7qpbejKCfRqugMgyG3MHXXRKcAkwQLRs9fj8",
"start_checkpoint":
"3NK2tkzqqK5spR2sZ7tujjqPksL45M3UUrcA4WhCkeiPtnugyE2x",
"lock_checkpoint":
"3NK5G8Xqn1Prh3XoTyZ2tqntJC6X2nVwruv5mEJCL3GaTk7jKUNo",
"epoch_length": "1769"
},
"next_epoch_data": {
"ledger": {
"hash": "jx7XXjRfJj2mGXmiHQmpm6ZgTxz14udpugyFtw4DefJFpie7apN",
"total_currency": "166537000000001000"
},
"seed": "2vbmZfpiFd34aGzNCkBaRwbz6Tnibe5JfdwCxjgJYapri2dhsrNf",
"start_checkpoint":
"3NLdAqxtBRYxYbCWMXxGu6j1hGDrpQwGkBDF9QvGxmtpziXQDADu",
"lock_checkpoint":
"3NLyAPo6YCqijZpDgtsz1iHUr1D81mjef3PYWiDBWhfwpgESmM9f",
"epoch_length": "979"
},
"has_ancestor_in_same_checkpoint_window": true,
"block_stake_winner":
"B62qpB1oBckU1wXkRFxcwwCheq3chiKtVhY6T3q8aQiwJUaoCjpqgXS",
"block_creator":
"B62qnyS9n4evSWTfHBpEYkpd7rNW2B2Tq5tEunQ6AB3vfdwjZtzs7Hh",
"coinbase_receiver":
"B62qnyS9n4evSWTfHBpEYkpd7rNW2B2Tq5tEunQ6AB3vfdwjZtzs7Hh",
"supercharge_coinbase": true
},
"constants": {
"k": "290",
"slots_per_epoch": "7140",
"slots_per_sub_window": "7",
"delta": "0",
"genesis_state_timestamp": "1609355670000"
}
}
},
"protocol_state_proof": "",
"staged_ledger_diff": "",
"delta_transition_chain_proof": "",
"current_protocol_version": "1.1.0",
"proposed_protocol_version": ""
},
"hash": "3NLsuS28AVWwMRwhfPaa2WA6Mkqse9TiH2aMuc18Aj5y65sawzP5"
},
"staged_ledger": "",
"just_emitted_a_proof": true,
"transition_receipt_time": "2021-01-20T22:00:40.131764Z"
}
2021-01-21 02:35:30 UTC [Info] No locked tokens in the delegator/delegatee account, applying supercharged coinbase
2021-01-21 02:36:30 UTC [Warn] validation callback timed out before we could respond
2021-01-21 02:36:26 UTC [Error] validation timed out :(
2021-01-21 02:36:31 UTC [Info] Protocol_state_proof proving time(ms)
2021-01-21 02:36:34 UTC [Info] Build breadcrumb on produced block
2021-01-21 02:36:34 UTC [Info] Successfully produced a new block
2021-01-21 02:36:36 UTC [Info] Generated transition $state_hash was accepted into transition frontier
state_hash: "3NK5orWoG2sugCLZ5YZGbyL8Z7VhVTZk94EpXm1KxA2LdLGT7GXv"
2021-01-21 02:36:36 UTC [Info] Updating new available work took 9.598731994628906 ms
2021-01-21 02:36:36 UTC [Info] Determining next slot to produce block

@yes-filippova I can see your block with state_hash 3NK5orWoG2sugCLZ5YZGbyL8Z7VhVTZk94EpXm1KxA2LdLGT7GXv in our logs. It was added but is not part of the canonical chain. There was a re-org and a few blocks including yours was removed from the canonical chain. Not sure why it is not in the explorer

@yes-filippova I can see your block with state_hash 3NK5orWoG2sugCLZ5YZGbyL8Z7VhVTZk94EpXm1KxA2LdLGT7GXv in our logs. It was added but is not part of the canonical chain. There was a re-org and a few blocks including yours was removed from the canonical chain. Not sure why it is not in the explorer

@bidulemachin could you check in the log file if there is a log that says Internally generated block $state_hash cannot be rebroadcast because it's not a valid time to do so ($timing). I don't see your block in any of our logs

@bidulemachin could you check in the log file if there is a log that says Internally generated block $state_hash cannot be rebroadcast because it's not a valid time to do so ($timing). I don't see your block in any of our logs

@deepthiskumar  add the block to my points then https://minaexplorer.com/wallet/B62qoXw9yG2phLtp5xJq1nVfwgyp8gaUpKJoEwpp214n4uyWEd9qWos

@deepthiskumar  add the block to my points then https://minaexplorer.com/wallet/B62qoXw9yG2phLtp5xJq1nVfwgyp8gaUpKJoEwpp214n4uyWEd9qWos

Hey @deepthiskumar, will you evaluate block productions according to your own nodes? Because I have 2 blocks that I also produced but are not visible. I hope that I do not need to apply for this. You will make the final evaluation based on your own nodes and your own correct data?

Hey @deepthiskumar, will you evaluate block productions according to your own nodes? Because I have 2 blocks that I also produced but are not visible. I hope that I do not need to apply for this. You will make the final evaluation based on your own nodes and your own correct data?

@NarayanaSupramati in the logs I see two blocks produced. the second one with hash 3NKZNvVEWXdCZxyB3ni5psa7p22jzzGfcgZH3B8hvX8JoCwGZnqS is not in the any of our nodes' logs. For this block the proving time was 1m36s which is a min longer than the proving time for the previous block produced. The block broadcast was logged at 21:19:53 (hh:mm:ss) and the slot 9641 ends at 21:20:30. Possible that the block was not gossiped well. Validation of a block is also quite expensive and can take ~10secs. I also see during block production the node caters to transition chain requests and other block validation that could cost some time (https://github.com/MinaProtocol/mina/issues/7446), but I believe there was high resource contention during this block (given previous block proof was generated in 30 secs). I don't see any gossip message from your IP in our logs or the seed nodes during that time, so could be possible that your node did not have many connected peers

@NarayanaSupramati in the logs I see two blocks produced. the second one with hash 3NKZNvVEWXdCZxyB3ni5psa7p22jzzGfcgZH3B8hvX8JoCwGZnqS is not in the any of our nodes' logs. For this block the proving time was 1m36s which is a min longer than the proving time for the previous block produced. The block broadcast was logged at 21:19:53 (hh:mm:ss) and the slot 9641 ends at 21:20:30. Possible that the block was not gossiped well. Validation of a block is also quite expensive and can take ~10secs. I also see during block production the node caters to transition chain requests and other block validation that could cost some time (https://github.com/MinaProtocol/mina/issues/7446), but I believe there was high resource contention during this block (given previous block proof was generated in 30 secs). I don't see any gossip message from your IP in our logs or the seed nodes during that time, so could be possible that your node did not have many connected peers

@rozum-dev could you check in the log file if there is a log that says Internally generated block $state_hash cannot be rebroadcast because it's not a valid time to do so ($timing)
state_hash is 3NLT6UuxVFqbEVsEboP8Fkg13E6PNk2ogBjhs3yQAXntWFdLHC7x

@rozum-dev could you check in the log file if there is a log that says Internally generated block $state_hash cannot be rebroadcast because it's not a valid time to do so ($timing)
state_hash is 3NLT6UuxVFqbEVsEboP8Fkg13E6PNk2ogBjhs3yQAXntWFdLHC7x

@rozum-dev nevermind I see 3NLT6UuxVFqbEVsEboP8Fkg13E6PNk2ogBjhs3yQAXntWFdLHC7x in the main chain. please check the explorer again

@rozum-dev nevermind I see 3NLT6UuxVFqbEVsEboP8Fkg13E6PNk2ogBjhs3yQAXntWFdLHC7x in the main chain. please check the explorer again

@deepthiskumar check explorer for what? Do you mean I should see a block produced there? There's no new blocks appeared, only the last from a week ago

@deepthiskumar check explorer for what? Do you mean I should see a block produced there? There's no new blocks appeared, only the last from a week ago

Or somehow account such cases in the leaderboard for my producer B62qqbBXkgopUgBgvGtfzCgGRXQYcLrxKezWgmqGndbvPnuUhnCeWYV
"hash": "3NLT6UuxVFqbEVsEboP8Fkg13E6PNk2ogBjhs3yQAXntWFdLHC7x"

@rozum-dev Mistook this as your block. This is the parent block. Could you check your logs for Internally generated block $state_hash cannot be rebroadcast because it's not a valid time to do so ($timing), "Broadcasting new state over gossip net for the state hash of your block? Or please attach the log file that has your block information

Or somehow account such cases in the leaderboard for my producer B62qqbBXkgopUgBgvGtfzCgGRXQYcLrxKezWgmqGndbvPnuUhnCeWYV
"hash": "3NLT6UuxVFqbEVsEboP8Fkg13E6PNk2ogBjhs3yQAXntWFdLHC7x"

@rozum-dev Mistook this as your block. This is the parent block. Could you check your logs for Internally generated block $state_hash cannot be rebroadcast because it's not a valid time to do so ($timing), "Broadcasting new state over gossip net for the state hash of your block? Or please attach the log file that has your block information

@deepthiskumar this is an output for $ sudo docker logs --follow mina -f --tail 10000 for the time block's been produced.
Tell me pls if it's enough
mina 2.log.zip

@deepthiskumar this is an output for $ sudo docker logs --follow mina -f --tail 10000 for the time block's been produced.
Tell me pls if it's enough
mina 2.log.zip

Relevant logs:
2021-01-20 10:04:00 UTC [Info] Successfully produced a new block
2021-01-20 10:04:03 UTC [Info] Generated transition $state_hash was accepted into transition frontier
	state_hash: "3NKKfLZinDCe5ECRhqKvQM8mUvyuRYxtTJo7YVUv1tLhjqtqdKQ1"
...
2021-01-20 10:04:25 UTC [Info] Broadcasting new state over gossip net

Relevant logs:
2021-01-20 10:04:00 UTC [Info] Successfully produced a new block
2021-01-20 10:04:03 UTC [Info] Generated transition $state_hash was accepted into transition frontier
	state_hash: "3NKKfLZinDCe5ECRhqKvQM8mUvyuRYxtTJo7YVUv1tLhjqtqdKQ1"
...
2021-01-20 10:04:25 UTC [Info] Broadcasting new state over gossip net

@mrmr1993 yes you can see it in the archive I provided above. pls take a look

@mrmr1993 yes you can see it in the archive I provided above. pls take a look

@bidulemachin could you check in the log file if there is a log that says Internally generated block $state_hash cannot be rebroadcast because it's not a valid time to do so ($timing). I don't see your block in any of our logs

Yes I found it
{
"timestamp": "2021-01-20 15:39:06.408234Z",
"level": "Error",
"source": {
"module": "Mina_lib",
"location": "File "src/lib/mina_lib/mina_lib.ml", line 1331, characters 26-53"
},
"message": "Internally generated block $state_hash cannot be rebroadcast because it's not a valid time to do so ($timing)",
"metadata": {
"external_transition": {
"data": {
"protocol_state": {
"previous_state_hash": "3NK7SDKMFtVer1JhU3WL8y5qQ88bMSvDQakLyu9gokdqWMHPuyV3",
"body": {
"genesis_state_hash": "3NLxYrjb7zmHdoFgBrubCN8ijM8v7eT8kvLiPLc9DHt3M8XrDDEG",
"blockchain_state": {
"staged_ledger_hash": {
"non_snark": {
"ledger_hash": "jxMGm5iNjKYNuMR6FBPG8ttCroX7Sej4a2p1FbekHcxLtjKQQ2Q",
"aux_hash": "UXRmyd2wYLGDcipQWkRTW6x3PfCvT6gAGkpRQqRqZCANbA4wtc",
"pending_coinbase_aux": "XUEeguj2J8Bj3knNEQz2GSFPTsdphoXMVdxeAQU5ZMUvafvTFC"
},
"pending_coinbase_hash": "2n1kgZMPBEbwG4NB4cjhfbWafVgPkRfu8TQrVg7Jq6Wep6rtYXA3"
},
"snarked_ledger_hash": "jwyae99tBovrVdfHxpVHwM2R44W8zF1C4Ea6HzM1ydcYVFTRchf",
"genesis_ledger_hash": "jxX6VJ84HaafrKozFRA4qjnni4aPXqXC2H5vQLKSryNpKTXuz1R",
"snarked_next_available_token": "2",
"timestamp": "1611156930000"
},
"consensus_state": {
"blockchain_length": "2713",
"epoch_count": "1",
"min_window_density": "1",
"sub_window_densities": [
"2",
"2",
"3",
"0",
"2",
"2",
"1",
"0",
"1",
"1",
"1"
],
"last_vrf_output": "jVNdtsghN2yQDDNGU_D8rmfD1uPFC-ixwWSIc5wTBAA=",
"total_currency": "166902600000001000",
"curr_global_slot": {
"slot_number": "10007",
"slots_per_epoch": "7140"
},
"global_slot_since_genesis": "10007",
"staking_epoch_data": {
"ledger": {
"hash": "jxX6VJ84HaafrKozFRA4qjnni4aPXqXC2H5vQLKSryNpKTXuz1R",
"total_currency": "165950000000001000"
},
"seed": "2vb1Mjvydod6sEwn7qpbejKCfRqugMgyG3MHXXRKcAkwQLRs9fj8",
"start_checkpoint": "3NK2tkzqqK5spR2sZ7tujjqPksL45M3UUrcA4WhCkeiPtnugyE2x",
"lock_checkpoint": "3NK5G8Xqn1Prh3XoTyZ2tqntJC6X2nVwruv5mEJCL3GaTk7jKUNo",
"epoch_length": "1769"
},
"next_epoch_data": {
"ledger": {
"hash": "jx7XXjRfJj2mGXmiHQmpm6ZgTxz14udpugyFtw4DefJFpie7apN",
"total_currency": "166537000000001000"
},
"seed": "2vaQinqP8sDNgY4saCLoVobRK7yyznh1mNq7Nk1pByo17Qwnf7zR",
"start_checkpoint": "3NLdAqxtBRYxYbCWMXxGu6j1hGDrpQwGkBDF9QvGxmtpziXQDADu",
"lock_checkpoint": "3NK7SDKMFtVer1JhU3WL8y5qQ88bMSvDQakLyu9gokdqWMHPuyV3",
"epoch_length": "945"
},
"has_ancestor_in_same_checkpoint_window": true,
"block_stake_winner": "B62qpNmgVUZ9vCzVwHX7qw329yMnmMNnYKG8KmhRWdbbPzYEN4qEQoo",
"block_creator": "B62qpNmgVUZ9vCzVwHX7qw329yMnmMNnYKG8KmhRWdbbPzYEN4qEQoo",
"coinbase_receiver": "B62qpNmgVUZ9vCzVwHX7qw329yMnmMNnYKG8KmhRWdbbPzYEN4qEQoo",
"supercharge_coinbase": true
},
"constants": {
"k": "290",
"slots_per_epoch": "7140",
"slots_per_sub_window": "7",
"delta": "0",
"genesis_state_timestamp": "1609355670000"
}
}
},
"protocol_state_proof": "",
"staged_ledger_diff": "",
"delta_transition_chain_proof": "",
"current_protocol_version": "1.1.0",
"proposed_protocol_version": ""
},
"hash": "3NKFzrh7sa1oZgeNcx8UbVc5Xrm1WDMSFPNHwtZfQA1zxfDgR7cz"
},
"host": "<IP_ADDRESS>",
"peer_id": "12D3KooWRQjjWVu78nW5kVwaamkmjfXj8ofR7YXnCwaAWCqMFgoC",
"pid": 2918,
"port": 8302,
"state_hash": "3NKFzrh7sa1oZgeNcx8UbVc5Xrm1WDMSFPNHwtZfQA1zxfDgR7cz",
"timing": "1 slots too late"
}
}
Any idea about the root cause ? Something wrong in my setup ?

@bidulemachin could you check in the log file if there is a log that says Internally generated block $state_hash cannot be rebroadcast because it's not a valid time to do so ($timing). I don't see your block in any of our logs

Yes I found it
{
"timestamp": "2021-01-20 15:39:06.408234Z",
"level": "Error",
"source": {
"module": "Mina_lib",
"location": "File "src/lib/mina_lib/mina_lib.ml", line 1331, characters 26-53"
},
"message": "Internally generated block $state_hash cannot be rebroadcast because it's not a valid time to do so ($timing)",
"metadata": {
"external_transition": {
"data": {
"protocol_state": {
"previous_state_hash": "3NK7SDKMFtVer1JhU3WL8y5qQ88bMSvDQakLyu9gokdqWMHPuyV3",
"body": {
"genesis_state_hash": "3NLxYrjb7zmHdoFgBrubCN8ijM8v7eT8kvLiPLc9DHt3M8XrDDEG",
"blockchain_state": {
"staged_ledger_hash": {
"non_snark": {
"ledger_hash": "jxMGm5iNjKYNuMR6FBPG8ttCroX7Sej4a2p1FbekHcxLtjKQQ2Q",
"aux_hash": "UXRmyd2wYLGDcipQWkRTW6x3PfCvT6gAGkpRQqRqZCANbA4wtc",
"pending_coinbase_aux": "XUEeguj2J8Bj3knNEQz2GSFPTsdphoXMVdxeAQU5ZMUvafvTFC"
},
"pending_coinbase_hash": "2n1kgZMPBEbwG4NB4cjhfbWafVgPkRfu8TQrVg7Jq6Wep6rtYXA3"
},
"snarked_ledger_hash": "jwyae99tBovrVdfHxpVHwM2R44W8zF1C4Ea6HzM1ydcYVFTRchf",
"genesis_ledger_hash": "jxX6VJ84HaafrKozFRA4qjnni4aPXqXC2H5vQLKSryNpKTXuz1R",
"snarked_next_available_token": "2",
"timestamp": "1611156930000"
},
"consensus_state": {
"blockchain_length": "2713",
"epoch_count": "1",
"min_window_density": "1",
"sub_window_densities": [
"2",
"2",
"3",
"0",
"2",
"2",
"1",
"0",
"1",
"1",
"1"
],
"last_vrf_output": "jVNdtsghN2yQDDNGU_D8rmfD1uPFC-ixwWSIc5wTBAA=",
"total_currency": "166902600000001000",
"curr_global_slot": {
"slot_number": "10007",
"slots_per_epoch": "7140"
},
"global_slot_since_genesis": "10007",
"staking_epoch_data": {
"ledger": {
"hash": "jxX6VJ84HaafrKozFRA4qjnni4aPXqXC2H5vQLKSryNpKTXuz1R",
"total_currency": "165950000000001000"
},
"seed": "2vb1Mjvydod6sEwn7qpbejKCfRqugMgyG3MHXXRKcAkwQLRs9fj8",
"start_checkpoint": "3NK2tkzqqK5spR2sZ7tujjqPksL45M3UUrcA4WhCkeiPtnugyE2x",
"lock_checkpoint": "3NK5G8Xqn1Prh3XoTyZ2tqntJC6X2nVwruv5mEJCL3GaTk7jKUNo",
"epoch_length": "1769"
},
"next_epoch_data": {
"ledger": {
"hash": "jx7XXjRfJj2mGXmiHQmpm6ZgTxz14udpugyFtw4DefJFpie7apN",
"total_currency": "166537000000001000"
},
"seed": "2vaQinqP8sDNgY4saCLoVobRK7yyznh1mNq7Nk1pByo17Qwnf7zR",
"start_checkpoint": "3NLdAqxtBRYxYbCWMXxGu6j1hGDrpQwGkBDF9QvGxmtpziXQDADu",
"lock_checkpoint": "3NK7SDKMFtVer1JhU3WL8y5qQ88bMSvDQakLyu9gokdqWMHPuyV3",
"epoch_length": "945"
},
"has_ancestor_in_same_checkpoint_window": true,
"block_stake_winner": "B62qpNmgVUZ9vCzVwHX7qw329yMnmMNnYKG8KmhRWdbbPzYEN4qEQoo",
"block_creator": "B62qpNmgVUZ9vCzVwHX7qw329yMnmMNnYKG8KmhRWdbbPzYEN4qEQoo",
"coinbase_receiver": "B62qpNmgVUZ9vCzVwHX7qw329yMnmMNnYKG8KmhRWdbbPzYEN4qEQoo",
"supercharge_coinbase": true
},
"constants": {
"k": "290",
"slots_per_epoch": "7140",
"slots_per_sub_window": "7",
"delta": "0",
"genesis_state_timestamp": "1609355670000"
}
}
},
"protocol_state_proof": "",
"staged_ledger_diff": "",
"delta_transition_chain_proof": "",
"current_protocol_version": "1.1.0",
"proposed_protocol_version": ""
},
"hash": "3NKFzrh7sa1oZgeNcx8UbVc5Xrm1WDMSFPNHwtZfQA1zxfDgR7cz"
},
"host": "<IP_ADDRESS>",
"peer_id": "12D3KooWRQjjWVu78nW5kVwaamkmjfXj8ofR7YXnCwaAWCqMFgoC",
"pid": 2918,
"port": 8302,
"state_hash": "3NKFzrh7sa1oZgeNcx8UbVc5Xrm1WDMSFPNHwtZfQA1zxfDgR7cz",
"timing": "1 slots too late"
}
}
Any idea about the root cause ? Something wrong in my setup ?

2982 block is red
The node was synchronized. There was definitely no lag in the blocks. Everything was exactly the same as in esplorer. The snark wasn't turned on. Home PC. The block was created but it was created in red. The session was started in tmux and I can't get the logs right now. For what reason the block is red and my block was taken away https://minaexplorer.com/block/3NK4AaNkLVnNRkffW5LEsd4YuAVZ2e8kWWNtfpHqMsss2L3Dzzqc
slot and time are the same.  i got fresh peers. correct block height and max observed block height.

2982 block is red
The node was synchronized. There was definitely no lag in the blocks. Everything was exactly the same as in esplorer. The snark wasn't turned on. Home PC. The block was created but it was created in red. The session was started in tmux and I can't get the logs right now. For what reason the block is red and my block was taken away https://minaexplorer.com/block/3NK4AaNkLVnNRkffW5LEsd4YuAVZ2e8kWWNtfpHqMsss2L3Dzzqc
slot and time are the same.  i got fresh peers. correct block height and max observed block height.

It is not displayed that I produced the block. This is the third time. There is a log only for today, I hope this is enough.
The node is synchronized, snark didn't start.
https://minaexplorer.com/wallet/B62qmGwjXbSMkN45XAC5PAxAQaKNBm1pAEpt9ZD6PQN2qgP2ctSWFH4
1.txt

It is not displayed that I produced the block. This is the third time. There is a log only for today, I hope this is enough.
The node is synchronized, snark didn't start.
https://minaexplorer.com/wallet/B62qmGwjXbSMkN45XAC5PAxAQaKNBm1pAEpt9ZD6PQN2qgP2ctSWFH4
1.txt

@bugz34btc this is not an issue and is perfectly normal. This issue is related to blocks that were never broadcast or gossiped. See this for more of the explanation as to why https://minaprotocol.com/docs/troubleshooting#why-was-my-block-orphaned--why-did-i-not-get-a-block-reward

@bugz34btc this is not an issue and is perfectly normal. This issue is related to blocks that were never broadcast or gossiped. See this for more of the explanation as to why https://minaprotocol.com/docs/troubleshooting#why-was-my-block-orphaned--why-did-i-not-get-a-block-reward

Hi!
I also have missing blocks. My /root/.coda-config/mina-rejected-blocks.log log is attached:
mina-rejected-blocks.log
Thanks for looking at it.

Hi!
I also have missing blocks. My /root/.coda-config/mina-rejected-blocks.log log is attached:
mina-rejected-blocks.log
Thanks for looking at it.

I have 1 missing block. See logs below
``2021-01-14 10:32:30 UTC [Info] Producing new block with parent $breadcrumb
breadcrumb: {
"validated_transition": {
"data": {
"protocol_state": {
"previous_state_hash":
"3NKmns1bFoWuHrHR8EzaQqAcrH2Y14S7jTz1d31Lku4yM1NC7Gyz",
"body": {
"genesis_state_hash":
"3NLxYrjb7zmHdoFgBrubCN8ijM8v7eT8kvLiPLc9DHt3M8XrDDEG",
"blockchain_state": {
"staged_ledger_hash": {
"non_snark": {
"ledger_hash":
"jwKwmgGhUPu7arZwZcUHMTtMJtbZDH6LyaAngNSfQQ9uUt4UuYy",
"aux_hash":
"V8YvcVwGVakW3NqRvPSKABhQE93D1CphyENX5ZcYBAQ5cCq5Rs",
"pending_coinbase_aux":
"WeLYrxS7dvYFwGKyBe9xbZofZQwYhXNUyZoem8YDsuRReKW6a7"
},
"pending_coinbase_hash":
"2n1PcX9VBMBbWkUB56c9qAjGgQMNqgwB44pnKqvZjkkVCPNdUwGW"
},
"snarked_ledger_hash":
"jx37bLKCGfa72gsnCSUthdaSSSsy6Ax7asYzczsFFfRUDsBz7r3",
"genesis_ledger_hash":
"jxX6VJ84HaafrKozFRA4qjnni4aPXqXC2H5vQLKSryNpKTXuz1R",
"snarked_next_available_token": "2",
"timestamp": "1610619090000"
},
"consensus_state": {
"blockchain_length": "1711",
"epoch_count": "0",
"min_window_density": "1",
"sub_window_densities": [
"2", "4", "2", "3", "4", "3", "3", "1", "2", "3", "1"
],
"last_vrf_output": "nmatQSEnc5hWvsaaSdZHvTPQ8XjWgyjwXwLdgYARBAA=",
"total_currency": "166515800000001000",
"curr_global_slot": {
"slot_number": "7019",
"slots_per_epoch": "7140"
},
"global_slot_since_genesis": "7019",
"staking_epoch_data": {
"ledger": {
"hash": "jxX6VJ84HaafrKozFRA4qjnni4aPXqXC2H5vQLKSryNpKTXuz1R",
"total_currency": "165950000000001000"
},
"seed": "2va9BGv9JrLTtrzZttiEMDYw1Zj6a6EHzXjmP9evHDTG3oEquURA",
"start_checkpoint":
"3NK2tkzqqK5spR2sZ7tujjqPksL45M3UUrcA4WhCkeiPtnugyE2x",
"lock_checkpoint":
"3NK2tkzqqK5spR2sZ7tujjqPksL45M3UUrcA4WhCkeiPtnugyE2x",
"epoch_length": "1"
},
"next_epoch_data": {
"ledger": {
"hash": "jxX6VJ84HaafrKozFRA4qjnni4aPXqXC2H5vQLKSryNpKTXuz1R",
"total_currency": "165950000000001000"
},
"seed": "2vb1Mjvydod6sEwn7qpbejKCfRqugMgyG3MHXXRKcAkwQLRs9fj8",
"start_checkpoint":
"3NK2tkzqqK5spR2sZ7tujjqPksL45M3UUrcA4WhCkeiPtnugyE2x",
"lock_checkpoint":
"3NK5G8Xqn1Prh3XoTyZ2tqntJC6X2nVwruv5mEJCL3GaTk7jKUNo",
"epoch_length": "1712"
},
"has_ancestor_in_same_checkpoint_window": true,
"block_stake_winner":
"B62qqhR2FRfU5u3XHjn8ewxToVKrW8NuA1nF8VrkY7M2VYDqa5hHn4c",
"block_creator":
"B62qqhR2FRfU5u3XHjn8ewxToVKrW8NuA1nF8VrkY7M2VYDqa5hHn4c",
"coinbase_receiver":
"B62qqhR2FRfU5u3XHjn8ewxToVKrW8NuA1nF8VrkY7M2VYDqa5hHn4c",
"supercharge_coinbase": true
},
"constants": {
"k": "290",
"slots_per_epoch": "7140",
"slots_per_sub_window": "7",
"delta": "0",
"genesis_state_timestamp": "1609355670000"
}
}
},
"protocol_state_proof": "",
"staged_ledger_diff": "",
"delta_transition_chain_proof": "",
"current_protocol_version": "1.1.0",
"proposed_protocol_version": ""
},
"hash": "3NKAnAmFfLmLfc3UutpgP5j8XggqC5SZpda9JoBwGhRta4p8T9WM"
},
"staged_ledger": "",
"just_emitted_a_proof": true,
"transition_receipt_time": "2021-01-14T10:22:50.497735Z"
}
2021-01-14 10:33:21 UTC [Info] Build breadcrumb on produced block
2021-01-14 10:33:21 UTC [Info] Successfully produced a new block
2021-01-14 10:33:23 UTC [Info] Generated transition $state_hash was accepted into transition frontier
state_hash: "3NLTrad7LGcBMMhZkJzinxZvew5SGbSVNjm5psBmVJ2MoEzg1ihh"``

I have 1 missing block. See logs below
``2021-01-14 10:32:30 UTC [Info] Producing new block with parent $breadcrumb
breadcrumb: {
"validated_transition": {
"data": {
"protocol_state": {
"previous_state_hash":
"3NKmns1bFoWuHrHR8EzaQqAcrH2Y14S7jTz1d31Lku4yM1NC7Gyz",
"body": {
"genesis_state_hash":
"3NLxYrjb7zmHdoFgBrubCN8ijM8v7eT8kvLiPLc9DHt3M8XrDDEG",
"blockchain_state": {
"staged_ledger_hash": {
"non_snark": {
"ledger_hash":
"jwKwmgGhUPu7arZwZcUHMTtMJtbZDH6LyaAngNSfQQ9uUt4UuYy",
"aux_hash":
"V8YvcVwGVakW3NqRvPSKABhQE93D1CphyENX5ZcYBAQ5cCq5Rs",
"pending_coinbase_aux":
"WeLYrxS7dvYFwGKyBe9xbZofZQwYhXNUyZoem8YDsuRReKW6a7"
},
"pending_coinbase_hash":
"2n1PcX9VBMBbWkUB56c9qAjGgQMNqgwB44pnKqvZjkkVCPNdUwGW"
},
"snarked_ledger_hash":
"jx37bLKCGfa72gsnCSUthdaSSSsy6Ax7asYzczsFFfRUDsBz7r3",
"genesis_ledger_hash":
"jxX6VJ84HaafrKozFRA4qjnni4aPXqXC2H5vQLKSryNpKTXuz1R",
"snarked_next_available_token": "2",
"timestamp": "1610619090000"
},
"consensus_state": {
"blockchain_length": "1711",
"epoch_count": "0",
"min_window_density": "1",
"sub_window_densities": [
"2", "4", "2", "3", "4", "3", "3", "1", "2", "3", "1"
],
"last_vrf_output": "nmatQSEnc5hWvsaaSdZHvTPQ8XjWgyjwXwLdgYARBAA=",
"total_currency": "166515800000001000",
"curr_global_slot": {
"slot_number": "7019",
"slots_per_epoch": "7140"
},
"global_slot_since_genesis": "7019",
"staking_epoch_data": {
"ledger": {
"hash": "jxX6VJ84HaafrKozFRA4qjnni4aPXqXC2H5vQLKSryNpKTXuz1R",
"total_currency": "165950000000001000"
},
"seed": "2va9BGv9JrLTtrzZttiEMDYw1Zj6a6EHzXjmP9evHDTG3oEquURA",
"start_checkpoint":
"3NK2tkzqqK5spR2sZ7tujjqPksL45M3UUrcA4WhCkeiPtnugyE2x",
"lock_checkpoint":
"3NK2tkzqqK5spR2sZ7tujjqPksL45M3UUrcA4WhCkeiPtnugyE2x",
"epoch_length": "1"
},
"next_epoch_data": {
"ledger": {
"hash": "jxX6VJ84HaafrKozFRA4qjnni4aPXqXC2H5vQLKSryNpKTXuz1R",
"total_currency": "165950000000001000"
},
"seed": "2vb1Mjvydod6sEwn7qpbejKCfRqugMgyG3MHXXRKcAkwQLRs9fj8",
"start_checkpoint":
"3NK2tkzqqK5spR2sZ7tujjqPksL45M3UUrcA4WhCkeiPtnugyE2x",
"lock_checkpoint":
"3NK5G8Xqn1Prh3XoTyZ2tqntJC6X2nVwruv5mEJCL3GaTk7jKUNo",
"epoch_length": "1712"
},
"has_ancestor_in_same_checkpoint_window": true,
"block_stake_winner":
"B62qqhR2FRfU5u3XHjn8ewxToVKrW8NuA1nF8VrkY7M2VYDqa5hHn4c",
"block_creator":
"B62qqhR2FRfU5u3XHjn8ewxToVKrW8NuA1nF8VrkY7M2VYDqa5hHn4c",
"coinbase_receiver":
"B62qqhR2FRfU5u3XHjn8ewxToVKrW8NuA1nF8VrkY7M2VYDqa5hHn4c",
"supercharge_coinbase": true
},
"constants": {
"k": "290",
"slots_per_epoch": "7140",
"slots_per_sub_window": "7",
"delta": "0",
"genesis_state_timestamp": "1609355670000"
}
}
},
"protocol_state_proof": "",
"staged_ledger_diff": "",
"delta_transition_chain_proof": "",
"current_protocol_version": "1.1.0",
"proposed_protocol_version": ""
},
"hash": "3NKAnAmFfLmLfc3UutpgP5j8XggqC5SZpda9JoBwGhRta4p8T9WM"
},
"staged_ledger": "",
"just_emitted_a_proof": true,
"transition_receipt_time": "2021-01-14T10:22:50.497735Z"
}
2021-01-14 10:33:21 UTC [Info] Build breadcrumb on produced block
2021-01-14 10:33:21 UTC [Info] Successfully produced a new block
2021-01-14 10:33:23 UTC [Info] Generated transition $state_hash was accepted into transition frontier
state_hash: "3NLTrad7LGcBMMhZkJzinxZvew5SGbSVNjm5psBmVJ2MoEzg1ihh"``

Collecting logs to see how frequently blocks are getting rejected due to timing

Collecting logs to see how frequently blocks are getting rejected due to timing

Collecting logs to see how frequently blocks are getting rejected due to timing

Which logs? I've already provided logs from the time block got produced (see in my previous message). Attaching mina-rejected-blocks.log too (see below) but it doesn't include that timeframe (Jan 14) cuz I've recently updated a docker image to a newer version.
mina-rejected-blocks.log

Collecting logs to see how frequently blocks are getting rejected due to timing

Which logs? I've already provided logs from the time block got produced (see in my previous message). Attaching mina-rejected-blocks.log too (see below) but it doesn't include that timeframe (Jan 14) cuz I've recently updated a docker image to a newer version.
mina-rejected-blocks.log

Hello. Again, when producing a block, I see that the block was produced successfully, but was not counted. I have already lost 5 blocks and could not get into the top 100 participants. I attach the log. I hope for understanding and fixing the bug. Good luck.
https://github.com/abukatov/Share/blob/main/mina-rejected-blocks.log
2021-01-28 10:37:59 UTC [Info] Sending transition_chain_proof to $peer
peer: {
"host": "<IP_ADDRESS>",
"peer_id": "12D3KooWRVUWspmwjxA9tp6Njsb2qZLiDZiSeJLCrLU2nw5woVFx",
"libp2p_port": 8302
}
2021-01-28 10:38:30 UTC [Info] Producing new block with parent $breadcrumb
breadcrumb: {
"validated_transition": {
"data": {
"protocol_state": {
"previous_state_hash":
"3NKuYotRrXgo7fBhbEM1MytHRtVMDRDS22JA9b8A13pF6xG8mW3S",
"body": {
"genesis_state_hash":
"3NLxYrjb7zmHdoFgBrubCN8ijM8v7eT8kvLiPLc9DHt3M8XrDDEG",
"blockchain_state": {
"staged_ledger_hash": {
"non_snark": {
"ledger_hash":
"jxRpwEVTXNfndixCh6UpFDbuwcQ2i3Wj3gx6yJkuAgdWS9AviQ9",
"aux_hash":
"VmcNP48Ym8kFrJ2Bi4RVE3hmyQ7FUPC6YVvq7ZeYd9fC2okFFg",
"pending_coinbase_aux":
"WuV5G9BFibFUQToRpzzojyLyJcZgiAEKTsxb4wPNqj9cWnAKTz"
},
"pending_coinbase_hash":
"2mzcuZapHMuGuSn5E3moYg7sC1w2KwFQJf2BcZv6rySyP3i823WL"
},
"snarked_ledger_hash":
"jwjNVnabHqA5A9HvLUv5Qip4vux2FghP6m9LbaQHZjb3adVRjXa",
"genesis_ledger_hash":
"jxX6VJ84HaafrKozFRA4qjnni4aPXqXC2H5vQLKSryNpKTXuz1R",
"snarked_next_available_token": "2",
"timestamp": "1611829590000"
},
"consensus_state": {
"blockchain_length": "4122",
"epoch_count": "1",
"min_window_density": "1",
"sub_window_densities": [
"3", "1", "4", "2", "3", "1", "2", "4", "3", "5", "4"
],
"last_vrf_output": "LB5iudKAXVJ7cOkPIL4zmbDK-ndJ-H6mHamp150tdAA=",
"total_currency": "167359000000001000",
"curr_global_slot": {
"slot_number": "13744",
"slots_per_epoch": "7140"
},
"global_slot_since_genesis": "13744",
"staking_epoch_data": {
"ledger": {
"hash": "jxX6VJ84HaafrKozFRA4qjnni4aPXqXC2H5vQLKSryNpKTXuz1R",
"total_currency": "165950000000001000"
},
"seed": "2vb1Mjvydod6sEwn7qpbejKCfRqugMgyG3MHXXRKcAkwQLRs9fj8",
"start_checkpoint":
"3NK2tkzqqK5spR2sZ7tujjqPksL45M3UUrcA4WhCkeiPtnugyE2x",
"lock_checkpoint":
"3NK5G8Xqn1Prh3XoTyZ2tqntJC6X2nVwruv5mEJCL3GaTk7jKUNo",
"epoch_length": "1769"
},
"next_epoch_data": {
"ledger": {
"hash": "jx7XXjRfJj2mGXmiHQmpm6ZgTxz14udpugyFtw4DefJFpie7apN",
"total_currency": "166537000000001000"
},
"seed": "2vavBR2GfJWvWkpC7yGJQFnts18nHaFjdVEr84r1Y9DQXvnJRhmd",
"start_checkpoint":
"3NLdAqxtBRYxYbCWMXxGu6j1hGDrpQwGkBDF9QvGxmtpziXQDADu",
"lock_checkpoint":
"3NL4Eis1pS1yrPdfCbiJcpCCYsHuXY3ZgEzHojPnFWfMK9gKmhZh",
"epoch_length": "2354"
},
"has_ancestor_in_same_checkpoint_window": true,
"block_stake_winner":
"B62qqXnsmUNXibyfBwf3ADyopwW1tTJDPZWnFhRdFfjRHjRjp634AzL",
"block_creator":
"B62qqiuSjeZDbTrxvcnxXfq4Z4R9CsFC9BCuuVuHiwUQ6ErNyJ717Az",
"coinbase_receiver":
"B62qqiuSjeZDbTrxvcnxXfq4Z4R9CsFC9BCuuVuHiwUQ6ErNyJ717Az",
"supercharge_coinbase": true
},
"constants": {
"k": "290",
"slots_per_epoch": "7140",
"slots_per_sub_window": "7",
"delta": "0",
"genesis_state_timestamp": "1609355670000"
}
}
},
"protocol_state_proof": "",
"staged_ledger_diff": "",
"delta_transition_chain_proof": "",
"current_protocol_version": "1.1.0",
"proposed_protocol_version": ""
},
"hash": "3NKsEZ3XDVhryZuQjtTPnbXGes3pX2qbwSgJcx2TUbnzzUNTyths"
},
"staged_ledger": "",
"just_emitted_a_proof": true,
"transition_receipt_time": "2021-01-28T10:28:13.035300Z"
}
2021-01-28 10:38:30 UTC [Info] No locked tokens in the delegator/delegatee account, applying supercharged coinbase
2021-01-28 10:38:45 UTC [Warn] failed to reset stream (this means it was probably closed successfully): $error
error: {
"commit_id": "[DIRTY]d075f83d26490f6510bbb14bbfe3c771256257b5",
"string":
"RPC #83765 failed: "internal RPC error error: unknown stream_idx""
}
2021-01-28 10:39:39 UTC [Info] Protocol_state_proof proving time(ms): 54844
2021-01-28 10:39:48 UTC [Info] Build breadcrumb on produced block: 9655
2021-01-28 10:39:48 UTC [Info] Successfully produced a new block
2021-01-28 10:39:51 UTC [Info] Generated transition $state_hash was accepted into transition frontier
state_hash: "3NKVLZ6iiSkCg9MmCKzR9dtTPvPmSRuJnhLkMEUCvNrMcRBMuRQ5"
2021-01-28 10:39:51 UTC [Info] Updating new available work took 10.658740997314453 ms
2021-01-28 10:39:53 UTC [Info] Determining next slot to produce block
2021-01-28 10:39:53 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6609
2021-01-28 10:39:53 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6610
2021-01-28 10:39:53 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6611
2021-01-28 10:39:53 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6612
2021-01-28 10:39:53 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6613
2021-01-28 10:39:53 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6614
2021-01-28 10:39:53 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6615
2021-01-28 10:39:53 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6616
2021-01-28 10:39:53 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6617
2021-01-28 10:39:53 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6618
2021-01-28 10:39:53 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6619
2021-01-28 10:39:53 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6620
2021-01-28 10:39:53 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6621
2021-01-28 10:39:53 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6622
2021-01-28 10:39:54 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6623
2021-01-28 10:39:54 UTC [Info] Broadcasting new state over gossip net
2021-01-28 10:39:54 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6624
2021-01-28 10:39:54 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6625
2021-01-28 10:39:54 UTC [Info] Saw block with state hash $state_hash
state_hash: "3NKVLZ6iiSkCg9MmCKzR9dtTPvPmSRuJnhLkMEUCvNrMcRBMuRQ5"
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6626
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6627
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6628
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6629
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6630
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6631
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6632
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6633
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6634
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6635
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6636
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6637
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6638
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6639
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6640
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6641
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6642
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6643
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6644
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6645
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6646
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6647
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6648
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6649
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6650
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6651
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6652
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6653
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6654
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6655
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6656
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6657
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6658
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6659
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6660
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6661
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6662
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6663
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6664
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6665
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6666
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6667
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6668
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6669
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6670
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6671
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6672
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6673
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6674
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6675
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6676
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6677
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6678
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6679
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6680
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6681
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6682
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6683
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6684
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6685
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6686
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6687
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6688
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6689
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6690
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6691
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6692
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6693
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6694
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6695
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6696
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6697
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6698
2021-01-28 10:39:55 UTC [Info] Producing block in 89 slots
2021-01-28 10:39:55 UTC [Info] Received a block from $sender
sender: {
"Remote": {
"host": "<IP_ADDRESS>",
"peer_id": "12D3KooWM1dHTpuJ7q6WAKzRixpNvjd4daxZXUFwHgLf8aG679Br",
"libp2p_port": 8302
}
}
2021-01-28 10:40:01 UTC [Info] Sending transition_chain_proof to $peer
peer: {
"host": "<IP_ADDRESS>",
"peer_id": "12D3KooWGGdjQAHaxsw6ZBJ6mXYhxyahZGzW1TA8Jt9iuG1FKYbf",
"libp2p_port": 8302
}

Hello. Again, when producing a block, I see that the block was produced successfully, but was not counted. I have already lost 5 blocks and could not get into the top 100 participants. I attach the log. I hope for understanding and fixing the bug. Good luck.
https://github.com/abukatov/Share/blob/main/mina-rejected-blocks.log
2021-01-28 10:37:59 UTC [Info] Sending transition_chain_proof to $peer
peer: {
"host": "<IP_ADDRESS>",
"peer_id": "12D3KooWRVUWspmwjxA9tp6Njsb2qZLiDZiSeJLCrLU2nw5woVFx",
"libp2p_port": 8302
}
2021-01-28 10:38:30 UTC [Info] Producing new block with parent $breadcrumb
breadcrumb: {
"validated_transition": {
"data": {
"protocol_state": {
"previous_state_hash":
"3NKuYotRrXgo7fBhbEM1MytHRtVMDRDS22JA9b8A13pF6xG8mW3S",
"body": {
"genesis_state_hash":
"3NLxYrjb7zmHdoFgBrubCN8ijM8v7eT8kvLiPLc9DHt3M8XrDDEG",
"blockchain_state": {
"staged_ledger_hash": {
"non_snark": {
"ledger_hash":
"jxRpwEVTXNfndixCh6UpFDbuwcQ2i3Wj3gx6yJkuAgdWS9AviQ9",
"aux_hash":
"VmcNP48Ym8kFrJ2Bi4RVE3hmyQ7FUPC6YVvq7ZeYd9fC2okFFg",
"pending_coinbase_aux":
"WuV5G9BFibFUQToRpzzojyLyJcZgiAEKTsxb4wPNqj9cWnAKTz"
},
"pending_coinbase_hash":
"2mzcuZapHMuGuSn5E3moYg7sC1w2KwFQJf2BcZv6rySyP3i823WL"
},
"snarked_ledger_hash":
"jwjNVnabHqA5A9HvLUv5Qip4vux2FghP6m9LbaQHZjb3adVRjXa",
"genesis_ledger_hash":
"jxX6VJ84HaafrKozFRA4qjnni4aPXqXC2H5vQLKSryNpKTXuz1R",
"snarked_next_available_token": "2",
"timestamp": "1611829590000"
},
"consensus_state": {
"blockchain_length": "4122",
"epoch_count": "1",
"min_window_density": "1",
"sub_window_densities": [
"3", "1", "4", "2", "3", "1", "2", "4", "3", "5", "4"
],
"last_vrf_output": "LB5iudKAXVJ7cOkPIL4zmbDK-ndJ-H6mHamp150tdAA=",
"total_currency": "167359000000001000",
"curr_global_slot": {
"slot_number": "13744",
"slots_per_epoch": "7140"
},
"global_slot_since_genesis": "13744",
"staking_epoch_data": {
"ledger": {
"hash": "jxX6VJ84HaafrKozFRA4qjnni4aPXqXC2H5vQLKSryNpKTXuz1R",
"total_currency": "165950000000001000"
},
"seed": "2vb1Mjvydod6sEwn7qpbejKCfRqugMgyG3MHXXRKcAkwQLRs9fj8",
"start_checkpoint":
"3NK2tkzqqK5spR2sZ7tujjqPksL45M3UUrcA4WhCkeiPtnugyE2x",
"lock_checkpoint":
"3NK5G8Xqn1Prh3XoTyZ2tqntJC6X2nVwruv5mEJCL3GaTk7jKUNo",
"epoch_length": "1769"
},
"next_epoch_data": {
"ledger": {
"hash": "jx7XXjRfJj2mGXmiHQmpm6ZgTxz14udpugyFtw4DefJFpie7apN",
"total_currency": "166537000000001000"
},
"seed": "2vavBR2GfJWvWkpC7yGJQFnts18nHaFjdVEr84r1Y9DQXvnJRhmd",
"start_checkpoint":
"3NLdAqxtBRYxYbCWMXxGu6j1hGDrpQwGkBDF9QvGxmtpziXQDADu",
"lock_checkpoint":
"3NL4Eis1pS1yrPdfCbiJcpCCYsHuXY3ZgEzHojPnFWfMK9gKmhZh",
"epoch_length": "2354"
},
"has_ancestor_in_same_checkpoint_window": true,
"block_stake_winner":
"B62qqXnsmUNXibyfBwf3ADyopwW1tTJDPZWnFhRdFfjRHjRjp634AzL",
"block_creator":
"B62qqiuSjeZDbTrxvcnxXfq4Z4R9CsFC9BCuuVuHiwUQ6ErNyJ717Az",
"coinbase_receiver":
"B62qqiuSjeZDbTrxvcnxXfq4Z4R9CsFC9BCuuVuHiwUQ6ErNyJ717Az",
"supercharge_coinbase": true
},
"constants": {
"k": "290",
"slots_per_epoch": "7140",
"slots_per_sub_window": "7",
"delta": "0",
"genesis_state_timestamp": "1609355670000"
}
}
},
"protocol_state_proof": "",
"staged_ledger_diff": "",
"delta_transition_chain_proof": "",
"current_protocol_version": "1.1.0",
"proposed_protocol_version": ""
},
"hash": "3NKsEZ3XDVhryZuQjtTPnbXGes3pX2qbwSgJcx2TUbnzzUNTyths"
},
"staged_ledger": "",
"just_emitted_a_proof": true,
"transition_receipt_time": "2021-01-28T10:28:13.035300Z"
}
2021-01-28 10:38:30 UTC [Info] No locked tokens in the delegator/delegatee account, applying supercharged coinbase
2021-01-28 10:38:45 UTC [Warn] failed to reset stream (this means it was probably closed successfully): $error
error: {
"commit_id": "[DIRTY]d075f83d26490f6510bbb14bbfe3c771256257b5",
"string":
"RPC #83765 failed: "internal RPC error error: unknown stream_idx""
}
2021-01-28 10:39:39 UTC [Info] Protocol_state_proof proving time(ms): 54844
2021-01-28 10:39:48 UTC [Info] Build breadcrumb on produced block: 9655
2021-01-28 10:39:48 UTC [Info] Successfully produced a new block
2021-01-28 10:39:51 UTC [Info] Generated transition $state_hash was accepted into transition frontier
state_hash: "3NKVLZ6iiSkCg9MmCKzR9dtTPvPmSRuJnhLkMEUCvNrMcRBMuRQ5"
2021-01-28 10:39:51 UTC [Info] Updating new available work took 10.658740997314453 ms
2021-01-28 10:39:53 UTC [Info] Determining next slot to produce block
2021-01-28 10:39:53 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6609
2021-01-28 10:39:53 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6610
2021-01-28 10:39:53 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6611
2021-01-28 10:39:53 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6612
2021-01-28 10:39:53 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6613
2021-01-28 10:39:53 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6614
2021-01-28 10:39:53 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6615
2021-01-28 10:39:53 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6616
2021-01-28 10:39:53 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6617
2021-01-28 10:39:53 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6618
2021-01-28 10:39:53 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6619
2021-01-28 10:39:53 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6620
2021-01-28 10:39:53 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6621
2021-01-28 10:39:53 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6622
2021-01-28 10:39:54 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6623
2021-01-28 10:39:54 UTC [Info] Broadcasting new state over gossip net
2021-01-28 10:39:54 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6624
2021-01-28 10:39:54 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6625
2021-01-28 10:39:54 UTC [Info] Saw block with state hash $state_hash
state_hash: "3NKVLZ6iiSkCg9MmCKzR9dtTPvPmSRuJnhLkMEUCvNrMcRBMuRQ5"
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6626
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6627
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6628
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6629
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6630
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6631
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6632
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6633
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6634
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6635
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6636
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6637
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6638
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6639
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6640
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6641
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6642
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6643
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6644
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6645
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6646
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6647
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6648
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6649
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6650
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6651
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6652
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6653
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6654
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6655
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6656
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6657
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6658
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6659
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6660
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6661
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6662
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6663
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6664
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6665
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6666
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6667
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6668
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6669
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6670
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6671
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6672
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6673
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6674
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6675
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6676
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6677
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6678
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6679
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6680
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6681
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6682
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6683
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6684
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6685
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6686
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6687
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6688
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6689
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6690
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6691
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6692
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6693
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6694
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6695
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6696
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6697
2021-01-28 10:39:55 UTC [Info] Checking VRF evaluations at epoch: 1, slot: 6698
2021-01-28 10:39:55 UTC [Info] Producing block in 89 slots
2021-01-28 10:39:55 UTC [Info] Received a block from $sender
sender: {
"Remote": {
"host": "<IP_ADDRESS>",
"peer_id": "12D3KooWM1dHTpuJ7q6WAKzRixpNvjd4daxZXUFwHgLf8aG679Br",
"libp2p_port": 8302
}
}
2021-01-28 10:40:01 UTC [Info] Sending transition_chain_proof to $peer
peer: {
"host": "<IP_ADDRESS>",
"peer_id": "12D3KooWGGdjQAHaxsw6ZBJ6mXYhxyahZGzW1TA8Jt9iuG1FKYbf",
"libp2p_port": 8302
}

Hello Mina Team!
I want to participate in the network development working group.
I have completed all available tasks, technical and public.
The same problem for me.
Discord Id: AndiiCapital#8941
My node continues to work.
I ask you to take the time to my question and check this information.
My problem is with mining (block producer).
I ask you to check my node and solve this problem in the mine network.
You have your own archive nodes, you can see all the information on the node.
My node is constantly and steadily working, I keep track of the time when the block should be mined, for the entire time of the pressure, my node was able to get 2 blocks.
4 blocks my node did not mine. The node was functioning.
I want to tell you about the last block, which again passed my node.
The block was numbered 3950.
If you rely on the technical characteristics, then in terms of hardware I have even higher.
Insert :I'm aware that the minimum requirements to run a node in this testnet are: at least a 8-core processor, at least 16GB of RAM, and at least 1 Mbps network connectio.
My characteristics: at least a 8-core processor, at 32GB/SSD 240GB.
If the producer blocks are mined randomly, then apparently I'm not that lucky. Blocks producer take nodes with large ironHigh characteristics are taken by blocks.
Look at my node and evaluate the work of my node
I hope for your understanding and credit for my node.
But at minaexplorer.com my block doesn't exists –
https://minaexplorer.com/wallet/B62qruw9vKuF6Z5o8bTYZpCEhKTmqpgpYzXiVHT8nJmFx4g1xEYeSDE
Thanks.

Hello Mina Team!
I want to participate in the network development working group.
I have completed all available tasks, technical and public.
The same problem for me.
Discord Id: AndiiCapital#8941
My node continues to work.
I ask you to take the time to my question and check this information.
My problem is with mining (block producer).
I ask you to check my node and solve this problem in the mine network.
You have your own archive nodes, you can see all the information on the node.
My node is constantly and steadily working, I keep track of the time when the block should be mined, for the entire time of the pressure, my node was able to get 2 blocks.
4 blocks my node did not mine. The node was functioning.
I want to tell you about the last block, which again passed my node.
The block was numbered 3950.
If you rely on the technical characteristics, then in terms of hardware I have even higher.
Insert :I'm aware that the minimum requirements to run a node in this testnet are: at least a 8-core processor, at least 16GB of RAM, and at least 1 Mbps network connectio.
My characteristics: at least a 8-core processor, at 32GB/SSD 240GB.
If the producer blocks are mined randomly, then apparently I'm not that lucky. Blocks producer take nodes with large ironHigh characteristics are taken by blocks.
Look at my node and evaluate the work of my node
I hope for your understanding and credit for my node.
But at minaexplorer.com my block doesn't exists –
https://minaexplorer.com/wallet/B62qruw9vKuF6Z5o8bTYZpCEhKTmqpgpYzXiVHT8nJmFx4g1xEYeSDE
Thanks.

The same problem (with freezed node) in 0.3.0 release.
My node had freezing at 5488 block (status: synced, peers count = 51)
Last received block:
2021-02-06 15:19:53 UTC [Info] Received a block from $sender
sender: {
"Remote": {
"host": "<IP_ADDRESS>",
"peer_id": "12D3KooWLs712oWHq9qx73KpczCoEyHCZZHPJF3aR7UzNV7getTV",
"libp2p_port": 57968
}
}
Some errors about peers:
2021-02-06 12:40:54 UTC [Error] failed to connect to peer err=failed to dial 12D3KooWMbrGk3JQKzGGaCqkM1cAPuJs1Ws
kcybLgjBwSVaQZqbV: all dials failed
And validation errors:
2021-02-06 13:32:08 UTC [Error] validation timed out :frowning:
2021-02-06 13:32:18 UTC [Warn] validation callback timed out before we could respond

My node had freezing at 5776 block (status: synced, peers count = 58). Actual is 5808.
sender: {
"Remote": {
"host": "<IP_ADDRESS>",
"peer_id": "12D3KooWPaPX9c163tYgJ998UFTAM7j4sK9VKw5C1Qtwxbn2tc27",
"libp2p_port": 8302
error: {
"commit_id": "[DIRTY]5493fe344b40b6e668315fb8db4aa7eef4f1963f",
"string":
"RPC #323383 failed: "internal RPC error error: unknown stream_idx""
peer: {
"host": "<IP_ADDRESS>",
"peer_id": "12D3KooWQe4UikFtJp1qzbnsdrwgrWee6jQMTeeV1EhDi3dMjv6n",
"libp2p_port": 8302

Request for logs while spamming with txns is out to the QA net

Request for logs while spamming with txns is out to the QA net
Hi!

Discord Id: svd#2372
I already sent my mina-rejected-blocks.log to logs @ o1labs.org.
There are my full logs: https://bit.ly/3u3zEjP

Thanks all for sending us the logs.
From the preliminary investigation we saw that blocks were being rejected for being a slot too late. Furthermore, some of these were not due to resource constraints that could cause block generation to be slow and therefore rejected. To understand the impact of this, we introduced a new log file mina-rejected-blocks.log that has all the rejected blocks a node has seen. These are the blocks produced by other block producers that are generated well within the slot duration but end up being rejected by the node.
We analyzed more than 680 rejected-blocks logs that the community members sent us during testworld and found that there were quite a few blocks being rejected that could have filled empty slots in the main chain. Here are some numbers:
Number of slots for which 1 or more blocks were rejected due to invalid time (late by 1 or more slots) = 1263
There can be multiple blocks per slot and only one of them will get accepted.
However, number of missed slots due to blocks being rejected for arriving late= 236 out of 3215.
So ~7.5% of slots missed because of the latency+block production time+validation time.
We continued our investigation on this by testing the block rejected rate in different network conditions by varying the number of nodes connected, gossip traffic in the network as part of the qa task force.
Case 1: fewer number of nodes - QATF members sent 244 log files for slots 20000 - 22000. 61 slots were missed due to blocks being late. A total of 1255 total missed slots. 4.6% of missed slots due to block being late
Case2: Increased network traffic by sending lots of transactions. QATF members sent 175 log files from slot 22000 - 22900. 18 missed slots due to blocks being late. A total of 604 total missed slots. 2.98% of missed slots due to blocks being late
The slot fill rate was more or less the same as the observation on testworld. The % of slots missed due to blocks being delayed is less (~7.5% on testworld) but we also received a lot fewer log files. Also, we don't know how many node were connected during this period but the node count was certainly less than what it was in testworld since only QATF members were connected to encore
We also have an issue #7446 describing a possible fix to alleviate this issue.
Since the fix would impact the block production flow and the way network messages are processed, the plan is to do this after main-net.
Until fixed, this requires the assumed honest stake to increase by 7.5%, if an adversary were to fix this issue and keep their fix privately on their own nodes. Applying it to the protocol's initial constants' implied honesty assumption of 65%, this increases the assumption by up to 70.27% until the issue is fixed, still within threshold for launch.
Closing this issue and tracking https://github.com/MinaProtocol/mina/issues/7446


Issue with Horizontal server list addon

The Horizontal server list addon is not functioning correctly. It puts the close, resize and minimize button below the server lists. This is a problem because some actions get put behind the close button and I cannot click them or it will close discord. For instance in the theme repo area it puts the close, resize, and minimize button in front of the escape button shown here https://i.imgur.com/UVsrOBf.gif This plugin used to put the 3 buttons above the horizontal list like usual and it had no issues. Am I doing something wrong and is their something I can toggle when making a custom theme that makes it look like normal? When making the theme it does show the buttons in the correct area as shown here https://i.imgur.com/Y4qilbJ.gif

Your two links are two completely different configs. One has the Windows Titlebar optional import enabled and the other doesn't.

The one that shows the 3 buttons on top are on the website itself. The one that shows it underneath the horizontal server list is in discord. On the theme editor I don't see anything about tilebar. It's it a discord setting out something?

Your two links are two completely different configs. One has the Windows Titlebar optional import enabled and the other doesn't.
And the issue you're displaying isn't with Horizontal Server List but rather a incompatibility between the Windows Titlebar addon and Theme/Plugin Repo plugins

I see what you are talking about but I do not have that option enabled on the editor https://i.imgur.com/L2P6IJJ.gif I am still seeing it underneath the Horizontal server list.

Oooh, I see the issue. I forgot to remove the Windows Titlebar out of the required imports object array.
Give it a couple minutes to rebuild and it should be good to go.

Okay that problem is fixed! However it is now clear for some reason for instance https://www.youtube.com/watch?v=JwHwIB1EkSU


Query parameter decoding not correctly handling plus signs

How I tripped on this: noticed that my actor names were coming into the LRS as {"name": "Mary+Sue"}, not {"name": "Mary Sue"} as I would have expected. I verified all other forms of decoding on either end of the situation, it's a specific problem with the native JS decodeURIComponent, and all modern browsers provide URLSearchParams instead to get more accurate behavior.  I propose PR #174 to address this.

Hi @gavbaa, thank you for raising the issue and providing a suitable fix. (Plus tests are always a bonus!)
It was originally implemented this way to cater for IE11 support, but I am very much in the mindset now of requiring projects to polyfill instead of the library catering for old tech.
I'll give this a review and get it released soon 😊

Fixed in #174, will be released shortly. Thanks again for the contribution!

Released in 1.2.1 🎉


feat: Make SPI service providers configurable from env variables

Description
Since the spi-e2e-setup script is being used in infra-deployments preview script, it would be nice to be able to have the SPI providers configured with actual values using environment variables.
Type of change

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration
Checklist:

[x] I have performed a self-review of my code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] I have added meaningful description with JIRA/GitHub issue key(if applicable), for example HASSuiteDescribe("STONE-123456789 devfile source")
[ ] I have updated labels (if needed)

/retest