mint_client_sanity flake: losing 1msat balance

 fedimint-test-all-wasm32-unknown-unknown-ci> 00:00:43 2024-05-02T20:26:22.963236Z  INFO mint_client_sanity: receiver info info={"federation_id":"7267eec085ffc1b2c3289a043741bc39e8036930679e71c387d9a1011b6eb151","network":"regtest","meta":{"federation_name":"testfed"},"total_amount_msat":19,"total_num_notes":10,"denominations_msat":{"1":5,"2":3,"4":2}}
fedimint-test-all-wasm32-unknown-unknown-ci> 00:00:43 thread 'main' panicked at modules/fedimint-mint-tests/src/bin/mint-client-sanity.rs:75:5:
fedimint-test-all-wasm32-unknown-unknown-ci> 00:00:43 assertion `left == right` failed
fedimint-test-all-wasm32-unknown-unknown-ci> 00:00:43   left: 19
fedimint-test-all-wasm32-unknown-unknown-ci> 00:00:43  right: 20
fedimint-test-all-wasm32-unknown-unknown-ci> 00:00:43 note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
fedimint-test-all-wasm32-unknown-unknown-ci> ## FAIL: mint_client_sanity

https://github.com/fedimint/fedimint/actions/runs/8929917822/job/24528829288
It could be related to another flake we've seen where we are sometimes losing a note.

Ping @elsirion for awareness.

> retry-success.sh ./scripts/tests/mint-client-sanity.sh

reproduces on my system after a long while

Interesting observation: if you restore the client from passphrase it has correct balance. So it's definitely the client that lost something.

there is Cmd::run_with_logging, you can't use out_string then

oh there is also FM_DEVIMINT_CMD_INHERIT_STDERR which will have stderr inherit

Yeah, I run with retry-success.sh env RUST_LOG=fm=debug,info FM_DEVIMINT_CMD_INHERIT_STDERR=1 ./scripts/tests/mint-client-sanity.sh, and also wrote a test in which a client remints over and over 1msat and it's the fastest way to trigger. Eventually the note disappears.

Argh. Yesterday I was able to repro 2 times, and today I'm running it for 2 hours already and nothing.

Ok, I think I've found something.
So one of the transactions was rejected:
2024-05-03T19:03:40.933046Z DEBUG task{name="state machine executor"}:state_machine_transition{operation_id=0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505}: fm::client::reactor: Finished executing state transition operation_id=0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505 terminal=true outcome=Inactive { dyn_state: OperationState { operation_id: OperationId(0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505), state: Rejected(64f6dfaeef2f6c4f50c800b888a0b52569f5bf957068580358bf0a170a7e0a5f, "The transaction had an invalid input") } }

and later the refund transactions seem to also get rejected:
2024-05-03T19:03:40.960222Z DEBUG task{name="state machine executor"}:state_machine_transition{operation_id=0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505}: fm::client::reactor: Executing state transition operation_id=0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505
2024-05-03T19:03:40.960325Z DEBUG task{name="state machine executor"}:state_machine_transition{operation_id=0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505}: fm::client::reactor: Finished executing state transition operation_id=0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505 terminal=true outcome=Inactive { dyn_state: Input(MintInputStateMachine { common: MintInputCommon { operation_id: OperationId(0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505), txid: 45b20314ea996f24e28df4737c3ed5467c2413dbfae2f96f124abccecbd35cd1, input_idx: 5 }, state: Error(MintInputStateError { error: "Refund transaction 64f6dfaeef2f6c4f50c800b888a0b52569f5bf957068580358bf0a170a7e0a5f was rejected" }) }) }
2024-05-03T19:03:40.960360Z DEBUG task{name="state machine executor"}: fm::client::reactor: State transition complete operation_id=0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505 outcome_active=false total=29
2024-05-03T19:03:40.960369Z DEBUG task{name="state machine executor"}:state_machine_transition{operation_id=0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505}: fm::client::reactor: Executing state transition operation_id=0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505
2024-05-03T19:03:40.960447Z DEBUG task{name="state machine executor"}:state_machine_transition{operation_id=0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505}: fm::client::reactor: Finished executing state transition operation_id=0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505 terminal=true outcome=Inactive { dyn_state: Input(MintInputStateMachine { common: MintInputCommon { operation_id: OperationId(0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505), txid: 45b20314ea996f24e28df4737c3ed5467c2413dbfae2f96f124abccecbd35cd1, input_idx: 4 }, state: Error(MintInputStateError { error: "Refund transaction dcca71e514a0bc6be10655b0b4f200164e60c0ab767c0980fd393f4a6b369eb4 was rejected" }) }) }
2024-05-03T19:03:40.960474Z DEBUG task{name="state machine executor"}: fm::client::reactor: State transition complete operation_id=0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505 outcome_active=false total=28
2024-05-03T19:03:40.960481Z DEBUG task{name="state machine executor"}:state_machine_transition{operation_id=0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505}: fm::client::reactor: Executing state transition operation_id=0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505
2024-05-03T19:03:40.960550Z DEBUG task{name="state machine executor"}:state_machine_transition{operation_id=0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505}: fm::client::reactor: Finished executing state transition operation_id=0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505 terminal=true outcome=Inactive { dyn_state: Input(MintInputStateMachine { common: MintInputCommon { operation_id: OperationId(0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505), txid: 45b20314ea996f24e28df4737c3ed5467c2413dbfae2f96f124abccecbd35cd1, input_idx: 3 }, state: Error(MintInputStateError { error: "Refund transaction 5bf1403a7b0e8023d0a6f940c59ad60904f9c81132a02baaa1310a0f8e7254ae was rejected" }) }) }
2024-05-03T19:03:40.960578Z DEBUG task{name="state machine executor"}: fm::client::reactor: State transition complete operation_id=0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505 outcome_active=false total=27
2024-05-03T19:03:40.960586Z DEBUG task{name="state machine executor"}:state_machine_transition{operation_id=0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505}: fm::client::reactor: Executing state transition operation_id=0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505
2024-05-03T19:03:40.960685Z DEBUG task{name="state machine executor"}:state_machine_transition{operation_id=0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505}: fm::client::reactor: Finished executing state transition operation_id=0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505 terminal=true outcome=Inactive { dyn_state: Output(MintOutputStateMachine { common: MintOutputCommon { operation_id: OperationId(0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505), out_point: OutPoint { txid: 5bf1403a7b0e8023d0a6f940c59ad60904f9c81132a02baaa1310a0f8e7254ae, out_idx: 0 } }, state: Aborted(MintOutputStatesAborted) }) }
2024-05-03T19:03:40.960712Z DEBUG task{name="state machine executor"}: fm::client::reactor: State transition complete operation_id=0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505 outcome_active=false total=26
2024-05-03T19:03:40.960719Z DEBUG task{name="state machine executor"}:state_machine_transition{operation_id=0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505}: fm::client::reactor: Executing state transition operation_id=0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505
2024-05-03T19:03:40.960825Z DEBUG task{name="state machine executor"}:state_machine_transition{operation_id=0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505}: fm::client::reactor: Finished executing state transition operation_id=0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505 terminal=true outcome=Inactive { dyn_state: Output(MintOutputStateMachine { common: MintOutputCommon { operation_id: OperationId(0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505), out_point: OutPoint { txid: 64f6dfaeef2f6c4f50c800b888a0b52569f5bf957068580358bf0a170a7e0a5f, out_idx: 0 } }, state: Aborted(MintOutputStatesAborted) }) }
2024-05-03T19:03:40.960851Z DEBUG task{name="state machine executor"}: fm::client::reactor: State transition complete operation_id=0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505 outcome_active=false total=25
2024-05-03T19:03:40.960858Z DEBUG task{name="state machine executor"}:state_machine_transition{operation_id=0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505}: fm::client::reactor: Executing state transition operation_id=0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505
2024-05-03T19:03:40.960960Z DEBUG task{name="state machine executor"}:state_machine_transition{operation_id=0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505}: fm::client::reactor: Finished executing state transition operation_id=0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505 terminal=true outcome=Inactive { dyn_state: Output(MintOutputStateMachine { common: MintOutputCommon { operation_id: OperationId(0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505), out_point: OutPoint { txid: 927ffd0c048e358e7ce70a623525a51a42e55a8e4611658b74a5816fd731cfe3, out_idx: 0 } }, state: Aborted(MintOutputStatesAborted) }) }
2024-05-03T19:03:40.960987Z DEBUG task{name="state machine executor"}: fm::client::reactor: State transition complete operation_id=0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505 outcome_active=false total=24
2024-05-03T19:03:40.960995Z DEBUG task{name="state machine executor"}:state_machine_transition{operation_id=0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505}: fm::client::reactor: Executing state transition operation_id=0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505
2024-05-03T19:03:40.961096Z DEBUG task{name="state machine executor"}:state_machine_transition{operation_id=0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505}: fm::client::reactor: Finished executing state transition operation_id=0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505 terminal=true outcome=Inactive { dyn_state: Output(MintOutputStateMachine { common: MintOutputCommon { operation_id: OperationId(0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505), out_point: OutPoint { txid: dcca71e514a0bc6be10655b0b4f200164e60c0ab767c0980fd393f4a6b369eb4, out_idx: 0 } }, state: Aborted(MintOutputStatesAborted) }) }
2024-05-03T19:03:40.961124Z DEBUG task{name="state machine executor"}: fm::client::reactor: State transition complete operation_id=0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505 outcome_active=false total=23
2024-05-03T19:03:40.961131Z DEBUG task{name="state machine executor"}:state_machine_transition{operation_id=0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505}: fm::client::reactor: Executing state transition operation_id=0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505
2024-05-03T19:03:40.961226Z DEBUG task{name="state machine executor"}:state_machine_transition{operation_id=0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505}: fm::client::reactor: Finished executing state transition operation_id=0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505 terminal=true outcome=Inactive { dyn_state: Output(MintOutputStateMachine { common: MintOutputCommon { operation_id: OperationId(0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505), out_point: OutPoint { txid: 057bc08733c1467c2ce734640315cd33789eae17d8011f7dace76c931ca50612, out_idx: 0 } }, state: Aborted(MintOutputStatesAborted) }) }
2024-05-03T19:03:40.961251Z DEBUG task{name="state machine executor"}: fm::client::reactor: State transition complete operation_id=0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505 outcome_active=false total=22
2024-05-03T19:03:40.961258Z DEBUG task{name="state machine executor"}:state_machine_transition{operation_id=0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505}: fm::client::reactor: Executing state transition operation_id=0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505
2024-05-03T19:03:40.961326Z DEBUG task{name="state machine executor"}:state_machine_transition{operation_id=0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505}: fm::client::reactor: Finished executing state transition operation_id=0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505 terminal=true outcome=Inactive { dyn_state: Input(MintInputStateMachine { common: MintInputCommon { operation_id: OperationId(0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505), txid: 45b20314ea996f24e28df4737c3ed5467c2413dbfae2f96f124abccecbd35cd1, input_idx: 0 }, state: Error(MintInputStateError { error: "Refund transaction e7bf13a0153efadc1f6224ad87120d05d8bfc02fc2605823e7b4fa580d62eaa9 was rejected" }) }) }
2024-05-03T19:03:40.961354Z DEBUG task{name="state machine executor"}: fm::client::reactor: State transition complete operation_id=0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505 outcome_active=false total=21
2024-05-03T19:03:40.961360Z DEBUG task{name="state machine executor"}:state_machine_transition{operation_id=0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505}: fm::client::reactor: Executing state transition operation_id=0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505
2024-05-03T19:03:40.961458Z DEBUG task{name="state machine executor"}:state_machine_transition{operation_id=0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505}: fm::client::reactor: Finished executing state transition operation_id=0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505 terminal=true outcome=Inactive { dyn_state: Output(MintOutputStateMachine { common: MintOutputCommon { operation_id: OperationId(0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505), out_point: OutPoint { txid: e7bf13a0153efadc1f6224ad87120d05d8bfc02fc2605823e7b4fa580d62eaa9, out_idx: 0 } }, state: Aborted(MintOutputStatesAborted) }) }
2024-05-03T19:03:40.961485Z DEBUG task{name="state machine executor"}: fm::client::reactor: State transition complete operation_id=0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505 outcome_active=false total=20
2024-05-03T19:03:40.961493Z DEBUG task{name="state machine executor"}:state_machine_transition{operation_id=0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505}: fm::client::reactor: Executing state transition operation_id=0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505
2024-05-03T19:03:40.961561Z DEBUG task{name="state machine executor"}:state_machine_transition{operation_id=0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505}: fm::client::reactor: Finished executing state transition operation_id=0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505 terminal=true outcome=Inactive { dyn_state: Input(MintInputStateMachine { common: MintInputCommon { operation_id: OperationId(0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505), txid: 45b20314ea996f24e28df4737c3ed5467c2413dbfae2f96f124abccecbd35cd1, input_idx: 2 }, state: Error(MintInputStateError { error: "Refund transaction 057bc08733c1467c2ce734640315cd33789eae17d8011f7dace76c931ca50612 was rejected" }) }) }
2024-05-03T19:03:40.961588Z DEBUG task{name="state machine executor"}: fm::client::reactor: State transition complete operation_id=0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505 outcome_active=false total=19
2024-05-03T19:03:40.961595Z DEBUG task{name="state machine executor"}:state_machine_transition{operation_id=0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505}: fm::client::reactor: Executing state transition operation_id=0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505
2024-05-03T19:03:40.961660Z DEBUG task{name="state machine executor"}:state_machine_transition{operation_id=0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505}: fm::client::reactor: Finished executing state transition operation_id=0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505 terminal=true outcome=Inactive { dyn_state: Input(MintInputStateMachine { common: MintInputCommon { operation_id: OperationId(0f60d05db2e0ae76f7411ec01823809ee7fc339bafb7d26848608b94aad7d505), txid: 45b20314ea996f24e28df4737c3ed5467c2413dbfae2f96f124abccecbd35cd1, input_idx: 1 }, state: Error(MintInputStateError { error: "Refund transaction 927ffd0c048e358e7ce70a623525a51a42e55a8e4611658b74a5816fd731cfe3 was rejected" }) }) }

The reason why we might be missing why the transaction rejection was missed on a higher level is using reissue --no-wait and then dev wait-completion.
However it's unclear why would it get rejected in the first place (possibly we're trying to spend the output too fast?), and then why would refund fail (something retried?).
I'm guessing

With more logging #5206:
e5e23}: fm::client::reactor: Finished executing state transition operation_id=372704d93438d666c8a84d3084a15a99c111d8b6fed204b8e61c3dfe9a3e5e23 terminal=true outcome=In
active { dyn_state: OperationState { operation_id: OperationId(372704d93438d666c8a84d3084a15a99c111d8b6fed204b8e61c3dfe9a3e5e23), state: Rejected(0x6d8df69ecbdd3daf022
c10d422b071a0352891ba179df6a2c1fdf3628f147ac9, "The transaction had an invalid input: 1-The note is already spent") } }

and then:
2024-05-03T23:33:28.356161Z  WARN task{name="state machine executor"}:state_machine_transition{operation_id=372704d93438d666c8a84d3084a15a99c111d8b6fed204b8e61c3dfe9a3
e5e23}: fm::client::module::mint: Refund transaction rejected. Notes probably lost. err=The transaction had an invalid input: 1-The note is already spent refund_txid=1
1ea6763ec3feeec06ca1c0c28232410b90e8e28b23ecda42679b5a4be957a3a

OK, makes sense. Input was already spent, trying to refund it fails the same way.
Now ... why is it spent?

How am I going to figure out at which stage the input became spent...
It could be that receiver on spend somehow produced a faulty note.
Or reissue either spent a note and somehow left it as spendable, or produced an already spent note.

@dpc I wasn't sure if the this is the same root issue that caused this wasm flake, but feel free to close as a dupe if you think it is.
https://github.com/fedimint/fedimint/issues/5210

dev call: dpc and eric both can reproduce (but it takes hours to hit)

OK, so I think I have a clue:
We submit a tx 35b5c2d3d99215baae9f71...:
2024-05-07T06:13:10.251926Z  WARN fm::client: Submitting transaction txid=35b5c2d3d99215baae9f71ce0d2dbaf8d194ddb366c32a9364b4121492c89a36 transaction=Transaction { inputs: [V0(MintInputV0 { amount: Amount { msats: 1 }, note: Note {
 nonce: Nonce(PublicKey(bdc6118ac1e9eb8256280a64d0fed75055a539ca0e03f22f182094ed517b3a5b2c599be048b101f731da233b60ebe78e266ba231facc1d029ff5ccdd4fed688b)), signature: Signature(G1Affine { x: 0x070263239e67d52a687e9ea9f278e496568a645
051a1988661bd1049330f892ac2efb170be1d25190c771bd8eec1f242, y: 0x138c65c09827904b677760e9aa63827b4ac94bf7266af5a8a16bd6c9303ec9a15e8b774c5f2a317366f7779676d7e555, infinity: Choice(0) }) } })], outputs: [V0(MintOutputV0 { amount: Amou
nt { msats: 1 }, blind_nonce: BlindNonce(BlindedMessage(G1Affine { x: 0x08ad882bd7dba54672ac3f3b1959053cbea841a9138813233d888641d0430e176cd358ff31b5229b5c9fe49d62aebe9d, y: 0x08a00ad7b821de685f43f005a60966ee43129ab8b685c0762ccb962b5
77069391b9434e463d6c5bf9e34e166cbb9e1eb, infinity: Choice(0) })) })], nonce: [94, 92, 27, 204, 71, 80, 103, 154], signatures: NaiveMultisig([Signature(85f260e71f598539b74bac6b377ab3f6414d9f50bef5e1ae616359f5d2aea1fa965eb9de7e4e5f67e
7d3139f37a6de118254fbbd624eaf96f611aa4515983108)]) }

The output state machine says it succeeded:
2024-05-07T06:13:10.494432Z DEBUG task{name="state machine executor"}:state_machine_transition{operation_id=611aee7b8f9bb7772fbdcac1139d91eaa679f9776c1fe51907ebe278023cc474}: fm::client::reactor: Finished executing state transition
operation_id=611aee7b8f9bb7772fbdcac1139d91eaa679f9776c1fe51907ebe278023cc474 terminal=true outcome=Inactive { dyn_state: Output(MintOutputStateMachine { common: MintOutputCommon { operation_id: OperationId(611aee7b8f9bb7772fbdcac11
39d91eaa679f9776c1fe51907ebe278023cc474), out_point: OutPoint { txid: 0x35b5c2d3d99215baae9f71ce0d2dbaf8d194ddb366c32a9364b4121492c89a36, out_idx: 0 } }, state: Succeeded(MintOutputStatesSucceeded { amount: Amount { msats: 1 } }) })
 }

But the input state machine fails:
2024-05-07T06:13:10.494471Z DEBUG task{name="state machine executor"}: fm::client::reactor: State transition complete operation_id=611aee7b8f9bb7772fbdcac1139d91eaa679f9776c1fe51907ebe278023cc474 outcome_active=false total=7
2024-05-07T06:13:10.494521Z DEBUG task{name="state machine executor"}: fm::client::reactor: Started new active state machine. operation_id=85e9b85c0b8d6839ea54e8c019a3cd476c5c4da50997923b4f5d45d41c95694b total=8 transitions_num=2
2024-05-07T06:13:10.494527Z DEBUG task{name="state machine executor"}:state_machine_transition{operation_id=611aee7b8f9bb7772fbdcac1139d91eaa679f9776c1fe51907ebe278023cc474}: fm::client::reactor: Executing state transition operation
_id=611aee7b8f9bb7772fbdcac1139d91eaa679f9776c1fe51907ebe278023cc474
2024-05-07T06:13:10.494592Z DEBUG task{name="state machine executor"}:state_machine_transition{operation_id=611aee7b8f9bb7772fbdcac1139d91eaa679f9776c1fe51907ebe278023cc474}: fm::client::reactor: Finished executing state transition
operation_id=611aee7b8f9bb7772fbdcac1139d91eaa679f9776c1fe51907ebe278023cc474 terminal=true outcome=Inactive { dyn_state: OperationState { operation_id: OperationId(611aee7b8f9bb7772fbdcac1139d91eaa679f9776c1fe51907ebe278023cc474),
state: Rejected(0x35b5c2d3d99215baae9f71ce0d2dbaf8d194ddb366c32a9364b4121492c89a36, "The transaction 35b5c2d3d99215baae9f71ce0d2dbaf8d194ddb366c32a9364b4121492c89a36 had an invalid input at index 0: 1-The note is already spent (Nonc
e(PublicKey(bdc6118ac1e9eb8256280a64d0fed75055a539ca0e03f22f182094ed517b3a5b2c599be048b101f731da233b60ebe78e266ba231facc1d029ff5ccdd4fed688b)))") } }

Output state machines wait only for AWAIT_OUTPUT_OUTCOME_ENDPOINT. But input state machines await on on transaction_update_stream.
Which suggests that our idempotency of tx submission is borked. Or I messed up something and confused myself.
@elsirion

:thinking:
diff --git a/fedimint-server/src/net/api.rs b/fedimint-server/src/net/api.rs
index 8fc49260f1..4a1cccfcc4 100644
--- a/fedimint-server/src/net/api.rs
+++ b/fedimint-server/src/net/api.rs
@@ -114,11 +114,10 @@ impl ConsensusApi {

         debug!(%txid, "Received mint transaction");

+        // Create read-only DB tx so that the read state is consistent
+        let mut dbtx = self.db.begin_transaction_nc().await;
         // we already processed the transaction before
-        if self
-            .db
-            .begin_transaction()
-            .await
+        if dbtx
             .get_value(&AcceptedTransactionKey(txid))
             .await
             .is_some()
@@ -126,9 +125,6 @@ impl ConsensusApi {
             return Ok(txid);
         }

-        // Create read-only DB tx so that the read state is consistent
-        let mut dbtx = self.db.begin_transaction_nc().await;
-
         // We ignore any writes, as we only verify if the transaction is valid here
         dbtx.ignore_uncommitted();

Great catch! That could very well be the culprit, starting the test script on my server too.

#5226 but I'm still able to repro, so either red herring or didn't help. Will keep debugging.

Taking a step back https://github.com/fedimint/fedimint/issues/5195#issuecomment-2097657170 doesn't explain losing money imo since the output was processed successfully. There's probably another anomaly that causes the loss, what I'd expect is that output processing failed since you noticed earlier that money reappears when recovering from seed.

The good part is that I'm improving the debugability of note tracing in the client.

Why does everything looks always OK?
The note 523bf09c2d77add20b0a957477e6 get created as an output to reissue of other notes in the sender:
2024-05-08T07:24:59.365868Z DEBUG task{name="state machine executor"}:state_machine_transition{operation_id=fd6e3bb89c30af2526bd513e57a16af5a2b6d08d14682b35996eb1ebd43e14a4}: fm::client::module::mint: Adding new note from transactio
n output amount=1 msat note=SpendableNote { signature: Signature(G1Affine { x: 0x0fb47ea115f2ec53a20bdc67350b31b4d6ce475da8f2440b8dd466e824ecb27c081b22eb7e6c5b08981d977af6666a31, y: 0x15ab63eb442bdc80fea2c00d78ffbc6fb83ab7d1d44b1ffd
813d88eba0bd4e204d500b61c8c105349d155b26615ebc36, infinity: Choice(0) }), spend_key: KeyPair(#19268168bfca36e0), nonce: Nonce(PublicKey(523bf09c2d77add20b0a957477e6d95a2d3a00a014a225ea31bf50be13da876f08d9e6523b895912c00c7c4ad8c32f60
c1623ea6437d0ddc427051ed76b5a87e)) }

Then it got spent as OOB as AgEEMOTxLABTAQEBr7R+oRXy7FOiC9xnNQsxtNbOR12o8kQLjdRm6CTssnwIGyLrfmxbCJgdl3r2ZmoxoRvISk5SvtAr8owWHSf6NjNOKLOI7GwA4r/WdTXO+U8=:
2024-05-08T07:24:59.694719Z DEBUG fm::client::module::mint: Spending note as oob amount=1 msat note=SpendableNote { signature: Signature(G1Affine { x: 0x0fb47ea115f2ec53a20bdc67350b31b4d6ce475da8f2440b8dd466e824ecb27c081b22eb7e6c5b0
8981d977af6666a31, y: 0x15ab63eb442bdc80fea2c00d78ffbc6fb83ab7d1d44b1ffd813d88eba0bd4e204d500b61c8c105349d155b26615ebc36, infinity: Choice(0) }), spend_key: KeyPair(#19268168bfca36e0), nonce: Nonce(PublicKey(523bf09c2d77add20b0a9574
77e6d95a2d3a00a014a225ea31bf50be13da876f08d9e6523b895912c00c7c4ad8c32f60c1623ea6437d0ddc427051ed76b5a87e)) }
...
2024-05-08T07:24:59.708802Z  INFO mint_client_sanity: sender adding to notes for the receiver info={"federation_id":"30e4f12c5b44533480e725c920356f1fa8a12d26387eeef9775fec701053b33f","network":"regtest","meta":{"federation_name":"te
stfed"},"total_amount_msat":9999988,"total_num_notes":67,"denominations_msat":{"1":2,"2":9,"4":8,"8":2,"16":5,"32":3,"64":2,"128":2,"256":2,"512":3,"1024":3,"2048":2,"4096":3,"8192":2,"16384":2,"32768":3,"65536":2,"131072":2,"262144
":2,"524288":3,"1048576":3,"2097152":2}} notes=AgEEMOTxLABTAQEBr7R+oRXy7FOiC9xnNQsxtNbOR12o8kQLjdRm6CTssnwIGyLrfmxbCJgdl3r2ZmoxoRvISk5SvtAr8owWHSf6NjNOKLOI7GwA4r/WdTXO+U8=

Then the receiver reissues it in tx 77bde4dec3fa430c67d0c34c0f030f810f57d6df89515d439534895f285c89ff :
2024-05-08T07:25:05.255727Z DEBUG fm::devimint: > fedimint-cli --data-dir=/tmp/devimint-657359-466/clients/receiver-0 reissue --no-wait AgEEMOTxLABTAQEBr7R+oRXy7FOiC9xnNQsxtNbOR12o8kQLjdRm6CTssnwIGyLrfmxbCJgdl3r2ZmoxoRvISk5SvtAr8owW
HSf6NjNOKLOI7GwA4r/WdTXO+U8=
...
2024-05-08T07:25:05.349580Z  WARN fm::client: Submitting transaction txid=77bde4dec3fa430c67d0c34c0f030f810f57d6df89515d439534895f285c89ff transaction=Transaction { inputs: [V0(MintInputV0 { amount: Amount { msats: 1 }, note: Note {
 nonce: Nonce(PublicKey(523bf09c2d77add20b0a957477e6d95a2d3a00a014a225ea31bf50be13da876f08d9e6523b895912c00c7c4ad8c32f60c1623ea6437d0ddc427051ed76b5a87e)), signature: Signature(G1Affine { x: 0x0fb47ea115f2ec53a20bdc67350b31b4d6ce475
da8f2440b8dd466e824ecb27c081b22eb7e6c5b08981d977af6666a31, y: 0x15ab63eb442bdc80fea2c00d78ffbc6fb83ab7d1d44b1ffd813d88eba0bd4e204d500b61c8c105349d155b26615ebc36, infinity: Choice(0) }) } })], outputs: [V0(MintOutputV0 { amount: Amou
nt { msats: 1 }, blind_nonce: BlindNonce(BlindedMessage(G1Affine { x: 0x0e14a6c2bdc546dd90a2956a461c88279fb092069a21198b49ca43c6ac6584e641583869480a4886270af20b1b1f2251, y: 0x058020fec2fc4f03813c3e9878585ac4954d7013ab57d0025c0c4f4a4
7202ab2404c3ced021d8956598d655fce827161, infinity: Choice(0) })) })], nonce: [203, 129, 244, 181, 108, 225, 25, 238], signatures: NaiveMultisig([Signature(07c5350d92b169bf7f69ce612ef823532e8362f1f3fc4538d37855e50f9ead8a3cad9c40198cd
3f6ffd3369ee2938af56073efac21e4f457e363f25782e2f845)]) }

but it fails:
2024-05-08T07:25:05.668673Z DEBUG task{name="state machine executor"}:state_machine_transition{operation_id=449a45443466a6ec5b0ef60d60b23b274eed9dd1f3ab2d13fef7c3d6cd5ea1b4}: fm::client::reactor: Finished executing state transition
operation_id=449a45443466a6ec5b0ef60d60b23b274eed9dd1f3ab2d13fef7c3d6cd5ea1b4 terminal=true outcome=Inactive { dyn_state: OperationState { operation_id: OperationId(449a45443466a6ec5b0ef60d60b23b274eed9dd1f3ab2d13fef7c3d6cd5ea1b4),
state: Rejected(0x77bde4dec3fa430c67d0c34c0f030f810f57d6df89515d439534895f285c89ff, "The transaction 77bde4dec3fa430c67d0c34c0f030f810f57d6df89515d439534895f285c89ff had an invalid input at index 0: 1-The note is already spent (Nonc
e(PublicKey(523bf09c2d77add20b0a957477e6d95a2d3a00a014a225ea31bf50be13da876f08d9e6523b895912c00c7c4ad8c32f60c1623ea6437d0ddc427051ed76b5a87e)))")

Adding more server-side logging to track when exactly this note got spent.

It seems the issue is server side:
First the server does the process_input check, without commiting:
2024-05-10T03:56:55.091881Z DEBUG fm::module::mint: Marking note as spent nonce=02143e021511766d280b0cacdd15a2f19dd6851aef1d21e055299b5fc9ecc8e8a9

Then the consensus processes the tx and everything commits:
2024-05-10T03:56:55.196421Z DEBUG fm::module::mint: Marking note as spent nonce=02143e021511766d280b0cacdd15a2f19dd6851aef1d21e055299b5fc9ecc8e8a9

But on the next submit_tx with the exact same transaction:
2024-05-10T03:56:55.319340Z DEBUG fm::module::mint: Marking note as spent nonce=02143e021511766d280b0cacdd15a2f19dd6851aef1d21e055299b5fc9ecc8e8a9
2024-05-10T03:56:55.319368Z DEBUG fm::module::mint: Note was already spent nonce=02143e021511766d280b0cacdd15a2f19dd6851aef1d21e055299b5fc9ecc8e8a9

which seems the code failed to detect that transaction was processed, but somehow (in the same dbtx) sees the note was marked as spent, which happened on the same write tx when tx was processed by the consensus.
Either we experience dirty-reads, or we somehow failed to write accepted tx, but marked inputs as spent, or something very weird is going on.

@elsirion  @m1sterc001guy
OMG.
There's nothing guaranteeing that the db view does not change from under us, since we are not running with "serializable" isolation level. Just read through https://stackoverflow.com/a/4036063/134409 .
By default we are only getting "read committed". We could update to "repeatable reads", but it doesn't really matter here.
The sequence is as follows:

submit checks if the tx is accepted in the db &AcceptedTransactionKey, it isn't
process consensus commits its transaction inserting both &AcceptedTransactionKey and NonceKey
submit checks if the nonce is spent, and it is

For some reason I never really internalized that the db can't guarantee consistent read-view of the database in the transaction. Only after seeing it step by step in the log now, with all the needed data it clicked.
Please confirm if this makes sense, or maybe it's just too late and I lost the plot.
The simple solution here, would be to change the code to check for AcceptedTransactionKey both before processing on submit_transaction and after, if there was any error.

@dpc @m1sterc001guy I think we (at least me) always operated under the assumption that RocksDB provides snapshot isolation :grimacing: Didn't we even have a whole test suite for that? Imo we should just get the closest to that we can, dealing with lower isolation levels manually isn't worth it at all.

Yeah I believe we have snapshot isolation. When creating the rocksdb transaction, we set the snapshot like this. Here was the commit.
async fn begin_transaction<'a>(&'a self) -> RocksDbTransaction {
        let mut optimistic_options = OptimisticTransactionOptions::default();
        optimistic_options.set_snapshot(true);

        let mut write_options = WriteOptions::default();
        // Make sure we never lose data on unclean shutdown
        write_options.set_sync(true);

        let mut rocksdb_tx =
            RocksDbTransaction(self.0.transaction_opt(&write_options, &optimistic_options));
        rocksdb_tx
            .set_tx_savepoint()
            .await
            .expect("setting tx savepoint failed");

        rocksdb_tx
    }

As Eric pointed out, we have a set of tests for detecting the other anomalies present at lower isolation levels.

Fixing that would probably require serializability?

Yes I believe so. Snapshot isolation provides a read-consistent view of the database at the time of the transaction creation. Since in your example above, the data that tx1 is operating over changes during the transaction, it will not have an up to date version of key1 at step 4.
I've been moving this week so I haven't been able to keep up with this whole thread, but do we think our isolation expectations have changed because of this bug? Perhaps we can discuss on the dev call.

BTW. https://github.com/fedimint/fedimint/issues/5271 is another place confirm something is of. Otherwise - how could you in the same db tx selected a note, and then fail to return it when you're deleting it.
Thought this sounds like even repeatable reads not working.

Calling set_snapshot will not affect the version of Data returned by get methods.

https://docs.rs/rocksdb/latest/rocksdb/struct.TransactionOptions.html#method.set_snapshot
This looks sus to me.
@m1sterc001guy

We've found the issue with @m1sterc001guy ! :partying_face: :tada: :taco:

@m1sterc001guy @elsirion https://github.com/fedimint/fedimint/pull/5279


Not have the Field Filled Out

Is there a way to just have the field not be filled out? I have added this to my code, but the button still seems to be filling out the field even though it shouldn't be. Making this button entirely useless. This seems like a bug in the plugin?
formContext.data.entity.attributes.get('new_pcf_button').setValue(null);
formContext.data.entity.attributes.get('new_pcf_button').setSubmitMode("never");

Never mind finally worked.


Unexpected type error for accessing known existent members in a ReadonlyArray with noUncheckedIndexedAccess

Bug Report
🔎 Search Terms
non-existent members, ReadonlyArray
Referenced https://github.com/microsoft/TypeScript/issues/13778#issuecomment-950196264 .
🕗 Version & Regression Information
noUncheckedIndexedAccess was introduced in 4.1.
⏯ Playground Link
Playground link with relevant code
💻 Code

const x: ReadonlyArray<number> = [0, 1, 2];
const c: number = x[0]; /* Unexpected `Type 'number | undefined' is not assignable to type 'number'.
  Type 'undefined' is not assignable to type 'number'.`
*/

🙁 Actual behavior
It returned number | undefined type for all members of the ReadonlyArray.
🙂 Expected behavior
Expected known members to be typed number.

I think this is expected, x[10] is also number | undefined since your type doesn't retain length information.  [0,1,2] as const retains the length information and therefore x[0] is typed as number.

That's right, ReadOnlyArray is just a generic array type and doesn't include the length of the array as part of the type, nor does the control flow analyzer track that information (hence why noUncheckedIndexedAccess is optional).  If you had gotten this array as an argument to a function call, for example, nothing in its typing says those elements have to be present, so the error is correct.

ReadOnly also doesn't mean immutable. You could get back an array from a function that still is able to mutate it later.
The behavior you expect can be achieved from
const x = [0, 1, 2] as const;
const c: number = x[0];


Fix typo - "Databas" -> "Database"

What this PR solves / how to test
Fixes typo in d1 execute
Author has addressed the following

Tests

[ ] TODO (before merge)
[ ] Included
[x] Not necessary because: no existing test and small typo fix

E2E Tests CI Job required? (Use "e2e" label or ask maintainer to run separately)

[ ] I don't know
[ ] Required / Maybe required
[x] Not required because: no e2e tests covering

Changeset (Changeset guidelines)

[ ] TODO (before merge)
[x] Included
[ ] Not necessary because:

Public documentation

[ ] TODO (before merge)
[ ] Cloudflare docs PR(s):
[x] Not necessary because: just a typo fix

Thanks!


replace deprecated spdx-license-list package by autogenerated module

The spdx-license-list package is deprecated (#131).
This PR adds a script that generates a module from the latest spdx license list https://raw.githubusercontent.com/spdx/license-list-data/master/json/licenses.json
I think this is easier than creating an extra package and maintaining it.

geez - 5000 lines of dict :/ maybe this mandates putting it somewhere separate after all - what do you think @FynnBe?

as discussed offline: we'll commit the raw json instead.
(sure it's a lot of lines, but only text after all 😉)

also, just to keep a record: we decided to only allow for licenceIDs to be specified (for simplicity and to avoid typos, also @k-dominik : conda-forge does the same)

After a discussion with @FynnBe and @constantinpape we decided to include the .json file from the official source, and load it dynamically instead of ingesting it as a python file via a script.


Lite: Softmax Op Refactored

1:> Code refactored to become more compact and reusable
2:> Error message rectified

@nutsiepully : Please help conclude this PR, it is been long pending, TIA!

@nutsiepully : Please help review the PR, TIA!

@nutsiepully : please conclude this PR, TIA!

@nutsiepully , @jianlijianli : Gentle Reminder!


Sparse checkout option for zopen build

I'm working with the z/OS Tools & Toys repo, which is very large due to a lot of non-code content that is stored there.  All the code is stored in a single subdirectory, so it would be nice if I could tell zopen to only checkout the contents of that directory.  It would speed up the clone process as well as reducing disk space on my build system.

I propose inventing new zopen vars for this, ZOPEN_STABLE_SUBDIRS/ZOPEN_STABLE_SUBDIRS:
export ZOPEN_STABLE_URL="https://github.com/git/git"
export ZOPEN_STABLE_DEPS="curl git make m4 perl autoconf automake help2man texinfo xz zlib openssl expat gettext gzip tar coreutils zoslib diffutils ncurses bash sed libpcre2"
export ZOPEN_STABLE_SUBDIRS="dir1 dir2"

If set, it zopen build would run these commands:
git clone -n --depth=1 --filter=tree:0 $repo $dir
cd $dir
git sparse-checkout set --no-cone $ZOPEN_STABLE_SUBDIRS
git checkout

Adding a few links on the topic probably for my own future reference.
https://github.blog/2020-01-17-bring-your-monorepo-down-to-size-with-sparse-checkout/
https://github.blog/2020-01-13-highlights-from-git-2-25/#sparse-checkout-management-made-easy
https://github.blog/2021-11-10-make-your-monorepo-feel-small-with-gits-sparse-index/
https://github.blog/2021-11-15-highlights-from-git-2-34/#sparse-index

The madness continues... there's something called 'partial clone' too 😵💫
https://github.blog/2020-12-21-get-up-to-speed-with-partial-clone-and-shallow-clone/


arduino-language-server: 0.7.4 -> 0.7.5

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://github.com/arduino/arduino-language-server/releases.
meta.description for arduino-language-server is: An Arduino Language Server based on Clangd to Arduino code autocompletion
meta.homepage for arduino-language-server is: https://github.com/arduino/arduino-language-server
meta.changelog for arduino-language-server is: https://github.com/arduino/arduino-language-server/releases/tag/0.7.5
Updates performed

Golang update

To inspect upstream changes
Impact

Checks done (click to expand)

built on NixOS
The tests defined in passthru.tests, if any, passed
found 0.7.5 with grep in /nix/store/8pbrcpifrmcig03lwi52klgz0afn2gqn-arduino-language-server-0.7.5
found 0.7.5 in filename of file in /nix/store/8pbrcpifrmcig03lwi52klgz0afn2gqn-arduino-language-server-0.7.5

Rebuild report (if merged into master) (click to expand)

1 total rebuild path(s)

1 package rebuild(s)

First fifty rebuilds by attrpath
arduino-language-server

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/8pbrcpifrmcig03lwi52klgz0afn2gqn-arduino-language-server-0.7.5 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A arduino-language-server https://github.com/r-ryantm/nixpkgs/archive/faf2e6345043630fbd986a6773e026743db93549.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/8pbrcpifrmcig03lwi52klgz0afn2gqn-arduino-language-server-0.7.5
ls -la /nix/store/8pbrcpifrmcig03lwi52klgz0afn2gqn-arduino-language-server-0.7.5/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

1 package built:

arduino-language-server

Maintainer pings
cc @BattleCh1cken for testing.

Result of nixpkgs-review pr 250871 run on x86_64-linux 1

1 package built:

arduino-language-server

Builds and runs on x86_64-linux.


Failed to click on warehouse in act2

Details
Failed to click on warehouse in act2 continuously
Version
6242d17
Logs
1

same here act 2 is basically broken because the stash cannot be accessed and it creates a loop of walking around until the game times out then it repeats on new game

it's working fine for me, please try latest commit


Question about read aggregation by barcode

Hi!
I am attempting to run TRUST4 on a single-cell V(D)J data set and it is running rather slowly, so I was peeking under the hood to see if there were any easy opportunities for parallelization. I noticed that even in the single-cell case, the main call to TRUST4 assembles a full list of reads, roughly annotates them, and then sorts them, but when we have barcodes, don't we want to group them by barcode and then sort and process? I have 250 million reads, so this is quite slow as a serial process, but if I were to split my inputs by barcode and run TRUST4 in parallel on each of them, I shouldn't have an issue, correct?
Basically, is there any sharing of information between reads before they are separated by barcodes that I need to watch for, or can I treat each barcode as a bulk sample and process them that way?
Thanks for any insight!

For what it's worth, here is my script in case I am doing something dumb with any of the arguments. I find that the algorithm starts to hang in the phase of TRUST4 where it is reading in and processing the reads.
#!/usr/bin/bash

donor_id="CD001"
exp_id="control_5cancer"

gdTCR_dir="/mnt/data0/TCR_Discovery_Platform/SequencingData/gdTCR"
ref_dir="/mnt/data0/References"

run-trust4 \
    -u $gdTCR_dir/fastqs/merged_${donor_id}_${exp_id}_R2.fastq.gz \
    -f $ref_dir/hg38_bcrtcr.fa \
    --ref $ref_dir/human_IMGT+C.fa \
    -o ${donor_id}_${exp_id} \
    --od $gdTCR_dir/output/TRUST4/${exp_id} \
    -t 24 \
    --barcode $gdTCR_dir/fastqs/merged_${donor_id}_${exp_id}_R1.fastq.gz \
    --barcodeWhitelist $ref_dir/737K-august-2016.txt \
    --UMI $gdTCR_dir/fastqs/merged_${donor_id}_${exp_id}_R1.fastq.gz \
    --readFormat bc:0:15,um:16:-1,r1:0:-1 \
    --outputReadAssignment \
    --stage 1 \

Yes, that's actually a function I'm planning to work on for a while, but need to find a large block of time to implement that.
For your implementation, I think the proper way is to run fastq-extract first, that will also generate error-corrected barcodes. Then you can split the reads based on barcode bunches, and run "run-trust4" with the option "--noExtraction".
Since your data is VDJ amplified, the "--repseq" option may help with the speed without hacking the multithreading.

Amazing! Thank you!
It seemed like the sort of next step to take this program, so I'll take a look and see if I can contribute anything as well!
Thanks for the advice on the flags, I'll give them a try.


South African JSE Missing

Please could you add the South African JSE market to the list?

Added South Africa screener
https://tvdb.brianthe.dev/


Consolidates endpoints and schema with latest meetings on UC-Post-Partum-001-Registration_PostPartum_and_InfantCare

Closes https://govstack-global.atlassian.net/browse/CON-21?atlOrigin=eyJpIjoiNzA0ODAyZGJkYWI5NGJlMGFhMjAyZDUzODk4OWNjM2UiLCJwIjoiaiJ9

Feedback from call:
Decision made to change the sequence for signatures to avoid storing "drafts" that need to be cleaned up later. This will require a redesign of endpoints and schemas.

Will synchronize with the latest version of the spreadsheet and merge this.


[Core] Design discussion: Refactor component/backend organisation

The current implementations grew out from experimental design around multi-backend support. The get_backend() checks are undesirable and do not make for readable implementations.
This issue is meant to collect design improvements.
Proposal 1: Components will be reorganised into a base component and backend-specific sub-classes.
A package tensorflow_components/pytorch_components will mirror the folder structure of the base components and contain the specific implementations.
Advantages: Avoid backend checks in implementations, clearly separates backends from interfaces
Disadvantages: Multiplies number of components, potentially irritating to see mirrored folder structure of
components/
       memories/base_memory
       tf_components/memories/tf_memory
       pytorch_components/memories/pytorch_memory

versus keeping everything in one folder memories (which would make imports more difficult for the package:
components/
       memories/
            base_memory
            tf_memory
            pytorch_memory

A possible solution avoiding all extra backend-prefixed classes (such as TfMemory) would be to use dynamic imports. So the folder structure would be something like the first one suggested above:
E.g.
components/
    memories/
        memory.py (<- some abstract base class, no graph_fns)
        prioritized_replay.py (<- will dynamically import graph_fns located  in `tf(or pytorch)/prioritized_replay.py`)
        tf/
            prioritized_replay.py (<- contains only tf graph_fns, no classes)
        pytorch/
            prioritized_replay.py (<- contains only pytorch graph_fns, no classes)

We would then only need one extra line per backend-agnostic Component-class file (e.g. prioritized_replay.py in the main memories directory):
_BACKEND_MOD = importlib.import_module(
    "rlcore.components.memories."+get_backend()+".prioritized_replay"
)

All graph_fns inside the main class file would look like this:
\@graph_fn
def _graph_fn_insert(self, records):
    return _BACKEND_MODE.insert_function(records)

The actual backend-dependent implementation of insert_function would be in the tf or pytorch folders.
This way, no import statements would have to be conditional anymore in any files. And we would not need any backend-specific sub-classing, which could potentially mess up the class hierarchy.

Counterpoint:

The above would not solve variable creation which is backend-dependent, non-static / relies on inheritance, and would still need backend checks in  prioritized_replay.py  in this case. Otherwise one would have to significantly refactor all get variable methods.
Readability (Personal opinion): Would have to jump between graph fns and stubs instead of reading all relevant code in a single class which would be my expectation.


Fix CPU unit test failures on recent macOS ARM platforms

Fixes the macOS ARM CPU unit tests for https://github.com/AcademySoftwareFoundation/OpenColorIO/issues/1946.
For the string comparison failures I add a utility function called StringFloatVecClose that converts the string values to float and compares with a threshold.
There is still a metal unit test failure happening on my machine, haven't looked into the cause of it.

Thank you Mark!  By the way, if the branch name contains "macarm" it will run the macARM CI Action.

@doug-walker FYI, GitHub now provides arm runners for free for OSS projects. See https://github.blog/changelog/2024-01-30-github-actions-introducing-the-new-m1-macos-runner-available-to-open-source/.

Yes, thank you @JeanChristopheMorinPerso!  I've been watching OIIO's journey on macARM and saw the CI WG notes as well. But no one has had time yet to update OCIO's Actions to use it.  I just wanted to make sure Mark was aware that we do currently have a paid Action he could use to test his work, if it's helpful.

I tried renaming my branch and it automatically closed this, I'll open an new PR


As a front-end developer, I want access to common tools.

What happened (or feature request):
Feature request.
What you expected to happen:
Many users in the role of site-builder, themer, and/or front-end developer are used to having access to tools (e.g. sass, less, gulp, grunt, compass, etc.) readily available. For local development, our images should support having these tools available.
Anything else do we need to know:
Some of these tools are a simple matter of installing the package by default. Others may have some additional configuration required to make them usable out of the gates.
Related source links or issues (like source JIRA issue):
Proposed Next Actions

[ ] Feedback from @tannerjfco would be appreciated given his extensive experience here on Drupal 7 and 8 with the front-end teams as well as the supporting infrastructure/tooling.
[ ] Get additional feedback from a handful of members in the Drupal and Wordpress community to get a better sense of nuance/detail.
[ ] Document the relevant packages and configurations needed.
[ ] Update Acceptance criteria.
[ ] Make this ticket actionable.

Closing because of the following:

No movement on this in over 3 months
Individual feature requests will likely move to other issues.


Make project creation optional in gcs to bq example

Next steps: break out resources in separate files, collapse resources into a single project, then write proper examples and tests.


Unbreak the build when WITH_OSL is OFF (default setting)

Inclusion of renderer/kernel/shading/closures.h should be guarded by APPLESEED_WITH_OSL macro because that header file includes OSL headers.
Lest us forget to test the builds against default settings. ;-)

Good catch, thanks!


feature: 支持接入点中 cityid 和 regionid 配置为空 (closed #46)

Codecov Report

Merging #66 (04e7f68) into V2.1.342rc (d0ff04a) will increase coverage by 0.00%.
The diff coverage is 75.00%.

@@             Coverage Diff             @@
##           V2.1.342rc      #66   +/-   ##
===========================================
  Coverage       58.92%   58.93%
===========================================
  Files             394      394
  Lines           25901    25901
===========================================
+ Hits            15263    15264    +1
+ Misses          10638    10637    -1

Impacted Files
Coverage Δ

apps/backend/views.py
56.83% <0.00%> (ø)

apps/node_man/models.py
74.03% <100.00%> (ø)

apps/node_man/serializers/ap.py
92.98% <100.00%> (ø)

apps/node_man/handlers/job.py
86.60% <0.00%> (+0.24%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d0ff04a...04e7f68. Read the comment docs.


New grading tasks: logical-and-or-not and lazy-evaluation

logical-and-or-not
A bunch of different boolean expressions are tested to cover as many error-cases as I was able to find/think of.
lazy-evaluation
Built on logical operators of course. The grading checks if basic boolean expressions still work followed by if lazy evaluation works by having endless loops in conditions which should be skipped if lazy evaluation was implemented successfully.

@CAS-ual-TY Awesome! Thanks a lot. Any clue why the docker build fails?

Yes. I made a trivial mistake which I believe is the cause of it. Lets see if it works now.

Yup, I was right. This is good to go!


Implement Stream to TcpStream conversion

Mostly needed for STARTTLS, see usage at https://github.com/duesee/imap-flow/pull/174/files#diff-3f7e738903cc8204a4e3b1c8ab4db5812898e6bb5ad8b6c7b36df42e29481a8fR197

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Details

0 of 3 (0.0%)  changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+6.4%) to 64.477%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

src/stream.rs
0
3
0.0%

Totals

Change from base Build<PHONE_NUMBER>:
6.4%

Covered Lines:
893

Relevant Lines:
1385

💛  - Coveralls

This is the only missing PR needed by imap-client, all other changes can be done directly on imap-client.

Talked with Jakob and we are not sure we want to expose this. On the other hand, I don't want to block you from getting things done.
Proposal: Can you put this behind the expose_stream feature? Then we merge and evaluate again at a later time.


Fix teensy patch and compiler

CI will fail until library is rebuild with the changes of this PR, so we are merging anyway

@mergify backport rolling humble


bootstrap-wizard inside bootstrap-modal

Hello,
This is not an issue, it could be classified as a question or an enhancement. I am planning on using this wizard and attaching the same inside the body class of bootstrap modal. Would like to know if its common to have a bootstrap wizard inside a modal for create operation (Create/Edit). If so has anyone come across any issues or potential recommendations to start with. Would like to know before i give it a try on possible issues or better alternatives to achieve the same.

I'll happily wished to know how it works as i'm trying to make it work in a modal, and it doesn't seem to reply. Changed a little like biskit adviced, don't seem to work.
Any clue ?

Worked on this issue for far too long now.
The sollution for me was to load the wizard code after the modal has opened .
$( "#modal" ).on('shown', function(){  FormWizard.init(); });
FormWizard.init(); contains all wizard code.
Hopefully this helps somebody

Instead the following event worked for me shown.bs.modal  this way

$("#modal").modal('show').on('shown.bs.modal', function () {
FormWizard.init();
});


Add metrics endpoint and request metrics

Resolves https://github.com/ollama/ollama/issues/3144
This pull request is to add /metrics endpoint and generally used metrics.
It exposes default prometheus metrics and custom metrics for request endpoints.
This PR does not try to cover all metrics to keep it simple. If this looks good. I could add few more that will be useful.
How to test:
once Ollama server is running pull a model and list(or any other Ollama actions)
curl http://<IP_ADDRESS>:11434/metrics

example custom metrics(not all are shown since i tried only few commands):
curl http://<IP_ADDRESS>:11434/metrics | grep -i ollama
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  6664    0  6664    0     0   519k      0 --:--:-- --:--:-- --:--:--  542k
# HELP ollama_model_list_requests_total The total number of model list requets that have been attempted.
# TYPE ollama_model_list_requests_total counter
ollama_model_list_requests_total{action="list",status="OK",status_code="200"} 1
# HELP ollama_model_pull_requests_total The total number of model pulls that have been attempted.
# TYPE ollama_model_pull_requests_total counter
ollama_model_pull_requests_total{action="pull",status="OK",status_code="200"} 1
# HELP ollama_model_requests_total The total number of requests on all endpoints.
# TYPE ollama_model_requests_total counter
ollama_model_requests_total{action="all",status="OK",status_code="200"} 6

Full output:
# curl http://<IP_ADDRESS>:11434/metrics
# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 1.0458e-05
go_gc_duration_seconds{quantile="0.25"} 5.4042e-05
go_gc_duration_seconds{quantile="0.5"} 6.9792e-05
go_gc_duration_seconds{quantile="0.75"} 0.000121417
go_gc_duration_seconds{quantile="1"} 0.00423525
go_gc_duration_seconds_sum 0.019693795
go_gc_duration_seconds_count 42
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 9
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.22.1"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 2.429592e+06
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 9.359476e+07
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 11319
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 545225
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 3.465616e+06
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 2.429592e+06
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 6.660096e+06
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 5.136384e+06
# HELP go_memstats_heap_objects Number of allocated objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 12324
# HELP go_memstats_heap_released_bytes Number of heap bytes released to OS.
# TYPE go_memstats_heap_released_bytes gauge
go_memstats_heap_released_bytes 6.201344e+06
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 1.179648e+07
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 1.7125328358982313e+09
# HELP go_memstats_lookups_total Total number of pointer lookups.
# TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 0
# HELP go_memstats_mallocs_total Total number of mallocs.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 557549
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 4800
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 15600
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 123840
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 179520
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 5.554272e+06
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 1.004569e+06
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 786432
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 786432
# HELP go_memstats_sys_bytes Number of bytes obtained from system.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 1.7259536e+07
# HELP go_threads Number of OS threads created.
# TYPE go_threads gauge
go_threads 15
# HELP ollama_model_list_requests_total The total number of model list requets that have been attempted.
# TYPE ollama_model_list_requests_total counter
ollama_model_list_requests_total{action="list",status="OK",status_code="200"} 1
# HELP ollama_model_pull_requests_total The total number of model pulls that have been attempted.
# TYPE ollama_model_pull_requests_total counter
ollama_model_pull_requests_total{action="pull",status="OK",status_code="200"} 1
# HELP ollama_model_requests_total The total number of requests on all endpoints.
# TYPE ollama_model_requests_total counter
ollama_model_requests_total{action="all",status="OK",status_code="200"} 7
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 73.31
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1.048576e+06
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 9
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 2.942976e+08
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.71253222787e+09
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 2.536882176e+09
# HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
# TYPE process_virtual_memory_max_bytes gauge
process_virtual_memory_max_bytes 1.8446744073709552e+19
# HELP promhttp_metric_handler_requests_in_flight Current number of scrapes being served.
# TYPE promhttp_metric_handler_requests_in_flight gauge
promhttp_metric_handler_requests_in_flight 1
# HELP promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code.
# TYPE promhttp_metric_handler_requests_total counter
promhttp_metric_handler_requests_total{code="200"} 3
promhttp_metric_handler_requests_total{code="500"} 0
promhttp_metric_handler_requests_total{code="503"} 0

@yuliyantsvetkov  @jmorganca It would be great if you can take a look.

This would be great

I think if observability is going to be added it should use the latest OpenTelmetry libraries instead of prometheus_client, especially because they're now stable and there's a lot more support now. prometheus only supports metrics, also, whereas I see it would be very valuable to add traces for observability purposes.
Happy to chat more about it on discord

@joshcarp I checked the OpenTelemetry libraries and your suggestion makes sense. It would be better to use Otel libraries. I will update the PR or create new one when i get some free time.

There's also a company called traceloop which does LLM observability who are spearheading a lot of the LLM observability stuff in Otel AFAIK. Might be worth seeing what they're doing because a lot of this seems like they're implementing/standardiing

replaced by https://github.com/ollama/ollama/pull/6537


Refresh Data

[x] When a new workspace is created, that workspace should be selected
[x] When a user CRUDs projects, they should see the updated information right away

In develop


chore: remove expectWarning and expectNotice use

Ref https://github.com/sebastianbergmann/phpunit/issues/3775, phpunit will not convert E_* to exceptions, therefore E_* can no longer be expected.
Following https://github.com/sebastianbergmann/phpunit/issues/5062#issuecomment-1416362657 approach to migrate our tests to avoid their usage.

🙌
Just a question, won't this be the workaround moving forward in other libraries as well? If yes then putting this type of logic somewhere in a central place makes more sense.

@yash30201 we decided against moving this logic in a central place because we would be looking to change these asserts to not expect warnings & notices in future.


scripts with type="text/ecmascript-6" are not  extracted

Putting type="text/ecmascript-6" in script tag like this:

results in ouput file containing original

Where you found this mime-type? links to spec https://tools.ietf.org/html/rfc4329 and thoughts http://www.2ality.com/2011/08/javascript-media-type.html

In Intellij idea:
https://intellij-support.jetbrains.com/hc/en-us/community/posts/207086715-Javascript-formatting-inside-html

I think standard-only will be type="module", not type="text/babel" or type="text/ecmascript-6" or type="es6" or even some kind of jsx. And their fix in intellij-idea is just another bundler/handler.. not real spec mime-type, because we can set it to type="text/coffeescript" or type="text/jsx".
So we must just wait their decision ;) I'd prefer to stay with standard and dont blow it with all other types.

well, anyway it's all workaround, hopefully, temporary. So nobody would be hurt much by blowing it with any sorts of types.

@bushuyev to make it faster, you can make pull like in commit linked before and maybe they accept it)

My pull request was  already merged here: https://github.com/PolymerLabs/crisper/commit/3e043d560797fdd6fecfae6ee1052aa8d43d5f5f
Somebody just need to bump the version I guess.

@wojtkowiak :+1: thanks!


Improve error message for invalid attribute values

Currently, when the user types in:
input type="number" min=0 max=100;

or
script src="pinkie.js" defer=true;

or
.col-sm-2 {}

they get the error "expected string". This is technically correct but I think we can do better.

For number literals (integer or float), suggest wrapping the number in quotes.

Alternatively, we can accept the number literal as-is, but I think that can be confusing because then factoring out a variable could change the result. See https://github.com/lambda-fairy/maud/issues/208#issuecomment-688756499.

For boolean literals, suggest removing the =true bit altogether.

For example, 0x2a and 0.99999999999999999999999 will be parsed as 42 and 1.0 respectively.

Actually, while Literal::* methods do lose precision, impl FromStr for TokenStream seems to be able to round-trip number literals loselessly. There rest is pretty easy.

Ah, that's interesting. I'm still against allowing that, though, as it means that factoring out a variable can change the result:
html! { 0x2a }  // "0x2a"

let ponk = 0x2a;
html! { (ponk) }  // "42"

But ponk is not a literal, it's an identifier. It shouldn't be a subject to suggestions in the first place.

To clarify, I mean that so far it's been true that adding parentheses () to a literal doesn't change it:
html! { "ponk" }  // "ponk"
html! { ("ponk") }  // "ponk"

But allowing bare number literals will break this property:
html! { 0x2a }  // "0x2a"
html! { (0x2a) }  // "42"

#265 improves the error message for non-string literals in element bodies and attribute values.
It still leaves numbers in class/ID shorthand (.col-sm-2). I'll file a follow-up issue to address that.


is it compatible with backbone?

+1

NB: I'm not the author.
I don't think it is based on a couple quick observations.

There's no $.ajax()
Backbone uses ajax() internally for it's sync() method on collections and models.
In another issue someone is requesting .on() event method.
IE delegating method binding to a parent DOM element. Backbone Views use this to bind to their root (this.el in the View) element, and dispatch to it's events: { ... } member.

Keeping ajax out of this lib makes a lot of sense to me, it's beyond the scope in my opinion. For Backbones purposes I think it would make more sense to rewrite it's sync() method to use the modern agreed upon XHR2 API, in keeping with this (sprint) libraries avoidance of legacy cruft.
The event binding it more problematic. If it is added it would be a good fit for use with Backbone I think.

Here are some infos about backbone without jquery.
https://github.com/jashkenas/backbone/wiki/Using-Backbone-without-jQuery

+1
could help sprint.js adoption rate. stats look great.


chore(main): release 0.1.14

:robot: I have created a release beep boop
0.1.14 (2024-11-28)
Other

release 0.1.14 (ea94cf9)

Dependency Updates

deps: update build digest to d0ef938 (#441) (044d782)
deps: update module github.com/onsi/gomega to v1.36.0 (#442) (ceb7392)

This PR was generated with Release Please. See documentation.

:robot: Created releases:

v0.1.14
:sunflower:


How do I update 1pl ?

I think go has changed some things. How do I update 1pl ?
I already noticed that go get doesn't work for 1pl and
that I need to use go install, which is also documented here.
But how do I update a go command? This here doesn't work:
$ go get -u all
go: go.mod file not found in current directory or any parent directory.
        'go get' is no longer supported outside a module.
        To build and install a command, use 'go install' with a version,
        like 'go install example.com/cmd@latest'
        For more information, see https://golang.org/doc/go-get-install-deprecation
        or run 'go help get' or 'go help install'.

To update 1pl, go install github.com/ichiban/prolog/cmd/1pl@latest will do the job.
$ $(go env GOPATH)/bin/1pl
Top level for ichiban/prolog v1.2.0
This is for testing purposes only!
See https://github.com/ichiban/prolog for more details.
Type Ctrl-C or 'halt.' to exit.
?-
$ go install github.com/ichiban/prolog/cmd/1pl@latest
$ $(go env GOPATH)/bin/1pl
Top level for ichiban/prolog v1.2.1
This is for testing purposes only!
See https://github.com/ichiban/prolog for more details.
Type Ctrl-C or 'halt.' to exit.
?-

Ok, thx.


Can I build a engine for Llama model, with dtype bfloat16 and int8 weight ?

Thanks for this excellent project!
I can generate a bfloat16 model or an int8 weight model，but wehn I tried the following commands:
python ./examples/llama/build.py --model_dir ./Mixtral-8x7B-Instruct-v0.1/
--use_inflight_batching
--enable_context_fmha
--use_gemm_plugin
--world_size 2
--tp_size 2
--dtype bfloat16
--use_gpt_attention_plugin bfloat16
--use_gemm_plugin bfloat16
--use_weight_only
--output_dir engine
I get an error message:
"
[12/23/2023-16:05:41] [TRT-LLM] [I] Loading weights from HF LLaMA...
Traceback (most recent call last):
File "/workspace/github/TensorRT-LLM/./examples/llama/build.py", line 906, in
build(0, args)
File "/workspace/github/TensorRT-LLM/./examples/llama/build.py", line 850, in build
engine = build_rank_engine(builder, builder_config, engine_name,
File "/workspace/github/TensorRT-LLM/./examples/llama/build.py", line 690, in build_rank_engine
load_from_hf_llama(tensorrt_llm_llama,
File "/workspace/github/TensorRT-LLM/examples/llama/weight.py", line 352, in load_from_hf_llama
print(torch.tensor(v))
TypeError: can't convert np.ndarray of type numpy.void. The only supported types are: float64, float32, float16, complex64, complex128, int64, int32, int16, int8, uint8, and bool.
"
Please tell me how to solve this problem? Thanks!

@ztxz16
Hi,
can you share the branch and github commit you are using to hit with this issue?
Thanks
June

@ztxz16
Hi,
can you share the branch and github commit you are using to hit with this issue?
Thanks June

main branch,
commit id: a75618df24e97ecf92b8899ca3c229c4b8097dda

The engineers working on Mixtral are on holiday this week - they'll be back in January. Is it a blocking issue?

@ztxz16 Do you still have the problem? If not, we will close it soon.


install_module_from_forge_on should require a version constraint string

I have been writing something like this when I always want the latest version of a forge module:
install_module_from_forge_on(host, mod, '> 1')

This feels a bit silly.  It would be preferable if #install_module_from_forge, #install_module_from_forge_on and #module_version_from_requirement didn't require a version constraint string and it being absent indicates that the latest forge version should be installed.

I'd say it should default to > 0 for maximum compatibility. However, I'm working on deprecating this entire gem and moving to another. There I already have it default to nil and then it doesn't pass in the version at all, trusting puppet module install to do the right thing.
https://github.com/voxpupuli/beaker_puppet_helpers/blob/422c1ec0dce9a391927a6e0b2c1f7f040fff49bd/lib/beaker_puppet_helpers/module_utils.rb#L9-L29

I wasn't aware of beaker_puppet_helpers, maybe we could add a link in the README from this module?  How soon do you think voxpupuli-test will be able to switch over?

It's still very fresh, but I opened https://github.com/voxpupuli/voxpupuli-acceptance/pull/31. I'm at the point where I should be doing some testing with it to see if it can replace the various gems. However, we're also in the middle of preparing for Foreman 3.2 so I probably won't have time to do so in the next week or 2.


Installation fails on macOS 10.15.2 for cchardet

OSX 10.15.2
Python 3.6.0
Issue
Installation fails due to cchardet build failure:
Running setup.py clean for cchardet
Failed to build cchardet
Installing collected packages: cchardet
    Running setup.py install for cchardet ... error

clang: warning: libstdc++ is deprecated; move to libc++ with a minimum deployment target of OS X 10.9 [-Wdeprecated]
    clang: warning: libstdc++ is deprecated; move to libc++ with a minimum deployment target of OS X 10.9 [-Wdeprecated]
    ld: library not found for -lstdc++
    clang: error: linker command failed with exit code 1 (use -v to see invocation)
    error: command '/usr/bin/clang++' failed with exit status 1

Upgrading to Python 3.7.6 solved the issue :)


add PSRAM write demonstration

I had some troubles starting work with the FPGA on my ICE-V board; the blinky example was too trivial, the factory_bitstream too complex, and it did not help that I am using VHDL, not Verilog. Having finally found my footing, I decided to share a intermediate demo that writes into the PSRAM in hopes that it will help whoever might come after me.

Thanks for the contribution - glad to see some VHDL in there too!


Multi-batch write testing on a single-node

Unit-test code only, self-merging on green build.

This change made the test suite fail with:
[messaging] 2015/02/21 20:27:45 EOF
--- FAIL: TestClient_Publish_ErrConnectionRefused (0.23s)
	client_test.go:122: unexpected error: Post http://<IP_ADDRESS>:44662/messaging/messages?topicID=0&type=100: EOF

@benbjohnson -- any comment? This is not 100% reproducible.

Unit test code only, self-merging.


Import wallet by private key

Hi there,
I've been using the Atomic wallet to store my TON tokens. Unfortunately, I've encountered an issue where it won't allow me to send my assets and displays an error message stating, "Block explorer is currently unavailable." I've tried various methods and even reached out to their support team for assistance, but my problem persists.
As you may know, you have access to your private key in your atomic wallet. However, the private key for TON wallet on Atomic wallet is a lengthy 128-character base, while most wallets that support private key imports typically use a 64-character base, such as Trust Wallet and Safepal.
My question is, are there any wallets that support the import of a wallet using a 128-character private key? Or do you have any suggestions to help me resolve this issue?
I appreciate your assistance in advance.
Answer
It should be possible to cut off the last half of key (64 HEX characters), which is actually public key, and use the rest in other wallets.

Original

Hey @akegaviar
You're experiencing transcript error. It seems your query would be best dealt with by the support team.
I'll have to refer you to the official support Requests team with the ticket ID SFW07638
Please see the link below to our dedicated support line:
Safepal Support Requests
Note; Click on the live chat icon at the bottom corner of the page to initiate chat with a live support agent.


iOS: Unable to display center popup when transitioning from a view presented with .fullScreenCover.

Attempting to display a center popup from a view presented using .fullScreenCover results in the popup not being shown.

Hey @RizwanaDesai,
So this is the problem we're aware of - there's no option to show anything on top of Apple's sheets.
However, you can replace fulscreenCover with our implementation - check out popupFillsEntireScreen modifier.

" there's no option to show anything on top of Apple's sheets." - Can you give me some documentation for this? Previously we were displaying a custom view over top the page as the popup.

hey @itsthisjustin,
What I meant is that it's impossible with a current implementation of the library. And here's why:
We use the implementPopupView modifier to ensure that any popup will be built on top of any view. However, the Apple sheets create another "layer", that should be on the top of the current view. And this is where the problem lies - since our PopupView was declared earlier, it's impossible to place it on the top of the sheet.

Therefore I think it's better to replace fullscreenSheet with our BottomPopup + contentFillsEntireScreen modifier. On Monday / Tuesday, we're going to implement a change that overlay will be always a layer below CentrePopup (sorry, we are busy this week as we're working full-time for a company)

@RizwanaDesai, @itsthisjustin feature was implemented in the version 2.3.0

Awesome thanks!

In version 2.3.0, I don’t see the contentFillsEntireScreen modifier?

Hey @Monkey-X-Byte,
Nah, it’s still there. However, it’s available only for BottomSheetPopup in the configurePopup method.
Let me know if it solved your problem and if not, I’m here to help you ;)

Hey @Monkey-X-Byte,
Nah, it’s still there. However, it’s available only for BottomSheetPopup in the configurePopup method.
Let me know if it solved your problem and if not, I’m here to help you ;)

thanks


kernel_config: add 4.1 kernel support

Fixes #144
Current kata containers can't run with kernel 4.1 built from current x86 config,
it will report error:
$ docker run -ti --runtime kata busybox sh
docker: Error response from daemon: oci runtime error: rpc error: code = Internal
 desc = Could not run process: container_linux.go:348: starting container process
 caused "process_linux.go:402: container init caused \"open /dev/ptmx: no such
file or directory\"".

This is caused by bogus devpts mount options. When run container with docker,
docker will assign a default devpts mount for every container which equals to
command below:
$ mount -t devpts -o nosuid,noexec,newinstance,ptmxmode=0666,mode=0620,gid=5 \
devpts /dev/pts

This requires kernel config CONFIG_DEVPTS_MULTIPLE_INSTANCES=y to work properly
under kernel-4.1, but this option is already removed from latest kernel.
It's better to add it back for support older kernel than current 4.14.
Signed-off-by: Wei Zhang<EMAIL_ADDRESS>

lgtm.
Note that you need to update the following though:
Check Changes in kernel
kernel/configs/x86_64_kata_kvm_4.14.x
Kernel directory has changes check kernel/kata_config_version changed
ERROR: Please bump version in kernel/kata_config_version

lgtm  and indeed a problem as we tested

lgtm but you have to bump the kernel version as @jodh-intel mentioned

Version updated!

/cc @Pennyzct, @nitkon.

@WeiZhang555 It's a bit weird that we add kernel configs in x86_64_kata_kvm_4.14.x to support 4.1 kernel. Maybe we can put the option in a separate file like x86_64_kernel_compat, which can contain all the removed old kernel configs that we need for older kernels. If we just put it in x86_64_kata_kvm_4.14.x, I'm afraid it can easily get lost when we update the file. And because we do not test old kernels in CI, we cannot be notified when it's lost.

@bergwolf I agree it could be a bit weird.
You know that we don't maintain a kernel 4.1 config, so adding this redundant line to kernel-4.14 config  could be the simplest way for user to configure its own kernel: just copy the config file to the kernel source directory and rename it to .config, then replying on make menuconfig to do auto-tuning magic.
This one configuration is the only missing one identified for kernel 4.1, maybe later we will have more for kernel 3.10 or kernel 3.19 or xxx.  I'm not sure how to make the x86_64_kernel_compat, user need to combine content of x86_64_kernel_compat and x86_64_kata_kvm_4.14.x? sounds a little bit complicated.
Another topic is, we really need a baseline of guest kernel version, it would be better we can test the minimum allowed kernel version in CI.

@WeiZhang555 The problem with adding it to 4.14.x is that when we update kernel configs (e.g. for kernel major version upgrade), most likely we will just overwrite it with a working .config. Then the extra old obsolete kernel config option is dropped and we cannot find it out unless someone pays close attention to it during reviewing.
So yes, it will be a bit more complicated for old guest kernel users if they do not want to use the guest kernel version we are constantly building/shipping. They need to append x86_64_kernel_compat to the main kata kernel config file. But it is just little price to pay to get a working kernel config file, compared with building and testing the old kernel themselves.

@WeiZhang555 Since this is removed from the latest kernel, I think it makes sense to put this in a separate compatibility file as @bergwolf pointed to make sure we dont lose this option again.

@bergwolf @amshinde That makes a lot of sense! Updated as you suggested. Please take another look at this.


send_to fails due to invalid SockAddr

I am using SocketAddrV4.into() which is passed to socket.send_to.
And according to https://github.com/rust-lang/socket2/blob/master/src/sockaddr.rs#L238
sin_len is always 0 which causes
thread 'tokio-runtime-worker' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 22, kind: InvalidInput, message: "Invalid argument" }', src/main.rs:55:43

is there any quick fix to this?

I was wrong, it is not the root cause of error described earlier.

Could you share some code and context (OS, etc.) because it's highly unlike the socket address conversion is at fault here.

yeah, this error code refers to EINVAL (linux.man)[https://linux.die.net/man/3/sendto].
I am using set_header_included meaning that I am manually creating a tcp ip packet.

solution to the problem is to use little endian for checksums and ip_len on macos
as per

https://cseweb.ucsd.edu//~braghava/notes/freebsd-sockets.txt
https://stackoverflow.com/a/7160206
Steven's "Unix Network Programming" Volume I Chapter 25 "Raw socket" 2nd edition


Vertical notification animation broken

Donate 💰 to fund this issue

You can donate funding to this issue. We receive the money once the issue is completed & confirmed by you.
100% of the funding will be distributed between the Filament core team to run all aspects of the project.
Thank you in advance for helping us make maintenance sustainable!

While playing with the demo site at v3.2.2, I noticed that the Notification Animation does slide in animated, but the direction seems to be reversed. It slides in from the left-to-right. It would be perfect for it to be corrected back to right-to-left.
Thanks.

@fseesys, could you please create a bug report for what you're experiencing?
I can't reproduce it in the demo.

@fseesys, could you please create a bug report for what you're experiencing? I can't reproduce it in the demo.

Sorry for replying late, but it seems to be fixed in v3.2.27 of the demo.
Thanks!

@zepfietje I'm not sure if this is related, if you need to me open a new issue please let me know.
We have the following code:
$quote->wasRecentlyCreated
    ? $this->redirectRoute('quotes.edit', $quote->id)
    : Notification::make()->title(__('general.saved'))->success()->send();

$this->dispatch('log:fresh');

The event is used to refresh our audit log. The audit log has its own Livewire component.
When the notification is sent, it drops down from the top right to the bottom right of the screen. As soon as we remove the dispatch or remove #[On] from the audit log component, the notification slides in from the right, as always does.

Once this issue is fixed, the issue you're experiencing will probably be fixed too, @ju5t.
If you want to look into a fix, any help is much appreciated since I've spent a lot of time on this one already.

@zepfietje I will schedule some time to look into it tomorrow. Can't promise I find a solution though, my JS is a bit rusty. Are there any other places to look for the animation aside from notification.js#L76?

No worries, @ju5t, any insights are welcome!
You linked the right place indeed. It worked fine in v2, but couldn't get it to work with Livewire v3 yet.

@zepfietje I found something.
When we're not dispatching the log:fresh event there are 3 requests in the network tab:

We call our action. This will send the notification.
The second request has the notificationsSent event.
And finally we see notificationClosed when the notification disappears.

When we dispatch our log:fresh event it's different. There are two notificationSent events. The first is combined with our event, the other is on its own.
It's related to NotificationsServiceProvider.php#L52. This uses:
on('dehydrate', function (Component $component) {
...
});

In our situation, two components are dehydrating at about the same time. The Notification component hasn't had the chance yet to pull the notification from the session. This results in two events getting dispatched.
Our component is called ActivityLog. As a workaround I added:
if ($component instanceof ActivityLog) {
    return;
}

This prevents notifications being sent from that component and solved it for us. Obviously this is not a solution, I haven't had the time to think about that yet. And I don't know if this is the cause of other issues with notifications either.
Food for thought :)

I spent some more time on this today, but I can't find a clean solution.
What worked for our issue is 'abusing' sessions to see if we've dispatched the event yet. Perhaps someone with more knowledge on the Livewire internals can come up with a cleaner way of doing this. Preferably so we can ditch the dehydrate logic altogether as it's somewhat redundant.
Here's our code to try out. In the service provider:
if (session()->get('filament.notificationsSent', false) === false) {
    $component->dispatch('notificationsSent');
    session()->put('filament.notificationsSent', true);
}

And inside the Notifications Livewire component we add this to the pullNotificationsFromSession() method:
session()->forget('filament.notificationsSent');

I spent some more time on this today, but I can't find a clean solution.
What worked for our issue is 'abusing' sessions to see if we've dispatched the event yet. Perhaps someone with more knowledge on the Livewire internals can come up with a cleaner way of doing this. Preferably so we can ditch the dehydrate logic altogether as it's somewhat redundant.
Here's our code to try out. In the service provider:
if (session()->get('filament.notificationsSent', false) === false) {
    $component->dispatch('notificationsSent');
    session()->put('filament.notificationsSent', true);
}

And inside the Notifications Livewire component we add this to the pullNotificationsFromSession() method:
session()->forget('filament.notificationsSent');

Could you explain better how you did this?

@mateusgalasso this is not user land code. It needs to be solved in Filament.


install cli failure

This is autogenerated. Please review and update as needed.
Describe the bug
Command Name
az aks install-cli
Errors:
IncompleteRead(19316736 bytes read, 30847904 more expected)
Traceback (most recent call last):
python3.8/site-packages/knack/cli.py, ln 215, in invoke
    cmd_result = self.invocation.execute(args)
cli/core/commands/__init__.py, ln 631, in execute
    raise ex
cli/core/commands/__init__.py, ln 695, in _run_jobs_serially
    results.append(self._run_job(expanded_arg, cmd_copy))
cli/core/commands/__init__.py, ln 688, in _run_job
    six.reraise(*sys.exc_info())
lib/python3.8/site-packages/six.py, ln 703, in reraise
    raise value
...
cli/command_modules/acs/custom.py, ln 278, in _urlretrieve
    f.write(req.read())
lib/python3.8/http/client.py, ln 467, in read
    s = self._safe_read(self.length)
lib/python3.8/http/client.py, ln 610, in _safe_read
    raise IncompleteRead(data, amt-len(data))
http.client.IncompleteRead: IncompleteRead(19316736 bytes read, 30847904 more expected)

To Reproduce:
Steps to reproduce the behavior. Note that argument values have been redacted, as they may contain sensitive information.

Put any pre-requisite steps here...
az aks install-cli

Expected Behavior
Environment Summary
macOS-10.15.3-x86_64-i386-64bit
Python 3.8.2
Installer: HOMEBREW

azure-cli 2.5.1

Additional Context

aks

seems downloading issue.

Looks like a transient download failure. Please reopen if it's consistent.


Fixes typos and markdown syntax in  the documentation

What does this PR do?
Fixes typos and some markdown syntax in docs.
Motivation
To provide a better documentation for the community.
More

[ ] Yes, I have tested the PR using my local account setup  (Provide any test evidence report under Additional Notes)
[x] Yes, I ran pre-commit run -a with this PR
[ ] Yes, I have added a new example under examples to support my PR (when applicable)
[ ] Yes, I have updated the Pages for this feature

Note: Not all the PRs required examples and docs.
For Moderators

[ ] E2E Test successfully complete before merge?

Additional Notes
It just fixes some typos and syntaxes. No TF code was modified or features implemented.

Thanks!


[SPARK-5123][SQL] Expose only one version of the data type APIs (i.e. remove Java-specific APIs)

Having two versions of the data type APIs (one for Java, one for Scala) requires downstream libraries to also have two versions of the APIs if the library wants to support both Java and Scala. I took a look at the Scala version of the data type APIs - it can actually work out pretty well for Java out of the box.
As part of the PR, I created a sql.types package and moved all type definitions there. I then removed the Java specific data type API along with a lot of the conversion code.

Test build #25141 has started for   PR 3925 at commit 8eb5dc9.

This patch merges cleanly.

Test build #25141 has finished for   PR 3925 at commit 8eb5dc9.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25141/
Test FAILed.

Is this something where we need to make sure to include upgrade details in the release notes?

Yes - is there a way to label these things in JIRA?

Test build #25142 has started for   PR 3925 at commit 4674585.

This patch merges cleanly.

cc @marmbrus, @yhuai for SQL changes, and @mengxr, @jkbradley for MLlib changes ...

Test build #25142 has finished for   PR 3925 at commit 4674585.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25142/
Test FAILed.

Test build #25145 has started for   PR 3925 at commit a2bb038.

This patch merges cleanly.

Test build #25145 has finished for   PR 3925 at commit a2bb038.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25145/
Test FAILed.

That's a very cool idea to make a unified DataType API for jvm-based language. Less code maintenance will be made!

SQL changes look good to me.

Test build #25169 has started for   PR 3925 at commit a1c1864.

This patch merges cleanly.

Test build #25170 has started for   PR 3925 at commit 0cc1a5d.

This patch merges cleanly.

Test build #25169 has finished for   PR 3925 at commit a1c1864.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25169/
Test FAILed.

Test build #25170 has finished for   PR 3925 at commit 0cc1a5d.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25170/
Test FAILed.

I pushed a fix for the failing schema tests. However, looks like sbt+master is failing HiveThriftServer2Suite that is not caused by this PR.

Test build #25173 has started for   PR 3925 at commit 5b77c6a.

This patch merges cleanly.

Test build #25173 has finished for   PR 3925 at commit 5b77c6a.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25173/
Test FAILed.

Test build #25184 has started for   PR 3925 at commit c130913.

This patch merges cleanly.

@rxin  MLlib changes look fine to me, and FWIW the other parts did too.  My only remaining comments are ordering imports, but I'll leave those out for now.  LGTM

Test build #25184 has finished for   PR 3925 at commit c130913.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25184/
Test PASSed.

Test build #25193 has finished for   PR 3925 at commit e98a7c0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25193/
Test PASSed.

Test build #25260 has started for   PR 3925 at commit 603919f.

This patch merges cleanly.

Test build #25260 has finished for   PR 3925 at commit 603919f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25260/
Test PASSed.

See https://github.com/apache/spark/pull/3958


Fix controlboard singleton keys

Closes #89

Ok! Can you just change the PR name? The PR name will become an item in the changelog, and "Fix controlboard singleton keys" is quite confusing. I think using a name like the one of the issue https://github.com/robotology/gz-sim-yarp-plugins/issues/89 is much more clear.

Merging!


wfnews-1528 show saved location on wfnews map

A few general cleanup bits to address but otherwise OK

did some clean up as advised. Will merge in