Refactor Transaction Receipt/Result/State

Per today's design review call, Noam and Iddo both questioned the current design of TransactionResult. Maybe what we really want to know is: 1. was the transaction "effective" or not (Iddo's term, i.e., syntactically valid), 2. did the transaction succeed or fail?, and 3. was a fee charged for it or not? These things are not immediately visible from the current design. Do we want to rethink the design?

I'm a bit confused about this issue.
A transaction is effective / syntactically valid if it made its way to the mesh and invalid if it was rejected before that as there are validity checks when it is submitted by a client to a node for broadcasting. See TransactionState.REJECTED.
I think that as soon as it is TransactionState.PENDING on its journey to the mesh, it means it passed syntactic validation and is in the pool.
The TransactionState enum is design to capture all possible rejections which can be considered types of failures before it gets to STF execution. What's may be confusing to reviewers here is that there are pre-stf validation checks (designed by Noam) so a transaction may be rejected by-stf due to non-syntactic reasons such as counter and balance. However, all of these failures are captured by TransactionState.
@noamnelke @tal-m
Regarding execution failure. The receipt should have all the information regarding the other questions in the issue which are different questions than syntactic validity. So, I don't understand the concern described based on the current design - it is immediately clear exactly what is the transaction status from the receipt, and the TransactionStatus enum should cover all possible cases and questions raised above regarding execution and there's also a field in the receipt for how much gas was consumed. So that is fully addressed.
A receipt is generated even if execution 'fails' because gas is consumed and the tx affected the global state in that way, even though the intent of the tx was not applied. So, even a transaction that was no applied - e.g funds were not moved according what it specifies, affects the global state and generates a receipt. This is good because the receipt is the only way for clients to know when it was executed (layer # of STF) and why it wasn't applied - and TransactionResult field tells you exactly that.
I think Iddo and Tal argued we should expose the ordered list of input transactions to the STF in each layer via the global state API, and have an index in the tx receipt that indicates the location of the tx in this ordered list so we can show users what txs were executed before or after it in the layer. As the ordered list is per layer, this gives us all the ordered txs across all layers which Tal mentioned.

A transaction is effective / syntactically valid if it made its way to the mesh and invalid if it was rejected before that as there are validity checks when it is submitted by a client to a node for broadcasting.

My understanding, based on our conversation, is that it's not this simple. As one example, a transaction may be included in two different layers (#8). It may not be "effective" the first time it's included, because, e.g., the nonce is too low, but it may become "effective" in a later layer, or vice-versa. The same tx in a later layer may not even be visible to/processed by the STF, depending how we design the protocol.

there are pre-stf validation checks (designed by Noam) so a transaction may be rejected by-stf due to non-syntactic reasons such as counter and balance... all of these failures are captured by TransactionState.

Agree with this, but it's worth pointing out that TransactionState is not canonical because one node might reject a tx that another node admits into its txpool/puts into a block.

we should expose the ordered list of input transactions to the STF in each layer via the global state API, and have an index in the tx receipt that indicates the location of the tx in this ordered list

See #34

If a transaction has the wrong nonce it will should be filtered out from the list of txs for execution by the stf... In that case, it won't be executed but will be in the mesh in that layer. The way the STF is defined now it only picks transactions from the layer-1 it is running on and not any other transactions.

Agree

The way the STF is defined now it only picks transactions from the layer it is running on and not any other transactions. So in this case, it can't be executed by the STF in the future. So it can't become effective a layer later unless we completely change the way the STF is defined

Disagree: a miner may decide to include the same tx in a block in a later layer - this has nothing to do with the STF

it's worth pointing out that TransactionState is not canonical because one node might reject a tx that another node admits into its txpool/puts into a block
Regarding your second comment - this is not true - all nodes that run the same protocol can not reject a transaction which is in a verified layer on the mesh. Once a transaction is in the mesh and there's consensus about it being on the mesh by honest nodes then they must execute it according to the STF algo which is all nodes must follow unless they want to fork from the state of all other honest nodes.

Agree

So, TransactionState is canonical and must always be the same for all honest nodes

Disagree: the decision whether and on what basis to admit a transaction to the txpool, or to include it in a block, is not under protocol and different nodes are free to implement different policies around it. You're right that once a tx is in the mesh and confirmed by consensus it becomes canonical, but up to this point transaction state is not canonical and different nodes may have different opinions:
// NOT CANONICAL:
        UNDEFINED = 0; // default state
        REJECTED = 1; // rejected pre STF processing due to, e.g., bad data
        INSUFFICIENT_FUNDS = 2; // rejected pre STF processing by funds check
        CONFLICTING = 3; // rejected pre STF due to conflicting counter

// SHOULD BE CANONICAL AMONG ALL HONEST NODES WHO SAW THE BLOCK:
        PENDING = 4; // included in a block on the mesh. Pending processing by STF

// CANONICAL:
        PROCESSED = 5; // processed by the STF. Processing results are in the tx receipt

This case where the same tx can appear in different layer is a PITA to deal with as it complicates things. e.g. a transaction needs to have a list of one or more mesh layers it appears in and not only one and the apis need to support this. It would be great if we can avoid this but I'm not sure it is possible

Frankly I don't think we can make any decisions here regarding API or data model until we have a decision on this question!

Hopefully the following will help answer your questions:
Conceptually, I think of Spacemesh as having two layers: the consensus layer and the VM layer. The consensus layer is completely VM agnostic (up to syntactic considerations, see Syntactic Validity of Transactions.
Consensus layer
The output of this layer is the transaction ledger: an ordered list of transactions. It has the following properties:

all honest nodes agree on the exact list of transactions (both contents and order)
every transaction appears exactly once.
The consensus layer is responsible for ordering and deduplicating transactions. Blocks and layers are implementation artifacts of the consensus layer. They do have some effect on the output---for example, our consensus algorithm  can only dedup and order transactions after agreeing on an entire layer, so the output granularity is one layer at a time. However, the VM layer doesn't really have to care about this.

Note that if a transaction appears in multiple blocks in a layer, the consensus algorithm will output it only once. Similarly, for a transaction that appears in two different layers, the consensus output will contain only the first occurrence. In particular, this means that once a transaction is in the mesh, the exact same transaction can't be "resubmitted" later --- a new transaction has to be created.
Rollbacks (reorgs)
In case of self-healing, it's possible that some portion of history is reversed). If this happens, the consensus layer updates the VM layer that there was a reset to some specific transaction in the past, and then outputs the ordered list of transactions starting from the reset point.
VM Layer
The VM layer is responsible for maintaining global state (account balances, app code, app variables, etc.) and executing the state transition function (STF). The STF accepts a single transaction, and executes it, potentially modifying the global state.
Transaction execution
The VM layer calls the STF on every transaction in the order it receives them from the consensus layer, modifying the global state as it does so.
The VM executes every transaction that it receives from the consensus layer. Some of these may be "totally ineffective" (have no effect on the global state) and some may change the global state, but still be ineffective as far as the user is concerned (e.g., if a TX runs out of gas, it does change the global state, because the gas fee is still deducted from some account, but it probably  didn't do what the user wanted.). Let's call this kind of transaction partially effective.
Rollbacks
If the VM layer is notified of a rollback, it is responsible for rolling back the global state to whatever it was at the reset point. It can then execute transactions normally.
Transaction results
Given the above, we can now classify the state of a transaction:

Not yet in mempool
Rejected from mempool for "heuristic" reasons
In mempool
Accepted to mesh. In this case, we have a "VM receipt"

totally ineffective (no change in global state). Reasons can be

bad nonce
insufficient funds
conflicting counter
etc.

effective or partially ineffective. In this case the receipt contains the changes in global state. In addition, if it's partially ineffective, it can have a reason (e.g., gas ran out)

Syntactic Validity of Transactions
Theoretically, we can make the consensus layer  totally VM-agnostic, which means it would allow completely invalid transactions in the mesh. However, to prevent DoS attacks, we do want to filter some things at the consensus layer already. So transactions that are syntactically-invalid (i.e., they would be totally ineffective regardless of the global state when they are executed by the VM) can be filtered out at the consensus layer. This means that we would consider blocks containing such transactions as syntactically invalid.
Heuristic rejection of transactions
We may also want to prevent transactions that do depend on state from making into the mempool (such as ones that have conflicting counters). In this case, honest nodes will ignore the transactions, but they could still be included into the consensus layer by malicious nodes (or even by honest nodes, in case our assumptions are not satisfied --- for example during a network split). This kind of transaction can be resubmitted later, since it is filtered before it makes it into the consensus layer, so doesn't violate the uniqueness property of the transaction ledger.

I agree with most of what you wrote in your comment and I think it is quite close to our proposed STF and tx results / status api  but there are few caveats and things to address and be a bit more precise about:

all honest nodes agree on the exact list of transactions (both contents and order)
every transaction appears exactly once.

I don't think you can exclude the time element (layer) from this definition. The agreement on the content is always in the context of a layer number. All honest nodes agree on the contents up to a specific layer. e.g. the verified layer.

The consensus layer is responsible for ordering and deduplicating transactions. Blocks and layers are implementation artifacts of the consensus layer. They do have some effect on the output---for example, our consensus algorithm can only dedup and order transactions after agreeing on an entire layer, so the output granularity is one layer at a time. However, the VM layer doesn't really have to care about this.

I find it a bit misleading to call a layer an implementation artifact where it is actually the time ticks that the consensus works in and vital to consensus as my point above explains. So, it is not enough that the consensus creates a prefix list of ordered transactions. Because the consensus verifies tx in chunks (all the tx in a layer) once a layer is verified, it is useful to also describe the input batch to the STF as input list for an input layer. It is not true that the VM layer doesn't care about layers. The way we specify the STF it must care about layer as it always work with input from layer n-1 and outputs state for layer n. Without such knowledge, the STF will not be able to create receipts that describe the deterministic time of execution of the tx which is critical piece of data in a crypto-currency. (e.g. block # for processed txs in bitcoin and ethereum). So it is undesirable from practical considerations to define the vm layer as layer agnostic. The layer is the deterministic tick of the decentralized computer that determines computation time. Without for example layer number in tx receipts there's no way to know when a tx was actually processed as the index in the global ordered lists of tx doesn't give us any notion of time.
It is not enough to say that the output granularly is one layer at the time. More exact definition of the STF is required - can you please comment on what is defined in the data and api smip where we attempt to properly define the STF? It will be useful if you can say what is wrong / bad there than needs to be changed.
I find the other comments in the consensus layer great and we should add them to spec. To be a bit more specific, I think that what you argue means that a specific transaction can only appear in 1 stf input-list in a layer but not in multiple ones. So, once a tx is in the input list of the STF, it will at least be what you call partially applied. meaning, the stf will attempt to execute it.

Rollbacks
This is exactly what it is very useful to define the STF ticks as layers. In case of rollback, the consensus layer instructs the vm layers to execute the STF from layer n.

Transaction results
Can you please relate to the proposed state? I think that we pretty much agree on this but you are talking about adding another state for inclusion in the mem pool.

enum TransactionStateType {
        UNDEFINED = 0; // default state
        REJECTED = 1; // rejected pre STF processing due to, e.g., bad data
        INSUFFICIENT_FUNDS = 2; // rejected pre STF processing by funds check
        CONFLICTING = 3; // rejected pre STF due to conflicting counter
        PENDING = 4; // included in a block on the mesh. Pending processing by STF
        PROCESSED = 5; // processed by the STF. Processing results are in the tx receipt
    }

it is also not entirely accurate that we'll have a receipt if a tx was accepted to the mesh because the STF may have not executed yet for that layer. Transactions which are on the mesh will have a receipt later - this is why we propose state  PROCESSED = 5 above which you don't mention in your summary. So considering your feedback and this point, I think the possible states are:
enum TransactionStateType {
UNDEFINED = 0; // default state
REJECTED = 1; // rejected pre STF processing due to, e.g., bad data
MEM_POOL = 2; // accepted to the mem-pool and broadcasted by node via gossip
INSUFFICIENT_FUNDS = 3; // rejected pre STF processing by funds check
CONFLICTING = 4; // rejected pre STF due to conflicting counter
PENDING = 5; // included in a block on the mesh. Pending processing by STF
PROCESSED = 6; // processed by the STF. Processing results are in the tx receipt
}
Regarding tx receipt data you mention, can you please review to what we proposed in the smip:
enum TransactionResult { // the results of STF transaction processing
UNDEFINED = 0;
EXECUTED = 1; // executed w/o error by the STF
BAD_COUNTER = 2; // unexpected transaction counter
RUNTIME_EXCEPTION = 3; // app code exception
INSUFFICIENT_GAS = 4; // out of gas
INSUFFICIENT_FUNDS = 5; // failed due to sender's insufficient funds
}
I think it captures all possible stf execution results and it is pretty similar to what you describe but more formal.

Syntactic Validity of Transactions
Theoretically, we can make the consensus layer totally VM-agnostic, which means it would allow completely invalid transactions in the mesh. However, to prevent DoS attacks, we do want to filter some things at the consensus layer already. So transactions that are syntactically-invalid (i.e., they would be totally ineffective regardless of the global state when they are executed by the VM) can be filtered out at the consensus layer. This means that we would consider blocks containing such transactions as syntactically invalid.

Didn't we review and approved Noam's design for pre-stf filleting for 0.3.5 in the product summit in 12/2019? I believe pre-filtering is currently implemented in the testnet codebase and has been so for a long time. If any pre-filtering as implemented for 0.3.5 then we want to spec it now.

Rollbacks (reorgs)
In case of self-healing, it's possible that some portion of history is reversed. If this happens, the consensus layer updates the VM layer that there was a reset to some specific transaction in the past, and then outputs the ordered list of transactions starting from the reset point.

Agree with Aviv here that this reset should only be defined for an entire layer, not for a specific transaction. The behavior of the STF on an individual transaction, or a portion of a layer, is not well defined due to the way we handle fees and rewards, pooled and on a per-layer basis. In other words, the STF cannot be "reset" to an intermediate point in a layer.

Accepted to mesh. In this case, we have a "VM receipt"

As Aviv pointed out, there should be a "pending" state - added to a block but the block hasn't been approved/added to the mesh yet.

transactions that are syntactically-invalid (i.e., they would be totally ineffective regardless of the global state when they are executed by the VM) can be filtered out at the consensus layer

Is this possible in practice? As discussed on the design review call, it's not possible for simple coin transfers as they're designed now, since every signature matches some account's private key - and we cannot know ahead of time whether that account won't have some balance when the STF attempts to execute the tx.
As for smart contract/"app" transactions, we need @yaronwittenstein's input on this. This question probably reduces to some version of, "Are there bytecode patterns that we can definitively call invalid pre-execution?" You could attempt to syntactically parse Wasm bytecode and make sure it's valid, but this doesn't preclude the possibility of using other bytecode formats/targeting other VMs in the future.

This is why I wanted to define global-state as meta-mash which includes the proper more narrow global-state (state of account and app db) as well as execution side-effects (receipts and app events), but we decided to call all the data and the side-effects of execution global state

Thinking about this more, I do see some value in a distinction here between a more narrowly-defined "global state" (basically, state of all accounts) and a broader "meta-mesh" (for lack of a better term) that includes tx receipts, app events, etc.

Regarding TransactionState, we unfortunately have a confusing mix of concerns the way this is set up right now:
// NOT CANONICAL:
    UNDEFINED = 0; // default state
    MEMPOOL = 1; // admitted to mempool
    REJECTED = 2; // rejected from mempool on heuristic grounds (could be further broken out into different reasons for rejection)

// SHOULD BE CANONICAL AMONG ALL HONEST NODES WHO SAW THE BLOCK:
    PENDING = 3; // included in a block on the mesh. Pending processing by STF

// CANONICAL:
    PROCESSED = 4; // processed by the STF. Processing results are in the tx receipt

The PROCESSED state could potentially be dropped, on the grounds that the existence or absence of a receipt for this tx contains this information.

This makes sense.
Regarding the precise stages a tx passes once it gets into the mempool:

It is in the mempool, but not yet in any block
It is in a syntactically-valid block, but not yet in any block that is contextually valid with high confidence.
It is in a contextually-valid block with high confidence, but cannot be output by the consensus layer yet because other blocks in the same or earlier layers do not yet have high confidence (this can be caused, for example, if a previous layer's hare protocol terminates after the next layer's hare protocol terminates).
It was output by the consensus layer.
It is executed by the STF

I'm guessing  "mined into a block" means (2) and "added to the mesh" means (3)?
Note that while it's true that technically some time passes between (4) and (5), in our current implementation (in which every node runs the STF) I expect this to be a matter of milliseconds at most, so it won't really matter from a user perspective. (In a future version, if/when we support a separate role for state validation, the separation will be more meaningful.)
Regarding the layer number, I have no objection to including it as input to the STF. (indeed, as you say, in our protocol we will always process entire layers at a time, and rollbacks will always happen at layer granularity). However, the STF is well defined for executing one transaction at a time.
I would think the STF's API would contain (very loosely) something like the following:
ExecuteTX(tx): executes the transaction, updating the state returns TX receipt
Commit(): commits the current state to allow rollbacks to this point. Returns rollback handle
Rollback(handle): rollback the state to the given commit point.

The node will call ExecuteTX(tx) on every transaction output from the consensus layer, and Commit() at the end of every layer.
Of course, the "rollback handles" can just be the layer numbers, and if our code can be significantly simplified or optimized by allowing an ExecuteTX(tx[]) which receives an array, that's fine too --- but I'm not totally sure that's the case.

I think it makes sense to have a clean separation of the consensus and VM layers that Tal describes. The PROCESSED state could potentially be dropped, on the grounds that the existence or absence of a receipt for this tx contains this information, which gets us most of the way there.

Ok makes sense.

Node service: can tell us that it admitted a tx to the mempool, or rejected it (and on what grounds), and when it was mined into a block that was submitted to the network
Mesh service: can tell us when a tx was mined into a block, and when that block has been approved/added to the mesh.

We already defined NodeService.SubmitTransaction(Transaction) returns (TransactionState); so the node does return that. But I think this is immediate and represent the tx pre-filtering results. I think that the mesh service should be the one to return state of tx that didn't fail pre-filtering when submitted and not the node. It is the concern of the mesh service to report when a tx was included in the mesh.

Global state service: can tell us when a tx, and the block/layer containing it, was run through the VM, and the outcome of that execution (receipt)

Yes, I think we all agree to this.
I still believe we should go over with tal over the spec of the STF (including layer being an input) in the smip to confirm that the design is good and finally agree on definitions and the stf.

However, the STF is well defined for executing one transaction at a time.
I would think the STF's API would contain (very loosely) something like the following:

ExecuteTX(tx): executes the transaction, updating the state returns TX receipt
Commit(): commits the current state to allow rollbacks to this point. Returns rollback handle
Rollback(handle): rollback the state to the given commit point.

The node will call ExecuteTX(tx) on every transaction output from the consensus layer, and Commit() at the end of every layer.

The problem with this definition of the STF is that it does not account for fees and rewards. This is why I keep saying the STF is not well-defined for executing one transaction at a time. In the protocol as I understand it, you cannot just execute one transaction in isolation, because fees from all of the transactions in a layer are pooled and divided up as part of the processing of a layer. (This is how it was explained to me, and how it was documented. If this is not the case, let's talk about it.)

It is all pretty well defined here: https://github.com/spacemeshos/SMIPS/issues/13 -> The STF.

The consensus layer outputs a list of sorted de-duplicated tx for processing - tx input list per layer.
The STF executes these TX in the order in the list, updating global state and producing receipt for each tx. The STF input is this list and the layer #, so it can put that layer # in the produced receipts. If a tx fails, the accounts and app state need to be rolled back - this is obvious but gas may be paid in some cases.
When tx processing done, a bunch of other things are computed - e.g. rewards and tx fees are distributed between the miners that included the txs processed by the stf in a block they published in the layer - see the smip.
No other code can modify global state - by definition the STF executes all global state modifying code.

I agree with one caveat:

The STF executes these TX in the order in the list, updating global state and producing receipt for each tx

As Tal pointed out, global state is not really updated after each individual tx is executed. That doesn't really happen until commit is called at the end of the layer.

It depends what you call the 'global state'. As we said above it includes the accounts and apps state db + all the side-effect of tx execution (receipts and events). So, the global state is updated as each tx is executed and ofcourse if there's an exception that it is rolled back. But it has to be understood that transaction n gets as input the global state after transaction n-1 has been executed as n-1 may have changed an account or app state that is input or ouput of tx n. There is no other way to execute these transactions. So commit, does need to happen after each executed tx and roll-back means not updating app or account state in the db in case of tx exception. So the global state is actually updated after each tx is executed but as Anton mentioned this can be an in-memory representation that is committed to storage only once when the last tx was processed.

It is impossible to execute correctly n txs that have shared input and some shared output using an initial state s. They must be executed in steps otherwise execution can't be correct. e.g. tx n-1 deplated an account balance used as input to tx n... tx n should fail as its input is the global state after tx n-1 was executed.

Maybe it will all be better if we can review the definition in the smip and argue if any specific requirement there needs to be changed and not discuss this generally...

By definition of the STF, I had something more like this in mind: https://community.spacemesh.io/t/formal-specification-of-stf/88.

@lrettig Good catch --- You're right that the fees and rewards do need more information than just the transactions. However, this doesn't mean that transactions aren't executed one at a time --- it just means that the STF needs to get some more information from the consensus layer (in particular, it needs to know the set of blocks in each layer, and the assignment of transactions to blocks, including repeated transactions).
The most general way to do this is to have the consensus layer give "auxiliary" information to the STF. Basically, a call such as:
NewAuxInfo(auxinfo): doesn't return anything

This would be called before processing the first transaction for a layer (with "auxinfo" being the set of blocks in the layer).  We might also include in AuxInfo information about published ATXs (e.g., if publishing an ATX gives a reward even without generating a block).
Note that in our case the rewards and fees only apply to future layers, so the STF doesn't need the set of blocks for layer i in order execute the transactions in layer i.

Note that in our case the rewards and fees only apply to future layers, so the STF doesn't need the set of blocks for layer i in order execute the transactions in layer i.

Makes sense - as long as we don't allow the VM to supply a running smart contract with any additional, contextual info based on the block that a tx lives in, e.g., a blockhash (as in ethereum).

Right. Although as I wrote it, the STF actually does have this information.  Is there a good use-case for letting smart contracts have this additional info?

Is there a good use-case for letting smart contracts have this additional info?

I think we all agree that smart contracts need access to global state: reading and writing storage, reading balances, reading code, etc.
For mesh data, it's not so immediately obvious how it could be useful. Off the top of my head, a lot of eth smart contracts use block hash as a first order source of randomness. There are also Ewasm host functions for block number, difficulty, block gas limit, etc. - allowing Spacemesh smart contracts to read, say, number of blocks per layer, or number of transactions per block would allow some degree of introspection along these same lines.

It is super important for smart contract methods implementation to have access to layer number as part of the context. Many use cases that involve time-lock and changes of data over time from creation - e.g. NFT evolution. I bet that every non trivial eth game and NFT is using block number at least once. It is also critical for DEFI apps such as token auctions. So layer # is the abs minimum we know we need in the context. Additional mesh info is NTH but not MH.

@lrettig Good catch --- You're right that the fees and rewards do need more information than just the transactions. However, this doesn't mean that transactions aren't executed one at a time --- it just means that the STF needs to get some more information from the consensus layer (in particular, it needs to know the set of blocks in each layer, and the assignment of transactions to blocks, including repeated transactions).
The most general way to do this is to have the consensus layer give "auxiliary" information to the STF. Basically, a call such as:
NewAuxInfo(auxinfo): doesn't return anything

This would be called before processing the first transaction for a layer (with "auxinfo" being the set of blocks in the layer). We might also include in AuxInfo information about published ATXs (e.g., if publishing an ATX gives a reward even without generating a block).

It doesn't really matter - what matter is what the overall set of inputs that the STF needs and it includes the layer blocks.

Note that in our case the rewards and fees only apply to future layers, so the STF doesn't need the set of blocks for layer i in order execute the transactions in layer i.

Here again Tal - can you please review the specific proposed definition of the STF in the SMIP? It states clearly how rewards are computed and when. The current proposal is for the STF that works on txs from layer n to also compute the rewards for smeshers for layer n. It makes sense because it computes the total TX fees charged in the TX in layer n when it processes the transactions and can assign it as a step after tx processing (as proposed - please read again). If we want to change this then we need clear definitions and specs regarding to what layer does the STF of layer n compute the rewards from in case we want it to be different than what is proposed in the smip. Note, that if for some reason we need to change the rewards definition of the STF then it will need as input all the blocks of the layer != n that it needs to compute the rewards for. We must converge and have a good spec that we all feel good about but it feels like information that is currently written there is ignored or general statements are made that contradict it without an alternative proposal. So, if we need to change something from the current proposal for the stf. e.g. how rewards and tx fee are computed then we need alternative concrete proposals to consider beyond high-level statements. Can we please have that?

Can we take the conversation thread about STF definition to the SMIP thread? https://github.com/spacemeshos/SMIPS/issues/13

Sure and it will be helpful if any new comments can propose changes to the current stf high-level definition in the smip so we can converge.

Is this issue still relevant? What did we agree that needed to be changed if anything?

Reviewing where we're at right now in the API, we have TransactionState (in the TransactionService) which tracks the tx "journey" across concerns but knows nothing about the STF:
https://github.com/spacemeshos/api/blob/11c8309f8ef4341097ee99b33830264a8511d16c/proto/spacemesh/v1/tx_types.proto#L40-L55
Separately, we have the TransactionReceipt (in the GlobalStateService) that contains the results of the STF processing
https://github.com/spacemeshos/api/blob/11c8309f8ef4341097ee99b33830264a8511d16c/proto/spacemesh/v1/types.proto#L45-L61
To answer the initial questions I posed, above:

was the transaction "effective" or not (Iddo's term, i.e., syntactically valid)

All transactions in spacemesh are syntactically valid. Any transaction with a TransactionResult >= TRANSACTION_RESULT_EXECUTED was "effective" in the sense that it caused gas to be spent and/or value to be transferred.

did the transaction succeed or fail?

Yes iff TransactionResult == TRANSACTION_RESULT_EXECUTED

was a fee charged for it or not?

See TransactionResult.fee
Does this all sound correct?

Does this all sound correct?

I'm taking no response to indicate assent. Closing this as no further action is pending here. Please reopen if I've missed something.

Since the job of the API will be only to deliver the binary transaction - I think the whole conversation regarding the scheme isn't relevant. The decoding of a Receipt into a JSON will happen inside the smapp / explorer using the svm_codec.wasm


Publish latest docker images for native-image proxy

Add docker image publishing for native-image proxies to the circleci build, for pushes to master. This should work... but will need to check once merged to master.

Will merge and check how this goes on master.

Latest docker images for native-image proxy with dev-mode, in-memory, postgres, and cassandra all publishing automatically now.


docs(migration): add table with docker stats metrics mapping

Could we add mapping to sumologicschema as part of this PR, or write some config to do this for customers? Should be pretty easy task and customers won't have to migrate their dashboards right now

We could, but is it worth it? I mean, for other sources there would be probably a similar situation, so do we want to add all that there?

We should ensure that switching from Installed Collector to OTC will be as seamless as possible

@sumo-drosiek I added the subprocessor, please review.


[SMP-2368] WIP SMP 0.12.0 release notes

WIP SMP 0.12.0 release notes
RN list: https://harness.atlassian.net/wiki/spaces/PD/pages/21590474754/Release+Notes+tickets+for+0.12.0+release
For reviewers, a preview is available.
What Type of PR is This?

[ ] Issue
[x] Feature
[ ] Maintenance/Chore

If tied to an Issue, list the Issue(s) here:

Issue(s)

House Keeping
Some items to keep track of. Screen shots of changes are optional but would help the maintainers review quicker.

[x] Tested Locally
[ ] Optional Screen Shoot.

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step behind the Harness VPN and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app. Current Draft URL is: https://658483121eb6072f1ec8b314--harness-developer.netlify.app

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step behind the Harness VPN and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app. Current Draft URL is: https://6584b2b92473eb4aa587ebc5--harness-developer.netlify.app

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step behind the Harness VPN and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app. Current Draft URL is: https://6584e0974f641568b803174d--harness-developer.netlify.app

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step behind the Harness VPN and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app. Current Draft URL is: https://6584e3e358d63a693b20bad4--harness-developer.netlify.app

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step behind the Harness VPN and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app. Current Draft URL is: https://6584e7a8b2a7d76a6b4e8d26--harness-developer.netlify.app

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step behind the Harness VPN and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app. Current Draft URL is: https://6585dab430feee0436925a76--harness-developer.netlify.app

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step behind the Harness VPN and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app. Current Draft URL is: https://6585e1f356d734074bd1a19f--harness-developer.netlify.app

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step behind the Harness VPN and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app. Current Draft URL is: https://6585e276b8d635072dd203a3--harness-developer.netlify.app

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step behind the Harness VPN and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app. Current Draft URL is: https://658af15eb2a7d7b4494e8e96--harness-developer.netlify.app

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step behind the Harness VPN and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app. Current Draft URL is: https://658b2e8a727b008fcd888adc--harness-developer.netlify.app

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step behind the Harness VPN and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app. Current Draft URL is: https://658de6f86ac4f12b893bce18--harness-developer.netlify.app

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step behind the Harness VPN and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app. Current Draft URL is: https://6598938f3f31ad6a3a1d9411--harness-developer.netlify.app

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step behind the Harness VPN and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app. Current Draft URL is: https://659c356a59b43b033093a84a--harness-developer.netlify.app


RuntimeError: mat1 and mat2 shapes cannot be multiplied (154x1024 and 768x320)

Hi there, great package. I'm really enjoying it so far.
I'm getting this error on the sd_solver node with controlnet.
Both my sd_latent_noise_generate and sd_image_to_points nodes are set to 512x512. The conditioning seems to be irrelevant as does the controlnet model. Skipping controlnet and using the image as an input for the scheduler's latents works as expected.
Happy to upload the hip if it helps.
Windows 10, Houdini 19.5
Invalid source [/obj/geo1/sd_solver1/solver](node:/obj/geo1/sd_solver1/solver)
Error: Python error: Traceback (most recent call last):
File "<stdin>", line 46, in <module>
File "C:/Users/adam.krebs/Houdini Packages/MLOPs-main/scripts/python\sdpipeline\image_solve.py", line 72, in run
down_block_res_samples, mid_block_res_sample = controlnet(latent_model_input, t.to(torch.float16), encoder_hidden_states=text_embeddings, controlnet_cond=controlnet_conditioning_image, conditioning_scale=controlnet_scale, return_dict=False,)
File "C:/Users/ADAM~1.KRE/Dropbox/PC(2)~1/DOCUME~1/houdini19.5/scripts/python\torch\nn\modules\module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "C:/Users/ADAM~1.KRE/Dropbox/PC(2)~1/DOCUME~1/houdini19.5/scripts/python\diffusers\pipelines\stable_diffusion\pipeline_stable_diffusion_controlnet.py", line 134, in forward
return_dict,
File "C:/Users/ADAM~1.KRE/Dropbox/PC(2)~1/DOCUME~1/houdini19.5/scripts/python\torch\nn\modules\module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "C:/Users/ADAM~1.KRE/Dropbox/PC(2)~1/DOCUME~1/houdini19.5/scripts/python\diffusers\models\controlnet.py", line 529, in forward
cross_attention_kwargs=cross_attention_kwargs,
File "C:/Users/ADAM~1.KRE/Dropbox/PC(2)~1/DOCUME~1/houdini19.5/scripts/python\torch\nn\modules\module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "C:/Users/ADAM~1.KRE/Dropbox/PC(2)~1/DOCUME~1/houdini19.5/scripts/python\diffusers\models\unet_2d_blocks.py", line 870, in forward
cross_attention_kwargs=cross_attention_kwargs,
File "C:/Users/ADAM~1.KRE/Dropbox/PC(2)~1/DOCUME~1/houdini19.5/scripts/python\torch\nn\modules\module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "C:/Users/ADAM~1.KRE/Dropbox/PC(2)~1/DOCUME~1/houdini19.5/scripts/python\diffusers\models\transformer_2d.py", line 270, in forward
class_labels=class_labels,
File "C:/Users/ADAM~1.KRE/Dropbox/PC(2)~1/DOCUME~1/houdini19.5/scripts/python\torch\nn\modules\module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "C:/Users/ADAM~1.KRE/Dropbox/PC(2)~1/DOCUME~1/houdini19.5/scripts/python\diffusers\models\attention.py", line 316, in forward
**cross_attention_kwargs,
File "C:/Users/ADAM~1.KRE/Dropbox/PC(2)~1/DOCUME~1/houdini19.5/scripts/python\torch\nn\modules\module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "C:/Users/ADAM~1.KRE/Dropbox/PC(2)~1/DOCUME~1/houdini19.5/scripts/python\diffusers\models\attention_processor.py", line 248, in forward
**cross_attention_kwargs,
File "C:/Users/ADAM~1.KRE/Dropbox/PC(2)~1/DOCUME~1/houdini19.5/scripts/python\diffusers\models\attention_processor.py", line 375, in __call__
key = attn.to_k(encoder_hidden_states)
File "C:/Users/ADAM~1.KRE/Dropbox/PC(2)~1/DOCUME~1/houdini19.5/scripts/python\torch\nn\modules\module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "C:/Users/ADAM~1.KRE/Dropbox/PC(2)~1/DOCUME~1/houdini19.5/scripts/python\torch\nn\modules\linear.py", line 114, in forward
return F.linear(input, self.weight, self.bias)
RuntimeError: mat1 and mat2 shapes cannot be multiplied (154x1024 and 768x320)
.

Any advice? Thanks!

mlops controlnet error.zip

What is your MLOPS_MODEL set to? For ControlNet it needs to be 1.5, eg: runwayml/stable-diffusion-v1-5
Check out the example file > controlnet! :)

Ah, I'm just going off of Mo's youtube video which showed it using $MLOPS_MODEL. I didn't change anything about the default models.
    MLOPS                                  C:/Users/adam.krebs/Houdini Packages/MLOPs-main
    MLOPS_MODEL                            C:/Users/adam.krebs/Houdini Packages/MLOPs-main/checkpoints/stabilityai-_-stable-diffusion-2-1
    MLOPS_MODELS                           C:/Users/adam.krebs/Houdini Packages/MLOPs-main/checkpoints

If I change the Unet Model, Scheduler Model and ControlNet model to runwayml/stable-diffusion-v1-5, I get:
OSError: Error no file named config.json found in directory C:/Users/adam.krebs/Houdini Packages/MLOPs-main/checkpoints/runwayml-_-stable-diffusion-v1-5.

Here's the directory listing:
C:\Users\adam.krebs\Houdini Packages\MLOPs-main\checkpoints\runwayml-_-stable-diffusion-v1-5>dir
 Volume in drive C is OS
 Volume Serial Number is ###

 Directory of C:\Users\adam.krebs\Houdini Packages\MLOPs-main\checkpoints\runwayml-_-stable-diffusion-v1-5

04/26/2023  01:14 PM    <DIR>          .
04/26/2023  01:14 PM    <DIR>          ..
04/26/2023  01:14 PM    <DIR>          feature_extractor
04/26/2023  01:14 PM               543 model_index.json
04/26/2023  01:14 PM    <DIR>          safety_checker
04/26/2023  01:14 PM    <DIR>          scheduler
04/26/2023  01:14 PM    <DIR>          text_encoder
04/26/2023  01:14 PM    <DIR>          tokenizer
04/26/2023  01:14 PM    <DIR>          unet
04/26/2023  01:14 PM    <DIR>          vae

FWIW I was able to run the example file and use the Multi_ControlNet example, but every time I change an input it opens the "Fetching 15 files" dialog

Is this still unresolved? If so please re-open.

I'm still seeing it, but I ended up just forking the example per your advice. Thanks


Implement tests for getExpirationTime() to cover null and invalid file Id

Added comprehensive tests for getExpirationTime() to handle negative scenarios:

Handles null FileId inputs.
Validates behavior for invalid or non-existent FileIds.

Closes #106

@manishdait thank you very much for the contribution!


Audit log when MFA has been used (success and fail)

We need to have a full audit log of user activities. This is already the case for "logged in" as well as "failed login" events. But it doesn't specifically register this in the context of backup codes. We should provide event hooks for https://github.com/silverstripe/silverstripe-auditor to pick up (and implement those listeners in the auditlog module)
ACs

MFA activities are logged to a point that it's useful for an auditing perspective
logged in an access restricted, append-only external logging service which is already part of CWP

Notes:

Initial list of MFA actions (may not definitive):
Success login, which method (including recovery codes)
Failed login, which method was attempted (including recovery codes)
Compounding failed attempts
Admin functions, like reseting backup codes, adding or removing registered methods
Skipped registration
This should cater for backend logs and db work. We'll handle any CMS views of this information in separate issue

@Firesphere has started a PR for the auditor module at https://github.com/silverstripe/silverstripe-auditor/pull/27


Rollback previous commit

https://github.com/ponylang/ponyc/commit/bc927a619a139cd1270a723613652b4506005ae1 should be undone as I forgot about an important protection that means that the edge case I identified was incorrect. Muted actors don't mark their queue as empty so that edge case can't happen.

Actually the edge case is correct.


Adds a relation id parameter in get_requirer_csrs_with_no_certs

Description
Adds a relation id parameter in get_requirer_csrs_with_no_certs
Checklist

[x] My code follows the style guidelines of this project.
[x] I have performed a self-review of my own code.
[ ] I have made corresponding changes to the documentation.
[x] I have added tests that validate the behaviour of the software.
[x] I validated that new and existing unit tests pass locally with my changes.
[x] Any dependent changes have been merged and published in downstream modules.
[x] I have bumped the version of any required library.

Double approved!!


Need to document various ways to install packages

https://docs.microsoft.com/en-us/nuget/consume-packages/overview-and-workflow covers the workflow. We need the exact commands and UI ways of installing the packages documented at one place.

(Merging #554).
Also, this should be an install topic under Consume. Without it, the TOC looks like it's missing something. Should be a fuller scenario than the quickstart, perhaps adding multiple packages, one or two of which contains more than just a DLL.

https://docs.microsoft.com/en-us/nuget/consume-packages/ways-to-install-a-package


Release version 0.2.0

This is basically ready, except for the final nrf52832-hal 0.8 release. Two things left to do, before this can be released:

[ ] Upgrade nrf52832-hal dependency to version 0.8, preferably in a separate PR.
[ ] Rebase this PR and finalize the changelog update (commit is currently marked as WIP).

cc @jamesmunns

Back at you @hannobraun

The new version has been published.


Feature: framing headers control

The C@E platform supports manually-controlled framing headers (like Content-Length), but this is only stubbed out in Viceroy.
The implementation of this feature is quite subtle, due to interactions with Hyper. Ideally, we could share the code between Viceroy and production C@E by open-sourcing it.

Somewhat related to https://github.com/fastly/Viceroy/issues/19

Noting that we need this for our purposes too, we're setting content-length explicitly and use manual framing mode because of that.


Suavex rpc request library

Could we merge some of this with #57

merged this into #57 to replace the contents of the encoding methods in the Bundle lib

closing this issue by merge with #57


Cannot publish a container's port(s) to the host with --net option.

Please read this important information about creating issues.
If you are reporting a new issue, make sure that we do not have any duplicates already open. You can ensure this by searching the issue list for this repository. If there is a duplicate, please close your issue and add a comment to the existing issue instead.
If you suspect your issue is a bug, please edit your issue description to include the BUG REPORT INFORMATION shown below. If you fail to provide this information within 7 days, we cannot debug your issue and will close it. We will, however, reopen it if you later provide the information.
This is an automated, informational response.
Thank you.
For more information about reporting issues, see https://github.com/docker/docker/blob/master/CONTRIBUTING.md#reporting-other-issues

BUG REPORT INFORMATION
Use the commands below to provide key information from your environment:
docker version:
docker info:
uname -a:
Provide additional environment details (AWS, VirtualBox, physical, etc.):
List the steps to reproduce the issue:
1.
2.
3.
Describe the results you received:
Describe the results you expected:
Provide additional info you think is important:
----------END REPORT ---------
#ENEEDMOREINFO

When using --net=container you have to publish ports from the container
that actually owns the namespace.  We filed this, too and we told "working
as intended".
On Mon, Oct 5, 2015 at 8:05 AM, Bear Huang<EMAIL_ADDRESS>wrote:

When I run
docker run -p 8000:8000 --net=container:81cc21830839 centos /bin/bash
it won't publish 8000 to host, nor --net=host.
docker version
Client version: 1.7.1
Client API version: 1.19
Package Version (client): docker-1.7.1-108.el7.centos.x86_64
Go version (client): go1.4.2
Git commit (client): 3043001/1.7.1
OS/Arch (client): linux/amd64
Server version: 1.7.1
Server API version: 1.19
Package Version (server): docker-1.7.1-108.el7.centos.x86_64
Go version (server): go1.4.2
Git commit (server): 3043001/1.7.1
OS/Arch (server): linux/amd64
docker info
Containers: 17
Images: 426
Storage Driver: devicemapper
Pool Name: docker-8:1-525892-pool
Pool Blocksize: 65.54 kB
Backing Filesystem: extfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 12.7 GB
Data Space Total: 107.4 GB
Data Space Available: 9.442 GB
Metadata Space Used: 19.94 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.128 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.93-RHEL7 (2015-01-28)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.10.0-229.7.2.el7.x86_64
Operating System: CentOS Linux 7 (Core)
CPUs: 2
Total Memory: 3.362 GiB
Name: gicc-bearh-1
ID: 5PEC:WZUG:XD3N:W36D:NAM5:362V:J5KD:PGE4:OVLS:DUE7:4SNN:WPJQ
—
Reply to this email directly or view it on GitHub
https://github.com/docker/docker/issues/16765.

Sorry, I missed uname -a:
Linux gicc-bearh-1 3.10.0-229.7.2.el7.x86_64 #1 SMP Tue Jun 23 22:06:11 UTC 2015 x86_64 x86_64 x86_64 GNU/Linu
So, I cannot publish any additional ports when using --net option? Would this be possible a new feature?
Thanks.

My issues was closed "working as intended", though I disagreed.
On Mon, Oct 5, 2015 at 8:22 AM, Bear Huang<EMAIL_ADDRESS>wrote:

Sorry, I missed uname -a:
Linux gicc-bearh-1 3.10.0-229.7.2.el7.x86_64 #1 SMP Tue Jun 23 22:06:11
UTC 2015 x86_64 x86_64 x86_64 GNU/Linu
So, I cannot publish any additional ports when using --net option? Would
this be possible a new feature?
Thanks.
—
Reply to this email directly or view it on GitHub
https://github.com/docker/docker/issues/16765#issuecomment-145567724.

/cc @mavenugo @mrjana PTAL

ping @mavenugo @mrjana @aboch @sanimej

I'm also on same issue and quite disappointed. I'll outline my case and maybe it would make any sense:
I need to set up a build server which fetches sources over VPN connection. For some reasons I cannot (or don't want) to install VPN client on host. So my intention was to have a vpn-client as a docker container, then run build-server container with --net=vpn-client to make build server access sources code and to publish ports on build-server to access it's web-ui.
Is there an 'intended' way to achieving this in docker?


Add Documentation for remult-cli in the Official Remult Docs

remult-cli is a powerful command-line utility designed to streamline the generation of Remult entities from a PostgreSQL database. Currently, it appears that there is no mention of remult-cli in the official documentation. I propose that we add a section dedicated to this CLI tool to help users understand its functionality and how to integrate it into their Remult projects.
Suggested Locations:
As a subsection under the "Getting Started" guide
As a standalone guide within the "Integrations" section
@jycouet @noam-honig, I'd love to hear your thoughts on where this information would be most effectively placed. Thank you!

Hi @Yedidyar pls check out #237.
Do you have any suggestions before I publish?

@yoni-rapoport looks good


Sources that @Autowire Source Cannot be Aggregated

I receive an error message "Field channels in org.springframework.cloud.stream.app.tcp.source.TcpSourceConfiguration required a single bean, but 2 were found: 1- Source 2- .Processor

See http://stackoverflow.com/questions/42346872/how-to-handle-message-using-spring-cloud-stream-app-starter-tcp/42347372#comment71855225_42347372
One solution might be to configure the output channel by name so it is bound to the actual channel at runtime.

Could this possibly be caused by 2 components/apps in the same package? - see https://github.com/spring-cloud/spring-cloud-stream/issues/785. That is an issue with classpath scanning.

Ah, could be; I didn't ask him that; I just pushed a PR for tcp (it was missing a test too).

Although, he was aggregating a custom app with the TCP source so it would be odd if he used the same package.

Thanks @garyrussell - looks like this is an offshoot of https://github.com/spring-cloud/spring-cloud-stream/commit/6c861135a3d4b4efb42d594af5df74c665598c8b - we now register the bindings as beans in the parent, but obviously that throws off injection in the children (seemed simpler to do that registration at the time, but the flipside is that this is causing injection issues). Seems easy to fix if we just query the proper child context for retrieving the binding.

Hi, I'm the one who have created the stackoverflow post. Here's my configurations. I hope it can helps!
MessagingTcpDasApplication.java
package tcp.das;
import org.springframework.cloud.stream.app.tcp.source.TcpSourceConfiguration;
[...]
@SpringBootApplication
@ComponentScan("org.springframework.cloud.stream.app.tcp")
public class MessagingTcpDasApplication {
   public static void main(String[] args) {
   	new AggregateApplicationBuilder()
   		.from(TcpSourceConfiguration.class)
   		.to(DasDataHandler.class).run(args);
   }
}

DasDataHandler.java
package tcp.das.handler;
[...]
@EnableBinding(Processor.class)
public class DasDataHandler {
	  @Transformer
	  public String handle(String inStream) {
		  System.out.println("test"+inStream);
	    return "test";
	  }
}

application.properties
spring.main.web-environment=false
tcp.port=13100
spring.cloud.stream.bindings.output.destination= iwls.newData

build.gradle
dependencies {
	compile('org.springframework.cloud:spring-cloud-starter-stream-rabbit')
	compile('org.springframework.cloud.stream.app:spring-cloud-starter-stream-source-tcp:1.1.1.RELEASE')
	compile('org.springframework.boot:spring-boot-devtools')
	testCompile('org.springframework.boot:spring-boot-starter-test')
}

Stack trace massage

APPLICATION FAILED TO START

Description:
Field channels in org.springframework.cloud.stream.app.tcp.source.TcpSourceConfiguration required a single bean, but 2 were found:

org.springframework.cloud.stream.messaging.Source: defined in null
tcp.das.handler.DasDataHandler_1.org.springframework.cloud.stream.messaging.Processor: a programmatically registered singleton

Action:
Consider marking one of the beans as @Primary, updating the consumer to accept multiple beans, or using @Qualifier to identify the bean that should be consumed

Thanks @rattek,
This confirms my comment on https://github.com/spring-cloud/spring-cloud-stream/issues/819#issuecomment-281195091 - we have a bug we need to fix.


[Bug]: Storage Write API client hanging forever on shutdown

What happened?
The write client hangs forever on shutdown
Issue Priority
Priority: 0
Issue Component
Component: io-java-gcp

Can you link to the failing test for posterity? And this P0 is now closed since you mitigated, correct?


[feaLib] ignore statements without marked glyphs

Another instance where it feels like there is a discrepancy between the spec and feaLib's behaviour.
Per the spec (5.f.ii. Specifying exceptions to the Chain Sub rule):

The backtrack and lookahead sequences may be omitted, but there must be at least one marked glyph or glyph class.

However the GSUB_5_formats.fea test case consists of ignore sub statements that do not have any marked glyphs or glyphclasses.
My understanding is that in the absence of any marked glyphs, feaLib chooses to interpret all glyphs/classes as belonging to the input sequence.
So: Is this intentional? If so, is there any particular rationale?
cc @simoncozens, who added this particular test case.

Interesting; you're correct, given the input,
feature cool {
    ignore sub e x;
} cool;

makeotf produces a type-6 lookup where 'e' is the input sequence and 'x' is lookahead. feaLib, on the otherhand, produces a type-6 lookup where both 'e' and 'x' are part of the input sequence.
I'm not really sure what to think about this? I suppose for my purposes it makes more sense to copy the feaLib behaviour, if it is intentional, but I think I would also print a warning indicating that this syntax is ambiguous.
also for further fun: if I change the above to,
feature cool {
    ignore sub e x;
    sub y e' by f;
} cool;

then the ignore statement gets promoted to type 6, but then the second statement seems to get added to the same subtable, which is probably a bug? the ttx looks like,
<LookupList>
      <!-- LookupCount=2 -->
      <Lookup index="0">
        <LookupType value="6"/>
        <LookupFlag value="0"/>
        <!-- SubTableCount=1 -->
        <ChainContextSubst index="0" Format="1">
          <Coverage>
            <Glyph value="e"/>
          </Coverage>
          <!-- ChainSubRuleSetCount=1 -->
          <ChainSubRuleSet index="0">
            <!-- ChainSubRuleCount=2 -->
            <ChainSubRule index="0">
              <!-- BacktrackGlyphCount=0 -->
              <!-- InputGlyphCount=2 -->
              <Input index="0" value="x"/>
              <!-- LookAheadGlyphCount=0 -->
              <!-- SubstCount=0 -->
            </ChainSubRule>
            <ChainSubRule index="1">
              <!-- BacktrackGlyphCount=1 -->
              <Backtrack index="0" value="y"/>
              <!-- InputGlyphCount=1 -->
              <!-- LookAheadGlyphCount=0 -->
              <!-- SubstCount=1 -->
              <SubstLookupRecord index="0">
                <SequenceIndex value="0"/>
                <LookupListIndex value="1"/>
              </SubstLookupRecord>
            </ChainSubRule>
          </ChainSubRuleSet>
        </ChainContextSubst>
      </Lookup>
      <Lookup index="1">
        <LookupType value="1"/>
        <LookupFlag value="0"/>
        <!-- SubTableCount=1 -->
        <SingleSubst index="0">
          <Substitution in="e" out="f"/>
        </SingleSubst>
      </Lookup>
    </LookupList>

In this particular case, I think you are better off following the spec and erroring here. On feaLib side, we need to figure out if feaLib was always not following makeotf here or one if the two changed its behavior and act accordingly.

but then the second statement seems to get added to the same subtable, which is probably a bug?

Does it matter from the shaping engine prescriptive whether it is one subtable or two? I’d assume it gives the same output regardless.

I think it does in the case of ignore statements: an ignore statement works by inserting a subtable that matches the input, but has no associated lookups. When evaluating the lookup, the shaping engine stops at the first matching subtable.

I test fonts produced by feaLib and makeotf and HarfBuzz gives me the same output:
feature rlig {
    ignore sub e x;
    sub y e' by f;
} rlig;

$ hb-shape a1.ttf "ye yex yea" --no-clusters --no-positions
[y|f|space|y|e|x|space|y|f|a]
$ hb-shape a2.ttf "ye yex yea" --no-clusters --no-positions
[y|f|space|y|e|x|space|y|f|a]

https://github.com/fonttools/fonttools/pull/2950

So I remain really confused by that example I posted, it really doesn't seem to compile how I expect and I have a nagging suspicion that it is producing the correct result accidentally, although I don't trust my intuition very much here. I'm going to look at it with a fresh brain tomorrow, and will open a separate issue if I identify a possible problem.


enhance Q1+ metadata

For Q0 we ask some mandatory metadata but for Q1+ we rely entirely on STAC (so the metadata that we have for Q0 is not mantained).
Should we require a README.md file for all datasets? Or should we enforce metadata fields in the STAC catalog (can do this for Q2 with our own extension, but not for Q1, so maybe the best option is to ask for README.md always, this also simplifies upgrading from Q0 to Q1 and to Q2). Any thoughts @Patrick1G ?

improve Q1+ overall, using https://github.com/earthpulse/eotdl/issues/122 as use case


Library covers are very jagged after update

Steps to reproduce

Have older version installed (0.16.5 in my case). Device doesn't seem to matter.
Update to new version
Go to library

Expected behavior
Library covers should appear clearly like in previous version:

Actual behavior
Coverart is jagged, as if antialiasing in a game is turned off.

Refreshing the covers doesn't fix it and happens on multiple devices.
Crash logs
No response
Mihon version
0.17.0
Android version
2 Android 14s and 1 Android 11
Device
Asus Zenfone 9, Samsung Galaxy Tab Active 5, Boox Leaf
Other details
No response
Acknowledgements

[X] I have searched the existing issues and this is a new ticket, NOT a duplicate or related to another open or closed issue.
[X] I have written a short but informative title.
[X] I have gone through the FAQ and troubleshooting guide.
[X] I have updated the app to version 0.17.0.
[X] I have updated all installed extensions.
[X] I will fill out all of the requested information in this form.

Could be that you had higher quality covers cached but now the covers are lower quality, which would be either a source or extension issue (some have a setting for cover quality).

Shouldn't refreshing the covers fix it then? Most of them are MD btw, with quality set to original.

So, I looked into it a bit more after BrutuZ mentioned extension settings. Simply refreshing didn't fix it. Instead, it seems like the original quality setting in the Mangadex extension caused this. Setting it to medium quality fixes it. So maybe rather than a too low quality, the higher quality thumbnails can't be rendered properly anymore in Mihon 0.17.0, hence the 'no anti-aliasing' effect that I described?


Pass custom error data

This is more of an idea than a finalized PR - I'm not sure if this approach is the best one, and supporting it fully would require changes to the other community Strategies.
I have been trying to get custom data passed through when throwing an error inside an authentication call (same as this discussion).
Usually in these situations, I create a custom error class where I can add data to an error message:
export class FancyError extends Error {
  constructor(message, data) {
    super(message);
    this.data = data;
  }
}

throw new FancyError('Uh oh', {moreData:'yes'})

try{
  throw
} catch(error) {
console.log(error.message) // Uh oh
console.log(error.data) // {moreData:'yes'}
}

The problem is that remix-auth and at least remix-auth-form pass errors around by calling new AuthorizationError(error.message) instead of passing through the original error object. This strips away any custom functionality.
This PR adds a simple errors[] list to the AuthorizationError class, and allows passing an errors list into authenticate.failure.
To actually make use of this, you'd have to throw new AuthorizationError(message,errors) inside your authentication call, and update your strategy to pass error.errors as the final parameter of failure.
Here's remix-auth-form with that enabled:
export class FormStrategy extends Strategy {
  name = 'form';

  async authenticate(request, sessionStorage, options) {
    let form = await request.formData();

    let user;
    try {
      user = await this.verify({ form, context: options.context });
    } catch (error) {

      let message = error.message;
      return await this.failure(
        message,
        request,
        sessionStorage,
        options,
        error.errors,
      );
    }

    return this.success(user, request, sessionStorage, options);
  }
}

And how I'd use it in the app:
authenticator.use(
  new FormStrategy(async ({ form }) => {
    const api = new Api();

    let email = form.get('email');
    let password = form.get('password');

    try {
      return await api.login(email, password);
    } catch (error) {
      // this assumes my api is sending in error.data.errors - you could set any list of strings as the second argument
      throw new AuthorizationError(error.message, error.data?.errors);
    }
  }),
  'user-pass'
);

Forgot to mention, but the errors object is completely optional in failure and AuthorizationError - so there'd be no breaking changes.

If this was accepted, I'd be happy to file PRs with the strategies listed in the discussion 👍🏼

This was added in v3.4.0


Top Level Await Support on Server Side

This bumps the base required Node version to 14.8 as to support top level await in Remix build.
Closes: #4348

[ ] Docs
[ ] Tests

Testing Strategy: TBA

I think the reason to target 14 instead of a newer version was to support more deployment platforms, some Serverless platforms still don't support newer Node.js versions. It would be great to review this first.
I checked and Vercel and Netlify both support v16, if the rest also support v16 maybe Remix could upgrade the required Node version to v16.
Another thing to consider is that v14 is still under maintenance mode until the 2023-04-30, so this could be another reason to keep support for v14 until that version reach end of life.

@sergiodxa Granted that we're on security support I think it's be sensible to assume greater than 14.8 for SaaS platforms too. This is only a Node semver minor bump.
All secure or even supported installations of Node 14 would support top-level await.
Technically the only secure version is really Node 14.20.1 currently anyway.

esbuild does not appear to support top level await https://github.com/evanw/esbuild/issues/253


Link error when ‘#define HAVE_CIPHER_CHACHA20 0’

#define SQLITE3MC_USE_RAND_S 0
#define HAVE_CIPHER_AES_128_CBC 0
#define HAVE_CIPHER_AES_256_CBC 0
#define HAVE_CIPHER_CHACHA20 0
#define HAVE_CIPHER_SQLCIPHER 0
#define HAVE_CIPHER_RC4 0
#define HAVE_CIPHER_ASCON128 1
#define CODEC_TYPE CODEC_TYPE_ASCON128
Unresolved external '_chacha20_rng' referenced from SQLITE3MC.OBJ
Unresolved external '_STORE32_BE' referenced from SQLITE3MC.OBJ

#define SQLITE3MC_USE_RAND_S 0
#define HAVE_CIPHER_AES_128_CBC 0
#define HAVE_CIPHER_AES_256_CBC 0
#define HAVE_CIPHER_CHACHA20 0
#define HAVE_CIPHER_SQLCIPHER 0
#define HAVE_CIPHER_RC4 0
#define HAVE_CIPHER_ASCON128 1
#define CODEC_TYPE CODEC_TYPE_ASCON128
Unresolved external '_chacha20_rng' referenced from SQLITE3MC.OBJ Unresolved external '_STORE32_BE' referenced from SQLITE3MC.OBJ

The cipher scheme Ascon128 uses parts of the code of the ChaCha20 implementation, namely the random number generator. Therefore you get unresolved external references, if you disable the cipher ChaCha20.
I will adjust the code, so that the ChaCha20 implementation will be included, even if the cipher ChaCha20 was disabled.
For the time being you can enable the ChaCha20 cipher as a work around.

Commit 26c5c5bd3bd6fc8fe24caf647809891865e6e1b0 fixes the issue.

Thank you.


Added check for apex AMP and unit tests for Horovod + AMP

What does this PR do?
Fixes #3111.
Before submitting

[X] Was this discussed/approved via a Github issue? (no need for typos and docs improvements)
[X] Did you read the contributor guideline, Pull Request section?
[X] Did you make sure your PR does only one thing, instead of bundling different changes together? Otherwise, we ask you to create a separate PR for every change.
[ ] Did you make sure to update the documentation with your changes?
[X] Did you write any new necessary tests?
[X] Did you verify new and existing tests pass locally with your changes?
[X] If you made a notable change (that affects users), did you update the CHANGELOG?

PR review
Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.
Did you have fun?
Make sure you had fun coding 🙃

Looks like test failures are unrelated:
tests/core/test_results.py::test_result_obj_predictions_ddp_spawn


Extend command: m365 aad user list with --deleted flag

Usage
m365 aad user list  --deleted
Description
Extend command: m365 aad user list with an additional option --deleted to retrieve deleted users from the recycle bin.
Options

Option
Description

--deleted
Use to retrieve deleted users

Additional Info
Can be retrieved using a get on https://graph.microsoft.com/v1.0/directory/deletedItems/microsoft.graph.user, more background here: https://docs.microsoft.com/en-us/graph/api/directory-deleteditems-list?view=graph-rest-1.0&tabs=http

If possible, then I would like to work on this one.

All yours @Jwaegebaert! We appreciate your help! Let's keep the PRs separate. It's easier for us to review them that way, especially as they affect two different commands.

Okaj great, then I would like to start on this one if possible.

All yours @Jwaegebaert. Thank you! 👏


replace is not working

Hello Angel,
replace function is not working.
Example code from https://medoo.in/api/replace is executed as:
UPDATE "account" SET "type" = REPLACE("type", '', ''), "column" = REPLACE("column", '', '') WHERE "user_id" > 1000
So without values to replace.

Yes, it is a bug. It fixed on 0ae17978b1b8.

Thanks, can you create stable version 1.4.6?

@adminko-svk it will possibly be with v1.5 soon.


[Improvement] In StorageLayoutException.java and StorageLayoutException.java no passing final variable in format string

What would you like to be improved?
StorageLayoutException.java:22: error: [FormatStringAnnotation] All variables passed as @FormatString must be final or effectively final
super(message, args);
StorageLayoutException.java:34: error: [FormatStringAnnotation] All variables passed as @FormatString must be final or effectively final
super(cause, message, args);
See https://errorprone.info/bugpattern/FormatStringAnnotation for more information
How should we improve?
No response

@justinmclean I think this issue can be closed now. Because it has been fixed on https://github.com/datastrato/gravitino/pull/1746.

Closing as it has been fixed.

@justinmclean Maybe forgot to click the close button.


CNBC does not support HEAD requests

Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2017.09.11. If it's not, read this FAQ entry and update. Issues with outdated version will be rejected.

[x] I've verified and I assure that I'm running youtube-dl 2017.09.11

Before submitting an issue make sure you have:

[x] At least skimmed through the README, most notably the FAQ and BUGS sections
[x] Searched the bugtracker for similar issues including closed ones

Possibly related: #13222

What is the purpose of your issue?

[x] Bug report (encountered problems with youtube-dl)

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['http://www.cnbc.com/live-tv/the-profit/full-episode/top-10-rules-for-success/1015334979858', '--verbose', '--ap-mso', 'DTV', '--ap-username', 'PRIVATE', '--user-agent', 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36', '--hls-prefer-native']
Type TV provider account password and press [Return]:
[debug] Encodings: locale cp1252, fs mbcs, out cp437, pref cp1252
[debug] youtube-dl version 2017.09.11
[debug] Python version 3.4.4 - Windows-10-10.0.14393
[debug] exe versions: ffmpeg N-82324-g872b358, ffprobe N-82324-g872b358
[debug] Proxy map: {}
[generic]<PHONE_NUMBER>858: Requesting header
WARNING: Could not send HEAD request to http://www.cnbc.com/live-tv/the-profit/full-episode/top-10-rules-for-success/1015334979858: HTTP Error 404: Not Found
[generic]<PHONE_NUMBER>858: Downloading webpage
WARNING: Falling back on generic information extractor.
[generic]<PHONE_NUMBER>858: Extracting information
ERROR: Unsupported URL: http://www.cnbc.com/live-tv/the-profit/full-episode/top-10-rules-for-success/1015334979858
Traceback (most recent call last):
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmp24lzs8ny\build\youtube_dl\YoutubeDL.py", line 776, in extract_info
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmp24lzs8ny\build\youtube_dl\extractor\common.py", line 434, in extract
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmp24lzs8ny\build\youtube_dl\extractor\generic.py", line 2964, in _real_extract
youtube_dl.utils.UnsupportedError: Unsupported URL: http://www.cnbc.com/live-tv/the-profit/full-episode/top-10-rules-for-success/1015334979858

Culprit is the HEAD request:
wget --spider http://www.cnbc.com/live-tv/the-profit/full-episode/top-10-rules-for-success/1015334979858 -U "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36"
Spider mode enabled. Check if remote file exists.
--2017-09-13 00:41:17--  http://www.cnbc.com/live-tv/the-profit/full-episode/top-10-rules-for-success/1015334979858
Resolving www.cnbc.com (www.cnbc.com)... <IP_ADDRESS>
Connecting to www.cnbc.com (www.cnbc.com)|<IP_ADDRESS>|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
Remote file does not exist -- broken link!!!

wget --delete-after http://www.cnbc.com/live-tv/the-profit/full-episode/top-10-rules-for-success/1015334979858 -U "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36"
--2017-09-13 00:50:15--  http://www.cnbc.com/live-tv/the-profit/full-episode/top-10-rules-for-success/1015334979858
Resolving www.cnbc.com (www.cnbc.com)... <IP_ADDRESS>
Connecting to www.cnbc.com (www.cnbc.com)|<IP_ADDRESS>|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: '1015334979858'

1015334979858                                         [ <=>                                                                                ] 129.75K  --.-KB/s    in 0.01s

2017-09-13 00:50:15 (9.06 MB/s) - '1015334979858' saved [132869]

Removing<PHONE_NUMBER>858.

bump, any ideas what could fix this?

@keithah This line needs to be changed

any update on this? i am still getting the same error with CNBC videos.

@cookieguru Sorry for the late response, but what do I change that line too?

@keithah If you're looking for a quick and dirty fix then change HEAD to GET here.

looks like you could just use this pull request to achieve what you want as well:
https://github.com/rg3/youtube-dl/pull/18086

Unfortunately, still fails:
https://gist.github.com/keithah/f249d0282f74be1415db2ef3444a9309
So it's not just the HEAD requests.

@keithah Look at line 3.  It's not using the CNBC extractor because said extractor doesn't have support for that URL pattern.  Updating the regex on line 41 should do it.

some progress!
https://gist.github.com/keithah/3788844981c5586466807b80e72d807a
Now failing at the info extractor..

Change line 62 to this:
            r'data-mpx-id=["\'](\d+)', webpage, display_id,

Although to make this worthy of a PR it should actually be a new class

@cookieguru Still no luck, now getting a 404 on a totally valid link, but farther!
https://gist.github.com/keithah/72930668ef9793b1cca6c7baab6037da
Tried a video which shouldn't require auth too:
https://www.irccloud.com/pastebin/ImKzOoKs/
I tried to email you too, in case you wanted to take this off GitHub.

I have no idea what SMIL data is or where to find it; that's probably the data that contains the path to the m3u8 file.
Evidently this new live-tv URL structure isn't just for URLs but also contains a new page structure and/or video-loading mechanism.


Added example 52 which returns Citations from the Grounding Context

Motivation and Context
With this PR, I am adding an example which has prompt fine tuned to return citation at the end of the response in markdown format. This will help other developers of SK to return Bing or Microsoft 365 Copilot like citations or references in their responses.

@microsoft-github-policy-service agree

Hi @sandeepvootoori Thanks for the contribution! This looks like a neat semantic function.
I don't think KernelSyntaxExamples is quite the right place for this -- that project is meant to show how to use specific kernel features, whereas this is a great semantic function that could be built for SK.
Can I recommend publishing this in a new standalone repo, as a new Skill, and linking back to SK? We'll happily highlight and showcase it as part of a demo or blog post.


Bug: No bare URIs breaks file links with triple slashes

[x] I have verified that I am on the latest version of the Linter

Describe the Bug
A file link with the slashes /// is broken after linting:
E.g. file:///C:/Untitled.md becomes <file:///>C:/Untitled.md
How to Reproduce
Steps to reproduce the behavior:

Activate the rule "No Bare URIs"
Create a note with a file link, e.g. file://C:/Untitled.md
Lint the file

Excerpt from the log
# Untitled

file:///C:/Untitled.md

Running No Bare URLs
no-bare-urls: 0.09999999403953552 ms
# Untitled

<file:///>C:/Untitled.md

Expected Behavior
Brackets are around the link, not after file:///
Expected output:
<file:///C:/Untitled.md>

Screenshots

Device

[x] Desktop
[ ] Mobile

Additional Context
Add any other context about the problem here.

Hey @Martin-Milbradt . It looks like I can reproduce this. I will see if I can figure out what is going on. It looks like the regex is matching correctly based on the regex tester I am using, but I am getting the result that you are. Hopefully this can be resolved soon.

Looks like it was a simple fix. I will go ahead and push that up.

This change has been merged into master and should go out in the next release. Please let us know if it does not work on master or in the next release.


Don't run tests on reference unnecessarily

Checker should see if all commands in a test case use JSON evaluation and skip all the reference steps when reference is not used for evaluation at all.

Fixed, although a bit lazily. Generate results still expects there to be a reference results dictionary - the current solution just puts the student results there if reference is not needed for evaluation.


[not really a bug] identical_ok is deprecated in huggingface-hub's upload_file

In the huggingface-hub dependency, the identical_ok argument has no effect in upload_file (and it will be removed soon)
See
https://github.com/huggingface/huggingface_hub/blob/43499582b19df1ed081a5b2bd7a364e9cacdc91d/src/huggingface_hub/hf_api.py#L2164-L2169
It's used here:
https://github.com/huggingface/datasets/blob/fcfcc951a73efbc677f9def9a8707d0af93d5890/src/datasets/dataset_dict.py#L1373-L1381
https://github.com/huggingface/datasets/blob/fdcb8b144ce3ef241410281e125bd03e87b8caa1/src/datasets/arrow_dataset.py#L4354-L4362
https://github.com/huggingface/datasets/blob/fdcb8b144ce3ef241410281e125bd03e87b8caa1/src/datasets/arrow_dataset.py#L4197-L4213
We should remove it.
Maybe the third code sample has an unexpected behavior since it uses the non-default value identical_ok = False, but the argument is ignored.

Resolved via https://github.com/huggingface/datasets/pull/4937.


Question: direction detection? ("radar")

Hi, I'm in the planning stage of adding Home Assistant to my house, and am planning for two camera's, both are near doors (front door, back door).
Is it feasable (and what is involved?) to have Viseron detect movement towards or away from the camera? And if so, could it fire an event that an object is approaching/moving away from door/zone? Goal here is to switch lights on/off, or do something else based on a person coming/going.
Basically act in many ways similar to radar.

Hi!
You could setup several zones and then act depending on ther activation order.
I use Viseron to activate my lights around the door using the object_detected entities in Home Assistant, but direction is a bit more tricky.

Thanks! Yes, I can see this could potentially work 👍


Add user context to device handle

Global variables should be avoided, each callback registration should have a user context argument to be passed to the callback when invoked.
There are callbacks that do not have a user context argument, uGnssPosGetStreamedStart is an example.
This change introduces the ability to attach a user context to the device handle, it will allow singleton callbacks such as uGnssPosGetStreamedStart to be able to extract the context without refactoring the registration functions and the callback signatures.
    void *context = "A";
    if (uDeviceOpen(&gDeviceCfg, &device_handle) == 0) {
        uDeviceSetUserContext(device_handle, context);
        assert(uDeviceGetUserContext(device_handle) == context);
    }

Hello,
Please tell me if you prefer I go over all callbacks and add  a variant with user context instead or in addition.
Thanks,

Hi there, that's a very neat idea indeed, many thanks for this.
Just to be clear: your proposal is that a context pointer would be added to the device structure and then the user may retrieve that from within their callback, since the callbacks in pretty much all cases get the device handle.  A usage example might be:
// Define my user context.
typedef struct {
    ...
} MyContext_t;

static const uDeviceCfg_t gDeviceCfg {
    ....
}

// My callback for uGnssPosGetStreamedStart().
static void myPosStreamedCallback(uDeviceHandle_t gnssHandle,
                                  int32_t errorCode,
                                  int32_t latitudeX1e7, int32_t longitudeX1e7,
                                  int32_t altitudeMillimetres,
                                  int32_t radiusMillimetres,
                                  int32_t speedMillimetresPerSecond,
                                  int32_t svs,
                                  int64_t timeUtc)
{
    MyContext_t *pContext = (MyContext_t *) uDeviceGetUserContext(gnssHandle);

   // Do stuff here, employing my user context as necessary.
}

...

// Somewhere in main() or beyond

uDeviceHandle_t devHandle = NULL;
MyContext_t context = {0};

uPortInit();
uDeviceInit();

if (uDeviceOpen(&gDeviceCfg, &devHandle) == 0) {
    // Attach user context
    uDeviceSetUserContext(devHandle, &context);

    ...

     // Do something that results in a callback, e.g.
    uGnssPosGetStreamedStart(devHandle, 1, myPosStreamedCallback);

    ...

}

As you say, none of this requires a change to any APIs so I'm very happy to accept it.  I'll just make a few comments syntax-wise  to match our naming conventions, plus a minor addition to the comment in the function header, and I will also run a check through our style-checking script to be sure it doesn't want to re-align.

Hello @RobMeades,
Yes, this is the exact use case.
I will be happy to make any convention modification, I tried my best to align :)
Thanks,
Alon

Thanks for the comments, if you have a script or a sequence to check these issues, I will be glad to send a github actions to check these automatically.

Thanks for the comments, if you have a script or a sequence to check these issues, I will be glad to send a github actions to check these automatically.

We just run python astyle.py from the root of the ubxlib directory and doxygen from the root of the ubxlib directory, but of course you need AStyle (3.1) and Doxygen installed for that.

Done!  Thanks again, really appreciate both the idea and the PR.


Clear GPU memory when using sklearn api

Is there any way to clear the GPU memory after training a model using the sklearn like api?
This does not work unfortunately:
model = mx.model.FeedForward(
     softmax,
     num_epoch=num_epoch,
     learning_rate=0.01)

model.fit(X=data_set)

del model

Most training executes asynchronously, so after model.fit finishes, some of the memory could still be alive. However, they will indeed be eventually released and there is no need to directly call del


The problem of adding to HA

I followed the instructions, but when adding it, it says in the panel that the card was not found (I attach a screenshot)
Tried through it HACH did not help, the same thing writes that it is not available

I think you manual resource is wrong. It has backslashes in the front and forward-slashes later on in the path.
Remove it and just rely on the HACS path and see if that fixes it.

@Xach-s-usami  have you been able to resolve the issue?

@Xach-s-usami  have you been able to resolve the issue?

No, I'm still puzzling over how to solve this case, I deleted everything and did it in stages

@Xach-s-usami , It seems like from time to time HACS frontend stop to load. What I have done is to copy the card out and directly under /config/www/ and added it manual.
/local/lux-power-distribution-card.js

thank you for your help, the problem turned out to be that my sensors are registered with a serial number (but this is not accurate)
I deleted everything, cleared the cache, and then reloaded via Hass
And began to manually write the lines
and I got it all thanks for the help)))
I sit and admire

thank you for your help, the problem turned out to be that my sensors are registered with a serial number (but this is not accurate)
I deleted everything, cleared the cache, and then reloaded via Hass
And began to manually write the lines
and I got it all thanks for the help)))
I sit and admire


Sending input in Cargo run mode errors with wrong-type-argument

When I send input to an inferior process in Cargo run mode, upon hitting RET, I get (wrong-type-argument number-or-marker-p nil) in comint-add-to-input-history.  It seems that comint-send-input doesn't work unless comint variables are initialized by a call to the comint-mode function.  Additionally, rustic-cargo-run-mode calls kill-all-local-variables, which undoes all initializations in comint-mode, so we somehow need to arrange that initialization to happen after rustic-cargo-run-mode.
Unfortunately, I don't know enough about Elisp to know how to make that happen, but here's a workaround for the time being.  It sets up an around-advice that initializes comint-mode before rustic-cargo-run-mode, then saves and restores comint-related local variables during the initialization of rustic-cargo-run-mode.
(defun my-rustic-cargo-run-mode-comint-fix (orig-fun &rest args)
  (comint-mode)
  (let* ((comint-bindings (cl-loop
                           for (sym . val) in (buffer-local-variables)
                           if (string-prefix-p "comint-" (symbol-name sym))
                           collect (cons sym val)))
         (ret-val (apply orig-fun args)))
    (cl-loop for (sym . val) in comint-bindings
             do (make-local-variable sym)
                (set sym val))
    ret-val))
(advice-add 'rustic-cargo-run-mode
            :around
            #'my-rustic-cargo-run-mode-comint-fix)

System Details:
System: macOS X Big Sur
Emacs version: GNU Emacs 27.2 (build 1, x86_64-apple-darwin18.7.0, NS appkit-1671.60 Version 10.14.6 (Build 18G95)) of 2021-03-28
Rustic version: rustic-20211018.1934 (pulled from melpa)

I think it makes sense to keep this open since the fix is still optional.
For anybody who is interested -> https://github.com/brotzeit/rustic#run

@psibi will add some docs on compilation-mode and comint-mode for 'cargo run'. If somebody has issues with polymode please open a new issue.
@jun0 thanks!


Json file cleanup

Solves #178
Could you check if this fixes the temp_out.txt cleanup please @jmafoster1 ? Interested to see how it runs on Ubuntu.

Another PR remove the use of the logger in the JSON frontend, and I'm not sure what the motivation to change the test temporary files like this?

Looking at the original #178, it looks like I just wanted to make sure that files created by running pytest didn't accidentally get committed to the repo, but if they're no longer created, then that's fine :)


CRAYSAT-1882: Further improvements to full-system boot and shutdown procedures

Description
Improve the steps in the "Shut Down and Power Off the Management
Kubernetes Cluster" section of the "System Power Off Procedures". This
includes the following changes:

Include more complete example output from the ncn-power stage of
sat bootsys shutdown, so that the admin knows what to expect.
Add sub-steps describing what to do in known exceptional circumstances
that can occur during the ncn-power stage of shutdown.
Improve console monitoring instructions, similarly to how they were
improved for the boot process.
Remove unnecessary ipmitool commands that query for power status of
NCNs using ipmitool. This is already done by the sat bootsys shutdown --stage ncn-power command.

Improve the steps in the "Power On and Start the Management Kubernetes
Cluster" section of the "System Power On Procedures". This includes the
following changes:

Show the successful output of the ncn-power stage first. Then show
exceptional conditions in sub-steps. Improve the wording.
Ensure all log messages in example output are appropriately prefixed
with the log level as they will be with the latest version of sat.
Improve documentation around monitoring node console logs to include a
tail command and describe using screen in more detail.
Remove the Ceph troubleshooting from the platform-services stage
since it has been moved to the ncn-power stage.
Add a general reminder to re-run the platform-services stage if any
unexpected errors occur.

Checklist

[x] If I added any command snippets, the steps they belong to follow the prompt conventions (see example).
[x] If I added a new directory, I also updated .github/CODEOWNERS with the corresponding team in Cray-HPE.
[x] My commits or Pull-Request Title contain my JIRA information, or I do not have a JIRA.

Reminder to @rustydb and @nrockershousen: this is just going to a feature branch at this point. When we have the updated sat code ready for inclusion in the CSM 1.6 manifest, we will create a PR to merge the feature/CRAYSAT-1740 branch of docs-csm into release/1.6. That should be later this week or early next week.
This PR is mostly for review amongst the SMEs.


Add sample app that uses search

We could generate some icons assets just to get rid of default one (it is easier then to find the right app on the phone).

@d-kunin good feedback! I made a bunch of changes to address the issues you found


Add section on Metrics to the documentation

Codecov Report

Merging #1912 (8dbdc0b) into main (757c888) will not change coverage.
The diff coverage is n/a.

:exclamation: Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.
@@           Coverage Diff           @@
##             main    #1912   +/-   ##
=======================================
  Coverage   87.12%   87.12%
=======================================
  Files          83       83
  Lines        9052     9052
  Branches      537      537
=======================================
  Hits         7887     7887
  Misses       1165     1165


Fix coverage test on Windows

Relying on $PWD made this invalid on Windows

@swift-ci please test Windows platform

@swift-ci please smoke test macOS platform

@swift-ci please smoke test Linux platform

Yea sure I can clean that up

Here's the follow up! https://github.com/apple/swift/pull/32443


Replacement of keptn-contrib/keptn-service-template with template in Go and for Keptn >0.5.0

The reference repository for building a keptn-service is out-of-date: https://github.com/keptn-contrib/keptn-service-template
This template need to be updated to a service written in Go and with deployment/service manifests that are supported by Keptn >0.5.0. Besides, a guideline for listening to Keptn events, building and deploying a custom service need to be provided in the Readme.md.
Finally, this new template has to be referenced in: https://keptn.sh/docs/0.5.0/reference/custom-service/

What if we just use one of the existing simpler services as a reference?

When updating the keptn-service-template we should also add the following to the tutorial:

How to use (read and write) cloud events (eventbroker, distributor, etc...)
How to make a good git repo for a service (such that a user can contribute it to keptn-contrib)
Link to example repos (e.g., notification-service, alexa-service)


[core] Add the "measurement.trackjvm" property to measure JVM statistics

when the status thread is enabled. This tracks the GC count, thread count,
memory used and average system load (if implemented) over time.
Related to #717

Can you add a descriptive blurb into workloads/workload_template?

Looks like an Acumulo test failed. Failed tests:   workloadd(com.yahoo.ycsb.db.accumulo.AccumuloTest): Didn't get enough total cells. expected:<20000> but was:<19660>

Well that's not good. Maybe race condition caused by a missing flush/close? I know that client library has a bug where it still defaults to client side buffering.
Let me take a look and rerun the test.

Had to fix a typo. This time a Cassandra failure:
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.12.4:test (default-test) on project cassandra2-binding: Error occurred in starting fork, check output in log -> [Help 1]


Contributing to scientific python page

Description: What does this implement/fix? Explain your changes.
Create the contributing to scientific python page.
Reference: Issue or Pull Request (PR)

[ ] Towards #88

Reminders

[x] Did you review the CONTRIBUTING GUIDE?
[x] Did you reference an issue or another pull request?

Any other comments?

@BerylKanali: we need to add this to the menu.
For menu items, use this file:
snowlake-v1.2/_data/navigation.yml

https://ericmjl.github.io/software-testing-open-source-and-data-science/talk.html#/


Added Min/Max Between

Coverage decreased (-0.9%) to 21.307% when pulling b9e6127a24f17c9b46c5c07489b5c2ad5ccf0465 on minmaxbetween into da5e1e205c7f3b50b1401376cf3b6fbaf8166bb6 on master.


Custom Image from local rep

How load custom image from my local rep. with animation

Use whatever Library/SDK you use, and in its progress update block/callback, update the loader's progress


dotnet: added shell completion scripts

Description of changes
I've added shell completions scripts from official microsoft page to dotnet package
Things done
I've checked bash and fish shells, completions works fine in there. When I tried to start zsh from nix-shell -p zsh it started complaining about missing config and stuff like that and completions didn't work. Would be nice if someone using zsh daily would test this PR. Other options would be to ship it as-is (my choice) or delete zsh completion from PR (if other reviewers decide so).

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Hey @NixOS/dotnet :wave: , could you please take a look?

Hello, I'll check it. I'm using zsh so I can test with that as well

Can confirm it works on zsh:

@gbtb I'm sorry, I tried and failed to resolve this conflict with the web editor.  would you mind rebasing it?

@gbtb I'm sorry, I tried and failed to resolve this conflict with the web editor. would you mind rebasing it?

I've rebased this PR on top of the latest master.