Better communication/clarity on phase-1 validation and serialization criteria

Overview
A perennial issue on a team I manage has been understanding what exactly comprises phase-1 validation. I would like to see a clear, plain English listing of the criteria that must be satisfied by a script context or transaction in order to actually function on the main net.
Full disclaimer: I've not fully read the specs or the implementation myself and this issue is, in part, to request a resource to which I can direct developers to in lieu of a code-dive or spec-read.
Current Understanding
The consensus among my coworkers is that:

Phase-1 validation is "everything besides script execution" -- which is too vague to actually build off of.
Phase-1 validation is specified in the formal specifications -- which is too much of a deep dive
Certain invariants/constraints are ensured by serialization, and these are neither explicitly specified nor readily apparent in the code base. These also appear to be defined on a Tx type rather than a TxInfo, meaning there's a lot of extra things that would be unsuitable for simply validating a TxInfo or ScriptContext.
Alternate implementations (even if the behavior is technically "specification compliant") would be considered invalid by IO, meaning there may be some de facto "standard" behavior that is not specified.

Our use case
The reason for why we need this is to be able to construct script contexts in Haskell and know whether or not they are realistic to what we would see in practice. My understanding is that tools like EmulatorTrace have failed to perform this sufficiently -- failing to completely normalize Values, for example. This leads to situations where developers are unsure whether they need to check for phase-1 validity in phase-2.
We would, in particular, like to be able to have a set of checks like isValueNormalized, is transactionBalanced, areOutputsSorted, and so forth, that we can combine into something that corresponds to isPhase1Valid. The final property should be that isPhase1Valid is true if and only if a given transaction/script context can

A.) Be constructed by the actual implementation
B.) Pass through to phase 2 validation

We'd be happy to write these if needed, but they would ideally be provided for us. It's insufficient to fully serialize a transaction using the various utilities available, because we want to be able to define partially-invalid script contexts at certain stages in the development process, but also understand which validation checks would fail on a given object.
Finally, we'd want to partially auto-generate script contexts, possibly exhaustively within certain bounds, so we'd want these checks to be fast. Full serialization would ostensibly be much slower than a pure isPhase1Valid function, and partial validation would certainly be faster than either if we only cared about a subset of checks.
Ideal Outcome

[ ] Detail exactly what checks are performed during phase-1 validation, in plain English, with back-references to implementation/spec if applicable, and add this to the "Ledger Explanations" webpage
[ ] Detail exactly what invariants emerge during serialization, and to what extent these are considered mandatory -- i.e., whether alternate implementations SHOULD or MUST uphold these conditions -- and add this to the "Ledger Explanations".

[ ] If alternate implementations are required to uphold these invariants and if they are not listed in the formal spec, add them.

[ ] Provide helper utilities and tests that developers can utilize to build libraries such as those described above
[ ] Add these checks to EmulatorTrace and other IO utilities.
[ ] Test these functions/findings against the actual implementation.

Hi @peter-mlabs , thanks for bring this up, this is a great topic.

What is "phase 1 validation"
The definition really is "everything besides script execution", but that's not vague at all. If you look at the Alonzo specification, Figure 9, the line:
evalScripts tx sLst

is what we mean by phase 2. The rest of the spec is phase 1. The specs contain a lot of plain English prose describing everything, but please let us know if you think something is missing. You say that the specs are too much of a deep dive, but that is our best effort of explaining the ledger validation. For folks creating transactions (as opposed to blocks), the topmost rule for validation is the LEDGER rule from the specs. Any failure in this rule, except for evalScripts tx sLst, is a phase 1 failure. Most Plutus developers can probably ignore the stake delegation and governance rules, leaving just UTXOW, UTXO, and UTXOS (see Figure 78 of the Shelley spec for the rule hierarchy, but note that the Shelley spec has no UTXOS rule). Moreover, you may not care much about UTXOW (the checks surrounding the authorizations of the transactions), and since UTXOS is mostly just the phase 2 check (it also calls as governance sub-rule), perhaps the UTXO rule is nearly all you care about.
I am myself unfamiliar with the "Ledger Explanations" webpage and EmulatorTrace. If I am given the time, I've dreamed of writing a ledger wiki myself, with high level explanations that I could give to, for example, a new ledger team member, etc. Could you provide me a link to what you mean by the "Ledger Explanations" webpage?
Invariants/constraints ensured by serialization
This is a very interesting topic, and one that comes up a lot when we make the rules. We do have a wire specification for every ledger era (See the table at the top of this repo's readme, the CDDL column provides links to all the wire spec), but sometimes our schema cannot be fully captured by CDDL. We try to list these constraints as comments in the CDDL file, but I'm sure there are huge gaps. Filling in these gaps sounds like a great idea to me.
I'm confused, though, why you mention validating TxInfo or ScriptContext. Those are not really validated in any sense I can figure, they are constructed by the ledger. Perhaps you are after "the serialization constraints which effect the script context"?
Alternate implementations considered invalid by IO due to behavior that is not specified.
Our goal has always been that the formal ledger specs, together with the wire specification, provide all the details needed for an alternate implementation. We may fall short of this goal, but that's always been the motivation. If you find gaps, I would be thrilled to get github issues for them. Having an alternate implementation would extremely healthy, as we'd have more confidence in how well we've achieved our goal.
Set of checks
Since phase 1 validation is the bulk of this codebase, this quite a task. That said, we do have a lot of re-usable code, and maybe in the future we can try steer things towards re-usable checks for Plutus developers.  For example, here is a hunk of the Alonzo UTXO rule, which probably includes several of the "phase 1 checks" that you care about:
https://github.com/input-output-hk/cardano-ledger/blob/dd4b6e31a05e3c02c6f00f0d53cf20e3b8b74468/eras/alonzo/impl/src/Cardano/Ledger/Alonzo/Rules/Utxo.hs#L533-L553

Regarding EmulatorTrace there has been significant effort in the last couple months to make it more faithful to the actual ledger rules. Previously it had a separate implementation that had diverged from the ledger and we know how much trouble this caused developers. We have changed it to use the code directly from ledger to make sure it is faithful to the behavior on chain. Its probably not perfect yet, but hopefully much better than before.
I realize the response to github issues has been spotty, but we really do want feedback on problems with the emulator. We have increased staffing and are trying to stay on top of the github issues.

Thank you both for your prompt responses.

The specs contain a lot of plain English prose describing everything, but please let us know if you think something is missing. You say that the specs are too much of a deep dive, but that is our best effort of explaining the ledger validation.

Understood. I'll be taking a closer look at these myself, but its been a perennial re-hashing that any time someone's been tasked on "figuring out what phase-1 is", its been a struggle -- perhaps after I've had a look myself, I'll have more direct insight into what, if anything, could be improved.

Could you provide me a link to what you mean by the "Ledger Explanations" webpage?

Just the Read The Docs page here (linked in the README of this repo). I think it would be a logical place for saying things "Values are normalized" and explaining exactly what that means -- with respect to non-duplicate entries, sorted entries, always containing an ADA entry, etc.

I'm confused, though, why you mention validating TxInfo or ScriptContext.

I am aware -- and fully agree -- that this terminology is imprecise, but I don't have a great vocabulary to distinguish my intended meaning. What I mean by "validating a TxInfo" or "validating a ScriptContext" is:

partitioning the set of inhabitants of those types into ones that could actually be seen on-chain during script execution and ones that couldn't,
having a function (for example) isPhase1Valid :: TxInfo -> Bool that returns true if it can be seen during phase 2 validation, and false otherwise.

For instance, including an invalid currency symbol in a transaction would produce an TxInfo that is invalid. A txInfoMint that doesn't actually reflect the delta between inputs an outputs would be invalid. A TxInInfo that contains a Value with a negative amount would be invalid.
The crux of the issue, as I see it, is that there are inhabitants of the types that (in a perfect world) wouldn't be possible to construct; but we don't live in that world, and thus there are certain values of those types that are "invalid" for a given use case.
And to be explicit, the reason this is difficult to develop around is because it's misleading about what we do need to check for in our scripts. If it is legal for non-normalized Values -- which are totally possible to construct via PlutusLedgerApi.V1 -- to reach the chain, then we end up needing to verify within our scripts that the assumed normalization holds. Otherwise we end up with issues like this, where testing environments diverge from the actual ledger in ways that we may not be able to feasibly detect without a full dive into the ledger spec and CDDL wire format.
I was under the impression that doing so wasn't intended to be a prerequisite for contributing to production-worthy script development. If you disagree, and do think that the specification and CDDL should be required reading for all smart contract developers, I won't take offense and will adjust my expectations accordingly.
This extends beyond just the fully-implemented validator or minting policy level, too -- there's no reason to be quickcheck-gening invalid Values, TxInfos, or ScriptContexts in most cases, since it (ideally) shouldn't be possible to construct such values in the first place, and there's correspondingly no reason to be writing logic to guard against these cases that are impossible in practice.
Does this all make sense?
Another relevant issue from a coworker is here.

Set of checks

Continuing on the above, and forgive my vague understanding of all of this: the "phase 1 checks I care about" seem like they extend beyond things that are represented in the code base as actual checks, and do indeed get into a more vague territory of serialization. It's not so much a is-this-transaction-valid "validation" that I'm concerned with, as it is a "is this value of this type something that my plutus script could possibly ever encounter".
So I suppose that what I'm looking for may not be what is traditionally considered phase 1 validation, but still falls under the definition of "everything before phase 2". What vocabulary would you use to describe the situation here?

EmulatorTrace updates

Very glad to hear that more effort is being put into this. We've been working with (and on) various tools that were meant to fill in some of the gaps, but none have quite lived up what we hoped EmulatorTrace could be.
We're working on PCB, and this is the area where the issues above have bitten us most recently. We're finding it useful to partially specify incomplete, obviously invalid script contexts to feed to scripts at certain stages in development. For instance, if a script should validate based only on what is contained in txInfoMint, then we don't want to force uses to fully specify a valid set of TxInInfo and TxOut, or if a validator delegates its logic to another script we don't even necessarily need the Spending script purpose to refer to a TxOutRef present as an input; but sometimes we do want all of these sorts of checks to have a guarantee that a known-invalid ScriptContext is invalid for the particular reasons we think it is.

any time someone's been tasked on "figuring out what phase-1 is", its been a struggle

It's probably true that we never explicitly state that "phase-1 failures" are all the failures except for the single phase-2 failure. It's not something that needed a formal definition in the spec, but it is language that we use a lot and I agree that that point should be explicitly stated somewhere.

Read The Docs page

Ah! That documentation is very incomplete and really only explains the multi-assets. The point about "Values are normalized" is mentioned in the spec (you can find in page "CAUTION" in the Alonzo spec to find it), or at least the ADA part. Perhaps we can add more details there.

partitioning the set of inhabitants of those types into ones that could actually be seen on-chain during script execution and ones that couldn't

So you would like to know which parts of the transaction context are in the ledger state vs those that are in the transaction? (and those in the middle, like how transaction inputs are "resolved" but the UtxO to include the outputs inside the context, for example).

function (for example) isPhase1Valid :: TxInfo -> Bool that returns true if it can be seen during phase 2 validation

I think I am following, and I'm sorry about all your struggles. I don't think such a function is possible without changing the signature to something like isPhase1Valid :: Tx -> UTxO -> StuffImForgetting -> TxInfo -> Bool, and applying many of the ledger rules. I think the heart of the issue is:

where testing environments diverge from the actual ledger

When developing the ledger code for the Alonzo era, we were assuming that the Plutus tooling would save folks from the nuances of things like worrying about Value being in cannonical form, etc.

I was under the impression that doing so wasn't intended to be a prerequisite for contributing to production-worthy script development.

I agree with you, as I said above, the plan was for the tooling to be a shield from worrying about these details.

Does this all make sense?

It does, I fully agree that there is an incredible amount of technical details surrounding the transaction context. I'm sorry it's still not made easy by the tooling.

"phase 1 checks I care about" seem like they extend beyond things that are represented in the code base as actual checks, and do indeed get into a more vague territory of serialization.

You very well might be correct. A couple example, if you have them easily at hand, would be nice.

may not be what is traditionally considered phase 1 validation, but still falls under the definition of "everything before phase 2". What vocabulary would you use to describe the situation here?

I myself probably say "phase 1 validation" somewhat informally to mean any of the ledger rules except running the Plutus evaluator. And I say things like "that is guarded on the wire" referring to things you cannot do because of the logic in the deserialization. I think the Plutus team uses the same language. It's very enlightening to see the troubles you are having, and concepts that were not obviously name-worth to use when developing the ledger for Alonzo. I myself have only written pretty trivial Plutus scripts, but I would love to do more and gain more empathy for the hard parts.

I think we're getting on the same page here!

So you would like to know which parts of the transaction context are in the ledger state vs those that are in the transaction? (and those in the middle, like how transaction inputs are "resolved" but the UtxO to include the outputs inside the context, for example).

I don't think that is what I mean.
I want to know which values/inhabitants of certain types or in certain contexts are "invalid" from either the perspective of (using your terminology) "phase 1 validation" or "guarded on the wire", and specifically why they are invalid.
For instance, (pseudo-code)
v :: Value
v = { "ADA" :
         { "" : 0 }
    , "FOO" :
         { "BAR" : 100
         , "BAZ" : -100
         }
     }

would be a "valid" inhabitant of the PlutusLedgerApi.V1 Value type if it appeared in the txInfoMint, representing the burning of 100 tokens with currency symbol FOO and token name BAZ, and the minting of 100 tokens with the currency symbol FOO and token name BAR.
But this would be "invalid" if it was used as the value of a TxOut, since a TxOut can't carry a negative value. T
This means that when we're writing scripts, and doing something akin to v' = ctx.txInfo.txInfoInputs.txInInfoResolved.txOutValue on-chain, we're never going to have v == v'; but if we have v'' = ctx.txInfo.txInfoMint, we could have v == v''.
And I want to know this, because if I'm writing a function that operates on Values that was something like
if
  v has a negative value
then
  launch the nukes
else
  don't

this function would be valid to include in scripts if it's called only on values like v'' (i.e., as derived from txInfoMint); but generating arbitrary TxInfos and running them through a test suite might flag such a usage as world-ending.
So your comment that

I don't think such a function is possible without changing the signature to something like isPhase1Valid :: Tx -> UTxO -> StuffImForgetting -> TxInfo -> Bool, and applying many of the ledger rules

makes sense, but I don't know if we're talking about the exact same thing here. I want to know if, given any possible  ledger state, and any possible spec-compliant implementation, whether I'd ever launch the nukes; I can tell just from the fact that t :: TxOut whether or not I will ever have negative Value entries from t.txOutValue.
I'll try to keep this issue updated as I read through the specs and possibly get some more concrete examples.

I think we're getting on the same page here!

yes indeed, it's becoming clearer and clearer!
I think the following might help (and I can try to get to it early next week): Go through each type inside TxInfo and think about the space of values in each type, and which one can be ruled out without all the context of the ledger state. So things like 1) blanket rules (no negatives unless minting, etc), and 2) things forbidden on the wire. I can list this out in this issue, and maybe that could be the start of a helpful document or a helper function.

For reference, the Plutus V2 context looks like:
https://github.com/input-output-hk/plutus/blob/9ef6a65067893b4f9099215ff7947da00c5cd7ac/plutus-ledger-api/src/PlutusLedgerApi/V2/Contexts.hs#L78-L92
data TxInfo = TxInfo
    { txInfoInputs          :: [TxInInfo]
    , txInfoReferenceInputs :: [TxInInfo]
    , txInfoOutputs         :: [TxOut]
    , txInfoFee             :: Value
    , txInfoMint            :: Value
    , txInfoDCert           :: [DCert]
    , txInfoWdrl            :: Map StakingCredential Integer
    , txInfoValidRange      :: POSIXTimeRange
    , txInfoSignatories     :: [PubKeyHash]
    , txInfoRedeemers       :: Map ScriptPurpose Redeemer
    , txInfoData            :: Map DatumHash Datum
    , txInfoId              :: TxId
    }

These types are from the Plutus library, and the ledger translation into them.  The Plutus types do protect the users from some of the details mentioned in this issue (such as whether or not a Value is in canonical form or not). I'm less familiar with all the protections in the Plutus library, but I can give you the perspective from the ledger code.

TxInInfo - This type is what I (and maybe this is just me) like to call a "resolved" input, ie a transaction input together with the corresponding transaction output from the UTxO. I will ignore the transaction outputs for the moment, since we will be addressing them when we get to txInfoOutputs. A transaction input is a transaction ID and an index. A priori (in other words, without looking at the surrounding ledger state, only looking at the values in the type) there are no values to rule out. The ID is a hash, anything is possible, and the index can in theory be anything.
TxOut - A transaction output is an address, a Value, possible a datum or a datum hash, and possibly a script. Addresses use a custom binary format (ie not describable in CBOOR) defined here. A priori, I cannot think of any values that meet the custom binary representation that we could rule out without the ledger state. We go over Value below. A priori, I also do not see any values in Datum, DatumHash, or Script that we can rule out. We can rule out, however, that an output contains both a datum hash and a datum (enforced by the wire spec).

Values in an output - We can rule out any negative values.

txInfoFee :: Value - The fee must contain only lovelace.
txInfoMint :: Value - The lovelace field must be zero.
[DCert] - I'll skip this one for now, since 1) it's rather involved, 2) I'm not sure anyone is interested in it, and 3) I suspect the answer is "there isn't anything interesting to rule out a priori (but we can come back to this if it turns out it is relevant).
Map StakingCredential Integer - The integer must be non-negative, and would certainly never be bigger than the total number of lovelace. The stake credential can a priori be any value in the type.
POSIXTimeRange. The lower bound could be anything at or below the upper bound. If you knew the current POSIX time, you could throw out any value above the current time plus the time horizon (I'd need to lookup what that is, I think it is something like 3k/f-many slots).
[PubKeyHash] - We can't rule out anything here without seeing the transaction.
Map ScriptPurpose Redeemer - We can't rule out anything here without seeing the transaction, except for the empty map.
Map DatumHash Datum - This one is perhaps a bit tricky. We do require that the datum hash be correct (ie if you hash the datum, you get the corresponding key in the map), but this depends on how the datum is serialized (CBOR is not unique).
TxId - anything goes (as mentioned above in TxInInfo).

I think the TLDR here is that Value is the type that contains nearly all of the checks that y'all want.
Did this help?

Thanks, Jared -- this is a very good start to what we're looking for.
A few comments follow below. I'm making some assumptions, so please flag if anything seems out of place -- its entirely possible that things that I'm inferring as "today's convention" were intentionally left open-ended.

These types are from the Plutus library, and the ledger translation into them. The Plutus types do protect the users from some of the details mentioned in this issue (such as whether or not a Value is in canonical form or not). I'm less familiar with all the protections in the Plutus library, but I can give you the perspective from the ledger code.

Just to clarify, the Value type from PlutusLedgerApi.V2 does not ensure canonical form (or at least not how I understand it). This may be a better issue to file against that library, but a few notable things:
-- Non-hex currency symbols are partial; they can be constructed, but lead to
-- exceptions when evaluated
ghci> Value $ fromList [("Not Hex", fromList [("",100)])]
Value (Map [(*** Exception: NotHexit 'N'

-- Currency symbols must have an even number of digits, or lead to a runtime exception:
ghci> Value $ fromList [("ABC", fromList [("",100)])]
Value (Map [(*** Exception: UnpairedDigit

-- Duplicate currency symbol entries are not combined
ghci> Value $ fromList [("BCDE", fromList [("",100)]),("BCDE", fromList [("",10)])]
Value (Map [(bcde,Map [("",10)]),(bcde,Map [("",100)])])

-- Duplicate token name entires are not combined
ghci> Value $ fromList [("BCDE", fromList [("",10), ("",100)])]
Value (Map [(bcde,Map [("",10),("",100)])])

Also:

I'm sure there are other bounds on the currency symbol/token name being within certain bound with respect to length, but I'm don't know the details off the top of my head
0 ADA entries should be present in all values, but are not (although this might be fixed already in the issue linked above)

Are you saying that there is another Plutus library with a Value type that does ensure these sorts of things? In Plutarch (which is what we use for most of our on-chain code), there are phantom types for sortedness and negative/non-negative/positive guarantees, but nothing at the PlutusLedgerApi.V* level AFAIK.

TxInInfo

Should the txOutRefId have the format of a SHA256 hash? It's type is just BuiltinByteString. This, to me, would indicate the the byte string must be of a certain length to be valid. If this is handled somewhere under the hood, what happens if a too-long or too-short byte string is passed (note: I could be totally misunderstanding how BuiltinByteString works).
Shouldn't the txOutRefIdx be non-negative? It's type is Integer.
Additionally, the txInfoInputs :: [TxInInfo] are sorted lexically. This is important to note, because

A.) It's not documented anywhere as far as I've seen, except in a github comment
B.) There's a CIP trying to remove this restriction
C.) This doesn't hold for outputs

Also, the txInfoInputs shouldn't contain duplicates.
This also can't be empty -- this was noted in one of the 30-something CIPs, because unless this is said explicitly, there wouldn't be anything preventing someone from paying fees via rewards withdrawals.

TxOut

As above, it seems like there are restrictions on "valid" Addresses, but its type is, essentially, BuiltinByteString. I don't know of any isValidAddress function, and my local hoogle searches don't turn anything like this up.
WRT datums, the OutputDatum type seems to cover the cases fine.
WRT scripts, ScriptHash is again a BuiltinByteString; the same comments as for CurrencySymbol apply.
Also, (in contrast with the txInfoInputs field), the txInfoOutputs field is not sorted.
This probably can't be empty, right? Is there some weird situations with staking, perhaps, where a UTxO containing the exact amount needed for the fee could be passed as the input and not need to list an output?

Values in an output

Must contain minAda.

txInfoFee

Also must be positive

[DCert]

Agreed that this is lower priority for now, but part of the reason nobody cares about these is because nobody understands them :joy:. Some immediate notes:

The various data constructors end up wrapping bytestrings. I think most of these are addresses, and end up having the same considerations with SHA-256 hashes as above.
Staking pointers (which lack even basic haddocks, but that's another story) is a product type of three Integers. I'm pretty sure these all have to be either non-negative or strictly positive.
DCertPoolRetire has an Integer referring to an epoch. This should likely be positive.
Ostensibly, we'd never see duplicates in the list on-chain?

Map StakingCredential Integer

I don't think there's much new here that hasn't been covered above, but to be explicit:

There are ostensibly restrictions on shape, duplicates, and perhaps ordering of these. These may be more of a criticism of the Map type, however.

POSIXTimeRange

I've not dealt with this type much, but it is defined as type POSIXTimeRange = Interval POSIXTime, and it appears that it doesn't actually enforce that the lower bound is indeed earlier than the upper bound -- I assume this is guarded by the wire on-chain?
ghci> Interval (LowerBound (Finite $ POSIXTime 100) True ) (UpperBound (Finite $ POSIXTime 10) False)
Interval {ivFrom = LowerBound (Finite (POSIXTime {getPOSIXTime = 100})) True, ivTo = UpperBound (Finite (POSIXTime {getPOSIXTime = 10})) False}

[PubKeyHash]

Aside from the comments about hashes vs. any BuiltinByteString above, these should probably not contain duplicates. Are they ordered? Must this list be NonEmpty?

Map ScriptPurpose Redeemer

I disagree that we can't rule things out.
For ScriptPurpose, generally:

The Minting currency symbol has to appear in the txInfoFee value.
The Spending TxOutRef has to appear in the transaction inputs
Does the Rewarding staking credential need to appear in txInfoWdrl?
Does the Certifying DCert need to appear in the txInfoDCert list?

For the Redeemers, I assume that this is indeed unrestricted (assuming the constructor(s) for BuiltinData do not allow invalid terms to be constructed).
But for the map in general, I'm assuming there are additional restrictions. Off the top of my head:

I'm assuming that including one or more redeemers is up to the discretion of the transaction submitter, but this map should not include more entries than there are transaction inputs.
Do Reward and Certifying scripts actually take redeemers?
Is there sorting?
No duplicates?

Map DatumHash Datum

This does sound tricky. I'm not sure of the details, but perhaps a validation function could take a witness function or phantom type to ensure DatumHash <-> Datum coherence?
I think I also read in the inline-datum CIP that "extra datums" could be passed. There's been some recurring folklore for new comers that this means

"The transaction submitter can include extra data that is not associated with any UTxO in `txInfoInputs"

but what it really means (as I've been told) is that a datum that is not strictly necessary for validation can be included provided it is attached to a transaction input or output.
So the DatumHash must match something in txInfoInputs, txInfoReferenceInputs, or txInfoOutputs; and thus, there is also a restriction on the maximum number of entries in this map.
Is this map sorted?
Duplicates are probably not allowed, right?

The actual rectification of this issue might be more in the domain of PlutusLedgerApi, and I'd be happy to open an issue there as well -- but first we'd need to narrow down:

Which of these are "forbidden on the wire"
Which of these are blanket rules
Which of these are just not having expressive enough types (i.e., BuiltinByteString)
Which of these are genuinely validated (in the form of "there exists a function in the ledger that checks if a given value is valid, and rejects it if not")

And regardless of whether the PlutusLedgerApi library implements these sorts of things or not, it'd be extremely helpful to have a straight-forward list of these somewhere for alternate implementations of these types/terms. It seems like most of these should be in the spec, but my impression is that some really are undocumented and down to the implementation (like [TxInInfo] being sorted, although I could be wrong.)
It seems, to me, that the overall solution here is probably:

Better documentation (both from the ledger side and in the haddocks of PlutusLedgerApi)
Newtypes and smart constructors where we can easily check things
validation functions where newtypes and smart constructors won't work

Thoughts?

the Value type from PlutusLedgerApi.V2 does not ensure canonical form

I was vague, sorry. By canonical form, I meant handling the zero valued fields. I thought this what was normalizeValue was for, but poking around I don't see it being used. I see master is already different than the release branch, so yea, maybe bring that up with the Plutus folks.

Should the txOutRefId have the format of a SHA256 hash?

The transaction ID is a blake2b 256 hash, so 32 bytes.

Shouldn't the txOutRefIdx be non-negative? It's type is Integer.

You are correct. apologies that I'm more familiar with the ledger types (where it is a Word64) and was mostly going from memory above.

This also can't be empty -- this was noted in one of the 30-something CIPs, because unless this is said explicitly, there wouldn't be anything preventing someone from paying fees via rewards withdrawals.

That's true, I was focusing on the TxIn and forgot about the container. this is a ledger rule.
It's not enforced to preventing someone from paying fees via rewards withdrawals. It's the replay protection for everything inside a transaction.

Additionally, the txInfoInputs :: [TxInInfo] are sorted lexically. This is important to note, because
A.) It's not documented anywhere as far as I've seen, except in a github comment
B.) There's a CIP trying to remove this restriction
C.) This doesn't hold for outputs
Also, the txInfoInputs shouldn't contain duplicates.

That's true, in the ledger it is a set, making these properties much more obvious. But this information is lost in the translation to the script context.

it seems like there are restrictions on "valid" Addresses, but its type is, essentially, BuiltinByteString. I don't know of any isValidAddress function, and my local hoogle searches don't turn anything like this up.

The ledger may not (unless you count fromCBOR, which would be wonky). Such a function probably exists in many other places (the wallet, for instance), but we could add something to ledger and give it a clear name like you suggest.

Also, (in contrast with the txInfoInputs field), the txInfoOutputs field is not sorted.

indeed, it is a list in the ledger

This probably can't be empty, right? Is there some weird situations with staking, perhaps, where a UTxO containing the exact amount needed for the fee could be passed as the input and not need to list an output?

It can indeed be empty, for the reason you mention. but it's not too weird, you can always pay more in fees that you are required.

Must contain minAda.

That's true, but I thought that this was too much toward the "ledger rule" side of the spectrum to be considered here. it requires know the current protocol parameters, you can't just look at the value to make the determination.

Also must be positive

true.

[DCert]
The various data constructors end up wrapping bytestrings. I think most of these are addresses, and end up having the same considerations with SHA-256 hashes as above

If you replace SHA-256 with blake2b 224 (28 bytes), then yes.

Staking pointers (which lack even basic haddocks, but that's another story) is a product type of three Integers. I'm pretty sure these all have to be either non-negative or strictly positive

non-negative, yes. Pointer addresses are nearly un-used, and will almost certainly be deprecated soon.

DCertPoolRetire has an Integer referring to an epoch. This should likely be positive.

yep

Ostensibly, we'd never see duplicates in the list on-chain?

unfortunately not, which is arguably a bug. we currently allow you to de-register a stake credential, re-register it, then deregister it again, all in the same transaction. which leads to duplicates. maybe we will disallow this in a future era.

There are ostensibly restrictions on shape, duplicates, and perhaps ordering of these. These may be more of a criticism of the Map type, however.

yep, most of this comes from Map.

POSIXTimeRange
I've not dealt with this type much, but it is defined as type POSIXTimeRange = Interval POSIXTime, and it appears that it doesn't actually enforce that the lower bound is indeed earlier than the upper bound -- I assume this is guarded by the wire on-chain?

this is guarded by the ledger rules, the transaction must fall between the (slot) interval.

[PubKeyHash]
Aside from the comments about hashes vs. any BuiltinByteString above, these should probably not contain duplicates. Are they ordered? Must this list be NonEmpty?

It's a set in the ledger, so correct, no dups and ordered lexicographically. it can be empty.

For ScriptPurpose, generally:
The Minting currency symbol has to appear in the txInfoFee value.

did you mean txInfoFee? the fee is just lovelace. did you mean an output? maybe you mean that the policy ID must show up in a resolved input or an output?

The Spending TxOutRef has to appear in the transaction inputs

true!

Does the Rewarding staking credential need to appear in txInfoWdrl

yes

Does the Certifying DCert need to appear in the txInfoDCert list?

yes

But for the map in general, I'm assuming there are additional restrictions. Off the top of my head:
I'm assuming that including one or more redeemers is up to the discretion of the transaction submitter, but this map should not include more entries than there are transaction inputs

I'm not sure I follow. The number of redeemers is dictated by the number of plutus script hashes in the transaction. But note that you can't tell from looking at a script hash if it is a plutus script or not, you need the transaction witnesses to figure that out. There's a CIP right now to deal with the awkwardness of this fact. But you'd need exactly one redeemer for each of: plutus input, plutus mint, plutus cert, plutus withdrawal.

Map DatumHash Datum
This does sound tricky. I'm not sure of the details, but perhaps a validation function could take a witness function or phantom type to ensure DatumHash <-> Datum coherence?

The ledger does guarantee that the hash matches, but I assume you want to make a context yourself and know that it is valid. I don't know what the best thing would be to do, you need to cache the serialization somehow.

I think I also read in the inline-datum CIP that "extra datums" could be passed. There's been some recurring folklore for new comers that this means

The only "supplementary" datums that are allowed are those that correspond to newly created output, and those that are in an output corresponding (in the UTxO) to a reference input used in the transaction.

Is this map sorted?
Duplicates are probably not allowed, right?

it's a map in the ledger, so yes it is sorted and no dups are not allowed (remember, the ledger caches the serialization of the datums and would never reserialize).

The actual rectification of this issue might be more in the domain of PlutusLedgerApi, and I'd be happy to open an issue there as well -- but first we'd need to narrow down:
Which of these are "forbidden on the wire"
Which of these are blanket rules
Which of these are just not having expressive enough types (i.e., BuiltinByteString)
Which of these are genuinely validated (in the form of "there exists a function in the ledger that checks if a given value is valid, and rejects it if not")

I think once our list settles down, we could go through each one and answer these questions. It's a fun exercise, but it definitely requires me to look at everything from a perspective that I am not myself used to.

And regardless of whether the PlutusLedgerApi library implements these sorts of things or not, it'd be extremely helpful to have a straight-forward list of these somewhere for alternate implementations of these types/terms. It seems like most of these should be in the spec, but my impression is that some really are undocumented and down to the implementation (like [TxInInfo] being sorted, although I could be wrong.)

Most of the things we have discussed can be derived from the formal spec and the wire spec. But, as I mention above, the perspective here is very different, and so things are not stated in the way that is helpful for you and the other folks looking for the possible space of values. Exacerbating the problem is the fact that the ledger spec is written operationally, and the denotational semantics can sometimes be obscured (though it makes it much easier to compare the spec with the code).

It seems, to me, that the overall solution here is probably:
Better documentation (both from the ledger side and in the haddocks of PlutusLedgerApi)
Newtypes and smart constructors where we can easily check things
validation functions where newtypes and smart constructors won't work
Thoughts?

The ledger team has never yet had the breathing room to build out a proper, user-friendly API. our biggest "consumer" is the consensus layer, and the bulk of our effort has gone to that. But I realize that many folks have struggled to use the ledger code as an API. I think your suggestions are good, and would be a part of the API. The cardano-api (in the node repository) does have an API for some of the ledger functionality, but it also has not been able yet to spend the time add in all the features that are really needed.
I guess I (probably naively) thought that most of these issues that we've talked about were not really ledger issues, but issues for the Plutus tooling.
My biggest take-away from your last post is: the ledger has more restrictive types that those in the context, and this information is lost during the conversion.
I understand why you would want all this validation, since you want to make arbitrary contexts so that you can test Plutus scripts. There's another conversation about what is and is not a good idea to assume inside the context of a Plutus script. This came up in the CIP that you mentioned, about using lists instead of sets for inputs, but all the same concerns apply here.

Thanks again Jared! I'll try to keep this response shorter to not take up too much more of your time.

Can you re-confirm this statement for me?

The transaction ID is a blake2b 256 hash, so 32 bytes.

The haddocks say SHA256. Is this incorrect? If so, I'll open a ticket on that repo.

RE: minAda

of course, your comments make sense. But I think my point was more that we know some ADA has to be present.

RE: DCert, s/sha256/blake2b

As above, going down and unwrapping the types leads to a validator hash being listed as a SHA256. Can you confirm that this is incorrect? If so, I'll open a ticket on that repo.

Did you mean txInfoFee?

Sorry, no -- I meant txInfoMint and also that the txInfoMint must reflect the delta between inputs and outputs.

RE: Redeemers

I think my point was that if we had r redeemer entries in the map, and x1, x2, x3, x4 plutus inputs, plutus mints, plutus certs, and plutus withdrawals, one of the two following situations would probably hold (but I'm not sure which, off the top of my head -- I'm sure this is explained adequately elsewhere)

Either r = x1 + x2 + x3 + x4, or
0 <= r <= x1 + x2 + x3 + x4

I.e., a TxInfo would be invalid if r didn't obey whichever of these actually holds.

RE: Closing thoughts

Agreed with everything you said. I think the line gets blurred between ledger vs. tooling, because when the official tooling is inadequate and can't be relied on as a ground-truth, we have to turn back to the ledger. And, as you've mentioned, it seems like the ledger team has not only a different set of priorities due to different consumers, but also a different set of types they're working with (sets vs lists, etc.)
But yes, I think we've now got a solid understanding of what each other are working with. My mind is currently wondering whether this repo was the wrong place for this, though -- this was where I was pointed to when I asked the question "what is phase-1 validation", but it seems like most of my issues are actually more related to the  plutus-ledger-api types being too permissive and lacking utility functions where specific types would be unfeasible.
In your opinion:

Is there still something worthwhile for the cardano-ledger team to do with respect to the above?

If so, what do you think the steps are, and is there any help needed?
If not, whose door should I knock on next?

Is it realistic to expect the plutus-ledger-api team to make these changes? I realize this might be out of your wheelhouse, but perhaps you'd know better who to ask (short of me opening up a generic issue and linking to this on that repo). My concern is that simply changing lists to Set where needed would break quite a bit.

I'll try to keep this response shorter to not take up too much more of your time.

Conversations like this are my favorite part of my job. And improving life for Plutus developers is arguably the most important thing we can do for Cardano right now. So don't slow down on my account! :)

Can you re-confirm this statement for me?

Absolutely. The only hash that we use in the cardano ledger is blake2b. The link that you shared points to incorrect documentation, thank you very much for letting us know (opening an issue would be grand, if you don't I will). Note that the Shelley ledger spec specifies the hashes in appendix A.1 (sometimes we use 224, and sometimes we use 256).

RE: minAda

ah, yes, it is safe to assume it is not zero. (unless you are on some testnet where the protocol parameter dictating the cost was set to zero).

I meant txInfoMint and also that the txInfoMint must reflect the delta between inputs and outputs.

ah yes, that is another good check. relatedly, you could do the whole "preservation of ADA" check as well, if you are okay with the validation function taking the protocol parameters that determine the deposits amounts (stake cert registration and stake pool registration). This is the "consumed == produced" check in the specs.

Either r = x1 + x2 + x3 + x4, or

Yes, that is another good check. The trouble, though, will be determining which script hashes correspond to Plutus scripts (in the absence of have the scripts in the context). And there is one gotcha to be aware of, though it's a rather odd case. If there is a repeated certificate inside a transaction (such as the situation I mentioned earlier, where someone de-registers, re-registers, and de-registers again), only the  first such certificate needs the redeemer.

My mind is currently wondering whether this repo was the wrong place for this, though

I'm very appreciative of seeing y'all's perspective, I'm glad you raised this here so that I was able to gain insight. It's definitely also
a good topic for plutus-ledger-api.

Is there still something worthwhile for the cardano-ledger team to do with respect to the above?
If so, what do you think the steps are, and is there any help needed?

Thank you for the offer! I think we are definitely honing in on a helper validation function that someone could write without too much effort. After we've settled on what it should look like (maybe we're there now), I can make a clear list of what to check. I can also check with the Plutus team and see if they have an opinion on where this should live. I love the idea of gaining new contributors, though, so if you want to do it to that would be fun too.

My concern is that simply changing lists to Set where needed would break quite a bit.

indeed, we can only do that with a new plutus language version.

Opened the issue, thanks for confirming.
But yes, in terms of the helper validation function, the dream would be having a suite of such functions that can cover all of the different bits. I mentioned this above briefly, but the reason why we'd want a suite rather than (only) one big function is because the checks we're after are broadly split into:

Things that don't make sense under any circumstances (i.e., negative value entries in the txInfoFee, 0 entries in the txInfoMint, etc.)
Things that only make sense in certain contexts, but that context might not be fully specified (i.e., your txInfoMint has to actually reflect the value minted in the TxInfo, but if you've only specified the txInfoMint and not inputs or outputs, there's not any a priori reason why a non-zero txInfoMint would necessarily be invalid.)

When our testing utilities are building a TxInfo for the purpose of testing a script, we'd want to avoid the nonsensical cases in the former, but not always the latter (as opposed to say, tooling, that might need to detect those nonsense cases itself).
So something like (psuedo-code) txInfoMint == [("ABCD", "FOO", 0])" would be rejected in all cases, but we'd have some nuance as to whether
TxInfo {
  txInfoMint = [("ABCD", "FOO", 100)],
  txInfoInputs = [],
  txInfoOutputs = [],
  (...)
}

would be rejected, since we might not want to bother fully specifying the entire TxInfo if our script only looks at txInfoMint.
This extends to your comment about protocol parameters and ledger state, as well. Sometimes we might want to pass those explicitly, sometimes we might want to rely on sensible defaults, or sometimes we might want to turn off all checks that can't be conclusively determined without the extra info.
In terms of our own contributions, I'd have to check with the client and see where priorities lie. I think there's a strong case to be made that many of our testing functions could be made much better with these sorts of checks -- 10000 quickcheck cases don't pack as big of a punch if half of them are nonsense! :)
Let me discuss with my team and get back to you; but collaboration with the plutus team would be good, having it totally handled for us would be great!
Thanks again, Jared

Related discussion.

This is a very long thread, I'm going to attempt to try to respond to it below, please correct me if I've misrepresented anyone.
I think Peter's primary desires are:

To know whether certain ScriptContexts cannot occur in practice, because the transactions that would give rise to them are invalid, or for other reasons. This is in order to simplify his validators by not handling the impossible cases.
To test validator scripts by generating (manually or automatically) ScriptContexts that match what will be produced on-chain.

I think these are quite different desires so I'll address them separately.

In order to tell people what ScriptContexts can't exist, I think we have three options:

Enforce properties in the types, e.g. use NonEmpty if a list can't be empty.

This is somewhat useful, we largely don't do it because we are trying to keep the types relatively simple. But maybe something we could look into more. We'd definitely be open to suggestions on this front!
However, this doesn't help people (much) who're using something like Plutarch where you're doing everything in an untyped way anyway (see the below discussion of Value for an example).
It's also difficult to do this for more involved properties like "this value must appear in this other map", or at least not without some type gymnastics that probably complicate things much more than is warranted.

Document any invariants in the Haddock.

This is good to do. We should probably do this more. The reason we have not done this more is partly because TxInfo is "just" a translation of what's in the ledger. So if you know that, you know "how transactions work" then you know how TxInfo works (mostly). We have generally assumed that dapp developers will know how transactions work.
I think what this conversation has shown me is that this assumption is too strong. Yes, you need to know something about how transactions work in order to do anything useful. But finding that information (particularly if you need precision) is difficult and amounts to option 3 below. It's probably worth duplicating the information to make it easier to access if nothing else.

Tell people to read the spec.

I would definitely be in favour of doing more of 1 and 2. However, I think we can't cover everything. The ledger spec is big, and almost every aspect of it bears on what you can see in a TxInfo, because the question of "what is a valid TxInfo?" is pretty much "what is a valid transaction?" and that's the subject of a large part of the ledger spec! "What kinds of withdrawal can coexist?", "What kinds of reference input are legal?", there are a very large number of such questions, depending on what you care about, and ultimately the ledger spec is the source of truth. I don't think there's a way to summarise that easily without losing precision (which you might need!).
I don't really know what to do here. I think many of the specific questions asked by Peter are low-enough-hanging fruit that it's worth including the answers in more places, but ultimately if you want to know the details of how some corner of transaction validation works, you're going to have to go to the spec.

I don't really know what to do about testing. I think the approach of generating ScriptContexts is reasonable, but it's not the approach that we thought people would take (or that we've taken ourselves), largely because it is difficult to generate a correct ScriptContext, as you've observed. The attitude we have historically taken is more like this:

Generating a ScriptContext is hard, let the ledger do it.
How do we make the ledger do it? By submitting transactions.
How do we generate valid transactions? Well, we have to generate valid transactions as part of our application, so we can generate actions and test that the resulting transactions work.

i.e. test via submitting transactions that actually look like the transactions your application will submit.
This is unsatisfying in a number of ways. It would be nice to be able to test the validator script more in isolation, and it would be nice to be able to test variants and malicious interactions. But I don't think anyone has an easy answer for that today, although people are thinking about it.
To be honest, I am not optimistic about the idea of generating valid TxInfos. However, the ledger does have some support for generating random transactions. It's possible that you might be able to use that, I know that e.g. the Hydra team is.

Finally, a special note on Value. Value is an interesting type, because it is supposed to be treated as a finitely-supported function. The implementation as a map is an implementation detail, and Values with or without 0 entries are equivalent.
The right answer here IMO is to write your operations at the correct level of abstraction. The plutus helpers for Value do this: they are "finitely-supported-function operations" implemented for the map representation, and hence they don't care about 0 entries. And in that spirit, the question of whether or not the representation contains them is a free choice for the ledger.
It's unfortunate that tools such as Plutarch force people to break the abstraction and operate on the representation. Ironically, this is a case where we did try to present people with types that enforce the correct usage.

Another thing: I think it is a real awkward point that this sits between the ledger and plutus. It really is the case that if you want to write a dapp, you need to know quite a lot about the ledger, more than any other kind of user, and the only way to get that information is by reading the spec. I have been arguing for some time that I would like to have some kind of medium-detail ledger conceptual documentation that at least covered some of the concepts, even if it didn't go into the full detail that you need the spec for.

I have been arguing for some time that I would like to have some kind of medium-detail ledger conceptual documentation that at least covered some of the concepts, even if it didn't go into the full detail that you need the spec for.

indeed! @michaelpj knows I am also very fond of this idea. (You can find in page "dreamed of" in this long conversation to see me expressing the same thing).

Thanks for your response @michaelpj; I'm very appreciative that you've obviously taken the time to read through the extended discussion between Jared and I.
I think that your first three points are spot on, and I would say that they're basically a spectrum from "a perfect world" to "what we'll suffer through if we must" from the perspective of developers. One of the things that drew me to Haskell, which I am admittedly more new to than most of my colleagues, is that illegal states can be made un-representable. Doing so at the type level is the holy grail, because then you know a total function is only operating on exactly what it needs to. You don't need any error or Left "Error: Impossible state"'s in this world.
Newtype with smart constructors can help at the value level, but then you're still littering the code with Mabyes or Either's that you wouldn't need in a perfect world.
Reading the spec, IMO, should be probably something that every Cardano dApp developer should do once, but its hard to find the time when we're struggling against so many other things. But its very difficult to keep all of that in our heads -- and that's another reason I was drawn to Haskell. I don't have a great memory, and having types around -- expressive, precise types -- meant that I no longer had to worry about exactly whether I needed to check that my strings were a certain length before passing them to a certain function.
I totally agree that there are some things -- particularly the interdependencies in the TxInfo -- that would be difficult to express in Haskell without resorting a bunch of tricks for hacking in pseudo-dependent types, and causing an unfriendly API in the process. But for places where this happens, having validation functions and smart constructors would be paramount to making sure we're not forgetting anything.
Your comment

However, this doesn't help people (much) who're using something like Plutarch where you're doing everything in an untyped way anyway (see the below discussion of Value for an example).

I wouldn't say Plutarch does anything in an untyped way at all. Plutarch carries around information in the type level for values now to indicate whether they're sorted, positive, non-zero, or neither, and this makes things much easier to work with than the last time I used plutus-tx.
And regardless, the abstractions in plutus-tx are leaky: it it perfectly legal to construct illegal Values since the newtype data constructors are exported, and perfectly legal to construct a TxInfo with, say, a fee field with an unsorted negative value. In Plutarch, this is prohibited at the type level. I don't want to have to check during code reviews whether every usage of a Value is properly sorted or normalized when the type system can do that for me.

The issue that I have with your comments regarding generating script contexts and TxInfo is that this is something we need. We have an inordinate amount of utility functions that operate on script contexts and TxInfos that don't qualify as complete scripts that would not make sense to do on chain. And the constraints of the actual chain (e.g., using something like plutip) mean that we're now writing double or triple the code, because we've got to write on-chain, off-chain, infrastructure, and wrappers to bundle everything into validators and can't get nearly the same level of expressiveness in error messages.
I don't have actual measurements, but I am positive that applying parameters to a plutus function and evaluating it will be much faster than building an entire transaction just to test one helper function. And even more so, we don't want to generate fully valid transactions. We want to generate just the parts of the TxInfo that we care about.  We want to generate Values and CurrencySymbols and values for the txInfoMint  field that we know make sense.
And just to be clear -- testing Validator scripts and minting policies is a very small percentage of what we want to do. It's important, for sure, but testing the utility functions are more critical. Its very difficult to compose a program with confidence when all of your requisite parts are totally untested.

And for your final comment:
I don't disagree that dApp development teams need SMEs on the spec, but it is a waste of time for every developer to need to have a copy of the spec sitting on their desk at all times just to write a legal inhabitant of a type. We're putting in more and more time working around this and demonstrating that it is possible to represent a large swath of this low-hanging fruit computationally.
Of course, the spec will always be the source of truth. A conceptual document will help, but its not as useful as wrapping that knowledge up and expressing in computationally.

For testing, I can recommend looking into https://github.com/mlabs-haskell/plutus-simple-model, it uses cardano-ledger directly but is otherwise really just like EmulatorTrace AFAICT.

It's unfortunate that tools such as Plutarch force people to break the abstraction and operate on the representation. Ironically, this is a case where we did try to present people with types that enforce the correct usage.

There's a very good reason for it: Substantial efficiency improvements.
Dealing with Value with no guarantees is much less efficient than if you have guarantees. E.g. https://github.com/input-output-hk/plutus/pull/4710 happened because there was no guarantee about 0s in the Value. In Plutarch, we (ab)use this information to give you efficient operations over Values. E.g. checking real equality is trivial if you know both are sorted, similar things are true for comparisons.
The operations in plutus-ledger-api use union on the underlying Maps before operating on them, but this is obviously quite inefficient.
If you have unnormalised maps, you can generally do the comparisons in O(n * log n) time for n entries.
If you have normalised maps, it's generally O(n).

I think it would be helpful to try and extract from this thread a list of specific places where we can adopt strategy 1 or 2.

But its very difficult to keep all of that in our heads

I mean, you only need to remember about the pieces you actually work with. But also it is simply true that Cardano transactions are pretty complicated, and if you are in the game of making Cardano transactions you can't really avoid that, although it depends how much of the complexity you use.

We want to generate just the parts of the TxInfo that we care about. We want to generate Values and CurrencySymbols and values for the txInfoMint field that we know make sense.

I think this is perhaps more promising angle. The local conditions on specific fields are more likely to be documentable or enforceable. The non-local conditions on the transaction as a whole would be quite difficult to replicate without duplicating the entirety of the ledger rules. Examples:

A type/smart constructors for valid CurrencySymbols: easy enough
A smart constructor for TxInfo that checks that the transaction balances etc. etc.: difficult without replicating the ledger rules entirely

We're putting in more and more time working around this and demonstrating that it is possible to represent a large swath of this low-hanging fruit computationally.

To be clear, there are two sources of pain in this duplication of the ledger logic:

Working it out and writing it
Maintaining it

And regardless, the abstractions in plutus-tx are leaky: it it perfectly legal to construct illegal Values since the newtype data constructors are exported

I guess we thought people would use that power responsibly. Perhaps it just needs a bigger warning.

Also on the subject of generation... I think a collection of quickcheck/hedgehog generators for the types in plutus-ledger-api would be very useful, and is something we'd be happy to maintain upstream. I mention this because I suspect you might already have written such a thing :)

I'm not up-to-date on the latest in Plutarch.

Plutarch carries around information in the type level for values now to indicate whether they're sorted, positive, non-zero, or neither, and this makes things much easier to work with than the last time I used plutus-tx.

My question is: why do you care about such things? My guess is: because you're operating on the underlying representation, and you want to write code that is faster by making use of those properties, rather than writing the more generic code that would work regardless.
I am fundamentally torn here. I think it's better software engineering to not write code that's so exposed to the tiny implementation details, but also I understand that performance is a key concern.

There's a very good reason for it: Substantial efficiency improvements.

I'm not saying there's no no reason for it, I'm just pointing out that you're breaking the abstraction that we gave you!


hallucinations

feature
Hello just wanted to say that the model works great in general although it seems to have an issue with visual and textual hallucinations for example if i ask the model what color is the car in the image is in an image that has no car in it just hallucinates something up similar things happen with text only requests where the model hallucinates facts all the time and the strange thing is is that it does this more often than vicuna the model it is based on and fixing these issues would make the model a lot more reliable  and could possibly be done though better instruction fine tuning on examples like i listed above
Anyway thanks for reading  my request i think llava has great potential to become an open source gpt 4 and eventually maybe even surpass it

Hi @Bensolz, thanks for the interest in our work and for the great feedback. The hallucination is definitely one of the most important weaknesses that we are striving to tackle with.  It is harder to tackle due to the existence of a separate vision encoder.  Please stay tuned for future updates, thanks!

thanks for the reply interestingly i noticed that mini gpt 4 seems to be less vulnerable to this and that uses Blip-2 s vision encoder so maybe just using a better vision encoder  and the latest checkpoint of vicuna could mitigate the issues

No Fine Tune Can Fix Hallucination. We need Architecture Redesign.


Update DevCGlobalDirectory.md

fixed typo for 'Hyderabad'. Additionally added details for DevC Bengaluru, DevC Vellore, DevC Ahmedabad and DevC Mumbai

What does this PR do?
Provides details for DevC Vellore, DevC Bengaluru, DevC Ahmedabad and DevC Mumbai and fixes typo for Hyderabad
Any relevant issues this PR addresses? (optional)
#
Any background context you want to add?
None

The community is still active and might be helpful for someone looking to
join in.
On Wed, Jul 29, 2020, 7:54 AM Saransh Kataria<EMAIL_ADDRESS>wrote:

@saranshkataria requested changes on this pull request.
I don't see the point of adding Bangalore here if there are no leads
currently?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/fbdevelopercircles/FbDevcCommunityContent/pull/152#pullrequestreview-457151162,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AGM42BUCERTKK4M5XZ5RTDDR56B6DANCNFSM4PKWL6KA
.

@sangeetagupta2068 we have the Facebook website for that. The intent for the Github file is either to find the Github org for the community or the Github profile for the lead.

Thanks, Saransh! No worries. I can make the changes and resubmit the PR for
this. Can you also have a look at the issue I have raised for this file?
On Thu, Jul 30, 2020, 3:47 AM Saransh Kataria<EMAIL_ADDRESS>wrote:

@sangeetagupta2068 https://github.com/sangeetagupta2068 we have the
Facebook website for that. The intent for the Github file is either to find
the Github org for the community or the Github profile for the lead.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/fbdevelopercircles/FbDevcCommunityContent/pull/152#issuecomment-665959158,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AGM42BVIGEPQGAFBQ46W3CDR6CNX3ANCNFSM4PKWL6KA
.


Where would those projects fit?

Hey there,
I started working on those two a while ago:

https://github.com/karfau/runex (for "executing" a function in a module without much boilerplate)
https://github.com/karfau/node-cli-arguments-options (side project for comparing features/capabilities of some cli option/argument parsers)

If I would know where they would fit into your TOC I would be happy to file a PR, but I'm just not sure.
By the way would you also accept a PR that actually adds an easily maintainable TOC to your README? :wink:

hi! thanks for your interest!
re: runex - is this similar to execa? maybe put it there?
re: node-cli-args-options - this is interesting! honestly i dont do a great job of this on the cheatsheet as i was forced to split things between "Frameworks" and "Argument Parsing". I would put a link to this under "argument parsing" and just use the same description you gave here
re: TOCs, yes i prefer using https://github.com/technote-space/toc-generator these days but curious what options u have

ok :) use your judgment!
the toc-generator one is quite easy to set up. give it a try, ive had good experiences so far


vue3 how to use

Hi @pengyinghao, thank you for the information. :smile:

No reply came, then this abandoned issue is closed.


allDay event disappear

In Day View,  allDay event is not shown, when dayStartHour > 0
Ex.  dayStartHour="6"
I'm sure it's because of position calculations default from 0 hour.

Good catch! I'll get that fixed in the next release, thanks for reporting 😄

Was this fixed? I'm having the same issue with version<EMAIL_ADDRESS>


Does not compile on Apple Silicon

It returns the following after running TARGET=x86_64-linux-musl make
g++ -no-pie   -g -O2 -DIN_GCC  -DCROSS_DIRECTORY_STRUCTURE  -fno-strict-aliasing -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wmissing-format-attribute -Woverloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings   -DHAVE_CONFIG_H  -o cc1 c/c-lang.o c-family/stub-objc.o attribs.o c/c-errors.o c/c-decl.o c/c-typeck.o c/c-convert.o c/c-aux-info.o c/c-objc-common.o c/c-parser.o c/c-fold.o c/gimple-parser.o c-family/c-common.o c-family/c-cppbuiltin.o c-family/c-dump.o c-family/c-format.o c-family/c-gimplify.o c-family/c-indentation.o c-family/c-lex.o c-family/c-omp.o c-family/c-opts.o c-family/c-pch.o c-family/c-ppoutput.o c-family/c-pragma.o c-family/c-pretty-print.o c-family/c-semantics.o c-family/c-ada-spec.o c-family/c-ubsan.o c-family/known-headers.o c-family/c-attribs.o c-family/c-warn.o c-family/c-spellcheck.o i386-c.o glibc-c.o \
	  cc1-checksum.o libbackend.a main.o libcommon-target.a libcommon.a ../libcpp/libcpp.a ../libdecnumber/libdecnumber.a libcommon.a ../libcpp/libcpp.a ./../intl/libintl.a -liconv  ../libbacktrace/.libs/libbacktrace.a ../libiberty/libiberty.a ../libdecnumber/libdecnumber.a   -L/Users/jlricon/gits/musl-cross-make/build/local/x86_64-linux-musl/obj_gcc/./gmp/.libs -L/Users/jlricon/gits/musl-cross-make/build/local/x86_64-linux-musl/obj_gcc/./mpfr/src/.libs -L/Users/jlricon/gits/musl-cross-make/build/local/x86_64-linux-musl/obj_gcc/./mpc/src/.libs -lmpc -lmpfr -lgmp   -L./../zlib -lz
clang: warning: argument unused during compilation: '-no-pie' [-Wunused-command-line-argument]
Undefined symbols for architecture arm64:
  "_host_hooks", referenced from:
      c_common_no_more_pch() in c-pch.o
      toplev::main(int, char**) in libbackend.a(toplev.o)
      gt_pch_save(__sFILE*) in libbackend.a(ggc-common.o)
      gt_pch_restore(__sFILE*) in libbackend.a(ggc-common.o)
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[3]: *** [cc1] Error 1
make[2]: *** [all-gcc] Error 2
make[1]: *** [obj_gcc/gcc/.lc_built] Error 2
make: *** [all] Error 2

See following gcc bug

So it looks like we need a backport of the GCC fix to all supported GCC versions. How hard is that going to be? IMO it should strip out useless hunks like the line number changes so that the diff is minimal and ideally a single patch applies to all affected versions.

Still the same error with
0019-backport-MacOS-11.0-support-to-gcc-9.2.0.patch.txt which is expected as the error occurs in gcc for which the patch doesn't add -Wl,-undefined,dynamic_lookup

gcc-10.3.0 builds TARGET=aarch64-linux-musl and seems to work with minimal patching. I just noticed that the error reported by @jlricon is not related to the gcc bug. Following patch fixes that link error.
diff -ru a/gcc/config/host-darwin.c b/gcc/config/host-darwin.c
--- a/gcc/config/host-darwin.c	2021-04-08 13:56:28.000000000 +0200
+++ b/gcc/config/host-darwin.c	2021-04-20 23:05:04.000000000 +0200
@@ -22,6 +22,8 @@
 #include "coretypes.h"
 #include "diagnostic-core.h"
 #include "config/host-darwin.h"
+#include "hosthooks.h"
+#include "hosthooks-def.h"

 /* Yes, this is really supposed to work.  */
 /* This allows for a pagesize of 16384, which we have on Darwin20, but should
@@ -78,3 +80,5 @@

   return ret;
 }
+
+const struct host_hooks host_hooks = HOST_HOOKS_INITIALIZER;

Blindly copied from other host-*.c files.
For TARGET=aarch64-linux-musl support for {march,cpu,tune}=native has to be disabled:
diff -ru a/gcc/config/aarch64/aarch64.h b/gcc/config/aarch64/aarch64.h
--- a/gcc/config/aarch64/aarch64.h	2021-04-08 13:56:28.000000000 +0200
+++ b/gcc/config/aarch64/aarch64.h	2021-04-20 22:41:03.000000000 +0200
@@ -1200,7 +1200,7 @@
 #define MCPU_TO_MARCH_SPEC_FUNCTIONS \
   { "rewrite_mcpu", aarch64_rewrite_mcpu },

-#if defined(__aarch64__)
+#if defined(__aarch64__) && ! defined(__APPLE__)
 extern const char *host_detect_local_cpu (int argc, const char **argv);
 #define HAVE_LOCAL_CPU_DETECT
 # define EXTRA_SPEC_FUNCTION

I've have only build TARGET=aarch64-linux-musl and did minimal testing.

@jlricon if I may ask, what sort of config were you using to build this? I'm getting many errors with the clang compiler on the M1, especially in the GCC configure script, which checks if the compiler is "working".

Can you try using https://github.com/richard-vd/musl-cross-make/commit/b298706d99a30c72c26ba82c46b5bbd8550f5296? That is like https://github.com/richfelker/musl-cross-make/commit/fe915821b652a7fa37b34a596f47d8e20bc72338 with PRs https://github.com/richfelker/musl-cross-make/pull/148, https://github.com/richfelker/musl-cross-make/pull/140 and https://github.com/richfelker/musl-cross-make/pull/129 merged.

Can you try using richard-vd@b298706? That is like fe91582 with PRs #148, #140 and #129 merged.

I'm no the original commenter, but I just built this patch/commit and it finished without error (MacOS 12.4, M1 chip). I used TARGET=aarch64-linux-musl make -j so now I'm waiting for the laptop to cool back down. 😅

Thanks @hoosierEE for posting your result!
@richfelker could you please merge PRs #148, #140 and #129?

I'm getting this same error (Undefined symbols for architecture arm64: "_host_hooks") when building riscv64-linux-musl. Does anyone happen to know how to adjust the patches to fix the error with the riscv64-linux-musl target?


Project CLA

@ryanshiroma Can you please sign the project CLA for datacompy project? If you plan to submit a PR in the near future, CLA-Assistant bot will prompt to to sign. If you aren't planning to submit any new PRs to the project, pls use this link to sign: https://docs.google.com/forms/d/e/1FAIpQLSfwtl1s6KmpLhCY6CjiY8nFZshDwf_wrmNYx1ahpsNFXXmHKw/viewform
We appreciate your contributions to the project and your understanding on the contribution requirement for CLA.

@theianrobertson Can you please sign the project CLA for datacompy project? If you plan to submit a PR in the near future, CLA-Assistant bot will prompt to to sign. If you aren't planning to submit any new PRs to the project, pls use this link to sign: https://docs.google.com/forms/d/e/1FAIpQLSfwtl1s6KmpLhCY6CjiY8nFZshDwf_wrmNYx1ahpsNFXXmHKw/viewform
We appreciate your contributions to the project and your understanding on the contribution requirement for CLA.

@theianrobertson @ryanshiroma Just following up here. Pls let me know if you have questions.

@fdosani Please let me know if there is a better medium by which to contact @theianrobertson and @ryanshiroma to obtain the aforementioned CLA signatures. Thanks!

Signed

Hey sorry for the delay, signed.

Closing, this has been completed.


Add some convenience methods to EitherOrBoth

Ping

Makes total sense. Let's edit the docs then it's done? Thanks

Thanks!

bors: r+


[stable/maintainers]: add new reviewers and maintainers to Elasticsearch and Fluentd charts

Signed-off-by: Hector Fernandez<EMAIL_ADDRESS>
What this PR does / why we need it:
Add additional maintainers and reviewers to avoid piling up stale PRs or issues.
Special notes for your reviewer:
Checklist
[Place an '[x]' (no spaces) in all applicable fields. Please remove unrelated fields.]

[x] DCO signed
[x] Chart Version bumped
[ ] Variables are documented in the README.md
[ ] Title of the PR starts with chart name (e.g. [stable/chart])

/assign @maorfr @Towmeykaw

/approve

/lgtm
good luck!


Add support for github enterprise

Summary
Adding support for Github enterprise.  Resurrecting some of what was done in the v1 codebase in #252 and re-implementing in the v2 codebase. Fixes #329
Description for the changelog
#329 - Adds support for Github enterprise by adding the hostname (optionally port and protocol) to the config
Testing Done
I manually tested with a Github Enterprise Server instance and confirmed that I could log in, view repositories and branches, and create threat models.
Other info
There are a few TODO's that I'd like to address before this ships:

td.vue has a few hard coded references to www.github.com in the template files and i'm not sure the best way to plumb through the .env configuration here without just going straight to process.env and duplicating logic already present in td.server
Add some unit tests

I also ran into an issue while testing which seems like an issue in the status quo as well. The github API by default only returns the 30 repos and 30 branches. This is a problem in my environment as there are far more repos and branches than that. There probably needs to be some refactoring to support paging in the UI. I would like to avoid coupling that refactor with this change.

This is really really good to see - thankyou @danielpops
The v2-development branch is historical now, we just did not want to remove it, so did you want to rebase it back to main?

@danielpops did you want to raise an issue on the paging of 30 repos/branches?

The v2-development branch is historical now, we just did not want to remove it, so did you want to rebase it back to main?

Ha! That's what I get for not paying attention for 6 months 😅
Ya no worries I'll rebase the changes

@danielpops did you want to raise an issue on the paging of 30 repos/branches?

Will do

Also td.vue can ask td.server for the correct URL, I can look at this if you like?

If you can point me to an example of something similar, I'm happy to include it. If it gets more complicated, I'd appreciate your help 😅

I recall that I struggled to be able to test this last time ... my introductory period with github enterprise ran out ... I can try again

I'm happy to pair with you on this if you want

The express server has an api that you can extend at td.server/src/config/routes.config.js, probably an unauthenticated API is your best choice - a bit like healthz maybe

Hello @danielpops - do we want to merge this in if it is working? I can not test it myself, and I am happy on you testing it and giving the OK for merge.
What is the effect of the two fix-me?

@jgadsden sorry I hadn't circled back to this yet.

we can probably mutate this in the vuex store state.repo.selected or similar

I'm not sure how to achieve that OR the previous suggestion of adding an unauthenticated API, but I'm happy to do that with some guidance or pair on it with you or hand this off back to you :)

@jgadsden sorry I hadn't circled back to this yet. For the two fix-me's:

we can probably mutate this in the vuex store state.repo.selected or similar

I'm not sure how to achieve that OR the previous suggestion of adding an unauthenticated API, but I'm happy to do that with some guidance or pair on it with you or hand this off back to you :)

This can be ignored. It was because I accidentally signed in to the wrong
account.
[image: Bio]
Daniel Popescu | Security Engineer, Yelp Inc.
@.***
Yelp https://www.yelp.com | Engineering Blog
https://engineeringblog.yelp.com/
On Mon, Dec 5, 2022 at 4:36 PM 'dpopes' via svc-yelp-cde-ro github service
account inbox @.***> wrote:

@jgadsden https://github.com/jgadsden sorry I hadn't circled back to
this yet. For the two fix-me's:
we can probably mutate this in the vuex store state.repo.selected or
similar
I'm not sure how to achieve that OR the previous suggestion of adding an
unauthenticated API, but I'm happy to do that with some guidance or pair on
it with you or hand this off back to you :)
—
Reply to this email directly, view it on GitHub
https://github.com/OWASP/threat-dragon/pull/546#issuecomment-1338490432,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ATNNXUAVGPOICRALPOO7PQLWL2DCFANCNFSM6AAAAAARD7HGGE
.
You are receiving this because you commented.Message ID:
@.***>

Hello @danielpops , I will have a look and see if I can provide an API on the express server to supply the URL

Hello @danielpops - I have created pull-request #569 'Add server API for provider details' and added you as a reviewer. I think this will do what you need to get the configured hostname from the express server back-end

I think this will do what you need to get the configured hostname from the express server back-end

@jgadsden I had problems getting this to work, mostly due to my lack of experience with node (I wasn't able to figure out the right way to reference the async API in the vue files). I also had problems iteratively making changes and testing because I had to keep re-building in docker each time which made it difficult for me to try enough things quickly enough to figure it out on my own. I'm happy to ship this as is and hand it off to you to fix the base URL in a follow-up, or if you want to send some hints / diffs I'm happy to incorporate that into my branch. Sorry for the hand holding :(

Hello @danielpops
I hope you do not mind, but to merge this I think I need to back out the changes to README.md and env.md
github is saying that the changes are too complicated to merge, and I know we did a lot of work to these two files
Once we merge your changes in then I can reinstate your documentation, if that is OK with you?


[PROCESS] Proposal for a new process roll out protocol changes via RFCs

Motivation
Whenever we design and propose a protocol change, (e.g., a new table feature), we go through a preview / experimental phase during which the details of the protocol can undergo changes. Until the protocol change is completely tested and finalized, it cannot be guaranteed that the tables created with the preview feature will be forward compatible with any Delta reader/writer. However, today, this "preview phase" is not properly communicated as we directly merge any new proposal into the final protocol doc without proper annotation about its "preview" nature. This has led to a number of issues in the past. For example: CheckpointV2 was released in Delta 3.0, and immediately after that it was reported that certain details of the implemented protocol are different from the documented protocol. We had to update the protocol doc after the implementation and tables generated by that implementation were released in the wild.
New RFC Process Proposal
To avoid confusions like this, let us have a RFC process that clearly separates features in preview from finalized features with future compatibility guarantees. The proposed process is as follows.
RFC structure

Location: All preview protocol changes will be located in a new directory <delta_repo_root>/protocol_rfcs/ . Contents of the directory will be as follows:

In the directory, each proposed change will be stored in a markdown file named <table_feature_name>.md (or some other meaningful name that makes it easy to identify).

The directory will have index.md that will contain the list of current and past RFCs along with their status - opened on <date>, accepted on <date>, rejected on <date>.

Template of an RFC doc: Each doc must be in markdown format. Contents of the markdown:

Title: Name of the protocol change / table feature

Link to the Github issue with additional context and discussions

New or existing sections of the final protocol doc that need to be modified

For existing sections, the changed content must be made clear (e.g., new content highlighted in bold/italics, removed content u strikethrough).

Steps of rolling out a protocol change
1. Make initial proposal
Create a Github issue of type [Protocol Change Request].

This will be a new type of issue template (see the 2 existing ones) which will define these steps in it.
The description of the issue may have links to design docs, etc.
This issue will serve as the central location for all discussions related to the protocol change.

If the proposal comes with a prototype or other pathfinding, the changes should be in an open PR.
2. Add the RFC doc
After creating the issue and discussing with the community, if a basic consensus is reached that this feature should be implemented, then create a PR to add the protocol RFC before merging code in master.

Clone the RFC template above and create a new RFC markdown doc.
Cross-link with the issue with "see #xxx". DONT USE "closes #xxx" or "fixes #xxx" or "resolves #xxx" because we don't want the issue to be closed when this RFC PR is merged.

Changes related to the feature should only merge after the RFC is accepted. In addition, for table features, it is strongly recommended that any experimental support for the feature uses a temporary feature name with a suffix like -dev. This will communicate to the users that are about to use experimental feature with no future compatibility guarantee.
3. Finally, accept RFC into the protocol
For a RFC to be accepted, it must satisfy the following criteria:

There is a production implementation (for example, in delta-spark) of the feature that has been thoroughly well tested.
There is at least some discussion and/or prototype that ensure the feasibility of the feature in Delta Kernel

When the success criteria are met, then the protocol can be finalized by making a PR to make the following changes:

Closely validate that the protocol spec changes are actually consistent with the production implementation.
Cross-link the PR with the original issue with "closes #xxx" as now we are ready to close the issue.
Update protocol.md
Move the RFC doc to the accepted subdirectory, and update the state in index.md
Remove the temporary/preview suffix like -dev in the table feature name from all the code.

3'. Alternative: reject the RFC
If the RFC is to be rejected, then make a PR to do the following changes:

Moved the RFC doc to the rejected subdirectory
Update the state in index.md
Remove any experimental/preview code related to the feature.

Next steps to formalize this process

I will add create the RFC directory and add the RFC template. I will include this process as a markdown as well - UPDATE: done #2601
All future protocol changes, should follow this process. For example, new proposals like #2532 should go through this process.

Please leave your feedback and questions as comments. I will keep updating the proposal based on them. Thanks in advance.

I will add create the RFC directory and add the RFC template

Should we link to that PR here, e.g.

https://github.com/delta-io/delta/pull/2601 will create the RFC directory and add the RFC template

@tdas @ryan-johnson-databricks - great initiative coming up with a more formal process how features make it into the protocol as well as the implementations - this will strengthen the community quite a bit I believe.
Given that the Kernel implementations are still in a raw state, I think it is good to require "just" some implementation and a feasibility discussion on a kernel implementation. However going forward since the kernels may be considered reference implementations of the protocol, would it maybe make sense that there is an implementation in at least one kernel or that there is at least a committment form someone to add it such that the community (of connectors) can immediately start to benefit once a feature has landed?

That's a good point. And yes, once the kernels have stabilized (so that adding a new feature doesn't require adding missing pieces of kernel implementation first), RFC acceptance should be contingent on a working kernel implementation of the feature.

The first version of the proposal, I had put down kernel implementation as
a requirement, but then iterated to soften it up given the current Kernel
state. So I completely agree that when at least one of the Kernel
implementations mature, we should consider making that a stronger
requirement.
On Mon, Feb 5, 2024 at 4:26 PM Ryan Johnson @.***>
wrote:

That's a good point. And yes, once the kernels have stabilized (so that
adding a new feature doesn't require adding missing pieces of kernel
implementation first), RFC acceptance should be contingent on a working
kernel implementation of the feature.
—
Reply to this email directly, view it on GitHub
https://github.com/delta-io/delta/issues/2594#issuecomment-1928122947,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAFB5LBUNLTNSU65ZU2BJBTYSFFAZAVCNFSM6AAAAABCVMEDRGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMRYGEZDEOJUG4
.
You are receiving this because you were mentioned.Message ID:
@.***>

This process is now enforced. Closing this issue.


Document CAPO cluster templating in GS docs

https://docs.giantswarm.io/ui-api/kubectl-gs/template-cluster/
Follow up after https://github.com/giantswarm/giantswarm/issues/19400 is done.

Done with https://github.com/giantswarm/docs/pull/1309


Difficulty Adjustments

Changed scaling for HP/ATK on higher Difficulty
Added LevelPropertiesModified rule to reduce resources at higher difficulties

My play testing shows these work, but I need more feedback before making additional adjustments. So working and need you to scope for errors.


Update sp-update-jobstep-transact-sql.md

The job step has no owner, only the job itself.

Learn Build status updates of commit 68a813f:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs/relational-databases/system-stored-procedures/sp-update-jobstep-transact-sql.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the learn.microsoft.com contributor guidesPost your question in the Learn support channel

@markingmyname
Can you review the proposed changes? IMPORTANT: When the changes are ready for publication, add a #sign-off comment to signal that the PR is ready for the review team to merge.
#label:"aq-pr-triaged"
@MicrosoftDocs/public-repo-pr-review-team

#sign-off


Why is this happening wrong?

Figure 1 is ok, but Figure 3 is problematic. Mybatis recognizes #{name} as an attribute of the interface parameter.

@1397649957 Thanks for your reporting!
We can improve to become same behavior with bind variable(#{...}). I've submit PR for fixing this via #1487 . For details, please see #1487 .


Migrate from sshtools.j2ssh to a dependency that includes up-to-date algorithms

Describe the issue
Using SFTP resulted in this error:
2023-05-03 13:35:50,371 ERROR [starting Adapter ***] {} core.Adapter - Adapter [***] got error starting PipeLine
nl.nn.adapterframework.core.PipeStartException: could not start: Cannot open fileSystem: Cannot connect to the FTP server with domain [***]: (IOException) The socket is EOF

and later on (in another adapter)
2023-05-03 13:35:50,454 ERROR [Transport protocol b] {} transport.TransportProtocolCommon - The Transport Protocol thread failed
com.sshtools.j2ssh.transport.kex.KeyExchangeException: No suitable key exchange algorithm could be agreed.
2023-05-03 13:35:50,455 ERROR [starting Adapter ProcessFileOctopus] {} core.Adapter - Adapter [***] got error starting PipeLine
nl.nn.adapterframework.core.PipeStartException: could not start: Cannot open fileSystem: Cannot connect to the FTP server with domain [***]: (KeyExchangeException) No suitable key exchange algorithm could be agreed.

Reporter
@philipsens
Screenshots

Configuration
<SenderPipe name="Read">
  <FtpFileSystemSender name="ReadFile" action="read" host="${ftp_hostname}" port="${ftp_port}" authAlias="sftp_competensys" ftpType="SFTP" fileType="BINARY" passive="false" proxyTransportType="3" prefCSEncryption="aes256-ctr" prefSCEncryption="aes256-ctr" prot="S" remoteDirectory="${ftp_remoteDirectory}">
    <Param name="filename" sessionKey="originalMessage"/>
  </FtpFileSystemSender>
</SenderPipe>

Environment
FF! 7.9-20230425.190152: Frank2SittardGeleen
Running on 0ad118099ffe using Apache Tomcat/8.5.85
Java Version: OpenJDK Runtime Environment (1.8.0_362-b09)
Heap size: 131M, total JVM memory: 230M
Free disk space: 34GB, total disk space: 45GB
Up since: 2023-05-03 15:35:46 (6m)

Additional Context
https://stackoverflow.com/questions/35307393/the-transport-protocol-thread-failed-the-socket-is-eof-with-j2ssh-connecti

Is it possible to use FTPS instead of SFTP?


reCAPTCHA error. Invalid domain for site key.

I entereg correct API keys and added site url reCAPTCHA admin panel.
If i enabled reCAPTCHA phpfox admin panel im getting "Invalid domain for site key." error.

I noticed another you issue had with passing data over HTTP(s). Do you know if your server is blocking external calls to web apis?

I think no.

Im still waiting for this issue can you help me please?

Since we cannot reproduce this on our community, nor can we reproduce it locally, you need to get in touch with us and provide us with your sites URL and AdminCP login details.

Its now fixed after i recreate app and update api keys.
It was old api key for recaptcha v1.


Integrating JavaScript Library/Function into Flutter Web Application

I am trying to integrate Braintree payment processing into my Flutter web application. However, the Braintree client facing UI to collect payment information is all done in JavaScript. How would I go about adding this functionality? I basically need to display JavaScript in my web application.
Here is the documentation on Braintree: https://developers.braintreepayments.com/start/hello-client/javascript/v3

Try https://pub.dev/packages/easy_web_view

Hi @nyilkhan
Please usejs
If you have more questions,  Please see https://flutter.dev/community for resources and asking questions like this, you may also get some help if you post it on Stack Overflow.
Closing, as this isn't an issue with Flutter itself,if you disagree please write in the comments and I will reopen it
Thank you


[Docs] Remove as_str() call from event.menu_item_id()

There isn't a method as_str() associated to the MenuIdRef returned by menu_item_id()

What kind of change does this PR introduce? (check at least one)

[ ] Bugfix
[ ] Feature
[x] Docs
[ ] New Binding Issue #___
[ ] Code style update
[ ] Refactor
[ ] Build-related changes
[ ] Other, please describe:

Does this PR introduce a breaking change? (check one)

[ ] Yes. Issue #___
[x] No

The PR fulfills these requirements:

[ ] When resolving a specific issue, it's referenced in the PR's title (e.g. fix: #xxx[,#xxx], where "xxx" is the issue number)
[ ] A change file is added if any packages will require a version bump due to this PR per the instructions in the readme.

If adding a new feature, the PR's description includes:

[ ] A convincing reason for adding this feature (to avoid wasting your time, it's best to open a suggestion issue first and wait for approval before working on it)

Other information:

Nice catch, thanks!


fix(dialog): capture previously focused element immediately

Captures the previously-focused element immediately, instead of waiting until the animation is done. This fixes a regression from #3774, because if we wait for the animation to finish, the focus might have shifted, or the element could have been disabled.

Addressed the feedback @jelbourn.

@crisbeto Can you rebase?


label of each part is inconsistent with the raw partnet

In your voxelized models, each of them has several parts represented by channels of the voxel, but I cannot figure out what the label of each channel/part is. For instance, 36354.h5 (Chair) has 9 channels/parts, but in raw partnet, model 36354 has 6 level-1 annotations, 8 level-2 annos and 12 level-3 annos, non of them is 9:

For chair category, we processed the data using level-1 annotation and further split chair legs (chair_base in this example) based on connectivity.

I see. In addition, what's the mapping between channel index and part label? For instance, voxel[0, :, :, :] corresponds to back_surface, etc

The order should be consistent with the order specified in partnet annotation. But for the mapping between channel index and part label, I can't say for sure, since we give no emphasis on part semantic labels.

I see. Thanks a lot!


Player rebeuffers

If I go back 10 seconds in the playback it rebuffers which is annoying if I'm on limited data

Working with this issue, will be fixed in next release, thank you.


Refactoring build and launch configurations

add 3 Realsense cameras configuration
add NUC + Jeston Mate configuration
refactoring docker-compose YAML files to support multiple configurations
use environment variables instead of command-line options to change settings
update launch.sh for multiple configurations
refactoring prebuild-docker.sh, build-docker.sh for building by Github actions
update Readme
add .dockerignore files to prevent from rebuilding image when re-run tools/setup-thirdparty-repos.sh
add script to fix timezone issue with images pulled from dockerhub

@muratams
Could you review the usage of environment variables to control settings?
https://github.com/CMU-cabot/cabot/blob/dev-wip/README.md#env-file

I think there is no problem in the usage of environmental variables for now.
I have some comments on README.
The difference between CABOT_MODEL and CABOT_NAME might be unclear.
Why are the environmental variables that we must be careful to set placed at the end of .env file section (Others section)? I think these environmental variables are not necessarily to be set.

Updated the .env section

list required settings first (launch.sh checks these variables)
add descriptions to CABOT_MODEL and CABOT_NAME
move others to optional settings


Bug: listBatchMessages does not handle Date parameters

Describe the bug
it seems the listBatchMessagesmethod has an issue with dates :
it expects the before and after to be Date objects but when I try to send a date it says ERROR in listBatchMessages [Error: The 6th argument cannot be cast to type Double?
also it supposedly returns DecodedMessage objects where sent is of type Date but here I actually have decodedMessage.sent = 709725751.7279999 (what does this represent? It’s not a timestamp - this message has been sent recently)
Expected behavior
No response
Steps to reproduce the bug
No response

Closed, resolved with https://github.com/xmtp/xmtp-react-native/pull/71

Release in https://github.com/xmtp/xmtp-react-native/releases/tag/v1.6.1