Authz/Pre-Authz 2.0.0 Service Specification Proposal

Context
It has been 3 years since the original FCL specification outlining authorization services (known as authz & pre-authz) was defined. In combination, these two services aimed to resolve one or more signatures for a transaction using keys whose total signatory weight was ≥1000. Combined, these signatures would confirm that accounts involved in each transaction role had approved the transaction’s execution.  While FCL’s authorization services have since proven themselves to be extremely robust, it seems that the time has come where these services have begun to outlive their foreseen use-cases.
The problem
Recently, the community has voiced its concerns regarding shortcomings of the current authorization model.  Particularly, discussion raised in #1679 has elicited the need for change to how authorization services operate within FCL.  However, this PR has not yet been merged as it failed to address the underlying issue and only offered a short-term solution.  We have listened to these community concerns and are proposing robust upgrade to the FCL specification to help support these emerging requirements.
Proposal
This proposal outlines a refined, key-agnostic approach by which account authorizations would be resolved.  Namely, a new configuration for authz services in FCL would mean that signatures are no longer resolved for specific account keys, but rather via one or more signatures from any keys on the account.
In addition to adding the key-agnostic specification to FCL, Flow client libraries must continue to support the key-specific authz service structure for two main reasons:

Proposer authorizations are not key-agnostic by definition.  All transaction payload signatures are generated based on a predetermined proposer key.  This cannot change during the authorization step and must be established prior (i.e. pre-authz).  For this reason, there must remain a authorization method valid for the transaction proposer.
FCL must guarantee compatibility with existing wallets & authz service implementations.  In future iterations of the FCL protocol, it is possible that this could become a breaking change, but for now support must remain for key-specific authz services.

In this document, we will establish the following terminology:

key-specific authz services - the current authz service model
key-agnostic authz services - the proposed addition to the authz service model (remove/optional keyId field)

Authz Service Structure
Because of changes to the schema of the authz service, it is necessary that the authz schema is given a new f_vsn to identify whether wallets/clients can support this functionality.
For the revised authz service, the key-specific configuration would remain identical to its predecessor v1.0.0 implementation, however with the version bumped to v2.0.0.
{
    "f_vsn": 2.0.0
    "f_type": "Service",
    "id": "foobar#authz-http-post",
    "type": "authz",
    "identity": {
        "address": "f086a545ce3c552d",
        "keyId": 615
    },
    "method": "HTTP/POST",
    "endpoint": "https://example.com/authz",
    "params": { ... }
}

In the above example, FCL expects that https://example.com/authz will respond with a signature for keyId=615 on account f086a545ce3c552d.  No other keys’ signatures would be valid responses.
Key-agnostic authz services would discard this assumption, instead conforming closer to the protocol-level definition of account authorization.  Rather than defining authorizations in terms of account-key pairs, they would be defined as a collection signatures using any number of arbitrary keys from a given account.
For the v2.0.0 authz specification, the keyId would become an optional parameter.  When describing a key-agnostic authz service, the only difference would be the removal of the "keyId" field.
{
    "f_vsn": 2.0.0,
    "f_type": "Service",
    "id": "foobar#authz-http-post",
    "type": "authz"
    "identity": {
        "address": "f086a545ce3c552d",
    },
    "method": "HTTP/POST",
    "endpoint": "https://example.com/authz",
    "params": { ... }
}

The above authz service defines an HTTP/POST request to [https://example.com/authz](https://example.com/authz) which should resolve one or more signatures for the account f086a545ce3c552d using any keys.
Authz Service Response
For key-specific authz services, the response was a CompositeSignature object defined as follows:
{
    ...polling response...
    "data": {
        "f_vsn": "1.0.0",
        "f_type": "CompositeSignature",
        "addr": "0x358529c1f0220750",
        "keyId": "1"
        "signature": "0db1718281a5c5587b5b4b026f12aed576b3e2dec5a297fab17c40a2c1e46f1b93fe537934bf0d5208c7cabe9196f24d4a0ba34a45ec958f8f9295139624cf3d"
    }
    ...
}

The notable limitation in this response is that only a single signature may be provided.  The proposed response for the revised authz service would also allow arrays of signatures.
Imagine an example authz service which is responsible for resolving an authorization for 0x358529c1f0220750 (like the example in the previous subsection).  Assume that this account has two keys: both with 500 weight and that the is authz service is responsible for resolving signatures for both these keys.  The response from this service could be as follows:
{
    ...polling response...
    "data": [
        {
            "f_vsn": "1.0.0",
            "f_type": "CompositeSignature",
            "addr": "0x358529c1f0220750",
            "keyId": "1" // 500 weight
            "signature": "0db1718281a5c5587b5b4b026f12aed576b3e2dec5a297fab17c40a2c1e46f1b93fe537934bf0d5208c7cabe9196f24d4a0ba34a45ec958f8f9295139624cf3d"
        },
        {
            "f_vsn": "1.0.0",
            "f_type": "CompositeSignature",
            "addr": "0x358529c1f0220750",
            "keyId": "2" // 500 weight
            "signature": "0db1718281a5c5587b5b4b026f12aed576b3e2dec5a297fab17c40a2c1e46f1b93fe537934bf0d5208c7cabe9196f24d4a0ba34a45ec958f8f9295139624cf3d"
        }
    ]
    ...
}

In this case, the authz service successfully obtains a 1000-weight authorization by using a combination of two keys’ signatures.  It is natural that key-agnostic authz responses are not restricted to a single CompositeSignature.  This design is more compatible with the protocol-level definition of account authorizations: the account authorization process may require an aggregate of signatures from multiple keys as opposed to a single key.  In some designs this can prevent redundant authz services and unnecessary API requests.
It is not a requirement that the cumulative weight of the signatures is ≥1000.  Multiple authz services may have to work together in order to achieve this.
Of course, a single CompositeSignature passed as a plain object, and not an array, would still be sufficient as well; e.g.
"data": {
    "f_vsn": "1.0.0",
    "f_type": "CompositeSignature",
    "addr": "0x358529c1f0220750",
    "keyId": "1"
    "signature": "0db1718281a5c5587b5b4b026f12aed576b3e2dec5a297fab17c40a2c1e46f1b93fe537934bf0d5208c7cabe9196f24d4a0ba34a45ec958f8f9295139624cf3d"
}

Multi-signer transactions?  Is there enough weight?
One question that may arise is the following: how does the removal of explicit account-key pairs from authz services affect multi-signer transactions.  Specifically, how can it be ensured that the combined signature resolution of all authz services would result in a satisfactory weight for transaction approval (1000)?
Ultimately, these concerns are not relevant.  It’s up to the authn-provider/wallet to provide a collection of authz services whose combination generate signatures of cumulative weight ≥1000 - this concern is out of the scope of FCL.  Largely speaking, these guarantees should likely hold true due to the natural configuration of the wallet-controlled APIs which these services rely upon.
However, should these services require additional metadata for key-orchestration, this can be defined by the pre-authz step and passed through "params" field of the authz service.  This may be a keyId the service is expected to sign with, a minimum signatory weight expected to be generated by the service, or other fields based on the application.
Overlapping Keys
One guarantee made by key-specific authz services is that two different authz services will not generate signatures using overlapping keys.  One question may be: how do we make the same guarantee with key-agnostic authz services.
This is not a concern for the same reasons as above with multi-signer transactions - key orchestration will tend to be an inherent guarantee of the wallet’s API endpoints themselves, and should it not be, metadata can be configured via authz service "params".
Compatibility Considerations
Ensuring backward compatibility is critical when making changes to the FCL spec. Wallets which wish to switch to key-agnostic authz service strategies may do so as long as the FCL client they are interacting with supports this feature.  The following outlines the version negotiation process.
Service Version Selection
The wallet and the client (FCL) will negotiate whether they are using the revised authz schema using the following technique:

The wallet will advertise all of the FCL services & versions of these which they are compatible with during the authentication step.  For example:
[
	...services...,
	{
		"f_vsn": "1.0.0",
		"f_type": "Service",
		"type": "authz"
		...
	},
	{
		"f_vsn": "2.0.0",
		"f_type": "Service",
		"type": "authz"
		...
	}
]

When executing a given service (i.e. execService('authz')), the FCL client will choose to execute the greatest compatible version supported by the wallet.

Using these two steps, it is possible to incrementally upgrade services from both the wallet & FCL while maintaining both-sided backwards compatibility.
So, to implement key-agnostic authz services from the wallet’s standpoint they will need to do add additional versioned services for the following:

pre-authz - If a wallet wishes to return key-agnostic authz services during their pre-authorization step, they must provide an additional v2.0.0 pre-authz service.  The prior v1.0.0 service should remain as well to support older clients.
authz - If a wallet wishes for their default authz service to be key-agnostic, they must provide an additional v2.0.0 authz service.  However, it is important to note that key-agnostic services are not valid for proposer authorizations.  If a wallet offers a key-agnostic authz service to the user, it is expected that the proposer authorization will be overridden by a key-specific authz service during a pre-authz step.  If this does not happen, the transaction will fail

FCL-JS Filtering
Unfortunately, FCL-JS does not implement service version selection prior to v1.7.0.  The service addition technique outlined above will not work for FCL versions <1.7.0.  As a workaround, wallets will have to use the fclVersion field provided in the authentication (sign in) step before they choose to provide the additional authz/pre-authz services.

Can we make this a FLIP please? Very hard to read and comment like this.

Hey @bluesign , thanks for your feedback on this proposal.  I'll be moving this to a FLIP shortly.

I am not sure why v2 auth is needed? If wallet has to know the client version. ( not sure when 1.7.0 released )

Versioning this proposal has been a really tricky problem.  Essentially, versioning of FCL specifications/features should not be FCL-JS specific.  We should be trying to avoid this tight coupling with the JavaScript library & FCL-JS versions as it makes it difficult to implement the FCL spec on different platforms (i.e. Swift, Kotlin, etc.).  This is supposed to be solved by the versioning system for FCL services (i.e. the v2 authz you referred to).
Unfortunately, this implementation has always been bugged (see https://github.com/onflow/fcl-js/pull/1785).  So if we used versioned services (v2) with the current FCL-JS implementation, it would break.  I.e. it isn't until this bug fix is released, 1.7.0, that versioned services become compatible.
Like I said... really nasty problem... and none of the solutions here feel really satisfactory.  I would have liked to avoid checking fclVersion on the wallet end, but to me this one feels like it sets the best precedent for future changes.

AFAIK Blocto's solution was backwards compatible? Just adding extra keyId, if exists just replacing keyId if signature is not proposer signature.

Blocto's solution is backwards compatible in the sense that their modified FCL would work with existing wallets.  However, it is not backwards compatible for the reverse situation - wallets using the keyId replacement will be incompatible with previous versions of FCL.  Could these wallets do a version check for FCL > 1.7.0... yeah, probably... but this goes full circle to the 1st point I addressed.

thanks @jribbink,
The issue I see is the requirement of providing v1 services with v2. ( for proposer go to v1 service etc ) Maybe I misunderstood that part.
My focus is not at v1 or v2 distinction, but if we make a v2 service maybe we can make it v1 compatible. ( so provider with v2 service doesn't have to also provide v1 )

moving discussion to https://github.com/onflow/flips/pull/215


Added my example code involving Debezium

Added my example code "MySQL Debezium Kafka Pinot CDC Flow" @ https://github.com/nmertaydin/mysql-debezium-vs-maxwell-kafka-pinot

@nmertaydin Applied, thanks a lot!

😭 Deploy PR Preview 5530328 failed. Build logs

🤖 By surge-preview

Oops @jpechane is this something known? ❓

No worries, @nmertaydin; sometimes the clean-up of the staging environment fails for reasons unrelated to the specific change at hand. I just cleaned it up. Thanks for adding your example!

No worries, @nmertaydin; sometimes the clean-up of the staging environment fails for reasons unrelated to the specific change at hand. I just cleaned it up. Thanks for adding your example!

You're welcome. 🙇


Commit current package-lock from npm i

(relevant discussion)
This is the package-lock generated by running our base npm install using NPM 6.4.1, which is the version that ships with Node 8.15.0, which is the version we have specified via asdf... at least, this is how it's generated on my machine, so I'd appreciate it if someone else could try an npm i on this branch and see if any changes show up!
The file is also generated the same way when I use NPM 6.7.0, the version recommended to manually install in the README.
This does not appear to involve the issue described in mbta/dotcom-legacy#2825.

Closing, as it turns out not everyone actually generates the same lockfile even when using identical versions of Node/NPM. This may be down to differences in exactly what already exists in your node_modules at the time you run the command. The diff appears harmless, at any rate, so we don't have to dig deep into this right now.


Azure Cosmos DB Gremlin graph support and documentation misleading here's why.

The Microsoft documentation for Azure Cosmos regarding using Grimlin API for their graph database is running on a really old version of .NET which is .NET 3 which is a bit misleading, here is the doc I'm referring to:
here
However what really is absolute here is that the Gremlin .NET nuget package is way ahead (3.6.1) on what currently supported by Azure Cosmos which is an older version (3.4.13) which caused me a headache and a lot of time to figure it out, I had to dig deep to realize that the latest version isnt supported by azure cosmos.
supported version
Available version
I am very well aware that in the quick start guide it is specifically said to download the older Gremlin .NET package however it is on .NET 3 so any developer who would want to make their application to run on .NET 6 would fall into the same issue as I have where you go to the nuget package manager via Visual Studio and just download assuming the fact that Microsoft Azure would definitely supported the latest version

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 2700fc00-4872-6fcf-65e0-ab9fd5fde2a1
Version Independent ID: e828bbb5-2c22-17f2-32aa-aa3773c55806
Content: Build an Azure Cosmos DB .NET Framework, Core application using the Gremlin API
Content Source: articles/cosmos-db/graph/create-graph-dotnet.md
Service: cosmos-db
Sub-service: cosmosdb-graph
GitHub Login: @manishmsfte
Microsoft Alias: mansha

A note has been added to the Clone the sample application section indicating that the API for Apache Gremlin does not support the latest driver. It also includes a link to the support drivers.

#please-close


Missing example or documentation

How do I use this? How do I round-trip a toml file? Which struct/enum is an "entry point" of the library?

What do you recomment to use? This library or tomllib?


Multiple window restore.

I'm working on migrating code to another project and it helps to have both projects open.  However when I exit only one window is restored when relaunching.
Please add better support for working with multiple projects at the same time.

/needsMoreInfo

@cheako, sounds like you need to check out the setting window.restoreWindows, setting to All.

It's not enough, even if it is the vscode method, to have a configuration option.
On Exit(I.E. File->Exit) there should be a check for more than one window open and a confirmation dialog with a checkbox(or radio for the case of never show again) to restore all windows.


Wrong Icons In the Prompt

I got wrong icons in the prompt. Although I tried to fix it myself, I can't really find the cause of this. If this is because of the codepoint, how could I find out which icon is wrong? Could someone help?

Thanks!

@bigapple10027 - All of the codepoints that P9k uses are defined here:
https://github.com/bhilburn/powerlevel9k/blob/master/functions/icons.zsh
So, just to be clear, you're saying you have this exact same set-up on another Mac with the same version of P9k, and it all works fine? My guess, honestly, is that the code-points in the font you are using, Roboto Mono, changed.
When you have P9k installed, you have access to get_icon_names in your terminal. Try running that on both systems. If you see a different icon for the same name, then my bet is you have different versions of the font installed.

Hi @bhilburn
Thanks a lot for the detailed information. Yeah, as you said, the icon with the same code point is indeed different on the two Macs. However, I'm wondering how can figure out which font is overwritten it? How do you guys usually debug and fix this problem? Thanks!

@bigapple10027 - I'm honestly not sure, to be honest with you. On Linux systems, you could use fontconfig tools, but I'm not familiar with what tools you can use on macOS to do the same :-/
Are you still trying to solve this? Have you found anything?


The type or namespace name "CSharp" does not exist in the namespace "Microsoft.CodeAnalysis"

Can be fixed by commenting the first line in "SerializedObjectUtils.cs"

fixed in 0.1.1, thanks!


Installation issue: starter-suid doesn't have setuid bit set

nextflow run christopher-hakkaart/nf-core-demo -profile test,singularity -r main

triggers .
ERROR ~ Error executing process > 'NFCORE_DEMO:DEMO:INPUT_CHECK:SAMPLESHEET_CHECK (samplesheet_test_illumina_amplicon.csv)'

Caused by:
  Process `NFCORE_DEMO:DEMO:INPUT_CHECK:SAMPLESHEET_CHECK (samplesheet_test_illumina_amplicon.csv)` terminated with an error exit status (1)

Command executed:

  check_samplesheet.py \
      samplesheet_test_illumina_amplicon.csv \
      samplesheet.valid.csv

  cat <<-END_VERSIONS > versions.yml
  "NFCORE_DEMO:DEMO:INPUT_CHECK:SAMPLESHEET_CHECK":
      python: $(python --version | sed 's/Python //g')
  END_VERSIONS

Command exit status:
  1

Command output:
  (empty)

Command error:
  INFO:    Environment variable SINGULARITYENV_NXF_TASK_WORKDIR is set, but APPTAINERENV_NXF_TASK_WORKDIR is preferred
  INFO:    Environment variable SINGULARITYENV_NXF_DEBUG is set, but APPTAINERENV_NXF_DEBUG is preferred
  ERROR  : Installation issue: starter-suid doesn't have setuid bit set

This is reproducible for nextflow run nf-core/sarek -profile test,singularity --outdir test_sarek -r 3.2.3 as well . --profile apptainer triggers the same

Fixed as per https://github.com/nesi/training-environment/commit/b9d6b53d997be96c78dbf82cb7b065586241a4e7 .i.e.

set enable_privileged_pods: to true


Enable riemann-elb-metrics to use IAM Instance profile

If no AWS keys are provided, use instance's profile.

Thanks @iramello!

Thanks @aphyr


Add guests

We want to give the option to add guests to a Team.
This might be a helpful resource: Using Power Automate And Graph API To Manage External Access To Teams
But we would need to do this in Azure Logic Apps.

fixed in release v3.0.0


add xid string validator

I've added an additional string validator to validate xid formatted strings.
Implements the feature requested by #2639
I've pretty much just done a global search on the codebase for cuid2/ulid, and added a matching entry for xid at each match, making slight changes where necessary.
Let me know if I'm missing anything!

Thanks this looks great! I'm merging this into the v4 branch. Some version of this will land in Zod 4, but (as a heads up) it may be a while, and this feature/API will likely undergo some changes in the interim.


package resolve does not save packages in cache

Right now resolving packages doesn't store packages in the cache.
This is in some situations problematic because "fully offline use" is not possible with the current implementation.
Except of the fact that it is also desired to cache packages to avoid unnecessary API calls.
Found out via this discussion.
See also this comment: https://github.com/apple/pkl/discussions/346#discussioncomment-8921705

My understanding is that packages are always cached unless --no-cache is specified.
Nevertheless, pkl project resolve --download could certainly be useful.

Instead of adding a --download argument, we're considering just always downloading (unless --no-cache is specified).

It’s about actively avoiding resolution, not actively avoiding download. When all I want is to download, I should never have to worry about PklProject.deps.json changing as a side effect. According to the docs, the download-package command already exists, so I’m not sure what you mean by “whole separate command”.

Avoiding resolution is fair enough. We were debating whether download-package should exist. Its current UI does not know about PklProject, through. This is why pkl download-package does not (yet?) work to download all project packages. Some more thinking required, but points taken.

Good point about pkl project resolve maybe not being deterministic in the future. But, I don't think it relates to "pkl project resolve should not populate the cache dir". Downloading dependencies is a matter of saving network I/O, and making subsequent pkl eval executions faster.
If we do want to lean on this being the method for "I want to download all my dependencies", we can maybe add a flag for "don't update already-resolved dependencies" in the future, if "latest" becomes a thing.
BTW: "I want to download all my dependencies" can also be a shell script:
cat PklProject.deps.json \
  | jq -c -r '.resolvedDependencies[] | select(.type == "remote") | (.uri[7:] + "::" + .checksums.sha256)' \
  | xargs -I {} pkl download-package {}

I guess I don’t mind if pkl project resolve also downloads packages, although I don’t really see the need either. I don’t think it saves much network I/O as packages will only be downloaded once in any case. Also, downloading as part of resolving won’t help all the other folks that didn’t run the resolve command themselves. Overall, not downloading by default feels more consistent with Pkl’s lazy attitude.
My gut feeling is that there should be an easy way to download all package dependencies without resolving. If the download-package command is here to stay, I’d expect it to be capable of this.

The resolve command does need to download in order to build the resolved dependency list. Since it's already downloading, it should also write them to the cache dir.

My gut feeling is that there should be an easy way to download all package dependencies without resolving. If the download-package command is here to stay, I’d expect it to be capable of this.

Fair; I can definitely see a mode in download-package that just downloads project dependencies. Maybe something like: pkl download-package --project=my-project/

Oh I thought that pkl project resolve would just download metadata. If it has to download the packages, it should definitely store them in the cache, at least by default.


Use h/w key derivation for witness key

This is not quite as straightforward as it might be because the h/w doesn't support the Ed25519 keys directly, instead I'm using the AES key it does support as a secure prng to seed HKDF which I steal a few bytes from for generating the friendly name, and then pass through to note's keygen func.
@smeiklej Does this look legit?

This does not look legit at all :).  Pseudorandom generators can only expand randomness, meaning if you start with a k-bit random string they can output a random string of length p(k) for some polynomial p.  If, as you're doing here, you start from a low-entropy seed (the current static string, but even the future proposed counter), it's easy for an attacker to guess keys, or even for witnesses to accidentally share keys and identities.
To be secure you'll need some source of entropy to get a (truly) random seed.  Things like the current time also don't work (too easy to guess), so hopefully there is some boot-time process that you can rely on to siphon off some random bits.

Thanks Sarah!

This does not look legit at all :). Pseudorandom generators can only expand randomness, meaning if you start with a k-bit random string they can output a random string of length p(k) for some polynomial p. If, as you're doing here, you start from a low-entropy seed (the current static string, but even the future proposed counter), it's easy for an attacker to guess keys, or even for witnesses to accidentally share keys and identities.

I would have though it should be hard for witnesses to accidentally share/guess keys - during the call to RPC.DeriveKey) the seed string is (I believe - @abarisani?) encrypted with the hardware's secret key which is unique per device.
Or are you saying that this is still produces an insufficient amount of entropy for the expansion?

To be secure you'll need some source of entropy to get a (truly) random seed. Things like the current time also don't work (too easy to guess), so hopefully there is some boot-time process that you can rely on to siphon off some random bits.

The runtime randomness is provided by a TRNG (w/ CAAM, w/ RNGB).
The key derivation is defined here and I think Al is mentioning that he his using a predictable rng (per device given that it's seeded with the device random + unique key) solely for the purpose of attestation key derivation.

Ah okay, this makes a lot more sense.  I still don't think it's a good idea though to use this custom blend of SHA256 and AES.  Really you should be using only standardized methods for deterministic RNG, as outlined at https://csrc.nist.gov/publications/detail/sp/800-90a/rev-1/final.  This means you could use CTR-DRBG with AES or Hash-DRBG or HMAC-DRBG with SHA256.  I'm not an expert in this space but believe that all these protocols come with a proof of security (with perhaps CTR-DRBG coming with the most caveats, from a quick skim of the literature).

Ah okay, this makes a lot more sense. I still don't think it's a good idea though to use this custom blend of SHA256 and AES. Really you should be using only standardized methods for deterministic RNG, as outlined at https://csrc.nist.gov/publications/detail/sp/800-90a/rev-1/final. This means you could use CTR-DRBG with AES or Hash-DRBG or HMAC-DRBG with SHA256. I'm not an expert in this space but believe that all these protocols come with a proof of security (with perhaps CTR-DRBG coming with the most caveats, from a quick skim of the literature).

We are not blending SHA256, the RNG is purely the output of AES with internal key and a diversifier, the reason for not implementing a different scheme is that we either cannot access the unique key (DCP, where it remains internal and can only be used for AES) or we wouldn't want to extract it to external RAM.
When using CAAM we will be able to have full memory encryption and that is where different schemes can be supported.

@AlCutter when CAAM is in place I'd suggest using the output of caam.DeriveKey with https://pkg.go.dev/github.com/canonical/go-sp800.90a-drbg (what do you think @smeiklej ?)

Yes, using a canonical implementation is even better!

Excellent, thanks both!
Ok, I'll park this PR until we get the rev2 boards with i.MX6UL and have the CAAM available, then I'll update.

Ah okay, this makes a lot more sense. I still don't think it's a good idea though to use this custom blend of SHA256 and AES. Really you should be using only standardized methods for deterministic RNG, as outlined at https://csrc.nist.gov/publications/detail/sp/800-90a/rev-1/final. This means you could use CTR-DRBG with AES or Hash-DRBG or HMAC-DRBG with SHA256. I'm not an expert in this space but believe that all these protocols come with a proof of security (with perhaps CTR-DRBG coming with the most caveats, from a quick skim of the literature).

We are not blending SHA256, the RNG is purely the output of AES with internal key and a diversifier, the reason for not implementing a different scheme is that we either cannot access the unique key (DCP, where it remains internal and can only be used for AES) or we wouldn't want to extract it to external RAM.
When using CAAM we will be able to have full memory encryption and that is where different schemes can be supported.

To correct my comment, the unique key derived with AES is fed to HKDF, used as DRGB for ECC key generation, this is the code I originally suggested to @AlCutter taken for our GoKey project:
// DeviceKey derives a device key, uniquely and deterministically generated for
// this SoC for attestation purposes.
func DeviceKey() (deviceKey *ecdsa.PrivateKey, err error) {
	iv := make([]byte, aes.BlockSize)
	key, err := imx6ul.DCP.DeriveKey([]byte(diversifierDev), iv, -1)

	if err != nil {
		return
	}

	salt := imx6ul.UniqueID()
	r := hkdf.New(sha256.New, key, salt[:], nil)

	return keygen.ECDSALegacy(elliptic.P256(), r)
}

@smeiklej in case we don't want to import canonical packages this still uses a standard expansion function as DRGB, does this look fine to you?

We have the new rev2 boards now, so it would be good to get this PR moving again.
I've tidied it up a bit in the meantime.
@smeiklej @abarisani

@smeiklej ping :)

If this is going to be a Google launch one day, it's probably worth asking ISE Crypto for their opinion as well?

Yup, good call - I've mentioned it to @meder.
I'm going to merge this as-is for now since it enables us to get on with other work this blocked, but it should be relatively easy to adapt to any changes to the key derivation coming from ISE.


nixos/tests: add check mariadb galera cluster

Motivation for this change
Moved mysql tests to subfolder.
Added check mariadb galera cluster.
cc @flokli @aanderse @worldofpeace @disassembler.
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[x] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[ ] Fits CONTRIBUTING.md.

@GrahamcOfBorg test mariadbGalera

@GrahamcOfBorg test mariadbGalera

@GrahamcOfBorg test mariadbGalera

@GrahamcOfBorg test mariadbGalera

@GrahamcOfBorg test mariadb-galera

@GrahamcOfBorg test mariadb-galera

@GrahamcOfBorg eval

@Izorkin Ran the test locally, it seems it's stuck at
galera_03_rsync # [  201.426407] mysqld[893]: 2020-04-04 22:46:55 0 [Note] WSREP: resuming provider at 10
galera_03_rsync # [  201.429545] mysqld[893]: 2020-04-04 22:46:55 0 [Note] WSREP: Provider resumed.
galera_04_rsync # [  142.918417] rsyncd[1436]: name lookup failed for <IP_ADDRESS>: Temporary failure in name resolution
galera_04_rsync # [  142.920801] rsyncd[1436]: connect from UNKNOWN (<IP_ADDRESS>)
galera_04_rsync # [  142.924294] rsyncd[1436]: rsync to rsync_sst/ from UNKNOWN (<IP_ADDRESS>)
galera_04_rsync # [  142.925897] rsyncd[1436]: receiving file list
galera_04_rsync # [  142.972423] rsyncd[1436]: sent 48 bytes  received 193 bytes  total size 41
galera_03_rsync # [  216.515459] mysqld[893]: 2020-04-04 22:47:10 0 [Note] WSREP: 0.0 (galera_03): State transfer to 1.0 (galera_04) complete.
galera_04_rsync # [  142.978348] mysqld[903]: 2020-04-04 22:47:10 0 [Note] WSREP: 0.0 (galera_03): State transfer to 1.0 (galera_04) complete.
galera_03_rsync # [  216.517765] mysqld[893]: 2020-04-04 22:47:10 0 [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: 5)
galera_03_rsync # [  216.521805] mysqld[893]: 2020-04-04 22:47:10 0 [Note] WSREP: Member 0.0 (galera_03) synced with group.
galera_03_rsync # [  216.523508] mysqld[893]: 2020-04-04 22:47:10 0 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 5)
galera_04_rsync # [  142.983994] mysqld[903]: 2020-04-04 22:47:10 0 [Note] WSREP: Member 0.0 (galera_03) synced with group.
galera_03_rsync # [  216.525931] mysqld[893]: 2020-04-04 22:47:10 1 [Note] WSREP: Synchronized with group, ready for connections
galera_03_rsync # [  216.527844] mysqld[893]: 2020-04-04 22:47:10 1 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
galera_04_rsync # [  143.125731] mysqld[903]: WSREP_SST: [INFO] Joiner cleanup. rsync PID: 961 (20200404 22:47:10.567)
galera_04_rsync # [  143.528222] rsyncd[961]: sent 0 bytes  received 0 bytes  total size 0
galera_04_rsync # [  143.670738] mysqld[903]: WSREP_SST: [INFO] Joiner cleanup done. (20200404 22:47:11.112)
galera_04_rsync # [  143.678394] mysqld[903]: 2020-04-04 22:47:11 0 [Note] WSREP: SST complete, seqno: 5
galera_04_rsync # [  143.703722] mysqld[903]: 2020-04-04 22:47:11 0 [Note] InnoDB: Using Linux native AIO
galera_04_rsync # [  143.705639] mysqld[903]: 2020-04-04 22:47:11 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
galera_04_rsync # [  143.707979] mysqld[903]: 2020-04-04 22:47:11 0 [Note] InnoDB: Uses event mutexes
galera_04_rsync # [  143.709711] mysqld[903]: 2020-04-04 22:47:11 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
galera_04_rsync # [  143.711215] mysqld[903]: 2020-04-04 22:47:11 0 [Note] InnoDB: Number of pools: 1
galera_04_rsync # [  143.712594] mysqld[903]: 2020-04-04 22:47:11 0 [Note] InnoDB: Using SSE2 crc32 instructions
galera_04_rsync # [  143.718026] mysqld[903]: 2020-04-04 22:47:11 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
galera_04_rsync # [  143.721941] mysqld[903]: 2020-04-04 22:47:11 0 [Note] InnoDB: Completed initialization of buffer pool
galera_04_rsync # [  143.724946] mysqld[903]: 2020-04-04 22:47:11 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
galera_04_rsync # [  143.730072] mysqld[903]: 2020-04-04 22:47:11 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=1635075
galera_04_rsync # [  143.838651] mysqld[903]: 2020-04-04 22:47:11 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
galera_04_rsync # [  143.841085] mysqld[903]: 2020-04-04 22:47:11 0 [Note] InnoDB: Creating shared tablespace for temporary tables
galera_04_rsync # [  143.844787] mysqld[903]: 2020-04-04 22:47:11 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
galera_04_rsync # [  143.849868] mysqld[903]: 2020-04-04 22:47:11 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
galera_04_rsync # [  143.852109] mysqld[903]: 2020-04-04 22:47:11 0 [Note] InnoDB: Waiting for purge to start
galera_04_rsync # [  143.904178] mysqld[903]: 2020-04-04 22:47:11 0 [Note] InnoDB: 10.3.22 started; log sequence number 1635084; transaction id 33
galera_04_rsync # [  143.907945] mysqld[903]: 2020-04-04 22:47:11 0 [Note] Plugin 'FEEDBACK' is disabled.
galera_03_rsync # [  217.454788] mysqld[893]: 2020-04-04 22:47:11 0 [Note] WSREP: 1.0 (galera_04): State transfer from 0.0 (galera_03) complete.
galera_04_rsync # [  143.914193] mysqld[903]: 2020-04-04 22:47:11 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
galera_03_rsync # [  217.456750] mysqld[893]: 2020-04-04 22:47:11 0 [Note] WSREP: Member 1.0 (galera_04) synced with group.
galera_04_rsync # [  143.919824] mysqld[903]: 2020-04-04 22:47:11 0 [Note] InnoDB: Buffer pool(s) load completed at 200404 22:47:11
galera_04_rsync # [  143.922910] mysqld[903]: 2020-04-04 22:47:11 0 [Note] Server socket created on IP: '<IP_ADDRESS>'.
galera_04_rsync # [  143.925181] mysqld[903]: 2020-04-04 22:47:11 0 [Note] WSREP: Signalling provider to continue.
galera_04_rsync # [  143.927311] mysqld[903]: 2020-04-04 22:47:11 0 [Note] WSREP: SST received: c2866648-76c5-11ea-af3c-e751edcdf91b:5
galera_04_rsync # [  143.929547] mysqld[903]: 2020-04-04 22:47:11 0 [Note] WSREP: 1.0 (galera_04): State transfer from 0.0 (galera_03) complete.
galera_04_rsync # [  143.931665] mysqld[903]: 2020-04-04 22:47:11 0 [Note] WSREP: Shifting JOINER -> JOINED (TO: 5)
galera_04_rsync # [  143.933587] mysqld[903]: 2020-04-04 22:47:11 0 [Note] WSREP: Member 1.0 (galera_04) synced with group.
galera_04_rsync # [  143.935348] mysqld[903]: 2020-04-04 22:47:11 0 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 5)
galera_04_rsync # [  143.937376] mysqld[903]: 2020-04-04 22:47:11 1 [Note] WSREP: Synchronized with group, ready for connections
galera_04_rsync # [  143.939631] mysqld[903]: 2020-04-04 22:47:11 1 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
galera_04_rsync # [  143.942303] mysqld[903]: 2020-04-04 22:47:11 0 [Note] Reading of all Master_info entries succeeded
galera_04_rsync # [  143.944319] mysqld[903]: 2020-04-04 22:47:11 0 [Note] Added new Master_info '' to hash table
galera_04_rsync # [  143.946434] mysqld[903]: 2020-04-04 22:47:11 0 [Note] /nix/store/kfiajz1r1cwxf5j9kmx2ma5cm42bf1l1-mariadb-server-10.3.22/bin/mysqld: ready for connections.
galera_04_rsync # [  143.949073] mysqld[903]: Version: '10.3.22-MariaDB'  socket: '/run/mysqld/mysqld.sock'  port: 3306  MariaDB Server
galera_04_rsync # [  143.988935] systemd[1]: Started MySQL Server.
galera_04_rsync # [  143.991279] systemd[1]: Reached target Multi-User System.
galera_04_rsync # [  143.992629] systemd[1]: Startup finished in 5.027s (kernel) + 2min 18.960s (userspace) = 2min 23.988s.

did you run it via nixosTests.mariadb-galera?
Also, I noticed a lot of spam about temporary failure in name resolution, while trying to do a reverse lookup of the internal IPs.
Could this be improved it we add the IPs and names to /etc/hosts, like it's done in  nixos/tests/nfs/kerberos.nix?
Also, doing that, we might be able to switch some of the IPs to hostnames.

did you run it via nixosTests.mariadb-galera?

Yes, it works for me
Added hostnames.

@GrahamcOfBorg test mariadb-galera

@GrahamcOfBorg test mariadb-galera-mariabackup
@GrahamcOfBorg test mariadb-galera-rsync

Pinging some more people controbuting to mysql for some more feedback.

@GrahamcOfBorg test mariadb-galera-mariabackup
@GrahamcOfBorg test mariadb-galera-rsync
@GrahamcOfBorg test mysql
@GrahamcOfBorg test mysql-autobackup
@GrahamcOfBorg test mysql-backup
@GrahamcOfBorg test mysql-replication


refactor setops

Pull Request Test Coverage Report for Build 456709002

65 of 115 (56.52%)  changed or added relevant lines in 2 files are covered.
8 unchanged lines in 2 files lost coverage.
Overall coverage increased (+45.3%) to 45.363%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

src/Flowpipes/setops.jl
4
5
80.0%

src/Flowpipes/reachsets.jl
61
110
55.45%

Files with Coverage Reduction
New Missed Lines
%

src/Flowpipes/reachsets.jl
2
46.93%

src/Flowpipes/setops.jl
6
52.05%

Totals

Change from base Build 455173930:
45.3%

Covered Lines:
1981

Relevant Lines:
4367

💛  - Coveralls

Pull Request Test Coverage Report for Build 456709002

65 of 115 (56.52%)  changed or added relevant lines in 2 files are covered.
8 unchanged lines in 2 files lost coverage.
Overall coverage increased (+45.3%) to 45.363%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

src/Flowpipes/setops.jl
4
5
80.0%

src/Flowpipes/reachsets.jl
61
110
55.45%

Files with Coverage Reduction
New Missed Lines
%

src/Flowpipes/reachsets.jl
2
46.93%

src/Flowpipes/setops.jl
6
52.05%

Totals

Change from base Build 455173930:
45.3%

Covered Lines:
1981

Relevant Lines:
4367

💛  - Coveralls


Trusty: Block on deprecated, expose provenance

Summary
This PR adds support for blocking pull requests introducing pull requests.
It also adds support to show provenance data (historical and sigstore) when trusty supplies it for the new dependencies.
Fixes #3379
Fixes #3382
Change Type
Mark the type of change your PR introduces:

[ ] Bug fix (resolves an issue without affecting existing features)
[x] Feature (adds new functionality without breaking changes)
[ ] Breaking change (may impact existing functionalities or require documentation updates)
[ ] Documentation (updates or additions to documentation)
[ ] Refactoring or test improvements (no bug fixes or new functionality)

Testing
Added a new test for the function that computes the provenance data
Heres the new improved section showing the deprecated flag and the provenance data:

Review Checklist:

[x] Reviewed my own code for quality and clarity.
[x] Added comments to complex or tricky code sections.
[ ] Updated any affected documentation.
[x] Included tests that validate the fix or feature.
[ ] Checked that related changes are merged.

coverage: 52.0% (+0.02%) from 51.98%
when pulling 896f131278e3c936605e76d28f1ed64c151e078a on puerco:deprecated
into 586144c00ddd9f86d829f52721cdee0540fca38b on stacklok:main.


Use type-safe sender-type casts

This PR takes the goal of adding type-safe sender type casts such that following code can be written:
var manager = new ExampleCommandManager<Supertype>();
var command = manager.commandBuilder("example")
    .senderType(Subtype.class)
    .handler(ctx -> {
        ctx.getSender(); // Subtype
    });
manager.command(command);

This is done through some generic changes and a couple casts that should be safe given the new generic type N holds the expectation N extends C, where C is the old type. This is safe, because by the time the command is called, the sender type should already be checked. N is therefore always going to be assignable to the type C (like how String is assignable to CharSequence, and (Class<CharSequence>) classInstance where Class<? extends CharSequence> classInstance = String.class is safe).

This should AFAIK be ABI-compatible with the previous code.

closing for #518


feat(output/tap): Output qualified resource name

This is an improvement suggestion for the TAP output. With only the resource kind in the output, it can be impossible to know which resource is being validated:
ok 48 - stdin (RoleBinding)
ok 49 - stdin (RoleBinding)
ok 50 - stdin (RoleBinding)
ok 51 - stdin (RoleBinding)

This PR replaces the resource kind by a "qualified" resource name formatted version/kind/namespace/name:
ok 48 - stdin (rbac.authorization.k8s.io/v1/RoleBinding/ns1/admin)
ok 49 - stdin (rbac.authorization.k8s.io/v1/RoleBinding/ns2/admin)
ok 50 - stdin (rbac.authorization.k8s.io/v1/RoleBinding/ns3/admin)
ok 51 - stdin (rbac.authorization.k8s.io/v1/RoleBinding/ns4/admin)

It also adds the signature to skipped resources:
ok 25 #skip - stdin

ok 24 - stdin (bitnami.com/v1alpha1/SealedSecret/some-ns/a-secret) # skip

Small comment on the format, otherwise nice improvement :100:


Add extendable test framework

The idea:

in tests/_commons.pynew test datasets can be added
in tests/test_cc6.pyone can specify if/which checks for this test dataset fail, and why
All test datasets referenced in tests/test_cc6.py:TestCC6Checks.test_all_cc6_checks will be checked by each individual cc6 check and unexpected check-failures reported

This should allow to quickly add new test datasets to the CI checks.
(A few problems I discovered in the check methods have been addressed as well.)

Thx, i'll have a look asap!


[Go Live][April 13]

Push to live

Docs Build status updates of commit fd81eb6:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

msteams-platform/apps-in-teams-meetings/enable-and-configure-your-app-for-teams-meetings.md
:white_check_mark:Succeeded
View

msteams-platform/apps-in-teams-meetings/meeting-app-extensibility.md
:white_check_mark:Succeeded
View

msteams-platform/bots/how-to/conversations/subscribe-to-conversation-events.md
:white_check_mark:Succeeded
View

msteams-platform/concepts/deploy-and-publish/appsource/prepare/teams-store-validation-guidelines.md
:white_check_mark:Succeeded
View

msteams-platform/tabs/how-to/authentication/auth-oauth-provider.md
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel


tbb update preparations

Changes in makefile.

/intelci: run

Are you sure that it is needed to keep compatibility with both tbbs .2 and .12? Will mkl_fpk will work with both tbbs?

Are you sure that it is needed to keep compatibility with both tbbs .2 and .12? Will mkl_fpk will work with both tbbs?
Why behavior made by this change isn`t compatible with MacOS?

Yeah, we need it to be able to build with old and new tbb packages.
mkl fpk won't work with different tbb versions but we are already solving this problem.
MacOS doesn't require any changes.
I'm glad you still with us! 👍

I'm glad you still with us! 👍
teabagk7 merged commit 10e7eda into oneapi-src:master 8 hours ago
as I see, you have no interest to discuss this question any more


[Bug]: jest overwrite global object and does not allow extension

Version
29.6.1
Steps to reproduce

clone the following most minimal repro: https://github.com/lemoustachiste/repro-jest-global-overwrite
npm install
npm test

Expected behavior
global.XmlHttpRequest is available as an instantiable class on the global object. It is also overwritable to a custom mock class
Actual behavior
The global object is not extendable to add custom, usually available constructors. Nor does it implement those regular properties/methods.
I have tried setting my mocked class with setupTest.js and in a beforeAll, to no avail.
Additional context
Overwriting the global class XmlHttpRequest with a custom class that allows avoiding internet round trips during tests and returns values for a specific URL as needed by the tested project
Note: this used to work in the previous installed version<EMAIL_ADDRESS>Issue arose as I was migrating to the latest version of jest.
I have little control on the actual source I am testing against as it is a rollup build.
Environment
System:
    OS: macOS 13.4.1
    CPU: (8) x64 Intel(R) Core(TM) i5-1038NG7 CPU @ 2.00GHz
  Binaries:
    Node: 16.20.0 - ~/.nvm/versions/node/v16.20.0/bin/node
    Yarn: 1.22.18 - /usr/local/bin/yarn
    npm: 9.8.0 - ~/.nvm/versions/node/v16.20.0/bin/npm
  npmPackages:
    jest: ^29.6.1 => 29.6.1

Can it be that you forgot to add testEnvironment: 'jsdom' to your Jest config? The default testEnvironment was changed in Jest 27. See the release blogpost.

@mrazauskas thanks for the quick support, that was it.
Sorry for missing out the simple config change...


[New Port Request] triton

Library name: triton
Library description: Triton is a Dynamic Binary Analysis (DBA) framework. It provides internal components like a Dynamic Symbolic Execution (DSE) engine, a dynamic taint engine, AST representations of the x86, x86-64, ARM32 and AArch64 Instructions Set Architecture (ISA), SMT simplification passes, an SMT solver interface and, the last but not least, Python bindings.
Source repository URL: https://github.com/JonathanSalwan/Triton
Project homepage (if different from the source repository): http://triton.quarkslab.com
Anything else that is useful to know when adding (such as optional features the library may have that should be included):
Triton dependencies are boost, capstone and z3 all of them already supported by vcpkg.
Triton has python bindings that should be optional when building.
Cheers

If you wish to add this port in the future, you can reopen this issue, but we're closing it for now.


[rv_dm,dv] Verify that writes to DMI registers are ignored while DM is not active (jtag_dmi_dm_inactive BLT)

This issue tracks the implementation of the jtag_dmi_dm_inactive block-level test, which is specified in the testplan for rv_dm.

estimate 8

Triaged for rv_dm. Assigning to M2.5 with https://github.com/lowRISC/opentitan/labels/Priority%3AP2 because this is an open V2 test.

@rswarbrick  @jdonjdon need to figure out with impare whether this is part of the RV_DM testplan already (and hence addressed).

Implemented in f2591927dd0e4c2a47edba40091b16c842f650aa (in PR #20016) last October.


Logout on get

https://github.com/Tivix/django-rest-auth/issues/279

Coverage decreased (-0.02%) to 97.046% when pulling 42ae22152adeb664c9df2d8b444454e486433bd3 on logout_on_get into 667e70c40f7c4ebccf3a63c12f0e10e5fd64983d on master.

Coverage decreased (-0.02%) to 97.046% when pulling cef8d679684a50690928ab8600bb5ca39a9ce5ee on logout_on_get into 667e70c40f7c4ebccf3a63c12f0e10e5fd64983d on master.


adding # to shebang in types-loops-tools/structure.py script

found missing # in shebang inside script types-loops-tools/structure.py

Thanks!


Panic on nvs_flash_deinit_partition when NvsCustom is dropped

The following code crashes with error code 4353 (ESP_ERR_NVS_NOT_INITIALIZED)
fn main() {
    esp_idf_sys::link_patches();

    match EspCustomNvsPartition::take("config") {
        Ok(ok) => {
            println!("got partition")
        }
        Err(e) => {
            println!("esp error {}", e.code())
        }
    }

    loop {
        thread::sleep(Duration::from_secs(30));
    }
}

got partition
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: EspError(4353)', /home/nick/.cargo/registry/src/github.com-1ecc6299db9ec823/esp-idf-svc-0.45.0/src/nvs.rs:119:74
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
abort() was called at PC 0x400ec3d6 on core 0
0x400ec3d6 - _ZN11panic_abort18__rust_start_panic5abort17hd168c25bed14c3b4E
    at ??:??

Backtrace: 0x40082bce:0x3ffb5b40 0x40085591:0x3ffb5b60 0x4008a30e:0x3ffb5b80 0x400ec3d6:0x3ffb5bf0 0x400ec3ca:0x3ffb5c10 0x400de515:0x3ffb5c30 0x400de4a4:0x3ffb5cb0 0x400e7707:0x3ffb5d20 0x400e75d9:0x3ffb5d50 0x400de237:0x3ffb5d80 0x400f4cf4:0x3ffb5db0 0x400f5467:0x3ffb5df0 0x400d3034:0x3ffb5e50 0x400d2a38:0x3ffb5e90 0x400d2b28:0x3ffb5eb0 0x4010ca43:0x3ffb5f00 0x400d2a14:0x3ffb5f20 0x400ebe31:0x3ffb5f40 0x400d2b57:0x3ffb5f60 0x400d40d3:0x3ffb5f90 0x4010dd68:0x3ffb5fb0
0x40082bce - panic_abort
    at /home/nick/IdeaProjects/esp32-tests/.embuild/espressif/esp-idf/release-v4.4/components/esp_system/panic.c:408
0x40085591 - esp_system_abort
    at /home/nick/IdeaProjects/esp32-tests/.embuild/espressif/esp-idf/release-v4.4/components/esp_system/esp_system.c:128
0x4008a30e - abort
    at /home/nick/IdeaProjects/esp32-tests/.embuild/espressif/esp-idf/release-v4.4/components/newlib/abort.c:46
0x400ec3d6 - _ZN11panic_abort18__rust_start_panic5abort17hd168c25bed14c3b4E
    at ??:??
0x400ec3ca - __rust_start_panic
    at ??:??
0x400de515 - rust_panic
    at ??:??
0x400de4a4 - _ZN3std9panicking20rust_panic_with_hook17he970e7307730fac7E
    at ??:??
0x400e7707 - _ZN3std9panicking19begin_panic_handler28_$u7b$$u7b$closure$u7d$$u7d$17h14ef4c42e31b3b43E
    at ??:??
0x400e75d9 - _ZN3std10sys_common9backtrace26__rust_end_short_backtrace17h06d82fdb5d0328ccE
    at ??:??
0x400de237 - rust_begin_unwind
    at ??:??
0x400f4cf4 - _ZN4core9panicking9panic_fmt17h3d71bf8d8318fafcE
    at ??:??
0x400f5467 - _ZN4core6result13unwrap_failed17hd42a26877a69ecb5E
    at ??:??
0x400d3034 - _ZN69_$LT$esp_idf_svc..nvs..NvsCustom$u20$as$u20$core..ops..drop..Drop$GT$4drop17h8f6a5910e1e0c86aE
    at ??:??
0x400d2a38 - _ZN5alloc4sync12Arc$LT$T$GT$9drop_slow17h593bebf4f09fa2edE
    at ??:??
0x400d2b28 - _ZN11esp32_tests4main17h15e60453b661ffe1E
    at ??:??
0x4010ca43 - _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17hc6be14b146155c8aE
    at ??:??
0x400d2a14 - _ZN3std2rt10lang_start28_$u7b$$u7b$closure$u7d$$u7d$17h7991eaaa6789e11bE.llvm.6876726762958448056
    at ??:??
0x400ebe31 - _ZN3std2rt19lang_start_internal17ha8c3fd326fb69b0fE
    at ??:??
0x400d2b57 - main
    at ??:??
0x400d40d3 - app_main
    at ??:??
0x4010dd68 - main_task
    at /home/nick/IdeaProjects/esp32-tests/.embuild/espressif/esp-idf/release-v4.4/components/freertos/port/port_common.c:141

I tried the same with nvs_flash_init() before the partition code, same problem
fn main() {
    esp_idf_sys::link_patches();

    let nvs = EspDefaultNvsPartition::take().unwrap();
    match EspCustomNvsPartition::take("config") {
        Ok(ok) => {
            println!("got partition")
        }
        Err(e) => {
            println!("esp error {}", e.code())
        }
    }

    let _ = nvs;

    loop {
        thread::sleep(Duration::from_secs(30));
    }
}

I saw the same problem seems to appear for nvs_flash_deinit()
https://github.com/esp-rs/esp-idf-svc/blob/ba1e19d59a0af0dd95ac6c67d0ed9d890059797d/src/nvs.rs#L63-L70

Have you created a custom partition labeled config before trying to use it?

I'm working successful with custom nvs partitions. You need to define the custom partition in the partitions.csv. In belows example it is the conf partition.
# ESP-IDF Partition Table
# Name,   Type, SubType, Offset,  Size, Flags
nvs,      data, nvs,     0x9000,  0x4000,
otadata,  data, ota,     0xd000,  0x2000,
phy_init, data, phy,     0xf000,  0x1000,
factory,  app,  factory, ,        0x14D000,
ota_0,    app,  ota_0,   ,        0x14D000,
ota_1,    app,  ota_1,   ,        0x14D000,
conf,     data, nvs,     ,        0x3000,

And as a second step you need to define the data storage layout and populate the partition with data. This can be done using the nvs_partition_gen.py tool.
~/esp/esp-idf/components/nvs_flash/nvs_partition_generator/nvs_partition_gen.py generate "../../weatherStationSecrets/anemometer/nvs.csv" conf.bin 0x3000 --outdir ./nvs

I would suggest to consult the ESP IDF C documentation for further details.

Thanks for the info that I have to create these tables before usage. I just started programming an ESP32 with this rust library. I didn't program anything in C before, but I'm currently trying with the bindings.
@taunusflieger I tried to flash your partition table with espflash and the option --partition-table, sadly with no luck.
Error: espflash::partition_table::invalid_partition_table

  × Failed to parse partition table
  ╰─▶ Invalid partition table

When I try to flash this simplified table, it works, but the SubType is not nvs.
# ESP-IDF Partition Table
# Name,   Type, SubType, Offset,  Size, Flags
nvs,      data, nvs,     0x9000,  0x6000,
phy_init, data, phy,     0xf000,  0x1000,
factory,  app,  factory,       ,  1M,
config,   data, fat,           ,  0x3000,

When I change the subType of the config partition to nvs, I get the same Invalid partition table error.

I got it fixed. I updated espflash from 1.7.1 to rc version 2.0.0-rc.3 and now this partition table is accepted and there is no panic anymore:
# ESP-IDF Partition Table
# Name,   Type, SubType, Offset,  Size, Flags
nvs,      data, nvs,     0x9000,  0x6000,
phy_init, data, phy,     0xf000,  0x1000,
factory,  app,  factory,       ,  1M,
config,   data, nvs,           ,  0x3000,

Thanks again for the help though 👍🏼


Separate Monitoring and health endpoints from function endpoint

FEATURE REQUEST:
Currently /healthz and /metrics are served on the same port as the function code. This is not optimal for different reasons.

This might create conflicts with function code, if the user doing some path based operations inside the function.
Istio with mutualTLS enabled will not work, as livenessprobe is failing and promethues can not get pod data.
Security People don't like it, as it is exposing data, which might provide insights

What you expected to happen:
Let's put /metrics and /healthz on a dedicated port. We can exclude this port from istio and do not have to expose.
Environment:

Kubernetes version (use kubectl version): all
Kubeless version (use kubeless version): all
Cloud provider or physical cluster: all

If this is fine for you, we can start working on it at the 28th. I am sure, we can make the change compatible to the current implementation, so it will not stop the 1.0 Release.

The liveness probe is done outside of the container. We could change the probe to curl and do it on localhost. This should work, but we still got the problem with potential collisions with function logic.
https://istio.io/help/faq/security.html#k8s-health-checks

what I am not sure I understand is why the health check would work if served in a different port. Does Istio mutual TLS only work in a specific port? Also it seems that they working on fixing the issue. My concern about serving the health check in a different port is that some runtimes may serve different ports (servers) in different threads. Reporting one of them as healthy might not mean that the  one serving the real requests is.
Also we cannot do the workaround of using curl since some of the runtimes doesn't have curl installed.
My suggestion is to move the /metrics to a different port like you suggest and make the liveness probe optional so you can disable it when using Istio. What do you think?
Also let's bring @murali-reddy to the discussion, I want to know his opinion as well.

https://github.com/istio/istio/issues/2628 has more details
We should stick to best practice or prescribed way of Kubernetes. I feel Istio folks should figure a way forward for interop with Kubernetes deployments. IMO we should wait and see the reolution on Istio for this issue before doing such a drastic change to Kubeless.

This might create conflicts with function code, if the user doing some path based operations inside the function.

This is relevent only in case of http triggers right? Should not user be supposed to do proper path to function at http triggers level.

From that issue:

It is likely to be ready in 0.8.1 release, which I expect to happen in a month.

So let's wait for them to show their solution.

I work on OpenFaaS and have an interest in this issue/topic too. What is the benefit in adding a second HTTP port for binding health/metrics endpoints?

This might create conflicts with function code, if the user doing some path based operations inside the function.

I was talking to Joe Beda about checkpoint/metric endpoints - in part of the project we've implemented a prefix for these endpoints i.e. /_/metrics and /_/health - have you considered that as a solution to resolve this point?

@murali-reddy Actually the problem occurs on all deployments, as mutualTLS is securing the service to service communication too (including communication between triggers and functions).
@andresmgot we can make the livenessprobe configurable via configmap to have a workaround in place as long as istio is not delivering the feature. (I don' trust them, istio 0.8 is still not in alpha). We should consider changing the path like @alexellis suggested.
For the metrics endpoint, we still have to have it on a dedicated port as we have to exclude this particular port to be intercepted by the istio sidecar.

I agree that the quickest solution that will unblock you @joek is having a field in the kubeless-config to disable healthchecks.
If I understood correctly, the /metrics endpoint is not blocking you since you can still do the request there using a proper certificate, isn't it?. Note that you should be able to also limit the access to that resource from outside the cluster with the Ingress controller. We can discuss a proper solution for that in a different issue but since it require changes in all the runtimes it will take longer.

I don't think we should switch off the healthchecks, we should have a option, to overwrite them in the deployment template. This will add more flexibility.
Regarding the metrics: I will check with the team one more option, which got into my mind. Fixing it in all runtimes should not be a problem, we can do it. I will open a ticket for the metrics, as soon as we checked out this option.


Update navbar.jade

This is how you need to label your menu. I have used the word "Challenges" when describing freecodecamp to people in actual speech. Also your own website says "Challenges required for certifications are marked with a *" on the list. You're using the word challenges as well. Please make challenges the word instead of map because your webpage will behave best when people know where to find their work. The words you use for labels are more important than almost any other decision you make regarding your design.

Please review our Guidelines for Contributing very carefully and then open a new PR, thank you!.


[Request] Add LibAdwaita/LibHandy bindings

It seems that every new GTK application uses LibAdwaita now, and the old ones are also ported to it.

Is this a request to add that bindings to the gintro package, or a statement that you are currently working on adding them?
For the later case: When you provide at least one working, high level example application then I think we can integrate it to gintro. High level means no use of cast and addr at least.
For the former case: I would have to ask Gnome/GTK core devs what LibAdwaita is and if it makes sense to add it. The reason is that I added a few libs by user request, including libhandy, libnice, webkitgtk and gstreamer. And it was not a really smart decision from my side: libnice is very low level and generated a lot of trouble, and for the other 3 libs the guys who requested it vanished very fast.

This is a request. I looked through other issues looking for an example of a request, but I didn't find them.

This is a request.

OK. When the GTK/Gnome devs think that adding makes sense then I will try to add it. With some luck it is only adding a line to gen.nim script. But of course I have to search for a C or Python example and convert that to Nim.
Will I be able to test an example on PC, or do I need a mobile device for testing? And does Adwaita lib work with GTK4 already?
I just remembered
https://github.com/StefanSalewski/gintro/issues/110
Never heard again from that guy also, and unfortunately that lib does not work on PC at all, so we have no example yet.

guys who requested it vanished

Yes, I found a request to add webkitgtk. I even contacted Adam Dingle in the mail to find out the details.
Can you explain why it was a bad decision? I did not understand the mechanism of gintro, but if it takes information from the GIR, do you need any additional manipulations? I mean, the whole point of GIR's existence is to be able to create automatic bindings without manual intervention.
LibAdwaita is not a low-level library. This is a library that extends the set of GTK widgets. It was originally created by purism to adapt GTK for their smartphone. But many widgets turned out to be extremely useful in desktop applications, after which LibHandy was renamed LibAdwaita(which shows that it is now a more general library, and not only for devices with a small screen)
As an experiment, I wanted to rewrite some simple GTK application to Nim(I'm trying to switch from Vala now). My choice fell on https://github.com/lainsce/khronos/ but it, like many, uses LibAdwaita.
Here are some examples that I found by going through the list of https://circle.gnome.org/
https://gitlab.gnome.org/World/Authenticator/-/blob/master/meson.build#L11
https://gitlab.gnome.org/somas/apostrophe/-/blob/master/meson.build#L48
https://gitlab.gnome.org/Cogitri/Health/-/blob/master/Cargo.toml#L16
https://github.com/GustavoPeredo/Font-Downloader/blob/master/meson.build#L13
https://gitlab.gnome.org/World/Shortwave/-/blob/master/meson.build#L16

why it was a bad decision?

Webkit was not that bad, was not much work to add it. But it is a very large lib, so it increases install time by a few seconds, and large libs have always some minimal risk that they will break the install process.
gstreamer and libnice causes much trouble, libnice maybe 40 hours of work for me, and gstreamer maybe 20 hours. Because both contained a lot low level stuff, and I do know nothing about that packages, so it was hard for me to port libnice example from C to Nim. But at least libnice is still used by someone from japan, and adding these packages improved the gen.nim bindings generator. But it is unclear for what someone will use Nim with gstreamer or webkitgtk.

LibHandy was renamed LibAdwaita

Interesting, I was not aware of it.

trying to switch from Vala now

OK, then you know GTK at least. The largest burden are request for packages from people with no GTK knowledge. Learning GTK is much work, and chances that newcomers will learn GTK and Nim are very tiny. I started the GTK4 book to make that possible, but the book is only 20% complete currently, and I am not sure if I will continue with it.

Do you have any chat in discord or tg about gintro?

Will I be able to test an example on PC, or do I need a mobile device for testing?

Ofc you dont, just the reason why handy was renamed to Adwaita is to show that it is no longer a library for devices with small screens, but just a GTK extension. (The standard theme is also called Adwaita)
And ye, ur gnome-settings using libhandy rigth now when you squeeze it
https://gitlab.gnome.org/GNOME/gnome-control-center/-/blob/master/meson.build#L102

Do you have any chat in discord or tg about gintro?

No, would not make too much sense for currently 1 to 2 active gintro users beside me, and I am not a guy who has the time to sit the whole day in front of a discord or IRC window. (Note, the other requested lib called GtkLayerShell was mobile only, so I was not able to test it.)
I will try to provide you with the LibAdwaita lib soon, will do first try tomorrow. I am a bit busy currently, as always...

First problem: Gentoo linux does not have LibAdwaita at all. So I have to install from sources, which sometimes works well, sometimes not. We will see tomorrow. LibAdwaita seems to be really a new development, see
https://aplazas.pages.gitlab.gnome.org/blog/blog/2021/03/31/introducing-libadwaita.html

OK, I was able to install latest libadwaita from git sources and I have added it to the gen.nim generator:
diff ~/gintrotest/tests/gen.nim gen.nim
1c1
< # High level gobject-introspection based GTK4/GTK3 bindings for the Nim programming language
---
> # High level gobject-introspection based GTK3/GTK4 bindings for the Nim programming language
3,4c3,4
< # v 0.8.9 2021-APR-11
< # (c) S. Salewski 2018, 2019, 2020, 2021
---
> # v 0.8.9 2021-MAR-16
> # (c) S. Salewski 2018
98,120d97
<   AdwInit = proc (argc: ptr cint; argv: ptr ptr cstring) {.cdecl.}
<
< proc tryInitAdw(): LibHandle =
<   # echo "Try to init gst"
<   when defined(windows):
<     const AdwLibName = "adwaita-1-0.dll"
<   elif defined(macosx):
<     const AdwLibName = "libadwaita-1.dylib"
<   else:
<     const AdwLibName = "libadwaita-1.so(|.0)"
<   let adwLib = loadLibPattern(AdwLibName) # dlopen
<   if adwLib != nil:
<     let adwInit = cast[AdwInit](adwLib.symAddr("adw_init"))
<     if adwInit != nil:
<       adwInit(nil, nil)
<     else:
<       echo "Can't init adw library!"
<   else:
<     echo "Can't load adw library!"
<   return adwLib
<   # unloadLib(adwLib)
<
< type
143,145d119
<
<
<
3317d3290
<   adwLib: LibHandle
3355,3357d3327
<   if namespace == "Adw" and adwLib == nil:
<     adwLib = tryInitAdw()
<
4063d4032
<     main("Adw") # replaces libhandy for GTK4
4107,4108d4075
<   if adwLib != nil:
<     unloadLib(adwLib)
4125c4092
< # 4125 lines
---
> # 4092 lines

Generally only adding the line "main("Adw") # replaces libhandy for GTK4" would have been necessary, but as libadwaita expects that init_adw() is called before that lib is used, the gtk core devs told me that in that case I should open the lib before using gobject-introspection on it. For some gst modules that is indeed necessary, for libadwaita it seems to work without too. The dopen() solution is not too nice, as we have to guess the actual lib names for linux, windows and mac.
The lib installs and compiles fine for me, but I was not yet able to provide an example. The old libhandy example does not work. The libadwaita provides an C example, but as that is not tiny I was not yet able to port it to Nim. I found no other examples, no Python one. The C example uses gtk builder. Unfortunately I do know nothing about libadwaita currently, so it may take me some time before I can provide a working Nim example.
You can install it with
nimble install gintro@#head
Maybe ask on Gnome forum or Nim forum for actual help with an example, I am really busy, I don't think I can do more today.

You may have to wait some time for a working example unfortunately, see
https://github.com/StefanSalewski/gintro/issues/123

All works. See #122 carousel example.


Extension displayed twice

After installation the extension is displayed twice like this :

Same thing in the integration list in admin.

May be fixed by commenting out 15th line in init.py - seems like Sentry registers modules itself just fine

I somehow missed this when it was opened, sorry...


Alpha of Circle

Hello! First of all: Great library - love it! Makes cross-platform development much easier.
Ran into just one issue: How can I set the alpha value of a circle so it becomes transparent ? So far the circles render solid for me.
Thanks in advance for your help!

Yes you can!
We support transparent color in circle and any shapes.
https://github.com/amay077/Xamarin.Forms.GoogleMaps/blob/5c7968cbeb89e58288e34e1163052ede19ee1906/XFGoogleMapSample/XFGoogleMapSample/Shapes2Page.xaml.cs#L43-L52

Thank you! Must've missed that part.


INSTALLATION FAILED: chart requires kubeVersion: >= 1.21.0-0 < 1.30.0-0 which is incompatible with Kubernetes ...

Trying to follow the instructions under: https://docs.reana.io/administration/deployment/deploying-locally/  (Deploying locally, For researchers).
During the installation step "helm install reana reanahub/reana --namespace reana --create-namespace --wait", I get
... Error: INSTALLATION FAILED: chart requires kubeVersion: >= 1.21.0-0 < 1.30.0-0 which is incompatible with Kubernetes v1.31.0

Hi @mcommer The support for Kubernetes 1.30 clusters will be declared in our Helm chart officially at the next REANA 0.9.4 release only.  (It works well locally, we just have not got to releasing the update yet.)
Until this is done, a quick workaround could be to deploy an older version of Kubernetes in your local Kind installation, for example:
$ kind delete cluster
$ kind create cluster --config kind-localhost-30443.yaml --image=kindest/node:v1.29.7

and follow the linked instructions.


Support _nodes/stats for elasticsearch 1.x series.

Follow pattern in nodes_info.go.

Pending acceptance I plan to issue a PR for v3 supporting elasticsearch 2.x.

Thank you. I'll hopefully find some time over the weekend for review.
You could double-check that the API contains all properties defined in the REST API spec for that endpoint.
Also notice that there is a Go generator that builds a skeleton for API's based on the REST API spec. You can find it in the generate-api branch. No need to implement everything manually. :-)

@olivere I've found a few discrepancies. I'll mention you again when all of the API spec has been met.

Take your time. I'll be out of office for a few days anyway.

@olivere Added unit test similar to that of what is already in the repo. Implemented the REST API and included it in the commit message based on https://github.com/elastic/elasticsearch/blob/1.7/rest-api-spec/api/nodes.stats.json
I did notice that the groups parameter is marked as a boolean, although it is a list as referenced in its documentation https://www.elastic.co/guide/en/elasticsearch/reference/1.7/cluster-nodes-stats.html

I'm sorry. I cannot merge this. There are a few problems.
First, please remove the iota values for Level. I tried that a few years ago, and it's just not worth it in the client. Simply let the caller pass a string.
Second, the Nodes Stats API has many endpoints. You need to use the one that matches the parameters given by the caller. You always assume /_nodes/{node_id}/stats/{metric}/{index_metric} is okay to use. But that doesn't work when {node_id} or {metric} isn't specified by the caller. We need to take this into consideration.
Third, although I didn't check in detail: Many of the structures returned by Nodes Stats API are already returned by Nodes Info API as well. I guess I have to figure out if duplicating the structs is really the better approach.
I'm thankful for your support. But it's hard to keep on top of all the endpoints ES provides and still keep everything under control. I think we'll have to have another round on Nodes Stats API.

Comment 1 I've addressed, but will wait to put it up for now.
For comment 2 I'm a bit confused. Looking at https://github.com/olivere/elastic/blob/release-branch.v2/nodes_info.go#L85 I'm doing the same here https://github.com/olivere/elastic/pull/314/files#diff-3910d97348111eb8bb2b1710f118a56dR156
When {node_id} or {metric} isn't specified, then _all is used in its place. Perhaps I'm not understanding. Can you give me a code example using my PR so I can understand what you are saying doesn't work?
Question 3, I have time to work on merging the two. Just let me know what you would like and I'll go forward. Once we have this solidified for ES 1.x then I'll continue working on the ES 2.x branch.

Sorry for the delay.
Regarding comment 2: I didn't see that you initialized metric and nodeId with "_all" in the initializer. Sorry for that.
Re question 3, it looks like both endpoints return the same structures. What needs to be done is first to check if that's a correct assumption by looking at the Java source code. It takes some experience to find the correct reference in the elasticsearch repository (use the current 2.x branch for elastic.v3). Then we'd need to check if the properties in our structs are in sync with those in the Java source.
I'll eventually do that, too. No worries if you don't feel like you can do that. Unfortunately, I'm rather busy at the moment, so it'll take some time.

Backport should be available in 2.0.54.


DROOLS-925 - fixed cannot deploy kie-server on wildfly 10.x.x

DROOLS-925 - fixed cannot deploy kie-server on wildfly 10.x.x due to HornetQ is to be deprecated.

@pokpitch Not sure if I missed your answer, but as Petr asks above, will this change work for Wildfly 8 and 9 as well?

Can one of the admins verify this PR? Comment with 'ok to test' to start the build.

-1 this does not work on Wildfly 8.2.0.Final and moreover it modifies file that affects both ee6 and ee7 (meaning both EAP and Wildfly) so will actually break all assemblies from JBoss
error that happened on wildfly 8.2.0.Final
Caused by: javax.xml.stream.XMLStreamException: ParseError at [row,col]:[2,1]
Message: Unexpected element '{urn:jboss:messaging-activemq-deployment:1.0}messaging-deployment'


Improve Home Manager support: profiles/suites, modules, extern, flake outputs

A really simple method of implementing #119.
This relies on a feature that I added in home-manager master for module options of extraSpecialArgs and sharedModules, nix-community/home-manager#1793. I could try and get that backported to 20.09. But I thought I'd get some feedback on these changes first.

bors try

Updated to match the lib changes.
Also I dropped hmActivationPackages and this time exported homeConfigurations with a hyphen seperating hosts and users. So all host/user pairs get exported as host-user.
I think this is significantly simpler. I only exported the activation packages in the first place due to nix's issues with nested outputs. So separating with a hyphen allows for easy debugging. And with the flk script, it really only affects someone who manually builds the activation package and even that isn't that bad.

This already is looking amazing, once you get it ready for review and it's approved, you can merge it yourself when your confident it's done. Good work.
bors delegate+

This already is looking amazing, once you get it ready for review and it's approved, you can merge it yourself when your confident it's done. Good work.

Wow thats cool! Thanks! I think I might try to get the sharedModules and extraSpecialArgs PR backported to 20.09 in home-manager so I can get this merged sooner.

Alternatively, I was thinking of upgrading the nixos flake to nixos-unstable for the time being, at least until the next stable release.

Ohh that makes things simpler. Can you do that as a separate commit? There might be other things to fix after upgrading, that doesn't make sense to include in this PR.

Yeah, I'll try to manage it soon. I'm working on cleaning up some other things as well. Hopefully by the end of the day.

I dropped the commit updating the flake, so this should theoretically merge cleanly once the flake is updated. But I can fix any other conflicts once you upgrade.

I added some more defaults to the hmConfig. Although, optimally, that would not be needed with #127.

I've update my local config to nixos-unstable and it seems to be working fine, after some fixes. However, I can't push it up yet, as it somehow broke the ci runner and all tests are failing with an exception. Once I figure it out, I'll push it.

I think the check for devshell on aarch64 will fail. Because nix flake check tries to evaluate it(not build) and nixUnstable is still broken for aarch64 on nixos-unstable - hydra hasn't updated the channel with the fix yet.

Yeah I saw that, but actually every single derivation is failing due to an exception in hercules-ci's cpp code. I am gonna try reverting the update to the hercules flake and see if that helps. If not, it may be related to the update to Nix. I believe I can work around it either way, hopefully won't take too long.

It seems fixing the ci-agent is going to be slightly more involved than I initially suspected. hercules-ci/hercules-ci-agent#291
We may have to rely on upstream to patch it before it works. I was thinking I could use a container as a workaround, but the containers seem to use the hosts nix-daemon, so it didn't pan out.
So the alternatives seem to be:

to simply update the revision and run an older version on the ci-runner for now until a fix is found (will probably have to go with this one)
let it stay broken and manually merge this and other PR's hoping a bunch of stuff doesn't break (probably not this one)
use a virtual machine (might work but performance may suffer, I'll experiment)

Is the ci runner on an old revision evaluate the checks ok? I think the first option is the best so at least we can keep changing things.

yes it seems to be working again after downgrading. Assuming all these checks pass, I'll push the changes with the nixos-unstable update.

Before we merge this, be sure to rebase after I push the update, because I'm also going to make a change to the bors.toml. After merging my last PR as a squash merge, I decided I don't like it, and am going to use a regular merge commit instead. BORS pulls the toml from the PR and not the main branch so it's important you have the new one.

Ok thats makes sense, will do. Yeah squash merges aren't great for larger PR's.

besides the merge conflicts, is this still blocked?

Yeah it requires features from home-manager master, so its blocked until devos updates. Or alternatively after 21.05, or a backport of the home-manager PR for shared modules and extraSpecialArgs

Okay, if really would like to see this merged and I'm getting pretty annoyed at how difficult it's been to update to unstable. Time allowing, I'm going to attempt to start porting CI to Hydra tomorrow so we can close #192 and get this moving.

I rebased and fixed conflicts. But I can't actually test this new version, so after the update I will have to rebase again and run through any errors that may have come up after the rebase.
Also I switched to use user@host pairs based on the pr in home-manager that adds --flake support to home manager: nix-community/home-manager#1856.

bors try

This is ready for final review. I updated the home-manager input in this PR(first commit), and rebased with core. Checks succeed for me. I added some docs to users/README.md, so I think thats the last thing to check - @blaggacao I would appreciate your input for that, you're usually good with docs..
bors try

I'm going to go ahead and merge, this should be complete.
bors r+

We are gonna need a follow up, cleanup/enhancement PR. I wouldn't mind doing it, if nobody else wishes to do so. I've mentions a copule places that need fixing.

I likely won't be able to test the fixes since I can't reproduce the issues.


Disable mask when certain characters entered

Hi,
I have such a case that I couldn't find an answer.
In our report parameter pages, we enable user to enter filter criteria in (masked) inputs like below;
*.2017 : Means all 2017 records
>1.000 : Means bigger than 1000
0506*60 : Means starts with 0506 and ends with 60
Users can also enter exact values without any filter characters (['*' , '!' , '>' , '<' , '=' , ';'])
What I want is to disable mask when these characters entered and re-enable it when these characters removed.
Thank you.

@newmangreen ,
Did you try with alternators?  Removing can also mean just using a mask which accepts everything
so something like (99-aaa-99)|({})  This case will show the first mask as long the input for it is valid, otherwise will revert to  the other.
Make a jsfiddle to make it more concrete.

https://jsfiddle.net/newmangreen/6unug1y4/
Alternators didn't help since;

I couldn't use custom alias within alternator definiton
If all characters entered complying with first mask, no key entry allowed to bypass first mask.

I developed a custom solution by listening keydown events and toggling mask.
One out-of-scope problem is that when removing inputmask, value is reverted to unmasked value although I used "autoUnmask:false" option.  See FIXME in fiddle.

@newmangreen ,
I will fix the unmasking issue when removing the mask for you

Thank you, I checked the fix, it is OK.


⚠ Jenkins Plugins Artifactory has degraded performance

In 2e9c5bf, Jenkins Plugins Artifactory (https://get.jenkins.io/plugins/artifactory/) experienced degraded performance:

HTTP code: 200
Response time: 2029 ms

Resolved: Jenkins Plugins Artifactory performance has improved in 4b59e81.


UserWarning: pyproject.toml is present in repo, but toml package is not installed. Cruft configuration may be ignored.

/Users/tekumara/.local/pipx/venvs/cruft/lib/python3.9/site-packages/cruft/_commands/utils/generate.py:104: UserWarning: pyproject.toml is present in repo, but `toml` package is not installed. Cruft configuration may be ignored.

occurs on a project with a pyproject.toml. We only use pyproject.toml for project settings, not for cruft.
To avoid this warning, would it be possible to bundle toml as a dependency of cruft?

It is bundled if you install cruft via cruft[pyproject]

Ah right thanks, I had no idea. Having it bundled as part of the main package would probably still be useful, I'm probably going to forget to include pyproject next time 😅


docker utilities

mostly useful for scripts, where you don't want to repeatedly run manual docker commands as you tweak things.
usage:
from agent0.utilities import check_docker
check_docker(restart=True)

I'm not sure this should live in elf-simulations. It is too specific to your setup (hard-coded paths) and to our general infra setup, which will almost certainly not be shared by the broader user-base. I think this might be better placed inside the infra repo. Open to discussion, though.

it seems this is mostly workarounds to workarounds of my local setup...
though if mine is a generic ubuntu setup...
do we want people to use the infra repo?

not wanted at the moment, since I'm using this in only a couple of scripts that aren't committed here.


Inner Outer example - MOD_HASH explanation unclear

https://chialisp.com/docs/common_functions#outer-and-inner-puzzles
keybase://chat/chia_network.public#chialisp/6660
As per Quexington in the above Keybase discussion, MOD_HASH is a tree hash of the example code with no curried arguments. This is relevant point to understanding the example, getting it to run and is unclear in the documentation due to omission.
Please consider changing  the following from:
"MOD_HASH which is the tree hash of this code"
To:
"MOD_HASH which is the tree hash of this code with no curried arguments"

This has been resolved in #88 thanks!


ping on windows workload does not work

What kind of request is this: bug
Steps to reproduce (least amount of steps as possible):

create a custom cluster with 1etcd & control linux node, 1 worker linux node and 1 windows linux node. (VXLan mode)
deploy 2 workloads- wk01 and wk02.
executed the below commands in exec shell of wk02.

ping google.com
ping <IP_ADDRESS>
ping <IP_ADDRESS>(ip address of the workload)
ping wk01

Actual Result:
The ping commands did not work.
Note: Host port and Node port also do not work.
Other details that may be helpful:
Environment information

Rancher version (rancher/rancher/rancher/server image tag or shown bottom left in the UI): rancher:master
Installation option (single install/HA): single

Cluster information

Cluster type (Hosted/Infrastructure Provider/Custom/Imported): custom
Kubernetes version (use kubectl version):

1.14.1

ping wk01 issue will be fixed in https://github.com/rancher/rancher/issues/19954
The others are security group issue. So please try again.

verified on rancher:master(05/07/2019)
the below command works.

curl www.google.com
curl dns entries also require FQDN.
curl on workloads- with FQDN works.
ping ip address of the workload from another workload pod.


UPDATE SMALLEST POSITIVE MISSING NUMBER

Have you read the Contributing Guidelines on Pull Requests?
yes
Description
UPDATE SMALLEST POSITIVE MISSING NUMBER
Checklist

[x] I've read the contribution guidelines.
[x] I've checked the issue list before deciding what to submit.
[x] I've edited the README.md and link to my code.

Related Issues or Pull Requests
#2995

C-Plus-Plus/README.md


Manage packages page minor UI fixes

package icon has massive width on edge until it loads
if a section of the page filters to empty, either hide the section or show a message inside it saying it has no packages
excess row of padding below the manage packages data

Account/Org filter should be wider (double the current)

Regarding the first bullet point (massive icon on Edge)--this appears to be some sort of bug with Edge. The max width is set and should work as expected.


Create Training Data

[x] Skip just create my own audio data

[x] Record voice

OR

[x] ~Figure out how Speaker Recognition works with Azure API, if not try alternatives~

[x] ~Look into pyAudioAnalysis~

[x] ~Test with own data~

[x] ~Join audio clips~

Micorsoft Azure Speech platform requires 8 hours for training data :( that's not happening

Looking into alternatives

Resemble.ai works perfectly. UI is a little clumsy and with a little bit of selenium automation we can get that to work


Extension telemetry log output channel

1.59.0-insider
955a1aca9a2e4473eec4f4bdc70506a3a7c43ca7
arm64
Out of the box I see two output channels named Log (Extension Telemetry):

There should be only one, and it should be below the separator as it is not specific to a specific extension. It's comparable to Log (Extension Host).
Also, I wonder whether it should be there all the time. We only show the core log (Log(Telemetry)) if you set the logging level to verbose. I think the extension telemetry log should behave the same.

The two logs appearing in separate places is definitely a bug but I cannot fix the log level nor the location of the log as this log is contributed by extensions and we have no APIs for placing an output channel in that location nor reacting to the log level.


Support efficient diffing for folks setting large immutable objects

I've started using Redux with Ractive lately in some specific cases where it makes sense, and so far it's been working well.  The interaction is pretty simple:
ractive.on('dispatch', actionType => store.dispatch({ type: actionType }))
ractive.on('dispatchInput', (actionType, node) => {
	store.dispatch({ type: actionType, payload: value(node) })
})
store.subscribe(() => ractive.set(store.getState()))

I have no complaints.  However, I know that this means that for this use case, Ractive could do much more efficient change detection, and it would be cool if that were implemented.
Assuming that the developer is using Redux correctly, objects and arrays in the model will be immutable.  This means that they can be checked for changes with === instead of comparing the values of all their properties.
Example:
{
	invoice: {
		lineitems: [{
			itemId: 1,
			price: 2.33
		}, {
			itemId: 2,
			price: 4.44
		}]
	},
	customerIsCreditRisk: false
}

When I fire some CUSTOMER_CHANGE action for which the reducer returns { ...oldState, customerIsCreditRisk: true }, the invoice object is exactly the same.  Because oldState.invoice === newState.invoice, there's no need to check all the lineitems.
But as you can see above, my Redux subscriber is setting the whole object every time.  I would love to be able to initialize the Ractive instance with pureModel: true (or some better name) to enable this more-efficient checking.

side discussion: obviously, using two-way binding breaks the idea of the pure reducer-driven state.  At the moment I'm just avoiding it by convention, and that's working fine.  It would be cool if there was a mode that would prevent the view from changing the model, or warn if you used two-way binding or something.  Not a big deal, though.  Instead of value="{{thingy}}" I'm using a decorator decorator="setValue:{{thingy}}" that updates the input when the model changes, and firing a dispatch event on-change="fire('dispatchInput', 'SET_NEW_TOPIC', event.node)" that my Redux/Ractive glue code catches and hands to the store with the value currently in the input.
I don't feel the need for first-class support of those ideas at the moment though, just tossing that context out.

How much you care about supporting this use case may be dependent on whether or not you consider Redux to be a fad or a useful way of describing state changes, and I think both viewpoints are fair.
I didn't see any discussion of the topic, and wanted to see what y'all thought!

Okay, let's start with how to disable twoway binding: you can simply pass twoway: false into your constructor options. See here for docs. To disable globally use Ractive.prototype.twoway = false; (or Ractive.defaults.twoway = false; if you prefer).

How much you care about supporting this use case may be dependent on whether or not you consider Redux to be a fad or a useful way of describing state changes, and I think both viewpoints are fair.

Ractive is a library for creating dynamic, interactive UI. We try not to be too opinionated and just provide great tools for building stuff how you want to build it. If Redux works for you, great. I would even support putting in the mode for strict equality check (more on this in a bit).
However, I do feel the need to make it clear that using immutable objects will not be faster or more efficient than simply responding to immediate changes as Ractive normally does. Perhaps imperceptibly so depending on your use case, but it seems to be a common misconception. To be blunt, React introduced immutability because dom-diffing across lists is a performance drag without it.
With immutability, you pay the cost of examining the equality of the entire affected branch when it creates a new object for you. So let's put in a flag on Ractive that tells it to use strict equality checking so you don't have to pay the cost a second time when you set high-level objects. :) 0.8 already introduced strict: true for observer options, so I think using that same flag as a Ractive option makes sense (I think implicitly then the observer would be strict true).
We'll need to look through the code, but I think the use is largely in Model and for Computations.

Ah, thanks for the link to twoway!  I obviously need to read down the initialization option list again.
I should clarify that I'm not using Redux because I'm dreaming of better performance, but because I have some views where the action/reducer model makes a lot of sense.  I recognize that in taking on this model, I am forcing inefficiency upon Ractive that would not occur during normal use - in normal use, I would call ractive.set('customerIsCreditRisk', true) and everything would be efficient.
So, having now taken on a state manipulation model that skips Ractive's normally very-efficient model, I'm asking for a workaround to make my use case as efficient as Ractive normally is.
Like I said, if you think the Redux reducer model doesn't have merit on its own, there is no compelling argument to support it in Ractive.

Redux is pretty cool. :sunglasses:
I think the changes would be minor to support improved integration, though that's just the theory at this point.

I wrote about using Ractive with Flux a while back. I used Reflux back then. I ended up doing two-way on the UI level, while the general data flow was one-way.
I once had twoway set to false to do one-way data flow but lost the perks that we normally get with Ractive (especially the "it just works" part). There's also the case of reusability of the components. My friends couldn't use my one-way components because they used two-way. One-way just broke their dataflow. And it's not easily noticeable that a component is one-way until someone uses it, something breaks, and they look at the source only to see twoway: false.

Ractive could do much more efficient change detection, and it would be cool if that were implemented

There's ractive.merge. Since your store will construct new data every time, built-in strict equality won't work. You need to have identifiable data, and provide a compare function to merge. Also, merge is really slower compared to set because it does comparisons.
As for quick and easy, there's ractive.set. However, set also has its trade-offs. You're setting totally different data. You end up throwing away a lot of things and re-rendering a lot of things like in https://github.com/ractivejs/ractive/issues/1905#issuecomment-104071325 . It might not be noticeable at first, but when the app grows huge, it happens.

I've taken a stab at making a diff that can be handed to ractive.set that would be more efficient than passing the entire new state object in: ractive-diff-generator.
I've started using this in my own view layer.
The worst case now is when items are removed from arrays.  Adding items is fine, if you do
ractive.set('ary', ['howdy'])
ractive.set('ary.1', 'whatever')

...then it just adds a new element to the array and ary => [ 'howdy', 'whatever' ].
But I don't see any way to remove elements from the array using set without passing in a whole new array.
For properties that used to be on the state object and were removed, I'm having to settle for setting them to null, since set ignores properties whose values are undefined.
Any thoughts?  Can this implementation be improved?  Would the maintainers consider bringing this behavior into the core module, or leave it as a third-party option?

For the removal case, have you looked at merge?

Hmmm... After looking at the project, I see that it's not handling the set, so merge is probably not useful. If you let the library handle application too, you might be able to squeeze out a little more performance with merges or splices.

I hadn't noticed that I could pass in a custom comparison function to merge.  That does make calling a combination of set plus a special-case calling merge for arrays a lot more attractive.

I've updated the library to take advantage of merge.  I didn't implement application in the library, but I did put this application code in the readme that should be copy/paste friendly - essentially this:
var someDiff = diff(oldState, newState)
ractive.set(someDiff.set)
someDiff.merge.forEach(function(arrayToMerge) {
    ractive.merge(arrayToMerge.keypath, arrayToMerge.array, {
        compare: function identity(o) {
            return o
        }
    })
})

Any thoughts on this implementation? https://github.com/TehShrike/ractive-diff-generator
Am I correct in thinking that this should be about as good as array performance will get (assuming immutable objects, and assuming that redux implies having to iterate over altered arrays at least once)?

The implementation looks reasonable to me. I think the only way you could get faster for a large tree that didn't change much would be to hook ractive's equality check to allow for ident equality.
It probably wouldn't take too much effort to allow a comparator to be passed to set. An init option would be even easier. Out of curiosity, have you done any benchmarking?

Passing a comparator to set or on init would be my ideal.
I have not done any benchmarking with Ractive yet.  Are there any well-established benchmarks I could fork or should I toss something together myself?  Are benchmarks run in node.js worth anything, or do they only make much sense when run in browsers?

Ractive has its own benchmark suite in perf, but I don't know that it would suite this case very well. It does a pretty decent job at allowing you to compare different versions.
I suspect that most of the time spent in a set would be syncing the DOM, so running in node would only show you the (probably much, much) smaller portion of time taken by Ractive's model change propagation.