Proposal: Building block API for cryptography

Note: Crypto as in cryptography, NOT cryptocurrency!!

Update: API Design proposal below: https://github.com/dapr/dapr/issues/4508#issuecomment-1260116173

In what area(s)?

/area runtime

Describe the proposal
This is a proposal for a new building block for Dapr to allow developers to leverage cryptography in a SAFE and consistent way. Goal is to expose an API that allows developers to ask Dapr to perform operations such as encrypting and decrypting messages, and calculating and verifying digital signatures.
Modern applications make extensive use of cryptography, which, when implemented correctly, can make solutions safer even in case data is compromised. Even more, in certain cases the use of crypto is required to comply with industry regulations (think banking) or even with legal requirements (GDPR). However, leveraging cryptography is hard: developers need to pick the right algorithms and options, and need to learn the proper way to manage and protect keys. Additionally, there are operational complexities when teams want to limit who has access to cryptographic key material.
Organizations have been increasingly started to leverage tools and services to perform crypto outside of applications. Examples include services such as Azure Key Vault, AWS KMS, Google Cloud KMS, etc. Customers may also use on-prem HSM products like Thales Luna. While those products/services perform the same or very similar operations, their APIs are very different.
This is an area where Dapr can help. Just like we're offering an abstraction on top of secret stores, we can offer an abstraction layer on top key vaults.
Using this building block, developers would be able to perform cryptographic operations without having to access raw key material. We would also offer a selection of algorithms that are configured correctly and forbid the usage of unsafe algorithms and operations. Algorithms available will depend on what the backend vaults support, but in general developers should always find AES (encrypt/decrypt only) and RSA; when supported, we can offer also ChaCha20-Poly1305 (encrypt/decrypt only) and ECC with ECDSA or EdDSA (sign/verify only).
Benefits would allow:

Making it easier for developers to perform cryptographic operations in a safe way. Dapr provides safeguards against using unsafe algorithms, or using algorithms with unsafe options.
Keeping keys outside of applications. Applications never see key material, but can request the vault to perform operations with the keys.
Allowing greater separation of concerns. By using external vaults, only authorized teams can access private/shared key materials.
Simplify key management and key rotation. Keys are managed in the vault and outside of the application, and they can be rotated without needing the developers to be involved (or even without restarting the apps).
Enabling better audit logging to monitor when operations are performed with keys in the vault.

The new building block would feature 7 APIs:

/encrypt: encrypts arbitrary data using a key stored in the vault. It supports symmetric and asymmetric ciphers, depending on the type of key in use (and the types of keys supported by the vault).
/decrypt: decrypts arbitrary data, performing the opposite of what /encrypt does.
/wrapkey: wraps keys using other keys stored in the vault. This is exactly like encrypting data, but it expects inputs to be formatted as keys (for example formatted as JSON Web Key) and it exposes additional algorithms not available when encrypting general data (like AES-KW)
/unwrapkey: un-wraps (decrypts) keys, performing the opposite of what /wrap does
/sign: signs an arbitrary message using an asymmetric key stored in the vault (we could also consider offering HMAC here, using symmetric keys, although not widely supported by the vault services)
/verify: verifies a digital signature over an arbitrary message, using an asymmetric key stored in the vault (same: we may be able to offer HMAC too)
/getkey: this can be used only with asymmetric keys stored in the vault, and returns the public part of the key

Different components would be developed to perform those operations on supported backends such as the products/services listed above. Dapr would "translate" these calls into whatever format the backends require. Dapr never sees the private/shared keys, which remain safely stored inside the vaults.
Additionally, we could offer a "local" crypto component where keys are stored as Kubernetes secrets and cryptographic operations are performed within the Dapr sidecar. Although this is not as secure as using an external key vault, it still offers some benefits such as using standardized APIs and separation of concerns/roles with regards to key management.

I support adding a cryptography API as it will hide a lot of complexity and provide best practices in an area where they are very needed.
I particularly find useful the "local" version of the component.

I support a "Building block for cryptography".   And I would like to add a feature to your list of features.  When using encrypted messages flowing between 2 services (that may be located who knows where, and not just locally, but it applies to a local cluster/host as well) end to end message encryption is highly desireable, especially for portions of the message that are seriously private.  In order for the sender and receiver to encrypt/decrypt the message (or portions of it (PII)) they need to share encryption keys -- this the the hard part.
Please implement a feature called Secure Conversation that allows the sender and receiver to securely communicate to exchange info containing the encryption key for each message.  There is a spec defined for this called WS-SecureConversation.  Described as "Following a pattern similar to TLS, WS-SecureConversation establishes a kind of session key."  See https://en.wikipedia.org/wiki/WS-SecureConversation and for a complete spec http://docs.oasis-open.org/ws-sx/ws-secureconversation/v1.4/ws-secureconversation.html.
Plus, please make it really easy to use through a simple API which I will leave to others to specify in detail.  It would be nice to have a really high level operation like EndToEndEncrypt(someMessage) that takes care of all the grubby secure conversation details/"session key" behind the scenes.   I believe that such a "session key" may be able to be obtained from Sentry or based on things that Sentry already does.  Then of course when an encrypted message is received by the receiver Dapr does the decryption behind the scenes as well.
Thanks for the highly useful idea!

Hey @georgestevens99  thanks for sharing your thoughts!
I would like to understand more about your requirements for E2E encryption in this case.
As you know, communication between Dapr-ized applications already uses TLS. Between Dapr sidecars, communications use gRPC with mTLS, which offers network-level encryption and mutual authentication. Communication between applications and the Dapr sidecar isn't encrypted, but that should happen locally.
Because of that, I don't know if the requirement for providing an API for E2EE communication would fit in the scope of this proposal, which is more focused on lower-level cryptographic primitives than higher-level, opinionated protocols.
If E2EE is necessary, however, I would consider implementing that by performing an ECDH key agreement between two applications and deriving that way a symmetric key that will be used with AES (or if hardware-accelerated AES isn't available, ChaCha20-Poly1305). I don't know if this necessarily needs to be implemented in Dapr itself, however: I feel that for true E2EE you would want to implement this in your applications: each service would expose an endpoint that is invoked to perform the ECDH agreement, and that allows obtaining the shared secret. This would happen with a regular service-to-service invocation between your applications, which would transparently pass through Dapr (ECDH is designed to be used over insecure channels so it's perfectly safe even if no encryption is present). If this were implemented within Dapr, additionally, then the sidecars would be able to sniff the shared encryption key, thus violating the E2EE requirement.
Perhaps we could provide an example of how to do this design (ECDH to derive a shared encryption key) even with current versions of Dapr :)

E2EE for a message (or parts of it) is suggested by the dapr documents for highly confidential info.  But dapr currently supplies no way to do such encryption.
While E2EE can certainly be done at a low level as is your idea, the fact of the matter is that there are vast numbers of developers that do not have the experience, knowledge, nor skills to use the "lower level crypto privatives" you are proposing.  Couple that with the fact that security is growing more and more challenging each year.   Thus in addition to having lower level primitives for those who have the skills to use them, we (our industry) also need a really, really simple to use and highly effective and highly secure E2EE for messages sent from a Service to a Dapr sidecar and beyond.  This will provide a solid, highly productive solution for the encryption for this network hop (and beyond) that is suggested in the Dapr docs.
In my experience, most developers haven't a clue as to what you said means, i.e. "If E2EE is necessary, however, I would consider implementing that by performing an ECDH key agreement between two applications and deriving that way a symmetric key that will be used with AES (or if hardware-accelerated AES isn't available, ChaCha20-Poly1305). ". And learning how to effectively use this low level stuff is non trivial and error prone and takes time.
So we need E2EE for developers that have little or no background in security.  So think of it as a higher level addition to your excellent proposal.

A fully-transparent E2EE would only be possible between Dapr sidecars, but Dapr sidecars already talk to each other using mTLS (which is enabled by default), which should provide equal levels of security and authentication.
Perhaps a middle-ground would be to implement an API that performs the ECDH between two services (technically, between two sidecars) and gives the application a shared secret they can then use as they see fit, including using the /encrypt endpoint of this proposal. However it should be clear that in this case the sidecars do have visibility over the encryption keys, even if for just an instant, so it's not true E2EE between the applications. I am not sure if that would void some of the security requirements you're talking about.
Or, we could just create some samples that demonstrate how to do what I described above: creating an endpoint in a service so that they can perform an ECDH agreement, then use that shared secret with the /encrypt endpoint.

Also worth pointing out that in all the cases above, both with things that are integrated with Dapr or implemented in the applications, there's always the problem of authenticating the other party. Dapr doesn't authenticate application IDs, so anyone could create an application with any ID: just because you're performing an ECDH with an application called foo, you don't have any guarantee that it's the foo you want to talk to.

@ItalyPaleAle -- Thanks so much for taking the time to write up the various alternatives that aim at fulfilling my requrements.  I really appreciate it.  It forced me to think through my requirements and also the limitations inherent in Dapr as it now stands.
I was hoping there was a way to use Dapr's future capabilities to implement industrial strength E2EE for messages.
However, given the caveats you pointed out in your text (many thanks for those!) I am abandoning my attempts to use Dapr for this.  Instead I am "biting the bullet" and planning on developing a small Security Utility service to at least deal with the encryption keys for the sender and receiver, and likely more.  That way I can use identity based security to authenticate the sender and receiver to the Security Utility to setup encryption keys and avoid the potential identity ambiguity you described in your last paragraph, plus also avoid a tiny "instant" you described where "the sidecars do have visibility over the encryption keys", thus resulting in better overall security.
Thus, I am withdrawing my request for extending your "Building block for cryptography" to support E2EE as well.
Based on our conversation thus far, it seems to me that Dapr could offer such a standalone, un-daprized (no sidecar) Security Utility for E2EE.  This would be highly useful to those of us who need to deal with very high levels of security.  Any thoughts on the possibility of a such a Dapr Security Utility?
Thanks again for your contribution to this conversation!

+1 for this proposal.
Particularly interested in this building block, as we want to offer a Bring Your Own Keys feature to our customers.
It would be super neat if a tenant in our multi-tenant system could share with us their key (one-time, super locked down SDLC for this part of the app which receives keys) and then we never get to see that key ever again because it is tucked away in a cloud vendor HSM/daprd, and our developers never get access to that key as they use an API something like this :
I would see a dapr API looking something like this
POST http://localhost:3500/v1.0/crypto/{tenant_id}/encrypt
REQ
{
  "unencrypted_payload" : "I am super secret tenant data. Its encrypted state will be used downstream in dapr State Management & dapr Pub Sub components"
}

RESPONSE
{
  "encrypted_payload" : "9yo&*Ti6TEo437togeg43qow347yga4pw3ogh7GOGo7gGWPW473y48p3T8Y"
}

downstream....
POST http://localhost:3500/v1.0/state/someStateComponent

[
  {
    "key": "weapon",
    "value": "9yo&*Ti6TEo437togeg43qow347yga4pw3ogh7GOGo7gGWPW473y48p3T8Y",
    "etag": "1234"
  }
]

👋🤖

@ItalyPaleAle Has there been any further discourse behind the scenes around moving this proposal forward? Cheers!

Not yet. We would welcome a POC + proposal from the community.

We have internal application encryption requirements for our project, and we would like to standardize the encryption operation across all the services. Since this new building block won't be available for a while, does it make sense for us to internally build the "local" version following the Output Binding Component spec?

@joelyen the biggest blocker to this is that we need to implement support for streaming first. These APIs need to be streaming-first so they can handle larger amounts of data without issue. You can try building this with the output bindings spec, but you may have issues if trying to process larger blobs of data.
Which operations are you specifically interested in?

We discussed this issue in the community call today. Here goes the general feedback on it (including some of my recommendations too):

Encrypt API over HTTP

POST http://localhost:3500/v1.0/crypto/{component}/encrypt
Payload goes directly into the body.

Response: RAW encrypted payload in the HTTP response body.

Decrypt API over HTTP

POST http://localhost:3500/v1.0/crypto/{component}/decrypt
Ciphertext goes direct into the HTTP body.

Response: RAW plaintext in the HTTP response body.

Built-in encryption component

Cipher information is part of the component
Fetches encryption key from secret store

apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: mycipher
spec:
  type: crypto.builtin
  version: v1
  metadata:
  - name: cipher
    value: AES256
  - name: secretStore
    value: mysecretstore
  - name: encryptionKeyName
    value: mysymmkey

Encryption components

apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: vault
spec:
  type: crypto.hashicorp.vault
  version: v1
  metadata:
  - name: vaultAddr
    value: "https://<IP_ADDRESS>:8200"
  - name: encryptionPath
    value: /transit/encrypt/mykey

Dapr encryption envelope
Dapr runtime will have an opinion on how to encrypt messages and minimize data sent/received from encryption services. It means that the Dapr runtime will generate on-time encryption keys and encrypt those with services instead of encrypting the whole payload. The payload is then encrypted with the on-time encryption key instead. The Dapr's encryption envelop will contain the ciphertext and the encrypted key (at least).

I think we need a way to also make envelope encryption exposed through Dapr. Some cloud providers (AWS, IBM) expose envelope encryption as a first class concept. Additionally, it is a common pattern for enabling secure high-throughput client side encryption. It might be a worth concept to have as a building block for distributed apps exposed through Dapr.
Context
So what is envelop encryption?[1], [2], [3]

Envelope encryption is the practice of encrypting plaintext data with a data key, and then encrypting the data key under another key.

Why even doing this? From the KMS Tutorial (GCP docs for envelope encryption is also pretty good):

AWS KMS can encrypt or decrypt data up to 4 KB in length. This can make it difficult to work with larger pieces of data. Additionally, when you encrypt data, it must be transferred over the network, processed by KMS, and returned; often, the associated network load and latency and complexity makes this undesirable.
A popular alternative to encrypting data directly with KMS is the use of envelope encryption . Rather than sending the data to KMS to be encrypted, we use KMS to generate a data key, which is then encrypted with a KMS-managed master key, or CMK. We then use this key - not the CMK itself - to encrypt and decrypt data, locally, where we are processing the data.

So the idea of envelope encryption is that

You generate a random key per encryption request to encrypt your data (your data encryption key or DEK). Data of any size can be encrypted locally, fast and unbounded by your cloud provided limits/quotas.
You encrypt your data with this DEK.
You encrypt this DEK with a key stored in your key management service, in a process called key wrapping. That will be your Key Encryption Key (or CMK, depending on the terminology 😉 ). Multiple DEK can be encrypted with the same KEK.
The DEK along with the data it wrapped (encrypted) are stored/sent together.

Why envelope encryption?

Reduced costs when working for cloud services - although some providers+SDKs will charge you for envelop encryption, the costs of using envelop encryption are lower than requesting keys and encryption of all data directly to the cloud provider.
Resiliency and increased TPS - cloud providers have limits on the rate of encryption operations one can requests and amount of data requests. Envelop encryption reduces those.
Decreased latency - encrypted data doesn't have to leave premises and are encrypted locally, saving transfer and latency fees
Consistency - cloud providers already offer this. By not offering an equivalent to this on Dapr we alienate those customers and decrease Dapr usefulness on this regard.

Support by cloud provider

AWS exposes envelope encryption as a first class concept in their SDK.

Azure Key Vault doesn't expose this directly in its APIs. It is handled transparently with its managed services and performed by its client libraries for storage

GCP doesn't explicitly offer envelop encryption as a first class concept in their SDK but its [Tink Cryptographic library][https://developers.google.com/tink/client-side-encryption] exposes it. That said, envelope encryption is the under-the-hood solution for encryption data with asymmetric cryptography (but that is expected).

IBM cloud exposes envelop encryption as an API

The proposal from @tmacam goes into details on how Dapr should have an opinion on the encryption envelope I mentioned earlier. Thanks, @tmacam!

After re-reading this proposal I noticed support for envelope encryption is present in the form of /wrapkey and /unwrapkey APIs. I do wonder if it makes sense to offer an abstraction of client-side encryption using envelope encryption under the hood.

Hey folks, I wanted to write down my thoughts next week once I’m back from vacation, but I had given this a bit more thought–and it also addresses @tmacam’s feedback.
I can flush this out in a full API design next week, but at a high level…
——
First, my original proposal was for exposing low-level primitives only. I think we still need to do that, and in addition to methods to encrypt/decrypt data with symmetric ciphers (what @artursouza already proposed), we also need:

Methods to encrypt and decrypt data with asymmetric ciphers
Methods to wrap/unwrap keys - this is essentially the “envelope encryption” @tmacam is talking about. These methods are encryption/decryption routines, but they allow for different encryption schemes that do not make sense when encrypting arbitrary messages.
Methods to sign data and verify signatures

I flushed these things out a bit more in the first post of this issue, with a bit more details.
(We could also consider offering methods to calculate hashes/digests, although those may be less important as they’re fairly common)
Second, I do think that we can provide additional value besides just offering these low-level primitives to implement on top of them higher-level APIs that implement common encryption schemes that make it easier for people who aren’t versed in the subject to encrypt and decrypt data safely.
This is aligned with what @tmacam is talking about too, as that’s getting into the realm of hybrid encryption, for example.
Third, I would recommend that the name of the key not be hardcoded in the component’s spec, but rather be passed as an argument to the encrypt/decrypt (and wrapkey/unwrapkey and sign/verify) methods. For example, /crypto/{component}/encrypt/{key name}). This is because people will want to use multiple keys, often for different purposes, and we should allow them to refer to any key that is present in the key vault. (Perhaps we could maintain an allowlist of keys if that’s needed as a metadata property?)
——
To sum things up, I think we should offer 2 sets of APIs:

/subtlecrypto offers the low-level primitives to encrypt/decrypt, sign/verify, wrap/unwrap keys. These are low-level functions that offer direct access to the ciphers and require a certain level of understanding of what one is doing. I like the “subtle” term, which I copied from the JS WebCrypto APIs, because it implies that it’s something to be treated with care. The MDN docs linked there have a nice warning along the lines of “this stuff is dangerous and you need to know what you’re doing”
/crypto would offer higher level, opinionated APIs to encrypt and decrypt data (no signatures or key wrapping) using a limited set of encryption schemes, including hybrid ones (e.g. RSA+AES).

From an implementation point of view, the higher-level APIs invoke the subtle APIs internally, and are only implemented in the runtime: no work needed on components-contrib, where only the low-level APIs are offered.
There are also a few more things that I would like to point out:

These APIs can deal with large amounts of data so support for streaming (which requires #4979) in the runtime will be essentially mandatory - lest users will be sending GBs of data to Dapr
While streaming is not a problem for most operations, we need to be careful in certain cases, such as when decrypting data that was encrypted with an authenticated cipher (such as AES-GCM, ChaCha20-Poly1305, or AES-CBC+SHA256), because we need to decrypt the entire payload before we can send it to the client, in case the authentication tag check fails (the authentication tag can only be computed only when the entire message has been processed, and we should not flush un-authenticated data to the client). For higher-level APIs we could get around this by chunking data and adding authentication tags periodically (see for example the DARE scheme)

We also need to agree on what algorithms we support and what modes of operation. I suggest (not all of them may be available if the key vaults don’t support them):

Symmetric ciphers: AES-GCM, AES-CBC, AES-CBC+SHA256, ChaCha20-Poly1305. For AES, we support keys 128, 192, and 256-bits long. For key wrapping, there’s also AES-KW
Asymmetric ciphers for data encipherment: RSA-OAEP (2048, 3072, 4096) - we should not support the old RSA defined in PKCS#1 v1.5
Signatures: RSA-PSS (PKCS#1 v2.1) and RSA-PKCS#1 v1.5, ECDSA (NIST curves), EdDSA (Curve25519 at least)

For the higher-level APIs, we should discuss what schemes we want to support, especially with regards to hybrid encryption.

Hey folks, I wanted to write down my thoughts next week once I’m back from vacation, but I had given this a bit more thought–and it also addresses @tmacam’s feedback.
I can flush this out in a full API design next week, but at a high level…
——
First, my original proposal was for exposing low-level primitives only. I think we still need to do that, and in addition to methods to encrypt/decrypt data with symmetric ciphers (what @artursouza already proposed), we also need:

Methods to encrypt and decrypt data with asymmetric ciphers
Methods to wrap/unwrap keys - this is essentially the “envelope encryption” @tmacam is talking about. These methods are encryption/decryption routines, but they allow for different encryption schemes that do not make sense when encrypting arbitrary messages.
Methods to sign data and verify signatures

I flushed these things out a bit more in the first post of this issue, with a bit more details.
(We could also consider offering methods to calculate hashes/digests, although those may be less important as they’re fairly common)
Second, I do think that we can provide additional value besides just offering these low-level primitives to implement on top of them higher-level APIs that implement common encryption schemes that make it easier for people who aren’t versed in the subject to encrypt and decrypt data safely.
This is aligned with what @tmacam is talking about too, as that’s getting into the realm of hybrid encryption, for example.
Third, I would recommend that the name of the key not be hardcoded in the component’s spec, but rather be passed as an argument to the encrypt/decrypt (and wrapkey/unwrapkey and sign/verify) methods. For example, /crypto/{component}/encrypt/{key name}). This is because people will want to use multiple keys, often for different purposes, and we should allow them to refer to any key that is present in the key vault. (Perhaps we could maintain an allowlist of keys if that’s needed as a metadata property?)
——
To sum things up, I think we should offer 2 sets of APIs:

/subtlecrypto offers the low-level primitives to encrypt/decrypt, sign/verify, wrap/unwrap keys. These are low-level functions that offer direct access to the ciphers and require a certain level of understanding of what one is doing. I like the “subtle” term, which I copied from the JS WebCrypto APIs, because it implies that it’s something to be treated with care. The MDN docs linked there have a nice warning along the lines of “this stuff is dangerous and you need to know what you’re doing”
/crypto would offer higher level, opinionated APIs to encrypt and decrypt data (no signatures or key wrapping) using a limited set of encryption schemes, including hybrid ones (e.g. RSA+AES).

From an implementation point of view, the higher-level APIs invoke the subtle APIs internally, and are only implemented in the runtime: no work needed on components-contrib, where only the low-level APIs are offered.
There are also a few more things that I would like to point out:

These APIs can deal with large amounts of data so support for streaming (which requires Migrate HTTP server from fasthttp to net/http #4979) in the runtime will be essentially mandatory - lest users will be sending GBs of data to Dapr
While streaming is not a problem for most operations, we need to be careful in certain cases, such as when decrypting data that was encrypted with an authenticated cipher (such as AES-GCM, ChaCha20-Poly1305, or AES-CBC+SHA256), because we need to decrypt the entire payload before we can send it to the client, in case the authentication tag check fails (the authentication tag can only be computed only when the entire message has been processed, and we should not flush un-authenticated data to the client). For higher-level APIs we could get around this by chunking data and adding authentication tags periodically (see for example the DARE scheme)

We also need to agree on what algorithms we support and what modes of operation. I suggest (not all of them may be available if the key vaults don’t support them):

Symmetric ciphers: AES-GCM, AES-CBC, AES-CBC+SHA256, ChaCha20-Poly1305. For AES, we support keys 128, 192, and 256-bits long. For key wrapping, there’s also AES-KW
Asymmetric ciphers for data encipherment: RSA-OAEP (2048, 3072, 4096) - we should not support the old RSA defined in PKCS#1 v1.5
Signatures: RSA-PSS (PKCS#1 v2.1) and RSA-PKCS#1 v1.5, ECDSA (NIST curves), EdDSA (Curve25519 at least)

For the higher-level APIs, we should discuss what schemes we want to support, especially with regards to hybrid encryption.

I am not sure about offering the low level crypto APIs because we will lose the portability cross components.  We should have a common denominator API that is compatible across cloud services first. The more we add to the request, the less portable it is. This was one of the outcomes of the discussion in the community call. /cc @yaron2
IMO, we should have /encrypt /decrypt /sign /verify but not /wrap /unwrap and not have any configuration via the request and move as much as possible to the component config instead.

I wasn't in the community call, but I am not sure I understand the concerns about portability.
All key vaults expose only crypto primitives as the lowest common denominator. Although some algorithms aren't universal, we can count on a core set to be present, including AES and RSA (AFAIK ChaCha20 isn't available anywhere, and support for ECDSA/EdDSA is limited).
When we implement this in Dapr, the most convenient thing (easier to maintain and less error-prone) would be to implement only the crypto primitives in the components. For example, implement methods to perform AES encryption/decryption with a given mode of operation and no further abstractions.
Additional abstractions should be implemented in the runtime instead to ensure that the behavior is consistent across all components (and portable, reliable, safe).
Higher level APIs are implemented in the runtime and interact with the components by leveraging the primitives.
For example, if we decide that we will offer a higher-level API to encrypt data that is opinionated, we can decide that it will work by encrypting data with an ephemeral key (data encryption key or DEK), which is then wrapped with the user key (key encryption key or KEK) stored in the vault. The flow would be:

runtime receives the request to encrypt data
runtime generates a random DEK
runtime invokes the component to wrap the DEK using the key stored in the vault - this is the only operation that needs to leverage the vault
runtime encrypts the data using the DEK and returns the wrapped DEK and the ciphertext to the client - it doesn't make sense to do this in the vault because of performance (need to send the entire data to the vault over the network) and cost

I don't think there's another way to implement such a routine in a portable and safe way.
Given the above, it probably makes sense to also expose the primitives directly, for example as "subtlecrypto", as they would already be implemented by the components. We can use the component metadata API to expose what algorithms are supported should people need "non-core" ones.
Access to the lower level APIs will also be a requirement for people who need compatibility with existing or external systems, as people may need to implement their own schemes. For example, imagine encrypting data in the server (with Dapr) and then having to decrypt it in a client (using JS).
This is for encryption/decryption. Computing and verifying signatures are by their own nature "subtle" because there are no commonly-used (or needed) higher level schemes.

I wasn't in the community call, but I am not sure I understand the concerns about portability.
All key vaults expose only crypto primitives as the lowest common denominator. Although some algorithms aren't universal, we can count on a core set to be present, including AES and RSA (AFAIK ChaCha20 isn't available anywhere, and support for ECDSA/EdDSA is limited).
When we implement this in Dapr, the most convenient thing (easier to maintain and less error-prone) would be to implement only the crypto primitives in the components. For example, implement methods to perform AES encryption/decryption with a given mode of operation and no further abstractions. Additional abstractions should be implemented in the runtime instead to ensure that the behavior is consistent across all components (and portable, reliable, safe).
Higher level APIs are implemented in the runtime and interact with the components by leveraging the primitives. For example, if we decide that we will offer a higher-level API to encrypt data that is opinionated, we can decide that it will work by encrypting data with an ephemeral key (data encryption key or DEK), which is then wrapped with the user key (key encryption key or KEK) stored in the vault. The flow would be:

runtime receives the request to encrypt data
runtime generates a random DEK
runtime invokes the component to wrap the DEK using the key stored in the vault - this is the only operation that needs to leverage the vault
runtime encrypts the data using the DEK and returns the wrapped DEK and the ciphertext to the client - it doesn't make sense to do this in the vault because of performance (need to send the entire data to the vault over the network) and cost

I don't think there's another way to implement such a routine in a portable and safe way.
Given the above, it probably makes sense to also expose the primitives directly, for example as "subtlecrypto", as they would already be implemented by the components. We can use the component metadata API to expose what algorithms are supported should people need "non-core" ones.
Access to the lower level APIs will also be a requirement for people who need compatibility with existing or external systems, as people may need to implement their own schemes. For example, imagine encrypting data in the server (with Dapr) and then having to decrypt it in a client (using JS).
This is for encryption/decryption. Computing and verifying signatures are by their own nature "subtle" because there are no commonly-used (or needed) higher level schemes.

I have used a "crypto API" before and the algorithm used was outside the code. It was defined when creating the encryption key in the encryption service. This way, the app just needs to encrypt passing the payload and keyname. I don't think Dapr would add value by providing low level primitives - there are plenty of libraries doing that today.

I don't think Dapr would add value by providing low level primitives - there are plenty of libraries doing that today.

The value Dapr provides is that it abstracts the various cloud key vaults. All libraries that offer the primitives expect the key material to be passed directly.

I don't think Dapr would add value by providing low level primitives - there are plenty of libraries doing that today.

The value Dapr provides is that it abstracts the various cloud key vaults. All libraries that offer the primitives expect the key material to be passed directly.

I support this...we are also using dapr in all our applications and this is one of our requirements...if dapr could orchestrate that that would be really helpful.
We have one service for all the data security-related requirement ( like encryption/decryption/signing XML) so it would be really helpful if we could orchestrate the backend kms system using dapr.

API design proposal
High-level vs subtle
The building block features 2 kinds of operations:

Low-level or "subtle" (term frequently used to indicate low-level crypto operations; one example is how in browsers, low-level operations are in the crypto.subtle object). These offer developers full control over the schemes that are used, allowing them to specify parameters, keys, modes of operations, etc.
As the "subtle" name implies, using these operations requires a certain level of understanding about what they do and how to use them safely; they are not meant to be consumed by the "general public".
Dapr offers these low-level operations for two reasons:

Just like it is Dapr's value proposition to, for example, offer a consistent API abstracting various state stores, we will offer a consistent API to interface with key vaults. It allows using key vaults from multiple cloud providers, as well as using keys stored as Kubernetes secrets, all while interfacing with the same API.
Developers working on applications that require interacting with existing and/or external solutions need these lower-level APIs to be able to maintain compatibility with them.

High-level operations. At launch, these will cover data encryption and decryption only.
When using these higher-leel methods, Dapr offers a great level of abstraction. Developers just need to provide a key (symmetric or asymmetric) and can then use Dapr to encrypt/decrypt data without having to worry about anything else. Dapr will choose the best ciphers and modes of operations, offering access to an encryption scheme that is secure and flexible.

Dapr encryption scheme: dapr.io/enc/v1
In the first version of the building block, we define 2 higher level operations to encrypt and decrypt data, in addition to low-level operations.

Sources: The encryption scheme that Dapr uses is heavily inspired by the Tink wire format (from the Tink library maintained by Google), as well as by Filippo Valsorda's age, and Minio's DARE.

The Dapr encryption scheme is optimized for processing data as a stream. Data is chunked into multiple parts which are encrypted independently. This allows us to return data to callers as a stream, even when decrypting messages, being confident that we are not flushing unverified data to the client.
Key
Each message is encrypted with a 256-bit symmetric File Key (FK) that is randomly generated by Dapr for each new message. The key must be generated as 32 byte of output from a CSPRNG (such as Go's crypto/rand.Reader) and must not be reused for other files.
The FK is wrapped using a key stored in a key vault (Key Encryption Key (KEK)) by Dapr. The result of the wrapping operation is the Wrapped File Key (WFK).  The algorithm used depends on the type of the KEK as well as the algorithms supported by the component: in order of preference:

Symmetric keys:

AES-KW (RFC 3394): AES-KW
AES-GCM: AES-GCM
The result of using AES-GCM is in the format iv || wfk || tag

RSA keys:

RSA OAEP with SHA-256: RSA-OAEP-256

In the future, we should explore how to add support for elliptic curve cryptography, for example Curve25519, which requires performing an ECDH key agreement.

Ciphertext format
The ciphertext is formatted as:
header || binary payload

Header
The header is human-readable and contains 4 items, each terminated by a line feed (0x0A) character:

Name and version of the encryption scheme used. Currently, this is always dapr.io/enc/v1
The algorithm used to wrap the file key, as well as any other parameters as needed (currently unused, but will be needed for ECDH)
The wrapped file key, base64-encoded
The MAC for the header, base64-encoded

Base64 encoding follows RFC 4648 §4 ("standard" format, with padding included but optional when decoding)

dapr.io/enc/v1
AES-KW
hGYjwDpWEXEymSTFZ95zgX8krElb3Gqyls67R8zJA3k=
pBDKLrhAWL7IAvDKBV/v7lmbTG6AEZbf3srUN0Pnn30=

The final line is the MAC for the header, which is computed with HMAC-SHA-256 over the previous 3 lines (including the final newline) with a key that is derived from the (plain-text) File Key with HKDF-SHA-256:
mac-key = HKDF-SHA-256(ikm = file key, salt = empty, info = "header")
MAC = HMAC-SHA-256(key = mac-key, message = first 3 lines of the header)

Note that there's one newline character (0x0A) at the end of the MAC, which concludes the header
Binary payload
The binary payload begins immediately after the header (after the 4th newline character) and it includes the binary header and each segment of data encrypted:
binary header || segment_0 || segment_1 || ... || segment_k

Binary header
The binary header is 8-bytes long:

Cipher suite
Nonce prefix

1 byte
7 bytes

0x00: AES-GCM 0x01: ChaCha20-Poly1305 (RFC 8439)
Random sequence of 7 bytes generated by a CSPRNG

Dapr will choose AES-GCM if running on a system that supports AES-NI CPU instructions, or ChaCha20-Poly1305 otherwise. We may change that behavior in the future.

Segments
The plaintext is chunked into segments 64KB (65,536 bytes) each; the last segment may be shorter. Segments must never be empty, unless the entire file is empty.
Each segment of plaintext is encrypted independently and stored together with its tag:
encrypted chunk || tag

Segments are encrypted with a Payload Key (PK) that is derived from the (plain-text) File Key and the nonce prefix:
payload-key = HKDF-SHA-256(ikm = file key, salt = nonce prefix, info = "payload")

Each segment is encrypted using a different 12-byte nonce / initialization vector:
nonce_prefix || i || last_segment

Where:

nonce_prefix (7 bytes) is the nonce prefix from the binary header
i (4 bytes) is the sequence number, as a 32-bit unsigned integer counter, encoded as big-endian. The first segment has sequence number 0, and it increases.
last_segment (1 byte) is 0x01 if this is the last segment, or 0x00 otherwise

Components
Components in dapr/components-contrib implement low-level primitives only, while all higher-level operations are performed by the runtime, so they are executed in a consistent way across all backends/services. This is because the job of the components is limited to actually interacting with the key vaults, and everything else is best handled by the runtime.
Components are to be placed in the crypto folder and must implement the SubtleCrypto interface:
// SubtleCrypto offers an interface to perform low-level ("subtle") cryptographic operations with keys stored in the vault
type SubtleCrypto interface {
    // GetKey returns the public part of a key stored in the vault
    // This method returns an error if the key is symmetric
    GetKey(
        // Context that can be used to cancel the running operation
        ctx context.Context,
        // Name (or name/version) of the key to use in the key vault
        key string,
    ) (
        // Object containing the public key
        // The type of the key can be inferred from the type of the object
        pubKey crypto.PublicKey,
        // Error
        err error,
    )

    // Encrypt a small message and returns the ciphertext
    Encrypt(
        // Context that can be used to cancel the running operation
        ctx context.Context,
        // Input plaintext
        plaintext []byte,
        // Encryption algorithm to use
        algorithm string,
        // Name (or name/version) of the key to use in the key vault
        key string,
        // Nonce / initialization vector
        // Ignored with asymmetric ciphers
        nonce []byte,
        // Associated Data when using AEAD ciphers
        // Optional, can be nil
        associatedData []byte,
    ) (
        // Encrypted ciphertext
        ciphertext []byte,
        // Authentication tag
        // This is nil when not using an authenticated cipher
        tag []byte,
        // Error
        err error,
    )

    // Decrypt a small message and returns the plaintext
    Decrypt(
        // Context that can be used to cancel the running operation
        ctx context.Context,
        // Input ciphertext
        ciphertext []byte,
        // Encryption algorithm to use
        algorithm string,
        // Name (or name/version) of the key to use in the key vault
        key string,
        // Nonce / initialization vector
        // Ignored with asymmetric ciphers
        nonce []byte,
        // Authentication tag
        // Ignored when not using an authenticated cipher
        tag []byte,
        // Associated Data when using AEAD ciphers
        // Optional, can be nil
        associatedData []byte,
    ) (
        // Decrypted plaintext
        plaintext []byte,
        // Error
        err error,
    )

    // WrapKey wraps a key
    // The key must have been already serialized in the appropriate format
    WrapKey(
        // Context that can be used to cancel the running operation
        ctx context.Context,
        // Key to wrap, already serialized in the right format if needed
        plaintextKey []byte,
        // Encryption algorithm to use
        algorithm string,
        // Name (or name/version) of the key to use in the key vault
        key string,
        // Nonce / initialization vector
        // Ignored with asymmetric ciphers
        nonce []byte,
        // Associated Data when using AEAD ciphers
        // Optional, can be nil
        associatedData []byte,
    ) (
        // Wrapped key
        wrappedKey []byte,
        // Authentication tag
        // This is nil when not using an authenticated cipher
        tag []byte,
        // Error
        err error,
    )

    // UnwrapKey unwraps a key
    // The consumer needs to unserialize the key in the correct format
    UnwrapKey(
        // Context that can be used to cancel the running operation
        ctx context.Context,
        // Wrapped key
        wrappedKey []byte,
        // Encryption algorithm to use
        algorithm string,
        // Name (or name/version) of the key to use in the key vault
        key string,
        // Nonce / initialization vector
        // Ignored with asymmetric ciphers
        nonce []byte,
        // Authentication tag
        // Ignored when not using an authenticated cipher
        tag []byte,
        // Associated Data when using AEAD ciphers
        // Optional, can be nil
        associatedData []byte,
    ) (
        // Key in plaintext (can be serialized)
        plaintextKey []byte,
        // Error
        err error,
    )

    // Sign a digest
    Sign(
        // Context that can be used to cancel the running operation
        ctx context.Context,
        // Digest to sign
        digest []byte,
        // Signing algorithm to use
        algorithm string,
        // Name (or name/version) of the key to use in the key vault
        // The key must be asymmetric
        key string,
    ) (
        // Signature that was computed
        signature []byte,
        // Error
        err error,
    )

    // Verify a signature
    Verify(
        // Context that can be used to cancel the running operation
        ctx context.Context,
        // Digest of the message
        digest []byte,
        // Signature to verify
        signature []byte,
        // Signing algorithm to use
        algorithm string,
        // Name (or name/version) of the key to use in the key vault
        // The key must be asymmetric
        key string,
    ) (
        // True if the signature is valid
        valid bool,
        // Error
        err error,
    )
}

A few notes about all methods above:

The algorithm should be represented as constant as defined by RFC 7518 ("JSON Web Algorithms (JWA)"). For the most part, Dapr components should not try to parse the value submitted by the user (unless the component is the "local" one that performs crypto operations directly), and pass whatever value directly to the key vault.
The key parameter can contain a version if keys can be versioned in the vault. The format should be name/version. If no version is specified, it's assumed to be the latest.
For operations that don't need to be performed on the vault,

Notes on WrapKey and UnwrapKey:

If the key need to be encoded (common with asymmetric keys), it needs to be encoded before being passed to the component. For example, in the runtime, RSA keys may be represented in a rsa.PrivateKey object, and need to be encoded in PKCS#1 format. Symmetric keys can be passed as-is, as they are normally stored in a byte slice already.
WrapKey and UnwrapKey can be implemented on top of Encrypt and Decrypt if the underlying key vault does not have a special operation for key wrapping/unwrapping.

Notes on Encrypt and Verify:

When using an asymmetric key, these operations can be performed using the public key without hitting the key vault. However, for consistency and to ensure that we always use the last version of the key, components should always perform them in the vault. Exception could be if the key has a specific version, in which case components may opt to download the public key, cache it, and perform the operation locally.

gRPC APIs
In the Dapr gRPC APIs, we are extending the runtime.v1.Dapr service to add new methods:
// (Existing Dapr service)
service Dapr {
  // SubtleGetKey returns the public part of an asymmetric key stored in the vault.
  rpc SubtleGetKey(SubtleGetKeyRequest) returns (SubtleGetKeyResponse);

  // SubtleEncrypt encrypts a small message using a key stored in the vault.
  rpc SubtleEncrypt(SubtleEncryptRequest) returns (SubtleEncryptResponse);

  // SubtleDecrypt decrypts a small message using a key stored in the vault.
  rpc SubtleDecrypt(SubtleDecryptRequest) returns (SubtleDecryptResponse);

  // SubtleWrapKey wraps a key using a key stored in the vault.
  rpc SubtleWrapKey(SubtleWrapKeyRequest) returns (SubtleWrapKeyResponse);

  // SubtleUnwrapKey unwraps a key using a key stored in the vault.
  rpc SubtleUnwrapKey(SubtleUnwrapKeyRequest) returns (SubtleUnwrapKeyResponse);

  // SubtleSign signs a message using a key stored in the vault.
  rpc SubtleSign(SubtleSignRequest) returns (SubtleSignResponse);

  // SubtleVerify verifies the signature of a message using a key stored in the vault.
  rpc SubtleVerify(SubtleVerifyRequest) returns (SubtleVerifyResponse);

  // Encrypt encrypts a message using the Dapr encryption scheme and a key stored in the vault.
  rpc Encrypt(stream EncryptRequest) returns (stream EncryptResponse);

  // Decrypt decrypts a message using the Dapr encryption scheme and a key stored in the vault.
  rpc Decrypt(stream DecryptRequest) returns (stream DecryptResponse);
}

// rpc SubtleGetKey(SubtleGetKeyRequest) returns (SubtleGetKeyResponse);

message SubtleGetKeyRequest {
  // Name of the component
  string component_name = 1 [json_name="component"];
  // Name (or name/version) of the key to use in the key vault
  string name = 2;
}

message SubtleGetKeyResponse {
  // Name of the component
  string component_name = 1 [json_name="component"];
  // Name (or name/version) of the key.
  // This is returned as response too in case there is a version.
  string name = 2;
  // Public key, encoded as SPKI (PEM)
  string public_key = 3 [json_name="publicKey"];
}

// rpc SubtleEncrypt(SubtleEncryptRequest) returns (SubtleEncryptResponse);

message SubtleEncryptRequest {
  // Name of the component
  string component_name = 1 [json_name="component"];
  // Message to encrypt.
  bytes plaintext = 2;
  // Algorithm to use, as in the JWA standard.
  string algorithm = 3;
  // Name (or name/version) of the key.
  string key = 4;
  // Nonce / initialization vector.
  // Ignored with asymmetric ciphers.
  bytes nonce = 5;
  // Associated Data when using AEAD ciphers (optional).
  bytes associated_data = 6 [json_name="associatedData"];
}

message SubtleEncryptResponse {
  // Name of the component
  string component_name = 1 [json_name="component"];
  // Encrypted ciphertext.
  bytes ciphertext = 2;
  // Authentication tag.
  // This is nil when not using an authenticated cipher.
  bytes tag = 3;
}

// rpc SubtleDecrypt(SubtleDecryptRequest) returns (SubtleDecryptResponse);

message SubtleDecryptRequest {
  // Name of the component
  string component_name = 1 [json_name="component"];
  // Message to decrypt.
  bytes ciphertext = 2;
  // Algorithm to use, as in the JWA standard.
  string algorithm = 3;
  // Name (or name/version) of the key.
  string key = 4;
  // Nonce / initialization vector.
  // Ignored with asymmetric ciphers.
  bytes nonce = 5;
  // Authentication tag.
  // This is nil when not using an authenticated cipher.
  bytes tag = 6;
  // Associated Data when using AEAD ciphers (optional).
  bytes associated_data = 7 [json_name="associatedData"];
}

message SubtleDecryptResponse {
  // Name of the component
  string component_name = 1 [json_name="component"];
  // Decrypted plaintext.
  bytes plaintext = 2;
}

// rpc SubtleWrapKey(SubtleWrapKeyRequest) returns (SubtleWrapKeyResponse);

message SubtleWrapKeyRequest {
  // Name of the component
  string component_name = 1 [json_name="component"];
  // Key to wrap, already serialized in the right format if needed.
  bytes plaintext_key = 2 [json_name="plaintextKey"];
  // Algorithm to use, as in the JWA standard.
  string algorithm = 3;
  // Name (or name/version) of the key.
  string key = 4;
  // Nonce / initialization vector.
  // Ignored with asymmetric ciphers.
  bytes nonce = 5;
  // Associated Data when using AEAD ciphers (optional).
  bytes associated_data = 6 [json_name="associatedData"];
}

message SubtleWrapKeyResponse {
  // Name of the component
  string component_name = 1 [json_name="component"];
  // Wrapped key.
  bytes wrapped_key = 2 [json_name="wrappedKey"];
  // Authentication tag.
  // This is nil when not using an authenticated cipher.
  bytes tag = 3;
}

// rpc SubtleUnwrapKey(SubtleUnwrapKeyRequest) returns (SubtleUnwrapKeyResponse);

message SubtleUnwrapKeyRequest {
  // Name of the component
  string component_name = 1 [json_name="component"];
  // Wrapped key.
  bytes wrapped_key = 2 [json_name="wrappedKey"];
  // Algorithm to use, as in the JWA standard.
  string algorithm = 3;
  // Name (or name/version) of the key.
  string key = 4;
  // Nonce / initialization vector.
  // Ignored with asymmetric ciphers.
  bytes nonce = 5;
  // Authentication tag.
  // This is nil when not using an authenticated cipher.
  bytes tag = 6;
  // Associated Data when using AEAD ciphers (optional).
  bytes associated_data = 7 [json_name="associatedData"];
}

message SubtleUnwrapKeyResponse {
  // Name of the component
  string component_name = 1 [json_name="component"];
  // Key in plaintext (may be serialized).
  bytes plaintext_key = 2 [json_name="plaintextKey"];
}

// rpc SubtleSign(SubtleSignRequest) returns (SubtleSignResponse);

message SubtleSignRequest {
  // Name of the component
  string component_name = 1 [json_name="component"];
  // Digest to sign.
  bytes digest = 2;
  // Algorithm to use, as in the JWA standard.
  string algorithm = 3;
  // Name (or name/version) of the key.
  string key = 4;
}

message SubtleSignResponse {
  // Name of the component
  string component_name = 1 [json_name="component"];
  // The signature that was computed
  bytes signature = 2;
}

// rpc SubtleVerify(SubtleVerifyRequest) returns (SubtleVerifyResponse);

message SubtleVerifyRequest {
  // Name of the component
  string component_name = 1 [json_name="component"];
  // Digest of the message.
  bytes digest = 2;
  // Signature to verify.
  bytes signature = 3;
  // Algorithm to use, as in the JWA standard.
  string algorithm = 4;
  // Name (or name/version) of the key.
  string key = 5;
}

message SubtleVerifyResponse {
  // Name of the component
  string component_name = 1 [json_name="component"];
  // True if the signature is valid.
  bool valid = 2;
}

// rpc Encrypt(stream EncryptRequest) returns (stream EncryptResponse);

message EncryptRequest {
  // Request details. Must be present in the first message only.
  EncryptRequestOptions options = 1;
  // Chunk of data of arbitrary size.
  common.v1.StreamPayload payload = 2;
}

message EncryptRequestOptions {
  // Name of the component
  string component_name = 1 [json_name="component"];
  // Name (or name/version) of the key.
  string key = 2;
  // Force algorithm to use: "aes-gcm" or "chacha20-poly1305" (optional)
  string algorithm = 3;
}

message EncryptResponse {
  // Chunk of data.
  common.v1.StreamPayload payload = 1;
}

// rpc Decrypt(stream DecryptRequest) returns (stream DecryptResponse);

message DecryptRequest {
  // Request details. Must be present in the first message only.
  DecryptRequestOptions options = 1;
  // Chunk of data of arbitrary size.
  common.v1.StreamPayload payload = 2;
}

message DecryptRequestOptions {
  // Name of the component
  string component_name = 1 [json_name="component"];
  // Name (or name/version) of the key.
  string key = 2;
}

message DecryptResponse {
  // Chunk of data.
  common.v1.StreamPayload payload = 1;
}

For the common.v1.StreamPayload message, see https://github.com/dapr/dapr/pull/5170

The Encrypt and Decrypt methods are stream-based. Dapr will read from the client until it has sufficient data, and will then send back the encrypted/decrypted data to the client. Clients must thus both send data to the RPC and listen for incoming messages. SDKs can offer to consumer methods to read the data as a stream (e.g. in Go, they accept an io.Reader and return an io.Reader)
HTTP APIs
The HTTP APIs are developed in a way that is the exact "port" of the gRPC "subtle" APIs, and the contents of the request and response bodies match exactly the fields in the gRPC APIs.
Methods are:

/v1.0/subtlecrypto/getkey ->  SubtleGetKey
/v1.0/subtlecrypto/encrypt ->  SubtleEncrypt
/v1.0/subtlecrypto/decrypt ->  SubtleDecrypt
/v1.0/subtlecrypto/wrapkey ->  SubtleWrapKey
/v1.0/subtlecrypto/unwrapkey ->  SubtleUnwrapKey
/v1.0/subtlecrypto/sign ->  SubtleSign
/v1.0/subtlecrypto/verify ->  SubtleVerify

Currently, higher-level APIs are not offered via HTTP. This is because it's not possible, using HTTP, to have a stream where Dapr sends back data to the client before the client's request is complete. Without that, we'd have to buffer the entire data in-memory in the runtime, which would be not practical when encrypting very large files.

Great proposal!
I'm struggling to get my head round the interoperation between the sidecar and the KeyVault / HSM, so maybe a sequence diagram might help aid my understanding of how it all hangs together :)

@olitomlinson does this help?

@ItalyPaleAle
It really does help, I'm a visual learner, so many thanks for taking the time to provide the diagram.

My understanding is that the high-level encrypt/decrypt is implementing client-side/envelope encryption, where the encryption of payloads happens in the dapr runtime, and NOT in the KeyVault / KSM? Have I got that right? (I'm only asking because there was talk previously in this proposal that mentioned client-side/envelope encryption explicitly, but it is not mentioned in the latest iteration of the proposal)

There was previous discussion around allowing key names to be dynamic :

Third, I would recommend that the name of the key not be hardcoded in the component’s spec, but rather be passed as an argument to the encrypt/decrypt (and wrapkey/unwrapkey and sign/verify) methods. For example, /crypto/{component}/encrypt/{key name}). This is because people will want to use multiple keys, often for different purposes, and we should allow them to refer to any key that is present in the key vault. (Perhaps we could maintain an allowlist of keys if that’s needed as a metadata property?)

Will this make it into the proposal? I can definitely see a use-case where our KMS stores a key-per-tenant, and being able to provide this programmatically at runtime when invoking the high-level APIs would be fantastic.
There is a similar thread here where we talk of introducing the notion of a metadata.tenant argument passed onto the State Management building block API surface to introduce a variable on each invoke which is propagated to the component configuration

Currently, higher-level APIs are not offered via HTTP. This is because it's not possible, using HTTP, to have a stream where Dapr sends back data to the client before the client's request is complete. Without that, we'd have to buffer the entire data in-memory in the runtime, which would be not practical when encrypting very large files.

So I'm reading this as you're trying to protect developers from themselves, and channeling them to use gRPC just incase they ever need to encrypt large files? Alternatively, could you allow HTTP but set a max request size soft-limit to some upper bound that you feel comfortable allowing? Rationale : HTTP is incredibly accessible for developers, and many do not use gRPC, do we want to alienate those developers who are not comfortable with gRPC?

My understanding is that the high-level encrypt/decrypt is implementing client-side/envelope encryption, where the encryption of payloads happens in the dapr runtime, and NOT in the KeyVault / KSM? Have I got that right?

Correct.

There was previous discussion around allowing key names to be dynamic

In the proposal I wrote above, the key name is always dynamic, meaning that components have access to all keys they are authorized to, and apps need to specify what key they want to use.
I think doing the opposite (hardcoding a key in the component's YAML) would also encourage bad practices, such as using the same key for data encipherment and signing.

So I'm reading this as you're trying to protect developers from themselves, and channeling them to use gRPC just incase they ever need to encrypt large files? Alternatively, could you allow HTTP but set a max request size soft-limit to some upper bound that you feel comfortable allowing? Rationale : HTTP is incredibly accessible for developers, and many do not use gRPC, do we want to alienate those developers who are not comfortable with gRPC?

Yes, this is a situation where we're essentially trying to protect the developers from themselves. The idea is that access to the high level APIs should happen using the Dapr SDKs (also because the code to perform the "back and forth" over the gRPC streams is not going to be very simple).
If there's strong demand for the API to be exposed over HTTP, I think it can be considered. But people should be aware that it does require all data to be loaded into memory first, ballooning the memory requirements of the Dapr sidecar.

We should close this issue here at this time and direct everyone's attention to the proposal at https://github.com/dapr/proposals/pull/3.
@yaron2 @mukundansundar

And everyone should express their support again on the official proposal over at https://github.com/dapr/proposals/pull/3!


Texture Malfunction (and JourneyMap)

I'm using the Iron Backpack mod. I set a slot on the menu to make a keybind link for opening a backpack and I use one of the mod's backpacks for the icon. When I restarted the client, though, it showed the "texturemissing" icon in its place. It happened several times in a row. At first I thought it was because I kept adding new mods between each time I restarted the client, but I closed it out then launched it again immediately and had the same problem. I also did a few other tests, and came to the conclusion that it is, indeed, the Iron Backpack mod. Bear in mind I'm using the latest 1.12.2 version of each of these. ...And as a side bug, when I set JourneyMap to open fullscreen with the keybind in MineMenu, nothing happens. I have to set a separate key.

I can't really do anything about Iron Backpack that's on their end.
As for JourneyMap, it should work. What Mine Menu version are you using?

I thought I was using the latest, but as it turns out I mistakenly downloaded 1.6.9 rather than 1.6.10. I'll download that and see if it gets fixed, then get back to you with the results.

As a side note, I can place waypoints - at least on single player, since that's the only time I'm able to get waypoints working. I just can't open the fullscreen map. Again, I'll let you know if updating fixes things.

I just downloaded the 1.6.10 and tried the JourneyMap fullscreen thing again, but still no luck. And the texture for the backpack still doesn't appear, but I'll take your advice and let Iron Backpacks know about it.


Filters

Where would you recommend adding a common filter? Thanks for releasing this project it provides a great start.

Hey, glad you like the project!
I would simply create a "filters" folder in the "common" directory and follow the same format as the rest of the common directories. Cheers!


Injecting middle-ware into the webpack-dev-server?

In my normal server using grunt-connect, I can do this fairly easily doing something like:
    connect: {
      options: {
        port: 8000
      },

      dist: {
        options: {
          keepalive: false,
          middleware: function (connect, options) {
            var middlewares = [];

            middlewares.push(modRewrite(modRewriteRulesConfig));
             ... other stuff ....

            return middlewares;
          }
        }
      }
    },

I want to add the same set of middleware to the dev server, which I currently have defined as:
    'webpack-dev-server': {
      options: {
        port: 8000,
        webpack: webpackDevConfig,
        publicPath: '/scripts/',
        contentBase: './<%= pkg.src %>/',
      },

      start: {
        keepAlive: true,
      }
    }

Is this possible, if so, how can I go about doing it? Thanks!

The above solution is really gross. Use https://github.com/webpack/webpack-dev-middleware, then (this is using grunt):
connect: {
  options: {
    hostname: '*'
    port: 80,

    middleware: function(connect, options) {
      return [
        // handles existing files within src directory
        connect.static(contentBase),

        // connect.directory(contentBase),
        require('serve-index')(contentBase),

        // route other files to webpack's in-memory dev server
        webpackMiddleware(webpack(webpackConfig))
      ];
    }
  }
}

@grrowl I'm not quite following your suggestion above. If webpack-dev-middleware is used does that remove the need to have the webpack-dev-server grunt task that @sokra lists above? If so, how are you getting livereload to understand that JS files have changed? Is a separate grunt-watch task needed for JS files to initiate the grunt-contrib-connect livereload server?
Right now my webpack.config.js output object is as follows:
output: {
    path: '/.tmp/assets',
    publicPath: 'http://localhost:8080/assets/scripts/',
    filename: '[name].js',
    contentBase: 'http://localhost:9000/',
  }

And my grunt-contrib-connect file:
  var webpack = require('webpack');
  var webpackConfig = require('../../webpack-config.js');
  var webpackMiddleware = require('webpack-dev-middleware');

...

options: {
      port: 9000,
      open: true,
      // keepalive: true,
      livereload: 35729,
      // Change this to '<IP_ADDRESS>' to access the server from outside
      hostname: '<IP_ADDRESS>'
    },
    livereload: {
      options: {
        middleware: function(connect, options) {
          return [
            connect.static('.tmp'),

            // route other files to webpack's in-memory dev server
            webpackMiddleware(webpack(webpackConfig), {
              publicPath: webpackConfig.output.publicPath,
              hot: true,
              stats: {
                colors: true
              },
            }),
          ];
        }
      }
    }

This setup allows me to run the connect server at http://<IP_ADDRESS>:9000 and successfully access webpack's in-memory compiled JS assets (without the need for different asset paths between dev/production) but whenever I change a JS file I have to manually reload the page.

Closing as there are solutions in the thread. If still relevant, please re-open at Stack Overflow.
@hershmire It might be preferable to use webpack's HMR or browsersync plugin over running livereload outside of webpack. You should get more or less the same effect this way while it should pick up JS changes (hard refresh at least).


The description of "[converted]" should probably be changed

As the indication "[converted]" is displayed in status line not only when the file is read (like in "cp1251.txt" [converted] 1L, 27C), when the conversion is made from 'fileencoding' to 'encoding', but also when the file is written (like in "cp1251.txt" [converted] 1L, 32C written), when the reverse conversion is made, from 'encoding' to 'fileencoding', the description of "[converted]" in doc/insert.txt, lines 2055-2056 should probably be changed to reflect that.
Vim: https://github.com/vim/vim-win32-installer/releases/download/v8.2.0735/gvim_8.2.0735_x86_signed.zip

On the other hand, since the description of "[converted]" is given in the "read-messages" (doc/insert.txt, line  2036) section, perhaps no connection of "[converted]" to writing files belongs in there.

There is no list to explain the message for writing.  I can add something at the help for ++enc


Adapt overlooked instances of zero-based column numbers

Is this a follow up from #6775?

Yes, I missed that after a newline the column count was being reset to 0.


Popup render issue in Chrome

If the popup receives an update than changes it's height whilst it is animating in then it goes bad:

Hm was not able to replicate so far, seems to be specific to Chrome.
Is this the same issue as https://bugs.chromium.org/p/chromium/issues/detail?id=457887 ?
Its over 1 year old, and they prioritize things oddly (eg. it took 4 years to fix background-size), so we may want to address this on our side.
Perhaps we could detect running in Chrome and producing some kind of  additional, delayed update after initial load?

@alanshaw mind checking if changes merged with #336 helped by any chance?

Seems to be fixed in gurrent master – https://github.com/ipfs-shipyard/ipfs-companion/pull/355#issuecomment-368478592
Feel free to reopen if happens again.


Windows 10 update not available after new release

For the past few weeks I have been getting prompts to update the Windows 10 client. When I try to apply the update, the installer does not have an install/upgrade option. It only has a repair option.
So far I've uninstalled and reinstalled from a fresh download. After the first update, the issue started appearing again.
I am on Windows 10, build 17083.
Keybase version is <IP_ADDRESS>

cc @zanderz

Can you do a keybase log send and post the ID? Thanks.

Report ID: acb1b8c8282f4ee62e97dd1c

Sure enough, those updates are failing, so the updater keeps pestering you. Since this happened over more than one version iteration, I wonder if you have some extra security settings/policies enabled? The installer seems to get stuck thinking files are in use. You should be able to get around this by uninstalling and then downloading again from keybase.io - your settings and device provisioning won't be affected. Upcoming versions may solve this, we're working on it. Thanks!

I don't intentionally have any security settings/policies in place. It is just normal windows settings. I'll continue applying the versions manually.

@matthewpucc One reason the upgrade failed is because files were still in use, caused by a bug we have since fixed. Are you able to upgrade yet? You may need to uninstall + reinstall. Sorry for the hassle.

@zanderz I was successfully able to update to <IP_ADDRESS> yesterday actually. Good job on the patch! I'll let you know if I see any failures in the future.


feat(caching): add rate triggered MyBatis database caching.

This PR introduces some application side in-mem database caching via a Caffeine cache
Originally we were going to cache all reads with a small TTL, but this was going to lead to a bad user experience (all writes would be eventually consistent) when people use the dashboard or CLI to make writes and expect ACID like behavior.
We have tweaked this logic so that Caching only occurs when values are being read repeatably.
We hypothesis that this will give users an ACID ux via the dashboard and CLI but also allow us to cache bursty repeat reads (Lots of servers or Lambdas doing the same thing over and over).

Coverage increased (+0.2%) to 54.424% when pulling 96b6ab36b0b434e0430765682de4c3fe0347d84e on feature/secure_data_caching into 318c1c4fc2a9d6efaafbbdd948226c7d1fa15214 on master.

Testing methodology
This branch was tested with the StsAuthAndReadSimulation.
Each iteration was tested with 25 distributed nodes and the following settings
NUMBER_OF_SERVICES_FOR_SIMULATION: 25
CREATE_IAM_ROLES: false
PEAK_USERS: 1360
RAMP_UP_TIME_IN_MINUTES:  30
HOLD_TIME_AFTER_PEAK_IN_MINUTES: 30

CMS was deployed with 50 r4.2xlarge instances
Iteration 1
CMS Settings
cms.mybatis.cache.enabled=false

Server Metrics

Results

Notes
In the Splunk logs lots of DB connection timeout errors and Riposte request timeout errors
Iteration 2
CMS Settings
cms.mybatis.cache.enabled=true
cms.mybatis.cache.global.repeatReadCounterResetInSeconds=1
cms.mybatis.cache.global.repeatReadThreshold=1

Server Metrics

Results

Notes
The cache behavior is not getting triggered enough, will increase the counter interval and try again.
In the Splunk logs lots of DB connection timeout errors and Riposte request timeout errors
Iteration 3
CMS Settings
cms.mybatis.cache.enabled=true
cms.mybatis.cache.global.repeatReadCounterResetInSeconds=10
cms.mybatis.cache.global.repeatReadThreshold=10

Server Metrics

Results

Notes
Errors gone and good results, but the caching behavior might be too loose and cause eventual consistency (BASE) issues in the dashboard where we want it to appear consistent (ACID).
Will do another iteration with tighter thresholds (2 and 2). If that iteration has the same results as this I will update the PR so that those are the default values.

Iteration 4
CMS Settings
cms.mybatis.cache.enabled=true
cms.mybatis.cache.global.repeatReadCounterResetInSeconds=2
cms.mybatis.cache.global.repeatReadThreshold=2

Server Metrics

Results

Notes
Everything looks good with the new settings, I optimistically updated the default settings already.

As a final test with the new default values I added the following line the the get method of the cache and clicked around the dashboard.
log.info("getObject - Mapper: {}, hit: {}, count: {}, threshold: {}", id, value != null, counter.getCount(), repeatReadThreshold);

With the following results.
[pool-5-thread-2] INFO  c.n.c.c.DatabaseCache - getObject - Mapper: com.nike.cerberus.mapper.CategoryMapper, hit: false, count: 1, threshold: 2
[pool-5-thread-3] INFO  c.n.c.c.DatabaseCache - getObject - Mapper: com.nike.cerberus.mapper.RoleMapper, hit: false, count: 1, threshold: 2
[ForkJoinPool-1-worker-4] INFO  c.n.c.c.DatabaseCache - getObject - Mapper: com.nike.cerberus.mapper.SafeDepositBoxMapper, hit: false, count: 1, threshold: 2
[pool-5-thread-1] INFO  c.n.c.c.DatabaseCache - getObject - Mapper: com.nike.cerberus.mapper.UserGroupMapper, hit: false, count: 1, threshold: 2
[pool-5-thread-1] INFO  c.n.c.c.DatabaseCache - getObject - Mapper: com.nike.cerberus.mapper.UserGroupMapper, hit: false, count: 1, threshold: 2
[pool-5-thread-1] INFO  c.n.c.c.DatabaseCache - getObject - Mapper: com.nike.cerberus.mapper.UserGroupMapper, hit: false, count: 1, threshold: 2
[pool-5-thread-1] INFO  c.n.c.c.DatabaseCache - getObject - Mapper: com.nike.cerberus.mapper.SafeDepositBoxMapper, hit: false, count: 1, threshold: 2
[pool-5-thread-1] INFO  c.n.c.c.DatabaseCache - getObject - Mapper: com.nike.cerberus.mapper.UserGroupMapper, hit: false, count: 2, threshold: 2
[pool-5-thread-1] INFO  c.n.c.c.DatabaseCache - getObject - Mapper: com.nike.cerberus.mapper.UserGroupMapper, hit: false, count: 2, threshold: 2
[pool-5-thread-1] INFO  c.n.c.c.DatabaseCache - getObject - Mapper: com.nike.cerberus.mapper.UserGroupMapper, hit: false, count: 2, threshold: 2
[pool-5-thread-1] INFO  c.n.c.c.DatabaseCache - getObject - Mapper: com.nike.cerberus.mapper.RoleMapper, hit: false, count: 1, threshold: 2
[pool-5-thread-1] INFO  c.n.c.c.DatabaseCache - getObject - Mapper: com.nike.cerberus.mapper.AwsIamRoleMapper, hit: false, count: 1, threshold: 2
[pool-5-thread-1] INFO  c.n.c.c.DatabaseCache - getObject - Mapper: com.nike.cerberus.mapper.AwsIamRoleMapper, hit: false, count: 1, threshold: 2
[pool-5-thread-4] INFO  c.n.c.c.DatabaseCache - getObject - Mapper: com.nike.cerberus.mapper.SafeDepositBoxMapper, hit: false, count: 1, threshold: 2
[pool-5-thread-1] INFO  c.n.c.c.DatabaseCache - getObject - Mapper: com.nike.cerberus.mapper.CategoryMapper, hit: false, count: 2, threshold: 2
[hystrix-event-processor-tp-3] INFO  c.n.c.e.p.LoggingEventProcessor - Event: GetSafeDepositBoxes Endpoint Called, Principal<EMAIL_ADDRESS>IP Address: _unknown, X-Forwarded-For: _unknown, Client Version: Dashboard/unknown, Method: GET, Path: /v2/safe-deposit-box, Action: 'Invoked GET request to uri /v2/safe-deposit-box triggering the GetSafeDepositBoxes endpoint class.', Originating Class: GetSafeDepositBoxes, SDB Name Slug: _unknown, Was Success: true, Trace ID: 3ef86baf62176dc5, Event Timestamp: Oct 21 2019, 06:51:40 PM +0000
[hystrix-event-processor-tp-4] INFO  c.n.c.e.p.LoggingEventProcessor - Event: GetSafeDepositBoxV2 Endpoint Called, Principal<EMAIL_ADDRESS>IP Address: _unknown, X-Forwarded-For: _unknown, Client Version: Dashboard/unknown, Method: GET, Path: /v2/safe-deposit-box/22bb1b10-d9f1-4406-a37a-4604936d76c4, Action: 'Fetch details for SDB with name: 'MySdb' and id: '22bb1b10-d9f1-4406-a37a-4604936d76c4'', Originating Class: GetSafeDepositBoxV2, SDB Name Slug: mysdb, Was Success: true, Trace ID: 9b5e755a63842015, Event Timestamp: Oct 21 2019, 06:51:40 PM +0000
[ForkJoinPool-1-worker-18] INFO  c.n.c.c.DatabaseCache - getObject - Mapper: com.nike.cerberus.mapper.SafeDepositBoxMapper, hit: false, count: 1, threshold: 2
[ForkJoinPool-1-worker-4] INFO  c.n.c.c.DatabaseCache - getObject - Mapper: com.nike.cerberus.mapper.SafeDepositBoxMapper, hit: false, count: 2, threshold: 2
[ForkJoinPool-1-worker-18] INFO  c.n.c.c.DatabaseCache - getObject - Mapper: com.nike.cerberus.mapper.PermissionsMapper, hit: false, count: 1, threshold: 2
[ForkJoinPool-1-worker-4] INFO  c.n.c.c.DatabaseCache - getObject - Mapper: com.nike.cerberus.mapper.PermissionsMapper, hit: false, count: 2, threshold: 2
[ForkJoinPool-1-worker-4] INFO  c.n.c.c.DatabaseCache - getObject - Mapper: com.nike.cerberus.mapper.SecureDataMapper, hit: false, count: 1, threshold: 2
[ForkJoinPool-1-worker-18] INFO  c.n.c.c.DatabaseCache - getObject - Mapper: com.nike.cerberus.mapper.SecureDataMapper, hit: false, count: 1, threshold: 2
[ForkJoinPool-1-worker-4] INFO  c.n.c.c.DatabaseCache - getObject - Mapper: com.nike.cerberus.mapper.SecureDataMapper, hit: false, count: 1, threshold: 2
[hystrix-event-processor-tp-7] INFO  c.n.c.e.p.LoggingEventProcessor - Event: ReadSecureData Endpoint Called, Principal<EMAIL_ADDRESS>IP Address: _unknown, X-Forwarded-For: _unknown, Client Version: Dashboard/3.34.0, Method: GET, Path: /v1/secret/app/mysdb/, Action: 'Listing keys under path: /v1/secret/app/mysdb/', Originating Class: ReadSecureData, SDB Name Slug: mysdb, Was Success: true, Trace ID: 84934a914b4f2c6f, Event Timestamp: Oct 21 2019, 06:51:40 PM +0000
[hystrix-event-processor-tp-8] INFO  c.n.c.e.p.LoggingEventProcessor - Event: GetSecureFiles Endpoint Called, Principal<EMAIL_ADDRESS>IP Address: _unknown, X-Forwarded-For: _unknown, Client Version: Dashboard/3.34.0, Method: GET, Path: /v1/secure-files/app/mysdb/, Action: 'List file metadata for SDB path: /v1/secure-files/app/mysdb/', Originating Class: GetSecureFiles, SDB Name Slug: _unknown, Was Success: true, Trace ID: 61def5f6e1a4b056, Event Timestamp: Oct 21 2019, 06:51:40 PM +0000
[ForkJoinPool-1-worker-4] INFO  c.n.c.c.DatabaseCache - getObject - Mapper: com.nike.cerberus.mapper.SafeDepositBoxMapper, hit: false, count: 1, threshold: 2
[ForkJoinPool-1-worker-4] INFO  c.n.c.c.DatabaseCache - getObject - Mapper: com.nike.cerberus.mapper.PermissionsMapper, hit: false, count: 1, threshold: 2
[ForkJoinPool-1-worker-4] INFO  c.n.c.c.DatabaseCache - getObject - Mapper: com.nike.cerberus.mapper.SecureDataMapper, hit: false, count: 1, threshold: 2
[hystrix-event-processor-tp-10] INFO  c.n.c.e.p.LoggingEventProcessor - Event: ReadSecureData Endpoint Called, Principal<EMAIL_ADDRESS>IP Address: _unknown, X-Forwarded-For: _unknown, Client Version: Dashboard/3.34.0, Method: GET, Path: /v1/secret/app/mysdb/test, Action: 'Reading secret with path: /v1/secret/app/mysdb/test', Originating Class: ReadSecureData, SDB Name Slug: mysdb, Was Success: true, Trace ID: 838952f514b424ba, Event Timestamp: Oct 21 2019, 06:51:50 PM +0000
[nioEventLoopGroup-3-10] INFO  c.n.c.c.DatabaseCache - getObject - Mapper: com.nike.cerberus.mapper.SafeDepositBoxMapper, hit: false, count: 1, threshold: 2
[nioEventLoopGroup-3-10] INFO  c.n.c.c.DatabaseCache - getObject - Mapper: com.nike.cerberus.mapper.PermissionsMapper, hit: false, count: 1, threshold: 2
[hystrix-event-processor-tp-9] INFO  c.n.c.e.p.LoggingEventProcessor - Event: WriteSecureData Endpoint Called, Principal<EMAIL_ADDRESS>IP Address: _unknown, X-Forwarded-For: _unknown, Client Version: Dashboard/3.34.0, Method: POST, Path: /v1/secret/app/mysdb/test, Action: 'Invoked POST request to uri /v1/secret/app/mysdb/test triggering the WriteSecureData endpoint class.', Originating Class: WriteSecureData, SDB Name Slug: mysdb, Was Success: true, Trace ID: 233805a1e3e89337, Event Timestamp: Oct 21 2019, 06:51:57 PM +0000
[pool-5-thread-1] INFO  c.n.c.c.DatabaseCache - getObject - Mapper: com.nike.cerberus.mapper.SecureDataMapper, hit: false, count: 1, threshold: 2
[ForkJoinPool-1-worker-4] INFO  c.n.c.c.DatabaseCache - getObject - Mapper: com.nike.cerberus.mapper.SafeDepositBoxMapper, hit: false, count: 1, threshold: 2
[ForkJoinPool-1-worker-4] INFO  c.n.c.c.DatabaseCache - getObject - Mapper: com.nike.cerberus.mapper.PermissionsMapper, hit: false, count: 1, threshold: 2
[ForkJoinPool-1-worker-4] INFO  c.n.c.c.DatabaseCache - getObject - Mapper: com.nike.cerberus.mapper.SecureDataMapper, hit: false, count: 1, threshold: 2
[hystrix-event-processor-tp-10] INFO  c.n.c.e.p.LoggingEventProcessor - Event: ReadSecureData Endpoint Called, Principal<EMAIL_ADDRESS>IP Address: _unknown, X-Forwarded-For: _unknown, Client Version: Dashboard/3.34.0, Method: GET, Path: /v1/secret/app/mysdb/test, Action: 'Reading secret with path: /v1/secret/app/mysdb/test', Originating Class: ReadSecureData, SDB Name Slug: mysdb, Was Success: true, Trace ID: ef55a97641984431, Event Timestamp: Oct 21 2019, 06:52:02 PM +0000
[ForkJoinPool-1-worker-4] INFO  c.n.c.c.DatabaseCache - getObject - Mapper: com.nike.cerberus.mapper.SafeDepositBoxMapper, hit: false, count: 1, threshold: 2
[ForkJoinPool-1-worker-4] INFO  c.n.c.c.DatabaseCache - getObject - Mapper: com.nike.cerberus.mapper.PermissionsMapper, hit: false, count: 1, threshold: 2
[hystrix-event-processor-tp-9] INFO  c.n.c.e.p.LoggingEventProcessor - Event: WriteSecureData Endpoint Called, Principal<EMAIL_ADDRESS>IP Address: _unknown, X-Forwarded-For: _unknown, Client Version: Dashboard/3.34.0, Method: POST, Path: /v1/secret/app/mysdb/test, Action: 'Invoked POST request to uri /v1/secret/app/mysdb/test triggering the WriteSecureData endpoint class.', Originating Class: WriteSecureData, SDB Name Slug: mysdb, Was Success: true, Trace ID: 5d3620911ba38114, Event Timestamp: Oct 21 2019, 06:52:14 PM +0000
[pool-5-thread-1] INFO  c.n.c.c.DatabaseCache - getObject - Mapper: com.nike.cerberus.mapper.SecureDataMapper, hit: false, count: 1, threshold: 2

I think this is ready to merge and be deployed and we can tweak numbers in the future if needed.


ADSync Profile Path

Hi,
One issue I noticed in adconnectdump.py script is that on Windows Server 2019 (probably on 2016 as well), the ADSync profile path is under:  c:\Windows\ServiceProfiles\ADSync\AppData\Local\Microsoft\Credentials
On which OS version did you try the python script ?
thanks

This was on server 2016, did you confirm changing the path fixed it for 2019? If so I can add it as additional search path, but don't have a 2019 box right now with ADSync

Also I attempted to support multiple paths in https://github.com/fox-it/adconnectdump/commit/2d1a01bdee5d85ed0f959e5c51aa46c87f216aa5, if you can try it with that version and let me know if it works, that'd be much appreciated.

This was on server 2016, did you confirm changing the path fixed it for 2019? If so I can add it as additional search path, but don't have a 2019 box right now with ADSync

yes - i confirm, this is why I raised this issue

Also I attempted to support multiple paths in 2d1a01b, if you can try it with that version and let me know if it works, that'd be much appreciated.

it works nicely, thanks for adapting the code

Ty for the report and for letting me know it's resolved! closing this one


Incorrectly setting a path, and then running paths will trigger a panic

Tried with a handful of versions of vault and safe. Should be reproducible across them, but my current specific versions are:
OS:
macOS High Sierra: 10.13.6

vault --version
Vault v0.9.6 ('7e1fbde40afee241f81ef08700e7987d86fc7242')

safe --version
safe v0.9.3

To reproduce run:
safe local --memory
Then:
safe set secret a/b/c x=yz
safe paths

You'll get a panic:
panic: runtime error: index out of range

goroutine 1 [running]:
main.main.func22(0x12f0420, 0x5, 0x1561b68, 0x1, 0x0, 0xc420475c01, 0x107cbe5)
	/tmp/build/01f561ac/gopath/src/github.com/starkandwayne/safe/main.go:1513 +0x5bb
main.(*Runner).Execute(0xc420048d40, 0x12f0420, 0x5, 0x1561b68, 0x0, 0x0, 0x0, 0x0)
	/tmp/build/01f561ac/gopath/src/github.com/starkandwayne/safe/runner.go:116 +0xa2
main.main()
	/tmp/build/01f561ac/gopath/src/github.com/starkandwayne/safe/main.go:2913 +0x30d0

safe tree will show:
.
└── secret
    ├──
    └── handshake

I think it's dying on the empty space. I also don't know how to recover from that, as in how to delete the now bad path.

This should be included in safe 0.9.5: https://github.com/starkandwayne/safe/releases/tag/v0.9.5

Cool Beans!


src/rmp/src/Restructure.cpp breaks when using system ABC

Describe the bug
I build OpenROAD ABC separately from OpenROAD to better cache builds, as ABC represents around 50% of OpenROAD's compile time from my experience and is changed very infrequently + mitigates https://github.com/The-OpenROAD-Project/OpenROAD/issues/2833, where the most frequent offender is the libabc.a creation.
The file src/rmp/src/Restructure.cpp has two imports:
#include "base/abc/abc.h"
#include "base/main/abcapis.h"

These two imports assume the headers are not being used as a system library.
This is the simplest workaround:
sed -i 's@#include "base/abc/abc.h"@#include <base/abc/abc.h>@' src/rmp/src/Restructure.cpp
sed -i 's@#include "base/main/abcapis.h"@#include <base/main/abcapis.h>@' src/rmp/src/Restructure.cpp

Expected Behavior
The #include should work regardless of the mode ABC is being used in.
Environment
Git commit: efe3d3e2e9b7e34052965890a8c5fb74358b2412
kernel: Darwin 22.3.0
os: macOS 13.2.1
cmake version 3.24.3

---

clang version 11.1.0
Target: x86_64-apple-darwin
Thread model: posix
InstalledDir: /nix/store/f22xbkm1ml2wvimnhz97ikbaniv4695v-clang-11.1.0/bin

To Reproduce
N/A
Relevant log output
No response
Screenshots
No response
Additional Context
No response

Can you add a -I flag to include the system header when you generate the cmake project.

Already do: didn't work
-DCMAKE_CXX_FLAGS=-I$ABC_PREFIX/include

I should note: the angled brackets did not work without the -I flag either iirc.

Ah, that's probably because we don't propagate compile flags to the abc subproject. I think there's some cmake function that will let you do that.
However, the only tier 1 supported version of ABC is the version bundled with OpenROAD so I think it would require an enhancement from you or another interested party to make this work more seamlessly in your flow.

To be clear, I am building the one in https://github.com/the-openroad-project/abc, but I getcha.
I'll see if I can come up with something that works for everybody.


Feature request to help combat the loss of inventories

I would like to request inventory backups. it would be ideal that inventories be saved/backed up when a user logs out. this i think would partially help combat the random inventory loss. as well as a command to back up all user inventories and a command to restore all or one users inventory and settings..

Inventories are saved to the disk when players leave the server. Further "backups" don't actually combat any issues, nor are they PWI's responsibility.


feat: fix async handling in jwt verify

I found a bug while developing https://sdjwt.co
in jwt verify method, It doesn't wait until its resolved. verifier can be async function so we need await keyword.

Lgtm,
Have you also checked the code if we have other mistreated async calls?
And I think all defined functions that can be passed to the sd-jwt instance should be async, since we don't know if the user needs it for his implementations. Or we define some type like Promise<string> | string

Yeah, I checked all code in core :) I guess now we're good.
Currently we treated passed functions as OrPromise which is Promise | type.


Store opensanctions data based on run IDs

Yeah it might be too simple but so far I haven't yet figured out how - it just keeps a list of history items and also puts it into a run folder. So if the items in the index file is limited to 100 and you want to go back 120, you just fetch the latest history, and then history file inside of the run folder for items[0] (the oldest run)....


Styled add to basket button

Hi, I wanted to practise contributing and came across your issue. Thanks.

Hey @tomahawk-jupiter, thanks for your interest.
Few things you need to take care of:

First make the issue assigned to yourself and then make the PR.
Attach a screenshot of the final work you have done.

Kindly do these changes and you will be good to go🚀
__
  Summary: :hammer_and_wrench: This code needs a fix


not finding json and flask error

whenever i run the py it nevers finds the json and when i try to run a actual command it gives a error about not finding flask

dm me at sleepy.wrld on discord


CMake: add native file dialog.

Add nativefiledialog entry and seperate lib entries with newlines for clearer git merge compatability

Considering engine does not use nativefiledialog it isnt really kosher to add it there. Git would take care of addition if you kept it in your repo. Though i can see that dependencies put on their own line would help you when merging so if ou removed nativefiledialog from this PR i would take it ;)


Market data API call failing

Ticker service uses the market data API at: https://lb.testnet.vega.xyz/markets-data
But this does not seem to work, although https://lb.testnet.vega.xyz/markets still does, so the server seems ok otherwise, and the markets data endpoint is still listed in the docs here: https://docs.fairground.vega.xyz/api/rest/data-node/api/v1/trading_data.html#operation/MarketsData
The API now returns:
{
  "code": 12,
  "message": "Not Implemented",
  "details": [
  ]
}

Perhaps this is relatred to data node split and a different server URL is needed?

Ok this is indeed because of data node split. REST API URLs now prefixed with /datanode/rest and seems ok
Commits:
https://github.com/vegaprotocol/ticker-service/commit/64561aa0c75dd8ef4362f8c8b1dbe447a757a25a
https://github.com/vegaprotocol/ticker-service/commit/031ddc88826689ec973d2e3fded82c58f32ce183


tag for 5.2.1 is missing

Do you want to request a feature or report a bug?
a bug
What is the current behavior?
the current version is 5.2.1 in npm repo, but when navigate to this github repo, but tag for 5.2.1.
**If the current behavior is a bug,
when you check the release, expand all tags, this is no tag for 5.2.1
What is the expected behavior?
provide a tag for 5.2.1 so that we can easy to check the source code for this version.
Please provide a git tag for latest version 5.2.1 so that we can check the source code.

I left a comment in https://github.com/nfl/react-helmet/issues/475 related to this.

You're better off using version 5.2.0, where I work currently we have encountered a memory-leak when using version 5.2.1 in a few server-rendered apps. Looks like the version could have been released unintentionally. Safe to say you should stay away from 5.2.1 until they've sorted it.

I left a comment in https://github.com/nfl/react-helmet/issues/475 related to this.

You're better off using version 5.2.0, where I work currently we have encountered a memory-leak when using version 5.2.1 in a few server-rendered apps. We were using Helmet.renderStatic() in the exact same way between both versions.
Looks like the version could have been released unintentionally. Safe to say you should stay away from 5.2.1 until they've sorted it.


Support ESM, add TypeScript types, add Prettier config

Closes #5

Oh, the types are unwanted here? Just so that I know if I should do another PR (either here or DefinitelyTyped).


Schema first: V12 Removed functionality to achieve two-way binding of input type fields

Is there an existing issue for this?

[X] I have searched the existing issues

Describe the bug
In V11 schema first, we were able to do the following to solve mismatch in input type field names between runtime type properties and GraphQL type fields:
BindComplexType<RuntimeFooInput>(b => b.To("FooInput").Field(f => f.FooProp).Name("fooField"))
In V12 schema first, no BindRuntimeType overload can achieve this, and we have not found another way to do it. Everything we've tried accomplishes resolver-style, i.e. one-way, bindings. Are we missing something?
We have sidestepped the issue temporarily by renaming the runtime type properties to match the corresponding GraphQL input type fields, but for our setup it comes with other issues that we would like to avoid in the future.
Steps to reproduce

N/A

Relevant log output
No response
Additional Context?
No response
Product
Hot Chocolate
Version
12.3.2

That is true... I have removed this functionality in favor of better automatic bindings. We will do another round of improvements soon for schema-first with another dot release. I will have a look at how we re-introduce this kind of bindings. Probably another overload on BindRuntimeType.

Please don't close this issue sir stale


combine author and admin in contracts

Having the author and admin as separate variables in the contract(s) means that we could instantiate a contract with the admin set to an inaccessible account. It makes more sense to have a single variable, admin, whom can instantiate and control the contract. If after instantiation the admin should be changed, a set_code_hash can be performed. This changes both author and admin in one hit, removing the risk of leaving one or the other or both set to an old account

it's just simpler 😎


Cannot use vendor descriptor for BLE GATT

Environment

Development Kit:  Macchina SuperB
Module or chip used:  ESP32-WROOM-32U
IDF version: v4.2
Build System: build.py
Compiler version: 8.4.0
Operating System: macOS
Power Supply: USB

Problem Description
I've been trying to set up my own BLE GATT service. I've managed to work my way through the existing documentation with some effort and get things working (almost). I'm able to successfully use my custom service and characteristics, but for some reason my custom descriptor is not showing.
Here is my BLE configuration:
index,uuid_len,uuid,perm,val_max_len,val_cur_len,value
0,16,0x2800,0x1,2,2,180A
1,16,0x2803,0x1,1,1,2
2,16,0x2A29,0x1,16,10,5865766f2c20496e632e
3,16,0x2803,0x1,1,1,2
4,16,0x2A26,0x1,16,6,76302e302e31
5,16,0x2800,0x1,16,16,338586FB4CAB422A87C479CFD0B2D2D2
6,16,0x2803,0x1,1,1,10
7,128,0x338586FC4CAB422A87C479CFD0B2D2D2,0x1,9,0,
8,128,0x338586FE4CAB422A87C479CFD0B2D2D2,0x1,1,1,0
9,16,0x2902,0x11,2,2,0000
10,16,0x2803,0x1,1,1,10
11,128,0x338586FC4CAB422A87C479CFD0B2D2D2,0x1,9,0,
12,128,0x338586FE4CAB422A87C479CFD0B2D2D2,0x1,1,1,0
13,16,0x2902,0x11,2,2,0000
14,16,0x2803,0x1,1,1,10
15,128,0x338586FD4CAB422A87C479CFD0B2D2D2,0x1,9,0,
16,128,0x338586FE4CAB422A87C479CFD0B2D2D2,0x1,1,1,0
17,16,0x2902,0x11,2,2,0000

Above, we can see the descriptors specified in the CSV: three 0x2902 (CCCD) and three 0x338586FE4CAB422A87C479CFD0B2D2D2 (my custom descriptor).
Expected Behavior
AT+BLEGATTSCHAR?
+BLEGATTSCHAR:"char",1,1,0x2A29,0x02
+BLEGATTSCHAR:"char",1,2,0x2A26,0x02
+BLEGATTSCHAR:"char",2,1,0x338586FC4CAB422A87C479CFD0B2D2D2,0x10
+BLEGATTSCHAR:"desc",2,1,1,0x338586FE4CAB422A87C479CFD0B2D2D2
+BLEGATTSCHAR:"desc",2,1,2,0x2902
+BLEGATTSCHAR:"char",2,2,0x338586FC4CAB422A87C479CFD0B2D2D2,0x10
+BLEGATTSCHAR:"desc",2,2,1,0x338586FE4CAB422A87C479CFD0B2D2D2
+BLEGATTSCHAR:"desc",2,2,2,0x2902
+BLEGATTSCHAR:"char",2,3,0x338586FD4CAB422A87C479CFD0B2D2D2,0x10
+BLEGATTSCHAR:"desc",2,3,1,0x338586FE4CAB422A87C479CFD0B2D2D2
+BLEGATTSCHAR:"desc",2,3,2,0x2902

Actual Behavior
AT+BLEGATTSCHAR?
+BLEGATTSCHAR:"char",1,1,0x2A29,0x02
+BLEGATTSCHAR:"char",1,2,0x2A26,0x02
+BLEGATTSCHAR:"char",2,1,0x338586FC4CAB422A87C479CFD0B2D2D2,0x10
+BLEGATTSCHAR:"char",2,2,0x338586FC4CAB422A87C479CFD0B2D2D2,0x10
+BLEGATTSCHAR:"char",2,3,0x338586FD4CAB422A87C479CFD0B2D2D2,0x10

When I connect to the device via an Android Tablet, I'm able to see all three characteristics and the CCCD for each (despite it not being listed in the output of the above AT command). However, my custom descriptor is not present.
If I remove my custom descriptors from the CSV but leave the CCCD descriptors, I see an improvement:
AT+BLEGATTSCHAR?
+BLEGATTSCHAR:"char",1,1,0x2A29,0x02
+BLEGATTSCHAR:"char",1,2,0x2A26,0x02
+BLEGATTSCHAR:"char",2,1,0x338586FC4CAB422A87C479CFD0B2D2D2,0x10
+BLEGATTSCHAR:"desc",2,1,1,0x2902
+BLEGATTSCHAR:"char",2,2,0x338586FC4CAB422A87C479CFD0B2D2D2,0x10
+BLEGATTSCHAR:"desc",2,2,1,0x2902
+BLEGATTSCHAR:"char",2,3,0x338586FD4CAB422A87C479CFD0B2D2D2,0x10
+BLEGATTSCHAR:"desc",2,3,1,0x2902

I also tried changing the first instance of my custom descriptor to 0x2901 (Characteristic User Description). When I do this, AT=BLEGATTSCHAR? reports 0x2901 and 0x2902 descriptors on the first characteristic, then none for the rest.
This would seem to indicate to me that something is going wrong while processing my custom descriptors. It seems processing my custom descriptor also prevents it from properly processing subsequent SIG descriptors correctly (though not entirely, since the client still reports the, it's just that the server won't show them via the AT command). Are vendor descriptors not supported?

Ok I tried using a descriptor UUID of 0x2910 which is not assigned in the Bluetooth SIG list. This succeeds. My best guess right now is there's a failure specifically for 128-bit characteristic UUIDs

Actually upon further investigation, although AT+BLEGATTSCHAR? shows the 0x2910 descriptor, none of my BLE clients can see it on the characteristic.

Does anyone have any info on support for 128-bit descriptor UUIDs?

Regarding how AT customizes the BLE service, you can refer to this document: https://docs.espressif.com/projects/esp-at/en/latest/Compile_and_Develop/How_to_customize_BLE_services.html. Hope it helps you.

@rfrowe For documentation on supporting 128-bit descriptor UUIDs, you can refer to: https://www.bluetooth.com/specifications/assigned-numbers/