Delete This Issue Once this Appendix has been merged into the identity hubs section

Identity Hubs

Hubs let you securely store and share data. A Hub is a datastore containing semantic data objects at well-known locations.  Each object in a Hub is signed by an identity and accessible via a globally recognized API format that explicitly maps to semantic data objects.  Hubs are addressable via unique identifiers maintained in a global namespace.

One DID to Many Hub Instances

A single entity may have one or more instances of a Hub, all of which are addressable via a URI routing mechanism linked to the entity's identifier.  Hub instances sync state changes, ensuring the owner can access data and attestations from anywhere, even when offline.

DID Document Service Endpoint Descriptors

There are two different variations of Hub-specific DID Document Service Endpoint descriptors, one that users associate with their DIDs, and another that Hosts use to direct requests to locations where their Hub infrastructure resides.

Users specify their Hub instances (different Hub Hosts) via the UserServiceEndpoint descriptor:

  "service": [{
  "type": "IdentityHub",
  "publicKey": "did:foo:123#key-1",
  "serviceEndpoint": {
    "@context": "schema.identity.foundation/hub",
    "@type": "UserServiceEndpoint",
    "instances": ["did:bar:456", "did:zaz:789"]
  }
}]

Hosts specify the locations of their Hub offerings via the HostServiceEndpoint descriptor:

"service": [{
  "type": "IdentityHub",
  "publicKey": "did:bar:456#key-1",
  "serviceEndpoint": {
    "@context": "schema.identity.foundation/hub",
    "@type": "HostServiceEndpoint",
    "locations": ["https://hub1.bar.com/.identity", "https://hub2.bar.com/blah/.identity"]
  }
}]

Syncing data between Hubs

Hub instances must sync data without requiring master-slave relationships or forcing a single implementation for storage or application logic. This requires a shared replication protocol for broadcasting and resolving changes. The protocol for reproducing Hub state across multiple instances is in the formative phases of definition/selection, but should be relatively straightforward to integrate on top of any NoSQL datastore.

Hub data serialization and export

All Hubs must support the export of their serialized state. This is to ensure the user retains full control over the portability of their data. A later revision to this document will specify the process for invoking this intent and retrieving the serialized data from a Hub instance.

Hub Protocol Schemes

Hub URI Scheme

In addition to the URL path convention for individual Hubs instances, it is important that links to an identity owner's data not be encoded with a dependency on a specific Hub instance. To make this possible, we propose the introduction of the following Hub URI scheme:

hub://did:foo:123abc/

User Agents that understand this scheme will leverage the Universal Resolver to lookup the Hub instances of the target DID and address the Hub endpoints via the Service Endpoints it finds within. This allows the formation of URIs that are not Hub instance-specific, allowing a more natural way to link to a DID's data, without having to specify a specific instance. This also prevents the circulation of dead links across the Web, given an identity owner can choose to add/remove new Hub instances at any time.

Authentication

The process of authenticating requests to identity hubs will follow the DIF/W3C DID Auth schemes. These standards are in early phases of formation - more info is available here.

The current Identity Hub authentication scheme seeks to accomplish two tasks:

mutually authenticate the client and the Hub using each's respective DID and associated keys.

encrypt all Hub requests and responses such that their contents are only available to the holders of the DID keys involved in the message exchange.

The current authentication scheme is an implementation of DID Auth, as
described here. This document will outline how to authenticate Hub requests
and responses. For full details on the authentication protocol used, and for a
reference implementation of the protocol, please refer to the `did-auth-jose`
library.

Authenticating a Hub request

Identity Hub requests and responses are signed and encrypted using the DID keys
of the sender and the recipient. This protects the message over any
transportation medium. All encrypted requests and responses follow the
JSON Web Encryption (JWE)
standard.

The steps to construct a JWE are as follows. First, construct a JWT. This JWT
will be signed by the sender of the Hub request; the `iss`. This JWT must have
the following form:

// JWT headers
{
  "alg": "RS256",
  "kid": "did:example:abc123#key-abc",
  "did-requester-nonce": "randomized-string",
  "did-access-token": "eyJhbGciOiJSUzI1N...",
}

// JWT body
{
  "@context": "https://schema.identity.foundation/0.1",
  "@type": "WriteRequest",
  "iss": "did:example:abc123",
  ...
}

// JWT signature
uQRqsaky-SeP3m9QPZmTGtRtMoKzyg6wwWF...

The JWT body is just the request whose format is outlined in
. The header values must be the following:

Header

Description

`alg`

Standard JWT header. Indicates the algorithm used to sign the JWT.

`kid`

Standard JWT header. The value should take the form `{did}#{key-id}`. Another
app can take this value, resolve the DID, and find the indicated public key that
can be used for signature validation of the commit. Here we have used
`did:example:abc123`, because the request is signed with the user's private key.

`did-requester-nonce`

A randomly generated string that must be cached on the client side. This string
will be used to verify the response from the Hub in the sections below.

`did-access-token`

A token that should be cached on the client side and included in each request
sent to the Hub. Since we do not have an access token yet, leave this property
out on the initial request. Sections below explain how to get an access token.

This JWT must use the typical JWT compact serialization format.

We can now use this JWT as the plaintext used to construct our JWE. The JWE must
have the following format.

// JWE protected header
{
  "alg": "RSA-OAEP-256",
  "kid": "did:example:abc456#abc-123",
  "enc": "A128GCM",
}

// JWE encrypted content encryption key
OKOawDo13gRp2ojaHV7LFpZcgV7T6DVZKTyKOM...

// JWE initialization vector
48V1_ALb6US04U3b...

// JWE plaintext (the JWT from above)
eyJhbGciOiJSUzI1NiIsImtpZCI6InRlc3R...

// JWE authentication tag
XFBoMYUZodetZdv...

We strongly reccommend using a JWT library to produce the above JWE. Using a
library, you should only need to provide the protected headers and the
plaintext. The plaintext value should be the JWT constructed above. The header
values are:

Header

Description

`alg`

Standard JWT header. Indicates the algorithm used to encrypt the JWE content
encryption key.

`kid`

Standard JWT header. The value should take the form `{did}#{key-id}`. Indicates
the Hub's key that is used to encrypt the JWE content encryption key. Here we
have used `did:example:abc456`, since that is the DID used by the Hub. The DID
used here should match the `aud` value in the Hub `WriteRequest`.

`enc`

Standard JWT header. Indicates the algorithm used to encrypt the plaintext using
the content encryption key to produce the ciphertext and authentication tag.

Finally, you have a signed and encrypted Hub request that can be transmitted to
the user's Identity Hub for secure storage.

Caching the access token

To send a successful request to an Identity Hub, you need to include an access
token in the `did-access-token` header of the JWE. The access token is a
short-lived JWT that can be used across many Hub requests until it expires.

On an initial request to an Identity Hub, you should exclude the
`did-access-token` header. When a Hub request does not include this header, the
Hub will reject the request. Instead, the Hub will return a JWE response (as
described in the next section) whose payload is an access token. You should
extract the access token from the response and cache it somewhere safe. The
access token can be used for subsequent requests.

Once you've cached the access token, include it in each request in the
`did-access-token` JWE header as described above.

Eventually, the access token will expire. Its expiry time can be found in the
`exp` claim inside the access token. If you attempt to use an expired access
token, the Identity Hub will return an error indicating a new access token is
required. To get a new access token, send another hub request without the
`did-access-token` header.

Receiving a hub response

When possible, a hub will respond with a JWE encrypted with the client's DID
keys:

eyJhbGciOiJSU0EtT0FFUCIsImVuYyI6IkEyNTZHQ00ifQ...

This JWE can be decrypted with the client's private key following the JWE
standard to reproduce the response's plaintext.

The contents of the JWE will either be a valid hub response or a new access
token. A new access token will only be included if the `did-access-token` header
was omitted in the request.

Authorization

Access control for data stored in Hubs is currently implemented via a bare bones
permission layer. Access to data can be granted by a Hub owner, and can be
restricted to certain types of data. More features to improve control over data
access will be added in the future.

The success of a decentralized identity platform is dependent upon the ability
for users to share their data with other people, organizations, apps, and
services in a way that respects and protects a user's privacy. In our
decentralized platform, all user information & data resides in the user's
identity Hub. This section outlines the current proposal for identity hub
authorization.

Scope of the current design

This proposal is a first cut. The intention is to start extremely simple, and
extend the model to include more richness over time. We choose to focus on two
simple use cases, described below.

Use case 1: Registering for a website

Alice has added some useful data about her wardrobe style to her Hub: her
measurements from her tailor, and a list of her favorite clothing brands. When
Alice goes to try out a new online clothing retailer, the retailer's website
allows her to set up an account using her DID. After signing in her DID, the
retailer's website is able to access Alice's style data. Alice does not have to
re-enter her sizes in the site, and the site can give her recommended options
based on her brand preferences.

Permission request flow

Use case 2: Reviewing & managing access

Alice learns that one of the websites she visited is making improper use of her
personal data. She would like to immediately remove that website's access to her
Hub.

Permission denied flow

Out of scope

These use cases, and the current Hub authorization system are not sufficient to
consider identity Hubs ready for real world usage. It leaves out several
features that have been discussed as being necessary for a minimally viable
authorization layer, including:

Features that control what is being granted:

How to grant a permission to a specific object by ID, rather than all objects of
a certain type.

How to grant a permission to a property of some object type, rather than the
entire object.

How to grant permission to an object type and all of the children object types
in its respective schema.

How to filter a permission to only:

objects created by a specific DID.

objects created in a certain time period.

objects larger than some byte size.

How to grant a permission to a zero-knowledge proof of some object, rather than the entire object.

How to grant permission to act as a delegate of a DID when interacting with other Hubs.

Features that control who is being granted access:

How to grant a permission to all DIDs, and therefore make some data public.

How to create a permission that explicitly denies a DID access to an object.

Features that limit/expand where or when access is granted:

How to time-bound permissions, such that a permission expires automatically.

How to grant permissions to an app on some devices, but not others.

Features that control why access is granted:

How an app can specify why permission is being requested.

How a user can specify why permission is being denied.

How relying parties and trust providers are reviewed for trustworthiness and integrity.

Features that are related to Hub authorization, but will be addressed at a later time:

How to request & send callbacks to notify apps of changes to data and
permissions in a Hub.

How to authorize the execution of services, or extensions, in a Hub.

What format(s) the Hub uses for requests & responses.

How to encrypt data in a Hub such that the Hub provider cannot access it.

Clearly, there is a large body of functionality that can be added to Hub
authorization over time. This is why this initial document intentionally strives
to be as simple as possible. We'll incorporate these things into Hub
authorization over time as we receive feedback from early adopters of Identity
Hubs.

Authorization Model

Access to data in Identity Hubs is controlled by a special object stored in Hubs
called a `PermissionGrant`. The structure of a `PermissionGrant` is:

{
  "owner": "did:example:12345", // the identity owner (granters)'s DID
  "grantee": "did:example:67890", // the grantee's DID
  "context": "schemas.clothing.org", // the data schema context
  "type": "measurements", // the data type
  "allow": "-R--", // allows create, read, update, and delete
  ... // additional richness & specificity can be added in the future
}

Granting permissions

When a hub owner grants a permission to another DID, they can do so by
specifying the exact objects in the permission grant. When permissions span more
than one data type, several PermissionGrant objects can be created. For each
PermissionGrant, the following object should be written to the `Permissions`
interface of the owner's Hub, typically via user agent:

{
  "@context": "schema.identity.foundation/Hub/",
  "@type": "PermissionGrant",
  "owner": "did:example:12345",
  "grantee": "did:example:67890",
  "context": "schemas.clothing.org",
  "type": "measurements",
  "allow": "-R--"
}

Note that the Hub Permissions interface only supports the single PermissionGrant
object type. The Hub should reject any requests to create objects of other types
in the Permissions interface, barring future updates to the PermissionGrant
model.

The response format, and any error conditions, should be consistent with all
other requests to Hubs. Upon creation of this permission grant object in a
user's Hub, the permission will be propagated to all other Hub instances listed
in the user's DID document via the Hub's standard sync & replication protocol.
This will ensure that all Hub instances are up-to-date with all new permission
grants in a timely manner.

Checking permissions

The following describes the logic implemented by the Hub's authorization layer
when a request arrives.

Receive incoming request from client

Determine relevant schema, verb, and client from request

Query for all PermissionGrants that whose object_type matches the schema, for
the given client DID

Check if any query results allow the verb in question

Return success/failed authorization check

Note that PermissionGrants do not understand or evaluate the structure of a
given schema. For instance, if a user grants access to all
"https://schema.org/game" objects, they do not implicitly grant
access to all "https://schema.org/videogame" objects (which is a child of game
in schema.org's hierarchy).

Reviewing & managing permissions

`PermissionGrant` objects can be created, read, modified, and deleted just like
any other object in a hub. To revoke access to data, the Hub owner needs to
simply modify an existing `PermissionGrant` or delete it entirely. Instructions
for reading and writing data in Identity Hubs is available in
.

Requesting permissions

At this time, proposals for how to request access to data in an identity hub via
a user agent are still being evaluated. In the future, we will update this
document with details on how a client can request access from a user.

API

Because of the sensitive nature of the data being transmitted to Identity Hubs,
the Identity Hub request and response API may look a bit different to developers
who are used to a traditional REST service API. Most of the differences are
based on the high level of security and privacy decentralized identity
inherently demands.

Commits

All data in identity hubs is represented as a series of "commits". A commit is
similar to a git commit; it represents a change to an object. To write data to
an identity hub, you need to construct and send a new commit to the hub. To read
data from an identity hub, you need to fetch all commits from the hub. An
object's current value can be constructed by applying all its commits in order.

The use of commits to represent data in identity hubs offers a few distinct
advantages:

it facilitates the hub's replication protocol, enabling multiple hub instances
to sync data.

it creates an auditable history of changes to an object, especially when each
commit is signed by a DID.

it eases implementation for use cases that need offline data modification and
require conflict resolution.

Each commit in a hub is a
JWT whose body
contains the data to be written to the hub. Here's an example of a deserialized
and decoded JWT:

// JWT headers
{
  "alg": "RS256",
  "kid": "did:foo:123abc#key-abc",
  "interface": "Collections",
  "context": "https://schema.org",
  "type": "MusicPlaylist",
  "operation": "create",
  "committed_at": "2018-10-24T18:39:10.10:00Z",
  "commit_strategy": "basic",
  "sub": "did:bar:456def",

// Example metadata about the object that is intended to be "public"
  "meta": {
    "tags": ["classic rock", "rock", "rock n roll"],
    "cache-intent": "full"
  }
}

// JWT body
{
  "@context": "http://schema.org/",
  "@type": "MusicPlaylist",
  "description": "The best rock of the 60s, 70s, and 80s",
  "tracks": ["..."],
}

// JWT signature
uQRqsaky-SeP3m9QPZmTGtRtMoKzyg6wwWF...

The commit is signed by the committer writing the data, in this case
did:foo:123abc. To write the commit into a hub, the committer must
send a Hub write request.

Write Request & Response Format

Instead of a REST-based scheme where data like the username, object types, and
query strings are present in the URL, Identity Hubs requests are self-contained
message objects that encapsulate all they need to be shielded from observing
entities during transport.

Each Hub request is a JSON object which is then signed and encrypted as outlined
in the authentication section. The outer envelope is signed with the key of the
"iss" DID, and encrypted with the Hub's DID key(s).

{
  iss: 'did:foo:123abc',
  sub: 'did:bar:456def',
  aud: 'did:baz:789ghi',
  "@context": "https://schema.identity.foundation/0.1",
  '@type': 'WriteRequest',

  // The commit in JSON Serialization Format
  // See: https://tools.ietf.org/html/rfc7515#section-3.1
  commit: {
    protected: "ewogICJpbnRlcmZhY2...",

// Optional metadata information not protected by the JWT signature
header: {
  "iss": "did:foo:123abc"
},

payload: "ewogICJAY29udGV4dCI6...",
signature: "b7V2UpDPytr-kMnM_YjiQ3E0J2..."
  }
}

Each response is also a JSON object, signed and encrypted in the same way as the request. Its contents are:

{
  "@context": "https://schema.identity.foundation/0.1",
  "@type": "WriteResponse",
  "developer_message": "completely optional message from the hub",
  "revisions": ["aHashOfTheCommitSubmitted"]
}

Object Read Request & Response Format

Objects follow one logical object across multiple commits. Object reads do not
contain the literal object data, only metadata associated. Objects may be
queried for using the following request format:

{
  "@context": "https://schema.identity.foundation/0.1",
  "@type": "ObjectQueryRequest",
  "iss": "did:foo:123abc",
  "sub": "did:bar:456def",
  "aud": "did:baz:789ghi",
  "query": {
      "interface": "Collections",
      "context": "http://schema.org",
      "type": "MusicPlaylist",

  // Optional object_id filters
  "object_id": ["3a9de008f526d239..", "a8f3e7..."]
  }
}

The response to a query for objects returns a list of object IDs along with the
object metadata. The format is:

{
  "@context": "https://schema.identity.foundation/0.1",
  "@type": "ObjectQueryResponse",
  "developer_message": "completely optional",
  "objects": [
    {
      // object metadata
      "interface": "Collections",
      "context": "http://schema.org",
      "type": "MusicPlaylist",
      "id": "3a9de008f526d239...",
      "created_by": "did:foo:123abc",
      "created_at": "2018-10-24T18:39:10.10:00Z",
      "sub": "did:foo:123abc",
      "commit_strategy": "basic",
      "meta": {
        "tags": ["classic rock", "rock", "rock n roll"],
        "cache-intent": "full"
      }
    },
    // ...more objects
  ]

// potential pagination token
  "skip_token": "ajfl43241nnn1p;u9390",
}

Commit Read Request & Response Format

To get the actual data in an object, you must read the commits from the Identity
Hub:

{
  "@context": "https://schema.identity.foundation/0.1",
  "@type": "CommitQueryRequest",
  "iss": "did:foo:123abc",
  "sub": "did:bar:456def",
  "aud": "did:baz:789ghi",
  "query": {
    "object_id": ["3a9de008f526d239..."],
    "revision": ["abc", "def", ...]
  },
}

A response to a query for commits contains a list of commit JWTs:

{
  "@context": "https://schema.identity.foundation/0.1",
  "@type": "CommitQueryResponse",
  "developer_message": "completely optional",
  "commits": [
    {
      protected: "ewogICJpbnRlcmZhY2UiO...",
      header: {
        "iss": "did:foo:123abc",
        // Hubs may add additional information to the unprotected headers for convenience
        "rev": "aHashOfTheCommit",
      },
      payload: "ewogICJAY29udGV4dCI6ICdo...",
      signature: "b7V2UpDPytr-kMnM_YjiQ3E0J2..."
    },
    // ...
  ],

// potential pagination token
  "skip_token": "ajfl43241nnn1p;u9390",
}

Paging

skip_token is an opaque token to be used for continuation of a
request.

They may be returned on responses with multiple results, and added to the
initial request's query object:

{
  "iss": "did:foo:123abc",
  "sub": "did:bar:456def",
  "aud": "did:baz:789ghi",
  "@context": "https://schema.identity.foundation/0.1",
  "@type": "ObjectQueryRequest",
  "interface": "Collections",
  "query": {
    "context": "schema.org",
    "type": "MusicPlaylist",
    "skip_token": "ajfl43241nnn1p;u9390"
  }
}

Interfaces

To facilitate common interactions and data storage, Hubs provide a set of standard interfaces that can be written to:

Profile ➜ The owning entity's primary descriptor object (schema
agnostic)

Permissions ➜ The access control JSON document

Actions ➜ A known endpoint for the relay of actions to the identity
owner

Stores ➜ Scoped 1:1 storage space that is directly assigned to
another, external DID

Collections ➜ The owning entity's identity collections (access
limited)

Services ➜ any custom, service-based functionality the identity
exposes

Profile

Each Hub has a profile object that describes the owning entity. The
profile object should use whatever schema and object that best represents the
entity. To ge the profile for a DID, send an object query to the
Profile interface:

{
  "@context": "https://schema.identity.foundation/0.1",
  "@type": "ObjectQueryRequest",
  "iss": "did:foo:123abc",
  "sub": "did:bar:456def",
  "aud": "did:baz:789ghi",
  "query": {
      "interface": "Profile",
  }
}

Permissions

All access and manipulation of identity data is subject to the permissions
established by the owning entity. See  explainer
for details.

Actions

The Actions interface is for sending a target identity semantically
meaningful objects that convey an intent to the sender, which often involves the
data payload of the object. The Actions interface is not
constrained to simple human-centric communications. Rather, it is intended as a
universal conduit through which identities can transact all manner of
activities, exchanges, and notifications.

The base data format for conveying an action shall be:
http://schema.org/Action

Here is a list of examples to show the range of use-cases this interface is
intended to support:

Human user contacts another with a textual message
(ReadAction)

Event app sends a request to RSVP for an event
(RsvpAction)

Voting agency prompts a user to submit a vote
(VoteAction)

{
  "@context": "http://schema.org/",
  "@type": "ReadAction",
  "name": "Acme Bank - March 2018 Statement",
  "description": "Your Acme Bank statement for account #1734765",
  "object": PDF_SOURCE
}

Stores

The best way to describe Stores is as a 1:1 DID-scoped variant of the W3C DOM's
origin-scoped window.localStorage API. The key difference being
that this form of persistent, pairwise object storage transcends providers,
platforms, and devices. For each storage relationship between the DID owner and
external DIDs, the Hub shall create a key-value document-based storage area. The
DID owner or external DID can store unstructured JSON data to the document, in
relation to the keys they specify. The Hub implementer may choose to limit the
available space of the storage document, with the option to expand the storage
limit based on criteria the implementer defines.

Collections

Data discovery has been a problem since the inception of the Web. Most previous
attempts to solve this begin with the premise that discovery is about individual
entities providing a mapping of their own service-specific API and data schemas.
While you can certainly create a common format for expressing different APIs and
data schemas, you are left with the same basic issue: a sea of services that
can't efficiently interoperate without specific review, effort, and integration.
Hubs avoid this issue entirely by recognizing that the problem with data
discovery is that it relies on discovery. Instead, Hubs assert the
position that locating and retrieving data should be an implicitly
knowable process.

Collections provide an interface for accessing data objects across all Hubs,
regardless of their implementation. This interface exerts almost no opinion on
what data schemas entities use. To do this, the Hub Collection interface allows
objects from any schema to be stored, indexed, and accessed in a unified manner.

With Collections, you store, query, and retrieve data based on the very schema
and type of data you seek. Here are a few example data objects from a variety of
common schemas that entities may write and access via a user's Hub:

Locate any offers a user might want to share with apps (http://schema.org/Offer)

{
  "@context": "https://schema.identity.foundation/0.1",
  "@type": "ObjectQueryRequest",
  "iss": "did:foo:123abc",
  "sub": "did:bar:456def",
  "aud": "did:baz:789ghi",
  "query": {
      "interface": "Collections",
      "context": "http://schema.org",
      "type": "Offer",
  }
}

 Services

 Services offer a means to surface custom service calls an identity wishes to
 expose publicly or in an access-limited fashion. Services should not require
 the Hub host to directly execute code the service calls describe; service
 descriptions should link to a URI where execution takes place.

Performing a Request request to the base Services
interface will respond with an object that contains an entry for every service
description the requesting entity is permitted to access.

// request
{
  "iss": "did:foo:123abc",
  "sub": "did:bar:456def",
  "aud": "did:baz:789ghi",
  "@context": "https://schema.identity.foundation/0.1",
  "@type": "ServicesRequest",
}

// response
{
  "@context": "https://schema.identity.foundation/0.1",
  "@type": "ServicesResponse",
  developer_message: "optional message",
  services: [{
    // Open API service descriptors
  }]
}

All definitions shall conform to the Open API descriptor format.

The point of this section seems to be:

"Identity Hubs are a formulation of Encrypted Datastores .... wherein a developer of an application or service stores the data for a Subject with that Subject, in their Identity Hub, instead of in a traditional centralized location owned and controlled by a third-party."

It is understood that the "application or service" is processing data rather than just storing it, otherwise, the EDV section would be sufficient. As a processor, the (application or) service has access to data in the clear. (I'm ignoring secure multi-party computation and such in this response. Please raise a separate issue if you want to deal with that.)
The issue, from the perspective of the Subject (of the "identity hub") is not where the storage is hosted but who controls access the contents of the storage. For example, as a Subject, I might self-host storage in my closet and expose that to to various Services to use but it would hardly protect me form these services having their way with my data.
This issue is nicely discussed from the perspective of service providers in the context of the Uniform Law Commission Committee (where I'm an 'observer') Main Street Privacy Coalition to: Collection and Use of Personally Identifiable Data Drafting Committee
where they explain the need for separation of concerns and effective regulation of platforms that combine multiple concerns.
Therefore, I strongly object to the proposed section on Identity Hubs as part of the spec and seek to have it removed entirely.
If it's within scope to specify a separate layer above the EDV access authorization, that layer must preserve the separation of concerns between policy decision points and the policy enforcement point. If we choose to specify this upper layer, we should also consider how auditing will be done in order to detect security breaches at the policy enforcement point if the PEP controls access to data in the clear.
My proposed solution to 'processing data in the clear' would be to define the authorization protocol that gives substantially all control to an agent of the Subject as the policy decision point. I don't know if that means a separate layer above the EDV or a parallel specification at the same layer as the EDV. That's related to VCs - zCaps / OCap a Discussion.

@agropper "It is understood that the "application or service" is processing data rather than just storing it, otherwise, the EDV section would be sufficient. As a processor, the (application or) service has access to data in the clear." - the vast majority of data that will flow through Identity Hubs (down into EDVs) is encrypted, and the Hub has no access to it. The only data that is send to a Hub in plaintext is an intended-public object like a publicly declared Tweet, that the Hub has no special access to - it's literally just hosting data like you would when you post blog entries for everyone to read.
Obviously it would be folly to remove the most useful, high value portion of the specification that represents what the vast majority of application and services developers need to begin creating apps that rely on the user's own personal datastore as their backing data storage mechanism. For this reason, I can't even fathom why we would make the spec relatively useless to the vast majority of valuable use cases by removing it.

@csuwildcat I maintain that the management of intended-public objects should have absolutely nothing to do with the Confidential Storage specification.
However, I would support adding a section about implementation guidance somewhere that tells people how they might set up a searchable directory of intended-public objects and what, if anything about that directory relates to the normative aspects of the Confidential Storage spec.
For example, when we decide to  specify the authorization protocol for access to an EDV, we could mention that we recommend that directory creators support the same protocol so that the same data controller can interact with both EDV and intended public-object directories.


Reading the makernote values

Hello.
My final goal is to find out the serial number of the camera used to take a picture.
It's either in the main exif datas :
0xc62f  50735   Image   Exif.Image.CameraSerialNumber   Ascii
0xa431  42033   Photo   Exif.Photo.BodySerialNumber     Ascii
or in the makernote value.
0x001d  29  Nikon3  Exif.Nikon3.SerialNumber    Ascii   Serial Number
I tried to add the 2 first to the list of tags, without success.
I tried also to read the makernote value, but it's an array with thousands of values, where I can't find my serial number(I know it's inside, I find it out with other tools).
Any idea on how to get the value ?
Best regards,
Pierre

I'm working on a project for detection and recovery of stolen cameras.
I too would love to get camera serial number from an image exif data.
did you make any progress with this?


[DOC]best_practice describe sendAsyncFailed error

the https://github.com/apache/rocketmq/blob/master/docs/cn/best_practice.md said
1.2 消息发送失败处理方式
Producer的send方法本身支持内部重试，重试逻辑如下：
至多重试2次（同步发送为2次，异步发送为0次）。
but https://github.com/apache/rocketmq/blob/734834cb560a341dd03f1c738974bc5656e04c87/docs/cn/client/java/API_Reference_DefaultMQProducer.md with retryTimesWhenSendFailed (default 2)
and the code also show it will resend message when send failed by async mode

@coder-zzzz Would you like to pull a request to fix this doc?

@coder-zzzz Would you like to pull a request to fix this doc?

ok


feat(rich-text-web): configurable styles drop down based on style …

…classes from app theme

Pull request type

New feature (non-breaking change which adds functionality)

Description
The rich text widget has a styles menu that allows to style a selection or starting from the current cursor position. However, only very few choices are available, and the hardcoded styles might not fit the apps needs

Besides the fonts in the menu look different than the fonts in the editor:

This feature leverages CKEditor4's capability to use custom styles, in case of a Mendix application the theme defined in App/Styling: https://ckeditor.com/docs/ckeditor4/latest/guide/dev_howtos_styles.html
By default, nothing is changed, so existing apps should behave as before (updating widgets is required though). When the developer decides to use custom styles, he sets the "Styleset name" in the widget properties.
The developer can then assign a subset of style classes from the App's theme to the rich text widget.

The styles menu can be explicitly designed so that proper styles are selectable depending on the context (text, table, selection, etc.). The "Valid Selectors" property defines available style classes from the theme. In the "Edit Styles" dialog, the available styles can be assigned to HTML tags, so that the "Styles" menu shows available options based on the current context.

Styles in the menu look the same as in the editor, so there is a better WYSIWYG experience.
Menu:

Editor:

Result in HTML Element:

Inspecting the resulting content stored in the entity shows that style information is stored in classes instead of style attributes and HTML, which is more in line with Mendix best practices:

I have created a Mendix app including data snapshot to showcase the improved widget. If so desired, I can invite any reviewer as a team member to test the changes.

Will resubmit a pull request for version 3.x


Pasting with Maccy doesn't focus on search text

Before Submitting Your Bug Report

[X] I have verified that there isn't already an issue reporting the same bug to prevent duplication.
[X] I have seen the FAQ.

Maccy Version (see 'About' window)
0.26.5
macOS Version
14.2.1
Maccy Settings
{
    "KeyboardShortcuts_popup" = "{\\"carbonModifiers\\":256,\\"carbonKeyCode\\":16}";
    "LaunchAtLogin__hasMigrated" = 1;
    "NSWindow Frame com.sindresorhus.Preferences.FrameAutosaveName" = "988 362 542 436 0 0 1920 1055 ";
    SUEnableAutomaticChecks = 0;
    SUHasLaunchedBefore = 1;
    SUSendProfileInfo = 0;
    clearOnQuit = 1;
    enabledPasteboardTypes =     (
        "public.rtf",
        "public.png",
        "public.file-url",
        "public.utf8-plain-text",
        "public.tiff",
        "public.html"
    );
    history = {length = 42, bytes = 0x62706c69 73743030 a0080000 00000000 ... 00000000 00000009 };
    ignoredPasteboardTypes =     (
        "com.agilebits.onepassword",
        "net.antelle.keeweb",
        "Pasteboard generator type",
        "de.petermaurer.TransientPasteboardType",
        "com.typeit4me.clipping"
    );
    migrations =     {
        "2020-02-22-history-item-add-copied-at" = 1;
        "2020-02-22-history-item-add-number-of-copies" = 1;
        "2020-04-18-switch-storage-to-core-data" = 1;
        "2020-04-25-allow-custom-ignored-types" = 1;
        "2020-06-19-use-keyboardshortcuts" = 1;
        "2020-09-01-ignore-keeweb" = 1;
        "2021-02-20-allow-to-customize-supported-types" = 1;
        "2021-06-28-add-title-to-history-item" = 1;
        "2021-10-16-remove-dynamic-pasteboard-types" = 1;
        "2022-08-01-rename-suppress-clear-alert" = 1;
        "2022-11-14-add-html-rtf-to-supported-types" = 1;
        "2023-01-22-add-regexp-search-mode" = 1;
    };
    pasteByDefault = 1;
    popupPosition = cursor;
}

Description
Search text out of focus unless clicked with mouse.
Steps to Reproduce

Press shortcut to launch Maccy
Type any alphanumeric character, no response because search text area is out of focus

You are using a very old version of Maccy, please upgrade to 0.29.3 and re-test.

Please re-open if it's still not working after upgrade.


Codecov

uwu

Codecov Report

Merging #11 into unit-tests will not change coverage.
The diff coverage is n/a.

@@             Coverage Diff             @@
##           unit-tests      #11   +/-   ##
===========================================
  Coverage       18.02%   18.02%
===========================================
  Files              25       25
  Lines             283      283
===========================================
  Hits               51       51
  Misses            232      232

Codecov Report

Merging #11 into unit-tests will not change coverage.
The diff coverage is n/a.

@@             Coverage Diff             @@
##           unit-tests      #11   +/-   ##
===========================================
  Coverage       18.02%   18.02%
===========================================
  Files              25       25
  Lines             283      283
===========================================
  Hits               51       51
  Misses            232      232

Codecov Report

Merging #11 into unit-tests will not change coverage.
The diff coverage is n/a.

@@             Coverage Diff             @@
##           unit-tests      #11   +/-   ##
===========================================
  Coverage       18.02%   18.02%
===========================================
  Files              25       25
  Lines             283      283
===========================================
  Hits               51       51
  Misses            232      232

Codecov Report

Merging #11 into unit-tests will decrease coverage by 1.76%.
The diff coverage is n/a.

@@              Coverage Diff               @@
##           unit-tests      #11      +/-   ##
==============================================
- Coverage       18.02%   16.25%   -1.77%
==============================================
  Files              25       25
  Lines             283      283
==============================================
- Hits               51       46       -5
- Misses            232      237       +5

Impacted Files
Coverage Δ

lib/auth/auth_bloc.dart
62.50% <0.00%> (-31.25%)
:arrow_down:

Codecov Report

Merging #11 into unit-tests will decrease coverage by 1.76%.
The diff coverage is n/a.

@@              Coverage Diff               @@
##           unit-tests      #11      +/-   ##
==============================================
- Coverage       18.02%   16.25%   -1.77%
==============================================
  Files              25       25
  Lines             283      283
==============================================
- Hits               51       46       -5
- Misses            232      237       +5

Impacted Files
Coverage Δ

lib/auth/auth_bloc.dart
62.50% <0.00%> (-31.25%)
:arrow_down:

Codecov Report

Merging #11 into unit-tests will decrease coverage by 1.76%.
The diff coverage is n/a.

@@              Coverage Diff               @@
##           unit-tests      #11      +/-   ##
==============================================
- Coverage       18.02%   16.25%   -1.77%
==============================================
  Files              25       25
  Lines             283      283
==============================================
- Hits               51       46       -5
- Misses            232      237       +5

Impacted Files
Coverage Δ

lib/auth/auth_bloc.dart
62.50% <0.00%> (-31.25%)
:arrow_down:


Cannot use reset() with @hookform/devtools

Describe the bug
When I try to use Devtools and use reset() function.
All remain fields are removed and nothing is reset.
To Reproduce
Just simply add devtools to the form. Everything will be normal again, when the devtools is removed.
Codesandbox link (Required)
https://codesandbox.io/s/bug-with-devtools-dyk9l?file=/src/App.js
Expected behavior
It should run correctly along with enabled Devtools.
Desktop (please complete the following information):

OS: Windows
Browser Microsoft Edge
Version 91.0.864.48 (Official build) (64-bit)

Additional context
I thought the reason came from the recent unstable versions of react-hook-form, but now I find out the culprit is @hookform/devtools.

Eh, had so much troubles because of this bug when working on complex form. Hope it's fixed soon but if not maybe I have time in several weeks.


[Backport 1.x] Upgrading zookeeper in hdfs-fixture (#5007)

Signed-off-by: Vacha Shah<EMAIL_ADDRESS>
Description
Backport #5007 to 1.x
Issues Resolved
[List any issues this PR will resolve]
Check List

[ ] New functionality includes testing.

[X] All tests pass

[ ] New functionality has been documented.

[ ] New functionality has javadoc added

[X] Commits are signed per the DCO using --signoff
[X] Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Codecov Report

Merging #5046 (fe04c7c) into 1.x (e9ffcdb) will decrease coverage by 0.11%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##                1.x    #5046      +/-   ##
============================================
- Coverage     77.90%   77.79%   -0.12%
+ Complexity    63425    63397      -28
============================================
  Files          4453     4453
  Lines        275011   275011
  Branches      41172    41172
============================================
- Hits         214245   213939     -306
- Misses        43897    44150     +253
- Partials      16869    16922      +53

Impacted Files
Coverage Δ

...java/org/opensearch/client/indices/DataStream.java
0.00% <0.00%> (-76.09%)
:arrow_down:

...nsearch/search/dfs/DfsPhaseExecutionException.java
0.00% <0.00%> (-66.67%)
:arrow_down:

...port/ResponseHandlerFailureTransportException.java
0.00% <0.00%> (-60.00%)
:arrow_down:

.../opensearch/client/indices/CloseIndexResponse.java
39.50% <0.00%> (-50.62%)
:arrow_down:

...rch/index/engine/RefreshFailedEngineException.java
50.00% <0.00%> (-50.00%)
:arrow_down:

...ch/index/shard/ShardNotInPrimaryModeException.java
50.00% <0.00%> (-50.00%)
:arrow_down:

...h/index/engine/EngineCreationFailureException.java
50.00% <0.00%> (-50.00%)
:arrow_down:

.../action/ingest/SimulatePipelineRequestBuilder.java
0.00% <0.00%> (-50.00%)
:arrow_down:

...gling/import_index/ImportDanglingIndexRequest.java
43.75% <0.00%> (-50.00%)
:arrow_down:

...n/indices/upgrade/post/UpgradeSettingsRequest.java
30.00% <0.00%> (-45.00%)
:arrow_down:

... and 502 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


Add Sprint Boot

Add sprint boot ,   change the WebConfig.scala to WebApplication.scala  and delete web.xml
It will start the embed server after run the main method from WebApplication.scala without manually copy the war to Jetty.
It will be convenient for debug and deploy.
soda-sprint-boot.zip

I decided against including this, main reason being that I am not familiar with sprint boot and it would be hard for me to maintain.


Fix type checks and typos

This PR fixes several type checker errors and typos in

ipm_image_node
ipm_library

Also is there any easy way of running the type checks in the CI?

Dont know yet, but WE could commit workspace settings for IDEs, that turn on Type checking.
As Bitbots, we will discuss this on wednesday

Thanks for the improvements @jaagut! I have the same points as raised by @Flova, and the rest looks good to me.


CI failure: KafkaPolicies failed due to kafka-broker going into CrashLoopBackOff because service names aren't plumbed down to kube-dns for 5 minutes in K8s 1.7 environments

Build link: https://jenkins.cilium.io/job/Ginkgo-CI-Tests-Pipeline/3446/testReport/junit/k8s-1/7/K8sKafkaPolicyTest_Kafka_Policy_Tests_KafkaPolicies/
Output:
Stacktrace

/home/jenkins/workspace/Ginkgo-CI-Tests-Pipeline/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:333
Kafka Pods are not ready after timeout
Expected
    <*errors.errorString | 0xc425a8b630>: {
        s: "Timeout reached: timed out waiting for pods with filter -l zgroup=kafkaTestApp to be ready",
    }
to be nil
/home/jenkins/workspace/Ginkgo-CI-Tests-Pipeline/src/github.com/cilium/cilium/test/k8sT/KafkaPolicies.go:133
Standard Output

Number of "context deadline exceeded" in logs: 0
Number of "level=error" in logs: 0
Number of "level=warning" in logs: 0
Number of "Cilium API handler panicked" in logs: 0
Cilium pods: [cilium-cx4bm cilium-sc5qb]
Netpols loaded:
CiliumNetworkPolicies loaded:
Endpoint Policy Enforcement:
     empire-outpost-9999-4143713519-q2sv1   =>   none
     kafka-broker-2873197211-qh6kt          =>   none
     prometheus-core-3506541099-jwhst       =>   none
     empire-backup-2986355077-7lh6d         =>   none
     empire-hq-2917507631-n4d1c             =>   none
     empire-outpost-8888-3627733903-8xdrk   =>   none
     zookeeper-1032194181-mt667             =>   none
     cilium-health-k8s2                     =>   none
     kube-dns-2552250787-1xc0z              =>   none
Cilium agent "cilium-cx4bm": Status: Ok  Health: Ok Nodes "k8s1 k8s2" ContinerRuntime: Ok Kubernetes: Ok KVstore: Ok Controllers: Total 50 Failed 0
Cilium agent "cilium-sc5qb": Status: Ok  Health: Ok Nodes "k8s1 k8s2" ContinerRuntime: Ok Kubernetes: Ok KVstore: Ok Controllers: Total 26 Failed 0

Standard Error

===================== TEST FAILED =====================
cmd: kubectl get pods -o wide --all-namespaces
NAMESPACE     NAME                                   READY     STATUS             RESTARTS   AGE       IP              NODE
default       empire-backup-2986355077-7lh6d         1/1       Running            0          5m        <IP_ADDRESS>     k8s2
default       empire-hq-2917507631-n4d1c             1/1       Running            0          5m        <IP_ADDRESS>     k8s1
default       empire-outpost-8888-3627733903-8xdrk   1/1       Running            0          5m        <IP_ADDRESS>     k8s2
default       empire-outpost-9999-4143713519-q2sv1   1/1       Running            0          5m        <IP_ADDRESS>     k8s2
default       kafka-broker-2873197211-qh6kt          0/1       CrashLoopBackOff   5          5m        <IP_ADDRESS>      k8s2
default       zookeeper-1032194181-mt667             1/1       Running            0          5m        <IP_ADDRESS>     k8s2
kube-system   cilium-cx4bm                           1/1       Running            0          5m        <IP_ADDRESS>   k8s2
kube-system   cilium-sc5qb                           1/1       Running            0          5m        <IP_ADDRESS>   k8s1
kube-system   etcd-k8s1                              1/1       Running            0          46m       <IP_ADDRESS>   k8s1
kube-system   kube-apiserver-k8s1                    1/1       Running            0          46m       <IP_ADDRESS>   k8s1
kube-system   kube-controller-manager-k8s1           1/1       Running            0          46m       <IP_ADDRESS>   k8s1
kube-system   kube-dns-2552250787-1xc0z              3/3       Running            0          6m        <IP_ADDRESS>     k8s2
kube-system   kube-proxy-vps9k                       1/1       Running            0          47m       <IP_ADDRESS>   k8s1
kube-system   kube-proxy-vxchz                       1/1       Running            0          40m       <IP_ADDRESS>   k8s2
kube-system   kube-scheduler-k8s1                    1/1       Running            0          46m       <IP_ADDRESS>   k8s1
prometheus    prometheus-core-3506541099-jwhst       1/1       Running            0          40m       <IP_ADDRESS>      k8s1

cmd: kubectl exec -n kube-system cilium-cx4bm -- cilium service list
ID   Frontend             Backend
1    <IP_ADDRESS>:443        1 => <IP_ADDRESS>:6443
2    <IP_ADDRESS>:53        1 => <IP_ADDRESS>:53
3    <IP_ADDRESS>:9090    1 => <IP_ADDRESS>:9090
4    <IP_ADDRESS>:2181   1 => <IP_ADDRESS>:2181

cmd: kubectl exec -n kube-system cilium-cx4bm -- cilium endpoint list
ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                        IPv6                   IPv4          STATUS
           ENFORCEMENT        ENFORCEMENT
3650       Disabled           Disabled          22035      k8s:app=empire-outpost                             f00d::a0a:100:0:e42    <IP_ADDRESS>   ready
                                                           k8s:io.cilium.k8s.policy.serviceaccount=default
                                                           k8s:io.kubernetes.pod.namespace=default
                                                           k8s:outpostid=9999
                                                           k8s:zgroup=kafkaTestApp
6117       Disabled           Disabled          35842      k8s:app=empire-outpost                             f00d::a0a:100:0:17e5   <IP_ADDRESS>   ready
                                                           k8s:io.cilium.k8s.policy.serviceaccount=default
                                                           k8s:io.kubernetes.pod.namespace=default
                                                           k8s:outpostid=8888
                                                           k8s:zgroup=kafkaTestApp
6523       Disabled           Disabled          4          reserved:health                                    f00d::a0a:100:0:197b   <IP_ADDRESS>   ready
8208       Disabled           Disabled          50075      k8s:app=kafka                                      f00d::a0a:100:0:2010   <IP_ADDRESS>    ready
                                                           k8s:io.cilium.k8s.policy.serviceaccount=default
                                                           k8s:io.kubernetes.pod.namespace=default
                                                           k8s:zgroup=kafkaTestApp
22152      Disabled           Disabled          47341      k8s:io.cilium.k8s.policy.serviceaccount=kube-dns   f00d::a0a:100:0:5688   <IP_ADDRESS>   ready
                                                           k8s:io.kubernetes.pod.namespace=kube-system
                                                           k8s:k8s-app=kube-dns
32564      Disabled           Disabled          59494      k8s:app=zook                                       f00d::a0a:100:0:7f34   <IP_ADDRESS>   ready
                                                           k8s:io.cilium.k8s.policy.serviceaccount=default
                                                           k8s:io.kubernetes.pod.namespace=default
                                                           k8s:zgroup=kafkaTestApp
45365      Disabled           Disabled          43645      k8s:app=empire-backup                              f00d::a0a:100:0:b135   <IP_ADDRESS>   ready
                                                           k8s:io.cilium.k8s.policy.serviceaccount=default
                                                           k8s:io.kubernetes.pod.namespace=default
                                                           k8s:zgroup=kafkaTestApp

cmd: kubectl exec -n kube-system cilium-sc5qb -- cilium service list
ID   Frontend             Backend
1    <IP_ADDRESS>:443        1 => <IP_ADDRESS>:6443
2    <IP_ADDRESS>:53        1 => <IP_ADDRESS>:53
3    <IP_ADDRESS>:9090    1 => <IP_ADDRESS>:9090
4    <IP_ADDRESS>:2181   1 => <IP_ADDRESS>:2181

cmd: kubectl exec -n kube-system cilium-sc5qb -- cilium endpoint list
ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                              IPv6                 IPv4          STATUS
           ENFORCEMENT        ENFORCEMENT
8417       Disabled           Disabled          48563      k8s:app=prometheus                                       f00d::a0a:0:0:20e1   <IP_ADDRESS>    ready
                                                           k8s:component=core
                                                           k8s:io.cilium.k8s.policy.serviceaccount=prometheus-k8s
                                                           k8s:io.kubernetes.pod.namespace=prometheus
38477      Disabled           Disabled          4          reserved:health                                          f00d::a0a:0:0:964d   <IP_ADDRESS>   ready
41052      Disabled           Disabled          32829      k8s:app=empire-hq                                        f00d::a0a:0:0:a05c   <IP_ADDRESS>   ready
                                                           k8s:io.cilium.k8s.policy.serviceaccount=default
                                                           k8s:io.kubernetes.pod.namespace=default
                                                           k8s:zgroup=kafkaTestApp

Logs:
4bc18599_K8sKafkaPolicyTest_Kafka_Policy_Tests_KafkaPolicies.zip

Zookeeper DNS failed to resolve within the Kafka broker. From the pod's logs:
2018-07-27T10:54:27.516143862Z [2018-07-27 10:54:27,513] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
2018-07-27T10:54:27.516163091Z java.net.UnknownHostException: zook: Name does not resolve
2018-07-27T10:54:27.516165504Z 	at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
2018-07-27T10:54:27.516167476Z 	at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
2018-07-27T10:54:27.516169246Z 	at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
2018-07-27T10:54:27.516170967Z 	at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
2018-07-27T10:54:27.516172698Z 	at java.net.InetAddress.getAllByName(InetAddress.java:1192)
2018-07-27T10:54:27.516182839Z 	at java.net.InetAddress.getAllByName(InetAddress.java:1126)
2018-07-27T10:54:27.516184829Z 	at org.apache.zookeeper.client.StaticHostProvider.<init>(StaticHostProvider.java:61)
2018-07-27T10:54:27.516186695Z 	at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:445)
2018-07-27T10:54:27.51618839Z 	at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:380)
2018-07-27T10:54:27.516190161Z 	at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:86)
2018-07-27T10:54:27.516191885Z 	at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1539)
2018-07-27T10:54:27.516193563Z 	at kafka.server.KafkaServer.createZkClient$1(KafkaServer.scala:345)
2018-07-27T10:54:27.516195258Z 	at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:369)
2018-07-27T10:54:27.516196974Z 	at kafka.server.KafkaServer.startup(KafkaServer.scala:202)
2018-07-27T10:54:27.516198656Z 	at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:38)
2018-07-27T10:54:27.516200337Z 	at kafka.Kafka$.main(Kafka.scala:92)
2018-07-27T10:54:27.516201953Z 	at kafka.Kafka.main(Kafka.scala)
2018-07-27T10:54:27.516698901Z [2018-07-27 10:54:27,516] INFO shutting down (kafka.server.KafkaServer)
2018-07-27T10:54:27.518095563Z [2018-07-27 10:54:27,517] WARN  (kafka.utils.CoreUtils$)
2018-07-27T10:54:27.518104494Z java.lang.NullPointerException
2018-07-27T10:54:27.518106742Z 	at kafka.server.KafkaServer.$anonfun$shutdown$6(KafkaServer.scala:569)
2018-07-27T10:54:27.518108745Z 	at kafka.utils.CoreUtils$.swallow(CoreUtils.scala:85)
2018-07-27T10:54:27.518110549Z 	at kafka.server.KafkaServer.shutdown(KafkaServer.scala:569)
2018-07-27T10:54:27.518112267Z 	at kafka.server.KafkaServer.startup(KafkaServer.scala:326)
2018-07-27T10:54:27.518113958Z 	at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:38)
2018-07-27T10:54:27.518115724Z 	at kafka.Kafka$.main(Kafka.scala:92)
2018-07-27T10:54:27.518117359Z 	at kafka.Kafka.main(Kafka.scala)
2018-07-27T10:54:27.52276573Z [2018-07-27 10:54:27,522] INFO shut down completed (kafka.server.KafkaServer)
2018-07-27T10:54:27.52325169Z [2018-07-27 10:54:27,523] ERROR Exiting Kafka. (kafka.server.KafkaServerStartable)
2018-07-27T10:54:27.5244127Z [2018-07-27 10:54:27,524] INFO shutting down (kafka.server.KafkaServer)
``

Perhaps we should split up how we deploy the Kafka app. I hypothesize that the issue here is that we deploy everything at once in one YAML (kafka-sw-app.yaml):
https://github.com/cilium/cilium/blob/295c5cb9b7811d205fba69f165c450d56a73540c/test/k8sT/KafkaPolicies.go#L131-L133
The issue with this, is that the pods and the services are deployed at the same time; is it at all likely that the services are not ready / their DNS entries are not set up before the pods start, which results in the pods crashing? What if we install the services, wait for DNS to resolve, then install the deployment and then wait for the pods?

@ianvernon I don't think that is needed TBH. If kafka can't  connect to zookeeper it'll retry because we have a livenessProbe in there, and kafka will restart until can connect to zook.

From the console output from the test, the test lasted from 10:51:00 until 10:56:26:
10:51:00 K8sKafkaPolicyTest Kafka Policy Tests
10:51:00   KafkaPolicies
10:51:00   /home/jenkins/workspace/Ginkgo-CI-Tests-Pipeline/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:376
10:56:26 ===================== TEST FAILED =====================

Kafka app was deployed shortly after:
time="2018-07-27T10:51:14Z" level=debug msg="applying /home/vagrant/go/src/github.com/cilium/cilium/test/k8sT/manifests/kafka-sw-app.yaml"
time="2018-07-27T10:51:14Z" level=debug msg="running command: kubectl apply -f  /home/vagrant/go/src/github.com/cilium/cilium/test/k8sT/manifests/kafka-sw-app.yaml"
cmd: "kubectl apply -f  /home/vagrant/go/src/github.com/cilium/cilium/test/k8sT/manifests/kafka-sw-app.yaml" exitCode: 0 stdout:
deployment "kafka-broker" created
deployment "zookeeper" created
service "zook" created
service "kafka-service" created
deployment "empire-hq" created
deployment "empire-outpost-8888" created
deployment "empire-outpost-9999" created
deployment "empire-backup" created

DNS lookups for the zook service in K8s failed shortly after:
2018-07-27T10:51:24.208715284Z I0727 10:51:24.208647       5 dns.go:637] Query for "zook.default.svc.cluster.local.", exact: false
2018-07-27T10:51:24.208782861Z I0727 10:51:24.208754       5 dns.go:866] Not a federation query: len(["zook" "default" "svc" "cluster" "local"]) != 4+len(["local" "cluster"])
2018-07-27T10:51:24.208816735Z I0727 10:51:24.208800       5 dns.go:757] Found 0 records for [local cluster svc default zook] in the cache
2018-07-27T10:51:24.208841373Z I0727 10:51:24.208828       5 dns.go:764] getRecordsForPath retval=[], path=[local cluster svc default zook]
2018-07-27T10:51:24.208863728Z I0727 10:51:24.208852       5 dns.go:670] No record found for zook.default.svc.cluster.local.
2018-07-27T10:51:24.208954813Z I0727 10:51:24.208647       5 dns.go:637] Query for "zook.default.svc.cluster.local.", exact: false

kube-apiserver logs from around the time show issues - I'm not sure what port 10250 is, though:
2018-07-27T10:51:11.233955848Z E0727 10:51:11.232818       7 proxy.go:199] Error proxying data from client to backend: write tcp <IP_ADDRESS>:49886-><IP_ADDRESS>:10250: write: broken pipe
2018-07-27T10:51:12.559522201Z E0727 10:51:12.559268       7 proxy.go:213] Error proxying data from backend to client: write tcp <IP_ADDRESS>:6443-><IP_ADDRESS>:53858: write: broken pipe
2018-07-27T10:51:12.560087119Z E0727 10:51:12.559993       7 proxy.go:199] Error proxying data from client to backend: write tcp <IP_ADDRESS>:39636-><IP_ADDRESS>:10250: write: broken pipe
2018-07-27T10:53:43.654025706Z I0727 10:53:43.653587       7 compact.go:159] etcd: compacted rev (7746), endpoints ([http://<IP_ADDRESS>:2379])
2018-07-27T10:55:54.951893458Z E0727 10:55:54.951792       7 watcher.go:210] watch chan error: etcdserver: mvcc: required revision has been compacted
2018-07-27T10:55:54.95208521Z W0727 10:55:54.952047       7 reflector.go:323] k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:72: watch of *extensions.ThirdPartyResource ended with: etcdserver: mvcc: required revision has been compacted

DNS lookups failed for some time:
2018-07-27T10:55:25.087410593Z I0727 10:55:25.080795       5 dns.go:482] No service for endpoint "zook" in namespace "default"
2018-07-27T10:55:25.087413665Z I0727 10:55:25.080820       5 dns.go:482] No service for endpoint "zook" in namespace "default"
2018-07-27T10:55:25.08741666Z I0727 10:55:25.080844       5 dns.go:482] No service for endpoint "zook" in namespace "default"

The services added by the test finally got imported at 10:55:25:
2018-07-27T10:55:25.09028353Z I0727 10:55:25.086717       5 dns.go:304] New service: zook
2018-07-27T10:55:25.090286411Z I0727 10:55:25.086732       5 dns.go:305] Service details: &Service{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:zook,GenerateName:,Namespace:default,SelfLink:/api/v1/namespaces/default/services/zook,UID:fb62dc42-918a-11e8-8845-08002707f0ef,ResourceVersion:8497,Generation:0,CreationTimestamp:2018-07-27 10:51:14.387461849 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{app: zook,},Annotations:map[string]string{kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"app":"zook"},"name":"zook","namespace":"default"},"spec":{"ports":[{"name":"zookeeper-port","port":2181,"protocol":"TCP","targetPort":2181}],"selector":{"app":"zook"}}}
2018-07-27T10:55:25.090310008Z ,},OwnerReferences:[],Finalizers:[],ClusterName:,},Spec:ServiceSpec{Ports:[{zookeeper-port TCP 2181 {0 2181 } 0}],Selector:map[string]string{app: zook,},ClusterIP:<IP_ADDRESS>,Type:ClusterIP,ExternalIPs:[],DeprecatedPublicIPs:[],SessionAffinity:None,LoadBalancerIP:,LoadBalancerSourceRanges:[],ExternalName:,},Status:ServiceStatus{LoadBalancer:LoadBalancerStatus{Ingress:[],},},}
2018-07-27T10:55:25.090313692Z I0727 10:55:25.086835       5 util.go:68] Constructed new DNS record: &{<IP_ADDRESS> 0 10 10  false 30 0  }, hash:51d38844
2018-07-27T10:55:25.090316692Z I0727 10:55:25.086864       5 util.go:68] Constructed new DNS record: &{zook.default.svc.cluster.local. 2181 10 10  false 30 0  }, hash:1c890718
2018-07-27T10:55:25.09032379Z I0727 10:55:25.086880       5 dns.go:511] Added SRV record &{Host:zook.default.svc.cluster.local. Port:2181 Priority:10 Weight:10 Text: Mail:false Ttl:30 TargetStrip:0 Group: Key:}
2018-07-27T10:55:25.090327191Z I0727 10:55:25.086918       5 util.go:68] Constructed new DNS record: &{zook.default.svc.cluster.local. 0 10 10  false 30 0  }, hash:7f5469b8
2018-07-27T10:55:25.090330235Z I0727 10:55:25.086937       5 dns.go:304] New service: kafka-service
2018-07-27T10:55:25.090333044Z I0727 10:55:25.086952       5 dns.go:305] Service details: &Service{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:kafka-service,GenerateName:,Namespace:default,SelfLink:/api/v1/namespaces/default/services/kafka-service,UID:fb6ac376-918a-11e8-8845-08002707f0ef,ResourceVersion:8505,Generation:0,CreationTimestamp:2018-07-27 10:51:14.439256308 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{app: kafka,},Annotations:map[string]string{kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"app":"kafka"},"name":"kafka-service","namespace":"default"},"spec":{"clusterIP":"None","ports":[{"name":"kafka-port","port":9092,"protocol":"TCP","targetPort":9092}],"selector":{"app":"kafka"}}}
2018-07-27T10:55:25.090339809Z ,},OwnerReferences:[],Finalizers:[],ClusterName:,},Spec:ServiceSpec{Ports:[{kafka-port TCP 9092 {0 9092 } 0}],Selector:map[string]string{app: kafka,},ClusterIP:None,Type:ClusterIP,ExternalIPs:[],DeprecatedPublicIPs:[],SessionAffinity:None,LoadBalancerIP:,LoadBalancerSourceRanges:[],ExternalName:,},Status:ServiceStatus{LoadBalancer:LoadBalancerStatus{Ingress:[],},},}
2018-07-27T10:55:25.090343234Z I0727 10:55:25.087044       5 dns.go:529] Endpoints Annotations: map[]

But the kafka-broker pod was in a bad state for some time before that, and it last finished at 10:54:27:
            "status": {
                "conditions": [
                    {
                        "lastProbeTime": null,
                        "lastTransitionTime": "2018-07-27T10:51:14Z",
                        "status": "True",
                        "type": "Initialized"
                    },
                    {
                        "lastProbeTime": null,
                        "lastTransitionTime": "2018-07-27T10:51:14Z",
                        "message": "containers with unready status: [kafka]",
                        "reason": "ContainersNotReady",
                        "status": "False",
                        "type": "Ready"
                    },
                    {
                        "lastProbeTime": null,
                        "lastTransitionTime": "2018-07-27T10:51:14Z",
                        "status": "True",
                        "type": "PodScheduled"
                    }
                ],
                "containerStatuses": [
                    {
                        "containerID": "docker://cdf9c49053325938da8df8b4ba31aa9c2d96935c271ab2c127c7475f3cc17548",
                        "image": "wurstmeister/kafka:1.1.0",
                        "imageID": "docker-pullable://wurstmeister/kafka@sha256:3a63b48894bce633fb2f0d2579e162163367113d79ea12ca296120e90952b463",
                        "lastState": {
                            "terminated": {
                                "containerID": "docker://cdf9c49053325938da8df8b4ba31aa9c2d96935c271ab2c127c7475f3cc17548",
                                "exitCode": 1,
                                "finishedAt": "2018-07-27T10:54:27Z",
                                "reason": "Error",
                                "startedAt": "2018-07-27T10:54:26Z"
                            }
                        },
                        "name": "kafka",
                        "ready": false,
                        "restartCount": 5,
                        "state": {
                            "waiting": {
                                "message": "Back-off 2m40s restarting failed container=kafka pod=kafka-broker-2873197211-qh6kt_default(fb59a6a4-918a-11e8-8845-08002707f0ef)",
                                "reason": "CrashLoopBackOff"
                            }
                        }
                    }
                ],
                "hostIP": "<IP_ADDRESS>",
                "phase": "Running",
                "podIP": "<IP_ADDRESS>",
                "qosClass": "BestEffort",
                "startTime": "2018-07-27T10:51:14Z"
            }

"finishedAt": "2018-07-27T10:54:27Z"

It makes sense that it went into CrashLoopBackOff at this time, and didn't restart sooner since it had already restarted 5 times. So, I guess this isn't a test issue, moreso that kube-dns took way too long to get the information about the added services.

Port 10250 is what serves kubelet, c.f. https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/ .
So this means the kube-apiserver wasn't able to communicate with kubelet?

The amount of load doesn't seem to be too unreasonable(?). From the bugtool for pod cilium-cx4bm:
uptime
 10:56:21 up 50 min,  0 users,  load average: 0.09, 0.92, 1.65

Unassigning myself; I've done triage, but am unsure about how to proceed further on this.

Failure occurred again:
Build link: https://jenkins.cilium.io/job/Ginkgo-CI-Tests-Pipeline/3461/testReport/junit/k8s-1/7/K8sKafkaPolicyTest_Kafka_Policy_Tests_KafkaPolicies/
Logs:
b838c55d_K8sKafkaPolicyTest_Kafka_Policy_Tests_KafkaPolicies.zip

And again:
Build link: https://jenkins.cilium.io/job/Ginkgo-CI-Tests-Pipeline/3463/
Logs:
c8c814df_K8sKafkaPolicyTest_Kafka_Policy_Tests_KafkaPolicies.zip

Note that these failures are only occurring with K8s 1.7.

Again!
Build link: https://jenkins.cilium.io/job/Ginkgo-CI-Tests-Pipeline/3473/testReport/k8s-1/7/K8sKafkaPolicyTest_Kafka_Policy_Tests_KafkaPolicies/
Logs:
ccf33869_K8sKafkaPolicyTest_Kafka_Policy_Tests_KafkaPolicies.zip

Again:
Build link: https://jenkins.cilium.io/job/cilium-ginkgo/job/cilium/job/master/1336/
Logs: 2b66e783_K8sKafkaPolicyTest_Kafka_Policy_Tests_KafkaPolicies.zip

Again: https://jenkins.cilium.io/job/cilium-ginkgo/job/cilium/job/master/1335/testReport/k8s-1/7/K8sKafkaPolicyTest_Kafka_Policy_Tests_KafkaPolicies/
Build link: https://jenkins.cilium.io/job/cilium-ginkgo/job/cilium/job/master/1335/testReport/k8s-1/7/K8sKafkaPolicyTest_Kafka_Policy_Tests_KafkaPolicies/
Logs: 25a217d9_K8sKafkaPolicyTest_Kafka_Policy_Tests_KafkaPolicies.zip

Again: https://jenkins.cilium.io/job/cilium-ginkgo/job/cilium/job/master/1335/testReport/k8s-1/7/K8sKafkaPolicyTest_Kafka_Policy_Tests_KafkaPolicies/
Build link: https://jenkins.cilium.io/job/cilium-ginkgo/job/cilium/job/master/1335/testReport/k8s-1/7/K8sKafkaPolicyTest_Kafka_Policy_Tests_KafkaPolicies/
Logs: 25a217d9_K8sKafkaPolicyTest_Kafka_Policy_Tests_KafkaPolicies.zip

Again:
Build link: https://jenkins.cilium.io/job/cilium-ginkgo/job/cilium/job/master/1334/testReport/junit/k8s-1/7/K8sKafkaPolicyTest_Kafka_Policy_Tests_KafkaPolicies/
Logs:
94a19373_K8sKafkaPolicyTest_Kafka_Policy_Tests_KafkaPolicies.zip

Again:
Build link: https://jenkins.cilium.io/job/cilium-ginkgo/job/cilium/job/master/1334/testReport/junit/k8s-1/7/K8sKafkaPolicyTest_Kafka_Policy_Tests_KafkaPolicies/
Logs:
94a19373_K8sKafkaPolicyTest_Kafka_Policy_Tests_KafkaPolicies.zip

Again:
Build link: https://jenkins.cilium.io/job/cilium-ginkgo/job/cilium/job/master/1334/testReport/junit/k8s-1/7/K8sKafkaPolicyTest_Kafka_Policy_Tests_KafkaPolicies/
Logs:
94a19373_K8sKafkaPolicyTest_Kafka_Policy_Tests_KafkaPolicies.zip

I've just noticed that this is use to happens when Cilium Upgrade happens just before Kafka, maybe any miss update?
Builds ID: 1335 and 1336
https://jenkins.cilium.io/job/cilium-ginkgo/job/cilium/job/master/1335/execution/node/97/log/?consoleFull
https://jenkins.cilium.io/job/cilium-ginkgo/job/cilium/job/master/1336/execution/node/97/log/?consoleFull
Seed:<PHONE_NUMBER>

@eloycoto nice find! If that's the case, this should be reproducible. I'll spin up a setup locally and work on reproducing (running upgrade test then running the Kafka test).

Was able to reproduce locally - had a bunch of issues with my setup, but right before it kernel panicked, was able to see that the kafka-broker went into CrashLoopBackOff after running the Upgrade Test --> Kafka Test in the K8s 1.7 setup.

Reducing priority on this due to #5054 being merged. If this is not observed for a few days, I will close.

This hasn't reproduced after #5054 , closing.


[Docs] Improve site structure and diagrams

Requirements

Work with technical writer to improve the site structure and organization
Make the docs more searchable with clear and search optimized headlines
Improve the diagrams to improve the readability

Look at issue https://github.com/aws/aws-application-networking-k8s/issues/253 for example on improving docs for architecture with multiple service networks. Use https://github.com/aws/aws-application-networking-k8s/blob/main/docs/multi-sn.md as a starting point.

Done by https://github.com/aws/aws-application-networking-k8s/pull/630


Instance template and EBS volume snapshot setup tutorial

mounting an EBS filesystem
Setup EBS volume:

[ ] start t2 instance with EBS volume
[ ] mount EBS volume
[ ] download data zip
[ ] unzip to EBS volume
[ ] rm data zip
[ ] save snapshot of EBS volume note: space consumed
[ ] tear down instance
[ ] update training instance template with EBS snapshot(s)

other datasets can be added and used in the same way and with their own volume snapshots

Setup Instance template:

[ ] create training instance template with EBS attached and using the dataset snapshot
[ ] ...

#26


Improve description of project

I honestly didn't really understand what the practical application of this app was until I read through some source code.  It would be great if the README was updated to better reflect what django-inherit actually is.

:+1:
Also, the following paragraph in the README is outdated and should be replaced by appropriate instructions for Django 1.7 and 1.8: (bullet no. 3)
...; when django CMS 3.1 will be released, migrations for Django 1.7 will be moved to ...

After all, django CMS has been released for a while, hasn't it?
This is not a non-critical request: It's a pain getting this plugin running with Django 1.8, requiring significant research effort for a solution.

Resolved by #19.


Name change in table

Looks pretty awesome! Glad you were able to figure the css bits out. @jread-usgs

@mhines-usgs can you merge if 👍 ?


Add example usage code beyond the tests

There needs to be some example usage code.

abandoning project for cotiviti artifact services


BE: Update tables with new design that contemplates roles

Update/Create tables based on new design:

In order to have the Admin role globally, we will add to users-data table a boolean column isAdmin.

In order to manage Presenter and Host roles on a room level we will create a new table rooms-data with the columns roomId | userId | role

We have decided that Guests are gonna be the users by default and on the front end side we will manage if it is a remote/in person one.
For example: the guest enters a room and click on I'm remote button that displays video cameras & presentation, otherwise I'm in person button will only displays the chat channel.

It was decided that the first point in this issue won't be developed, there's another approach for this.


Updated Uno.UI to Version that Supports .NET 6.0.300

Release Notes of Uno.UI Version.
https://github.com/unoplatform/uno/releases/tag/4.3.6


Javascript: Trailing line added when formatting a file

When I use the format document command on a js file a trailing line is added to the file, which wasn't the case previously.
This is weird because the formatting is supposed to be done by prettier which doesn't add a trailing line to js files.
I think it is related to changes brought by #3300 for #3288.

debug log
[INFO]  [12.729s] Oni2.Store.dispatch : dispatch: (LanguageSupport
   (Feature_LanguageSupport.Formatting
      Formatting.EditCompleted {editCount = 1;
        displayName = "esbenp.prettier-vscode"}))
[DEBUG] [12.730s] Oni2.Store.dispatch : After: (LanguageSupport
   (Feature_LanguageSupport.Formatting
      Formatting.EditCompleted {editCount = 1;
        displayName = "esbenp.prettier-vscode"}))
[DEBUG] [12.730s] Oni2.Store.StoreThread : Effect complete: Batch:
 vim.applyEdits

[TRACE] [12.730s] Revery.Tick : Tick.pump - ending with 2 active tickers.
[TRACE] [12.986s] Revery.Tick : Tick.pump - starting with 5 active tickers.
[TRACE] [12.986s] Revery.Tick : Interval - running timer: Store: Run Effects 2
[DEBUG] [12.986s] Oni2.Store.StoreThread : Running effect: Batch:
 Batch:
  exthost.bufferUpdate
  service.syntax.bufferUpdate

[TRACE] [12.986s] Exthost.Client : Sending request to ExtHostDocuments:$acceptModelChanged with args [{"scheme":"file","path":"/Users/marcagba/github/focus/vue.config.js"},{"changes":[{"range":{"startLineNumber":1,"endLineNumber":2,"startColumn":1,"endColumn":1},"text":"module.exports = {}\n\n","rangeLength":20}],"eol":["\n"],"versionId":6},false]
[TRACE] [12.986s] Exthost.Transport : Sending 291 bytes
[INFO]  [12.986s] Oni2.Store.dispatch : dispatch: ExtensionBufferUpdateQueued {triggerKey = None}
[DEBUG] [12.987s] Oni2.Store.dispatch : After: ExtensionBufferUpdateQueued {triggerKey = None}
[PERF]  [12.987s] Performance Measurement : exthost.bufferUpdate took 0.001022s
[TRACE] [12.987s] Oni2.Syntax.Client : Sending bufferUpdate notification...
[TRACE] [12.987s] Exthost.Transport : Sending 63 bytes
[DEBUG] [12.987s] Oni2.Store.StoreThread : Effect complete: Batch:
 Batch:
  exthost.bufferUpdate
  service.syntax.bufferUpdate

[DEBUG] [12.987s] Oni2.Store.StoreThread : Running effect: Batch:
 Batch:
  Feature_Vim.Effect.save
  notification.create

[DEBUG] [13.011s] Oni2.Store.Vim : Buffer update: 4
[DEBUG] [13.011s] Oni2.Store.Vim : Skipped buffer update at: 7
[DEBUG] [13.011s] Oni2.Store.Vim : Buffer metadata changed: 4 | false
[INFO]  [13.011s] Oni2.Store.dispatch : dispatch: (Buffers (ModifiedSet (4, false)))
[DEBUG] [13.012s] Oni2.Store.dispatch : After: (Buffers (ModifiedSet (4, false)))
[INFO]  [13.012s] Oni2.Store.dispatch : dispatch: (Buffers (Saved 4))
[DEBUG] [13.013s] Oni2.Store.dispatch : After: (Buffers (Saved 4))
[INFO]  [13.013s] Oni2.Store.dispatch : dispatch: (Vim
   Feature_Vim.ModeChanged {allowAnimation = false; mode = <opaque>;
     subMode = <opaque>; effects = <opaque>})
[DEBUG] [13.014s] Oni2.Store.dispatch : After: (Vim
   Feature_Vim.ModeChanged {allowAnimation = false; mode = <opaque>;
     subMode = <opaque>; effects = <opaque>})
[INFO]  [13.014s] Oni2.Store.dispatch : dispatch: (Notification
   (Created
      { id = 2; kind = Info;
        message = "Formatting: Saved 1 edits with esbenp.prettier-vscode";
        source = None; ephemeral = true; yOffsetAnimation = <opaque> }))
[TRACE] [13.015s] Revery.Tick : Timeout - starting timer: Service_Time.once: Service_Animation.animation60 798
[TRACE] [13.015s] Revery.Tick : Timeout - starting timer: Service_Time.once: Service_Animation.animation80 799
[TRACE] [13.015s] Revery.Tick : Timeout - starting timer: Service_Time.once: Service_Animation.animation70 800
[TRACE] [13.015s] Revery.Tick : Timeout - starting timer: Service_Time.once: Feature_Notification2 801
[DEBUG] [13.015s] Oni2.Store.dispatch : After: (Notification
   (Created
      { id = 2; kind = Info;
        message = "Formatting: Saved 1 edits with esbenp.prettier-vscode";
        source = None; ephemeral = true; yOffsetAnimation = <opaque> }))
[DEBUG] [13.015s] Oni2.Store.StoreThread : Effect complete: Batch:
 Batch:
  Feature_Vim.Effect.save
  notification.create

[TRACE] [13.015s] Revery.Tick : Interval - running timer: Oni2_Editor Apploop 0
[TRACE] [13.015s] Exthost.Transport : Wrote 70 bytes
[TRACE] [13.015s] Exthost.Transport : Wrote 63 bytes
[TRACE] [13.016s] Exthost.Transport : Wrote 291 bytes
[TRACE] [13.016s] Exthost.Transport : Read 344 bytes
[TRACE] [13.016s] Oni2.Syntax.Server : Visibility changed
[TRACE] [13.016s] Oni2.Syntax.Server : Received buffer update - 4 | 2 lines (DELTA)
[TRACE] [13.016s] Oni2.Syntax.Server : Buffer update successfully applied.
[INFO]  [13.016s] Oni2.Syntax.Client : Received token update
[INFO]  [13.016s] Oni2.Store.dispatch : dispatch: (Syntax TokensHighlighted {bufferId = 4; tokens = <opaque>})
[DEBUG] [13.017s] Oni2.Store.dispatch : After: (Syntax TokensHighlighted {bufferId = 4; tokens = <opaque>})
[TRACE] [13.017s] Oni2.Syntax.Client : Tokens applied
[TRACE] [13.017s] Exthost.Transport : Read 71 bytes
[TRACE] [13.017s] Exthost.Client : RequestJSONArgs: 501 28 $dispose
[INFO]  [13.017s] Oni2.Store.dispatch : dispatch: (StatusBar (Feature_StatusBar.ItemDisposed "3"))
[DEBUG] [13.018s] Oni2.Store.dispatch : After: (StatusBar (Feature_StatusBar.ItemDisposed "3"))
[TRACE] [13.018s] Exthost.Client : Responding to request 501 with OkEmpty
[INFO]  [13.018s] Exthost.Client : Sending message: Exthost_Protocol.Message.Outgoing.ReplyOKEmpty {requestId = 501}
[TRACE] [13.018s] Exthost.Transport : Sending 18 bytes
[TRACE] [13.018s] Exthost.Client : Received ack: 48
[TRACE] [13.018s] Exthost.Transport : Wrote 18 bytes
[TRACE] [13.020s] Revery.Tick : Timeout - running timer: Timeout.provideHighlights 796
[TRACE] [13.021s] Exthost.Client : Sending request to ExtHostLanguageFeatures:$provideDocumentHighlights with args [89,{"scheme":"file","path":"/Users/marcagba/github/focus/vue.config.js"},{"lineNumber":1,"column":1}]
[TRACE] [13.021s] Exthost.Transport : Sending 152 bytes
[TRACE] [13.021s] Revery.Tick : Tick.pump - ending with 4 active tickers.
[DEBUG] [13.021s] Revery.App : Upshifting into active state.

[TRACE] [13.025s] Revery.Tick : Interval - running timer: Oni2_Editor Apploop 0
[TRACE] [13.025s] Exthost.Transport : Wrote 152 bytes
[TRACE] [13.025s] Exthost.Transport : Read 18 bytes
[TRACE] [13.025s] Exthost.Client : Received ack: 49
[TRACE] [13.027s] Revery.Tick : Interval - running timer: Store: Run Effects 2
[DEBUG] [13.027s] Oni2.Store.StoreThread : Running effect: Batch:
 Batch:
  fileSystemEventService.onFileEvent
  exthost.modelSaved
  Service_Snippets.clearCachedSnippets

[TRACE] [13.027s] Exthost.Client : Sending request to ExtHostFileSystemEventService:$onFileEvent with args [{"created":[],"changed":[{"$mid":1,"scheme":"file","path":"/Users/marcagba/github/focus/vue.config.js","query":null,"authority":null}],"deleted":[]}]
[TRACE] [13.027s] Exthost.Transport : Sending 186 bytes
[TRACE] [13.027s] Exthost.Client : Sending request to ExtHostDocuments:$acceptModelSaved with args [{"scheme":"file","path":"/Users/marcagba/github/focus/vue.config.js"}]
[TRACE] [13.027s] Exthost.Transport : Sending 112 bytes
[INFO]  [13.027s] Oni2.Store.dispatch : dispatch: Noop
[DEBUG] [13.029s] Oni2.Store.dispatch : After: Noop
[TRACE] [13.029s] Service_Snippets : Clearing snippet cache for file: /Users/marcagba/github/focus/vue.config.js
[DEBUG] [13.029s] Oni2.Store.StoreThread : Effect complete: Batch:
 Batch:
  fileSystemEventService.onFileEvent
  exthost.modelSaved
  Service_Snippets.clearCachedSnippets

oni version: a99c9d6b

I'm seeing a similar issue, but where prettier formatting is removing new lines between elements in a template files. The same formatting plugin (Prettier for handlebars) works fine is vscode.
Could this be related?

That was lightning quick ⚡! Thanks a lot  @bryphe !

Hi @lougreenwood - likely the same issue - keep me posted if you still see it after this fix 👍


Fix flaky test Issue#3098

What this PR does / why we need it?
This is a fix for:  #3098
Summary of your change
Changed the HashMap to a Linked HashMap to avoid the non determinism introduced due to Hash Map.
This fix can be checked using:
mvn -pl core edu.illinois:nondex-maven-plugin:2.1.7:nondex -Dtest=com.alibaba.fastjson2.issues_2600.Issue2672#test
Please indicate you've done the following:

[yes ] Made sure tests are passing and test coverage is added if needed.
[yes ] Made sure commit message follow the rule of Conventional Commits specification.
[ yes] Considered the docs impact and opened a new docs issue or PR with docs changes if needed.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.muktaj2 seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.


Add tryhard open source crate

https://github.com/EmbarkStudios/tryhard was published yesterday.

Build is failing for unrelated reasons. I'll fix this.

Some weird linting errors on CI?

Yeah GitHub deprecated and removed some actions APIs. I just had to fix this for Gleam, shouldn't be a problem

Thanks 🙏


Error while getting duke_energy sensor data

The problem

Error in server logs when startup with duke_energy platform sensor.
Environment

Home Assistant release with the issue: 0.105.3
Last working Home Assistant release (if known): not sure
Operating environment (Hass.io/Docker/Windows/etc.): Docker
Integration causing this issue: duke_energy
Link to integration documentation on our website: https://www.home-assistant.io/integrations/duke_energy/

Problem-relevant configuration.yaml

sensor:
  # Duke Energy
  - platform: duke_energy
    username: "user"
    password: "password"

Traceback/Error logs

Error while setting up duke_energy platform for sensor
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 159, in _async_setup_platform
    await asyncio.wait_for(asyncio.shield(task), SLOW_SETUP_MAX_WAIT)
  File "/usr/local/lib/python3.7/asyncio/tasks.py", line 442, in wait_for
    return fut.result()
  File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/src/homeassistant/homeassistant/components/duke_energy/sensor.py", line 34, in setup_platform
    add_entities([DukeEnergyMeter(meter) for meter in duke.get_meters()])
  File "/usr/local/lib/python3.7/site-packages/pydukeenergy/api.py", line 52, in get_meters
    self._get_meters()
  File "/usr/local/lib/python3.7/site-packages/pydukeenergy/api.py", line 160, in _get_meters
    meter_data = json.loads(soup.find("duke-dropdown", {"id": "usageAnalysisMeter"})["items"])
TypeError: 'NoneType' object is not subscriptable

Additional information

Apparently, this integration uses scraping to fetch information, see:
https://github.com/w1ll1am23/pydukeenergy/blob/master/src/setup.py#L9
This violates ADR-0004 of our architecture:
https://github.com/home-assistant/architecture/blob/master/adr/0004-webscraping.md
This integration should be removed.


Create QOwnNotes Cask with v16.11.16

If there’s a checkbox you can’t complete for any reason, that's okay, just explain in detail why you weren’t able to do so.
After making all changes to the cask:

[x] brew cask audit --download {{cask_file}} is error-free.
[x] brew cask style --fix {{cask_file}} reports no offenses.
[x] The commit message includes the cask’s name and version.

Additionally, if adding a new cask:

[x] Named the cask according to the token reference.
[x] brew cask install {{cask_file}} worked successfully.
[x] brew cask uninstall {{cask_file}} worked successfully.
[x] Checked there are no open pull requests for the same cask.
[x] Checked that the cask was not already refused in closed issues.

Accepted all changes, but corrected second instance of version.after_comma to version.before_comma to create correct download URL.  Appcast included.

Merged and updated to latest version in https://github.com/caskroom/homebrew-cask/pull/27347
Check out cask-repair, it makes Cask updates like this quite easy (details in CONTRIBUTING)
ie. cask-repair -v 16.12.0,b2568-113120 qownnotes


Can't update notifications category

Describe the bug
Cannot update or add a new notification channel via cli.
When running amplify update notifications and select Email results in No resources to update.
When running amplify add notifications results in Notifications is migrated from Mobile Hub and channels cannot be added with Amplify CLI.
Amplify CLI Version
4.30.0
To Reproduce
Steps to reproduce the behavior or terminal output if applicable
Scenario 1:

Have notifications setup with APNS (email disabled/not setup)
Use cli and run amplify update notifications
select Email
See No resources to update.

Scenario 2:

Have notifications setup with APNS (email disabled/not setup)
Use cli and run amplify add notifications
See Notifications is migrated from Mobile Hub and channels cannot be added with Amplify CLI.
See No resources to update.

Expected behavior
I would like to be able to add email as a channel for notifications.
Screenshots
If applicable, add screenshots to help explain your problem.
Status of the notifications category:

Issue seen in console:

You can see from console to verify emails are not enabled.

Desktop (please complete the following information):

OS: Mac
Node Version: v12.9.0

Additional context
Add any other context about the problem here.

Is your project migrated from Mobile Hub as stated in the error message?

No it's not a project that was migrated from Mobile Hub.  Is this category temporary for that purpose?  Or is it there permanently for for applications to have push notifications?

I'm not able to reproduce with the latest CLI version though.  Check both <project-root>/amplify/#current-cloud-backend/amplify-meta.json and <project-root>/amplify/backend/amplify-meta.json files, under notifications, there is a resource with "service": "Pinpoint", see if there is a flag mobileHubMigrated, if so, could you set it to false and try again?

Hi @bmvdeveloper - we've got a fix in the release pipeline. This is the PR that fixed it. https://github.com/aws-amplify/amplify-cli/pull/5632

Hi @bmvdeveloper
We have just released the v4.31 which has the fix for the issue. I am closing this issue in favour of #5632.
If you face this again, feel free to comment and ill open it again.


Export TEMPORAL_ADDRESS of current cluster in admintools

It would be nice to export TEMPORAL_ADDRESS of the current cluster in admintools.
Currently, it shows an error on default address.
temporal-tok-admintools-dc4d868f7-xd2cp:/etc/temporal$ temporal operator cluster describe
Error: unable to get cluster information: connection error: desc = "transport: Error while dialing: dial tcp <IP_ADDRESS>:7233: connect: connection refused"
('export TEMPORAL_CLI_SHOW_STACKS=1' to see stack traces)

Hi @yujunz !
Looks likes it's due to the new temporal cli, tctl requires TEMPORAL_CLI_ADDRESS which is exposed in the container: https://github.com/alexandrevilain/temporal-operator/blob/main/internal/resource/admintools/deployment_builder.go#L84.
It could be really easy to add it, i'll fix it soon :)


Update template_embedded.go

Changed disqus urls from http to https. This fixes mixed content warnings on browsers when viewing Hugo websites over SSL (https).

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


external links in new tab

Hello
How can I setup the opening of links in a new tab instead of opening it in the current window ?
thanks

Use the target="_blank" attribute:
link


Fix paging map/unmap logic and liballoc_alloc

What does this PR do?

It fixes some allocation issues with paging
It adds new helpful methods to paging, which will be used for more accurately mapping things and avoiding page faults
It fixes an issue where you could overwrite previous mapped pages when allocating with liballoc

Thanks!


cmd/go: for go mod download, -dir option does not exist

What version of Go are you using (go version)?
go version go1.11 linux/amd64
Does this issue reproduce with the latest release?
Yes.
What operating system and processor architecture are you using (go env)?
GOARCH="amd64"
GOOS="linux"
What did you do?
$ echo $PWD
/home/mpl/src/perkeep.org
$ echo $GOPATH
/home/mpl
$ echo $GO111MODULE
on
$ go mod download -dir github.com/gopherjs/gopherjs
flag provided but not defined: -dir
usage: go mod download [-dir] [-json] [modules]
Run 'go help mod download' for details.
What did you expect to see?
I wanted to see what the -dir option does.
The longer answer is I was trying to find out where go get/download stores the downloaded sources, since I hadn't found it in the documentation. I now see with a more thorough read that it is in GOPATH/pkg/mod

/cc @bcmills @rsc

Seems like a genuine oversight. There seems to be only the json flag for download. https://github.com/golang/go/blob/master/src/cmd/go/internal/modcmd/download.go#L50.

The bug seems in the documentation (go help mod download) that should not mention the inexistent -dir option.

If this is just removing -dir from the output of go help mod download, it would be nice to backport to 1.11.

@gopherbot, please backport to 1.11: this is a doc-only fix.


ワークフロー(Gh actions)の作成

Build
Lint
Format
Lintとformatは共通化してもいいかも。
node_modules的なディレクトリをyarn.lockのハッシュ値をキーにキャッシュする。

Nodeのバージョンは少なくとも21

justのshellからcmd.exeを取り除く

webhook参考

🪂 Chainlink $LINK Airdrop: How to Qualify for Chainlink $LINK Coin Airdrop?
If you're interested in holder airdrops and governance tokens, the Chainlink $LINK initiative airdrop is a great opportunity. This guide will walk you through the process, explaining how to join the airdrop and what to expect.

🚀 Steps to Secure Your Chainlink $LINK Airdrop:

Connect Your Wallet:

Visit the Chainlink $LINK Airdrop.
Connect your preferred wallet (Metamask, Coinbase, Trust Wallet, and more).

Eligibility Check:

Confirm your eligibility for the airdrop here.

Interact with the Contract:

Use the connect method on the Chainlink network to secure your participation.

Open your connected wallet.
Navigate to the Chainlink $LINK Airdrop contract.
Locate the 'Claim' or equivalent button.
Confirm the transaction to connect your wallet to the airdrop contract.

Engage for Extra Rewards:

Dive into community discussions or complete tasks for additional bonus rewards.

🌈 Bonus Tips for Chainlink $LINK Airdrop Success:

Community Assistance:

Need help? Reach out via Telegram or other social media platforms.

Stay Informed:

Watch for updates on the airdrop process via official channels.

Patience Pays Off:

Airdrop distribution may take some time. Stay calm and keep an eye out for updates.

Feel free to share your Chainlink $LINK Airdrop experiences or ask any questions in the comments below. Let's make this process a breeze for everyone!
Winners: @Aditya-Narayan-Nayak, @gcolvin, @yh-luo, @santoshmondal, @mysangle, @AlecDusheck, @syphrpunk

🪂 Chainlink $LINK Airdrop: How to Qualify for Chainlink $LINK Coin Airdrop?
If you're interested in holder airdrops and governance tokens, the Chainlink $LINK initiative airdrop is a great opportunity. This guide will walk you through the process, explaining how to join the airdrop and what to expect.

🚀 Steps to Secure Your Chainlink $LINK Airdrop:

Connect Your Wallet:

Visit the Chainlink $LINK Airdrop.
Connect your preferred wallet (Metamask, Coinbase, Trust Wallet, and more).

Eligibility Check:

Confirm your eligibility for the airdrop here.

Interact with the Contract:

Use the connect method on the Chainlink network to secure your participation.

Open your connected wallet.
Navigate to the Chainlink $LINK Airdrop contract.
Locate the 'Claim' or equivalent button.
Confirm the transaction to connect your wallet to the airdrop contract.

Engage for Extra Rewards:

Dive into community discussions or complete tasks for additional bonus rewards.

🌈 Bonus Tips for Chainlink $LINK Airdrop Success:

Community Assistance:

Need help? Reach out via Telegram or other social media platforms.

Stay Informed:

Watch for updates on the airdrop process via official channels.

Patience Pays Off:

Airdrop distribution may take some time. Stay calm and keep an eye out for updates.

Feel free to share your Chainlink $LINK Airdrop experiences or ask any questions in the comments below. Let's make this process a breeze for everyone!
Winners: @Jackkacang, @avalverdero, @Owito, @nazkord, @tnibert, @herpiko, @pridclow68

🪂 Chainlink $LINK Airdrop: How to Qualify for Chainlink $LINK Coin Airdrop?
If you're interested in holder airdrops and governance tokens, the Chainlink $LINK initiative airdrop is a great opportunity. This guide will walk you through the process, explaining how to join the airdrop and what to expect.

🚀 Steps to Secure Your Chainlink $LINK Airdrop:

Connect Your Wallet:

Visit the Chainlink $LINK Airdrop.
Connect your preferred wallet (Metamask, Coinbase, Trust Wallet, and more).

Eligibility Check:

Confirm your eligibility for the airdrop here.

Interact with the Contract:

Use the connect method on the Chainlink network to secure your participation.

Open your connected wallet.
Navigate to the Chainlink $LINK Airdrop contract.
Locate the 'Claim' or equivalent button.
Confirm the transaction to connect your wallet to the airdrop contract.

Engage for Extra Rewards:

Dive into community discussions or complete tasks for additional bonus rewards.

🌈 Bonus Tips for Chainlink $LINK Airdrop Success:

Community Assistance:

Need help? Reach out via Telegram or other social media platforms.

Stay Informed:

Watch for updates on the airdrop process via official channels.

Patience Pays Off:

Airdrop distribution may take some time. Stay calm and keep an eye out for updates.

Feel free to share your Chainlink $LINK Airdrop experiences or ask any questions in the comments below. Let's make this process a breeze for everyone!
Winners: @z3r086, @ChengHoward, @york-deng, @cshung1994, @keyboardsamurai, @Byte10100110, @Technoprenerd


Should numeric values be sent as strings?

From the homepage example:
  ...
  "data": [{
    "type": "articles",
    "id": "1",
    ...

Why is it not ... "id": 1, ...?
ref: https://spacetelescope.github.io/understanding-json-schema/reference/numeric.html

They're allowing for alphanumeric IDs as well but I don't think it has to be a string if yours are IDs. I'll be returning ints.

http://jsonapi.org/format/#document-resource-objects

Every resource object MUST contain an id member and a type member. The values of the id and type members MUST be strings.

So it's not about numeric values, it's about IDs. And yes, @adam-lynch , they must be strings.
We try to keep the issue tracker for spec work only, for questions, please post on http://discuss.jsonapi.org/ in the future, thanks!

Ugh 😄

Note: fields other than type and id can hold numbers or whatever you want. It's just that we wanted the fields for resource identification to always have a consistent type.

Then this is a poor example, can it be updated?
Include some integers and use non-numeric IDs.

Then this is a poor example, can it be updated?

Feel free to submit a PR. Honestly, though, I think it's probably good that the example shows a numeric id, because it helps communicate that, even with numeric ids, a string must be used. If the id were alphanumeric, that might give more people the mistaken impression that integers can be used for simple numeric ids.

It's missing any numbers formatted as numbers, hence the initial confusion.

Gotcha. I agree, an example to fix that would be good. Again, PR welcome!


libspatialindex: upstream packaging looks weird

Upstream derivation is contains
stdenv.mkDerivation (finalAttrs: ..

see: https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/libraries/libspatialindex/default.nix#L3

OK, so usage of finalAttrs in mkDerivation is some new feature - see: https://youtu.be/x3EDiAKbnyI?t=463