RFC - @auth directive improvements

RFC - @auth directive improvements
This document will outline designs for a number of new features relating to authentication & authorization within the GraphQL Transform. The goal of these features is too fill holes and introduce new mechanisms that make protecting your valuable information easier.
Proposal 1: Replace 'queries' and 'mutations' arguments with 'operations'

Merged by #1262

Currently an @auth directive like this:
type Task @model @auth(rules: [{allow: owner}]) {
    id: ID!
    title: String
    owner: String
}

causes these changes to the following resolvers:

Query.getTask - Returns the post only if the logged in user is the post's owner.
Query.listTasks - Filter items such that only owned posts are returned.
Mutation.createTask - If an owner is provided via $ctx.args.input.owner and matches the identity of the logged in user, succeed. If no owner is provided, set logged in user as the owner, else fail.
Mutation.updateTask - Append a conditional expression that will only update the record if the logged in user is its owner.
Mutation.deleteTask - Append a conditional expression that will only delete the record if the logged in user is its owner.

In other words, the @auth directive currently protects the root level query & mutation fields that are generated for an @model type.
Problem: The 'queries' and 'mutations' arguments imply top level protection
GraphQL APIs are a graph and we need to be able to define access rules on any field, not just the top level fields.
Solution
I suggest replacing the queries and mutations arguments on the @auth directive with a single operations argument. This would be the new @auth directive definition.
directive @auth(rules: [AuthRule!]!) on OBJECT
input AuthRule {
    allow: AuthStrategy!
    ownerField: String # defaults to "owner"
    identityField: String # defaults to "cognito:username"
    groupsField: String
    groups: [String]

    # The new argument
    operations: [ModelOperation]

    # Old arguments
    queries: [ModelQuery] @deprecated(reason: "The 'queries' argument will be deprecated in the future. Please replace this argument with the 'operations' argument.")
    mutations: [ModelMutation] @deprecated(reason: "The 'mutations' argument will be deprecated in the future. Please replace this argument with the 'mutations' argument.")
}
enum AuthStrategy { owner groups }

# The new enum
enum ModelOperation { create update delete read }

# The old enums
enum ModelQuery { get list }
enum ModelMutation { create update delete }

This change generalizes the config such that it implies all read operations on that model will be protected. Not just the top level 'get' & 'list' queries. Auth rules that use the 'read' operation will be applied to top level query fields, @connection resolvers, top level fields that query custom indexes, and subscription fields. Auth rules that use the 'create', 'update', and 'delete' operation will be applied to createX, updateX, and deleteX mutations respectively. Those using queries & mutations will have the same behavior and those using operations will get the new behavior. The queries & mutations arguments will eventually be removed in a future major release.
Protect @connections by default

Merged by #1262

Once the change from queries/mutations -> operations has been implemented, we will want to go back and implement any missing authorization logic in @connection fields by default.
For example, given this schema:
type Post @model @auth(rules: [{allow: owner}], operations: [create, update, delete, read]) {
    id: ID!
    title: String
    owner: String
}
type Blog @model {
    id: ID!
    title: String
    # This connection references type Post which has auth rules and thus should be authorized.
    posts: [Post] @connection
}

The new code would add authorization logic to the Blog.posts resolver such that only owner's of the post would be able to see the posts for a given blog. It is important to note that the new logic will restrict access such that you cannot see records that you are not supposed to see, but it will not change any index structures under the hood. You will be able to use @connection with the new custom index features to optimize the access pattern and then use @auth to protect access within that table or index.
Proposal 2: Implement @auth on @searchable search fields
Github Issues

https://github.com/aws-amplify/amplify-cli/issues/460

Problem
Currently Query.searchX resolvers generated by @searchable are not protected by @auth rules.
Solution
The Elasticsearch DSL is very powerful and will allow us inject Elasticsearch query terms and implement authorization checks within Elasticsearch. This work will need to handle static & dynamic ownership and group based authorization rules. Any auth rule that includes the 'read' operation will protect the Query.searchX field.
Proposal 3: Make @auth protect subscription fields
Problem: @auth does not protect subscription fields.
type Post @model @auth(rules: [{allow: owner}]) {
    id: ID!
    title: String
    owner: String
}

Currently subscriptions are not protected automatically.
Solution
AppSync subscription queries are authorized at connect time. That means that we need to parameterize the subscription queries in such a way that any relevant authorization logic is included in the subscription query itself. In the case of ownership @auth, this means that the client must pass an owner as a query argument and the subscription resolver should verify that the logged in user and owner are the same.
For example, given this schema:
type Post @model @auth(rules: [{allow: owner}]) {
    id: ID!
    title: String
    owner: String
}

The following subscription fields would be output:
type Subscription {
    onCreatePost(owner: String): Post
    onUpdatePost(owner: String): Post
    onDeletePost(owner: String): Post
}

and when running a subscription query, the client must provide a value for the owner:
subscription OnUpdatePost($owner: String) {
    onUpdatePost(owner: $owner) {
        id
        title
    }
}

The proposed change would create a new subscription resolver for each subscription field generated by the @model. Each subscription resolver would verify the provided owner matches the logged-in identity and would fail the subscription otherwise.
There are a few limitation to this approach:

There is a limit of 5 arguments per subscription field.

e.g. a field onCreatePost(owner: String, groups: String, otherOther: String, anotherOwner: String, anotherListOfGroups: String): Post has too many fields and is invalid. To handle this the CLI can emit a warning prompting you to customize your subscription field in the schema itself.

Subscription fields are equality checked against published objects. This means that subscribing to objects with with multi-owner or multi-group auth might behave slightly differently than expected.

When you subscribe you will need to pass the full list of owners/groups on the item. Not just the calling identity.

As an example to point (2) above, imagine this auth rule:
type Post @model @auth(rules: [{allow: owner, ownerField: "members"}]) {
    id: ID!
    title: String
    members: [String]
}

Let's say that we want to subscribe to all new posts where I am a member.
subscription {
    onCreatePost(members: ["my-user-id"]) {
        id
        title
        members
    }
}

AppSync messages are published to subscriptions when the result of the mutation, to which the subscription field is subscribed, contains fields that equal the values provided by the subscription arguments. That means that if I were to publish a message via a mutation,
mutation {
    createPost(input: { title: "New Article", members: ["my-user-id", "my-friends-user-id"]}) {
        id
        title
        members
    }
}

the subscription started before would not be triggered because ["my-user-id", "my-friends-user-id"] is not the same as  ["my-user-id"]. I bring this up for clarity but I still think this feature is useful. Single owner & group based authorization will behave as expected.
Proposal 4: Field level @auth

Merged by #1262

Currently an @auth directive like this:
type Task @model @auth(rules: [{allow: owner}], queries: [get, list], mutations: [create, update, delete]) {
    id: ID!
    title: String
    owner: String
}

causes these changes to the following resolvers:

Query.getTask - Returns the post only if the logged in user is the post's owner.
Query.listTasks - Filter items such that only owned posts are returned.
Mutation.createTask - If an owner is provided via $ctx.args.input.owner and matches the identity of the logged in user, succeed. If no owner is provided, set logged in user as the owner, else fail.
Mutation.updateTask - Append a conditional expression that will only update the record if the logged in user is its owner.
Mutation.deleteTask - Append a conditional expression that will only delete the record if the logged in user is its owner.

In other words, the @auth directive currently protects the root level query & mutation fields.
Github Issues

https://github.com/aws-amplify/amplify-cli/issues/655
https://github.com/aws-amplify/amplify-cli/issues/111

Problem: You cannot protect @connection resolvers
For example, look at this schema.
type Task @model {
    id: ID!
    title: String
    owner: String
    notes: [Task] @connection(name: "TaskNotes")
}
# We are trying to specify that notes should only be visible by the owner but
# we are unintentially opening access via *Task.notes*.
type Notes @model @auth(rules: [{allow: owner}]) {
    id: ID!
    title: String
    task: Task @connection(name: "TaskNotes")
    owner: String
}

Since only top level fields are protected and we do not have an @auth directive on the Task model, we are unintentionally opening access to posts via Task.notes.
Solution
We discussed having @auth rules on OBJECTs automatically protect connection fields in proposal 1, but I also suggest opening the @auth directive such that it can be placed on both FIELD_DEFINITION and OBJECT nodes. This will result in an updated definition for @auth:
directive @auth(rules: [AuthRule!]!) on OBJECT, FIELD_DEFINITION
# ...

You may then use the @auth directive on individual fields in addition to the object type definition. An @auth directive used on an @model OBJECT will augment top level queries & mutations while an @auth directive used on a FIELD_DEFINITION will protect that field's resolver by comparing the identity to the source object designated via $ctx.source.
For example, you might have:
type User @model {
    id: ID!
    username: String

    # Can be used to protect @connection fields.
    # This resolver will compare the $ctx.identity to the "username" attribute on the User object (via $ctx.source in the User.posts resolver).
    # In other words, we are authorizing access to posts based on information in the user object.
    posts: [Post] @connection(name: "UserPosts") @auth(rules: [{ allow: owner, ownerField: "username" }])

    # Can also be used to protect other fields
    ssn: String @auth(rules: [{ allow: owner, ownerField: "username" }])
}
# Users may create, update, delete, get, & list at the top level if they are the
# owner of this post itself.
type Post @model @auth(rules: [{ allow: owner }]) {
    id: ID!
    title: String
    author: User @connection(name: "UserPosts")
    owner: String
}

An important thing to notice is that the @auth directives compares the logged-in identity to the object exposed by $ctx.source in the resolver of that field. A side effect of (1) is that an @auth directive on a field in the top level query type doesn't have much meaning since $ctx.source will be an empty object. This is ok since @auth rules on OBJECT types handle protecting top level query/mutation fields.
Also note that the queries and mutations arguments on the @auth directive are invalid but the operations argument is allowed. The transform will validate this and fail at compile time with an error message pointing you to the mistake. E.G. this is invalid:
type User @model {
    # @auth on FIELD_DEFINITION is always protecting a field that reads data.
    # Fails with error "@auth directive used on field User.ssn cannot specify arguments 'mutations' and 'queries'"
    ssn: String @auth(rules: [{allow: owner}, mutations: [create], queries: [get]])

    # No one but the owner may update/delete/read their own email.
    email: String @auth(rules: [{allow: owner}, operations: [update, delete, read])
}

The implementation for allowing operations in field level @auth directives is a little different.

create - When placed on a @model type, will verify that only the owner may pass the field in the input arg. When used on a non @model type, this does nothing.
update - When placed on a @model type, will verify that only the owner may pass the field in the input arg. When used on a non @model type, this does nothing.
delete - When placed on a @model type, will verify that only the owner may set the value to null. Only object level @auth directives impact delete operations so this will actually augment the update mutation and prevent passing null if you are not the owner.
read - Places a resolver on the field (or updates an existing resolver in the case of @connection) that restricts access to the field. When used on a non-model type, this still protects access to the resolver.

Proposal 5: And/Or in @auth rules
Github Issues

https://github.com/aws-amplify/amplify-cli/issues/317

Problem
Currently all @auth rules are joined via a top level OR operation. For example, the schema below results in rules where you can access Post objects if you are the owner OR if you are member of the "Admin" group.
type Post @model @auth(rules: [{ allow: owner }, { allow: groups, groups: ["Admin"] }]) {
    id: ID!
    title: String
    author: User @connection(name: "UserPosts")
    owner: String
}

It would be useful if you could organize these auth rules using more complex rules combined with AND and OR.
Solution
We can accomplish this by adding to the the @auth definition.
directive @auth(rules: [TopLevelAuthRule!]!) on OBJECT, FIELD_DEFINITION
input TopLevelAuthRule {
    # For backwards compat, any rule specified at the same level as an "and"/"or" will be joined via an OR.
    allow: AuthStrategy!
    ownerField: String # defaults to "owner"
    identityField: String # defaults to "cognito:username" for UserPools, "username" for IAM, "sub" for OIDC
    groupsField: String
    groups: [String]

    # This only exists in top level rules and specifies operations for all the rules even when combined with and/or.
    # Neseted "operations" tags are not allowed because it would confuse evaluation logic.
    operations: [ModelOperation]

    # New recursive fields on AuthRule
    and: [AuthRule]
    or: [AuthRule]
}
input AuthRule {
    allow: AuthStrategy!
    ownerField: String # defaults to "owner"
    identityField: String # defaults to "cognito:username" for UserPools, "username" for IAM, "sub" for OIDC
    groupsField: String
    groups: [String]

    # New recursive fields on AuthRule
    and: [AuthRule]
    or: [AuthRule]
}
enum AuthStrategy { owner groups }
# Reduces get/list to read. See explanation below.
enum ModelOperation { create update delete read }

This would allow users to define advanced auth configurations like:
type User
  @model
  @auth(rules: [{
    and: [
      { allow: owner },
      { or: [
        { allow: groups, groups: ["Admin"] },
        { allow: owner, ownerField: ["admins"] },
      }
    ],
    operations: [read]
  }]) {
  id: ID!
  admins: [String]
  owner: String
}
# Logically: ( isOwner && ( isInAdminGroup || isMemberOfAdminsField ) )

The generated resolver logic will need to be updated to evaluate the expression tree.
Proposal 6: Deny by default mode.
Github Issues

https://github.com/aws-amplify/amplify-cli/issues/344

Problem: There is currently no way to specify deny access by default for Amplify APIs.
If you create an API using a schema:
type Post @model {
    id: ID!
    title: String!
}

then the generated create, update, delete, get, and list resolvers allow access to any request that includes a valid user pool token (for USER_POOL auth). This proposal will introduce a flag that specifies that all operations should be denied by default and thus all fields that do not contain an explicit auth rule will be denied. This will also change the behavior of create mutations such that the logged in user identity is never added automatically when creating objects with ownership auth.
Solution: Provide a flag that enables deny by default
By adding a DenyByDefault flag to parameters.json or transform.conf.json will allow users to specify whether fields without an @auth directive will allow access or not. When deny by default is enabled the following changes will be made.

Mutation.createX resolvers will no longer auto-inject the ownership credential when the ownership credential is not provided when creating objects. Users will have to supply the ownership credential from the client and it will be validated in the mutation resolver (this happens already when you provide the ownership credential in the input).
All resolvers created by a @model without an @auth directive will be denied by default.
All resolvers created by @searchable on a @model without an @auth will be denied by default.
All resolvers created by @connection that return a @model with an @auth AND that do not have their own field level @auth will be denied by default.

For example, with deny by default enabled
type Post @model {
    id: ID!
    title: String
    author: User @connection(name: "UserPosts")
    comments: [Comment] @connection(name: "PostComments")
}
type User @model @auth(rules: [{allow: owner}]) {
    id: ID!
    username: String!
    posts: [Post] @connection(name: "UserPosts")
}
type Comment @model {
    id: ID!
    content: String
    post: [Comment] @connection(name: "PostComments")
}

This mutation would fail:
mutation CreatePost {
    createPost(...) {
        id
        title
    }
}

This mutation would succeed:
mutation CreateUser {
    createUser(input: { username<EMAIL_ADDRESS>}) { # Assuming the logged in user identity is the same
        id
        title
    }
}

This top level query would succeed but Post.comments would fail.
query GetUserAndPostsAndComments {
    getUser(id: 1) { # succeeds assuming this is my user.
        posts { # succeeds because the @auth on User authorizes the child fields
            items {
                title
                comments { # fails because there is no auth rule on Post, Comment, or the Post.comments field.
                    items {
                        content
                    }
                }
            }
        }
    }
}

More details coming soon

Write custom auth logic w/ pipeline functions

E.G. Only allow a user to create a message if they are a member of the chat.
GH issues

https://github.com/aws-amplify/amplify-cli/issues/318
https://github.com/aws-amplify/amplify-cli/issues/372
https://github.com/aws-amplify/amplify-cli/issues/891

Enable IAM auth within the API category

Automatically create and attach IAM policies to the Amplify Auth & Unauth IAM roles.
GH issues

https://github.com/aws-amplify/amplify-cli/issues/891
https://github.com/aws-amplify/amplify-cli/issues/624

Request for comments
This document details a road map for authorization improvements in the Amplify CLI's API category. If there are use cases that are not covered or you have a suggestion for one of the proposals above please comment below.

You will be able to use @connection with the new custom index features to optimize the access pattern and then use @auth to protect access within that table or index.

Loving it.

The proposed change would create a new subscription resolver for each subscription field generated by the @model. Each subscription resolver would verify the provided owner matches the logged-in identity and would fail the subscription otherwise.

Fantastic!

You may then use the @auth directive on individual fields in addition to the object type definition. An @auth directive used on an @model OBJECT will augment top level queries & mutations while an @auth directive used on a FIELD_DEFINITION will protect that field's resolver by comparing the identity to the source object designated via $ctx.source

I'm drooling a bit at this point...

This proposal will introduce a flag that specifies that all operations should be denied by default and thus all fields that do not contain an explicit auth rule will be denied. This will also change the behavior of create mutations such that the logged in user identity is never added automatically when creating objects with ownership auth.

Thank you please.
Fantastic proposals! I think the level of granular control this will enable will make it easier to maintain regulatory compliance (e.g. HIPAA, PCI) with the graphql-transformer and generally make schema design simpler. Looking forward to seeing it in production.

Great write up and love the proposals. One big area is the idea of multi-tenancy and not relying on Cognito Groups but "tenantId" or some other arbitrary field to determine tenant ownership. This might be covered by enabling custom Auth logic with pipeline functions. This is how I'm solving it currently, but it's a manual process and a directive to cover this case would be nice!

The "Deny by default" mode would be perfect. For a secure multi-tenant app I would rather have all of my queries/mutations protected by default and need to be explicitly opened up for a user/group, so that a schema/resolver mistake on my end doesn't automatically mean compromised user data.

These are all fantastic and I can't wait for them to be implemented. I think proposal 1 needs to get implemented asap so we don't have to refactor too much. I also think proposal 5 is very important feature that's lacking from Amplify. Having to write pipeline resolvers + CFN for every operation because you need to verify the tenant sucks the joy out of my life!

Please consider the #805 issue. Perhaps the auth directive could include a flag that would generate an IAM policy or IAM role that would allow other resources (eg. Lambda) to perform mutations and queries? The other role (eg. Lambda) would need to be responsible for setting the owner field correctly.
On one hand, I recognize that Amplify prefers to keep everything within its own ecosystem, however, it would be nice if there were simple hooks to allow other resources to interact with the Amplify stack and leverage the graphQL endpoint.
It seems like a common use-case that the user might create a DynamoDB object and then a backend system would update it or delete it.

Proposal 3 to secure subscriptions is a must have for us (and I’m really curious to know if I could as of today us the console to attach a particular resolver to a subscription that would secure it).
Regarding proposal 4, I would like being able to define a field that is read only for certain users and writable for others. I am not sure that this proposal take this use case into account.
Finally, regarding @ajhool remarks right above, I would like to add that making the auth directive compatible with a setup with AWS_IAM would be great ! (for apps that have an auth/unauth setup)
Thank you @mikeparisstuff and the team, please keep going like this, we love amplify :) !

Allow Owner not working On IAM ...

Proposals 1 & 5 implemented in #1262

Proposal 3 would help companies working with Amplify/AppSync subscriptions maintain compliance (PCI, HIPAA, privacy, etc) more easily. Please consider prioritizing it for your next iteration!

@mikeparisstuff If it's not too late, you REALLY need to change auth owner validation to be based on the cognito sub and not the username.
Usernames can be reassigned, and if that happens, you could have a potential situation where person can access a record owned by the person who used to own that record.
The sub value does not change, ever, and should be the unique identifier for validation of ownership.
AWS response explains this pretty clearly:
https://forums.aws.amazon.com/thread.jspa?threadID=243796

@mikeparisstuff - I'm wondering if Proposal 5 could be expanded to support conditions more complex than just and/or?
A particular use case I'd love to see @auth support, is arbitrary field value comparison.
For example, i have a model (let's call it "node") which has various states: ["start","middle","end"]. I want user's to only have access to certain nodes, based on the node's state and the group they belong to.
Rather than a static and/or field, something like condition could be used to future proof, which could encapsulate any possible operator (which could be rolled out over time): and, or, eq, lt, gt, in, etc.
type Node
    @model
    @auth(rules: [
      // allow Admins full access
      { allow: groups, groups: [ "Admin" ] },

      // allow employees in group "pre-process" to access Node's in state "start"
      // Example 1 - static comparison. explicitly set group name, and expected value of `state` field
      { allow: groups, groups: ["pre-process"],
        // -- here's an expansion on the and/or logic proposed in "Proposal 5"
        // note the use of `condition`, `eq`, `field`, `val`
        condition: {
          eq: [
            // value of `state` field must equal 'pre-process'
            { field: 'state' },
            { val: 'pre-process'}
          ]
        },

      // Example 2 - dynamic comparison.
      {
        allow: groups,
        condition: {
          eq: [
            // value of `state` field, must equal value of groupField concatenated to 'state-processors-'
            // e.g. someone in group 'state-processors-start' could access nodes with state `start`
            { field: 'state' },
            { join: [
                // static value
                { val: 'state-processors-'},
                // variable reference, in this case it's the value of the groupInput
                { ref: 'group' }
              ]
            }
          ]
        }
      }
    ]) {

    id: ID!
    state: State!
}

enum State {
    Start,
    Middle,
    End
}

This is pretty rough - but just sketching out a possible solution. I also can see the case for this creating too much complexity, but the only alternative right now (that I see) seems to be writing a custom resolver for every model, which is complex in it's own way.

A common use case that is currently a little bit awkward is a DynamoDB object where each user should have only one (eg. UserSettings). I usually implement that setting the id field to the owner field. It would be really nice if the Get resolver recognized that the ownerField was also the ID field and returned the object. I believe that similar logic is used in the list query when { allow: owner, operations: [get, list ] }, where the resolver queries on the ownerField.
For instance:
type UserSettings @model
@auth(rules: [
    { allow: owner, operations: [get, update] },
    { allow: groups, groups: ["Admin"] }
]) {
  owner: ID
  favoriteColor: String!
}

I believe the 2 current ways to achieve this query are:
a.
// must add "list" to the auth model operations
const result = await API.graphql(graphqlOperation(queries.listUserSettings));
const { items } = result.listUserSettings;
const userSettings = (items.length === 1) ? items[0] || null;
If( null === userSettings) {
throw 'No user settings found'
}
b.
// I'm not sure if the id field is the field that amplify chooses for "owner"
const { id } = await Auth.currentAuthenticatedUser();
await API.graphql(graphqlOperation(queries.getUserSettings, { owner: id }));
Neither of those are bad at all and this definitely isn't high priority, but it would be nice to have in the backend and it is common enough that it would make sense to support natively IMHO

Another common use case that I don't think is currently possible with the @auth directive (please correct me if I'm wrong, I'd really like to use it!):
A persisted object that an Admin creates/updates, but that everybody should be able to get/list. It would be nice if there group named  EVERYBODY or AUTHENTICATED or ALL that simply contained all authenticated users.
type Product @model
  @auth(rules: [
    { allow: groups, groups: ["Admin"] },
    { allow: groups, groups: [ _EVERYBODY_ ], operations: [ get, list ] }
  ]
  {
    id: ID!
    name: String
    description: String
  }

It seems unnecessarily complex that we would create a Cognito group called "everybody" or "customers" and automatically add all users to that group when they sign up, but maybe that is the better design pattern?

Hi folks, might be the wrong thread or place for this but somewhat related - can anyone explain to me the rationale for choosing Cognito:username as the implicit "owner" if none is provided, as implemented in the auto gen'ed resolvers, as opposed to the identity's "sub"? My guess is one's as good as the other and the former is more readable, but I do wonder about the impact of username changing/reassignment etc.
Googling a little on this pulled up this thread for example, which suggests that "sub" would be a much more resilient identifier for auth? For example, if a user is deleted and another created with the same username, the current auth means they'll get access to all the first user's entities..?
https://stackoverflow.com/questions/39223347/should-i-use-aws-cognito-username-or-sub-uid-for-storing-in-database

@jmorecroft it's purely a mistake on the Amplify team's part.  The sub should always be used to uniquely identify a user.  I already posted about this here in this same thread.

I had posted a similar bug (in the amplify js repo), where the Storage object is connected to a transient id: https://github.com/aws-amplify/amplify-js/issues/1787 , not sure if that is still the case.

Protecting a @connection field by queries is done by using this pattern:
type User @model {
    id: ID!
    username: String

    posts: [Post]
      @connection(name: "UserPosts")
      @auth(rules: [{ allow: owner, ownerField: "username" }])
}
type Post @model(queries: null) { ... }

How can we protect, that someone else adds a new Post to any other User?
Say I'm logged in as 'Cersei' and do the following mutation:
mutation addPostToOtherUser {
  createPost(input: { title:"My evil Post", userPostId:"Daenerys"} ) {...}
}

I would like to protect against this.

I'm using the new @auth 'operations' argument to achieve:

allow read (get/list) access to any unauthenticated or authenticated user.
allow only an authenticated user to create, update, delete.

my setup:
{allow: owner, operations: [create, update, delete]}
Outcome:

allow read (get/list) access to any unauthenticated or authenticated user.
-- it does work for authenticated user.
--it does NOT work for unauthenticated user. error: "no current user".

allow only an authenticated user to create, update, delete.
-- it does work for authenticated user, allow to perform operations.
-- it does work for unauthenticated user, does not allow to create an object.

Since `{allow: owner} will impose OwnerStrategy for ALL operations, I would expect my setup to exclude the read operation from the OwnerStrategy. Because it is excluded I would expect it to function just as it would without the OwnerStrategy. Regardless of a user or not, it should produce the read.

I am interested in Proposal 5, as I would like to have a group membership override the owner priveleges. For example, a banned user (still owner of a post, but not currently allowed to edit/delete due to being in the "Banned Users" group).

Can @auth support be extended to S3Objects (storage) category? Last time I checked the storage category (https://github.com/aws-amplify/amplify-cli/blob/master/packages/amplify-category-storage/Readme.md) creates 3 folders - private, protected, and public. Objects in private are only accessible to the user who uploaded them. Protected is accessible to any authenticated user, and public is public access.
The authentication for this is also through identity management I believe, not cognito sub.
It seems kind of divorced from the @auth rules set in the GraphQL schema, though access to the S3Object stored in dynamodb is controlled by the @auth rules. The actual objects stored in S3 are not. This is a point of confusion and perhaps can be considered as part of this RFC or #766.
From my high level point of view, data in S3 should be controlled under the same access rules set by this @auth directive.

Can @auth support be extended to S3Objects (storage) category? Last time I checked the storage category (https://github.com/aws-amplify/amplify-cli/blob/master/packages/amplify-category-storage/Readme.md) creates 3 folders - private, protected, and public. Objects in private are only accessible to the user who uploaded them. Protected is accessible to any authenticated user, and public is public access.
The authentication for this is also through identity management I believe, not cognito sub.
It seems kind of divorced from the @auth rules set in the GraphQL schema, though access to the S3Object stored in dynamodb is controlled by the @auth rules. The actual objects stored in S3 are not. This is a point of confusion and perhaps can be considered as part of this RFC or #766.
From my high level point of view, data in S3 should be controlled under the same access rules set by this @auth directive.

+1 for this. I have a use case that I need to control S3 access based on the same Cognito/Auth Groups that govern my API access. I need Admin level access, group level access, or individual level access based on logged in user. I'm trying to set up S3 Bucket policies based on Cognito User Pool groups but I'm having to do it manually at the moment.

Wait a second, per item number 3, where "Currently subscriptions are not protected automatically." does this mean that protected data can leak to anybody who subscribes to a data type? (issue also referenced, here: #1766  )
Let's say the owner of the website defines their auth module as follows:
type Todo @model @auth(rules: [{ allow: owner }]) {

And then I go to their website and run this code from their site (that developer has to be running the code from the client because amplify-js doesn't support server side execution).
import Amplify, { API, graphqlOperation } from 'aws-amplify';
import * as subscriptions from './graphql/subscriptions';

// Subscribe to creation of Todo
const subscription = API.graphql(
    graphqlOperation(subscriptions.onCreateTodo)
).subscribe({
    next: (todoData) => console.log(todoData)
});

// Stop receiving data updates from the subscription
subscription.unsubscribe();

Do I now have access to every single Todo object that gets created on their site? Even though the developer added the @auth rule for "owner" only with the clear goal of keeping that user's Todo (or, say, healthcare records) information private?
Please tell me that I'm misunderstanding something here because this is a makes a mockery of the efforts I've taken to secure my website, if the Amplify team is aware of such a terrible vulnerability and doesn't make any effort to document it, aside from a proposal in an RFC. Why isn't this documented AS CLEARLY AS POSSIBLE SO THAT ANYBODY WHO CARES ABOUT THE SECURITY OF THEIR "VALUABLE INFORMATION" CAN TAKE STEPS TO MITIGATE IT OR SIMPLY NOT TOUCH THIS PRE-ALPHA FRAMEWORK UNTIL IT IS CLOSER TO PRODUCTION READY
Why is there even an @auth directive provided if it doesn't currently prevent subscriptions from funneling all of the data into the hands of anybody who wants it?
Would anybody who uses amplify in production and protects valuable information please send me their website url?

@mikeparisstuff here's some feedback on our use case regarding auth and subscriptions. I hope it helps. It's a pretty straight forward scenario.
Consider a slack clone. We have a message table or type. Every message in inserted with the groups and individuals that can see it (according to the auth rules). The identity claims are custom and injected pre token generation, via Cognito triggers.
When a new message is created, we hope that all the users that are authorized to read it, will get notified through an onCreateMessage subscription. The case right now is that every client that is subscribed gets the notification. Clearly that cannot scale and compromises security.
I understand there is custom work that can be done through vtl. Our suggestion is that all the groups and individuals that are authorized to read the record, would receive an event. It is what I expected the case would be before diving in.
Hope this helps.

To anybody who is currently using Amplify in production, you are probably exposed to the data leak vulnerability discussed in #1766 and I highly recommend that you add this fix to your graphql schema. Amplify automatically creates Subscription resolvers that do not respect @auth, so an attacker can simply Subscribe to your data model's endpoint and they will receive all/site-wide OnCreate / OnUpdate events for that data model. Those events include the data, even if the attacker is not authorized to see that data. The fix is to disable Subscription autogeneration by adding the below line to your code:
@model(subscriptions: null)
This fix appears to be undocumented (the vulnerability is also undocumented) but it comes from the Amplify team. If your app uses Subscriptions, then you will currently need to create a custom Subscription resolver with custom Authentication logic -- or use some other form of security in depth.

Hey @ajhool as discussed in the other thread #1766 we have added documentation covering this. Please see Authorizing Subscriptions. That being said, we do still want to support these features so please reply with details on use cases and scenarios that you would like to support so we can work it into the roadmap and schedule.

@mikeparisstuff here's some feedback on our use case regarding auth and subscriptions. I hope it helps. It's a pretty straight forward scenario.
Consider a slack clone. We have a message table or type. Every message is inserted with the groups and individuals that can see it (according to the auth rules). The identity claims are custom and injected pre token generation, via Cognito triggers.
When a new message is created, we hope that all the users that are authorized to read it, will get notified through an onCreateMessage subscription. The case right now is that every client that is subscribed gets the notification. Clearly that cannot scale and compromises security.
I understand there is custom work that can be done through vtl. Our suggestion is that all the groups and individuals that are authorized to read the record, would receive an event. It is what I expected the case would be before diving in.
Hope this helps.

@rawadrifai With this model, how would you see yourself doing the rules? Essentially you have public & private chat rooms in this model. Would you just blanket apply the query permissions to subscriptions? Do the mutation permissions get ignored? Or is it something that you add onto the @auth directive?
For example one hypothetical method could be that you specify named groups that might be allowed to receive messages:
type Message @auth({ subscribers: ["GroupA", "GroupB"]}) {
  id: ID!
  content: String
}

But realistically this wouldn't be that flexible so each room would have an id with arguments (per @mikeparisstuff comments in the original proposal) so maybe you do something like:
type Message @auth({ subscribers: [{RoomX: GroupA, RoomY: GroupB}]}) {
  id: ID!
  content: String
}

This would mean that anyone from GroupA who did a onCreateMessage(id:"RoomX") would be granted access. Another option might be saying only owners to objects they created can receive updates to them:
type Message @auth({ subscribers: owner}) {
  id: ID!
  content: String
}

These are just some thoughts to spark a little more dialogue on your use cases and requirements. Getting more specifics on what exact controls you'd like to use for granting or restricting access will help us know the MVP for such a feature.

@undefobj thank you for this. Here's a subset of my model below. The idea is that every user when they sign up, they do after an invite from someone in the "Community" (similar to slack).
A community has multiple users. A user can belong to a single community.
A Channel can be private or public.
Public channels can be viewed by everyone. Private channels only by their members.
Conversations have a users field that tell you who are the users in a given conversation.
A conversation is a DM as opposed to a channel.
# Users that belong to the community (group name in id) have read access

type Community @model @auth(rules: [
  { allow: owner },
  { allow: groups, groupsField: "id", operations: [read] }
]) {
  id: String!
  owner: String
  description: String
  enabled: Boolean
  users: [User] @connection(name: "communityUsers")
}

# The owner of the user has full control
# Users that belong to the same COMMUNITY have read access
# The groups field should be the name of the COMMUNITY

type User @model @auth(rules: [
  { allow: owner },
  { allow: groups, groupsField: "group", operations: [read] }
]) {
  id: String!
  group: String
  firstName: String
  lastName: String
  dobMonth: String
  dobDay: Int
  dobYear: Int
  address1: String
  address2: String
  country: String
  region: String
  city: String
  postalCode: String
  email: String
  enabled: Boolean
  community: Community @connection(name: "communityUsers")
  memberships: [Membership] @connection(name: "userMemberships")
}

# The owner of the conversation has full control
# Users of the conversation have read access (2 or more people conversation)

type Conversation @model @auth(rules: [
  { allow: owner },
  { allow: owner, ownerField: "users", operations: [read] }
]) {
  id: ID!
  users: [String]
  archived: String
  messages: [Message] @connection(name: "ConversationMessages", sortField: "createdAt")
}

# The owner of the message has full control
# The users field should be an array of users of a conversation (if this message is for a conversation)
# The group field should be the name of the CHANNEL (if this message is for a channel)

type Message @model(queries:null) @auth(rules: [
  { allow: owner },
  { allow: groups, groupsField: "group", operations: [read] },
  { allow: owner, ownerField: "users", operations: [read] }
]) {
  id: ID!
  owner: String
  createdAt: String
  content: String
  group: String
  users: [String]
  archived: String
  conversation: Conversation @connection(name: "ConversationMessages")
  channel: Channel @connection(name: "ChannelMessages")
  comments: [Comment] @connection(name: "MessageComments", sortField: "createdAt")
}

# The owner of a Channel has full control
# The Channel created should have a group field that controls access.
# If the channel is public the group should be the name of the COMMUNITY.
# If the channel is private the group should be the name of the CHANNEL itself.

type Channel @model @auth(rules: [
  { allow: owner },
  { allow: groups, groupsField: "group", operations: [read] }
]) {
  id: String!
  description: String
  archived: String
  group: String
  type: ChannelType
  messages: [Message] @connection(name: "ChannelMessages", sortField: "createdAt")
  memberships: [Membership] @connection(name: "channelMemberships")
}

# Maintains the channel users
# Also we query this table to know what groups a user belongs to

type Membership @model @auth(rules: [
  { allow: owner },
  { allow: groups, groupsField: "group", operations: [read] }
])
{
  id: String!
  group: String
  user: User @connection(name: "userMemberships")
  channel: Channel @connection(name: "channelMemberships")
}

One caveat for inserting new messages, that has to do with auth too, is that I am not sure how to prevent users who do not belong to a channel from creating a message with channelId that they shouldn't use.
So basically, pre insert, I would like to check if the "channelId" they're using is one of the values of the cognito groups they're in.

I have another suggestion that if not implemented now, might be to keep in mind for the future. At the risk of sounding too wishful, from a development perspective, it would be very convenient to have s3 objects adopt the auth of a reference field in the schema.
The most common use case is someone storing a reference for an object in a database, Dynamo in my case. When accessing the s3 object the desired outcome is to apply the same rules as the @auth.

@mikeparisstuff The documentation additions look good, although the main definition is still incorrect:

@auth
Object types that are annotated with @auth are protected by a set of authorization rules. Currently, @auth only supports APIs with Amazon Cognito User Pools enabled. You may use the @auth directive on object type definitions and field definitions in your project’s schema.

When using the @auth directive on object type definitions that are also annotated with @model, all resolvers that return objects of that type will be protected.

The bolded part says that all resolvers are protected, but actually Subscription resolvers aren't protected.
I still don't think that documentation is enough and that the current unprotected Subscriptions should be opt-in, not opt-out. I also think that this basic security strategy of requiring developers to explicitly opt-in to public resources should be used in all future Amplify feature releases.
Anyways, I'll stop pestering about the public subscription resolvers, but I do think there is a lesson there that should be applied to the @auth directive...
It's the same lesson that AWS learned from the PUBLIC-by-default S3 buckets, which caused so many data leaks. If the default protection for a resource is Public, then the developer should need to explicitly add a PUBLIC flag in code.
For instance, if you don't specify an @auth flag on an /@/model, that model is a public resource (I think). I can't think of any application with a publicly readable/writeable/updatable/deletable data object, so the "convenience" of this default is dubious.
IMO, the @auth flag should always be mandatory, perhaps there could be an @auth(PUBLIC) flag that would replicate the current behavior without an @auth directive.
TLDR; The @auth flag should be mandatory on all /@/models & functions. Use @auth(PUBLIC) to opt-in to the current default behavior. Use the same concept of explicit auth on all resources, including Storage

@mikeparisstuff I tried following documentation and changed my code to be exactly as it says in the doc but I get a bunch of errors when I create subscriptions. Can you please share actually working example?

I suspect that comments in the vtl files should have ## instead of # in the example
is it correct that data source is new and set to name: Local type: None?
should { "ok": true } be replaced with $util.toJson($ctx.result)?

I tried all sorts of fixes but end up getting various errors from "Cannot return null for non-nullable type:" to "you don't have an operation field in your request mapping template" to ""Unsupported operation 'Subscription'."" etc.
Looked through docs but type subscription is not a type for DynamoDB and default resolvers for subscriptions are not even in the deployment resolvers folder together with query/mutation/etc resolvers.
can someone give me some hints how to resolve this? I would appreciate some sort of end-to-end working example.

Oh boy, so the @searchable directive is also not protected by @auth and this is also not documented anywhere except for this RFC despite the first issue (#460) being posted in November 2018?
Documentation: "When using the @auth directive on object type definitions that are also annotated with @model, all resolvers that return objects of that type will be protected."
Searchable resolver: Provides unprotected search of all private data to anybody who wants it even when using @auth. (No documentation, no CLI warnings)
Subscription resolvers: Provides an unprotected streaming data pipeline of private data updates/creates/deletes to anybody who wants it even when using @auth. (Insufficient documentation, no CLI warnings, opt-out behavior).
AWS' willingness to push out zero-security features with little to no documentation or warning to developers is incredibly troubling, particularly as we depend on this framework in production and we value our customers' data security.
Even worse, as seen in the discussion on the #460 issue, the Amplify team actually tried to close that issue as being resolved... and it is marked as a Feature Request instead of as a Bug... Again, the issue is that when you use the @searchable directive, it exposes a search resolver of private data to anybody who wants it... Despite the behavior not being intuitive, contrary to the documentation, and it being a clearly outrageous security vulnerability, it's considered a feature request and not a bug?
"Well, you can just make a custom resolver, yourself" -- maybe so, but AWS provides zero indication that you would need to while presenting all of these features as working seamlessly with each other and as being protected by @auth.
AWS is presenting security as a feature while actually treating security as a Feature Request.
Amplify is a very useful framework that has grown tremendously, but it is clear that the team needs to take immediate steps to fix current vulnerabilities / documentation, implement better security audits when pushing out new features, and shift to a "secure-by-default" design strategy for all future features.

Hey @mikeparisstuff - do you have any update on the likely direction with @auth and subscriptions?

They way a user knows they received a message, is by subscribing to onCreateMessage. My expectation was that they would receive the messages that they can read. Check out the permissions on the Message type.

@rawadrifai We've been reading through the feedback on the design here, and one thing that might be a misunderstanding is the way that authorization works for subscriptions. It is at connection time, and a resolver will run granting or denying the subscription request. So to take your use case where the client is subscribing to onCreateMessage we would actually need to require arguments in that subscription which would give the resolvers enough context for them to perform authorization logic.
For example in might be onCreateMessage(userId: ID!) or onCreateMessage(roomId:ID!) which basically says all messages for a specific user or room. Channels/conversations are just generalizations of this. Here's an example in the AppSync docs of how this might look: https://docs.aws.amazon.com/appsync/latest/devguide/security-authorization-use-cases.html#real-time-data
With this in mind, my current thinking is that by adding an explicit ownership or channel protection this could make the input fields on a subscription required as outlined above. I've got to think a bit more on field level settings.

One caveat for inserting new messages, that has to do with auth too, is that I am not sure how to prevent users who do not belong to a channel from creating a message with channelId that they shouldn't use. So basically, pre insert, I would like to check if the "channelId" they're using is one of the values of the cognito groups they're in.

This is a more straightforward to to, since unlike subscriptions mutations are not long held connections and we can do conditional checks on membership before the write occurs. That's what @auth does today.

Hello everyone - We have a version of this functionality which has been published to the following tag:
npm install -g @aws-amplify/cli@authSubRelease

In this implementation, when a model is protected using the owner auth strategy, each subscription request will require that the user is passed as an argument to the subscription request. If the user field is not passed, the subscription connection will fail. In the case where it is passed, the user will only get notified of updates to records for which they are the owner.
Alternatively, when the model is protected using the static group auth strategy, the subscription request will only succeed if the user is in an allowed group. Further, the user will only get notifications of updates to records if they are in an allowed group. Note: You don't need to pass the user as an argument in the subscription request, since the resolver will instead check the contents of your JWT token.
For example suppose you have the following schema:
type Post @model
@auth(rules: [{allow: owner}])
{
  id: ID!
  username: String
  postname: String
  content: String
}

This means that the subscription must look like the following or it will fail:
subscription onCreatePost(username: “Bob”){
  postname
  content
}

In the case of groups if you define the following:
type Post @model
@model @auth(rules: [{allow: groups, groups: ["Admin"]}]) {
{
  id: ID!
  username: String
  postname: String
  content: String
}

Then you don’t need to pass an argument, as the resolver will check the contents of your JWT token at subscription time and ensure you are in the “Admin” group.
Finally, if you use both owner and group authorization then the username argument becomes optional. This means the following:

If you don’t pass the user in, but are a member of an allowed group, the subscription will notify you of records added.
If you don’t pass the user in, but are NOT a member of an allowed group, the subscription will fail to connect.
If you pass the user in who IS the owner but is NOT a member of a group, the subscription will will notify you of records added of which you are the owner.
If you pass the user in who is NOT the owner and is NOT a member of a group, the subscription will not notify you of anything as there are no records for which you own

Dynamic groups have no impact to subscriptions, so you will not get notified of any updates to them.
Next week we will add capability that if your type doesn’t already have username the Transformer will automatically add this for you. There are also still a few edge cases with field level auth we need to validate for subscriptions, however the behavior will be that the field will return null if the field auth is not a subset of the object type auth. In the meantime please have a look at the early version published to the tag above. Our aim is to do final validations over the next few days and publish to the latest build next week.

It is great to see that subscriptions are now covered by auth.
Will future features follow this same trajectory of releasing the feature and then adding security a year later? The @connection directive was also released before it was protected by auth as was/is @searchable and those were also not documented.
Security is a weakest-link problem so the design strategy is more important than the individual features and I really hope that the Amplify team would acknowledge that leaving undocumented public autogenerated subscriptions open for a year (they're still open until this branch is merged) was not a good strategy and indicate that better security audits will be used for future features.
I'm sorry for continuing to make these  posts even though they aren't receiving official responses but I think basic security is critically important and official responses would be appreciated.

@undefobj  This is a great start.
Please keep in mind the use case where Cognito groups do not suffice.
For example, I interject a login request to append the groups a user belongs to. I get those from Dynamo. So to check whether a user belongs in a group or not, please do so with the token not a call to Cognito. I assume this is the case with this release.

@rawadrifai the Static Group checks are done against claims in the JWT which Cognito gives after a user logs in, so it's not a call to Cognito during connection time. The reason we cannot do Dynamic Group auth is unlike a query or mutation there isn't an item to check against for a subscription (e.g. no authorization metadata exists). I'm still thinking about how to expand that in the future, you would need some sort of a user-to-tag mapping in a DynamoDB table. In the meantime we're trying to pull in this request https://github.com/aws-amplify/amplify-cli/issues/1899 as part of this so you can use alternative claims for the subscription authorization.

@undefobj I am a bit confused.
I use the pre token generation trigger in Cognito to inject the groups after pulling them from Dynamo. So they're in the token, and they work fine for @auth models. Will this update to subscriptions be able to read the groups from the jwt if done as described?

@rawadrifai that's the idea, we're trying to see if you can pass in a groupsField (or something like that) into the auth rule for subscriptions which would correspond to your claim.

@rawadrifai we have published to the tag again:
sudo npm install -g @aws-amplify/cli@authSubRelease --unsafe-perm=true

I've also just submitted a PR for the new functionality: https://github.com/aws-amplify/docs/pull/868
You should be able to use this for documentation before we merge this to latest build. We're still running final stability tests but are trying to get this out ASAP. Your testing would assist us in expediting this. Note that the custom claims for either the user identity (identityClaim) or the groups (groupClaim) are also part of this PR. While the default behavior is to protect subscriptions, customers have an explicit control now to make them public if they wish to keep previous behavior. Also we've added some automatic inspection to flag a warning if you're not using @auth on any @model, listing the ones that are unprotected, with information on how to learn how to add more controls. This happens before an $amplify push sends updates to the cloud allowing you to abort if you made a mistake. Finally there are protections in place if you're using per-field level @auth to protect sensitive data inside a model from being sent over subscriptions.

Is there a way ATM to give public permissions to a query, without any Cognito authentication?
Something like:
type Post @model @auth( rules: [{allow: public}], operations: [read] ) {
  id: ID!
  title: String!
}

Is there a way ATM to give public permissions to a query, without any Cognito authentication?
Something like:
type Post @model @auth( rules: [{allow: public}], operations: [read] ) {
  id: ID!
  title: String!
}

This is coming very soon as part of https://github.com/aws-amplify/amplify-cli/pull/1916

@rawadrifai Can't remember if this is mentioned in issue comments or code directly, but the PR does not support dynamic group authorisation (yet), only static group auth.

@undeobj I’ve implemented an alternative design without the recursive nesting. I’ve added a property and=“name” to each auth rule that should be combined. Then when an auth rule matches I increment a counter with the name as the key. After all rules are run I check which counters are set and compare their value to how many rules have that key in total. If there is a mismatch I set it as unauthorised.

@undefobj no worries. We’re still building with amplify at this point too deep, hoping there would be support for dynamic groups. It’s impossible to design something like slack with static groups only. Let us know if anything changes.

Hey @undefobj , thanks for all your work on this.
There haven't been any updates or comments on Proposal 4 in a long time, is that still in the roadmap? any progress on it yet?
thanks again

One more question, is there a way to define auth rules based on property values? For example, allow User B to view User A's status only if User B is present in User A's approvedFollowers list

@aej11a field level authorization has been implemented for some time now. See docs here.

@amirmishani Thank you! Missed that somehow

@undefobj I opened this (sorry for posting on the RFC) https://github.com/aws-amplify/amplify-cli/issues/2189. If there is no intention to support dynamic groups with subscriptions in the near future, please let us know. If you have an alternative solution staying within aws, also pls comment on the open request.

Hello everyone, an update on auth based off of feedback from the RFC: today we released multi-auth functionality which includes explicit controls and overrides for public & private access, provider overrides, and mixing/matching authorization controls. You can read more here:
https://aws-amplify.github.io/docs/cli-toolchain/graphql#public-authorization
https://aws-amplify.github.io/docs/cli-toolchain/graphql#private-authorization
https://aws-amplify.github.io/docs/cli-toolchain/graphql#combining-authorization-rules
https://aws-amplify.github.io/docs/cli-toolchain/graphql#allowed-authorization-mode-vs-provider-combinations

@undefobj Good updates. Any news on subscriptions and dynamic group auth? If it's not looking likely pls let us know.

@undefobj In the docs you mention:

Note identityField is being deprecated for identityClaim.

Not sure I understand if there's a difference between identityField and identityClaim. Can I just swap identityField with identityClaim in the example below?
type User
  @model
  @key(fields: ["tenantId", "userId"])
  @auth(
    rules: [
      {
        allow: owner
        ownerField: "userId"
        identityField: "sub"
      }
      {
        allow: owner
        ownerField: "tenantId"
        identityField: "custom:tenant_id"
      }
    ]
  ) { ... }

@amirmishani no difference in functionality just an alias for clarity

@rawadrifai probably not because it doesn't make sense for Subscriptions. We spent several hours whiteboarding this and it's not the same use case as a request/response operation. You can dynamically grant access at connection time in your token as previously mentioned.

Hi,
I have an application where users create reports that can be private or shared with on or several users.
I first thought I could use an editors field in reports to manage the set of allowed users on a specific report.
I encountered the following limitation on subscription (detailed in proposal 3) :

Subscription fields are equality checked against published objects. This means that subscribing to objects with with multi-owner or multi-group auth might behave slightly differently than expected.
When you subscribe you will need to pass the full list of owners/groups on the item. Not just the calling identity.

I cannot use groups since each reports can be shared to one or several users independently.
Is there anything planned to manage this kind of use case ?
Thanks

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

I'm still waiting to see if we can get Proposal 5 and have "And" in @auth rules.

@amirmishani Same here. This is the last thing I am waiting for to release my app.

Hi there, any update on Proposal 5 "And" in @auth rules? @mikeparisstuff ?

Sorry for being politically correct, but what moron proposed the idea to combine get and list into a single read permission?

Hi, is there anything new about granting read access for non-authenticated users?

In proposal 3 - add a new sub-proposal  or something -

the subscription started before would not be triggered because ["my-user-id", "my-friends-user-id"] is not the same as ["my-user-id"]. I bring this up for clarity but I still think this feature is useful. Single owner & group based authorization will behave as expected.

Single and group owner should behave as expected but sometimes we need a if-found condition. Add a new if-in-group owner. Allow subscriptions if a user is present in a member array is very frequently needed use case especially when the members list could be dynamic (users can be added or removed )
Example whatsapp application
https://github.com/aws-amplify/amplify-js/issues/7534#issuecomment-760412322

In proposal 3 - add a new sub-proposal  or something -

the subscription started before would not be triggered because ["my-user-id", "my-friends-user-id"] is not the same as ["my-user-id"]. I bring this up for clarity but I still think this feature is useful. Single owner & group based authorization will behave as expected.

Single and group owner should behave as expected but sometimes we need a if-found condition. Add a new if-in-group owner. Allow subscriptions if a user is present in a member array is very frequently needed use case especially when the members list could be dynamic (users can be added or removed )
Example whatsapp application
https://github.com/aws-amplify/amplify-js/issues/7534#issuecomment-760412322

I'm trying to create a YouTube-like resources hub where some sources are public and others are private.
If I use the current public @auth directive, all resources are then accesible to the public but only a subset of them should be.
For example:
type Resource
	@model
	@auth(
		rules: [
			{ allow: groups, groupField: "visibleTo", operations: [read] }
			{ allow: groups, groupField: "editors", operations: [update, read] }
			{ allow: groups, groups: ["admin"], operations: [create, update, read, delete] }
		]
	) {
	id: ID!
	title: String!
	description: String!
	videoUrl: String
	presentationUrl: String
	editors: [String]!
	visibleTo: [String]!
}

The above would look at the visibleTo array for the names of groups that have access to read the resources. The editors array would contain the names of Cognito groups that have access to edit resources. Admins can perform CRUD operations.
With the above example, how does one enable public access without writing countless custom resolvers/lambda functions?
There is no public group assigned to unauthenticated users that receive their credentials (without signing in to the app) via Auth.getCredentials().
Any suggestions on how we can enable public/private access that can change? It would be great if there was a way add a public group to the editors/viewers fields that resolves to public access.

@kylekirkby I'm sorry I don't have time for a more thorough research or a better answer. You should look at both the "custom claims" in the amplify cli graphql transformer documentation, and pre-token trigger in coginito. I used the latter to inject group claims for federated users that are linked to coginto identity pool users. I haven't tried this with public users, I suspect one of these features might be your best path.

@ryanhollander - Thanks for your reply! I think the pre token trigger method would work as long as unauthenticated requests sent to app sync respect the custom identity claims. The pre token trigger function would have to inject a cognito:groups value of public for unauth users.  I’ll have a play with it and see if I can get something working.

@kylekirkby Good luck. FYI, You can use the Amplify CLI to create the trigger w/Lambda through the Auth Flow.

@ryanhollander cheers! I used the cli to add a pre sign up trigger hoping to be able to link External Provider accounts with Cognito Users only to find that the AdminLinkUsers api function is broken :/
https://blog.ilearnaws.com/2020/08/06/error-cognito-auth-flow-fails-with-already-found-an-entry-for-username/

@kylekirkby briefly, I just implemented this using the pre-sign-up trigger and adminLinkProviderForUser. I did this using the Auth Flow and a few lambdas. There were a lot of gotcha's! What I thought might take a morning took me almost 3 days. I did not experience this particular issue, though I read about it alot, it's possible the amplify js eats the issue or cognito was fixed, not sure. FWIW: I'm using Amplify's FederatedLogin.
What I ended up doing was a tad convoluted. The issue is that you can't give the trigger lambda the permissions for the user pool. This is because it would create a circular dependency in the build. The workaround is to create a second lambda, give it the auth permissions via the cli, and manually add the execute permissions to the second lambda for the trigger via the cloudformation json (because there is no dependency for this as IAM doesn't care if your resource exists). Let the trigger call the second lambda for any cognito functions. What I found was I needed two of these, a pre-sign-up to link the federated user to the pool user, and a pre-token to inject the group assignments from the linked cognito user into the token of the federated user. For this project I am using the user pool as the primary user store but users can only login with their domain email through a federated login (for now).
Anyway, I might get around to posting more detail and code on this at some point...

@ryanhollander nice one for the write up! Sounds familiar! I’ve done the whole create a secondary lambda to be called from a post confirmation trigger in another project. I managed to get the pre signup trigger to work with wildcard permissions without needing to add to another lambda function. I know the full cognito pool arn should be used but that was a quick work around and I assumed that since the lambda is triggered directly from Cognito itself, it should be safe some what to assume the correct pull is used.
As for the @auth situation. I’m thinking it might be easier to separate out the PrivateResources into its own type. And give full public read access to a Resource type (public).
As for the storage of the videos in S3 I think I’m going to need to write a custom query to fetch a pre-signed url from s3 if the user belongs to a valid group for that resource.

@kylekirkby with the amplify cli you can give the lambda the exact pool arn for each env with 0 overhead after you do the 1 custom config. Another type/model is always an option, both paths have advantages. I've done that signed URL to S3 scheme too. It's not hard to implement and works great. I think I pulled most of the code from a blog post somewhere.

@ryanhollander how did you arrange the bucket for user uploads? Did you use the Amplify Storage feature and then set uploads to private?

@kylekirkby I used the Amplify Storage and amplify.js on the front end, I store the reference in DynamoDB. I then use the aws sdk in a lambda to generate a signed URL with a 24 hour expiration and a custom URL shortener (lambda/s3) to generate a short, branded link pointing to the signed url. This is what we send to the user when they request a document. The user is interacting with an Alexa skill in this case, and the skill sends the link via SMS or email immediately upon request.

@ryanhollander Thanks for the insight!

Proposals 1 & 5 implemented in #1262

But Proposal 5 is still not working. I still can not implement multi tenancy with user roles as proposed in #317.
My query is
@auth(
rules: [
{ allow: groups, groups: ["admin"] }
{ allow: groups, groupsField: "tenantId", groupClaim: "tenantId" }
]
)
It still generates
#if( !($isStaticGroupAuthorized == true || $isDynamicGroupAuthorized == true || $isOwnerAuthorized == true) )
    $util.unauthorized()
  #end

Or am I missing anything. Can anyone please help me? :)
@mikeparisstuff

Proposals 1 & 5 implemented in #1262

But Proposal 5 is still not working. I still can not implement multi tenancy with user roles as proposed in #317.
I want only the admin to have access on the type and only within the tenant.
My query is
 @auth(
   rules: [
     { allow: groups, groups: ["admin"] }
     { allow: groups, groupsField: "tenantId", groupClaim: "tenantId" }
   ]
 )

It still generates
#if( !($isStaticGroupAuthorized == true || $isDynamicGroupAuthorized == true || $isOwnerAuthorized == true) )
    $util.unauthorized()
  #end

Or am I missing something. What have I done wrong? could someone please help me? :)
@mikeparisstuff @undefobj

Can anyone help me in this ?
It is a pretty common use case, I am sure someone might have found solution for group-owner if-found match criteria ..
https://github.com/aws-amplify/amplify-cli/issues/1043#issuecomment-762596291

I believe something like this might help address some of the concerns brought up in other issues. Some users - including myself - are looking at graphql interfaces as a channel for discussing granting permissions to various table items based on those implementations' @auth rules. However, if we could have auth set by an enum, we could have control over the publicity of an item from within that item instead of by the table.

enum PermissionType @auth( rules: [
  { key: "PUBLIC", allow: public, operations: [ read ] },
  { key: "PRIVATE", allow: groups, groups: ["Admins", "Editors"] }
]) {
  PUBLIC
  PRIVATE
}

type Resource @model {
  id: ID! @primaryKey
  status: PermissionType! @index(name: "Resources_by_Status", queryField: "resourcesByStatus")
  name: String!
}

type Node @model {
  id: ID! @primaryKey
  status: PermissionType! @index(name: "Nodes_by_Status", queryField: "nodesByStatus")
  resources: [Resource] @hasMany(indexName: "Resources_by_Node", fields: ["id"])
  edges: [Edge] @manyToMany(relationName: "EdgeNode")
}

type Edge @model {
  id: ID! @primaryKey
  nodes: [Node] @manyToMany(relationName: "EdgeNode")
}

This would allow us to toggle permissions on an item by simply updating a single item, rather than having to delete a private item, create a public one and reconnect them to their various connections...
Would love to hear your thoughts on an approach such as this

@loganpowell - this looks ideal. We've currently had to write custom resolvers to check if the user is using the unauth role, which, if they are, we don't return private resources. If the user is logged in then the permissions for resources are entirely based on their group membership.

@kylekirkby do you have a GSI that has the resources indexed by auth role? Might you share your custom resolver?

@kylekirkby do you have a GSI that has the resources indexed by auth role? Might you share your custom resolver?

Hi @loganpowell ,
So currently our Resource type has a private boolean field. This is then used to prevent users from seeing private resources in queries via custom VTL code. Currently, I've only managed to get this working in the response resolver, which means when you expect 12 resources in a query you get 12  - any that were private.
What I'm looking at now is updating the client-side queries to filter based on the private value so the correct number of resources are returned to unauth users. It would be much cleaner to get the req resolver filtering these based on the use of the unauth role but, as you probably know, documentation and guides are severely lacking for VTL + Amplify.
So in
amplify/backend/api/yourAPIName/resolvers/Query.resourcesByCreatedDate.res.vtl
I've got this:
#if( $ctx.identity.userArn == $ctx.stash.unauthRole )
  #set( $items = [] )
  #foreach( $item in $ctx.result.items )
    #set( $isPrivate = $util.defaultIfNull($item.private, false) )
    #if( $isPrivate == false )
      $util.qr($items.add($item))
    #end
  #end
  #set( $ctx.result.items = $items )
#end

#if( $ctx.error )
  $util.error($ctx.error.message, $ctx.error.type)
#end
$util.toJson($ctx.result)

If the user is using the unauth role, then private resources are not returned. You'll have to add this logic to all of your resource queries by overriding the amplify generated resolvers.
There may be bits I'm missing but I hope this helps! This has been one hell of an issue to solve for me!


Markdown layout error

✅ It's right!
Learning done: 21/24
Words in test: 11
Chotiskazal.Bot.Interface.Markdown:
➕ envelope
➕ mound
...

fixed


fix: int ids break get one query

fixes https://github.com/panter/ra-data-prisma/issues/19

:tada: This PR is included in version 1.5.1 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


RYA-119 Added MongoDB Column Visibility (called Document Visibility).

Description
Re-creating PR that was closed before.
This adds a new field to each document called documentVisibility which
uses a boolean expression to determine if the user can access the document.
The boolean expression is in Disjunctive Normal Formal so that the expression's grouping is simplified enough that MongoDB can run Set operations on it to determine if
the document is viewable.  The expression is stored as an array in MongoDB.
The classes in the package "org.apache.rya.mongodb.document.visibility" are pretty much a copy of the classes that are used by "org.apache.accumulo.core.security.ColumnVisibility" so we don't need to have an Accumulo dependency in a MongoDB component.  If anyone is against that then let me know.
Tests
Unit Tests
Links
Jira
Checklist

[x] Code Review
[x] Squash Commits

People To Review
@pujav65
@amihalik
@isper3at
@DLotts

asfbot build


Move pyright.lua to correct location under lspconfig

Pyright.lua was not moved to the correct subfolder after the refactoring, this was my mistake.

@h-michael The recently merged PR had the pyright.lua file under the wrong directory, this fixes it. My bad for not catching the change earlier.


Add upstage document reader

Description
This PR introduces a new document reader tailored for the Upstage API, leveraging the Layout-Analyzer Model. It encompasses essential functionalities for document retrieval, response parsing, and management of various split types.
New Package?
Did I fill in the tool.llamahub section in the pyproject.toml and provide a detailed README.md for my new integration or package?

[x] Yes
[ ] No

Version Bump?
Did I bump the version in the pyproject.toml file of the package I am updating? (Except for the llama-index-core package)

[ ] Yes
[x] No

Type of Change
Please delete options that are not relevant.

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

[ ] Added new unit/integration tests
[ ] Added new notebook (that tests end-to-end)
[x] I stared at the code and made sure it makes sense

Suggested Checklist:

[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas
[x] I have made corresponding changes to the documentation
[ ] I have added Google Colab support for the newly added notebooks.
[x] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[x] New and existing unit tests pass locally with my changes
[x] I ran make format; make lint to appease the lint gods

@logan-markewich Thank you for your feedback and approval. I have pushed the changes with the renamed function as per your suggestion. I will ensure that the code passes the tests before merging it.

After a thorough internal review, I'll resend the PR for your consideration, and I truly appreciate your patience and input throughout this process.


profile add RpcInstanceDetails profile to debug rpc consumption problem

What problem does this PR solve?
Picked from the 2.1 branch, only the RPC profile-related code was selected.
https://github.com/apache/doris/pull/39852
https://github.com/apache/doris/pull/40117
 -  RpcAvgTime:  11.850ms
                          -  RpcCount:  10
                          -  RpcMaxTime:  86.891ms
                          -  RpcMinTime:  15.200ms
                          -  RpcSumTime:  118.503ms
                          -  SerializeBatchTime:  13.517ms
                          -  SplitBlockDistributeByChannelTime:  38.923ms
                          -  SplitBlockHashComputeTime:  2.659ms
                          -  UncompressedRowBatchSize:  135.19  KB
                          -  WaitForDependencyTime:  0ns
                              -  WaitForRpcBufferQueue:  0ns
                        RpcInstanceDetails:
                              -  Instance  85d4f75b72a9ea61:  Count:  4,  MaxTime:  36.238ms,  MinTime:  12.107ms,  AvgTime:  21.722ms,  SumTime:  86.891ms
                              -  Instance  85d4f75b72a9ea91:  Count:  3,  MaxTime:  11.107ms,  MinTime:  2.431ms,  AvgTime:  5.470ms,  SumTime:  16.412ms
                              -  Instance  85d4f75b72a9eac1:  Count:  3,  MaxTime:  7.554ms,  MinTime:  3.160ms,  AvgTime:  5.066ms,  SumTime:  15.200m

Issue Number: close #xxx

Related PR: #xxx
Problem Summary:
Check List (For Committer)

Test

[ ] Regression test
[ ] Unit Test
[ ] Manual test (add detailed scripts or steps below)
[ ] No need to test or manual test. Explain why:

[ ] This is a refactor/code format and no logic has been changed.
[ ] Previous test can cover this change.
[ ] No colde files have been changed.
[ ] Other reason

Behavior changed:

[ ] No.
[ ] Yes.

Does this need documentation?

[ ] No.
[ ] Yes.

Release note

None

Check List (For Reviewer who merge this PR)

[ ] Confirm the release note
[ ] Confirm test cases
[ ] Confirm document
[ ] Add branch pick label

run buildall

run buildall

run buildall

run buildall


dynamometer-infra test failed

The TestDynamometerInfra failed,it may be need revise.
> Configure project :
  Building version '0.1.4' (value loaded from 'version.properties' file).

> Task :dynamometer-infra:test

com.linkedin.dynamometer.TestDynamometerInfra > classMethod FAILED
    java.net.UnknownHostException at TestDynamometerInfra.java:146

3 tests completed, 1 failed

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':dynamometer-infra:test'.
> There were failing tests. See the report at: file:///home/hxh/hadoop/dynamometer-0.1.3/dynamometer-infra/build/reports/tests/test/index.html

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 46s
45 actionable tasks: 7 executed, 38 up-to-date

It continues to pass on TravisCI: https://travis-ci.org/linkedin/dynamometer
Can you provide more information about the failure:

Does it occur consistently or only intermittently?
Can you provide a full stack trace and any other relevant logs from the test report?

Hi, @xkrogen ,

Does it occur consistently or only intermittently?
yes,In my cluster, It occurs every time when i compile dynamometer by ./gradlew build

Can you provide a full stack trace and any other relevant logs from the test report?
sure,The complete log is as follows:

Parallel execution with configuration on demand is an incubating feature.
> Configure project :
  Building version '0.1.4' (value loaded from 'version.properties' file).
> Task :dynamometer-infra:test
com.linkedin.dynamometer.TestDynamometerInfra > classMethod FAILED
    java.net.UnknownHostException at TestDynamometerInfra.java:146

3 tests completed, 1 failed
FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':dynamometer-infra:test'.
> There were failing tests. See the report at: file:///home/hxh/hadoop/dynamometer-0.1.3/dynamometer-infra/build/reports/tests/test/index.html

* Try:
Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Exception is:
org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':dynamometer-infra:test'.
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:103)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:73)
        at org.gradle.api.internal.tasks.execution.OutputDirectoryCreatingTaskExecuter.execute(OutputDirectoryCreatingTaskExecuter.java:51)
        at org.gradle.api.internal.tasks.execution.SkipUpToDateTaskExecuter.execute(SkipUpToDateTaskExecuter.java:59)
        at org.gradle.api.internal.tasks.execution.ResolveTaskOutputCachingStateExecuter.execute(ResolveTaskOutputCachingStateExecuter.java:54)
        at org.gradle.api.internal.tasks.execution.ValidatingTaskExecuter.execute(ValidatingTaskExecuter.java:59)
        at org.gradle.api.internal.tasks.execution.SkipEmptySourceFilesTaskExecuter.execute(SkipEmptySourceFilesTaskExecuter.java:101)
        at org.gradle.api.internal.tasks.execution.FinalizeInputFilePropertiesTaskExecuter.execute(FinalizeInputFilePropertiesTaskExecuter.java:44)
        at org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:88)
        at org.gradle.api.internal.tasks.execution.ResolveTaskArtifactStateTaskExecuter.execute(ResolveTaskArtifactStateTaskExecuter.java:62)
        at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:52)
        at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:54)
        at org.gradle.api.internal.tasks.execution.ExecuteAtMostOnceTaskExecuter.execute(ExecuteAtMostOnceTaskExecuter.java:43)
        at org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:34)
        at org.gradle.execution.taskgraph.DefaultTaskGraphExecuter$EventFiringTaskWorker$1.run(DefaultTaskGraphExecuter.java:248)
        at org.gradle.internal.progress.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:336)
        at org.gradle.internal.progress.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:328)
        at org.gradle.internal.progress.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:199)
        at org.gradle.internal.progress.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:110)
        at org.gradle.execution.taskgraph.DefaultTaskGraphExecuter$EventFiringTaskWorker.execute(DefaultTaskGraphExecuter.java:241)
        at org.gradle.execution.taskgraph.DefaultTaskGraphExecuter$EventFiringTaskWorker.execute(DefaultTaskGraphExecuter.java:230)
        at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$TaskExecutorWorker.processTask(DefaultTaskPlanExecutor.java:123)
        at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$TaskExecutorWorker.access$200(DefaultTaskPlanExecutor.java:79)
        at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$TaskExecutorWorker$1.execute(DefaultTaskPlanExecutor.java:104)
        at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$TaskExecutorWorker$1.execute(DefaultTaskPlanExecutor.java:98)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionPlan.execute(DefaultTaskExecutionPlan.java:623)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionPlan.executeWithTask(DefaultTaskExecutionPlan.java:578)
        at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$TaskExecutorWorker.run(DefaultTaskPlanExecutor.java:98)
        at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63)
        at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:46)
        at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:55)
Caused by: org.gradle.api.GradleException: There were failing tests. See the report at: file:///home/hxh/hadoop/dynamometer-0.1.3/dynamometer-infra/build/reports/tests/test/index.html
        at org.gradle.api.tasks.testing.AbstractTestTask.handleTestFailures(AbstractTestTask.java:596)
        at org.gradle.api.tasks.testing.AbstractTestTask.executeTests(AbstractTestTask.java:477)
        at org.gradle.api.tasks.testing.Test.executeTests(Test.java:520)
        at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:73)
        at org.gradle.api.internal.project.taskfactory.StandardTaskAction.doExecute(StandardTaskAction.java:46)
        at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:39)
        at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:26)
        at org.gradle.api.internal.AbstractTask$TaskActionWrapper.execute(AbstractTask.java:788)
        at org.gradle.api.internal.AbstractTask$TaskActionWrapper.execute(AbstractTask.java:755)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter$1.run(ExecuteActionsTaskExecuter.java:124)
        at org.gradle.internal.progress.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:336)
        at org.gradle.internal.progress.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:328)
        at org.gradle.internal.progress.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:199)
        at org.gradle.internal.progress.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:110)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeAction(ExecuteActionsTaskExecuter.java:113)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:95)
        ... 30 more
* Get more help at https://help.gradle.org
BUILD FAILED in 43s
44 actionable tasks: 6 executed, 38 up-to-date

The pasted log is just a Gradle stack trace and doesn't have any info about the test itself. Can you please include output from the test report ( file:///home/hxh/hadoop/dynamometer-0.1.3/dynamometer-infra/build/reports/tests/test/index.html )

Ahh, I see. It looks like it's having trouble connecting to the mirror site to download the Hadoop tarball. Where you are running this, is the access to the internet restricted? That could explain its inability to download.
When you run the test, can you specify the system property dyno.hadoop.bin.path with a path to a Hadoop tarball you supply? This will instruct the test not to download the tarball.

Hi, @xkrogen ,

is the access to the internet restricted?
my  network is working properly.

can you specify the system property dyno.hadoop.bin.path with a path to a Hadoop tarball you supply?

Is it specified when ./gradlew build is executed?

Is it specified when ./gradlew build is executed?

Yes, like ./gradlew test -Ddyno.hadoop.bin.path=/path/to/hadoop.tar.gz

That's a good way,Thank you very much.

OK, great!


Add more options to procstatus queries

for gh-36

I'm fine with the code. The variable names bug me though, e.g., "query" is vague and "compiledRegex" does not reflect its purpose, as a regex to retrieve dimension value.

Renamed query to pattern. I will address compiledRegex in another pr

@tsheasha can you review this for me? Thanks

ship it!

ship it!


New Dashboard Include Cumulative Tests Either Processed or Sent Out

New Dashboard does not seems to include a number for Cumulative Tests Either Processed or Sent Out as reported by gov.uk

@przemyslawpluta That is correct - the dashboard only displays the number of "tests processed", not those that have been made available or just sent out but not returned or processed.


Update docs today

Remove in-line requests from docs.
Check tutorial instructions to ensure up to date.
fern check documented.
auth-override documented.
iron-man-3 is confusing.

moving to another task for Docs.


Custom metrics with percentile calculations

Expected Behavior
I would like to be able to submit a custom metric from the lambda extension, and then be able to analyze that metric's percentile calculations once the metrics have arrived in DataDog. This would more closely match with my experience submitting Distribution metrics via DogStatsD.
Actual Behavior
I can only aggregate the metrics I generated through the Lambda extension by avg, min, max, sum, and count.
Steps to Reproduce the Problem

Send any custom metric from a Lambda function.
Percentile aggregations are not available:

Specifications

Datadog Go Lambda package version: 43
Go version: provided.al2

Hi @rclark, thanks for submitting this issue.  You can turn on percentiles for your distribution metrics from the UI directly as described here: https://docs.datadoghq.com/metrics/distributions/#enabling-advanced-query-functionality
Hope this helps! I'm going to go ahead and close this issue. If you need more assistance or have more questions, please feel free to reopen it.


Near zero-downtime production upgrades

We need a strategy to update production clusters safely to new Flynn versions. Currently the safest option is backup/restore, but this strategy does not allow writing to databases during an upgrade.

I did try the in-place update several times over the last month (just for testing) on single node flynn clusters. It tends to break the cluster completely, usually resulting in discoverd not being available.
Reboots don't fix the issue, and flynn-host fix --min-hosts=1 also failing with discoverd not being able to return host list.
Long story short, I think self diagnosis/healing should be addressed before this. A reboot (especially on a single node cluster) should be able to self diagnose and self heal in most cases. Such an ability would also improve the trust in Flynn, knowing it is able to recover from most situations. Downtime of 5 minutes is not optimal but not a big pain, but a broken cluster which needs manual intervention is a disaster for production environments.

@IsNull

I did try the in-place update several times over the last month (just for testing) on single node flynn clusters. It tends to break the cluster completely, usually resulting in discoverd not being available.

FYI I believe this was fixed during the migration to libcontainer (#3030, specifically this change). Previously every update would "resurrect" another discoverd job even though one was already running, causing issues.
If you continue to have the discoverd issue with in-place updates on releases since v20160711.0 then let us know.

We are still encountering the same issue with version v20170116.0p2.
see #4002

Flynn is unmaintained and our infrastructure will shut down on June 1, 2021. See the README for details.


Release 10.4.0 rc2

Add new version entry 10.4.0-rc2- this obsoletes https://github.com/owncloud-docker/enterprise/pull/30
This is the enterprise twin of https://github.com/owncloud-docker/server/pull/166

👍 LGTM, let us wait for CI

Build fails with testing cannot be enabled because of invalid enterprise key

obsoleted by #32


Remove deprecated description methods

As of Rust 1.42.0, the description method is deprecated. Instead, the
Display trait should be properly implemented (which is the case here),
so the description of the Error can be printed directly.

Thanks


perf(sourcemap): avoid copying sources

[!WARNING]
This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

#4542  👈
#4541
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @overlookmotel and the rest of your teammates on  Graphite


no api:   numpy.lib.stride_tricks.as_strided

Please support the api: numpy.lib.stride_tricks.as_strided.
as_strided() is powerful,it is  the array's view.
It can do every thing  with einsum(), such as conv .

Thanks for the request!
I love as_strided in NumPy. I used it a ton in grad school. And I take your point about convolutions: in fact, if you look at the file lax_reference.py, which is a NumPy-based implementation of the basic JAX primitives (used for testing against), you can see the convolutions are implemented using stride_tricks composed with einsum. However, in the conv case, the computation that technique generates isn't as efficient as calling a dedicated convolutional kernel. (If it were, folks wouldn't spend so long hand-optimizing or compiler-engineering for convolutions!)
So while you can express conv using as_strided composed with einsum, it may not be very broadly useful for performance reasons.
More generally, as_strided isn't as relevant in JAX as it is in, say, NumPy because while NumPy provides an operational semantics in terms of memory buffers (and hence e.g. views), JAX's (and XLA's) functional semantics are based on values. That is, even setting aside fancy strides, a simple expression like x[:2] produces a new value for which the ultimate physical representation is not guaranteed to be related to (e.g. a view of) the physical representation of x. Choices of what to materialize in memory, how to lay it out, etc. are all left to the compiler.
For example, in classic NumPy the programmer is in charge of execution details, so if we're evaluating something like (x[:10].max(), x[:10].min()) we might like to know that the x[:10] subexpressions aren't going to cause any copies. Instead, we're going to get a view on the original array x and just loop over it twice (once to compute the max and once to compute the min).
But with JAX/XLA, under a jit, not only would we avoid creating copies or intermediate arrays, but the two loops would likely be fused together so that we compute both the max and the min in a single pass over the first 10 elements of x. Those kinds of operational optimizations are only possible because we're not telling the computer exactly what to do, and instead giving the compiler freedom to choose how things are computed.
All that is to say that while we could provide an as_strided API, it wouldn't have the same "how this computation is performed" guarantees that NumPy has. But it still may be useful from an expressiveness point of view (e.g. to build 'windowed views' of data, in a way that's familiar to NumPy experts).
After writing all that, I just noticed that #3171 is essentially the same feature request, and I left a very similar answer there (but more than 2 years ago). So take a look at that for a little more info!
But instead of closing the request like #3171, I think we should consider adding an as_strided API, with clear warnings about the lack of operational semantics guarantees. It may be a lower priority item though...
What do you think?

Thanks for your reply so much!  You are so patient！Nice！@mattjj


Revert "api: augment console.log with proper Sentry breadcrumb"

This reverts commit a60a97a1ab5440e1b8221998a4c84623ded035eb.
Despite the commit order, this was originally written and tested
against the 0.x interface. In 1.x, these identifiers are no longer
available with this import method.
A future commit is planned which will incorporate this functionality
correctly, alongside Flow type declarations to ensure this does not
recur elsewhere.

Merged, thanks!


Potential error in example code.

Should these say stringValue?

Correct. It should have been stringValue. Do you want to contribute a fix?


Implementing Additional Ensemble Algorithms in the Project

Millésime was tested using only two ensemble algorithms - RandomForrestRegressor and Bagging. There are other algorithms we could explore for comparing performance.
The file where the models are being tested is app/services/python/generate_model.py. In addition to the implementation, it would be interesting to have a graph showing the MSE and R2 score for each algorithm to validate our choice among the options.

Hi! Could you assign this for me?

Closed #2

Perfect!