[WIP] Sideposting draft

This PR is a work in progress, following discussions in #1089, #795, and #536.
Questions:

POSTing

Expressing linkage between (fresh) resources in the request document

[x] Should we use JSON pointers or temp ids?

[x] Should temp ids be unique throughout a document or just unique per type?
[x] What key should we use for temp ids? tempid? tid? tmpid?

[ ] Should sideposted resources be within the included member or within some new ad-hoc member? (see #1215)

Expressing linkage between (fresh) resources in the request document and persisted resources in the response document

[x] How do we tell the client which created resources correspond to which resources in the request? Keep the temp ids in the response

[ ] Should the temp id be in each resource object in the response, or should there be a single, document-level temp id => permanent id mapping object?

[ ] What should happen if a client uses ?include on their POST and (for some reason) doesn't list in that param all the relationship paths for which they sideposted resources?
[ ] Should created sideposted resources in the response be within the included member or within some other ad-hoc member? (see #1215)

Supported linkage graphs for sideposting

[ ] Should we put restrictions on the kinds of graphs allowed or should we leave it to the server implementations?
[ ] Should we put minimum requirements of the kind of graphs a server must support?
[ ] Should we offer further guidance on responding to unsupported sideposting requests? (Unsupported graph, etc.)

Updating/deleting related resources

[x] Should we allow updates of related resources while POSTing a new primary resource? Not in 1.1, but maybe down the line
[x] Should we allow deletion of related resources while POSTing a new primary resource? Not in 1.1, but maybe down the line

PATCHing

[ ] Should we allow updates of related resources while PATCHing a resource endpoint?
[ ] Should we allow deletion of related resources while PATCHing a resource endpoint?

Other

[ ] Should we allow bulk creation? (which is simply a matter of relaxing the constraint on the primary data being a single resource object).
[ ] If so, should we allow bulk updates? deletions?

It's well worded and I found it easy to understand.
Is this for creation of new items only? Does it cover update/delete scenarios?

Awesome start!
Some general thoughts:

I agree with @brainwipe that we should think about if/how this applies to PATCH on resource endpoints, and PATCH/POST on relationship endpoints. We don't have to include rules for all those endpoints in this PR, but we should make sure that whatever design we come up with could logically be extended to them.

Where do the created side-posted resources end up in the response? In the included array? If so, just consider that ?include can be used on a POST request to ask for related resources to be returned with the newly-created resource. For instance, if the client POSTs an article with a sideposted author to POST /articles?include=comments, the current spec says they should get back the just-POSTed article, and its comments (from the linkage in the POST), but NOT the sideposted author...maybe that's ok, actually.

I'm not sure how I feel about supporting pointers and temp ids...presumably we would only support one, right? If so, I think I'm leaning toward temporary ids, because they're probably easier for a client to generate than a pointer.

While we can require 403 forbidden from new servers that don't support sideposting, we should warn that 1.0 servers may return a different error. My guess is that most of them will return a 400, if we use an unrecognized pointer format (i.e., one missing an id key). However, if client side ids are in use and the pointer is just {type: "...", id: "client-generated-id"}, the server should technically return a 404, because it should ignore the presence of included, try to link a related resources based on the type/id, and not find the related resource. Some servers may instead create a link to a resource that doesn't exist (if their db doesn't do a foreign key check) and return a 200, without having created the related resource at all. To avoid this, and the 400 vs. 404 vs. 403 discrepancy, it might be good to always use a pointer with a temporary id, even if client-side ids are in use.

Just a tiny note that we shouldn't forget about validation messages. If something bad has happened, we should have a way to point to the attribute which was malformed.

Thanks for providing constructive feedback so quickly! :+1:

Is this for creation of new items only? Does it cover update/delete scenarios? @brainwipe

As it stands, this PR is only about resource creation, though the goal is to agree on a general mechanism to handle "compound requests". As @ethanresnick mentioned, we indeed have to make sure the retained mechanism generalizes to updates. [TODO: add example payloads for resource PATCH]

Where do the created side-posted resources end up in the response? @ethanresnick

The created related resources would probably be sent back within included. I haven't written that part yet as the exact way those resources are matched with those from the request depends on whether we use JSON pointers or temp ids. This is indeed totally in the scope of this PR.

If so, just consider that ?include can be used on a POST request to ask for related resources to be returned with the newly-created resource. @ethanresnick

Fair point about POSTs with ?include query params – I guess this is what initially drove you towards using an embedded member instead? Not sure whether this is an issue or not.

I'm not sure how I feel about supporting pointers and temp ids...presumably we would only support one, right? @ethanresnick

To clarify, I was merely mentioning that JSON pointers and temp ids are two possible solutions for linking resources while sideposting, not that we should support both (which would add unnecessary complexity IMO). JSON pointers seem semantically cleaner, at the expense of ease of implementation. Plus they would force us to introduce some kind of mechanism to link the newly created resources with the request payload. I personally do not feel strongly about this, and neither does @richmolj AFAIK.

While we can require 403 forbidden from new servers that don't support sideposting, we should warn that 1.0 servers may return a different error. [...] To avoid this, and the 400 vs. 404 vs. 403 discrepancy, it might be good to always use a pointer with a temporary id, even if client-side ids are in use. @ethanresnick
Fair point – this would make server implementations easier as well.

Finally, in the current wording, it's not clear to me if a resource in included can link to another sideposted resource in included. @ethanresnick

I intentionally didn't put any restriction on the kind of linking, leaving it to server implementations to decide what is a "supported sideposting request" (which resources are allowed, what kind of graph (possibly none if the server does not implement sideposting on that endpoint), etc.). I agree that it may be useful to offer guidance for responding to unsupported sideposting requests, though the current errors mechanism with JSON pointers should be sufficient. Thoughts?

Just a tiny note that we shouldn't forget about validation messages. If something bad has happened, we should have a way to point to the attribute which was malformed. @a-komarev

Again, the current errors mechanism with JSON pointers should be sufficient. Did you have a specific situation in mind?

Again, the current errors mechanism with JSON pointers should be sufficient. Did you have a specific situation in mind? @beauby

Should we point to the data.relationship.{name}.0 or to included.0?

@a-komarev Let's say you have the following payload (where the first sideposted tag has an invalid label attribute):
{
  "data": {
    "type": "articles",
    "attributes": {
      "title": "Sideposting with json:api"
    },
    "relationships": {
      "tags": {
        "data": [{ "type": "tags", "id": "9" },
                 { "pointer": "/included/0" },
                 { "pointer": "/included/1" }]
      }
    }
  },
  "included": [
    {
      "type": "tags",
      "attributes": {
        "label": nil // This tag will fail validation as the server does not allow nil labels for tags
      }
    },
    {
      "type": "tags",
      "attributes": {
        "label": "REST"
      }
    }
  ]
}

Then the error response would include an error about that attribute, pointing to where it is defined in the request payload, i.e. /included/0/attributes/label.

Awesome! ⛵

I think temp-ids are better (but I'm not totally against pointers).
As others have mentioned, it would be good to think of the update/delete scenarios too. I understand if you don't want to expand the scope too much here, but in many real-world uses all three would be needed.

Again, super-excited to see this PR! 🎉

Cases not covered by the draft yet:
PATCH /articles/123
I would suggest the following rule:

Any side-posted resource with a temp-id is considered a new resource, any other side-posted resource is considered an update. (Again, it would be the responsibility of the server to decide which related resources are allowed to be side-posted/side-updated.)

 Thoughts?

@beauby if you're inferring anything sent with id should be updated, how are you accomodate client-generated IDs?

@richmolj I think you misread, I was suggesting to infer that anything sent with a temp-id should be created (and the rest be updated), which solves the ambiguity that you're raising.

@beauby right, but I don't know if it should be created with the id matching the value of temp-id, or created with an auto-generated primary key.

With an auto-generated id, unless there is a client-generated id.
On Fri, 28 Jul 2017 at 22:38, Lee Richmond<EMAIL_ADDRESS>wrote:

@beauby https://github.com/beauby right, but I don't know if it should
be created with the id matching the value of temp-id, or created with an
auto-generated primary key.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/json-api/json-api/pull/1197#issuecomment-318756317,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACHPYrA03O9EKgOABiJErh-eLyLAtfimks5sSkbWgaJpZM4Ojdwi
.

--
Lucas Hosseini
<EMAIL_ADDRESS>

@beauby how do I tell if temp-id is a client-generated id, or a random uuid? How do I know "I should persist temp-id" versus "I should use temp-id only for linkage"?
For example, let's say I want a new Person to be created. I would use temp-id, but the server would assign an auto-generated primary key underneath the hood.
Now let's say I have a client-generated ID. It would be the exact same payload. How would the server know to persist that client-generated id, if the payloads are the same?
Or similarly, if I were to use id instead of temp-id, how does the server know to create this versus attempt an update to it?
To solve this, I propose a method (name can change) key at the same level:
# create with client-generated id
{ id: 's0m3uu1d', method: 'create', type: 'people' }

# create with auto-incremented primary key
{ temp-id: 's0m3uu1d', method: 'create', type: 'people' }

Not only does this solve the immediate issue, it matches a general RESTful pattern. We usually send our resource to the server with an HTTP verb. Now we are sending multiple resources to the server, and so we will send multiple verbs. This also allows for additional functionality:
# destroy the object
{ id: 's0m3uuid', method: 'destroy', type: 'people' }

# nil the foreign key
{ id: 's0m3uuid', method: 'disassociate', type: 'people' }

You could make the case method could be a known whitelist, or allow additional values (so we could see edge-case implementations like "delete this object and all it's children" etc.)

@richmolj To clarify, temp-ids are never to be persisted, and only exist to indicate linkage within a document.

For example, let's say I want a new Person to be created. I would use temp-id, but the server would assign an auto-generated primary key underneath the hood.

Yes.

Now let's say I have a client-generated ID. It would be the exact same payload. How would the server know to persist that client-generated id, if the payloads are the same?

No it would not – because the payload would contain both id and temp-id.

Or similarly, if I were to use id instead of temp-id, how does the server know to create this versus attempt an update to it?

The proposed rule is "If there is a temp-id, then it is a new resource, otherwise it is an existing resource, so it must be updated".

@beauby OK, making them both be sent for client-generated IDs now makes sense, thanks for clarifying 👍  I do disagree with this approach, though, as I find it more confusing and supports less use cases than something like the method approach above.

No it would not – because the payload would contain both id and temp-id.

I haven't thought too much about the method approach, so I'm not comparing the two, but, just in general: I really like temp-id being present even when id is too. Yes, it makes for a tiny bit of repetition in the client-generated id case (temp-id and id might hold the same value), but it creates uniform processing rules for both the client- and server-assigned id cases by separating what really can be orthogonal concerns—declaring that two unpersisted resources should be linked together and assigning resources a permanent id.
Checklist
@beauby I really like the checklist you made at the top of the issue. Going through it point by point:

Should we use JSON pointers or temp ids?

It seems like most people in this issue marginally leaning toward temp-ids. I agree with that wholeheartedly. I think we should commit to that and move on. As far as the temp-id object format goes, I actually like your original approach of {"temp-id": "...."} rather than {"temp-id": "....", "type": "...."}. I agree with you that the type isn't necessary, and I think including it maybe even obfuscates what these objects' true purpose is, and invites confusion.
I understand what you're saying about it being a little harder to spec without the type key though... I just went through that exercise myself. Still, I'd bite the bullet. Here's how I think think I'd approach that:

Create two subsections under the current "Resource Identifier Objects" section. One called "Permanent Resource Identifier Objects", which holds the content that's there today but clarifies that it's for identifying resources that have already been persisted, and one called "Temporary Resource Identifier Objects", which describes the {temp-id: ""} object format for linking resources that are just being created.

Just link to the definition or temporary resource identifier objects in the sideposting section.

Should we put restrictions on the kinds of graphs allowed or should we leave it to the server implementations?

I don't think I want to put restrictions in the sense of saying that the server is not allowed to support certain graphs. My comment was actually trying to suggest the opposite: that your current wording might be imposing restrictions you didn't mean to impose. In particular, the wording that could be imposing restrictions now is the part that says "The document MUST still respect the full linkage requirement." I believe that would forbid a graph where there's just one resource in included, and it's not connected to the primary resource (i.e., it's really just specifying a second, independent creation).
That said, we should probably have a discussion, conceptually, about whether we want restrictions before we get lost in the details of how to make sure the copy is imposing/not imposing the restrictions we want or don't want.

Should we put minimum requirements of the kind of graphs a server must support?

I edited your OP to add this to the checklist. I hope you don't mind. (I'd never edit a post to change/delete something you wrote, but I wanted to keep all the open questions in one checklist, so I thought just adding something would be ok.)
Basically, the idea would be that, even if we don't prevent servers from supporting any graphs, requiring them to support certain graphs might be useful. We already do this, e.g., by requiring conforming implementations to support sparse fieldsets, and the rationale is that, if all servers support a common baseline of functionality, it gives client libraries much more incentive to implement support for that functionality. I think the same argument would apply here. I also think there's a pretty obvious choice for what an appropriate baseline would be, given the actual, requested use cases that motivated this feature (create an invoice with its items, an author with their first post, etc). Quoting from my earlier comment, that baseline could be:

the extension would be required to support graphs in which the only links between the resources in the request document are relationships from the primary data to embedded resources. (That is, the embedded resources don't link to one another, and the primary data's resource objects, if there's more than one, don't link to one another. But embedded resources can link to other, pre-existing resources.)

Should we offer further guidance on responding to unsupported sideposting requests? (Unsupported graph, etc.)

Let's defer this until #1199 is solved.

Should sideposted resources (as well as the created sideposted resources in the response) be within the included member or within a new embedded member?

Tbh, embedded never totally made sense. I'd say we should just reuse included.

What should happen if a client uses ?include on their POST and (for some reason) doesn't list in that param all the relationship paths for which they sideposted resources?

This is another item I added to the checklist, and it's the one issue to figure out when using included. Should the server omit the sideposted resources from its response (more consistent with current ?include handling) or always include them (probably simpler to implement). Example:
POST /articles?include=comments

{
  "data": {
    "type": "articles",
    "relationships": {
      "author": {
        "data": { "temp-id": "2" }
      }
      "comments": {
        "data": [
          { "type": "comments", "id": "1" },
          { "temp-id": "1" },
        ]
      }
    }
  },
  "included": [
    { /* comment with temp-id 1 */},
    { /* author with temp-id 2 */},
  ]
}

Possible responses:

No author, honoring include:

{
  "data": {
    "type": "articles",
    "relationships": {
      "author": {
        "data": { "type": "people": "id": "1" }
      }
      "comments": {
        "data": [
          { "type": "comments", "id": "1" },
          { "type": "comments", "id": "2" },
        ]
      }
    }
  },
  "included": [
    { "type": "comments", "id": "1", /* .... */ },
    { "type": "comments", "id": "2", "temp-id": "1", /* ... */}
  ]
}

All sideposted resources are always in included; ?include is still honored in the sense of providing including extra resources at the requested paths (here, the comment with server-id 1):

{
  "data": {
    "type": "articles",
    "relationships": {
      "author": {
        "data": { "type": "people": "id": "1" }
      }
      "comments": {
        "data": [
          { "type": "comments", "id": "1" },
          { "type": "comments", "id": "2" },
        ]
      }
    }
  },
  "included": [
    { "type": "comments", "id": "1", /* .... */ },
    { "type": "comments", "id": "2", "temp-id": "1", /* ... */},
    { "type": "people", "id": "1", "temp-id": "2", /* .... */}
  ]
}

The tricky part with option 1 is that, if the no-longer included author had also linked to other sideposted resources, presumably those wouldn't be returned either — unless a different resource that still is part of included linked to those other sideposted resources too. Implementing that seems like a lot of extra complexity for little gain.

Should we allow updates of [related] resources when POSTing a new primary resource?

I'm inclined to say no, but I'd be interested to hear the use cases. My gut reaction is that updating an existing, resource at the same time as creating a new one is rarely going to need to happen transactionally, and I think we should focus sideposting on those cases where things really must happen transactionally (e.g. to satisfy consistency/validation constraints: "a user always needs an address, an author always needs at least one post, an invoice always needs at least one item, so I need to create them together").

Btw, I realize we're discussing a lot of issues in parallel here. Maybe we should create separate threads/issues for the pieces of this that can be considered independently, so the posts here don't get super long and hard to follow. E.g., restrictions + requirements on sideposted graphs could probably be its own issue, as could the interactions with ?include. Then, assuming we've settled on temp-ids (which I really hope), this issue could just focus on sanity checking this approach could extend well to all the other types of requests.

@beauby Nice work here! I really like the approach you're using.
A couple of comments though:

Should temp-id be renamed to tempId or tid so that it's easier to access in languages like JavaScript? Just at first glance, temp-id looks out of place.
Regarding the question if the temp IDs should be unique for the document or per type, and my vote is for unique per type.

Should temp-id be renamed to tempId or tid so that it's easier to access in languages like JavaScript?

I don't want to get bogged down bikeshedding names, but this comment does bring up something important to flag: JSON:API wants to stay away from adding any multi-word keys, so we don't have to take a stand one way or the other on camel casing vs. dasherizing. (See @dgeb's comment here about that.) Therefore, something like tempid or tid would be preferable.

Hi - just throwing in my two cents.

JSON pointers seem semantically cleaner, at the expense of ease of implementation.

Out of curiosity, what's the reasoning behind this? Given that we seem to be moving towards not including type in the linkage, I'm not sure what the practical differences are between {"tempid": "1"} and {"pointer": "/included/1"}.
I'd also like to point out that error responses would use JSON pointers to point to the included data. It would seem convenient that relationship linkage and error detail use the same pointer.

@rpkilby What I meant was that a JSON pointer expresses "this is the chunk of data I'm talking about here", whereas a temp id expresses "there is an abstract concept of temporary resource that I know of and I am referring to one of those".  The practical difference though is that in the response document, it is quite clear that adding temp ids signifies "this is the actual resource that you referred to as X", whereas adding a pointer makes it slightly less clear because it refers to a part of the request document, rather than a defined concept.

Also, re temp ids being easier to implement, my reading is that mostly applies to clients constructing requests. Like, if the client is building up a composite request (say, as the user goes through multiple screens of a form in a CMS), having each part of the form just give its new resource objects an opaque id is easier than it having to know where that resource object will end up in the request document (to make the pointer).

To add support for client creation of IDs over pointers: microservice/CQRS/eventing architectures need the client to create an id because a service accepting a POST/PUT (by strict definition) does not return data. In those cases, the unique identifier that the database uses never leaves the confines of the microservice.

It seems like this discussions has stalled on picking a name for the temp ids. Can I advocate for tid?  I think it is more consistent other common id modifiers - uuid and jid coming to mind.

Isn't it a little redundant to have the type twice on the request?
"relationships": {
  "tags": {
    "data": [
      { "type": "tags", "id": "9" },
      { "type": "tags", "temp-id": "1" },
      { "type": "tags", "temp-id": "2" }
    ]
  }
}

Here we could easily have the data for tags with only the id or tid, without repeating the type for every object, as it is already assigned two levels up in the body.

I'd also like to advocate for tid (or even tempid) since temp-id is not a valid variable / method name in a fair number of languages due to the dash.

FYI, I've opened a couple separate issues to discuss the various sub-problems for sideposting, per my comment here.
@beauby If you feel like I grouped the checklist items wrong, or left any out, in the splinter issues I opened, feel free to open new ones instead or comment on them and I'll gladly update the OPs.

I am coming to this pull request/discussion late but I wanted to share how we at movietickets.com handled "sideposting" with our hypermedia API (json:api as the media type).
A quick background, as director I pitched adopting json:api to the company and development team over 2 years ago and it is our official json based media type for our "Level 3 REST API". We currently have in the teens of 3rd party integrators with a good number of these being major companies with well known name recognition. So far the majority of these integrators have given us and the json;api specification "thumbs up" so that is a good thing. The one area where json:api was not pragmatic was creating a single resource at a time constraint - in the real world this was not going to work. I felt like creating another "wrapper" or "aggregate" resource just for creating a resource and it's related resources was not a real solution - why when the json;api specification itself already defines a standard on how a document can be composed of a primary resource with optional related resources. So after brainstorming we did come up with a solution that we documented for 3rd party integrators but had to add the disclaimer this was non-standard json:api with our "rationale" for being non-standard. Hence why this topic is "interesting" to me at movietickets.com...
The solution we are using for creating a resource and related resources is the following:

We use a standard json:api 1.0 compliant document.
All resources are created atomically, either they are all created or none are created.
If the client wants to create related resources it puts the related resources in the included section and creates "temporary" json;api resource linkage just for the request itself.
This leads to the convention for resource linkage interpretation:

If the referenced resource is in the document, the client intends to create a new resource,
If the referenced resource is not in the document, the client intends the referenced resource already exists in the system.

What we liked about this solution was the POST payload was a json:api 1.0 compliant document and felt straightforward to explain to 3rd party integrators as we did not introduce any new "keywords" to the specification.
An example showing how a 3rd party integrator would create a payment for an order in our system:
{
  "data": {
    "type": "order-payments",
    "relationships": {
      "payment-option": {
        "data": {
          "type": "payment-options",
          "id": "15-2544-CreditCard-Visa"
        }
      },
      "order-payment-items": {
        "data": [
          {
            "type": "order-payment-items",
            "id": "1"
          },
          {
            "type": "order-payment-items",
            "id": "2"
          },
          {
            "type": "order-payment-items",
            "id": "3"
          },
          {
            "type": "order-payment-items",
            "id": "4"
          },
          {
            "type": "order-payment-items",
            "id": "5"
          },
          {
            "type": "order-payment-items",
            "id": "6"
          }
        ]
      }
    }
  },
  "included": [
    {
      "type": "order-payment-items",
      "id": "1",
      "attributes": {
        "Value": "4111111111111111"
      },
      "relationships": {
        "payment-item": {
          "data": {
            "type": "payment-items",
            "id": "15-2544-CardNumber"
          }
        }
      }
    },
    {
      "type": "order-payment-items",
      "id": "2",
      "attributes": {
        "Value": "John"
      },
      "relationships": {
        "payment-item": {
          "data": {
            "type": "payment-items",
            "id": "15-2544-CardFirstName"
          }
        }
      }
    },
    {
      "type": "order-payment-items",
      "id": "3",
      "attributes": {
        "Value": "Doe"
      },
      "relationships": {
        "payment-item": {
          "data": {
            "type": "payment-items",
            "id": "15-2544-CardLastName"
          }
        }
      }
    },
    {
      "type": "order-payment-items",
      "id": "4",
      "attributes": {
        "Value": "05"
      },
      "relationships": {
        "payment-item": {
          "data": {
            "type": "payment-items",
            "id": "15-2544-CardExpirationMonth"
          }
        }
      }
    },
    {
      "type": "order-payment-items",
      "id": "5",
      "attributes": {
        "Value": "2020"
      },
      "relationships": {
        "payment-item": {
          "data": {
            "type": "payment-items",
            "id": "15-2544-CardExpirationYear"
          }
        }
      }
    },
    {
      "type": "order-payment-items",
      "id": "6",
      "attributes": {
        "Value": "33472"
      },
      "relationships": {
        "payment-item": {
          "data": {
            "type": "payment-items",
            "id": "15-2544-BillingPostalCode"
          }
        }
      }
    }
  ]
}

Now you can see why the current json;api constraint of creating 1 resource at a time was not pragmatic for us. BTW we also support this being a resource collection document for "bulk" creation purposes.
I don't know if this helps, I am not saying I am right, I just wanted to share what we are doing in the "real world" to meet our integration needs in the present...

@scott-mcdonald I believe this is roughly where the current draft (which has not been updated to the latest consensus) is headed (modulo the temp ids being explicitly marked as such). Have I missed something?

@beauby No sir you have not missed anything as what I came up with a year ago is indeed almost identical to what your proposal is with the only difference being the simplification of not needing the whole "temp identifier" paradigm instead relying on convention. I just wanted to share to see if that different way of thinking about it adds any value for discussion in your proposal - eliminating tempid basically...
In the end I'll abide by the standard as I am also a .NET implementer of the specification with JsonApiFramework and have been looking forward to a 1.1 version of the specification for a long time. I am still waiting for my revision added as a top-level member of resource pull request to be finalized since December of 2015...

Just want to share how "sideposting" was implemented in Oro Platform: Create and Update Related Resources Together with a Primary API Resource

@vsoroka Your approach for "sideposting" is the exact same approach we came for our hypermedia API at our company. Only difference was for the update (PATCH) use case we did not require a meta attribute for the updating of related resources, instead used the convention if the client "included" a resource then they meant for the included resource to be updated atomically with the primary resource.
I hope our approaches give the authors some insight to some prior art on this topic and it helps them in some way. I do look forward to a standard on this topic someday which would greatly improve json:api in my opinion.

@scott-mcdonald I appreciated reading your background on using the jsonapi spec and sideposting.   My solution in jsonapi_suite is similar to yours and the current spec. For a bit of my background, I wanted to create libraries that facilitated the development of common applications like the one from my tutorial:

This is why I ended up needing an explicit temp-id. The code for that app is something like this:
let department = new Department({ name: 'QA' });
let position = new Position({ title: 'Safety Inspector', department });
employee.positions.push(position);
// saves everything in one request
employee.save({ with: { positions: 'department' });

Your proposed spec could do this too. However, now I have this in-memory position object, and I need to assign it the server-minted id. In other words, I need the response to contain both the temporary identifier I sent to the server and the "real" id so I can update my in-memory objects and ensure my single-page-app has correct state.
This brings up what the current spec in this issue hasn't addressed yet - deletions and disassociations. Scenarios would be something like:

Deletion: Save the employee while deleting a previous position and adding a new one.

Disassociation: Given a position already has a department, remove the department from the position without destroying the underlying record (ie, make the foreign key nil). The most common use case for this is removing tags.

To solve these I added method:
{
  relationships: {
    department: {
      data: {
        id: '1',
        type: 'departments',
        method: 'disassociate' # can be create/update/destroy/disassociate
      }
    }
  }
}

I like this because it's just REST - I'm sending a resource to the server with a corresponding verb.
Hope this proposed spec evolves to the point it can actually power my sample app!

@richmolj Your jsonapi_suite website is impressive - excellent work! Thank you for the example demonstrating the reasoning behind "temporary identifiers". I understand the "temporary identifier" paradigm clearly now, you basically want to reconcile the client-side generated document your passed in the POST to the server-side generated document possibly returned in the POST call.
I think the JSON API specification is great and I want to see it succeed but after using it for real world purposes I believe it needs to evolve in some areas:

Revision as a top-level resource member for optimistic concurrency control and caching purposes. Think of revision as the etag per resource.
Sideposting per this proposal
Batch operations meaning multiple create/update/delete of resources as an atomic transaction.
Filtering standard

If the JSON API specification had these things, dare I say it would be awesome. It is a little disheartening that JSON API specification has not officially evolved at all since 1.0 release...

In 2015 we implemented the following solution.  It has been in use successfully since then (millions of transactions processed), with the following features:

Handled as a POST with a data holding an array, and a meta tag indicating it is a batch.
Each object in data has a meta tag indicating what the operation is for that object
For posted objects, each has a "_locId" - an ID that is local only to the message, so that other objects in the same message can reference them by id. These IDs are deliberately formatted to be non-overlapping with global IDs. They are also returned in the response to allow the client to know which local ID goes with which global ID (if of interest).
In our case, the server handles the request in a single transaction (atomic), which can be general (do adds, then patches, then deletes) or specific (i.e. custom handler), but depending on the nature of the transaction this may not be semantically essential
The response contains a data array with the results for each operation

We recently updated to use a 'BATCH' verb to respect the JSON API protocol (i.e. POSTing a data array was not valid), but that's pretty minor.  Don't ask me why we put underscores before some of the meta attributes. We probably felt we needed a bike-shed inside our bike-shed.
Sample
This sample posts a set of objects having the following structure:

survey-set

survey

survey-feature

POST /survey-sets (updates a survey-set and its related survey and survey-feature objects)
{
    "meta": {
        "is-batch": true,
        "batch-id": "1"
    },
    "data": [{
        "attributes": {},
        "relationships": {},
        "type": "survey-set",
        "id": "8",
        "meta": {
            "_op": "PATCH"
        }
    }, {
        "attributes": {
            "occupancy-type": "assembly,storage",
            "sprinkler-types": "dry"
        },
        "relationships": {},
        "type": "survey",
        "id": "8",
        "meta": {
            "_op": "PATCH"
        }
    }, {
        "attributes": {
            "type": "secondaryEntrance",
            "sub-type-ordinal": 4
        },
        "relationships": {
            "survey": {
                "data": {
                    "type": "survey",
                    "id": "8"
                }
            }
        },
        "type": "survey-feature",
        "id": "688",
        "meta": {
            "_op": "DELETE"
        }
    }, {
        "attributes": {
            "center-lat": 11.111,
            "center-lng": -11.111
        },
        "relationships": {},
        "type": "survey-feature",
        "id": "690",
        "meta": {
            "_op": "PATCH"
        }
    }, {
        "attributes": {
            "sub-type-ordinal": 4
        },
        "relationships": {},
        "type": "survey-feature",
        "id": "2483",
        "meta": {
            "_op": "PATCH"
        }
    }, {
        "attributes": {
            "type": "fireServiceConnection",
            "sub-type-ordinal": 1,
            "center-lat": 11.222,
            "center-lng": -11.222
        },
        "relationships": {
            "survey": {
                "data": {
                    "type": "survey",
                    "id": "8"
                }
            }
        },
        "type": "survey-feature",
        "meta": {
            "_op": "POST",
            "_locId": "_1"
        }
    }]
}

RESPONSE (201)
{
  "data": [
    {
// redacted -- all PATCHed and POSTed objects
    }
  ],
  "meta": {
    "is-batch": true,
    "batch-id": "1",
    "results": [
      {
        "_op": "PATCH",
        "type": "survey-set",
        "id": "8",
        "status": 200
      },
      {
        "_op": "PATCH",
        "type": "survey",
        "id": "8",
        "status": 200
      },
      {
        "_op": "DELETE",
        "type": "survey-feature",
        "id": "688",
        "status": 204
      },
      {
        "_op": "PATCH",
        "type": "survey-feature",
        "id": "690",
        "status": 200
      },
      {
        "_op": "PATCH",
        "type": "survey-feature",
        "id": "2483",
        "status": 200
      },
      {
        "_op": "POST",
        "type": "survey-feature",
        "id": "2484",
        "_locId": "_1",
        "status": 201
      }
    ]
  }

I have been struggling a bit with the name tempid. I think of this as a document-scoped secondary key that might be more clearly named key, which also has the advantage of not trying to fake a union of two words into one.
What do others think of calling this field key? I've been exploring this concept for use with my operations proposal, and would like to formalize it in the main 1.1 spec outside of the confines of sideposting.

Calling it key, or something more specific (such as batch-key or similar) seems more intuitive to me than temp-id. :+1:

Another logistics challenge on the server side is cross-referencing POSTs with temp ids.
For example (contrived):
POST: blog-post.first-comment => comment
POST: comment.post => blog-post

If this is to be created on the server in one batch, then both objects
will have a temp id, and one of the objects must be created first - this
is further compounded if the first-comment property is required, but that
truly is a completely out-of-scope issue for this.
This is not in scope for the message structure, I know, but one solution
is to 'split' one of the POST operations into a POST and a PATCH:
POST: blog-post.first-comment => null
POST: comment.post => blog-post
PATCH: blog-post.first-comment => comment

If this is a client responsibility, then the message format needs to support
having more than one operation for the same object (the blog-post in this
example).
There are other examples we've had where ordering of the operations
is important and also includes a need for multiple PATCHes to a pivot
object:
PATCH a to remove a=>b
DELETE b
POST c (b and c can't co-exist)
PATCH a to set a=>c

What do others think of calling this field key?

Honestly, key seems less clear to me, because it's so generic. tempid is exactly what it sounds like: a unique id applied to the resource temporarily (i.e., only for the duration of this request/response). That said, I care way more about picking something than I do about the exact name, so I'm fine going with key, esp. if that would work better for operations.

The reason we chose _locId (local-id would be clearer) is because in our
case the ID is assigned in the client when the object is created which can be
some time before it is submitted (i.e. the user creates something and edits
it for a while before submitting).
That's outside the scope of the issue here, but considering the context of the
use case here is perhaps valuable.
That said @ethanresnick , it's important to move this forward and key
would work just fine for us as well.

I agree with @ethanresnick - I prefer the more descriptive tempid than the generic key, but happy with anything that moves this along.

It seems like this discussions has stalled once again on picking a name for the temp ids. We were close to concluding tid was the way to go.  Can I continue to advocate for tid? I think it is more consistent other common id modifiers - uuid and jid coming to mind.

I prefer tempid as well.
I look at this from another perspective too: Whatever is decided means the name will most likely have to be "reserved" similar to type and id. So, in theory one wouldn't include this field name in their attributes to avoid conflicts. I feel like key (e.g. key/value objects) might conflict with existing implementations of JSON (just a feeling, no data to back this up).

Whatever is decided means the name will most likely have to be "reserved" similar to type and id.

This is not necessarily true - we do not actually need to reserve this as a field in the attributes/relationships/type/id namespace. It is analogous to allowing an attribute named meta, which is fine although perhaps odd. This member should never be used for sorting, filtering, etc.
Thanks for everyone's feedback re: key. I will follow through with my operations proposal to share another perspective. I don't think we'll be merging either proposal until both are considered together.
In the mean time, there are some other general concerns I still have about this PR:

We can't make the presence of key / tempid a requirement for linkage. Client-generated ids must be allowed for "included" resources, just as they can be allowed for primary resources. So type / id must also be allowed for linkage.

This proposal provides very little utility without also allowing a corresponding update (i.e. PATCH) solution that can also create resources. Take for example an invoice that "contains" invoice items. If an invoice can be created together with invoice items, it should also be able to be updated together with new invoice items. The nuance of specifying invoice items to create vs update seems thorny, especially when paired with the need to allow for client-generated ids.

I also agree with @ethanresnick  that key is not a clear self-documenting name for the client supplied document scoped temporary identifier for creating related resources. I was thinking about this proposal over the Thanksgiving break and wanted to throw out some thoughts/ideas that may help or may not help and be total crap:
Idea #1 - Keeping with the current design - what are we really asking the server to do? The client wants the server to return the original client created temporary identifier it created for document scoped linkage purposes with the newly created server identifiers. So with that said, any of the names like temp-id, tempid, tid, original-id, client-id, local-id, etc. in the created/updated resources would work and seem self-documenting.
Idea #2 - Thinking differently - what are we really asking the server to do? The client wants the mappings of client generated identifiers to server generated identifiers. And BTW, not all clients care about the temporary client-generated identifier mappings being echoed back - so only if the "client" asks for the mappings having some new document-level "section" that contains the mappings that we have been debating about in this proposal. Not sure what the section would be called so just picking mappings to pick something for now, it would look something like the following:
{
  "links": {...}
  "data": {...} or [...]
  "included": [...]
  "mappings": [{"type": "articles", "client-id": "123", "server-id": "567"}, or
               {"type": "articles", "client-id": "123", "id": "567"}, or
               {"type": "articles", "tid": "123", "id": "567"}, or
               {"type": "articles", "temp-id": "123", "id": "567"}, or whatever but something along these lines
               ...]
}

Maybe it goes beyond mappings but the results of the operation as well such as if it was created or updated or deleted(?) so something like the following:
{
  "links": {...}
  "data": {...} or [...]
  "included": [...]
  "operations": [{"type": "articles", "client-id": "123", "server-id": "567", "result": "Created"}, or
               {"type": "articles", "client-id": "123", "id": "567", "result": "Created"}, or
                 {"type": "articles", "tid": "123", "id": "567", "result": "Updated"}, or
                 {"type": "articles", "temp-id": "123", "id": "567", "result": "Created"},or whatever but something along these lines
                 ...]
}

This type of document would only be returned as the response of POST, PATCH. This idea seems self-documenting and explicit but maybe is overkill. Also I am not sure how a client could "ask" for this or it is always returned by the server...
Idea #3 - Maybe this "mapping" feature is not the responsibility of the specification at all but instead is an implementation detail of the hypermedia API server if this feature is desired. The hypermedia API domain model as designed could contain an attribute (or meta) that is intended to be a unique identifier initialized by a client just for this mapping purpose and is echoed back to the client or something along that lines.

@dgeb Thanks for weighing in here, Dan. I think it's really great + important to get your input as this is evolving. A couple questions:

We can't make the presence of key / tempid a requirement for linkage. Client-generated ids must be allowed for "included" resources, just as they can be allowed for primary resources. So type / id must also be allowed for linkage.

I'm not sure I follow this. Yes, we have to allow client-generated ids for all resources (even in included), but does that mean we can't also require that sideposted resources have a temp id, and that that temp-id be used for linkage, whether or not they have a client-generated id? That's what I proposed in my fourth point here, and it has some other advantages.

This proposal provides very little utility without also allowing a corresponding update (i.e. PATCH) solution that can also create resources... The nuance of specifying invoice items to create vs update seems thorny, especially when paired with the need to allow for client-generated ids.

Maybe this looks easier to specify in light of my answer above, but the permutations seem relatively straightforward:

Create primary resource(s) and create related ones (the classic sideposting case): do a POST with the primary resource to create in data and related resources to create in included (with tempids).

Update primary resource(s) and create related ones (first case you mentioned): same as above, except the method is PATCH.

Update primary resource(s) and update related ones (second case you mentioned): A PATCH, where the resources to update are in included. These resources to update are distinguished from resources to create (i.e., the case above), because they wouldn't have a tempid. This is the solution @beauby proposed here. We could allow this case in 1.1, or just say that servers should error on included resources with no tempids; then, we can specify this behavior later.

Create resources and update related ones: this is the last case in the matrix of (create, update) x (primary, included). If we wanted to support it -- and I'm not sure we really do -- we could use the same approach as above to differentiate the related resources that are updates vs creations.

Create resource and update an existing resource that's being linked to the new one:

@scott-mcdonald Thanks for weighing in as well! I'm not opposed to the idea of having a single mapping of tempids to permanent ids in the response (I actually explored that earlier). That said, I'm also fine with putting the tempid in each resource object in the response.
I've added new items to the issue checklist to cover these two options. For now, though, I don't think where to put the tempids in the response is the real sticking point, so we can probably come back to that at the end.

Yes, we have to allow client-generated ids for all resources (even in included), but does that mean we can't also require that sideposted resources have a temp id, and that that temp-id be used for linkage, whether or not they have a client-generated id?

@ethanresnick it feels like a strange requirement to force the usage of temporary ids just to identify included resources that should be created, especially when paired with a primary resource that's created with just a type and id. And then those temporary ids would have to be managed and returned by the server in the response. And all this ceremony is just to mark included resources as "to be created", when none of this is necessary when posting resources as primary data.

And all this ceremony is just to mark included resources as "to be created"

@dgeb there is a separate reason for temp ids, aside from the one you note, explained here.
On a separate note:

when none of this is necessary when posting resources as primary data

This is why I propose adding method (open to other names) - pair a resource with a verb just like you would for primary data. This allows us to support delete and disassociate as well, explained at the same link above.

And all this ceremony is just to mark included resources as "to be created", when none of this is necessary when posting resources as primary data to the same server that accepts client-generated ids.

@dgeb It's also for backwards compatibility, as I mentioned upthread:

if client side ids are in use and the pointer [to the included resource] is just {type: "...", id: "client-generated-id"}, the server should technically return a 404, because it should ignore the presence of included, try to link a related resources based on the type/id, and not find the related resource. Some servers may instead create a link to a resource that doesn't exist (if their db doesn't do a foreign key check) and return a 200, without having created the sideposted resource at all. To avoid this, and the 400 vs. 404 vs. 403 discrepancy, it might be good to always use a pointer with a temporary id, even if client-side ids are in use.

This is why I propose adding method (open to other names) - pair a resource with a verb just like you would for primary data. This allows us to support delete and disassociate as well, explained at the same link above.

@richmolj That is, indeed, more powerful. Once we add all that power, though, it's starting to feel like the full operations extension might be more appropriate for those use casses. Still, I don't want to foreclose the method option in the future. (My "inline operations" proposal from #795 was in the same vein.)
My preferred solution for 1.1 would be to always have the resources in included be created (i.e., only supporting the top two cases from here), and always have tempids. Then, if we want to support other actions on the included resources down the line, we can add a method key (which defaults to "create" if missing) or we can go the route of treating a missing tempid as a sign to update rather than create. Either way, we've preserved our options and the extensions compose nicely.

there is a separate reason for temp ids, aside from the one you note, explained here.

@richmolj I was specifically talking about a scenario in which a server supports client-generated ids. Sorry if I wasn't clear.

Once we add all that power, though, it's starting to feel like the full operations extension might be more appropriate for those use cases.

@ethanresnick I will make this a public proposal within a week. I can't expect others to discuss something that's not concrete and public.

@ethanresnick I will make this a public proposal within a week. I can't expect others to discuss something that's not concrete and public.

@dgeb Sounds good. I also haven't seen the proposal, so I won't speak more for it :) Also, I updated my comment above to explain the tempid thing better. Lmk if that makes sense.

Also, I updated my comment above to clarify my current position on tempids, since the old framing didn't really get at it. Lmk if that makes sense.

@ethanresnick Thanks for clarifying. I should say that there would be no ambiguity if side-posting was opted into explicitly as an extension (as per #1207). A server that accepts the extension would know exactly what to expect.

That is, indeed, more powerful. Once we add all that power, though, it's starting to feel like the full operations extension might be more appropriate for those use casses.

I look forward to seeing the full operations proposal. That said, you could make the case that this entire PR should go away in favor of the operations proposal - that was the direction https://github.com/json-api/json-api/issues/795 was going before https://github.com/json-api/json-api/issues/1089 was created to try and separate the two.
So what's the goal of this PR? I think there are three ways to define the goals of "sideposting" independent of operations:

Abstract: Ability to perform the same actions we already perform on primary data (GET/POST/PATCH/DELETE) at any level in the graph.
Practical: Support simple, common web forms like shown in https://github.com/json-api/json-api/issues/1089 and this tutorial. This is actually how I got involved - ember-data couldn't support simple forms, and all discussion of the issue led here.
Simple/Intuitive: From what I've seen so far, the operations proposal will be more complex than sideposting (such is its nature, as it supports more complex functionality). Sideposting simply mirrors the same payload we see on GET (tempid tangent aside).

The first two bullet-points would require a way to delete data. After that, there's a clear separation from how I think of operations - more complex scenarios where I need partial transactions, ordering of operations, etc...RPCish stuff that diverges from the general premise of "mirror the existing spec".
To the third bullet-point - can we support DELETE while still keeping things simple? I believe so - what could be simpler than "secondary data works the same way as primary data"? We interact with primary data by pairing a resource with a verb. I suggest pairing sideposted data with a verb as well. This seems intuitive (a familiar REST pattern and no implicit side-meaning of tempid), supports more scenarios (DELETE at minimum), and draws a clear distinction from operations.
One final note. Consider the nested web forms I have referenced above. If this PR does not support DELETE, that means my form submission will submit one payload on create/update and a significantly different payload if the user marks an item for deletion. That's going to be a pain point for developers and library authors alike - at that point I'd prefer avoiding sideposting altogether just to keep things consistent.

I should say that there would be no ambiguity if side-posting was opted into explicitly as an extension (as per #1207). A server that accepts the extension would know exactly what to expect.

@dgeb That's true, but, of course: 1) it would block sideposting on us figuring out that negotiation mechanism (which in turn might be blocked on other details of extensions and/or profiles) and 2) there's still the issue that APIs using server assigned ids are going to need temp ids. For those reasons, I'd much, much prefer to keep @beauby's proposal basically as is, address the remaining checklist items, and merge in some version of sideposting (that addresses just creating related resources on POST and PATCH) ASAP, and call that 1.1 beta. Ideally, we could do that by year end. Thoughts?
Even if you disagree with the plan above, can we try to define a concrete timeline + next steps progression for sideposting? (And hopefully for extensions/profiles too, which I believe I owe you a message on — although that conversation may be more stuck than sideposting, so perhaps it makes sense to proceed with sideposting first, which actually has been requested much more frequently.)

That said, you could make the case that this entire PR should go away in favor of the operations proposal - that was the direction #795 was going before #1089 was created to try and separate the two.

@richmolj I still think the reasons we split out this proposal out apply. Just to reiterate them for others in this thread (by quoting you and me from #795):

most people are looking for a solution to [Creating/updating a single resource with its relations in one go]. I think a minority care about [generically grouping multiple operations into a single request]. But arriving at a solution on [the former] is being blocked by [the latter].

Moreover, re the argument that we should do only operations because sideposting is duplicative:

we shouldn't take the "it's duplicative" argument to its extreme, as that would suggest removing most of the base spec, since all one really needs is operations.

I.e., everything can be read as duplicative of operations, but that's a whole different model, and it shouldn't prevent us from extending the existing payload structure to cover common cases.

The first two bullet-points would require a way to delete data.

This idea — basically, that a sideposting MVP requires deletion support — seems to be at the heart your comment. I'm curious to hear what others think about this.
As I see it, a simple rule for now that says "create the client-provided resources in included on POSTs and PATCHes" would solve a bunch of use cases and doesn't preclude at all, or make any messier, expanding support to cover a proposal like #1089 down the line.

@ethanresnick - well summarized. Let me take one more crack at clarifying my comment, since lack of delete support means my libraries wouldn't be able to use any of this (😭 ).

most people are looking for a solution to [Creating/updating a single resource with its relations in one go]. I think a minority care about [generically grouping multiple operations into a single request]. But arriving at a solution on [the former] is being blocked by [the latter].

I think the reason most people are looking for this solution is because it's very common to have a nested form in a single-page javascript application. It's also very common for those forms to have a delete button (it's in the examples I've given, here and here). So lack of delete support, to me, means jsonapi still cannot support common web forms and the original problem is not being solved. I think it would be a poor developer experience to say, "if your form has a delete button do it this way, otherwise do it this completely different way".
This also significantly impacts library developers. Let's say you're writing a client and want an API for "save this object with its relationships":
employee.save({ with: { positions: 'department' } })

The library author needs to go down an entirely different codepath if a position or department has been deleted. At that point, this PR is actually counterproductive - a library author would want to generate the same operations-style payload every time, rather than switching payloads conditionally.
I still very much want sideposting separate from operations, because the payload would be simpler and more intuitive...but if delete is not supported I'm not sure that's a viable option.
I'd actually like to ask the opposite - why wouldn't we want to support delete? It means we no longer have the implicit meaning of tempid (meaning, no create/update inference because you'll have an explicit verb saying create or update), enables more server-side opportunities (we could have a base list of verbs, but stuff like disassociate could be optionally supported by the server), has symmetry with the way we interact with primary data (the RESTful paradigm of pairing a resource with a verb)  and a clear separation of sideposting vs operations ("use operations whenever pairing a resource with a verb is not enough, such as when you need to control the order of writes"). What's the downside?

I'd actually like to ask the opposite - why wouldn't we want to support delete?

Supporting delete = bigger scope = more to find consensus on = who the hell knows when this will ship. At least that's how I've been thinking about it; getting consensus on anything re sideposting has been so hard that I really want the narrowest version possible.

So lack of delete support, to me, means jsonapi still cannot support common web forms and the original problem is not being solved... The library author needs to go down an entirely different codepath if a position or department has been deleted. At that point, this PR is actually counterproductive - a library author would want to generate the same operations-style payload every time, rather than switching payloads conditionally.

I definitely don't want something that's counterproductive. But, I'm not sure it's as bad as you describe. Delete is really two things: 1) delete resource entirely, or 2) disassociate it from the relationship. In the case of 1), how often does that need to be done in the same request? IMO, the main point of sideposting is to let creations that need to happen transactionally happen transactionally, and it seems like deleting a related resource (and retrying delete if needed) can usually happen in a separate request without issue. If the delete needs to happen for (e.g.) foreign key integrity, the db may take care of it on its own. For the second case (disassociation), there's an obvious way to do that today, which is to just replace the whole relationship by sending back data with the new contents. That's slightly more likely to run into concurrency issues than having separate updated_/removed_/added_data buckets would, but it's not as though having the separate keys makes you immune and, to fully prevent the problem, you need something like #600 either way. Also, replacing data isn't practical in some cases when the relationships contents are huge, but these are relatively rare. Don't get me wrong: I'm not saying that the absolute minimal sideposting approach I'm proposing is a cure all, just that it seems like a win over the status quo, and something we can build on, and that it's narrow enough that we won't regret adding it.

It means we no longer have the implicit meaning of tempid (meaning, no create/update inference because you'll have an explicit verb saying create or update)

I'm not proposing this switching logic for the minimal version. Quoting myself:

My preferred solution for 1.1 would be to always have the resources in included be created (i.e., only supporting the top two cases from here), and always have tempids. Then, if we want to support other actions on the included resources down the line, we can add a method key (which defaults to "create" if missing) or we can go the route of treating a missing tempid as a sign to update rather than create. Either way, we've preserved our options and the extensions compose nicely.

Hey some feedback from someone using

noSQL (MongoDB) and Elasticsearch databases.
a Rails 5 API  communicating with a Single Page React App using json:api and...
... @richmolj jsonapi_suite library featuring sideposting with method and tempid

The method concept allows to clearly and intuitively make the distinction between 2 different types of data model and their associated operations. I'm quite in favor of it.
I'm using MongoDB/mongoid as my primary DB/ORM, which is therefore quite close to JSON overall and my use cases also involve embedded data. Sometimes, there will be a need to mix embedded-relationships with foreign relationships and the method seems to be the best way to clear things up here
For embedded data like person <-> educations ...
person: {
  educations: [
    { title: "Some fun at MIT !", school_id: '5a207c88aba9cfa1a2f5943f', start_date: "2017-11-05", ... }
    { title: "Bored in High School", school_id: '5a207c88aba9cfa1a2f5943f', start_date: "2015-11-05", ... }
  ]
}

the sideposting requirement is to be able to create/destroy those associations whereas for foreign key based relations like education <-> school
education: {
  educations_id: '5a207c88aba9cfa1a2f5943f'
}
schools: {
  id: '5a207c88aba9cfa1a2f5943f',
  name: "Massachussets Institute of Technology"
}

...the requirement is to be able to associate/disassociate. And that naming makes it super-clear. This could also leave some flexibility for some people who might need even more special types of persistence (I'm quite often struggling because people designed their libraries for SQL and there is little support to extend the behavior to support more fashioned types of persistence like embedded date, etc.)
Actually in some cases of nested relationships (which is already the case in my example above) it makes sense to keep the method even in included data payloads. I have actually almost got to the point where I actually needed that in my application. Here is a gist of what I'm thinking about
And in those cases of associate/disassociate, yes it totally makes sense that Patch be able to updated those nested/embedded resources.
My answers to the draft suggestion would be

Yes use tempid
Use included, no need to invent something else
tempids in each resource as needed, and unicity of tempids on the whole payload
using ?include in POST : it should behave like ?include in normal GET operations IMO. It could be that sideposted resources are not persisted immediately (think of a 202 accepted). But actually I'd rather the spec stayed vague about that at this stage
location of created sideposted resources in the response : depending on 201, 202, etc. the answer is not really easy. Leave it vague ?

Regarding linkage graphs, @ethanresnick made some good points. I haven't got anything else to say / haven't though a lot yet

for PATCH + allow updates/destroy of related resources, this makes a lot of sense for embedded data. yes for me

Bulk creation/deletions : I would like the json:api specs to go that way, but this feels quite too much already (usually bulk writes means there could be a top-level document (even if something entirely artificial) grouping sub-documents. Also this is outside the scope of sideposting ? (but naturally if json:api is extended to bulk writes and we have sideposting, we should have bulk-sideposting :-) )

Supporting delete = bigger scope = more to find consensus on = who the hell knows when this will ship.

@ethanresnick I could not be more sympathetic to this point ❤ . I understand completely and agree - if we get stuck on this point, let's not hold up this PR.  That said - I've actually not heard any disagreement or downsides yet, and this was in scope for the prior sideposting issue that got 30 upvotes as well as the original scope of https://github.com/json-api/json-api/issues/795. We have something simple that enables more functionality and is already working for many real-world applications - let's at least ensure we have disagreement here before deciding it shouldn't be part of the debate.
Does anyone disagree with these points?:

Conceptually, sideposting should support DELETE.

To disagree with this, your opinion would be that sideposting should support create/update but use the separate operations API for DELETE.

If sideposting is going to support DELETE, the way to do so is to pair the sideposted resource with a verb.

I'm open to whatever naming and whitelisting of verbs here. The spec could even be super vague and leave verbs up to server implementation (which might prevent bogging this issue down with debate).

I would be surprised if there's disagreement on the first bullet-point. The second might be trickier, but until it bogs us down I believe it should remain in scope.

I'm not proposing this switching logic for the minimal version.

Thanks for this note. So to clarify - this PR would actually only support create, but NOT support update or delete. Is that your position? So we would support web forms, but only for the initial create...? I sympathize with wanting to trim scope to ship something, but that doesn't reach the level of MVP in my opinion.
Regarding your specific points on DELETE:

One of the primary users of jsonapi is ember-data, which has already been making decisions based around the concept of 'single request'.
I have an ETL process sending large amounts of data to my API - separate requests would complicate the client code and slow down my ETL.
I would have to write + test a separate endpoint just for the delete, versus piggybacking on the same sidepost request.
Sometimes these do need to run in the same transaction, such as "the parent requires at least one child to be valid". For example, given a menu requires a fish of the day, you can only delete the prior fish of the day if you are also adding a new one.
The data replacement is a non-starter for me, as it's immediately not viable with a basic "blog post has many comments" example (I had this problem the second time I wrote a JSONAPI, so it's hard for me to see this as rare).

This is all to say, of course we can come up with workarounds, but I have a hard time seeing these alternatives as preferable to a simple resource/verb pairing that's already the solution we use for primary data. My main argument here is "keep things simple".

That's slightly more likely to run into concurrency issues than having separate updated_/removed_/added_data buckets would, but it's not as though having the separate keys makes you immune and, to fully prevent the problem, you need something like #600 either way.

To be clear, I don't think anyone is proposing separate buckets anymore. The thought is to take this same PR and add method (or equivalent) to the relationships payload, as explained here.

@richmolj Can you give an example of your proposed sideposted DELETE request for my edification?

@scott-mcdonald you can see the full payload + explanation at https://jsonapi-suite.github.io/jsonapi_suite/concepts#sideposting

@richmolj Thank you! In that example the HTTP verb is PATCH and you have a method enumeration to specify the verb to perform to make it explicit what the client is requesting on the related resource - I like this approach! It is additive to the current specification and in my opinion simple and explicit  - if this PR adopts this fundamental approach having method in relationship then it makes sense to add all the verbs you demonstrated in your example (create, update, destroy, disassociate). Anyways that is my opinion.
Also I advocate for tid over tempid or temp-id if anybody is asking.
Also if the entire request is an atomic transaction, can we loosen the primary data constraint of it being a single object to being an array of objects for bulk sideposting? Or am I reaching for the stars...
BTW in your example you stated it is creating two related tags, but it looks like only one tag is in the json:api document.

can we loosen the primary data constraint of it being a single object to being an array of objects for bulk sideposting?

@scott-mcdonald though I certainly support this as a separate issue, going with the method approach would allow your primary data to be a dummy entry resource, then sidepost onto it. So the functionality would be enabled if not ideal.

@richmolj I think my use of bulk sideposting was not the best word choice. I simple meant the primary data can be an array of resources instead of a single resource for bulk change as an atomic operation which isn't what this PR is about.
I am also wondering if this method approach is essentially the same as what the owners are calling operations and thus will be solved by that proposal? I'll guess we will find out soon, maybe...

There is now a proposal for local ids, lids, that will make side-posting easier.
If you like this to be implemented, please support it here: https://github.com/json-api/json-api/pull/1244

Have there been any recent updates on this or Operations (#1254)? It's not clear (at least, to someone new to all this) whether this is still being considered, or whether Operations would obviate the need for sideposting entirely.

Hi there 😊 I would also be interested in knowing were we're at regarding this subject. Is work stil happening on this front? Anyway we can help? ❤

@simonc It's still in the plans. The next step is to add lid (local-id) to the spec, in 1.2.
You can follow along here:
https://github.com/json-api/json-api/pull/1244
(I know it's being actively worked on)

Hi there :)
First of all, thank you very much about your hard work which created that awesome spec. I discovered it a few months ago and I'm currently implementing it on Laravel to support both spec versions. That's why I wondered if there was some recent news about bulk operations.
Thanks in advance!

how does the atomic operation extension influence this?

how does the atomic operation extension influence this?

Is there any progress being made regarding this feature? Is there an active discussion group somewhere else?

How does the Atomic Operations extension not solve the problem sideposting was intended to achieve?  That extension is planned for specification version 1.1.

How does the Atomic Operations extension not solve the problem sideposting was intended to achieve? That extension is planned for specification version 1.1.

I think it fully does.
It should be possible to implement sideposting as proposed here as an extension, but I wouldn't be sure why we would need a second solution for the same problem.
I think this proposal was valuable input. Exploring this option helped to iterate on the problem, which result in Atomic Operations. This merge request could be closed now in my opinion.
@beauby Do you have any concerns closing it? Does someone else has?

I agree that this behavior could be implemented, but I still think sideposting would be valuable for the following reasons:

Easier to implement in client- and server-side libraries, due to its smaller scope
Doesn't bundle all requests up at the same endpoint. Commonly used app performance monitoring systems often use the endpoint name to separate out different resource types. For example, Skylight, that we're using, will rank endpoints by performance out of the box. Would need extra work for it to analyze a single-path API.
If all requests are POST to /operations, you cannot use many of the inherit assumptions about GET, POST, DELETE, etc. For example, just looking at the method, you don't know if it's safe to retry the request. Similarly, our client is configured to send GET requests to replica slave nodes if no POST/PUT/DELETE has been sent in the past 10 seconds. Such logic would need to be customized. We're also using the method type for a bunch of other internal logic, that wouldn't work with operations extension using only POST.

One example from our application is the Invoice model, with one or more embedded InvoiceItem models. These invoice items are embedded at the database level, so makes sense to create/update/delete them via the parent invoice. In this case, sideposting would be very useful.
PATCH /invoices/123
Content-Type: application/vnd.api+json

{
  "data": {
    "id": "123",
    "type": "invoices",
    "attributes": { "invoice_date": "2022-01-15", "amount" : 250 },
    "relationships": {
      "invoice_items": {
        "data": [{ "type": "invoice_items", "id": "14" }, { "type": "invoice_items", "lid": "a1" }]
      }
    }
  },
  "included": [
    {
      "type": "invoice_items",
      "id": "14",
      "attributes": { "label": "Tax", "amount": 50 }
    },
    {
      "type": "invoice_items",
      "lid": "a1",
      "attributes": { "label": "Hammer", "amount": 200 }
    }
  ]
}

lid is in 1.1, so I think this needs updated to use that instead of temp-id? https://github.com/json-api/json-api/pull/1244

With atomic operations, we're forcing all requests to go through one endpoint

@sandstrom Is it necessary?
My understanding is that atomic operations requires all requests to use POST and mention the extension name in the Content Type itself. Not sure if its limited to a single end-point.
cc: @dgeb

Closing this as I am not actively working on it, and a lot has changed since I stopped.


Upgrade kube-router to v1.5.0

Proposed Changes
This PR upgrades the kube-router dependency to v1.5.0.
It points to the k3s-io fork of kube-router because the netpol dual-stack feature is not integrated in upstream yet.
Types of Changes

Verification

There are no new tests but the upgrade should not break anything.
Testing

Linked Issues

https://github.com/k3s-io/k3s/issues/5715
User-Facing Change

NONE

Further Comments

This PR is not going to accomplish #5715. We have to update the k3s-io/kube-router to v1.5.0 first.

@dereknola it looks good to me? https://github.com/k3s-io/kube-router/tree/k3s-v1.5.0 is present and appears to be what the replace entry is pointing at.

The branch pointed to is now a horrible amalgamation of commits. Why don't we just update the master branch in our kube-router fork, then reapply the dual-stack commits. This is going to be a nightmare to maintain down the line.
...
This branch is 67 commits ahead, 81 commits behind cloudnativelabs:master.

@brandond I opened a PR to do that https://github.com/k3s-io/kube-router/pull/17

Okay @thomasferrandiz please update this PR with the commit a43bcd8511d263855338a2dcc4d7f409c918077b, I have rebase the v1.5.0 branch to properly track close to upstream, we are now only:
...
This branch is 2 commits ahead, 15 commits behind cloudnativelabs:master.
...
Which is much better :)

Rebased to fix conflicts

CI is green other than s390x which is out due to hardware issues. Merging.


Revert "Fix outcomes and 13 week calculation for Maternity-Paternity calculator"

Reverts alphagov/smart-answers#3022

Closing this for #3121 - we need to keep the content changes that were in #3022


Project Manager: Remove parent info in AssetItem's doc generation

Remove fields about parents in AssetItem.to_doc( ), so the comparison for update info doesn't identify AssetItems as needing update due to mismatching fields.
While trying to device some Python way to programmatically create new tasks, I noticed the update data Program manager was writing to Mongo seems a bit excessive. Digging into it, it seems in AssetItem.update_data(),  always identify the item to be updated because of the additional field 'parent' in doc and 'parents' in data.
Here's some debugging printouts for the two doc objects compared for update info, only with some reformatting for easier comparison.
[DEBUG] current item doc:
{
	'_id': ObjectId('617c63c9797e4a6b9d76c208'),
	'name': 'assets',
	'type': 'asset',
	'schema': 'openpype:asset-3.0',
	'data': {
		'parents': [], 'visualParent': None, 'tasks': {}, 'pixelAspect': 1.0, 'frameStart': 1001, 'handleEnd': 0, 'handleStart': 0, 'fps': 24.0, 'clipIn': 1, 'clipOut': 1, 'tools_env': [], 'resolutionHeight': 1188, 'frameEnd': 1100, 'resolutionWidth': 2112
	},
	'parent': ObjectId('617c367c797e4a6b9d76c207'),

}

[DEBUG] original item doc:
{
	'_id': ObjectId('617c63c9797e4a6b9d76c208'),
	'name': 'assets',
	'type': 'asset',
	'schema': 'openpype:asset-3.0',
	'data': {
		'visualParent': None, 'tasks': {}, 'resolutionHeight': 1188, 'handleEnd': 0, 'resolutionWidth': 2112, 'tools_env': [], 'clipIn': 1, 'frameStart': 1001, 'handleStart': 0, 'clipOut': 1, 'fps': 24.0, 'frameEnd': 1100, 'pixelAspect': 1.0
	}
}

Then the false positive 'update' info generated for bulk-write:
[INFO] bulk writes: [
UpdateOne({'_id': ObjectId('617c63c9797e4a6b9d76c208')}, {'$set': {'parent': ObjectId('617c367c797e4a6b9d76c207'), 'data.parents': []}}, False, None, None, None),
...
...

Removing them in the doc generation let the update comparison only find the AssetItems actually changed.
Testing notes:
Use cases tested:

Add a task
Add an asset
rename a task
copy tasks from one asset to another

And after shutting down OP and restart, the changes above were persisted.

I'm not sure that I understand the issue. Keys parent and data.parents are required for asset document.

Created new PR with proper fix https://github.com/pypeclub/OpenPype/pull/3083 . But very good catch, I wonder how did you find it out.

EDITED: I see the issue, but this is not how it should be fixed as you're skipping required asset document keys. I'll create new PR.

Right, removing them seemed not to be a proper fix.. but strangely when I tested the main functionalities of ProjectManager it seemed everything works? I was puzzled.
This is the first time I play with MongoDB api so I'm not surprised I was way off..

Created new PR with proper fix #3083 . But very good catch, I wonder how did you find it out.

As I mentioned in the description, I'm trying to come up with an API to programmatically create new tasks, so I was learning how it currently works by instrumenting the code to see the payload sent to Mongodb, in particular bulkWrite(). Then I noticed there are many UpdateOne() entries even if I changed a single thing.. then I started digging.  :D


[Phenotyping] Excluding patients with history of liver dieases

We need to exclude patients with existing history of gall bladder disease/cholelithiasis and/or prior liver impairment (alcoholic hepatitis or hepatitis a/b/c infections)

we may need to exclude those with gall bladder diseases as well.


left outer join

I'd like to do a LEFT OUTER JOIN between two streams
It looks like I could use DefinePattern. Is there any other better way?
Thanks,
Nestor

Hi Nestor, have you tried the LeftOuterJoin macro? https://github.com/Microsoft/Trill/blob/master/Sources/Core/Microsoft.StreamProcessing/StreamableAPI/StreamableExtensions.cs#L984

Great. Thanks!


After Merge

After the Merge I assume the code could not be checked in?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 54bffa91-2a14-2819-8f74-80fd1f79b9df
Version Independent ID: f1042837-8207-d07b-3d53-86ead3b6679c
Content: Configure the code upgrade service in Lifecycle Services (LCS) - Finance & Operations | Dynamics 365 | #MSDyn365FO
Content Source: articles/dev-itpro/lifecycle-services/configure-execute-code-upgrade.md
Service: dynamics-ax-platform
GitHub Login: @tariqbell
Microsoft Alias: tabell

Thank you for your question. @tariqbell, can you please take a look?

Hi @soerenpc could you elaborate a bit for us please? Merging branches in itself doesn't stop you checking in.

Sorry I meant;
"After the Merge I assume the code should not be checked in?"

Why do you think you shouldn't check it in? Typically you're completing all the uodate/upgrade work in the releases branch, when you're happy all is good, then you that Releases branch back into Main, check in, Main becomes the active branch again. If you keep working in Main while the Releases work is happening then it gets more complex, because Main will have some more new code, but when merging the branches it'll highlight the differences and ask what you want to do with each of them, so you can tell it to keep the newer code on the ones that changed.

Hi Tariq
But if I want to work on the Release branch then I need to map it was will I assume?
<EMAIL_ADDRESS>For now I had to merge it into Main (which is mapped):
<EMAIL_ADDRESS>I have seen this:
<EMAIL_ADDRESS>But it doesn’t mention that you need to map the Release folder?
Regards / Mvh, Søren Printz Christensen, Partner,+45 2447 6425
From: Tariq Bell<EMAIL_ADDRESS>Sent: 1. juli 2019 08:01
To: MicrosoftDocs/dynamics-365-unified-operations-public<EMAIL_ADDRESS>Cc: Søren Printz Christensen<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [MicrosoftDocs/dynamics-365-unified-operations-public] After Merge (#1395)
Why do you think you shouldn't check it in? Typically you're completing all the uodate/upgrade work in the releases branch, when you're happy all is good, then you that Releases branch back into Main, check in, Main becomes the active branch again. If you keep working in Main while the Releases work is happening then it gets more complex, because Main will have some more new code, but when merging the branches it'll highlight the differences and ask what you want to do with each of them, so you can tell it to keep the newer code on the ones that changed.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/MicrosoftDocs/dynamics-365-unified-operations-public/issues/1395?email_source=notifications&email_token=ALCN5XVGXL7NNIGNP47WT7DP5GMSTA5CNFSM4HZ3F4R2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODY5CKRQ#issuecomment-507127110, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ALCN5XSFRM4WBFNCYQ5NIW3P5GMSTANCNFSM4HZ3F4RQ.

Actually, the code upgrade service is used when upgrading from a pre-8.0 version - because in those earlier versions where we still had overlayering, the code upgrade service could help with model splitting/merging.
When you#re upgrading from one of those earlier versions - your development environment - it can't be upgraded, you had to deploy a new one. For example:
I'm currently running 7.3, I want to upgrade to 10.x. I run code upgrade service, it create a Releases\xxx branch for me.
Now I can't connect to that Releases branch from my old 7.3 development machine - it's running the wrong version of the application - the DevOps branch contains only my non-Microsoft code.
So I deploy a new 10.x development environment, map it to Releases\xxx and complete the work to get the code working again with the new version. Then when I'm happy it's working, I merge back into Trunk\Main - and at that point all the developers need to have 10.x development environments - because after that Trunk\Main will be 10.x.
This article doesn't explain that detail - it just says "Once the upgrade code in Releases\xxx compiles successfully..." - the main upgrade articles used to explain the process in more detail - but OneVersion has superseded that, and the articles have changed to reflect the new process (there's no code upgrade process anymore)

Hi Tariq
My scenario is from Ax2012!
The article has been written to support that as well:
<EMAIL_ADDRESS>Regards / Mvh, Søren Printz Christensen, Partner,+45 2447 6425
From: Tariq Bell<EMAIL_ADDRESS>Sent: 1. juli 2019 09:27
To: MicrosoftDocs/dynamics-365-unified-operations-public<EMAIL_ADDRESS>Cc: Søren Printz Christensen<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [MicrosoftDocs/dynamics-365-unified-operations-public] After Merge (#1395)
Actually, the code upgrade service is used when upgrading from a pre-8.0 version - because in those earlier versions where we still had overlayering, the code upgrade service could help with model splitting/merging.
When you#re upgrading from one of those earlier versions - your development environment - it can't be upgraded, you had to deploy a new one. For example:
I'm currently running 7.3, I want to upgrade to 10.x. I run code upgrade service, it create a Releases\xxx branch for me.
Now I can't connect to that Releases branch from my old 7.3 development machine - it's running the wrong version of the application - the DevOps branch contains only my non-Microsoft code.
So I deploy a new 10.x development environment, map it to Releases\xxx and complete the work to get the code working again with the new version. Then when I'm happy it's working, I merge back into Trunk\Main - and at that point all the developers need to have 10.x development environments - because after that Trunk\Main will be 10.x.
This article doesn't explain that detail - it just says "Once the upgrade code in Releases\xxx compiles successfully..." - the main upgrade articles used to explain the process in more detail - but OneVersion has superseded that, and the articles have changed to reflect the new process (there's no code upgrade process anymore)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/MicrosoftDocs/dynamics-365-unified-operations-public/issues/1395?email_source=notifications&email_token=ALCN5XSB6RLWDKQRLRRW6QTP5GWT7A5CNFSM4HZ3F4R2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODY5HTSI#issuecomment-507148745, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ALCN5XRFWC2IZB6FV5N2IFLP5GWT7ANCNFSM4HZ3F4RQ.

ah ok. If you're upgrading from AX2012, then merging back into Main and checking in, or working in the Releases branch directly and merging back later, make no difference - because the incoming code is the first version of your code for D365FFO so you won't break anything by checking in - the code probably won't work yet if you haven't already completed all the upgrade work, but it can still be useful to check it in, so you've got a safe saved copy of the work you've done so far.

Hi
Yes so as I wrote. No need to check in after merging Main of elements later are move to a different model or if an element overlayers..
Regards / Mvh, Søren Printz Christensen, Partner,+45 2447 6425
From: Tariq Bell<EMAIL_ADDRESS>Sent: 1. juli 2019 10:22
To: MicrosoftDocs/dynamics-365-unified-operations-public<EMAIL_ADDRESS>Cc: Søren Printz Christensen<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [MicrosoftDocs/dynamics-365-unified-operations-public] After Merge (#1395)
ah ok. If you're upgrading from AX2012, then merging back into Main and checking in, or working in the Releases branch directly and merging back later, make no difference - because the incoming code is the first version of your code for D365FFO so you won't break anything by checking in - the code probably won't work yet if you haven't already completed all the upgrade work, but it can still be useful to check it in, so you've got a safe saved copy of the work you've done so far.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/MicrosoftDocs/dynamics-365-unified-operations-public/issues/1395?email_source=notifications&email_token=ALCN5XROVML4G3KOBRVNMRTP5G5BJA5CNFSM4HZ3F4R2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODY5L4PA#issuecomment-507166268, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ALCN5XWLMGH3Y6DJCYVTURLP5G5BJANCNFSM4HZ3F4RQ.

I'm not sure I'm understanding your point? Is it wrong for the article to say that after merging the branches you should check in? We're expecting to work in Releases.. and then when code is completed, merge back to Main and check in.
The only thing that really matters is not to put too much effort into the code without checking in, so you dknt lose what you've done.


Make api_server_authorized_ip_ranges configurable.

Make api_server_authorized_ip_ranges configurable.
Fix #208 and #108

Looks good to me. I think it's better to include it in the test main.tf as well so the users can have a good example.


Deployments

We need an activity with a RecyclerView that lists all of the available deployments from the server. This activity will have Material Design CardViews inside of a RecyclerView.
Clicking on a card will give you details and actions to retrieve a given deployment. Each card will have a progress bar showing the state of a download.

We're getting cards in a recycle view showing the name and description of deployments coming from a real-live OpenMapKit Server deployments API.
Don't worry about the email fab on the bottom. It's a stub in case we want to have a button there.

Basic UI for Deployment Details

@hallahan these look great. I do especially like the download button on the bottom to download everything.

We're mostly there. I need to keep track of deployments that have been already downloaded so that the UI doesn't suggest downloading it again. Also, we need to show deployments that have completed the downloads in the UI when the device is offline so that the user can check them out.

At this point, I'm showing in the Deployments recycler view the deployments saved to external storage, and then the deployments show from the API.
I'm going to revamp the Deployments class to have a List and a HashMap referencing Deployment objects. When we get a deployment from the API that collides with the name of a deployment from disk, we are going to PUT overwrite what came from disk so that we have new info about it.
Internally we're going to reference the constructed Deployment object instead of a JSONObject, and this is cleaner.
Also, we're going to have a persisted boolean in a Deployment to show that it came from ExternalStorage. We'll have an api boolean if it came from OMK Server. We'll have both if we have both.
When I have all of this in there, then we can make a method in Deployment to check to see if the deployment being described by the API has been updated, based on file names and sizes.

Showing offline deployments is going smoothly. Next step is to have DeploymentDetails show the right state of the download when the activity is reloaded. I'm going to verify state of a download by looking at the file system and compare with what is stated in the deployment.json.

For all intensive purposes, we've got full deployment functionality working now. I'm going to pull into master where I'm at. I think we should leave this issue open, because there's a few bugs hanging out there like:

The Checkout FAB sometimes shows when it's not supposed to while a download is happening. Not sure why yet...
If I'm downloading many deployments at once, the list of deployments in DeploymentsActivity hangs, since that also needs an AsyncTask. I need a bigger limit on the thread pool.

I'm considering 1. and 2. edge cases. Let's file new issues if they come into play. Just completed 3. #138


(963) Use first class risk ratings in frontend for assessments

Now the risk information is automatically included in the Application data, we no longer need to hit the risks endpoint each time.

 Use first-class risk ratings in frontend


Fix Xcode 9 introduced leveldb warnings

$ pod spec lint FirebaseCommunity.podspec | grep leveldb
-> FirebaseCommunity (0.1.4)
- WARN  | [FirebaseCommunity/Database] xcodebuild:  leveldb-library/db/c.cc:274:56: warning: possible misuse of comma operator here [-Wcomma]
- NOTE  | [FirebaseCommunity/Database] xcodebuild:  leveldb-library/db/c.cc:274:21: note: cast expression to void to silence warning
- WARN  | [FirebaseCommunity/Database] xcodebuild:  leveldb-library/db/c.cc:275:56: warning: possible misuse of comma operator here [-Wcomma]
- NOTE  | [FirebaseCommunity/Database] xcodebuild:  leveldb-library/db/c.cc:275:21: note: cast expression to void to silence warning
- WARN  | [FirebaseCommunity/Database] xcodebuild:  leveldb-library/db/version_set.cc:1350:5: warning: code will never be executed [-Wunreachable-code]
- NOTE  | [FirebaseCommunity/Database] xcodebuild:  leveldb-library/db/version_set.cc:1349:7: note: silence by adding parentheses to mark code as explicitly dead
Analyzed 1 podspec.

Warnings are silenced by https://github.com/firebase/leveldb-library-podspec/pull/5 which we plan to push to CocoaPods trunk in the near future.

Run pod-update to get the 1.20 version of leveldb-library


add type=file

@MartijnR @cagulas you can now test this on stage, deployed the branch.

Ok, I am able to upload pdf files, only pdf files.  Is that the intention?  Am I supposed to be able to upload other types of files as well, like .doc, etc.?

You should be able to upload any type of file, the implementation does not restrict the type of file to be uploaded. That restriction will be something that would be controlled by the aggregate server.

Ok, then, it's not functioning correctly, because only pdf files can be uploaded.  All other file types are greyed out and cannot be selected for upload.

Ah thanks. A long time has passed since this was feature was developed.... It seems application/* no longer works in Chrome (still works in FF). We should find another media type.

I updated the issue (#33) with a new proposed mediatype value.