[0.8.1] Partial updates syntax for restful

[Cassie]

The current restful spec says that the partial updates syntax is tbd. I'd like to propose the following syntax:

When the "fields" parameter is specified on a url, only those fields will be looked at when updating an object. If a field is in the parameter list but not part of the post, that would be considered a removal of that field. If the field is in the post but not in the parameter list it would be ignored. If a field is in both places then it will be updated normally.

For example, 


PUT /appdata/john.doe/@self/@app?fields=foo
post body : {foo : 3}

The data store now has {foo : 3}


PUT /appdata/john.doe/@self/@app?fields=bar
post body : {bar : 5}

The data store now has {foo : 3, bar : 5}
(foo was not deleted because it wasn't in the fields list)


PUT /appdata/john.doe/@self/@app?fields=bar
post body : {foo : 5}

The data store now has {foo : 3} 
(we ignored foo because it wasn't in the fields list. bar was treated as a delete because it wasn't posted)


DELETE /appdata/john.doe/@self/@app?fields=bar

The data store now has {foo : 3} 
(nothing happened as bar was already deleted)


DELETE /appdata/john.doe/@self/@app

The data store now has {} 
(all fields were deleted because no fields param was specified)


This implementation is very trivial and has been completed within shindig.
- Cassie

[Cassie]

Oh - I also don't think this should be optional and would like to change it to a MUST in the spec.

I think is necessary because of the opensocial js apis. In order for the app data calls to be backed by the restful spec we need partial updates for appdata. The js apis only get a call of the form update(key, newValue). They don't know whether the key is new or established, they don't know whether there are any other keys in the datastore etc etc. Instead of having to request all data from the server, update the one key and then repost the whole thing the js apis can be simply translate into:

POST /appdata/@me/@self//@app?fields=key
post data: {key : newValue}

Again, the support is pretty trivial to implement on the server so I don't think this is much of a burden on containers.


- Cassie

[John Panzer]

+1, two clarifications/suggestions

1. Clarify interactions with ETags (when they are supported).  If you supply the ETag that you got from a "full" representation of a resource via an If-Match: header, but only update a subset of the fields, what happens?  (These are two separate resources so HTTP is basically silent on this point, we can decide to define it for convenience of point updates with optimistic concurrency, or leave it undefined for 0.8.1 and address it later.)

2. Does this address only top level fields in the resource in question (RIQ)?

(Note that in this case, the 'resource' in question is the resource named by /appdata/john.doe@self/@app?fields=bar, which is a _separate_ resource from the one named by /appdata/john.doe@self/@app.)

John Panzer (http://abstractioneer.org)

[Louis Ryan]

-1 to implied deletes by omitting fields on updates. Can we just require HTTP DELETE for any delete and everything else is an update/add?

I'm imagining try to explain the above in words to a gadget developer and the blank stare I get back.

[Cassie]

So basically there is no blanket overwriting of data? Updating an activity with {title : 'myactivity'} would only set the title fields and not touch anything else? Everything is a "partial update"?

If so, okay! I'm in!

- Cassie

[John Panzer]

+1 to implied field deletes by omitting fields on PUTs.  Louis, you're the one pushing for batch operations -- I'd think you'd be all over this.  Note that a PUT != update, it means "set the state to the state that I'm sending you".  Implicit deletes of fields are no more confusing than implicit deletes of characters in the middle of a string field when you PUT a new value to the server -- it's just state information.

I don't see what's the problem with the explanation; how about this:

"Okay, you do a GET of nickname and photo using &fields=nickname,photo.  Then you edit them (if you want to remove one, you just remove it).  Then when you're done, you do a PUT back to the original URL including &fields=nickname,photo so the server knows you're updating those fields. If you've removed one, the server removes it on its side. If it returns OK, you're done, and all changes are committed."

As opposed to

"First, you DELETE the one that you don't want any more using &fields=photo.  Then, you PUT the one you want to update using &fields=nickname.  You can do these in a batch, but there's no guarantee that both will complete of course, and you have to compose and send two requests."


John Panzer (http://abstractioneer.org)

[Kevin Marks]

The danger here is the @all or '*' case - we should explicitly exclude that as that makes it too easy for a client-side data transformation error to erase the whole entry (which is the problem specifying fieds is trying to solve - that it otherwise requires all client-side transformations to preserver unparsed fields).

[John Panzer]

-1 but on iPhone so bandwidth too limited to expound :)

--
John Panzer (http://abstractioneer.org)

[Louis Ryan]

From Cassies example

PUT /appdata/john.doe/@self/@app?

fields=bar
post body : {foo : 5}

The data store now has {foo : 3} 
(we ignored foo because it wasn't in the fields list. bar was treated as a delete because it wasn't posted)

this scares me as there was obvious intent in the call that differed from the protocol and the call succeeds anyway. Similarly if I don't specify a field in the fields list it gets deleted from the resource. Both of these cases seem far too easy for developers to hit. In the case of the implicit delete there isn't any indication in the response what fields were deleted. I like batching and performance optimizations but I fear the wrath of unintended consequences far more.

In situations where the object is more complex such as a Person/Contact the situation is much worse than for AppData. Most containers have fields specific to them  and a consumer of this API would need to know how to generate an appropriate fields param to cover all of the variations. For large objects this makes an update to a single field incredibly verbose. Many developers would make mistakes forming up the appropriate object and field list for all containers.

Could we simplify and say we don't support the fields param on PUT and only use the content of the body to perform explicit updates on those specified fields. This would allow clients to perform updates on objects for which they only have partial representations to begin with (because ironically they used the fields param on GET) or where they want to directly author the field without first fetching the object at all.

I believe in REST this would actually be an HTTP MERGE/PATCH which if thats the right semantic for what people should be using then lets support and encourage the use of that because its the right thing to use and forego PUT.

[Scott Seely]

I like this approach since I only have to state things once.

 

Is the proposal then:

PUT /appdata/john.doe/@self/@app?

post body : {foo : 5}

causes an update to foo

 

and

 

DELETE /appdata/john.doe/@self/@app?

fields=bar

causes the bar field to be removed?

[Louis Ryan]

Yup, thats pretty much it...

[John Panzer]

John Panzer (http://abstractioneer.org)

On Tue, Jul 29, 2008 at 11:37 AM, Cassie <do...@google.com> wrote:

The current restful spec says that the partial updates syntax is tbd. I'd like to propose the following syntax:

When the "fields" parameter is specified on a url, only those fields will be looked at when updating an object. If a field is in the parameter list but not part of the post, that would be considered a removal of that field. If the field is in the post but not in the parameter list it would be ignored. If a field is in both places then it will be updated normally.

For example, 


PUT /appdata/john.doe/@self/@app?fields=foo
post body : {foo : 3}

The data store now has {foo : 3}


PUT /appdata/john.doe/@self/@app?fields=bar
post body : {bar : 5}

The data store now has {foo : 3, bar : 5}
(foo was not deleted because it wasn't in the fields list)


PUT /appdata/john.doe/@self/@app?fields=bar
post body : {foo : 5}

The data store now has {foo : 3} 
(we ignored foo because it wasn't in the fields list. bar was treated as a delete because it wasn't posted)

The way to think about these URLs is that /appdata/john.doe/@self/@app?fields=bar defines a resource with a single field named "bar", which obviously overlaps with the resource named /appdata/john.doe/@self/@app.  In English, we're talking about "the 'bar' part of the appdata".  I don't think we should silently truncate data on a PUT (I really, really hate it when DBs do this to my strings; silent data loss is Bad).

I'd suggest that this result in a 400 Bad Request (with a reasonable error body explaining what went wrong), since you're trying to set a field that doesn't exist.  (Just as you'd get a 400 Bad Request if you tried to set the field "xyzzy" on a Person object, because field xyzzy doesn't exist.)

 

[John Panzer]

On Thu, Jul 31, 2008 at 12:50 AM, Louis Ryan <lr...@google.com> wrote:

From Cassies example

PUT /appdata/john.doe/@self/@app?

fields=bar
post body : {foo : 5}

The data store now has {foo : 3} 
(we ignored foo because it wasn't in the fields list. bar was treated as a delete because it wasn't posted)

this scares me as there was obvious intent in the call that differed from the protocol and the call succeeds anyway. Similarly if I don't specify a field in the fields list it gets deleted from the resource. Both of these cases seem far too easy for developers to hit. In the case of the implicit delete there isn't any indication in the response what fields were deleted. I like batching and performance optimizations but I fear the wrath of unintended consequences far more.

This scares me as well (see my proposal that this should result in a 400 Bad Request error; silent data loss is Bad).

But wait a second: "Similarly if I don't specify a field in the fields list it gets deleted from the resource".  No, if you leave a field off your filter it means "don't touch this field, it's not part of the object I'm working on at the moment".
 

In situations where the object is more complex such as a Person/Contact the situation is much worse than for AppData. Most containers have fields specific to them  and a consumer of this API would need to know how to generate an appropriate fields param to cover all of the variations. For large objects this makes an update to a single field incredibly verbose. Many developers would make mistakes forming up the appropriate object and field list for all containers.

Can you provide an example of a call where this is complex?  I'm trying to imagine one and failing, so I may be misunderstanding you.
 

Could we simplify and say we don't support the fields param on PUT and only use the content of the body to perform explicit updates on those specified fields. This would allow clients to perform updates on objects for which they only have partial representations to begin with (because ironically they used the fields param on GET) or where they want to directly author the field without first fetching the object at all.

I believe in REST this would actually be an HTTP MERGE/PATCH which if thats the right semantic for what people should be using then lets support and encourage the use of that because its the right thing to use and forego PUT.

This is the direction these discussions typically go in (partial updates -> generalized PATCH).  I really like the simple partial updates we have now though, as a generalized PATCH mechanism is complicated and I think unnecessary for 80% of all uses.

 

[John Panzer]

This breaks the semantics of PUT; you want PATCH.

John Panzer (http://abstractioneer.org)

[Louis Ryan]

On Thu, Jul 31, 2008 at 11:08 AM, John Panzer <jpa...@google.com> wrote:

On Thu, Jul 31, 2008 at 12:50 AM, Louis Ryan <lr...@google.com> wrote:

From Cassies example

PUT /appdata/john.doe/@self/@app?

fields=bar
post body : {foo : 5}

The data store now has {foo : 3} 
(we ignored foo because it wasn't in the fields list. bar was treated as a delete because it wasn't posted)

this scares me as there was obvious intent in the call that differed from the protocol and the call succeeds anyway. Similarly if I don't specify a field in the fields list it gets deleted from the resource. Both of these cases seem far too easy for developers to hit. In the case of the implicit delete there isn't any indication in the response what fields were deleted. I like batching and performance optimizations but I fear the wrath of unintended consequences far more.

This scares me as well (see my proposal that this should result in a 400 Bad Request error; silent data loss is Bad).

+1 on this :)
 

But wait a second: "Similarly if I don't specify a field in the fields list it gets deleted from the resource".  No, if you leave a field off your filter it means "don't touch this field, it's not part of the object I'm working on at the moment".

Yeah, I think I misread that part and so I recant. I think Cassie's proposal with appropriate error responses for inconsistent bodies is fine.

[Chris Chabot]

We have had a great discussion about this, and had a lot of input from everyone involved.

Do we reckon we are ready to vote on an actual proposal? And if so what would it be?

-- Chris

[John Panzer]

There may be a consensus here, but it's impossible to tell without an actual proposal to look at.  I liked Cassie's original proposal with the addition of an error if you try to shove fields in that don't fit into the current projection (to avoid silent data loss).  Could we vote on that?

John Panzer (http://abstractioneer.org)

[Louis Ryan]

If its not clear from my rantings above Im in agreement with John on +1 for Cassis proposal with the additional error handling.

[Cassie]

Okay, so I think Louis, John and I are all okay with this. (where
"this" is my original proposal plus error responses for inconsistent
bodies)
Scott Seely - I couldn't tell if you were a +1 on this or not.

So we've got 3, anybody else want to chime in on this thread so we can
close it up?
Thanks.

- Cassie

[Kevin Marks]

Can you restate it? I remember some tension about deletion of keys from your use cases - if you repost that with the clarified error cases that would help.

[Cassie]

Sure thing:

PUT /appdata/john.doe/@self/@app?fields=foo
post body : {foo : 3}

The data store now has {foo : 3}

PUT /appdata/john.doe/@self/@app?fields=bar
post body : {bar : 5}

The data store now has {foo : 3, bar : 5}
(foo was not deleted because it wasn't in the fields list)

PUT /appdata/john.doe/@self/@app?fields=bar
post body : {foo : 5}

returns a 400 error!
the post body contains a field that doesn't exist when "fields=bar" so
this request has a bad format.

PUT /appdata/john.doe/@self/@app?fields=bar
post body : {}

The data store now has {foo : 3}

(bar was treated as a delete because it wasn't posted)

DELETE /appdata/john.doe/@self/@app?fields=bar

The data store now has {foo : 3}
(nothing happened as bar was already deleted)

DELETE /appdata/john.doe/@self/@app

The data store now has {}
(all fields were deleted because no fields param was specified)

[Kevin Marks]

LGTM er, I mean +1

[Scott Seely]

I am -1 on this so far. As I understand things, a PUT is supposed to be idempotent. Because of this, I don’t like the idea of combining insert and update on the same verb.

 

From: Kevin Marks [mailto:kevin...@google.com]
Sent: Tuesday, August 12, 2008 12:36 PM
To: opensocial-an...@googlegroups.com
Cc: Scott Seely
Subject: Re: [0.8.1] Partial updates syntax for restful

 

LGTM er, I mean +1

[Scott Seely]

I am -1 on this so far. As I understand things from RFC 2616, a PUT is supposed to be idempotent. Because of this, I don’t like the idea of combining insert and update on the same verb. When I PUT on the object and then issue the same PUT again, how do I guarantee idempotent inserts? Typically, the actor doing the INSERT doesn’t know the ID of the item being added since that ID is dependent on the backing store.

 

I can guarantee a PUT is idempotent if I can pass in some unique ID, such as by doing a PUT to a specific URL that uses that resource. In the case below, that doesn’t happen.

 

With a POST, we don’t guarantee idempotent POST. It seems that we need to INSERT using POST, UPDATE using PUT.

 

I’m more than welcome to hearing opinions on why this is incorrect. As of now, I think that this is the wrong path.

 

From: Kevin Marks [mailto:kevin...@google.com]
Sent: Tuesday, August 12, 2008 12:36 PM
To: opensocial-an...@googlegroups.com
Cc: Scott Seely
Subject: Re: [0.8.1] Partial updates syntax for restful

 

LGTM er, I mean +1

[John Panzer]

Its still idempotent. There's no insert unless the field did not
exist previously; its standard PUT semantics.

[Scott Seely]

Given the thread, it's not clear if this discussion is solely for the
appData resource or if it is a general implementation pattern for
objects. Can someone clarify the intent of what we are being asked to
vote on?

[Patrick Mueller]

The "RESTful API Specification", http://code.google.com/apis/opensocial/docs/0.8/restfulspec.html
, section 10, says:

"Partial updates avoid the need to send full representations
of data on updates, especially for People and App Data."

On Aug 12, 2008, at 10:23 PM, Scott Seely wrote:

> Given the thread, it's not clear if this discussion is solely for the
> appData resource or if it is a general implementation pattern for
> objects. Can someone clarify the intent of what we are being asked to
> vote on?

Patrick Mueller
pmu...@gmail.com

[Scott Seely]

I just want to make sure that we are only being asked to update the
syntax around partial updates. In the examples, it looked like we were
pushing into an area where this would be applied to inserts.

I'm +1 for the update semantics.

-----Original Message-----
From: opensocial-an...@googlegroups.com
[mailto:opensocial-an...@googlegroups.com] On Behalf Of
Patrick Mueller
Sent: Wednesday, August 13, 2008 6:55 AM
To: opensocial-an...@googlegroups.com
Subject: Re: [0.8.1] Partial updates syntax for restful

[Patrick Mueller]

Some questions:

- someone previous asked about doing partial updates of 'inner'
fields. I assume an example would be something like setting the
"name.unstructured" field of Person. Or is this just for 'top-level'
fields; in this case, you couldn't JUST set "name.unstructured", you'd
need to set "name", whose value would be {name : { unstructured :
"Bob" }}

- is there a way to do this in XML?

- what is the JS API going to look like here?

- how do ETags factor in to handle concurrent/lost updates?

I'd say that something about this in general doesn't feel right, where
the ?fields={fieldList} bit is being treated like a resource itself -
someone phrased it well as a "projection" of a resource. Though
clearly you can use query string parameters with PUT and DELETE,
there's something about it that seems ... non-standard. Even though
it would end up being pretty non-performant to do it the way I'm about
to suggest, to update multiple fields, it seems ... RESTier. And that
way would be to handle field updates one at a time, where you don't
use a query string to specify th e field name, but some other
qualifier that moves it back into the main-line URL. Something like

/appdata/john.doe/@self/@app/@field/foo

What ETags mean in such a world is not clear.

On Aug 12, 2008, at 3:05 PM, Cassie wrote:

>
> Sure thing:
>
>
> PUT /appdata/john.doe/@self/@app?fields=foo
> post body : {foo : 3}
>
> The data store now has {foo : 3}
>
>
> PUT /appdata/john.doe/@self/@app?fields=bar
> post body : {bar : 5}
>
> The data store now has {foo : 3, bar : 5}
> (foo was not deleted because it wasn't in the fields list)
>
>
> PUT /appdata/john.doe/@self/@app?fields=bar
> post body : {foo : 5}
>
> returns a 400 error!
> the post body contains a field that doesn't exist when "fields=bar" so
> this request has a bad format.
>
>
> PUT /appdata/john.doe/@self/@app?fields=bar
> post body : {}
>
> The data store now has {foo : 3}
> (bar was treated as a delete because it wasn't posted)
>
>
> DELETE /appdata/john.doe/@self/@app?fields=bar
>
> The data store now has {foo : 3}
> (nothing happened as bar was already deleted)
>
>
> DELETE /appdata/john.doe/@self/@app
>
> The data store now has {}
> (all fields were deleted because no fields param was specified)

Patrick Mueller
pmu...@gmail.com

[John Panzer]

Just a note, the URL /foo?bar=1 and /foo/bar=1 are both equally RESTful and semantic (but the former is probably easier for frameworks to parse).  And, a projection of a resource is a reasonable resource in its own right. 

John Panzer (http://abstractioneer.org)