Adding fields to existing API

[Ben George]

When adding new fields to a  restful HTTP API, how can the API handle the absence of new fields in a PUT requests.

Given the resource /users exists, and GET returns { "username":"joeblogs", "address":"1 London Ave" }.  If the client does not know about the newly added address field, and PUTS { "username":"joeblogs" }, then what should the API do with address ?

Can we state that for a PUT any field not mentioned is not updated ?  Seems messy, and against the vibe of PUT.

We are stumped as how to approach this.  Versioning could be one approach, but that is a massive amount of work for our existing API, and some work for our client code too.  For what it is worth, this API is used by multiple apps internally (we have control over the clients also).

[mjpearson]

Versioning is the best idea, but it would make sense to raise a validation error to the client anyhow if required fields are not present.  If its too much hassle and you need backwards compatability, could merge the payload into your data source rather than explicitly saving it... or support atomic updates with the PATCH method.

[Jørn Wildt]

PUT is a somewhat funny method in the API world. The designers of PUT decided that it should have "full replacement" semantics - well, at least the client intention should be to "replace the target completely with the enclosed data". So in Ben's example the intention should be interpreted as "remove the address property"!

My understanding is that PUT was designed for a very document oriented approach: the client GETs the full resource representation, makes changes to it and PUT it all back again. That would work fine if the resource was backed by an HTML file, image or perhaps an office document. No one would try to make partial changes to a word document, right? Neither would anyone guess the state of an image and then try to send back partial content of the image without reading it first, right?

Now we are making APIs with HTTP and then something funny happens: we start to assume a fixed structure of the resource - the client becomes hard coded with the assumption that a resource represents, for instance, a "user" as above, and with that knowledge comes the natural idea of trying do partial updates of the resource (for instance only changing the user name) - or, as in the case of this discussion, the client is out of date and makes the wrong assumption about the structure of the resource.

So we have three similar problematic scenarios with PUT in the API space:

1) Clients unintentionally send back partial content because they make the wrong assumptions about the resource (because the client is not upgraded in-sync with the server).

2) Clients wants to do intentionally partial updates (because it is more efficient than sending back the whole modified resource representation)

3) The resource representation contains a lot of links and other server-added stuff that cannot (and should not) be modified by the client. So the client developer don't want to PUT all that unnecessary jazz back again.

So, what to do?

First of all - as I have come to understand it - PUT needs a preceding GET. The client MUST always do a GET of the resource, keeping and ignoring ALL unknown structural elements, apply its intended modifications and then PUT the complete modified representation back again. This is inefficient but safe and allows the server to evolve its resource representation over time without breaking clients.

Second - there is no such thing as a partial PUT. See for instance the discussion here http://lists.w3.org/Archives/Public/ietf-http-wg/2012JanMar/0653.html. It does even make sense when you want the server to be able to evolve without upgrading clients - as you will avoid the out-of-date problem Ben describes initially as long as you always combine GET with PUT.

Third - if you want to make partial changes to a resource, use PATCH with a suitable patch format, for instance JSON-PATCH.

And, for the fun of it, it should in fact be possible to do partial changes to an image (as opposed to what I said above). You just need a suitable patch language - something that could for instance apply image operators like "rotate 90 degress clockwise" or "replace all red with yellow" and so on. That could safely be applied without making a GET first.

And, yes, you can think of a simple JSON object as a simple patch format: if a property is not there, the corresponding target property is not modified, otherwise it is replaced with the property value. But it doesn't work well with arrays and nested objects - and its not a standard :-)

/Jørn

--
You received this message because you are subscribed to the Google Groups "API Craft" group.
To unsubscribe from this group and stop receiving emails from it, send an email to api-craft+...@googlegroups.com.
Visit this group at http://groups.google.com/group/api-craft.
For more options, visit https://groups.google.com/d/optout.

[Markus Lanthaler]

On Wednesday, June 18, 2014 2:21 PM, Ben George wrote:
> When adding new fields to a restful HTTP API, how can the API handle
> the absence of new fields in a PUT requests.
>
> Given the resource /users exists, and GET returns {
> "username":"joeblogs", "address":"1 London Ave" }. If the client does
> not know about the newly added address field, and PUTS {
> "username":"joeblogs" }, then what should the API do with address ?

Remove it

> Can we state that for a PUT any field not mentioned is not updated ?
> Seems messy, and against the vibe of PUT.

Nope. That would be a violation of the HTTP spec.

> We are stumped as how to approach this. Versioning could be one
> approach, but that is a massive amount of work for our existing API,
> and some work for our client code too. For what it is worth, this API
> is used by multiple apps internally (we have control over the clients
> also).

That's what PATCH was made for. You just need a patch media type with the right semantics. Luckily, there's already one in the making:

http://tools.ietf.org/html/draft-ietf-appsawg-json-merge-patch


HTH,
Markus


--
Markus Lanthaler
@markuslanthaler

[Mike Kelly]

On Wed, Jun 18, 2014 at 11:57 PM, Markus Lanthaler
<markus.l...@gmx.net> wrote:
> On Wednesday, June 18, 2014 2:21 PM, Ben George wrote:
>> When adding new fields to a restful HTTP API, how can the API handle
>> the absence of new fields in a PUT requests.
>>
>> Given the resource /users exists, and GET returns {
>> "username":"joeblogs", "address":"1 London Ave" }. If the client does
>> not know about the newly added address field, and PUTS {
>> "username":"joeblogs" }, then what should the API do with address ?
>
> Remove it
>
>> Can we state that for a PUT any field not mentioned is not updated ?
>> Seems messy, and against the vibe of PUT.
>
> Nope. That would be a violation of the HTTP spec.
>

Ben,

Unfortunately, a bad decision was made to alter the HTTP spec to make
it explicitly prevent the fairly common usage of 'partial-PUT' you're
describing.

The good news is, in practice you can safely disregard Mark's advice
(aswell as the spec) and use PUT in the way you're considering.

Cheers,
M

[Jørn Wildt]

Depending on the server side framework there is a catch to partial PUTs (and POSTs as well) - some frameworks handles PUT/POST payload by serializing them into class instances. In these situations the server code cannot distinguish between NULL and "Not in the payload" - so it can be difficult to know if a certain property should be ignored (not in the payload) or set to NULL.

It can of course be avoided by deserializing the payload into dictionaries (JSON objects) instead where a key can be either absent, NULL or a non-NULL value.

/Jørn

M

[Jonathan Roeder]

Client SDKs will often also represent Resources as static types, which can cause the same clouding of intent (omit means leave it alone or delete it?) from the client's call. One solve is to have the SDK present a static type for a POST or PUT, but a dictionary or dynamic type (if the language supports it) for a PATCH.

[Julie C]

I'm late to this party, resurrecting a dormant thread, but my team is currently having angst over this exact question so I'd like to just make sure I'm understanding the conclusion correctly.

To paraphrase the problem:  the server is theoretically able to add new attributes when it needs to without triggering a version change in the representation because the client can safely ignore what it doesn't understand when it GETs a resource.  But if the client goes to PUT back the representation it understands, well, from the server's perspective that's a partial-PUT.

Jorn, your first suggestion (way back above) is that the client should do a GET and then just re-write the values it wants to update and ship the whole thing back.  I get it, but if my client is using a serialization framework that deserializes payload into class instances and my client-side code doesn't have that attribute in its class definition, then there's no way for the client to re-serialize the missing attribute back into the PUT payload.

This suggests to me that the client can pretty much never do PUTs because it can never be confident that its understanding of the definition of the Resource is current - unless everything is versioned, all the time.

Which brings me to the conclusion that if one's client & server aren't tightly coupled in a deployment schedule kind of way, and one isn't doing explicit versioning (and up-revving with every single change), then PUT is pretty much useless and one can only use PATCH.  Using PUT requires versioning.

Have I done a 2+2=5 somewhere?

Thanks,

Julie

[Andrew Braae]

My understanding is also that PUT is pretty much useless unless:

- you are dealing with very simple resources that don't acquire new attributes over time

   (which probably disqualifies many business type resources, like Order, Customer, etc.)

- you are happy to wear the overhead of always shipping the entire resource across the wire

   (which probably disqualifies even simple resources like a Document object that contains a blob of binary data and some metadata like Author, since you can't update the Author without also sending the entire document)

I guess PUT would be really useful though for something like the state of a door!

PUT /doors/10034/isOpen

true

BTW if you do choose to move to PATCH then be aware there are at least two relevant media types. I've found studying these and trying to adapt your requirements to one of them is likely more rewarding than reinventing the wheel and coming up with your own:

JSON Merge Patch - https://tools.ietf.org/html/rfc7386 (mentioned by Markus above): wonderfully simple but does not deal well with things like chagning the contents of an array

JSON Patch - https://tools.ietf.org/html/rfc6902 - less natural but you can do more with it

[Jørn Wildt]

Hi Julie

That is pretty much the same conclusion I have come to: if you want to use PUT then you need to keep the whole document model and work on that - serializing into classes of fixed types won't work. See also http://soabits.blogspot.dk/2013/01/http-put-patch-or-post-partial-updates.html for a discussion.

If you work on strictly typed classes then PATCH seems like the way to go.

What you can do instead is to serialize into hashmaps/dictionaries and have classes on top of them as type safe facades. Something like this in C#:

class Cat : Dictionary<string, object>
{

  public string Name
  {

    get { return this["name"]; }

    set { this["name"] = value; }

  }

}

(in C# you can have even more fun if you start using dynamics).

If anyone is working with C# then my Ramone library (https://github.com/JornWildt/Ramone) handles some of this stuff (especially the work.in-progress part here https://github.com/JornWildt/Ramone/tree/master/Ramone.Hypermedia).

Some of the more flexible languages should be able to add all payload properties as dynamic properties on the fly - and thus be able to serialize back and forth safely (but I haven't tried that).

/Jørn

[Jørn Wildt]

>- you are happy to wear the overhead of always shipping the entire resource across the wire
>   (which probably disqualifies even simple resources like a Document object that contains a blob of binary data and some metadata like

> Author, since you can't update the Author without also sending the entire document)

Well, you can always have the binary blob as a separate resource ;-)

/Jørn

[Ben George]

Julie, totally get your problem, what technologies / framework are you using ?

If your using Java I can happily share some generic code that will
get-merge-put some json.

> You received this message because you are subscribed to a topic in the

> Google Groups "API Craft" group.

> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/api-craft/wsw3TJ5IP28/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to

[mca]

As long as you make your API changes backward-compatible, you'll have no problem supporting HTTP.PUT from existing deployed client apps.

"Absence of fields" occurs quite often in real life (e.g. date-time stamps supplied by servers, server-generated ids, etc.) and PUT doesn't change any of that. adding new elements in a response doesn't automatically REQUIRE them in PUT messages. it may, however, mean that servers need to be aware of existing implementations and take appropriate action when those implementations execute a valid PUT.

mca

+1.859.757.1449
skype:mca.amundsen
http://amundsen.com/blog/
http://twitter.com/mamund
https://github.com/mamund
http://linkedin.com/in/mamund

[Julie C]

Ben,

Thanks for the offer, but our client is C#/.NET (this week, anyway - we're in the very, very, early stages of prototyping.)  In future there's a strong possibility that a Java client will appear, but that's a while away still.

Cheers,

Julie

[Julie C]

Mike,

Are you suggesting that we're taking the meaning of "Full PUT" much too literally?  Let me paraphrase what I think you're saying, and please correct me where I'm misunderstanding.

1.  "Full PUT" != "specify all properties of resource".

2.  The representation that is received by GET does not have to match the representation that is sent by PUT.

3.  When the client PUTs a representation, the server should update the resource according to "full replacement" semantics, but that doesn't necessarily include updating all attributes.

So, harkening back to Andrew's document example from earlier in the thread...

If the resource on the server is fully:

Document {

  "content" : "blob",

  "create-date" : "Jan 1",

  "author" : "Andrew"

}

GET may send all of those things in the representation, but since "create-date" is system-assigned and not modifiable, it would be legit to define a "Full PUT" representation as:

Document {

  "content" : "different-blob",

  "author" : "Andrew"

}

So I think you're saying that as long as the server updates the document resource to replace Document.content with "different-blob" and Document.author with "Andrew", the fact that create-date isn't in the PUT and isn't getting modified does not make this request a Partial PUT?

Thanks,

Julie

[mca]

Julie C:

yep.

mca

+1.859.757.1449
skype:mca.amundsen
http://amundsen.com/blog/
http://twitter.com/mamund
https://github.com/mamund
http://linkedin.com/in/mamund

[Kevin Swiber]

Just throwing in a simple +1. :)

The write model may differ from the read model.

Sent from my iPhone

[Mike Kelly]

This all gets a lot simpler if you think about PUT as any idempotent update, and stop trying to distinguish "partial" from "non-partial".

Mike

http://twitter.com/mikekelly85
http://github.com/mikekelly
http://linkedin.com/in/mikekelly123

[Pedro Santos]

I have several times used PUT in this way, but if we consider PUT as an idempotent update, then how would one actually remove a property from a representation in a "standard" HTTP way?

Not making judgements on whether that makes sense as a use case (as I said up until know every single API I've come across that uses PUT uses it as an idempotent update), but if we want to differentiate between a null property and a non-existing property isn't "full replace" PUT the only way to remove properties from a resource representation?

[Mike Kelly]

You can expose another resource and allow the client to DELETE it. You are right though, it does make it awkward to do property removal with PUT... it's a trade-off.

[Owen Rubel]

You should be doing I/O checking on input data to the server and output data back to the client. And you can reject on too much input/output or just 'wipe' it