The proposal can be read at :
http://groups.google.com/group/opensocial-and-gadgets-spec/msg/f923cf8c2283562d?hl=en
(john if you have a link to a docs.google.com document, please
contribute it since the formatting is lost in this link)
Please vote +1 If you prefer the JSON-REST proposal over the JSON-RPC
proposal. Otherwise vote on the other thread.
The proposal with the most votes will become the batching format for
the 0.8.1 spec, so please vote so we can accept one of these two since
either is much better then the 0.8 one, but please also do so after
having evaluated the pro's and con's for each since we'll be using it
for a long time to come.
It looks like I need to withdraw my concern—I missed that bit. (My apologies.)
So, after reading the proposal a few more times, I’ll raise a new concern.
I think we need to update the spec to state that responses MUST maintain ordering for batched requests.
Why?
1. If id is not specified, then the results must come back in request order to allow for client side correlation.
2. If id is specified on one item but not on others, the results must come back in request order allow for client side correlation.
3. If id is specified on all items, we cannot require all ids to be unique. For example, imagine that someone passes in two items with the same id: a person and an activity. id values may have been derived from the ids in the system. To work with this possibility, we would again need to maintain ordering. Nothing prevents an implementation from deciding that a person and an activity can have the same id.
My point was this: the spec needs to require that servers match request order with response order.
I took point #3 as an example of why this would be true, even in the presence of ids.
Given your requirement, we could make order unimportant for case #3. I don’t think we should add a uniqueness constraint on ids. Every server implementation will have to handle the case where responses must come back in order when ids aren’t present. Requiring the ids to be unique means that server implementations will have to write extra code to reject a request that is otherwise fine without the ids. Because that id is only meaningful to the client, what does a server gain by validating uniqueness? I argue that the server doesn’t gain anything.
I completely agree that order should match between request batch and response batch. Order can be maintained when the result is an associative array or a regular array but if we want to use associative arrays as the result format we need to require unique id's
There are a couple of arguments for requiring unique id's
- Allows access of the batch result as an associative array using batchresult.<unique key> which means less code in general for clients
- Allows for using the output of one request as the input to a later one (subject to ordering). There have been a number of discussions on the value of joining in prior threads, the best use case being restricting app data to a paged subset of friends.
- A client can use the array of requests to do ordered iteration of an associative array of responses if needed, 'for each' style loops will also maintain iteration order on some platforms
I’m leery of the ‘unless’ below.
As you point out in a fork of this thread:
“All true, though note that the last two are really arguments for _allowing_ unique IDs to be sent from the client and requiring servers to honor them. (You get all the benefits of the last two without needing to make it required on the client.)”
And that was in response to Louis’ comment about why unique ids are valuable client side. I don’t think that a server should make assumptions about the uniqueness of the ids—just return things in order and echo back any ids. This would change the language to:
“Order of operations in the response MUST be the same as in the request. If operations have client provided ids, those ids must also be present in the same order as the requests.”
Is this OK?
I favor the JSON-REST format over the JSON-RPC format, but I would also like to see support for x-www-form-urlencoded POST/PUT requests for single entity updates. This allows us to sign the request body and I believe it what would be used in most instances of PUT/POST by developers.
Slammed w/ meetings the past couple of days, but I will follow up with more as soon as I can.
~Paul
In the light of two other items working their way through, OpenSocial Data Template and OpenSocial Data Pipelining, do we really need section 9 (request batching)? I think that Data Template and Data Pipelining offer a more elegant and appropriate solution to this problem because they open the door for evaluating scores of requests directly on the server as needed.
Further, while it would theoretically help process more requests faster, is there a real, demonstrated need for the batching feature?
Thanks for explaining this. You’re right that if we are going to have this, we might as well be consistent across implementations. Thanks!
Can you provide some exact use cases of batching from the gadget js? Could not one resource URI be created that represents the multiple being batched? For example, if there is a desire to batch both owner and viewer, rather than some complex form of batching multiple resources, couldn’t we create one resource URI that represented both (e.g. /people/owner,viewer?format=json)
~Paul
Our current version of our API supports multiple users simply by allowing the consumer to delimit the ids with semicolons (/users/{userid};{userid};{userid}…
). Couldn’t we do the same thing with this spec? This is what Sam Ruby recommends after all ;-).
Naturally there are a number n of entities on any non-trivial API and as n grows there are more permutations, but the argument makes the large leap that all permutations need to be supported by the API. I would argue just the opposite, that a permutation should not be supported unless a good use case for it exists, at which time, a resource should be defined for it.
Generic batching punts the responsibility of properly defining our resources (much like the fields parameters) and gives the consumer assumption that any requests can be batched even those that don’t make sense.
~Paul
I think what we are rapidly approaching here, is settling on a hybrid
approach.
There are a lot of people excited about the RESTful interface, but
there also those who think a RPC interface is a far better solution
and neither camp has been able to pull the other to their side in the
last 2 weeks.
As such currently it looks like we'll have both, a classic REST
interface for single requests, and a RPC interface too, which while it
can be construed as confusing, does cater to the desires of both sides
and does give the app developers the choice to which they prefer; Just
as long as they don't want to batch REST requests :)
So baring any -1's in the next 24 hours, i think we're ready to call
this a compromise and go for the (single request) RESTful service, and
a JSON-RPC service next to that.
-- Chris
So, how do all of the other google apis get away without a batching mechanism?
Using a comma delimited list to separate identifiers on a list of entities in a URI is not meant to be a solution for batching GETs across entity types, but it is a better solution for batching gets of multiple records of the same entity type. And this is exactly what I’m afraid of with this generic batching proposal, is that it is going be referred to and used when at times it would be better to ensure we properly define resources (and their response representations)
I concede that this batching proposal would be needed for if the desire is to get multiple resources of different types, but I would also argue that if a particular resource (such as an activity or a list of activities) always has the context of a user associated with the activities, then we could simply change the definition of the representation to include the user. It’s very common to include the related parent entity when a request is made for a child in a hierarchal relationship.
I’m afraid most of the time in implementing this generic batching is going to be spent handling the combinations that should not be allowed. Consideration of this should be considered outside of the context of Shindig and a particular containers implementation, but that’s a bit late I suppose.
I want to do an update, followed by a retrieval of some resource that might have been affected by the update either directly or indirectly. I therefore need to do the two in sequence (can't parallelize) and there's not necessarily an obvious composite resource to be dealt with. A somewhat fanciful example here would be updating one's own geographic position, then retrieving the list of friends within five miles of one's current position. I don't want to incur the cost of two HTTP round trips to do this.
Louis, Evan, Scott and I had a long discussion re: batching.
I do some use cases where batching is useful, but my main concern is that I want to ensure that we don’t default to batching requests as a solution when a resource could define the operation. Also, for GET requests of a list of entities of the same type, if we can create a resource that takes a delimited list of ids, I would prefer that (as discussed in another thread). Also, I don’t see an issue with defining the response body to a POST/PUT to include the updated resource (along with the location header on a POST).
So, I’m not opposed to batching (and see some value) as long as we do our due diligence and ask ourselves, “Could we define this operation as one resource?” As far as the semantics of REST vs RPC, I still lean toward the former, with the understanding that it may be a tad more difficult to parse the parameters out of the body of the entity.
We discussed support for x-www-form-urlencoded and mutipart/form-data for single entity PUT/POST as well. I’ll start another thread specific to that.
+1 for JSON-RPC
Couple of thoughts:
a) I think we are talking about a heterogeneous batching viz: composite data
b) If so, would it be a good idea to embed the operation in the batch separator ? (I think that is what Paul is talking about)
c) Agree with Paul that we should try to use REST semantics as far (and much) as possible
d) More import question is “How does REST handle heterogeneous batch operations ?”. Do we need to extend REST ?
e) Would be happy to participate in an f2f form (or visit cubicles ;o)) if needed.
Cheers
<k/>
A MAY means an implementation is free to do something (and for
compatibility, everybody else needs to be aware of the possiblility).
A SHOULD means that an implementation has to implement something
unless there's a really good reason not to. Things like security
problems or major performance issues where the community agrees to
ignore the standard for example.
MAY is what we are stating in this thread. We are not even in the spirit of a ‘SHOULD’. Please understand that RFC2119 means some very specific things. This isn’t like “you should brush and floss daily”. We are clearly demonstrating the behavior of a ‘MAY’. ShinDig thinks this will help, MySpace says ‘no.’ That’s a MAY, according to RFC2119.
From http://tools.ietf.org/html/rfc2119:
3. SHOULD This word, or the adjective "RECOMMENDED", mean that there
may exist valid reasons in particular circumstances to ignore a
particular item, but the full implications must be understood and
carefully weighed before choosing a different course.
5. MAY This word, or the adjective "OPTIONAL", mean that an item is
truly optional. One vendor may choose to include the item because a
particular marketplace requires it or because the vendor feels that
it enhances the product while another vendor may omit the same item.
An implementation which does not include a particular option MUST be
prepared to interoperate with another implementation which does
include the option, though perhaps with reduced functionality. In the
same vein an implementation which does include a particular option
MUST be prepared to interoperate with another implementation which
does not include the option (except, of course, for the feature the
option provides.)
Scott Seely |
architect |
email sse...@myspace.com |
Agreed. OPTIONAL is MAY. Remember, a time will come when we need to worry about conformance et al and at that time this would be important.
Cheers
<k/>
From:
opensocial-an...@googlegroups.com
[mailto:opensocial-an...@googlegroups.com] On Behalf Of Scott
Seely
Sent: Wednesday, September 03, 2008 3:28 PM
To: opensocial-an...@googlegroups.com
Subject: RE: [VOTE] JSON-REST batching format
MAY is what we are stating in this thread. We are not even in the spirit of a ‘SHOULD’. Please understand that RFC2119 means some very specific things. This isn’t like “you should brush and floss daily”. We are clearly demonstrating the behavior of a ‘MAY’. ShinDig thinks this will help, MySpace says ‘no.’ That’s a MAY, according to RFC2119.
+1 on MAY
+1 on MAY also