Proposal for 1.0: Cleanup of response structure in Data Pipeline

[goosemanjack]

Synopsis:

Data responses to all data pipeline tags must be clarified and aligned
with matching REST response. Special case tags <os:ViewerRequest>,
<os:OwnerRequest>, and <os:HttpRequest> should also be clarified.


In the errata wiki there is already reference to fixing text about the
response type. The text, however, appears to be in error. I have
added some text to flag the issue.

http://wiki.opensocial.org/index.php?title=Errata_in_v0.9#Clarify_return_type_for_os:ActivityRequest_and_os:PeopleRequest

The original text stated the response should follow the RPC spec
response format, when in fact it should follow the REST spec response
format. This flags another issue about the format mis-match between
the REST and RPC specs, which I'll start another thread on.

Because the data pipeline tags are supposed to be an abstraction on
top of the REST endpoints, the REST response format should be used.
We should also verify if the <os:Viewer> and <os:Owner> tags should
follow this format or be treated as special-case tags where only the
Person JSON data is returned. Currently we return the raw Person JSON
object, but could change to align with the wrapping response object as
defined in the REST spec:

http://www.opensocial.org/Technical-Resources/opensocial-spec-v09/REST-API.html#rfc.section.3.1

[Lane LiaBraaten]

I think the fact the the Overview references 'REST' is really a spec bug (i.e. it should be 'RPC').  Building data pipelining on top of REST is not what was intended.

Here's another spec bug from the <os:DataRequest> section [2]:

@method {string} The name of the REST endpoint and operation called. Implementations MUST support "people.get" and "activities.get", and MUST NOT support ".update", ".create", or ".delete" operations.

Clearly, we mean RPC here.

The proxied content section [1] does explicitly reference the RPC spec. A gadget may contain multiple data pipelining tags, which are combined and send to the developer's server using RPC batching. Without batching in REST,  how would we support multiple DP tags (and be able to correlate each responses with the proper tag/request?

So +1 for cleaning up these typos, but -1 on try to make DP work w/ REST in this iteration (certainly could be an extension if you want to explore the idea).

-Lane

[1] http://www.opensocial.org/Technical-Resources/opensocial-spec-v09/OpenSocial-Data-Pipelining.html#rfc.section.10

[2] http://www.opensocial.org/Technical-Resources/opensocial-spec-v09/OpenSocial-Data-Pipelining.html#DataRequest

[goosemanjack]

IMO it would be difficult to make the case that RPC would be the
underlying format instead of REST given that as of now RPC is a SHOULD
not a MUST. REST, I believe, is a MUST at this point (haven't looked
it up). Regardless, this whole problem goes away if we can align the
minor differences in response formats between RPC and REST response
formats.
--
clc

On Oct 7, 6:33 pm, Lane LiaBraaten <lliab...@google.com> wrote:
> I think the fact the the Overview references 'REST' is really a spec bug
> (i.e. it should be 'RPC').  Building data pipelining on top of REST is not
> what was intended.
>
> Here's another spec bug from the <os:DataRequest> section [2]:
>
> @method {string} The name of the REST endpoint and operation called.
> Implementations MUST support "people.get" and "activities.get", and MUST NOT
> support ".update", ".create", or ".delete" operations.
>
> Clearly, we mean RPC here.
>
> The proxied content section [1] does explicitly reference the RPC spec. A
> gadget may contain multiple data pipelining tags, which are combined and
> send to the developer's server using RPC batching. Without batching in
> REST,  how would we support multiple DP tags (and be able to correlate each
> responses with the proper tag/request?
>
> So +1 for cleaning up these typos, but -1 on try to make DP work w/ REST in
> this iteration (certainly could be an extension if you want to explore the
> idea).
>
> -Lane
>

> [1]http://www.opensocial.org/Technical-Resources/opensocial-spec-v09/Ope...
>
> [2]http://www.opensocial.org/Technical-Resources/opensocial-spec-v09/Ope...

>
> On Wed, Oct 7, 2009 at 4:36 PM, goosemanjack <cc...@myspace.com> wrote:
>
> > Synopsis:
>
> > Data responses to all data pipeline tags must be clarified and aligned
> > with matching REST response.  Special case tags <os:ViewerRequest>,
> > <os:OwnerRequest>, and <os:HttpRequest> should also be clarified.
>
> > In the errata wiki there is already reference to fixing text about the
> > response type.  The text, however, appears to be in error.  I have
> > added some text to flag the issue.
>

> >http://wiki.opensocial.org/index.php?title=Errata_in_v0.9#Clarify_ret...

>
> > The original text stated the response should follow the RPC spec
> > response format, when in fact it should follow the REST spec response
> > format.  This flags another issue about the format mis-match between
> > the REST and RPC specs, which I'll start another thread on.
>
> > Because the data pipeline tags are supposed to be an abstraction on
> > top of the REST endpoints, the REST response format should be used.
> > We should also verify if the <os:Viewer> and <os:Owner> tags should
> > follow this format or be treated as special-case tags where only the
> > Person JSON data is returned.  Currently we return the raw Person JSON
> > object, but could change to align with the wrapping response object as
> > defined in the REST spec:
>

> >http://www.opensocial.org/Technical-Resources/opensocial-spec-v09/RES...

[Lane LiaBraaten]

On Wed, Oct 7, 2009 at 8:38 PM, goosemanjack <cc...@myspace.com> wrote:

IMO it would be difficult to make the case that RPC would be the
underlying format instead of REST given that as of now RPC is a SHOULD
not a MUST.  REST, I believe, is a MUST at this point (haven't looked

it up).  Regardless, this whole problem goes away if we can align the
minor differences in response formats between RPC and REST response
formats.

I don't think that quite solves it.

Suppose I have two data piplining request for collections of people, say the owner's friends and the viewer's friends.  It's simple to send the two results in one request to the developer's sever using RPC batching - each is keyed with an id.

With REST, there's no way to send the ID over the wire, and no way to send multiple responses in a since request to the developer's server.

-Lane

[goosemanjack]

Client-side processing of DataPipeline (DP) tags is not mandated by
the spec. Reprocessing of existing previously registered data
pipeline tags is also not currently part of the spec (tho previously
proposed and flagged for re-proposal in 1.1). While it is true that
it would be more efficient to batch and send these requests as a
single unit from the client via a single XHR request, that is not
really a scenario we face currently. MySpace (and I believe Shindig
on Orkut) processes all the os:* data pipeline requests that sit in
front of REST API endpoints server-side. Since this is server-
processed it is not necessary to use JSON-RPC in any form to attain
the optimizations of an atomic request.

Additionally, RPC support is currently defined as a MAY, not a MUST,
so there is no guarantee that RPC will be available at all on a given
implementation. REST, on the other hand, is a MUST, so we know it
will be available. While the Javascript API supports a batching-
syntax, there is no requirement that the requests actually be
batched. In fact, all current MySpace implementations send the
requests individually to the REST endpoints for processing, then
reconstruct the result.

> Building data pipelining on top of REST is not what was intended.

Actually, it is what was intended when MySpace originally proposed the
tag language and data pipelining ages ago. As people tell me (it was
before my time at MySpace), data pipelining was proposed prior to the
original Javascript API. It may be that in conversations over at
Google the DP proposal was looked at thru the RPC lens, but that was
never the case at MySpace, so I don't think the above statement can be
made.

I believe we're in agreement in principle that the data format
specified in the various DP tags (ex: array of JSON objects) is not
accurate. The question is if the REST wrapper format or the RPC
wrapper format is the correct format to use when registering the
result with the client-side data context.

I see two distinct issues right now:

1. Data Pipeline tags do not have the correct return format identified
in the spec.

2. RPC and REST specs have subtle differences in the meta-format for
responses
(identified also in this thread:
http://groups.google.com/group/opensocial-and-gadgets-spec/browse_thread/thread/6dec034bf9d99f20/db8a98a0ca3eda22?hl=en
)

If we can resolve the meta-format inconsistencies between REST and
RPC, the debate over what the underlying endpoints are for
DataPipeline tags becomes moot.
--
clc

[Adam Winer]

On Thu, Oct 8, 2009 at 3:19 PM, goosemanjack <cc...@myspace.com> wrote:
>
> Client-side processing of DataPipeline (DP) tags is not mandated by
> the spec.  Reprocessing of existing previously registered data
> pipeline tags is also not currently part of the spec (tho previously
> proposed and flagged for re-proposal in 1.1).  While it is true that
> it would be more efficient to batch and send these requests as a
> single unit from the client via a single XHR request, that is not
> really a scenario we face currently.  MySpace (and I believe Shindig
> on Orkut) processes all the os:* data pipeline requests that sit in
> front of REST API endpoints server-side.

Shindig supports both server-side and client-side evaluation.

> Since this is server-
> processed it is not necessary to use JSON-RPC in any form to attain
> the optimizations of an atomic request.
>
> Additionally, RPC support is currently defined as a MAY, not a MUST,
> so there is no guarantee that RPC will be available at all on a given
> implementation.

Support for an RPC endpoint and protocol is a MAY. But the use of the
RPC *format* for proxied rendering is a MUST, and at least some of the
details of data pipelining in 0.9 were intended to refer to the RPC
format (e.g. the @method params listed above as Lane said, the format
of os:HttpRequest data).

[Evan Gilbert]

On Wed, Oct 7, 2009 at 4:36 PM, goosemanjack <cc...@myspace.com> wrote:

Synopsis:

Data responses to all data pipeline tags must be clarified and aligned
with matching REST response.  Special case tags <os:ViewerRequest>,
<os:OwnerRequest>, and <os:HttpRequest> should also be clarified.


In the errata wiki there is already reference to fixing text about the
response type.  The text, however, appears to be in error.  I have
added some text to flag the issue.

http://wiki.opensocial.org/index.php?title=Errata_in_v0.9#Clarify_return_type_for_os:ActivityRequest_and_os:PeopleRequest

The original text stated the response should follow the RPC spec
response format, when in fact it should follow the REST spec response
format.  This flags another issue about the format mis-match between
the REST and RPC specs, which I'll start another thread on.

If we align REST & RPC JSON responses, then is this issue closed? I'd rather not discuss whether it should use the JSON-REST or the JSON-RPC format if both should be the same.

 

Because the data pipeline tags are supposed to be an abstraction on
top of the REST endpoints, the REST response format should be used.
We should also verify if the <os:Viewer> and <os:Owner> tags should
follow this format or be treated as special-case tags where only the
Person JSON data is returned.  Currently we return the raw Person JSON
object, but could change to align with the wrapping response object as
defined in the REST spec:

http://www.opensocial.org/Technical-Resources/opensocial-spec-v09/REST-API.html#rfc.section.3.1

--

[Evan Gilbert]

On Thu, Oct 8, 2009 at 5:22 PM, Adam Winer <awi...@gmail.com> wrote:

On Thu, Oct 8, 2009 at 3:19 PM, goosemanjack <cc...@myspace.com> wrote:
>
> Client-side processing of DataPipeline (DP) tags is not mandated by
> the spec.  Reprocessing of existing previously registered data
> pipeline tags is also not currently part of the spec (tho previously
> proposed and flagged for re-proposal in 1.1).  While it is true that
> it would be more efficient to batch and send these requests as a
> single unit from the client via a single XHR request, that is not
> really a scenario we face currently.  MySpace (and I believe Shindig
> on Orkut) processes all the os:* data pipeline requests that sit in
> front of REST API endpoints server-side.

Shindig supports both server-side and client-side evaluation.

> Since this is server-
> processed it is not necessary to use JSON-RPC in any form to attain
> the optimizations of an atomic request.
>
> Additionally, RPC support is currently defined as a MAY, not a MUST,
> so there is no guarantee that RPC will be available at all on a given
> implementation.

Support for an RPC endpoint and protocol is a MAY.  But the use of the
RPC *format* for proxied rendering is a MUST, and at least some of the
details of data pipelining in 0.9 were intended to refer to the RPC
format (e.g. the @method params listed above as Lane said, the format
of os:HttpRequest data).

Proxied rendering with data pipelining data sent to the 3rd party server isn't possible without a batch format - the goal is to post (viewer, friends, activities, app data) to the developers server, to the URL specified in <Content href="...">

We had actually split out the JSON-RPC batch format into a separate spec element during the drafting process for 0.9 (see http://sites.google.com/site/opensocialdraft/Home/opensocial-batch-json-format,  https://sites.google.com/site/opensocialdraft/Home/data-pipelining), to clarify this, but seems like we folded the batch reference back into the RPC protocol for the final spec.

This doesn't address the inconsistency between the result formats for JSON-REST and JSON-RPC that Chris pointed out.